2024-09-18T09:57:40.501 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T09:57:40.524 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T09:57:40.611 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} email: skanta@redhat.com first_in_suite: false job_id: '7910840' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T09:57:40.611 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T09:57:40.612 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T09:57:40.612 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T09:57:40.614 INFO:teuthology.task.internal:Checking packages... 2024-09-18T09:57:40.636 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T09:57:40.636 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T09:57:40.636 INFO:teuthology.packaging:ref: None 2024-09-18T09:57:40.636 INFO:teuthology.packaging:tag: None 2024-09-18T09:57:40.636 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T09:57:40.636 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T09:57:40.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T09:57:40.779 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T09:57:40.781 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T09:57:40.787 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T09:57:40.788 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T09:57:40.830 INFO:teuthology.task.internal:Saving configuration 2024-09-18T09:57:40.844 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T09:57:40.864 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T09:57:40.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 09:51:27.527262', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T09:57:40.909 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 09:51:27.528321', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T09:57:40.909 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T09:57:40.916 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-18T09:57:40.916 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-18T09:57:40.916 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T09:57:41.007 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd81da7bac0>, signals=[15]) 2024-09-18T09:57:41.007 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T09:57:41.013 INFO:teuthology.task.internal:Opening connections... 2024-09-18T09:57:41.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T09:57:41.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T09:57:41.105 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T09:57:41.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T09:57:41.194 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T09:57:41.203 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-09-18T09:57:41.209 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-09-18T09:57:41.210 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T09:57:41.256 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2024-09-18T09:57:41.257 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2024-09-18T09:57:41.257 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-09-18T09:57:41.278 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-09-18T09:57:41.285 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-09-18T09:57:41.285 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="20.04" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T09:57:41.335 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T09:57:41.336 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T09:57:41.336 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=focal 2024-09-18T09:57:41.336 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=focal 2024-09-18T09:57:41.336 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-09-18T09:57:41.361 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T09:57:41.369 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T09:57:41.420 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T09:57:41.420 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-09-18T09:57:41.423 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-09-18T09:57:41.429 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T09:57:41.454 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T09:57:41.454 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-09-18T09:57:41.471 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-09-18T09:57:41.495 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T09:57:41.575 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T09:57:41.589 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T09:57:41.589 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T09:57:41.589 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T09:57:41.590 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T09:57:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T09:57:41.595 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-09-18T09:57:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T09:57:41.596 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-18T09:57:41.602 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-124-generic 2024-09-18T09:57:41.602 INFO:teuthology.task.kernel:Running kernel on smithi142: 5.4.0-124-generic 2024-09-18T09:57:41.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2024-09-18T09:57:41.643 INFO:teuthology.orchestra.run.smithi150.stdout:5.4.0-124-generic 2024-09-18T09:57:41.643 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.4.0-124-generic 2024-09-18T09:57:41.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-09-18T09:57:41.749 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-18T09:57:41.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-18T09:57:41.944 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T09:57:42.020 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T09:57:42.034 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T09:57:42.048 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T09:57:42.084 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T09:57:42.129 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T09:57:42.238 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T09:57:42.332 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T09:57:42.374 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T09:57:42.378 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T09:57:42.378 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T09:57:42.398 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T09:57:42.398 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T09:57:42.402 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T09:57:42.402 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T09:57:42.409 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T09:57:42.413 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T09:57:42.414 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T09:57:42.415 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T09:57:42.415 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T09:57:42.426 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T09:57:42.427 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T09:57:42.438 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T09:57:42.482 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T09:57:42.544 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T09:57:42.623 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T09:57:42.684 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T09:57:42.870 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T09:57:42.911 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T09:57:43.038 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T09:57:43.050 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T09:57:43.051 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T09:57:43.104 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T09:57:43.118 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T09:57:43.119 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T09:57:43.146 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T09:57:43.147 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T09:57:43.155 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T09:57:43.166 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T09:57:43.166 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T09:57:43.171 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T09:57:43.181 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T09:57:43.181 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T09:57:43.183 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T09:57:43.195 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T09:57:43.213 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T09:57:43.213 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T09:57:43.218 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T09:57:43.222 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T09:57:43.222 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T09:57:43.227 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T09:57:43.228 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T09:57:43.228 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T09:57:43.251 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T09:57:43.271 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T09:57:43.273 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T09:57:43.278 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T09:57:43.278 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T09:57:43.279 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T09:57:43.279 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T09:57:43.280 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T09:57:43.287 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T09:57:43.288 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T09:57:43.288 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T09:57:43.289 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T09:57:43.290 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T09:57:43.336 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T09:57:43.347 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T09:57:43.351 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T09:57:43.352 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T09:57:43.352 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T09:57:43.353 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T09:57:43.353 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T09:57:43.353 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T09:57:43.368 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T09:57:43.369 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T09:57:43.370 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T09:57:43.417 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T09:57:43.418 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T09:57:43.418 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T09:57:43.502 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T09:57:43.502 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T09:57:43.503 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T09:57:43.503 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T09:57:46.794 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 22.0 MB in 3s (7,455 kB/s) 2024-09-18T09:57:47.378 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 22.0 MB in 4s (6,064 kB/s) 2024-09-18T09:57:48.400 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T09:57:48.425 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T09:57:48.501 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T09:57:48.721 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-18T09:57:48.722 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-18T09:57:48.925 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2024-09-18T09:57:48.925 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T09:57:48.925 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T09:57:48.925 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T09:57:48.926 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2024-09-18T09:57:48.926 INFO:teuthology.orchestra.run.smithi142.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T09:57:48.927 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2024-09-18T09:57:48.927 INFO:teuthology.orchestra.run.smithi142.stdout: thermald 2024-09-18T09:57:48.948 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T09:57:48.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T09:57:48.985 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2024-09-18T09:57:48.985 INFO:teuthology.orchestra.run.smithi142.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T09:57:48.985 INFO:teuthology.orchestra.run.smithi142.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T09:57:48.986 INFO:teuthology.orchestra.run.smithi142.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T09:57:48.987 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2024-09-18T09:57:48.988 INFO:teuthology.orchestra.run.smithi142.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T09:57:49.048 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T09:57:49.170 INFO:teuthology.orchestra.run.smithi142.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T09:57:49.170 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 77.2 MB of archives. 2024-09-18T09:57:49.171 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T09:57:49.171 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T09:57:49.266 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-18T09:57:49.267 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-18T09:57:49.468 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-09-18T09:57:49.468 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T09:57:49.468 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T09:57:49.468 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T09:57:49.469 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-09-18T09:57:49.470 INFO:teuthology.orchestra.run.smithi150.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T09:57:49.470 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2024-09-18T09:57:49.470 INFO:teuthology.orchestra.run.smithi150.stdout: thermald 2024-09-18T09:57:49.526 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-09-18T09:57:49.526 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T09:57:49.526 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T09:57:49.527 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T09:57:49.528 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2024-09-18T09:57:49.529 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T09:57:49.583 INFO:teuthology.orchestra.run.smithi150.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T09:57:49.583 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 77.2 MB of archives. 2024-09-18T09:57:49.583 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T09:57:49.583 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T09:57:49.925 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T09:57:50.022 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T09:57:50.328 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T09:57:50.557 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T09:57:50.557 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T09:57:50.558 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T09:57:50.597 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T09:57:50.653 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T09:57:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T09:57:51.383 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 77.2 MB in 1s (68.4 MB/s) 2024-09-18T09:57:51.560 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T09:57:51.909 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T09:57:51.909 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T09:57:51.909 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T09:57:52.189 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T09:57:52.242 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T09:57:52.942 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 77.2 MB in 3s (23.8 MB/s) 2024-09-18T09:57:53.122 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T09:57:53.284 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T09:57:53.289 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:53.407 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:57:54.858 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T09:57:54.863 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:54.943 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:57:55.407 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T09:57:55.423 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:55.514 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:57:56.627 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T09:57:56.641 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:56.685 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:57:56.792 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T09:57:56.809 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:56.904 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:57:57.945 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T09:57:57.961 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:57:58.003 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:01.455 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:01.607 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:01.952 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:02.094 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:02.430 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T09:58:02.443 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T09:58:02.480 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T09:58:02.760 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:02.912 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:03.282 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:03.407 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:03.735 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T09:58:03.748 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T09:58:03.793 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T09:58:08.801 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T09:58:08.824 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:58:08.868 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:09.090 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T09:58:09.112 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T09:58:09.157 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:11.660 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:11.798 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:11.801 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T09:58:11.922 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T09:58:12.156 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T09:58:12.252 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:12.265 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T09:58:12.393 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:12.395 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T09:58:12.529 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:12.777 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T09:58:12.911 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:13.748 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:14.131 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:14.905 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:14.905 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:15.075 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:15.321 INFO:teuthology.orchestra.run.smithi142.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:15.458 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:16.268 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T09:58:16.394 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T09:58:16.521 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:16.592 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T09:58:16.593 INFO:teuthology.orchestra.run.smithi150.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:16.634 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T09:58:16.760 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T09:58:16.895 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T09:58:16.977 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T09:58:16.977 INFO:teuthology.orchestra.run.smithi142.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:38.728 INFO:teuthology.orchestra.run.smithi142.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T09:58:38.728 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub' 2024-09-18T09:58:38.742 INFO:teuthology.orchestra.run.smithi142.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T09:58:38.762 INFO:teuthology.orchestra.run.smithi142.stdout:Generating grub configuration file ... 2024-09-18T09:58:38.909 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T09:58:38.909 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub' 2024-09-18T09:58:38.924 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T09:58:38.942 INFO:teuthology.orchestra.run.smithi150.stdout:Generating grub configuration file ... 2024-09-18T09:58:39.106 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:39.126 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:39.271 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:39.307 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:39.507 INFO:teuthology.orchestra.run.smithi142.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T09:58:39.519 INFO:teuthology.orchestra.run.smithi142.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T09:58:39.704 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T09:58:39.716 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T09:58:41.990 INFO:teuthology.orchestra.run.smithi142.stdout:done 2024-09-18T09:58:42.205 INFO:teuthology.orchestra.run.smithi150.stdout:done 2024-09-18T09:58:42.435 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2024-09-18T09:58:42.489 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2024-09-18T09:58:42.489 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2024-09-18T09:58:42.489 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2024-09-18T09:58:42.489 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2024-09-18T09:58:42.489 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.196.194 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2024-09-18T09:58:42.490 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2024-09-18T09:58:42.491 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T09:58:42.491 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T09:58:42.491 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T09:58:42.491 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T09:58:42.491 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi142.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T09:58:42.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get clean 2024-09-18T09:58:42.570 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-18T09:58:42.631 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-09-18T09:58:42.690 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-09-18T09:58:42.690 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 20 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.4.0.196.194 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-09-18T09:58:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-09-18T09:58:42.692 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T09:58:42.692 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T09:58:42.692 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T09:58:42.692 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-18T09:58:42.692 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T09:58:42.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-09-18T09:58:42.720 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T09:58:42.726 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T09:58:42.743 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T09:58:42.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-18T09:58:42.837 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T09:58:42.888 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T09:58:42.893 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T09:58:42.911 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T09:58:43.006 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T09:58:44.603 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T09:58:44.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T09:58:44.700 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T09:58:44.733 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T09:58:44.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T09:58:44.829 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T09:58:44.915 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-18T09:58:44.915 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-18T09:58:45.025 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-18T09:58:45.025 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-18T09:58:45.147 INFO:teuthology.orchestra.run.smithi142.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T09:58:45.148 INFO:teuthology.orchestra.run.smithi142.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T09:58:45.149 DEBUG:teuthology.orchestra.run.smithi142:> dpkg -s linux-image-generic 2024-09-18T09:58:45.170 INFO:teuthology.orchestra.run.smithi142.stdout:Package: linux-image-generic 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Status: install ok installed 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Priority: optional 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Section: kernel 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Installed-Size: 20 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Architecture: amd64 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Source: linux-meta 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Version: 5.4.0.196.194 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T09:58:45.171 INFO:teuthology.orchestra.run.smithi142.stdout:Recommends: thermald 2024-09-18T09:58:45.172 INFO:teuthology.orchestra.run.smithi142.stdout:Description: Generic Linux kernel image 2024-09-18T09:58:45.172 INFO:teuthology.orchestra.run.smithi142.stdout: This package will always depend on the latest generic kernel image 2024-09-18T09:58:45.172 INFO:teuthology.orchestra.run.smithi142.stdout: available. 2024-09-18T09:58:45.172 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-09-18T09:58:45.180 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.wcsNgICJMJ 2024-09-18T09:58:45.181 DEBUG:teuthology.orchestra.run.smithi142:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wcsNgICJMJ 2024-09-18T09:58:45.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.wcsNgICJMJ 2024-09-18T09:58:45.257 INFO:teuthology.orchestra.run.smithi150.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T09:58:45.257 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T09:58:45.259 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-09-18T09:58:45.275 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.wcsNgICJMJ is 9KB 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 20 2024-09-18T09:58:45.281 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.4.0.196.194 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-09-18T09:58:45.282 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-09-18T09:58:45.283 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-09-18T09:58:45.287 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.dzTXgP8O5a 2024-09-18T09:58:45.287 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub/grub.cfg /tmp/tmp.dzTXgP8O5a 2024-09-18T09:58:45.290 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.wcsNgICJMJ 2024-09-18T09:58:45.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.dzTXgP8O5a 2024-09-18T09:58:45.345 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.dzTXgP8O5a is 9KB 2024-09-18T09:58:45.352 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T09:58:45.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.361 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.dzTXgP8O5a 2024-09-18T09:58:45.367 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.388 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T09:58:45.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo update-grub 2024-09-18T09:58:45.424 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T09:58:45.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T09:58:45.455 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T09:58:45.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo update-grub 2024-09-18T09:58:46.729 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub' 2024-09-18T09:58:46.742 INFO:teuthology.orchestra.run.smithi142.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T09:58:46.758 INFO:teuthology.orchestra.run.smithi142.stderr:Generating grub configuration file ... 2024-09-18T09:58:46.895 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub' 2024-09-18T09:58:46.915 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T09:58:46.929 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-09-18T09:58:46.974 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:46.986 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:47.200 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T09:58:47.212 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T09:58:47.379 INFO:teuthology.orchestra.run.smithi142.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T09:58:47.391 INFO:teuthology.orchestra.run.smithi142.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T09:58:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T09:58:47.608 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T09:58:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:done 2024-09-18T09:58:48.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo shutdown -r now 2024-09-18T09:58:49.087 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-09-18T09:58:49.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-09-18T09:59:18.791 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T09:59:18.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T09:59:18.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T09:59:19.092 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-18T09:59:19.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-18T09:59:19.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T09:59:53.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-09-18T10:00:02.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T10:00:02.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:00:19.109 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T10:00:20.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-09-18T10:00:28.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-18T10:00:28.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:00:32.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T10:00:32.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:00:33.040 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-09-18T10:00:33.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T10:00:33.682 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T10:00:33.682 DEBUG:teuthology.orchestra.run.smithi142:> uname -r 2024-09-18T10:00:33.729 INFO:teuthology.orchestra.run.smithi142.stdout:5.4.0-196-generic 2024-09-18T10:00:33.729 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T10:00:33.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T10:00:33.729 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T10:00:34.730 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T10:00:34.730 DEBUG:teuthology.orchestra.run.smithi142:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T10:00:34.834 INFO:teuthology.orchestra.run.smithi142.stdout:ttyS1 2024-09-18T10:00:34.865 DEBUG:teuthology.parallel:result is None 2024-09-18T10:00:36.707 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-18T10:00:37.771 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-18T10:00:37.771 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-196-generic"... 2024-09-18T10:00:37.771 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-18T10:00:37.777 INFO:teuthology.orchestra.run.smithi150.stdout:5.4.0-196-generic 2024-09-18T10:00:37.777 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T10:00:37.777 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T10:00:37.778 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T10:00:38.778 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-18T10:00:38.779 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T10:00:38.947 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-09-18T10:00:38.979 DEBUG:teuthology.parallel:result is None 2024-09-18T10:00:38.979 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T10:00:38.987 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T10:00:38.987 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T10:00:38.990 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T10:00:38.997 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T10:00:39.015 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T10:00:39.049 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T10:00:39.049 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T10:00:39.052 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T10:00:39.080 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T10:00:39.086 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T10:00:39.086 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T10:00:39.089 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T10:00:39.109 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:00:39.119 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:00:39.122 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:00:39.133 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:00:39.134 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T10:00:39.141 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T10:00:39.141 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T10:00:39.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T10:00:39.191 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T10:00:39.200 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T10:00:39.200 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T10:00:39.202 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T10:00:39.208 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T10:00:39.257 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T10:00:39.264 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:00:39.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T10:00:39.292 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T10:00:39.299 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T10:00:39.347 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:00:39.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T10:00:39.491 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-09-18T10:00:39.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-09-18T10:00:39.593 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T10:00:39.600 INFO:teuthology.task.internal:Starting timer... 2024-09-18T10:00:39.600 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T10:00:39.638 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T10:00:39.668 DEBUG:teuthology.task.selinux:Excluding smithi142: OS 'ubuntu' does not support SELinux 2024-09-18T10:00:39.668 DEBUG:teuthology.task.selinux:Excluding smithi150: OS 'ubuntu' does not support SELinux 2024-09-18T10:00:39.669 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T10:00:39.669 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T10:00:39.669 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T10:00:39.669 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T10:00:39.701 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T10:00:39.713 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T10:00:39.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi142.front.sepia.ceph.com,smithi150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T10:06:16.173 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi142.front.sepia.ceph.com'), Remote(name='ubuntu@smithi150.front.sepia.ceph.com')] 2024-09-18T10:06:16.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T10:06:16.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:06:16.282 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-09-18T10:06:16.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-09-18T10:06:16.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-18T10:06:16.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:06:16.502 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-18T10:06:16.626 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-18T10:06:16.627 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T10:06:16.637 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T10:06:16.638 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T10:06:16.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T10:06:16.641 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T10:06:16.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T10:06:16.674 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T10:06:16.674 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Command line: ntpd -gq 2024-09-18T10:06:16.675 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: proto: precision = 0.164 usec (-22) 2024-09-18T10:06:16.675 INFO:teuthology.orchestra.run.smithi142.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:06:16.676 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:06:16.676 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:06:16.676 INFO:teuthology.orchestra.run.smithi142.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen normally on 3 enp3s0f1 172.21.15.142:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen normally on 4 lo [::1]:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e0d%5]:123 2024-09-18T10:06:16.677 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:16 ntpd[13739]: Listening on routing socket on fd #22 for interface updates 2024-09-18T10:06:16.702 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T10:06:16.702 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Command line: ntpd -gq 2024-09-18T10:06:16.703 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: proto: precision = 0.092 usec (-23) 2024-09-18T10:06:16.703 INFO:teuthology.orchestra.run.smithi150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:06:16.704 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:06:16.704 INFO:teuthology.orchestra.run.smithi150.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:06:16.705 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:06:16.705 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T10:06:16.705 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T10:06:16.705 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T10:06:16.705 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen normally on 3 enp3s0f1 172.21.15.150:123 2024-09-18T10:06:16.706 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen normally on 4 lo [::1]:123 2024-09-18T10:06:16.706 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e35%5]:123 2024-09-18T10:06:16.706 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:16 ntpd[13789]: Listening on routing socket on fd #22 for interface updates 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:17.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:17 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:17.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:17 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:18.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:18.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:18.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:18.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:18 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:19.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:19 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:19.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:19.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:19.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:19 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:20 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:20.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:20.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:20.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:20.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:20.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:20.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:20 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:21 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:21 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:22.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:22 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:22.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:22.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:22.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:22.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:22.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:22.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:22 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.675 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stdout:18 Sep 10:06:23 ntpd[13739]: ntpd: time slew +0.000609 s 2024-09-18T10:06:23.676 INFO:teuthology.orchestra.run.smithi142.stdout:ntpd: time slew +0.000609s 2024-09-18T10:06:23.677 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T10:06:23.677 INFO:teuthology.orchestra.run.smithi142.stderr:18 Sep 10:06:23 ntpd[13739]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:06:23.703 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stdout:18 Sep 10:06:23 ntpd[13789]: ntpd: time slew +0.000483 s 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stdout:ntpd: time slew +0.000483s 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T10:06:23.704 INFO:teuthology.orchestra.run.smithi150.stderr:18 Sep 10:06:23 ntpd[13789]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T10:06:23.741 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T10:06:23.741 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2024-09-18T10:06:23.742 INFO:teuthology.orchestra.run.smithi142.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.742 INFO:teuthology.orchestra.run.smithi142.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.742 INFO:teuthology.orchestra.run.smithi142.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.742 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.786 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:06:23.787 INFO:teuthology.run_tasks:Running task install... 2024-09-18T10:06:23.796 DEBUG:teuthology.task.install:project ceph 2024-09-18T10:06:23.797 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T10:06:23.797 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T10:06:23.797 INFO:teuthology.task.install:Using flavor: default 2024-09-18T10:06:23.802 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T10:06:23.802 INFO:teuthology.task.install:extra packages: [] 2024-09-18T10:06:23.803 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-key list | grep Ceph 2024-09-18T10:06:23.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-key list | grep Ceph 2024-09-18T10:06:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T10:06:23.905 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T10:06:23.963 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T10:06:23.963 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T10:06:23.964 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T10:06:23.964 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:06:23.969 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T10:06:23.970 INFO:teuthology.orchestra.run.smithi142.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T10:06:23.970 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T10:06:23.970 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:06:24.108 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T10:06:24.108 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:06:24.116 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T10:06:24.116 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:06:24.223 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:06:24.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T10:06:24.229 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:06:24.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T10:06:24.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-18T10:06:24.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo apt-get update 2024-09-18T10:06:24.435 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T10:06:24.435 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T10:06:24.442 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T10:06:24.460 INFO:teuthology.orchestra.run.smithi142.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T10:06:24.480 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T10:06:24.487 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T10:06:24.501 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T10:06:24.509 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T10:06:24.525 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T10:06:24.533 INFO:teuthology.orchestra.run.smithi142.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T10:06:24.554 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T10:06:24.559 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T10:06:24.559 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T10:06:24.559 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T10:06:24.593 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T10:06:24.629 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T10:06:24.711 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T10:06:25.095 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 28.4 kB in 1s (44.5 kB/s) 2024-09-18T10:06:25.219 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 28.4 kB in 1s (37.2 kB/s) 2024-09-18T10:06:26.249 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T10:06:26.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:06:26.345 INFO:teuthology.orchestra.run.smithi142.stdout:Reading package lists... 2024-09-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T10:06:26.407 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:06:26.479 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-18T10:06:26.562 INFO:teuthology.orchestra.run.smithi142.stdout:Building dependency tree... 2024-09-18T10:06:26.563 INFO:teuthology.orchestra.run.smithi142.stdout:Reading state information... 2024-09-18T10:06:26.671 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-18T10:06:26.672 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-18T10:06:26.759 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout:The following additional packages will be installed: 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T10:06:26.761 INFO:teuthology.orchestra.run.smithi142.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T10:06:26.762 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T10:06:26.762 INFO:teuthology.orchestra.run.smithi142.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T10:06:26.763 INFO:teuthology.orchestra.run.smithi142.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: socat xmlstarlet 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout:Suggested packages: 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T10:06:26.764 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout:Recommended packages: 2024-09-18T10:06:26.765 INFO:teuthology.orchestra.run.smithi142.stdout: btrfs-tools 2024-09-18T10:06:26.862 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T10:06:26.863 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T10:06:26.863 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T10:06:26.864 INFO:teuthology.orchestra.run.smithi150.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T10:06:26.865 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: socat xmlstarlet 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-09-18T10:06:26.866 INFO:teuthology.orchestra.run.smithi150.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2024-09-18T10:06:26.867 INFO:teuthology.orchestra.run.smithi150.stdout: btrfs-tools 2024-09-18T10:06:26.881 INFO:teuthology.orchestra.run.smithi142.stdout:The following NEW packages will be installed: 2024-09-18T10:06:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T10:06:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T10:06:26.881 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T10:06:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T10:06:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T10:06:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T10:06:26.882 INFO:teuthology.orchestra.run.smithi142.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T10:06:26.884 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T10:06:26.885 INFO:teuthology.orchestra.run.smithi142.stdout:The following packages will be upgraded: 2024-09-18T10:06:26.886 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 librbd1 2024-09-18T10:06:26.949 INFO:teuthology.orchestra.run.smithi142.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T10:06:26.949 INFO:teuthology.orchestra.run.smithi142.stdout:Need to get 201 MB of archives. 2024-09-18T10:06:26.949 INFO:teuthology.orchestra.run.smithi142.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T10:06:26.949 INFO:teuthology.orchestra.run.smithi142.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T10:06:26.985 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-09-18T10:06:26.985 INFO:teuthology.orchestra.run.smithi150.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T10:06:26.986 INFO:teuthology.orchestra.run.smithi150.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T10:06:26.987 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2024-09-18T10:06:26.988 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 librbd1 2024-09-18T10:06:27.032 INFO:teuthology.orchestra.run.smithi142.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T10:06:27.033 INFO:teuthology.orchestra.run.smithi142.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T10:06:27.033 INFO:teuthology.orchestra.run.smithi142.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T10:06:27.043 INFO:teuthology.orchestra.run.smithi142.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T10:06:27.049 INFO:teuthology.orchestra.run.smithi142.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T10:06:27.073 INFO:teuthology.orchestra.run.smithi142.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T10:06:27.073 INFO:teuthology.orchestra.run.smithi142.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T10:06:27.079 INFO:teuthology.orchestra.run.smithi142.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T10:06:27.110 INFO:teuthology.orchestra.run.smithi142.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T10:06:27.112 INFO:teuthology.orchestra.run.smithi142.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T10:06:27.118 INFO:teuthology.orchestra.run.smithi142.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T10:06:27.120 INFO:teuthology.orchestra.run.smithi142.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T10:06:27.120 INFO:teuthology.orchestra.run.smithi142.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T10:06:27.122 INFO:teuthology.orchestra.run.smithi142.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T10:06:27.123 INFO:teuthology.orchestra.run.smithi142.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T10:06:27.123 INFO:teuthology.orchestra.run.smithi142.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T10:06:27.123 INFO:teuthology.orchestra.run.smithi142.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T10:06:27.128 INFO:teuthology.orchestra.run.smithi142.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T10:06:27.137 INFO:teuthology.orchestra.run.smithi150.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T10:06:27.137 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 201 MB of archives. 2024-09-18T10:06:27.137 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T10:06:27.137 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T10:06:27.142 INFO:teuthology.orchestra.run.smithi142.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T10:06:27.143 INFO:teuthology.orchestra.run.smithi142.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T10:06:27.143 INFO:teuthology.orchestra.run.smithi142.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T10:06:27.143 INFO:teuthology.orchestra.run.smithi142.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T10:06:27.145 INFO:teuthology.orchestra.run.smithi142.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T10:06:27.145 INFO:teuthology.orchestra.run.smithi142.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T10:06:27.145 INFO:teuthology.orchestra.run.smithi142.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T10:06:27.148 INFO:teuthology.orchestra.run.smithi142.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T10:06:27.148 INFO:teuthology.orchestra.run.smithi142.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T10:06:27.162 INFO:teuthology.orchestra.run.smithi142.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T10:06:27.163 INFO:teuthology.orchestra.run.smithi142.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T10:06:27.164 INFO:teuthology.orchestra.run.smithi142.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T10:06:27.179 INFO:teuthology.orchestra.run.smithi142.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T10:06:27.179 INFO:teuthology.orchestra.run.smithi142.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T10:06:27.180 INFO:teuthology.orchestra.run.smithi142.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T10:06:27.181 INFO:teuthology.orchestra.run.smithi142.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T10:06:27.181 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T10:06:27.184 INFO:teuthology.orchestra.run.smithi142.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T10:06:27.186 INFO:teuthology.orchestra.run.smithi142.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T10:06:27.188 INFO:teuthology.orchestra.run.smithi142.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T10:06:27.196 INFO:teuthology.orchestra.run.smithi142.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T10:06:27.197 INFO:teuthology.orchestra.run.smithi142.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T10:06:27.197 INFO:teuthology.orchestra.run.smithi142.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T10:06:27.209 INFO:teuthology.orchestra.run.smithi142.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T10:06:27.213 INFO:teuthology.orchestra.run.smithi142.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T10:06:27.213 INFO:teuthology.orchestra.run.smithi142.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T10:06:27.213 INFO:teuthology.orchestra.run.smithi142.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T10:06:27.263 INFO:teuthology.orchestra.run.smithi142.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T10:06:27.275 INFO:teuthology.orchestra.run.smithi142.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T10:06:27.282 INFO:teuthology.orchestra.run.smithi142.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T10:06:27.282 INFO:teuthology.orchestra.run.smithi142.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T10:06:27.286 INFO:teuthology.orchestra.run.smithi142.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T10:06:27.287 INFO:teuthology.orchestra.run.smithi142.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T10:06:27.298 INFO:teuthology.orchestra.run.smithi142.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T10:06:27.299 INFO:teuthology.orchestra.run.smithi142.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T10:06:27.315 INFO:teuthology.orchestra.run.smithi142.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T10:06:27.316 INFO:teuthology.orchestra.run.smithi142.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T10:06:27.327 INFO:teuthology.orchestra.run.smithi142.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T10:06:27.405 INFO:teuthology.orchestra.run.smithi142.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T10:06:27.406 INFO:teuthology.orchestra.run.smithi142.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T10:06:27.406 INFO:teuthology.orchestra.run.smithi142.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T10:06:27.406 INFO:teuthology.orchestra.run.smithi142.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T10:06:27.452 INFO:teuthology.orchestra.run.smithi142.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T10:06:27.453 INFO:teuthology.orchestra.run.smithi142.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T10:06:27.453 INFO:teuthology.orchestra.run.smithi142.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T10:06:27.453 INFO:teuthology.orchestra.run.smithi142.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T10:06:27.455 INFO:teuthology.orchestra.run.smithi142.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T10:06:27.456 INFO:teuthology.orchestra.run.smithi142.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T10:06:27.457 INFO:teuthology.orchestra.run.smithi142.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T10:06:27.460 INFO:teuthology.orchestra.run.smithi142.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T10:06:27.462 INFO:teuthology.orchestra.run.smithi142.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T10:06:27.465 INFO:teuthology.orchestra.run.smithi142.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T10:06:27.469 INFO:teuthology.orchestra.run.smithi142.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T10:06:27.485 INFO:teuthology.orchestra.run.smithi142.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T10:06:27.489 INFO:teuthology.orchestra.run.smithi142.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T10:06:27.492 INFO:teuthology.orchestra.run.smithi142.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T10:06:27.510 INFO:teuthology.orchestra.run.smithi142.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T10:06:27.512 INFO:teuthology.orchestra.run.smithi142.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T10:06:27.548 INFO:teuthology.orchestra.run.smithi142.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T10:06:27.549 INFO:teuthology.orchestra.run.smithi142.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T10:06:27.549 INFO:teuthology.orchestra.run.smithi142.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T10:06:27.549 INFO:teuthology.orchestra.run.smithi142.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T10:06:27.550 INFO:teuthology.orchestra.run.smithi142.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T10:06:27.551 INFO:teuthology.orchestra.run.smithi142.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T10:06:27.562 INFO:teuthology.orchestra.run.smithi142.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T10:06:27.562 INFO:teuthology.orchestra.run.smithi142.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T10:06:27.563 INFO:teuthology.orchestra.run.smithi142.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T10:06:27.566 INFO:teuthology.orchestra.run.smithi142.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T10:06:27.583 INFO:teuthology.orchestra.run.smithi142.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T10:06:27.584 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T10:06:27.584 INFO:teuthology.orchestra.run.smithi142.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T10:06:27.584 INFO:teuthology.orchestra.run.smithi142.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T10:06:27.586 INFO:teuthology.orchestra.run.smithi142.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T10:06:27.587 INFO:teuthology.orchestra.run.smithi142.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T10:06:27.592 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T10:06:27.634 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T10:06:27.647 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T10:06:27.715 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T10:06:27.773 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T10:06:27.779 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T10:06:27.808 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T10:06:27.949 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T10:06:27.957 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T10:06:27.980 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T10:06:27.987 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T10:06:27.988 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T10:06:27.992 INFO:teuthology.orchestra.run.smithi142.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T10:06:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T10:06:27.996 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T10:06:27.996 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T10:06:28.000 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T10:06:28.020 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T10:06:28.031 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T10:06:28.055 INFO:teuthology.orchestra.run.smithi142.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T10:06:28.057 INFO:teuthology.orchestra.run.smithi142.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T10:06:28.058 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T10:06:28.059 INFO:teuthology.orchestra.run.smithi142.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T10:06:28.079 INFO:teuthology.orchestra.run.smithi142.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T10:06:28.084 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T10:06:28.085 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T10:06:28.086 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T10:06:28.086 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T10:06:28.088 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T10:06:28.089 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T10:06:28.090 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T10:06:28.092 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T10:06:28.101 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T10:06:28.103 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T10:06:28.110 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T10:06:28.112 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T10:06:28.113 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T10:06:28.163 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T10:06:28.175 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T10:06:28.177 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T10:06:28.178 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T10:06:28.178 INFO:teuthology.orchestra.run.smithi142.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T10:06:28.186 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T10:06:28.187 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T10:06:28.188 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T10:06:28.189 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T10:06:28.195 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T10:06:28.197 INFO:teuthology.orchestra.run.smithi150.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T10:06:28.276 INFO:teuthology.orchestra.run.smithi150.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T10:06:28.276 INFO:teuthology.orchestra.run.smithi150.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T10:06:28.279 INFO:teuthology.orchestra.run.smithi150.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T10:06:28.281 INFO:teuthology.orchestra.run.smithi150.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T10:06:28.284 INFO:teuthology.orchestra.run.smithi150.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T10:06:28.284 INFO:teuthology.orchestra.run.smithi150.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T10:06:28.285 INFO:teuthology.orchestra.run.smithi150.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T10:06:28.611 INFO:teuthology.orchestra.run.smithi150.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T10:06:28.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T10:06:28.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T10:06:28.846 INFO:teuthology.orchestra.run.smithi142.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T10:06:28.847 INFO:teuthology.orchestra.run.smithi142.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T10:06:28.847 INFO:teuthology.orchestra.run.smithi142.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T10:06:28.847 INFO:teuthology.orchestra.run.smithi142.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T10:06:28.848 INFO:teuthology.orchestra.run.smithi150.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T10:06:28.849 INFO:teuthology.orchestra.run.smithi150.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T10:06:28.849 INFO:teuthology.orchestra.run.smithi150.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T10:06:28.910 INFO:teuthology.orchestra.run.smithi142.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T10:06:28.930 INFO:teuthology.orchestra.run.smithi150.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T10:06:28.935 INFO:teuthology.orchestra.run.smithi150.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T10:06:28.938 INFO:teuthology.orchestra.run.smithi150.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T10:06:28.942 INFO:teuthology.orchestra.run.smithi150.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T10:06:28.943 INFO:teuthology.orchestra.run.smithi150.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T10:06:28.947 INFO:teuthology.orchestra.run.smithi150.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T10:06:28.949 INFO:teuthology.orchestra.run.smithi150.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T10:06:29.024 INFO:teuthology.orchestra.run.smithi150.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T10:06:29.027 INFO:teuthology.orchestra.run.smithi150.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T10:06:29.098 INFO:teuthology.orchestra.run.smithi142.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T10:06:29.098 INFO:teuthology.orchestra.run.smithi142.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T10:06:29.188 INFO:teuthology.orchestra.run.smithi150.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T10:06:29.190 INFO:teuthology.orchestra.run.smithi150.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T10:06:29.190 INFO:teuthology.orchestra.run.smithi150.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T10:06:29.190 INFO:teuthology.orchestra.run.smithi150.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T10:06:29.191 INFO:teuthology.orchestra.run.smithi150.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T10:06:29.192 INFO:teuthology.orchestra.run.smithi150.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T10:06:29.205 INFO:teuthology.orchestra.run.smithi150.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T10:06:29.205 INFO:teuthology.orchestra.run.smithi150.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T10:06:29.206 INFO:teuthology.orchestra.run.smithi150.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T10:06:29.271 INFO:teuthology.orchestra.run.smithi150.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T10:06:29.353 INFO:teuthology.orchestra.run.smithi150.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T10:06:29.353 INFO:teuthology.orchestra.run.smithi150.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T10:06:29.354 INFO:teuthology.orchestra.run.smithi150.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T10:06:29.357 INFO:teuthology.orchestra.run.smithi150.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T10:06:29.358 INFO:teuthology.orchestra.run.smithi150.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T10:06:29.893 INFO:teuthology.orchestra.run.smithi150.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T10:06:30.041 INFO:teuthology.orchestra.run.smithi150.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T10:06:30.044 INFO:teuthology.orchestra.run.smithi150.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T10:06:30.049 INFO:teuthology.orchestra.run.smithi150.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T10:06:30.106 INFO:teuthology.orchestra.run.smithi150.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T10:06:30.305 INFO:teuthology.orchestra.run.smithi150.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T10:06:30.819 INFO:teuthology.orchestra.run.smithi142.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T10:06:30.823 INFO:teuthology.orchestra.run.smithi142.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T10:06:30.824 INFO:teuthology.orchestra.run.smithi142.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T10:06:31.026 INFO:teuthology.orchestra.run.smithi150.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T10:06:31.026 INFO:teuthology.orchestra.run.smithi150.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T10:06:31.038 INFO:teuthology.orchestra.run.smithi150.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T10:06:31.125 INFO:teuthology.orchestra.run.smithi150.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T10:06:31.126 INFO:teuthology.orchestra.run.smithi150.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T10:06:31.128 INFO:teuthology.orchestra.run.smithi150.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T10:06:31.298 INFO:teuthology.orchestra.run.smithi142.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T10:06:31.303 INFO:teuthology.orchestra.run.smithi150.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T10:06:31.423 INFO:teuthology.orchestra.run.smithi150.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T10:06:31.423 INFO:teuthology.orchestra.run.smithi150.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T10:06:31.956 INFO:teuthology.orchestra.run.smithi142.stdout:Fetched 201 MB in 4s (45.6 MB/s) 2024-09-18T10:06:32.097 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T10:06:32.152 INFO:teuthology.orchestra.run.smithi142.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T10:06:32.157 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T10:06:32.229 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:06:32.533 INFO:teuthology.orchestra.run.smithi150.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T10:06:32.538 INFO:teuthology.orchestra.run.smithi150.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T10:06:32.538 INFO:teuthology.orchestra.run.smithi150.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T10:06:32.560 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:32.716 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:06:32.849 INFO:teuthology.orchestra.run.smithi150.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T10:06:33.460 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:33.488 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 201 MB in 6s (34.4 MB/s) 2024-09-18T10:06:33.587 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:06:33.612 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T10:06:33.664 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T10:06:33.668 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T10:06:33.744 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:06:34.105 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:34.233 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:06:34.249 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T10:06:34.264 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:34.315 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:34.651 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rados. 2024-09-18T10:06:34.667 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:34.709 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:34.952 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:35.077 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:06:35.104 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T10:06:35.122 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:35.162 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:35.398 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T10:06:35.413 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:35.447 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:35.739 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T10:06:35.759 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:35.775 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T10:06:35.791 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:35.797 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:35.833 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:36.153 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T10:06:36.166 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rados. 2024-09-18T10:06:36.170 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T10:06:36.186 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:36.220 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T10:06:36.225 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:36.456 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T10:06:36.474 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:36.514 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:36.586 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T10:06:36.601 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:36.635 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:36.866 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T10:06:36.879 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T10:06:36.882 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T10:06:36.899 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:36.916 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:06:36.929 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:37.253 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T10:06:37.268 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T10:06:37.299 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T10:06:37.311 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:06:37.316 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:37.357 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:37.672 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T10:06:37.688 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T10:06:37.702 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T10:06:37.718 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T10:06:37.730 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:06:37.760 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T10:06:37.996 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T10:06:38.012 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:38.071 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:38.125 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T10:06:38.142 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T10:06:38.183 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:06:38.407 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T10:06:38.426 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T10:06:38.482 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:06:38.503 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T10:06:38.521 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T10:06:38.561 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:06:38.858 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T10:06:38.876 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T10:06:38.918 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:06:39.023 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T10:06:39.039 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:39.108 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:39.229 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T10:06:39.247 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T10:06:39.304 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:06:39.636 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T10:06:39.657 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:39.697 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T10:06:39.700 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:39.715 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T10:06:39.757 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:06:40.021 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T10:06:40.040 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:40.079 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:40.127 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T10:06:40.142 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T10:06:40.185 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:06:40.484 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T10:06:40.501 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T10:06:40.540 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:06:40.672 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T10:06:40.688 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:40.740 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:40.909 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package librgw2. 2024-09-18T10:06:40.927 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:40.967 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:41.266 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T10:06:41.282 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:41.324 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:41.736 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T10:06:41.753 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:06:41.753 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T10:06:41.771 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:41.812 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:41.819 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:06:42.131 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liboath0. 2024-09-18T10:06:42.147 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T10:06:42.181 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T10:06:42.218 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T10:06:42.239 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T10:06:42.281 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:06:42.492 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T10:06:42.510 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:42.558 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:42.709 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librgw2. 2024-09-18T10:06:42.725 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:42.867 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:43.029 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-common. 2024-09-18T10:06:43.044 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:43.112 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:43.628 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T10:06:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:43.678 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:44.039 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liboath0. 2024-09-18T10:06:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T10:06:44.097 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T10:06:44.433 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T10:06:44.452 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:44.491 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:44.978 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-common. 2024-09-18T10:06:44.997 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:45.036 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:45.625 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-base. 2024-09-18T10:06:45.642 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:45.694 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:46.588 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T10:06:46.606 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T10:06:46.646 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:06:47.042 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T10:06:47.059 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T10:06:47.100 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T10:06:47.394 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T10:06:47.410 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T10:06:47.444 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T10:06:47.625 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-base. 2024-09-18T10:06:47.641 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:47.672 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T10:06:47.687 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:47.693 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:47.738 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:48.110 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T10:06:48.125 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T10:06:48.167 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:06:48.487 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T10:06:48.506 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T10:06:48.545 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:06:48.613 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T10:06:48.632 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T10:06:48.671 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:06:48.882 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T10:06:48.899 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T10:06:48.940 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T10:06:49.074 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T10:06:49.090 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T10:06:49.132 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T10:06:49.252 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T10:06:49.310 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:06:49.460 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T10:06:49.479 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T10:06:49.518 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T10:06:49.647 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T10:06:49.663 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T10:06:49.705 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T10:06:49.771 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T10:06:49.788 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:06:49.829 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:50.024 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T10:06:50.044 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T10:06:50.082 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:06:50.218 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T10:06:50.234 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T10:06:50.309 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-mako. 2024-09-18T10:06:50.384 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T10:06:50.426 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:06:50.595 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T10:06:50.615 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T10:06:50.653 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:06:50.764 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T10:06:50.779 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T10:06:50.821 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:06:51.023 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T10:06:51.043 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T10:06:51.081 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T10:06:51.124 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T10:06:51.140 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T10:06:51.199 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:06:51.385 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T10:06:51.403 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T10:06:51.434 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:06:51.494 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webob. 2024-09-18T10:06:51.510 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T10:06:51.552 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:06:51.746 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T10:06:51.765 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T10:06:51.804 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T10:06:51.921 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T10:06:51.980 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:06:52.123 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T10:06:52.143 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T10:06:52.182 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:06:52.366 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T10:06:52.383 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T10:06:52.424 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T10:06:52.510 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-mako. 2024-09-18T10:06:52.526 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T10:06:52.568 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:06:52.769 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T10:06:52.784 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T10:06:52.830 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:06:52.904 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T10:06:52.921 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T10:06:52.962 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:06:53.172 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T10:06:53.188 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T10:06:53.230 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T10:06:53.257 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T10:06:53.273 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T10:06:53.307 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:06:53.555 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-paste. 2024-09-18T10:06:53.568 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webob. 2024-09-18T10:06:53.571 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T10:06:53.584 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T10:06:53.614 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:06:53.668 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:06:53.900 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T10:06:53.919 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T10:06:53.950 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:06:53.987 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T10:06:54.004 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T10:06:54.070 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:06:54.244 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T10:06:54.263 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T10:06:54.302 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:06:54.370 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T10:06:54.385 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T10:06:54.453 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T10:06:54.631 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T10:06:54.650 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T10:06:54.689 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T10:06:54.798 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T10:06:54.814 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T10:06:54.885 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:06:55.033 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T10:06:55.051 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T10:06:55.091 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:06:55.210 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T10:06:55.228 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T10:06:55.301 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T10:06:55.440 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T10:06:55.458 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T10:06:55.490 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:06:55.588 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-paste. 2024-09-18T10:06:55.607 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T10:06:55.646 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:06:55.852 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T10:06:55.868 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:55.910 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:55.965 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T10:06:55.981 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T10:06:56.023 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:06:56.248 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T10:06:56.266 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:56.306 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:56.360 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T10:06:56.376 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T10:06:56.418 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:06:56.705 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T10:06:56.724 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T10:06:56.754 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T10:06:56.778 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:56.819 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:57.082 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T10:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T10:06:57.140 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:06:57.507 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T10:06:57.527 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T10:06:57.565 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:06:57.623 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T10:06:57.639 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:57.688 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:57.935 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T10:06:57.953 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:58.002 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:59.207 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T10:06:59.227 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:59.271 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:59.650 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph. 2024-09-18T10:06:59.667 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:59.708 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:06:59.799 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T10:06:59.816 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:06:59.857 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:00.027 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T10:07:00.046 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:00.352 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:00.661 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T10:07:00.679 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:00.719 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:01.163 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T10:07:01.182 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:01.221 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:01.800 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package cephadm. 2024-09-18T10:07:01.818 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:02.196 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T10:07:02.214 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T10:07:02.252 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:07:02.539 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T10:07:02.555 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:02.589 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:02.881 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph. 2024-09-18T10:07:02.898 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:02.925 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T10:07:02.930 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:02.943 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T10:07:02.975 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T10:07:03.194 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T10:07:03.211 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T10:07:03.252 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T10:07:03.290 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:03.333 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:03.555 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T10:07:03.572 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T10:07:03.613 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:07:03.788 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T10:07:03.805 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:03.846 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:03.958 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-routes. 2024-09-18T10:07:03.977 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T10:07:04.016 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T10:07:04.335 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T10:07:04.351 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:04.394 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:04.433 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephadm. 2024-09-18T10:07:04.449 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:04.491 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:04.843 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T10:07:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T10:07:04.901 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:07:05.263 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T10:07:05.283 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:05.321 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:05.678 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T10:07:05.696 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T10:07:05.741 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T10:07:05.749 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T10:07:05.765 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T10:07:05.807 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:07:05.993 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T10:07:06.009 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T10:07:06.051 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:07:06.152 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T10:07:06.168 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T10:07:06.232 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:07:06.354 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T10:07:06.372 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T10:07:06.412 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:07:06.554 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T10:07:06.571 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T10:07:06.612 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T10:07:06.748 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-routes. 2024-09-18T10:07:06.767 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T10:07:06.806 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T10:07:07.159 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T10:07:07.174 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:07.217 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:07.931 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T10:07:07.948 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T10:07:07.997 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:08.517 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T10:07:08.522 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T10:07:08.535 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T10:07:08.543 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T10:07:08.575 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T10:07:08.580 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:07:08.918 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T10:07:08.936 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T10:07:08.945 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T10:07:08.963 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T10:07:08.976 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:07:09.003 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:09.329 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T10:07:09.345 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T10:07:09.387 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T10:07:09.695 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T10:07:09.712 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:09.786 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:10.629 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T10:07:10.646 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T10:07:10.704 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:10.923 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T10:07:10.940 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T10:07:10.981 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T10:07:11.249 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T10:07:11.268 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T10:07:11.283 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T10:07:11.300 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T10:07:11.307 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T10:07:11.342 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:07:11.686 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T10:07:11.704 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T10:07:11.720 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T10:07:11.736 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T10:07:11.743 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:11.778 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:07:12.114 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T10:07:12.131 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T10:07:12.181 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T10:07:12.475 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T10:07:12.485 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T10:07:12.492 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T10:07:12.504 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:12.533 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T10:07:12.543 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:12.861 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T10:07:12.881 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T10:07:12.920 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:07:13.231 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T10:07:13.250 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T10:07:13.288 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:07:13.601 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T10:07:13.621 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T10:07:13.659 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:07:13.662 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T10:07:13.681 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T10:07:13.712 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T10:07:14.004 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T10:07:14.020 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T10:07:14.057 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T10:07:14.062 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:07:14.076 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T10:07:14.115 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:07:14.399 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T10:07:14.415 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T10:07:14.456 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:07:14.785 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T10:07:14.801 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T10:07:14.843 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:07:15.098 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T10:07:15.115 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:15.156 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:15.179 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T10:07:15.195 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T10:07:15.237 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T10:07:15.442 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T10:07:15.462 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T10:07:15.516 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:07:15.565 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T10:07:15.581 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T10:07:15.623 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T10:07:15.899 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T10:07:15.916 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:07:15.957 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:16.003 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T10:07:16.019 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T10:07:16.052 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:07:16.381 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T10:07:16.397 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T10:07:16.438 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:07:16.593 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package jq. 2024-09-18T10:07:16.613 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:07:16.673 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:16.784 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T10:07:16.801 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T10:07:16.841 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:07:16.929 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package socat. 2024-09-18T10:07:16.948 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T10:07:16.987 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T10:07:17.162 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T10:07:17.178 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T10:07:17.220 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:07:17.316 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T10:07:17.335 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T10:07:17.374 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:07:17.565 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T10:07:17.581 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T10:07:17.623 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:07:17.696 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-test. 2024-09-18T10:07:17.714 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:17.754 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:18.300 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T10:07:18.317 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:18.383 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:18.694 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T10:07:18.710 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T10:07:18.786 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:07:19.139 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T10:07:19.157 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:07:19.222 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:19.500 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package jq. 2024-09-18T10:07:19.519 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:07:19.558 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:19.853 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package socat. 2024-09-18T10:07:19.869 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T10:07:19.911 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T10:07:20.249 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T10:07:20.268 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T10:07:20.307 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:07:20.673 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-test. 2024-09-18T10:07:20.690 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:20.731 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:22.250 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T10:07:22.266 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:22.307 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:22.658 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T10:07:22.675 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:07:22.715 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:23.263 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T10:07:23.282 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:23.321 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:23.716 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T10:07:23.735 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T10:07:23.792 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:07:24.121 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T10:07:24.141 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:07:24.179 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:25.361 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T10:07:25.377 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:07:25.419 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:25.784 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T10:07:25.790 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T10:07:25.800 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:07:25.809 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T10:07:25.842 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:25.848 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T10:07:26.193 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T10:07:26.211 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T10:07:26.251 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:07:26.329 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T10:07:26.348 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:26.387 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:26.570 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-attr. 2024-09-18T10:07:26.590 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T10:07:26.628 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T10:07:26.807 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T10:07:26.823 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T10:07:26.883 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:07:26.965 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T10:07:26.981 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T10:07:27.022 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:07:27.212 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T10:07:27.231 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:07:27.269 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:27.334 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T10:07:27.354 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T10:07:27.392 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T10:07:27.745 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T10:07:27.761 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T10:07:27.803 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:07:28.298 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T10:07:28.316 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T10:07:28.356 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T10:07:28.676 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T10:07:28.692 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T10:07:28.734 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T10:07:28.854 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T10:07:28.872 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T10:07:28.912 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T10:07:29.053 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T10:07:29.074 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T10:07:29.112 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:07:29.274 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T10:07:29.290 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T10:07:29.332 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:07:29.456 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-py. 2024-09-18T10:07:29.473 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T10:07:29.514 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:07:29.651 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-attr. 2024-09-18T10:07:29.671 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T10:07:29.709 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T10:07:29.825 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T10:07:29.846 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T10:07:29.884 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:07:30.070 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T10:07:30.088 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T10:07:30.128 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:07:30.203 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T10:07:30.220 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T10:07:30.261 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:07:30.456 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T10:07:30.473 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T10:07:30.514 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T10:07:30.589 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T10:07:30.606 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T10:07:30.647 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T10:07:30.876 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T10:07:30.894 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T10:07:30.934 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:07:30.984 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T10:07:31.001 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T10:07:31.042 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:07:31.328 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T10:07:31.345 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T10:07:31.403 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:07:31.404 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T10:07:31.421 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T10:07:31.462 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T10:07:31.790 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T10:07:31.807 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T10:07:31.848 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T10:07:32.099 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package radosgw. 2024-09-18T10:07:32.118 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:32.156 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:32.187 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T10:07:32.208 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T10:07:32.245 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:07:32.614 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-py. 2024-09-18T10:07:32.631 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T10:07:32.672 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:07:33.026 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T10:07:33.043 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T10:07:33.084 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:07:33.403 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T10:07:33.424 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T10:07:33.461 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:07:33.773 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T10:07:33.793 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T10:07:33.847 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T10:07:33.910 INFO:teuthology.orchestra.run.smithi142.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T10:07:33.927 INFO:teuthology.orchestra.run.smithi142.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:33.967 INFO:teuthology.orchestra.run.smithi142.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:34.184 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T10:07:34.201 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T10:07:34.242 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:07:34.284 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T10:07:34.537 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T10:07:34.557 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T10:07:34.595 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:07:34.609 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:07:34.735 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:07:34.861 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:07:35.054 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:35.344 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package radosgw. 2024-09-18T10:07:35.364 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:35.657 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user cephadm....done 2024-09-18T10:07:35.852 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:35.986 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:07:36.348 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T10:07:36.658 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:36.775 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T10:07:36.926 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:07:37.197 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T10:07:37.215 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:07:37.255 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:37.278 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T10:07:37.572 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T10:07:37.705 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:07:37.901 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:07:37.998 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T10:07:38.011 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:07:38.112 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:07:38.307 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:07:38.340 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:38.575 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:07:38.904 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:38.936 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user cephadm....done 2024-09-18T10:07:39.064 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:39.220 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:39.223 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:07:39.346 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:07:39.616 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T10:07:39.639 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:07:39.917 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:07:39.931 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:07:40.057 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:07:40.077 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T10:07:40.187 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:07:40.425 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T10:07:40.497 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T10:07:40.693 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:07:40.925 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:07:40.978 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:07:41.226 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T10:07:41.312 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:07:41.561 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:07:41.588 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:07:41.845 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:07:41.914 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T10:07:42.129 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:42.215 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T10:07:42.414 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:42.516 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:07:42.573 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:07:42.667 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:07:42.776 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:07:42.858 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:07:43.053 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:07:43.167 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:07:43.293 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:07:43.446 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:07:43.720 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T10:07:43.830 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:07:44.030 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:07:44.199 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T10:07:44.305 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:07:44.325 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:44.651 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:44.656 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:07:44.769 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T10:07:44.941 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:07:45.103 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:07:45.254 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:07:45.325 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T10:07:45.380 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T10:07:45.673 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:07:45.983 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:45.993 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T10:07:46.134 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:07:46.260 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:07:46.303 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:07:46.446 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:07:46.546 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:07:46.569 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:07:46.814 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:07:46.854 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T10:07:47.138 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:07:47.274 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:07:47.642 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:07:47.830 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:07:48.019 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T10:07:48.145 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:48.148 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T10:07:48.458 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:07:48.472 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:48.606 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T10:07:48.751 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:07:48.965 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:07:49.092 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:07:49.119 INFO:teuthology.orchestra.run.smithi142.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T10:07:49.217 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T10:07:49.387 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:07:49.513 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:07:49.536 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:07:49.853 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:49.873 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:07:49.979 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:07:50.130 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:07:50.215 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:07:50.341 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:07:50.465 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:07:50.467 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:07:50.785 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:07:50.883 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T10:07:51.053 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:51.178 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T10:07:51.209 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:07:51.360 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:07:51.636 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T10:07:51.904 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:07:52.205 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:07:52.578 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T10:07:52.642 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:52.760 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:52.841 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:07:52.886 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T10:07:52.967 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:07:53.271 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T10:07:53.310 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:07:53.563 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:07:53.678 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:07:53.804 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:07:53.881 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:53.955 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:07:54.007 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T10:07:54.315 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:07:54.350 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:54.476 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:07:54.599 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:54.758 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T10:07:54.828 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:54.954 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:07:55.272 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:56.231 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:56.335 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:56.356 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:56.461 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:07:56.516 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T10:07:56.620 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T10:07:56.746 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:07:56.872 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T10:07:56.901 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T10:07:57.165 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:07:57.210 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:07:57.291 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T10:07:57.537 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:07:57.592 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:57.687 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T10:07:57.737 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:57.960 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:07:57.960 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:07:58.014 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:58.173 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:07:58.436 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:58.491 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:58.651 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:07:58.655 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:07:58.957 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T10:07:59.002 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:07:59.267 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:07:59.619 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:59.746 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:07:59.888 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:00.039 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:00.040 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:00.165 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T10:08:00.166 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:08:00.326 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T10:08:00.452 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:08:00.466 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T10:08:00.578 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T10:08:00.793 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:00.896 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:08:00.927 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:01.037 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:08:01.070 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T10:08:01.373 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:01.524 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:01.745 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:08:01.746 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:08:01.965 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:02.200 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:02.428 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:08:02.534 INFO:teuthology.orchestra.run.smithi142.stdout:Adding group ceph....done 2024-09-18T10:08:02.747 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T10:08:02.952 INFO:teuthology.orchestra.run.smithi142.stdout:Adding system user ceph....done 2024-09-18T10:08:03.065 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:08:03.153 INFO:teuthology.orchestra.run.smithi142.stdout:Setting system user ceph properties....done 2024-09-18T10:08:03.178 INFO:teuthology.orchestra.run.smithi142.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T10:08:03.327 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T10:08:03.442 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:03.593 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:03.651 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T10:08:03.711 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:03.837 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:03.997 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T10:08:04.116 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:04.225 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:04.290 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T10:08:04.464 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:08:04.464 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:08:04.616 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:04.742 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:04.893 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:08:05.169 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:05.555 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T10:08:05.805 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:06.029 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:06.259 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:08:06.259 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:08:06.366 INFO:teuthology.orchestra.run.smithi150.stdout:Adding group ceph....done 2024-09-18T10:08:06.706 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:06.759 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user ceph....done 2024-09-18T10:08:06.926 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:08:06.926 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:08:06.960 INFO:teuthology.orchestra.run.smithi150.stdout:Setting system user ceph properties....done 2024-09-18T10:08:06.981 INFO:teuthology.orchestra.run.smithi150.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T10:08:07.131 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T10:08:07.365 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:07.434 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T10:08:07.865 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:07.884 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:08:07.884 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:08:07.991 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:08.266 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:08:08.266 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:08:08.334 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:08.460 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:08.636 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:08.852 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:08:08.852 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:08:08.884 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.292 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T10:08:09.304 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.438 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.556 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.698 INFO:teuthology.orchestra.run.smithi142.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.711 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:09.859 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T10:08:09.934 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:08:09.935 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:08:10.019 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T10:08:10.295 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T10:08:10.363 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:10.594 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:08:10.594 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:08:11.031 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:11.664 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:08:11.664 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:08:12.083 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:12.209 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:12.386 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:12.607 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:08:12.608 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:08:13.020 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:13.163 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:13.164 INFO:teuthology.orchestra.run.smithi142.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T10:08:13.314 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:13.457 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:08:13.584 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T10:08:13.744 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T10:08:13.916 INFO:teuthology.orchestra.run.smithi142.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T10:08:13.922 DEBUG:teuthology.parallel:result is None 2024-09-18T10:08:14.003 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T10:08:16.974 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T10:08:17.591 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T10:08:17.596 DEBUG:teuthology.parallel:result is None 2024-09-18T10:08:17.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:08:17.746 DEBUG:teuthology.orchestra.run.smithi142:> dpkg-query -W -f '${Version}' ceph 2024-09-18T10:08:17.769 INFO:teuthology.orchestra.run.smithi142.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:08:17.770 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:08:17.770 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T10:08:17.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:08:17.909 DEBUG:teuthology.orchestra.run.smithi150:> dpkg-query -W -f '${Version}' ceph 2024-09-18T10:08:17.931 INFO:teuthology.orchestra.run.smithi150.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:08:17.931 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:08:17.931 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T10:08:17.933 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T10:08:17.933 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:17.933 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T10:08:17.950 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:17.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T10:08:17.966 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T10:08:17.966 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:17.966 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T10:08:17.981 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T10:08:18.037 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:18.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T10:08:18.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T10:08:18.063 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T10:08:18.064 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:18.064 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T10:08:18.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T10:08:18.090 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:18.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T10:08:18.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T10:08:18.160 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T10:08:18.160 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:18.161 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T10:08:18.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T10:08:18.190 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:18.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T10:08:18.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T10:08:18.264 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T10:08:18.339 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T10:08:18.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph 2024-09-18T10:08:18.341 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-09-18T10:08:18.357 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T10:08:18.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T10:08:18.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T10:08:18.373 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T10:08:18.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T10:08:18.413 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T10:08:18.440 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T10:08:18.440 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T10:08:18.440 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7910840', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi142.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T10:08:18.441 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T10:08:18.443 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T10:08:18.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T10:08:18.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T10:08:18.510 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:18.510 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T10:08:18.515 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T10:08:18.515 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-18 10:04:48.810359874 +0000 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-18 10:04:48.614364394 +0000 2024-09-18T10:08:18.563 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-18 10:04:48.614364394 +0000 2024-09-18T10:08:18.564 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-18T10:08:18.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T10:08:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-18T10:08:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-18T10:08:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000434173 s, 1.2 MB/s 2024-09-18T10:08:18.577 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T10:08:18.585 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-09-18T10:08:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 367 Links: 1 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-18 10:04:49.234350094 +0000 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-18 10:04:49.014355168 +0000 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-18 10:04:49.014355168 +0000 2024-09-18T10:08:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-18T10:08:18.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T10:08:18.648 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-18T10:08:18.648 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-18T10:08:18.648 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.00042831 s, 1.2 MB/s 2024-09-18T10:08:18.649 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T10:08:18.657 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 373 Links: 1 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-18 10:04:49.654340406 +0000 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-18 10:04:49.426345665 +0000 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-18 10:04:49.426345665 +0000 2024-09-18T10:08:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-18T10:08:18.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T10:08:18.722 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-18T10:08:18.722 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-18T10:08:18.722 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000406155 s, 1.3 MB/s 2024-09-18T10:08:18.723 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T10:08:18.732 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 380 Links: 1 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-09-18 10:04:50.122329609 +0000 2024-09-18T10:08:18.738 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-09-18 10:04:49.890334962 +0000 2024-09-18T10:08:18.739 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-09-18 10:04:49.890334962 +0000 2024-09-18T10:08:18.739 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2024-09-18T10:08:18.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T10:08:18.794 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-09-18T10:08:18.794 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-09-18T10:08:18.794 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000368999 s, 1.4 MB/s 2024-09-18T10:08:18.795 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T10:08:18.804 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:18.804 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:18.804 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T10:08:18.811 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T10:08:18.811 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-09-18T10:08:18.818 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T10:08:18.818 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.818 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 369 Links: 1 2024-09-18T10:08:18.819 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.819 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-18 10:04:50.244492728 +0000 2024-09-18T10:08:18.819 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-18 10:04:50.028497711 +0000 2024-09-18T10:08:18.819 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-18 10:04:50.028497711 +0000 2024-09-18T10:08:18.819 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-18T10:08:18.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T10:08:18.878 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-18T10:08:18.878 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-18T10:08:18.878 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00044863 s, 1.1 MB/s 2024-09-18T10:08:18.880 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T10:08:18.889 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-09-18T10:08:18.896 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T10:08:18.896 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.896 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-18T10:08:18.896 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.897 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-18 10:04:50.640483594 +0000 2024-09-18T10:08:18.897 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-18 10:04:50.452487930 +0000 2024-09-18T10:08:18.897 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-18 10:04:50.452487930 +0000 2024-09-18T10:08:18.897 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-18T10:08:18.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T10:08:18.950 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-18T10:08:18.950 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-18T10:08:18.950 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000401048 s, 1.3 MB/s 2024-09-18T10:08:18.951 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T10:08:18.960 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 382 Links: 1 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-18 10:04:51.072473630 +0000 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-18 10:04:50.848478795 +0000 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-18 10:04:50.848478795 +0000 2024-09-18T10:08:18.967 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-18T10:08:18.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T10:08:19.023 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-18T10:08:19.024 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-18T10:08:19.024 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000358222 s, 1.4 MB/s 2024-09-18T10:08:19.024 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T10:08:19.032 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 6h/6d Inode: 387 Links: 1 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-18 10:04:51.500463757 +0000 2024-09-18T10:08:19.038 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-18 10:04:51.304468277 +0000 2024-09-18T10:08:19.039 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-18 10:04:51.304468277 +0000 2024-09-18T10:08:19.039 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-18T10:08:19.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T10:08:19.097 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-18T10:08:19.097 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-18T10:08:19.097 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000389258 s, 1.3 MB/s 2024-09-18T10:08:19.098 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T10:08:19.105 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:19.105 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T10:08:19.105 INFO:tasks.ceph:Generating config... 2024-09-18T10:08:19.106 INFO:tasks.ceph:[client] debug monc = 20 2024-09-18T10:08:19.106 INFO:tasks.ceph:[client] debug ms = 1 2024-09-18T10:08:19.106 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T10:08:19.106 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-18T10:08:19.107 INFO:tasks.ceph:[global] ms type = async 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T10:08:19.107 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T10:08:19.108 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T10:08:19.108 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T10:08:19.109 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T10:08:19.110 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T10:08:19.110 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T10:08:19.110 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T10:08:19.110 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T10:08:19.110 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T10:08:19.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T10:08:19.144 INFO:teuthology.orchestra.run.smithi142.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T10:08:19.147 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T10:08:19.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T10:08:19.239 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.142'), ('mon.c', '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]'), ('mon.b', '172.21.15.150')] 2024-09-18T10:08:19.239 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790],172.21.15.150', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-18T10:08:19.240 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:19.240 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T10:08:19.246 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.142 --addv c '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]' --add b 172.21.15.150 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:19.332 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:19.332 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: generated fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:19.332 INFO:teuthology.orchestra.run.smithi142.stdout:setting min_mon_release = octopus 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:epoch 0 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:min_mon_release 15 (octopus) 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:election_strategy: 1 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:19.333 INFO:teuthology.orchestra.run.smithi142.stdout:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:19.334 INFO:teuthology.orchestra.run.smithi142.stdout:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:19.334 INFO:teuthology.orchestra.run.smithi142.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T10:08:19.335 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T10:08:19.342 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bfa92b4c-6f79-422f-bfaa-515c68f959fb... 2024-09-18T10:08:19.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T10:08:19.345 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T10:08:19.376 INFO:teuthology.orchestra.run.smithi142.stdout:[global] 2024-09-18T10:08:19.376 INFO:teuthology.orchestra.run.smithi142.stdout: chdir = "" 2024-09-18T10:08:19.376 INFO:teuthology.orchestra.run.smithi142.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T10:08:19.376 INFO:teuthology.orchestra.run.smithi142.stdout: auth supported = cephx 2024-09-18T10:08:19.376 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: filestore xattr use omap = true 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: mon clock drift allowed = 1.000 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: osd crush chooseleaf type = 0 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: auth debug = true 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on old message = true 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on bug = true 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.377 INFO:teuthology.orchestra.run.smithi142.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon pg warn max object skew = 0 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default size = 2 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow primary affinity = true 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow pg remap = true 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on legacy crush tunables = false 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on crush straw calc version zero = false 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on no sortbitwise = false 2024-09-18T10:08:19.378 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on osd down out interval zero = false 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on too few osds = false 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon_allow_pool_size_one = true 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: osd default data pool replay window = 5 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon allow pool delete = true 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon cluster log file level = debug 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: debug asserts on shutdown = true 2024-09-18T10:08:19.379 INFO:teuthology.orchestra.run.smithi142.stdout: mon health detail to clog = false 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: mon host = "172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790],172.21.15.150" 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: mon client directed command retry = 5 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: mon election default strategy = 1 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay max = 1 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay probability = 0.005 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay type = mon 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject internal delays = 0.002 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject socket failures = 2500 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: ms type = async 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: fsid = bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout:[osd] 2024-09-18T10:08:19.380 INFO:teuthology.orchestra.run.smithi142.stdout: osd journal size = 100 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub load threshold = 5.0 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub max interval = 600 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd mclock profile = high_recovery_ops 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd recover clone overlap = true 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd recovery max chunk = 1048576 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug shutdown = true 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug op order = true 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify stray on activate = true 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.381 INFO:teuthology.orchestra.run.smithi142.stdout: osd open classes on start = true 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug pg log writeout = true 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: osd deep scrub update digest min age = 30 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: osd map max advance = 10 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: journal zero on create = true 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: filestore ondisk finisher threads = 3 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: filestore apply finisher threads = 3 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: bdev debug aio = true 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug misdirected ops = true 2024-09-18T10:08:19.382 INFO:teuthology.orchestra.run.smithi142.stdout: bdev async discard = True 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: bdev enable discard = True 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore allocator = bitmap 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore block size = 96636764160 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore fsck on mount = True 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore zero block detection = True 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluefs = 20 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluestore = 20 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: debug osd = 20 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: debug rocksdb = 10 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: mon client ping interval = 4 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: mon client ping timeout = 12 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T10:08:19.383 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd full ratio = 0.9 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd nearfull ratio = 0.8 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify cached snaps = True 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify missing on start = True 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd failsafe full ratio = 0.95 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd mclock override recovery settings = True 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd objectstore = bluestore 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd op queue = debug_random 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: osd op queue cut off = debug_random 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.384 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr] 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: debug mgr = 20 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: mgr/telemetry/nag = false 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: debug monc = 10 2024-09-18T10:08:19.385 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout:[mon] 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: debug paxos = 20 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon data avail warn = 5 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon mgr mkfs grace = 240 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd reporter subtree level = osd 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd prime pg temp = true 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.386 INFO:teuthology.orchestra.run.smithi142.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T10:08:19.387 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: # 1m isn't quite enough 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon_down_mkfs_grace = 2m 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_filestore_osds = false 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon min osdmap epochs = 25 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune interval = 2 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune min = 15 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon osdmap full prune txsize = 2 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon scrub inject missing keys = 0.05 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: mon scrub interval = 300 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: paxos service trim min = 5 2024-09-18T10:08:19.388 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout:[client] 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: rgw cache enabled = true 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable ops log = true 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable usage log = true 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: debug monc = 20 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a] 2024-09-18T10:08:19.389 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.c] 2024-09-18T10:08:19.390 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.b] 2024-09-18T10:08:19.390 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-09-18T10:08:19.390 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-09-18T10:08:19.390 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.391 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-09-18T10:08:19.392 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.393 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790],172.21.15.150" 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon client directed command retry = 5 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: mon election default strategy = 1 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay max = 1 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay probability = 0.005 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject delay type = mon 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject internal delays = 0.002 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms inject socket failures = 2500 2024-09-18T10:08:19.394 INFO:teuthology.orchestra.run.smithi150.stdout: ms type = async 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.395 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-09-18T10:08:19.396 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bdev async discard = True 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bdev enable discard = True 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore allocator = bitmap 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-09-18T10:08:19.397 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore zero block detection = True 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: mon client ping interval = 4 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: mon client ping timeout = 12 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify cached snaps = True 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify missing on start = True 2024-09-18T10:08:19.398 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock override recovery settings = True 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue = debug_random 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: osd op queue cut off = debug_random 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-09-18T10:08:19.399 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 10 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-09-18T10:08:19.400 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-09-18T10:08:19.401 INFO:teuthology.orchestra.run.smithi150.stdout: mon min osdmap epochs = 25 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune interval = 2 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune min = 15 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon osdmap full prune txsize = 2 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon scrub inject crc mismatch = 0.01 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon scrub inject missing keys = 0.05 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: mon scrub interval = 300 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: paxos service trim min = 5 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T10:08:19.402 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T10:08:19.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug monc = 20 2024-09-18T10:08:19.403 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-18T10:08:19.403 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-09-18T10:08:19.403 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-09-18T10:08:19.403 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-09-18T10:08:19.404 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T10:08:19.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T10:08:19.499 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T10:08:19.499 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:19.499 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T10:08:19.504 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:19.504 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T10:08:19.511 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:08:19.511 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:19.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T10:08:19.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T10:08:19.578 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:19.578 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:19.584 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:08:19.584 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:19.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T10:08:19.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T10:08:19.610 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:19.610 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:19.615 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T10:08:19.615 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T10:08:19.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T10:08:19.702 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T10:08:19.704 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T10:08:19.704 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T10:08:19.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-18T10:08:19.734 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-18T10:08:19.747 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T10:08:19.747 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-18T10:08:19.747 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T10:08:19.763 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:19.763 INFO:tasks.ceph:role: osd.0 2024-09-18T10:08:19.763 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:08:19.763 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:20.268 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:20.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-18T10:08:20.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T10:08:20.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T10:08:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:20.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T10:08:20.387 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:20.387 INFO:tasks.ceph:role: osd.1 2024-09-18T10:08:20.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:08:20.387 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:20.922 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:20.923 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:20.923 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:20.923 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:20.923 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:20.924 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-18T10:08:20.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T10:08:20.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T10:08:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:20.990 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:20.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T10:08:21.001 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:21.001 INFO:tasks.ceph:role: osd.2 2024-09-18T10:08:21.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:08:21.001 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T10:08:21.462 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:21.462 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:21.463 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:21.464 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2024-09-18T10:08:21.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T10:08:21.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T10:08:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:21.519 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:21.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:21.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.630+0000 7f5a90e08540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T10:08:21.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.630+0000 7f5a90e08540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T10:08:21.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.630+0000 7f5a90e08540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T10:08:21.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.630+0000 7f5a90e08540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T10:08:21.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.630+0000 7f5a90e08540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T10:08:21.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:21.634+0000 7f5a90e08540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T10:08:22.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:22.206+0000 7f5a90e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:22.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:22.982+0000 7f5a90e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:23.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:23.758+0000 7f5a90e08540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:24.616 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T10:08:24.634 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:24.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T10:08:24.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T10:08:24.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T10:08:24.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T10:08:24.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T10:08:24.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:24.713+0000 7f75b07f0540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T10:08:25.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:25.277+0000 7f75b07f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:26.053+0000 7f75b07f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:26.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:26.837+0000 7f75b07f0540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:27.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T10:08:27.717 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:27.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.749+0000 7fa4e34ba540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T10:08:27.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.749+0000 7fa4e34ba540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T10:08:27.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.749+0000 7fa4e34ba540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T10:08:27.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.749+0000 7fa4e34ba540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T10:08:27.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.749+0000 7fa4e34ba540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T10:08:27.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:27.753+0000 7fa4e34ba540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T10:08:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:28.313+0000 7fa4e34ba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:29.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:29.097+0000 7fa4e34ba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:29.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:29.881+0000 7fa4e34ba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:30.743 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T10:08:30.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T10:08:30.817 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:30.817 INFO:tasks.ceph:role: osd.3 2024-09-18T10:08:30.817 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:08:30.817 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:31.369 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:31.370 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:31.370 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:31.370 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:31.370 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:31.371 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-18T10:08:31.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-18T10:08:31.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T10:08:31.485 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:31.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-18T10:08:31.501 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:31.501 INFO:tasks.ceph:role: osd.4 2024-09-18T10:08:31.502 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:08:31.502 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:32.024 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:32.025 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:32.025 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:32.025 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:32.025 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:32.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-18T10:08:32.029 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-18T10:08:32.074 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-18T10:08:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:32.082 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:32.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-18T10:08:32.093 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-18T10:08:32.093 INFO:tasks.ceph:role: osd.5 2024-09-18T10:08:32.093 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:08:32.093 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T10:08:32.550 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T10:08:32.551 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T10:08:32.551 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T10:08:32.552 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-18T10:08:32.552 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-18T10:08:32.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-18T10:08:32.614 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T10:08:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:08:32.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:32.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.695+0000 7ff9320ac540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T10:08:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.699+0000 7ff9320ac540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T10:08:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.699+0000 7ff9320ac540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T10:08:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.699+0000 7ff9320ac540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T10:08:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.699+0000 7ff9320ac540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T10:08:32.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:32.703+0000 7ff9320ac540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T10:08:33.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:33.271+0000 7ff9320ac540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:34.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:34.055+0000 7ff9320ac540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:34.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:34.835+0000 7ff9320ac540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:35.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T10:08:35.718 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:35.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-18T10:08:35.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-18T10:08:35.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T10:08:35.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T10:08:35.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-18T10:08:35.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:35.799+0000 7f36df158540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-18T10:08:36.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:36.363+0000 7f36df158540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:37.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:37.143+0000 7f36df158540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:37.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:37.923+0000 7f36df158540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:38.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-18T10:08:38.797 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.827+0000 7f5ab39c5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-18T10:08:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.827+0000 7f5ab39c5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-18T10:08:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.827+0000 7f5ab39c5540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T10:08:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.827+0000 7f5ab39c5540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T10:08:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.827+0000 7f5ab39c5540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-18T10:08:38.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:38.831+0000 7f5ab39c5540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-18T10:08:39.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:39.383+0000 7f5ab39c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:40.175+0000 7f5ab39c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:40.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:40.955+0000 7f5ab39c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T10:08:41.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-18T10:08:41.864 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T10:08:41.865 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:41.865 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T10:08:41.882 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:41.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T10:08:41.940 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:41.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T10:08:41.950 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:41.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T10:08:41.966 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:41.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T10:08:41.976 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:41.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-18T10:08:41.987 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:41.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-18T10:08:42.045 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:42.045 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-18T10:08:42.051 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T10:08:42.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T10:08:42.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCVpupmx7m7JRAAjjdLa9B002mYxDcGhNJcCg== 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCYpupmmzrEKhAAH1v0meaNkZTeE27LVDgZMQ== 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCbpupm7vr9LBAADMMv2yiwpUYSqvgmGkv54g== 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCTpupmXAe7KRAAcpI0KkigJ9+J255NpgL2TQ== 2024-09-18T10:08:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCgpupmq0CqKRAAN+CKG0M28bP6Dtrliv/U3Q== 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCjpupmDsOzLxAAte2wpG09OoebJF3GEYnDFQ== 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCmpupmqYaEMRAAQ6/hh3alVeKf3eEUVdqhkA== 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-09-18T10:08:42.066 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQCTpupmo1mtKxAALsAWRKfdyjiVPOE2FMzXQw== 2024-09-18T10:08:42.067 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0] 2024-09-18T10:08:42.067 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCVpupmx7m7JRAAjjdLa9B002mYxDcGhNJcCg== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCYpupmmzrEKhAAH1v0meaNkZTeE27LVDgZMQ== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbpupm7vr9LBAADMMv2yiwpUYSqvgmGkv54g== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCTpupmXAe7KRAAcpI0KkigJ9+J255NpgL2TQ== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.3] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCgpupmq0CqKRAAN+CKG0M28bP6Dtrliv/U3Q== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.4] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCjpupmDsOzLxAAte2wpG09OoebJF3GEYnDFQ== 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.5] 2024-09-18T10:08:42.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCmpupmqYaEMRAAQ6/hh3alVeKf3eEUVdqhkA== 2024-09-18T10:08:42.069 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2024-09-18T10:08:42.069 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCTpupmo1mtKxAALsAWRKfdyjiVPOE2FMzXQw== 2024-09-18T10:08:42.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.156 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.206 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T10:08:42.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T10:08:42.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T10:08:42.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T10:08:42.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-18T10:08:42.615 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T10:08:42.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T10:08:42.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T10:08:42.725 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T10:08:42.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T10:08:42.749 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T10:08:42.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T10:08:42.858 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T10:08:42.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T10:08:42.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T10:08:42.942 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:42.944 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T10:08:42.991 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T10:08:42.992 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:08:42.992 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:08:42.994 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:08:42.994 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:08:42.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:08:42.996 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:08:42.996 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:08:42.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:08:42.999 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:08:42.999 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T10:08:42.999 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T10:08:42.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T10:08:43.001 INFO:tasks.ceph.mgr.x:Started 2024-09-18T10:08:43.001 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T10:08:43.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T10:08:43.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: get_monmap_and_config 2024-09-18T10:08:43.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: build_initial_monmap 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: monmap: 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:43.111827+0000 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:43.111827+0000 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:43.113 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:43.114 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 Processor -- start 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 -- start start 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: init 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d6409d800 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-a con 0x7f7d6409b5c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:08:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d64134600 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6406fb70 con 0x7f7d6409d800 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6409dbd0 con 0x7f7d64134600 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6409afa0 con 0x7f7d6409b5c0 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d63fff700 10 monclient(hunting): authenticate will time out at 812.037816s 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d62ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7d6409d800 0x7f7d64098950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60090/0 (socket says 172.21.15.142:60090) 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 learned_addr learned my addr 172.21.15.142:0/2555004600 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d6409b5c0 auth_method 0 2024-09-18T10:08:43.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.109+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.113+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.142:6789/0 conn(0x7f7d6409d800 legacy=0x7f7d64098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.113+0000 7f7d637fe700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.150:6789/0 conn(0x7f7d64134600 legacy=0x7f7d641369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d6409b5c0 auth_method 0 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:43.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d6409b5c0 auth_method 0 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:43.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:43.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d6409b5c0 auth_method 0 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:44.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:44.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): tick 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 monclient(hunting): continuing hunt 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 msgr2=0x7f7d64099060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d6409b5c0 0x7f7d64099060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:46.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.150:6789/0 conn(0x7f7d64134600 legacy=0x7f7d641369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.142:6789/0 conn(0x7f7d6409d800 legacy=0x7f7d64098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient: _add_conns ranks=[2,4] 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): picked mon.noname-b con 0x7f7d50003860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): picked mon.noname-c con 0x7f7d500063d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6409afa0 con 0x7f7d500063d0 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d50008f00 con 0x7f7d50003860 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d60ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:46.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d50003860 auth_method 0 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d500063d0 auth_method 0 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:46.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:08:46.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.109+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:08:46.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d50003860 auth_method 0 2024-09-18T10:08:46.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d500063d0 auth_method 0 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.313+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:08:46.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d50003860 auth_method 0 2024-09-18T10:08:46.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d500063d0 auth_method 0 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:46.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:46.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:46.717+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:08:47.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:47.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d50003860 auth_method 0 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d500063d0 auth_method 0 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:08:47.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:08:47.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:47.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:47.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:47.521+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:08:48.451 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:48.447+0000 7f21e052e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T10:08:49.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): tick 2024-09-18T10:08:49.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 monclient(hunting): continuing hunt 2024-09-18T10:08:49.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:08:49.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 msgr2=0x7f7d50005ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d50003860 0x7f7d50005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 msgr2=0x7f7d50008840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d500063d0 0x7f7d50008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d637fe700 1 -- 172.21.15.142:0/2555004600 reap_dead start 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): picked mon.noname-c con 0x7f7d50003c90 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d50006ce0 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d50008930 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6409c2a0 con 0x7f7d50006ce0 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5000fc40 con 0x7f7d50008930 2024-09-18T10:08:49.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 1 -- 172.21.15.142:0/2555004600 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6409afa0 con 0x7f7d50003c90 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d60ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d50003c90 auth_method 0 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_done global_id 4102 payload 274 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.150:6789/0 conn(0x7f7d50008930 legacy=0x7f7d5000d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> v1:172.21.15.142:6789/0 conn(0x7f7d50006ce0 legacy=0x7f7d50007180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 1 monclient: found mon.noname-c 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.109+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d50010ea0 con 0x7f7d50003c90 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.113+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 msgr2=0x7f7d500067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.113+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 msgr2=0x7f7d500067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.113+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f7d64098590 tx=0x7f7d54009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:49.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.113+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:08:49.251 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.247+0000 7f21e052e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T10:08:49.308 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.303+0000 7f21e052e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient: get_auth_request con 0x7f7d50003c90 auth_method 0 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 0 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient: ms_handle_reset current mon v2:172.21.15.150:3300/0 2024-09-18T10:08:49.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 msgr2=0x7f7d500067a0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.150:3300/0 conn(0x7f7d50003c90 0x7f7d500067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient: _add_conns ranks=[3,2] 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d4c003550 0x7f7d4c0059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c003550 addr v2:172.21.15.142:3300/0 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d4c006290 0x7f7d4c008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f7d4c006290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c008fe0 con 0x7f7d4c003550 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c009150 con 0x7f7d4c006290 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d61ffb700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.150:3300/0 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> v2:172.21.15.142:3300/0 conn(0x7f7d4c003550 0x7f7d4c0059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d4c006290 0x7f7d4c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d4c003550 auth_method 0 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d4c006290 auth_method 0 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d4c006290 msgr2=0x7f7d4c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d4c006290 0x7f7d4c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 monclient: found mon.noname-a 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c0093d0 con 0x7f7d4c003550 2024-09-18T10:08:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.317+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d4c006290 0x7f7d4c008740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:08:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d62ffd700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 msgr2=0x7f7d4c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d62ffd700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d4c0059c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7d500063a0 tx=0x7f7d54009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33f42288f268dd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54001750 con 0x7f7d4c003550 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: dump: 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: _finish_auth 0 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 5 monclient: authenticate success, global_id 4102 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d540018b0 con 0x7f7d4c003550 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 4 set_mon_vals no callback set 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 1 -- 172.21.15.142:0/2555004600 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54001c70 con 0x7f7d4c003550 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:08:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d61ffb700 10 monclient: dump: 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 10 monclient: shutdown 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 -- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 msgr2=0x7f7d4c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d4c0059c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7d500063a0 tx=0x7f7d54009040 comp rx=0 tx=0).stop 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d637fe700 1 -- 172.21.15.142:0/2555004600 reap_dead start 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 -- 172.21.15.142:0/2555004600 shutdown_connections 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2555004600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d4c0059c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 -- 172.21.15.142:0/2555004600 >> 172.21.15.142:0/2555004600 conn(0x7f7d64070ef0 msgr2=0x7f7d640712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 -- 172.21.15.142:0/2555004600 shutdown_connections 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 1 -- 172.21.15.142:0/2555004600 wait complete. 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 10 monclient: build_initial_monmap 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.321+0000 7f7d63fff700 10 monclient: monmap: 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:08:49.331 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:49.332 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:49.332 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:49.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:49.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:49.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:49.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 Processor -- start 2024-09-18T10:08:49.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 -- start start 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient: init 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d64094320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-a con 0x7f7d4c003550 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-c con 0x7f7d64094860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6408e5c0 0x7f7d6408e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): picked mon.noname-b con 0x7f7d6408e5c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d64094c30 con 0x7f7d4c003550 2024-09-18T10:08:49.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6406f720 con 0x7f7d6408e5c0 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d64096490 con 0x7f7d64094860 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d63fff700 10 monclient(hunting): authenticate will time out at 818.253139s 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d64094320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42398/0 (socket says 172.21.15.142:42398) 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 -- 172.21.15.142:0/2406858936 learned_addr learned my addr 172.21.15.142:0/2406858936 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): get_auth_request con 0x7f7d64094860 auth_method 0 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request con 0x7f7d4c003550 auth_method 0 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6408e5c0 0x7f7d6408e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): get_auth_request con 0x7f7d6408e5c0 auth_method 0 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 -- 172.21.15.142:0/2406858936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6408e5c0 msgr2=0x7f7d6408e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6408e5c0 0x7f7d6408e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 -- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 msgr2=0x7f7d64094320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d64094320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 monclient: found mon.noname-c 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.325+0000 7f7d627fc700 1 -- 172.21.15.142:0/2406858936 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64091b00 con 0x7f7d64094860 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d627fc700 1 -- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 msgr2=0x7f7d6408e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d627fc700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c015650 tx=0x7f7d5c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c015e50 con 0x7f7d64094860 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: dump: 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: _finish_auth 0 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 5 monclient: authenticate success, global_id 4103 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c020780 con 0x7f7d64094860 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c020b40 con 0x7f7d64094860 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:08:49.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 10 monclient: dump: 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d617fa700 4 set_mon_vals no callback set 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _renew_subs 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6408fa10 con 0x7f7d64094860 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _renew_subs 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6408ef10 con 0x7f7d64094860 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f7d5c003040 con 0x7f7d64094860 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7d5c021400 con 0x7f7d64094860 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.329+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d64072a80 con 0x7f7d64094860 2024-09-18T10:08:49.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.337+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f7d5c02ae60 con 0x7f7d64094860 2024-09-18T10:08:49.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.337+0000 7f7d4affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.337+0000 7f7d4affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:49.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.409+0000 7f7d63fff700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T10:08:49.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.409+0000 7f7d63fff700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T10:08:49.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.409+0000 7f7d63fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.409+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7d6409a570 con 0x7f7d64094860 2024-09-18T10:08:49.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d4affd700 1 -- 172.21.15.142:0/2406858936 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7d5c02a4a0 con 0x7f7d64094860 2024-09-18T10:08:49.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d4affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-18T10:08:49.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d4affd700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 10 monclient: shutdown 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 msgr2=0x7f7d6408e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c015650 tx=0x7f7d5c00b040 comp rx=0 tx=0).stop 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 shutdown_connections 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6408e5c0 0x7f7d6408e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d64094860 0x7f7d6408e080 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 --2- 172.21.15.142:0/2406858936 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d4c003550 0x7f7d64094320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 >> 172.21.15.142:0/2406858936 conn(0x7f7d64070ef0 msgr2=0x7f7d6409c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:49.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 shutdown_connections 2024-09-18T10:08:49.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:49.417+0000 7f7d63fff700 1 -- 172.21.15.142:0/2406858936 wait complete. 2024-09-18T10:08:49.423 INFO:teuthology.orchestra.run.smithi142.stderr:adjusted tunables profile to default 2024-09-18T10:08:49.424 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.419+0000 7f21e052e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T10:08:49.436 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T10:08:49.437 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:49.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T10:08:49.447 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:49.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T10:08:49.461 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-09-18T10:08:49.461 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T10:08:49.474 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:49.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T10:08:49.489 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:49.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-18T10:08:49.548 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:08:49.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-18T10:08:49.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 5aa843cd-ccc8-4450-87dd-76691eac5d04 0 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: get_monmap_and_config 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: build_initial_monmap 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: monmap: 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:49.657906+0000 2024-09-18T10:08:49.660 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:49.657906+0000 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:49.661 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 Processor -- start 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 -- start start 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: init 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:08:49.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f664809c910 0x7f6648096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-a con 0x7f664809c910 addr v2:172.21.15.142:3300/0 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6648098790 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-c con 0x7f6648098b90 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664809e390 con 0x7f6648098790 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664809e610 con 0x7f664809c910 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6648097ad0 con 0x7f6648098b90 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f664d39c700 10 monclient(hunting): authenticate will time out at 811.751720s 2024-09-18T10:08:49.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f66477fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.655+0000 7f66477fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50424/0 (socket says 172.21.15.150:50424) 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 learned_addr learned my addr 172.21.15.150:0/2557278340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): get_auth_request con 0x7f6648098b90 auth_method 0 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.142:3300/0 conn(0x7f664809c910 0x7f6648096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 10 monclient(hunting): get_auth_request con 0x7f664809c910 auth_method 0 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 >> v2:172.21.15.142:3300/0 conn(0x7f664809c910 msgr2=0x7f6648096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.142:3300/0 conn(0x7f664809c910 0x7f6648096fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 >> v1:172.21.15.142:6789/0 conn(0x7f6648098790 legacy=0x7f664809cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 monclient: found mon.noname-c 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664809d3f0 con 0x7f6648098b90 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f6646ffd700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.142:3300/0 conn(0x7f664809c910 0x7f6648096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 msgr2=0x7f6648135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 msgr2=0x7f6648135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:08:49.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f664809de80 tx=0x7f6640009840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:08:49.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.659+0000 7f66477fe700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:08:49.668 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.663+0000 7f21e052e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T10:08:49.716 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.711+0000 7f21e052e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T10:08:49.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f66477fe700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f66477fe700 10 monclient: get_auth_request con 0x7f6648098b90 auth_method 0 2024-09-18T10:08:49.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f66477fe700 0 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:08:49.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f66477fe700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient: ms_handle_reset current mon v2:172.21.15.150:3300/0 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 msgr2=0x7f6648135b40 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 1 --2- 172.21.15.150:0/2557278340 >> v2:172.21.15.150:3300/0 conn(0x7f6648098b90 0x7f6648135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient: _add_conns ranks=[2,1] 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 1 --2- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 0x7f6630005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f6630003550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6630006300 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663000a2a0 con 0x7f6630006300 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663000b4b0 con 0x7f6630003550 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6645ffb700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.150:3300/0 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 1 --2- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 0x7f6630005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): get_auth_request con 0x7f6630003550 auth_method 0 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient(hunting): handle_auth_done global_id 4108 payload 293 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 1 -- 172.21.15.150:0/2557278340 >> v1:172.21.15.150:6789/0 conn(0x7f6630006300 legacy=0x7f66300087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 1 monclient: found mon.noname-b 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:08:49.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.863+0000 7f6646ffd700 1 -- 172.21.15.150:0/2557278340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663000b730 con 0x7f6630003550 2024-09-18T10:08:49.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6646ffd700 1 -- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 msgr2=0x7f6630005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6646ffd700 1 --2- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 0x7f6630005990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f663800dc20 tx=0x7f6638008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b4be68abaec3005 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6646ffd700 1 --2- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 0x7f6630005990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f663800dc20 tx=0x7f6638008040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f663800fce0 mon_map magic: 0 v1 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663800f7c0 con 0x7f6630003550 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: dump: 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.872 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: _finish_auth 0 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663800f920 con 0x7f6630003550 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 5 monclient: authenticate success, global_id 4108 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:49.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 4 set_mon_vals no callback set 2024-09-18T10:08:49.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 10 monclient: shutdown 2024-09-18T10:08:49.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:49.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 -- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 msgr2=0x7f6630005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 --2- 172.21.15.150:0/2557278340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6630003550 0x7f6630005990 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f663800dc20 tx=0x7f6638008040 comp rx=0 tx=0).stop 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f66477fe700 1 -- 172.21.15.150:0/2557278340 reap_dead start 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 1 -- 172.21.15.150:0/2557278340 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f663800fce0 con 0x7f6630003550 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f6645ffb700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 -- 172.21.15.150:0/2557278340 shutdown_connections 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 -- 172.21.15.150:0/2557278340 >> 172.21.15.150:0/2557278340 conn(0x7f66480717f0 msgr2=0x7f66480a1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 -- 172.21.15.150:0/2557278340 shutdown_connections 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.867+0000 7f664d39c700 1 -- 172.21.15.150:0/2557278340 wait complete. 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient: build_initial_monmap 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient: monmap: 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:49.876 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 Processor -- start 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 -- start start 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient: init 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f664813e4d0 0x7f664813db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-c con 0x7f664813e4d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:49.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-b con 0x7f6648121d50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f664813ee50 0x7f6648141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): picked mon.noname-a con 0x7f664813ee50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66481234c0 con 0x7f664813ee50 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664813e8a0 con 0x7f6648121d50 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6648122f00 con 0x7f664813e4d0 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f664d39c700 10 monclient(hunting): authenticate will time out at 811.966499s 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50442/0 (socket says 172.21.15.150:50442) 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 -- 172.21.15.150:0/361004090 learned_addr learned my addr 172.21.15.150:0/361004090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): get_auth_request con 0x7f6648121d50 auth_method 0 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f664813ee50 0x7f6648141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f664813e4d0 0x7f664813db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): get_auth_request con 0x7f664813e4d0 auth_method 0 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): get_auth_request con 0x7f664813ee50 auth_method 0 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f6646ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 -- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f664813e4d0 msgr2=0x7f664813db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f664813e4d0 0x7f664813db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 -- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f664813ee50 msgr2=0x7f6648141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f664813ee50 0x7f6648141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient: _finish_hunting 0 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 monclient: found mon.noname-b 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.871+0000 7f66467fc700 1 -- 172.21.15.150:0/361004090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66481209e0 con 0x7f6648121d50 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f66467fc700 1 -- 172.21.15.150:0/361004090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 msgr2=0x7f6648141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f66467fc700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f663c00c5e0 tx=0x7f663c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00ce70 con 0x7f6648121d50 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: dump: 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: _finish_auth 0 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 5 monclient: authenticate success, global_id 4111 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _renew_subs 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6648122c20 con 0x7f6648121d50 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _renew_subs 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664813e0c0 con 0x7f6648121d50 2024-09-18T10:08:49.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c0117a0 con 0x7f6648121d50 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c011b60 con 0x7f6648121d50 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f66457fa700 4 set_mon_vals no callback set 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 10 monclient: dump: 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:49.887 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f663c01c460 con 0x7f6648121d50 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f663c012770 con 0x7f6648121d50 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.875+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66280050b0 con 0x7f6648121d50 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.879+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f663c01c8a0 con 0x7f6648121d50 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.879+0000 7f662effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.879+0000 7f662effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:49.900 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.895+0000 7f21e052e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T10:08:49.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.955+0000 7f664d39c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5aa843cd-ccc8-4450-87dd-76691eac5d04", "id": 0}] 2024-09-18T10:08:49.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.955+0000 7f664d39c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5aa843cd-ccc8-4450-87dd-76691eac5d04", "id": 0}] 2024-09-18T10:08:49.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.955+0000 7f664d39c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:49.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.955+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5aa843cd-ccc8-4450-87dd-76691eac5d04", "id": 0} v 0) v1 -- 0x7f66280053d0 con 0x7f6648121d50 2024-09-18T10:08:49.962 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:49.955+0000 7f21e052e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T10:08:49.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f662effd700 1 -- 172.21.15.150:0/361004090 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5aa843cd-ccc8-4450-87dd-76691eac5d04", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f663c01ca40 con 0x7f6648121d50 2024-09-18T10:08:49.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f662effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5aa843cd-ccc8-4450-87dd-76691eac5d04", "id": 0}] 2024-09-18T10:08:49.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f662effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:49.968 INFO:teuthology.orchestra.run.smithi150.stdout:0 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f664d39c700 10 monclient: shutdown 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f664d39c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 msgr2=0x7f6648141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.963+0000 7f664d39c700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f663c00c5e0 tx=0x7f663c007f60 comp rx=0 tx=0).stop 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 shutdown_connections 2024-09-18T10:08:49.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f664813ee50 0x7f6648141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6648121d50 0x7f6648141a40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 --2- 172.21.15.150:0/361004090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f664813e4d0 0x7f664813db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:49.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 >> 172.21.15.150:0/361004090 conn(0x7f66480717f0 msgr2=0x7f664809a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:49.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 shutdown_connections 2024-09-18T10:08:49.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:49.967+0000 7f664d39c700 1 -- 172.21.15.150:0/361004090 wait complete. 2024-09-18T10:08:49.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 9bedda37-ff2e-47a0-9fa0-1b62b9b934cb 1 2024-09-18T10:08:50.042 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:50.035+0000 7f21e052e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: get_monmap_and_config 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: monmap: 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:50.080811+0000 2024-09-18T10:08:50.083 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:50.080811+0000 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:50.084 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 Processor -- start 2024-09-18T10:08:50.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 -- start start 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: init 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44d009cef0 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f44d009e4a0 0x7f44d00a0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-a con 0x7f44d009e4a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44d009f9c0 0x7f44d0139990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-c con 0x7f44d009f9c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d009dac0 con 0x7f44d009cef0 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00a1660 con 0x7f44d009e4a0 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d009ed30 con 0x7f44d009f9c0 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d7239700 10 monclient(hunting): authenticate will time out at 812.174263s 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6237700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f44d009cef0 0x7f44d00a01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34454/0 (socket says 172.21.15.150:34454) 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44d009f9c0 0x7f44d0139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6237700 1 -- 172.21.15.150:0/4164552631 learned_addr learned my addr 172.21.15.150:0/4164552631 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): get_auth_request con 0x7f44d009f9c0 auth_method 0 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d5a36700 1 --2- 172.21.15.150:0/4164552631 >> v2:172.21.15.142:3300/0 conn(0x7f44d009e4a0 0x7f44d00a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d5a36700 10 monclient(hunting): get_auth_request con 0x7f44d009e4a0 auth_method 0 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d5a36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d5a36700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d5a36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 -- 172.21.15.150:0/4164552631 >> v2:172.21.15.142:3300/0 conn(0x7f44d009e4a0 msgr2=0x7f44d00a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 --2- 172.21.15.150:0/4164552631 >> v2:172.21.15.142:3300/0 conn(0x7f44d009e4a0 0x7f44d00a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 -- 172.21.15.150:0/4164552631 >> v1:172.21.15.150:6789/0 conn(0x7f44d009cef0 legacy=0x7f44d00a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 monclient: found mon.noname-c 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.079+0000 7f44d6a38700 1 -- 172.21.15.150:0/4164552631 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d009d7e0 con 0x7f44d009f9c0 2024-09-18T10:08:50.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d6a38700 1 -- 172.21.15.150:0/4164552631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009f9c0 msgr2=0x7f44d0139990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d6a38700 1 --2- 172.21.15.150:0/4164552631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009f9c0 0x7f44d0139990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f44d0067ab0 tx=0x7f44c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18a9c029fe676517 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 1 -- 172.21.15.150:0/4164552631 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c800e070 con 0x7f44d009f9c0 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: dump: 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.092 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 5 monclient: authenticate success, global_id 4114 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 1 -- 172.21.15.150:0/4164552631 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c800be40 con 0x7f44d009f9c0 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 4 set_mon_vals no callback set 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 1 -- 172.21.15.150:0/4164552631 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8012660 con 0x7f44d009f9c0 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:50.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d5235700 10 monclient: dump: 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: shutdown 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:50.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- 172.21.15.150:0/4164552631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009f9c0 msgr2=0x7f44d0139990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- 172.21.15.150:0/4164552631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009f9c0 0x7f44d0139990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f44d0067ab0 tx=0x7f44c8009780 comp rx=0 tx=0).stop 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- 172.21.15.150:0/4164552631 shutdown_connections 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- 172.21.15.150:0/4164552631 >> v2:172.21.15.142:3300/0 conn(0x7f44d009e4a0 0x7f44d00a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- 172.21.15.150:0/4164552631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009f9c0 0x7f44d0139990 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- 172.21.15.150:0/4164552631 >> 172.21.15.150:0/4164552631 conn(0x7f44d00a5800 msgr2=0x7f44d00a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- 172.21.15.150:0/4164552631 shutdown_connections 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- 172.21.15.150:0/4164552631 wait complete. 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: monmap: 2024-09-18T10:08:50.095 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:50.096 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 Processor -- start 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- start start 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: init 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 0x7f44d008de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-c con 0x7f44d009cef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-b con 0x7f44d009e4a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 0x7f44d008e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): picked mon.noname-a con 0x7f44d009f9c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00987d0 con 0x7f44d009f9c0 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d008fd00 con 0x7f44d009e4a0 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d009bfb0 con 0x7f44d009cef0 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.083+0000 7f44d7239700 10 monclient(hunting): authenticate will time out at 812.179943s 2024-09-18T10:08:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50456/0 (socket says 172.21.15.150:50456) 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 -- 172.21.15.150:0/2696576034 learned_addr learned my addr 172.21.15.150:0/2696576034 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): get_auth_request con 0x7f44d009e4a0 auth_method 0 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 0x7f44d008de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 10 monclient(hunting): get_auth_request con 0x7f44d009cef0 auth_method 0 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 0x7f44d008e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 10 monclient(hunting): get_auth_request con 0x7f44d009f9c0 auth_method 0 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 -- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 msgr2=0x7f44d008de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 0x7f44d008de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 -- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 msgr2=0x7f44d008e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 0x7f44d008e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 monclient: found mon.noname-b 2024-09-18T10:08:50.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 -- 172.21.15.150:0/2696576034 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8009a20 con 0x7f44d009e4a0 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6237700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 0x7f44d008de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d6a38700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 0x7f44d008e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 -- 172.21.15.150:0/2696576034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 msgr2=0x7f44d008e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d5a36700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f44c000d890 tx=0x7f44c0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c000b810 con 0x7f44d009e4a0 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: dump: 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.104 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 5 monclient: authenticate success, global_id 4117 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c000b970 con 0x7f44d009e4a0 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 10 monclient: _renew_subs 2024-09-18T10:08:50.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d0091240 con 0x7f44d009e4a0 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c000bcf0 con 0x7f44d009e4a0 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d4a34700 4 set_mon_vals no callback set 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 10 monclient: dump: 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.106 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 10 monclient: _renew_subs 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d00993f0 con 0x7f44d009e4a0 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f44c00148d0 con 0x7f44d009e4a0 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.087+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f44c001e550 con 0x7f44d009e4a0 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.095+0000 7f44d7239700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.095+0000 7f44d7239700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.095+0000 7f44d7239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.095+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0072a80 con 0x7f44d009e4a0 2024-09-18T10:08:50.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.095+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f44c0014d10 con 0x7f44d009e4a0 2024-09-18T10:08:50.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.099+0000 7f44c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.099+0000 7f44c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.175+0000 7f44d7239700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9bedda37-ff2e-47a0-9fa0-1b62b9b934cb", "id": 1}] 2024-09-18T10:08:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.175+0000 7f44d7239700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9bedda37-ff2e-47a0-9fa0-1b62b9b934cb", "id": 1}] 2024-09-18T10:08:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.175+0000 7f44d7239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.175+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9bedda37-ff2e-47a0-9fa0-1b62b9b934cb", "id": 1} v 0) v1 -- 0x7f44d0067790 con 0x7f44d009e4a0 2024-09-18T10:08:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44c67fc700 1 -- 172.21.15.150:0/2696576034 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9bedda37-ff2e-47a0-9fa0-1b62b9b934cb", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f44c001e830 con 0x7f44d009e4a0 2024-09-18T10:08:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9bedda37-ff2e-47a0-9fa0-1b62b9b934cb", "id": 1}] 2024-09-18T10:08:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44c67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:50.188 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 10 monclient: shutdown 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 msgr2=0x7f44d008e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f44c000d890 tx=0x7f44c0008040 comp rx=0 tx=0).stop 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 shutdown_connections 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44d009f9c0 0x7f44d008e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44d009e4a0 0x7f44d008e3a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 --2- 172.21.15.150:0/2696576034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44d009cef0 0x7f44d008de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 >> 172.21.15.150:0/2696576034 conn(0x7f44d00a5800 msgr2=0x7f44d009a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 shutdown_connections 2024-09-18T10:08:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.183+0000 7f44d7239700 1 -- 172.21.15.150:0/2696576034 wait complete. 2024-09-18T10:08:50.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 1097bd95-2d11-48af-b727-df403e6983a8 2 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: get_monmap_and_config 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: monmap: 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:50.308870+0000 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:50.308870+0000 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.311 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:50.312 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 Processor -- start 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 -- start start 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: init 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 0x7ff11012f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-b con 0x7ff110099af0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff11009c120 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:50.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff11009c550 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110099ec0 con 0x7ff11009c550 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100960e0 con 0x7ff11009c120 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11012f020 con 0x7ff110099af0 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff1166a1700 10 monclient(hunting): authenticate will time out at 812.403301s 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff114e9e700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff11009c120 0x7ff1100952c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34460/0 (socket says 172.21.15.150:34460) 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff114e9e700 1 -- 172.21.15.150:0/1509068984 learned_addr learned my addr 172.21.15.150:0/1509068984 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 1 --2- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 0x7ff11012f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): get_auth_request con 0x7ff110099af0 auth_method 0 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.307+0000 7ff11569f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 -- 172.21.15.150:0/1509068984 >> v1:172.21.15.150:6789/0 conn(0x7ff11009c120 legacy=0x7ff1100952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 -- 172.21.15.150:0/1509068984 >> v1:172.21.15.142:6789/0 conn(0x7ff11009c550 legacy=0x7ff1100959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 monclient: found mon.noname-b 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:08:50.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 -- 172.21.15.150:0/1509068984 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11009a350 con 0x7ff110099af0 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 -- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 msgr2=0x7ff11012f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff11569f700 1 --2- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 0x7ff11012f360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009910 tx=0x7ff10c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c911dff644cb8e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 1 -- 172.21.15.150:0/1509068984 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00ba70 con 0x7ff110099af0 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: dump: 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.319 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 1 -- 172.21.15.150:0/1509068984 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c00bbd0 con 0x7ff110099af0 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 1 -- 172.21.15.150:0/1509068984 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c011420 con 0x7ff110099af0 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:08:50.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff107fff700 10 monclient: dump: 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff1166a1700 5 monclient: authenticate success, global_id 4112 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff1166a1700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:50.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.311+0000 7ff1166a1700 4 set_mon_vals no callback set 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: shutdown 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 msgr2=0x7ff11012f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 --2- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 0x7ff11012f360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009910 tx=0x7ff10c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- 172.21.15.150:0/1509068984 shutdown_connections 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 --2- 172.21.15.150:0/1509068984 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff110099af0 0x7ff11012f360 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- 172.21.15.150:0/1509068984 >> 172.21.15.150:0/1509068984 conn(0x7ff1100a25a0 msgr2=0x7ff1100a2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- 172.21.15.150:0/1509068984 shutdown_connections 2024-09-18T10:08:50.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- 172.21.15.150:0/1509068984 wait complete. 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: monmap: 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 Processor -- start 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- start start 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: init 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 0x7ff11012dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-a con 0x7ff110099af0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-b con 0x7ff11009c120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff11009c550 0x7ff110071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): picked mon.noname-c con 0x7ff11009c550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11012b7f0 con 0x7ff110099af0 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11012d040 con 0x7ff11009c120 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11009d720 con 0x7ff11009c550 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff1166a1700 10 monclient(hunting): authenticate will time out at 812.411151s 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50464/0 (socket says 172.21.15.150:50464) 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 -- 172.21.15.150:0/722025907 learned_addr learned my addr 172.21.15.150:0/722025907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): get_auth_request con 0x7ff11009c120 auth_method 0 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 0x7ff11012dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff115ea0700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff11009c550 0x7ff110071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 10 monclient(hunting): get_auth_request con 0x7ff110099af0 auth_method 0 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff115ea0700 10 monclient(hunting): get_auth_request con 0x7ff11009c550 auth_method 0 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff115ea0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff115ea0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff115ea0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 -- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff11009c550 msgr2=0x7ff110071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff11009c550 0x7ff110071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 -- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 msgr2=0x7ff11012dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 0x7ff11012dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 monclient: found mon.noname-b 2024-09-18T10:08:50.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff114e9e700 1 -- 172.21.15.150:0/722025907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c009420 con 0x7ff11009c120 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.315+0000 7ff11569f700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 0x7ff11012dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff114e9e700 1 -- 172.21.15.150:0/722025907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 msgr2=0x7ff110070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff114e9e700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c5e0 tx=0x7ff100004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100005d40 con 0x7ff11009c120 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: dump: 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 5 monclient: authenticate success, global_id 4120 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1000070d0 con 0x7ff11009c120 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff100007490 con 0x7ff11009c120 2024-09-18T10:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 10 monclient: _renew_subs 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11012cd60 con 0x7ff11009c120 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1077fe700 4 set_mon_vals no callback set 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 10 monclient: dump: 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 10 monclient: _renew_subs 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11006ee80 con 0x7ff11009c120 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff10001c440 con 0x7ff11009c120 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ff100012830 con 0x7ff11009c120 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.319+0000 7ff1166a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1166a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1166a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff110072a80 con 0x7ff11009c120 2024-09-18T10:08:50.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7ff100017070 con 0x7ff11009c120 2024-09-18T10:08:50.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.323+0000 7ff1057fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.395+0000 7ff1166a1700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1097bd95-2d11-48af-b727-df403e6983a8", "id": 2}] 2024-09-18T10:08:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.395+0000 7ff1166a1700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1097bd95-2d11-48af-b727-df403e6983a8", "id": 2}] 2024-09-18T10:08:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.395+0000 7ff1166a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.395+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1097bd95-2d11-48af-b727-df403e6983a8", "id": 2} v 0) v1 -- 0x7ff1101292d0 con 0x7ff11009c120 2024-09-18T10:08:50.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1057fa700 1 -- 172.21.15.150:0/722025907 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1097bd95-2d11-48af-b727-df403e6983a8", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff100012400 con 0x7ff11009c120 2024-09-18T10:08:50.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1097bd95-2d11-48af-b727-df403e6983a8", "id": 2}] 2024-09-18T10:08:50.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1057fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:50.409 INFO:teuthology.orchestra.run.smithi150.stdout:2 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 10 monclient: shutdown 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 msgr2=0x7ff110070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff10000c5e0 tx=0x7ff100004a80 comp rx=0 tx=0).stop 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 shutdown_connections 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff11009c550 0x7ff110071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff11009c120 0x7ff110070ef0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 --2- 172.21.15.150:0/722025907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff110099af0 0x7ff11012dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.403+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 >> 172.21.15.150:0/722025907 conn(0x7ff1100a25a0 msgr2=0x7ff1100962d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:50.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.407+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 shutdown_connections 2024-09-18T10:08:50.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.407+0000 7ff1166a1700 1 -- 172.21.15.150:0/722025907 wait complete. 2024-09-18T10:08:50.420 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new d9381f04-413d-4382-a0b1-b73fca6b0d11 3 2024-09-18T10:08:50.457 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:50.451+0000 7f21e052e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T10:08:50.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.511+0000 7f164107b700 10 monclient: get_monmap_and_config 2024-09-18T10:08:50.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.511+0000 7f164107b700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.511+0000 7f164107b700 10 monclient: monmap: 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:50.515999+0000 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:50.515999+0000 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 Processor -- start 2024-09-18T10:08:50.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 -- start start 2024-09-18T10:08:50.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient: init 2024-09-18T10:08:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:08:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f163c09d800 0x7f163c098950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): picked mon.noname-a con 0x7f163c09d800 addr v2:172.21.15.142:3300/0 2024-09-18T10:08:50.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f163c09b5c0 0x7f163c098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): picked mon.noname-c con 0x7f163c09b5c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f163c0993d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c06fb70 con 0x7f163c0993d0 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c09afa0 con 0x7f163c09d800 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c09dbd0 con 0x7f163c09b5c0 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f164107b700 10 monclient(hunting): authenticate will time out at 812.609070s 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f163c09b5c0 0x7f163c098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f163c09b5c0 0x7f163c098e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50468/0 (socket says 172.21.15.150:50468) 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 -- 172.21.15.150:0/3101221571 learned_addr learned my addr 172.21.15.150:0/3101221571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:50.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): get_auth_request con 0x7f163c09b5c0 auth_method 0 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 1 --2- 172.21.15.150:0/3101221571 >> v2:172.21.15.142:3300/0 conn(0x7f163c09d800 0x7f163c098950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): get_auth_request con 0x7f163c09d800 auth_method 0 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 -- 172.21.15.150:0/3101221571 >> v2:172.21.15.142:3300/0 conn(0x7f163c09d800 msgr2=0x7f163c098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 --2- 172.21.15.150:0/3101221571 >> v2:172.21.15.142:3300/0 conn(0x7f163c09d800 0x7f163c098950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 -- 172.21.15.150:0/3101221571 >> v1:172.21.15.142:6789/0 conn(0x7f163c0993d0 legacy=0x7f163c135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.515+0000 7f163b7fe700 1 -- 172.21.15.150:0/3101221571 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c09c2a0 con 0x7f163c09b5c0 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 -- 172.21.15.150:0/3101221571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 msgr2=0x7f163c098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 --2- 172.21.15.150:0/3101221571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c098e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f163c06fd60 tx=0x7f162c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47db22b0e819ea78 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 1 -- 172.21.15.150:0/3101221571 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c009e70 con 0x7f163c09b5c0 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: dump: 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 5 monclient: authenticate success, global_id 4123 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 1 -- 172.21.15.150:0/3101221571 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00ba70 con 0x7f163c09b5c0 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 4 set_mon_vals no callback set 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 1 -- 172.21.15.150:0/3101221571 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00be30 con 0x7f163c09b5c0 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163affd700 10 monclient: dump: 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.527 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.528 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: shutdown 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- 172.21.15.150:0/3101221571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 msgr2=0x7f163c098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- 172.21.15.150:0/3101221571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c098e90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f163c06fd60 tx=0x7f162c009250 comp rx=0 tx=0).stop 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- 172.21.15.150:0/3101221571 shutdown_connections 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- 172.21.15.150:0/3101221571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c098e90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- 172.21.15.150:0/3101221571 >> v2:172.21.15.142:3300/0 conn(0x7f163c09d800 0x7f163c098950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- 172.21.15.150:0/3101221571 >> 172.21.15.150:0/3101221571 conn(0x7f163c070ef0 msgr2=0x7f163c0712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- 172.21.15.150:0/3101221571 shutdown_connections 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- 172.21.15.150:0/3101221571 wait complete. 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: build_initial_monmap 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: monmap: 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:50.533 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 Processor -- start 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- start start 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: init 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f163c0993d0 0x7f163c141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): picked mon.noname-a con 0x7f163c0993d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): picked mon.noname-b con 0x7f163c09b5c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f163c09d800 0x7f163c08def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): picked mon.noname-c con 0x7f163c09d800 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c06f720 con 0x7f163c0993d0 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c09f6d0 con 0x7f163c09b5c0 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c142640 con 0x7f163c09d800 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164107b700 10 monclient(hunting): authenticate will time out at 812.614466s 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50482/0 (socket says 172.21.15.150:50482) 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 -- 172.21.15.150:0/90159030 learned_addr learned my addr 172.21.15.150:0/90159030 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): get_auth_request con 0x7f163c09b5c0 auth_method 0 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163bfff700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f163c0993d0 0x7f163c141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164087a700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f163c09d800 0x7f163c08def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163bfff700 10 monclient(hunting): get_auth_request con 0x7f163c0993d0 auth_method 0 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164087a700 10 monclient(hunting): get_auth_request con 0x7f163c09d800 auth_method 0 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164087a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164087a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f164087a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 -- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f163c09d800 msgr2=0x7f163c08def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f163c09d800 0x7f163c08def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 -- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f163c0993d0 msgr2=0x7f163c141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f163c0993d0 0x7f163c141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.519+0000 7f163b7fe700 1 -- 172.21.15.150:0/90159030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c0094f0 con 0x7f163c09b5c0 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f163b7fe700 1 -- 172.21.15.150:0/90159030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 msgr2=0x7f163c0942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f163b7fe700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f162c006fd0 tx=0x7f162c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c006a90 con 0x7f163c09b5c0 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: dump: 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.538 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: _finish_auth 0 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 5 monclient: authenticate success, global_id 4126 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c011db0 con 0x7f163c09b5c0 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _renew_subs 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c0917e0 con 0x7f163c09b5c0 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c01a680 con 0x7f163c09b5c0 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:50.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f163a7fc700 4 set_mon_vals no callback set 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 10 monclient: dump: 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:50.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _renew_subs 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c141ca0 con 0x7f163c09b5c0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f162c01aa70 con 0x7f163c09b5c0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f162c01ad20 con 0x7f163c09b5c0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.523+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c091970 con 0x7f163c09b5c0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.527+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f162c031060 con 0x7f163c09b5c0 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.527+0000 7f1623fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.527+0000 7f1623fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:50.586 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:50.583+0000 7f21e052e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T10:08:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.603+0000 7f164107b700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d9381f04-413d-4382-a0b1-b73fca6b0d11", "id": 3}] 2024-09-18T10:08:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.603+0000 7f164107b700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d9381f04-413d-4382-a0b1-b73fca6b0d11", "id": 3}] 2024-09-18T10:08:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.603+0000 7f164107b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:50.603+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9381f04-413d-4382-a0b1-b73fca6b0d11", "id": 3} v 0) v1 -- 0x7f163c072a80 con 0x7f163c09b5c0 2024-09-18T10:08:50.908 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:50.903+0000 7f21e052e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T10:08:51.085 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.079+0000 7f21e052e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T10:08:51.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f1623fff700 1 -- 172.21.15.150:0/90159030 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9381f04-413d-4382-a0b1-b73fca6b0d11", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f162c005a00 con 0x7f163c09b5c0 2024-09-18T10:08:51.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f1623fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d9381f04-413d-4382-a0b1-b73fca6b0d11", "id": 3}] 2024-09-18T10:08:51.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f1623fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:51.151 INFO:teuthology.orchestra.run.smithi150.stdout:3 2024-09-18T10:08:51.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 10 monclient: shutdown 2024-09-18T10:08:51.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:51.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 msgr2=0x7f163c0942e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f162c006fd0 tx=0x7f162c016000 comp rx=0 tx=0).stop 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 shutdown_connections 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f163c09d800 0x7f163c08def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163c09b5c0 0x7f163c0942e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 --2- 172.21.15.150:0/90159030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f163c0993d0 0x7f163c141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 >> 172.21.15.150:0/90159030 conn(0x7f163c070ef0 msgr2=0x7f163c06e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 shutdown_connections 2024-09-18T10:08:51.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.147+0000 7f164107b700 1 -- 172.21.15.150:0/90159030 wait complete. 2024-09-18T10:08:51.158 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.155+0000 7f21e052e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T10:08:51.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 65c1f281-564d-4628-8323-7b9adf16621f 4 2024-09-18T10:08:51.216 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.211+0000 7f21e052e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T10:08:51.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: get_monmap_and_config 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: build_initial_monmap 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: monmap: 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:51.266397+0000 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:51.266397+0000 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.269 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:51.270 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:51.270 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:51.270 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:51.270 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:51.270 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 1 Processor -- start 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 1 -- start start 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: init 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6dc09cf70 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:51.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.263+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc09e3d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc137a00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0a1df0 con 0x7fb6dc09cf70 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc09e7f0 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc09dfb0 con 0x7fb6dc137a00 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient(hunting): authenticate will time out at 813.360097s 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50490/0 (socket says 172.21.15.150:50490) 2024-09-18T10:08:51.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e248e700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb6dc09cf70 0x7fb6dc0a0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34476/0 (socket says 172.21.15.150:34476) 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 -- 172.21.15.150:0/3372863038 learned_addr learned my addr 172.21.15.150:0/3372863038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): get_auth_request con 0x7fb6dc09e3d0 auth_method 0 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e2c8f700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e2c8f700 10 monclient(hunting): get_auth_request con 0x7fb6dc137a00 auth_method 0 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e2c8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e2c8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e2c8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 -- 172.21.15.150:0/3372863038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 msgr2=0x7fb6dc139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 -- 172.21.15.150:0/3372863038 >> v1:172.21.15.150:6789/0 conn(0x7fb6dc09cf70 legacy=0x7fb6dc0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient: _finish_hunting 0 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 monclient: found mon.noname-c 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 -- 172.21.15.150:0/3372863038 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0a1650 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 -- 172.21.15.150:0/3372863038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09e3d0 msgr2=0x7fb6dc0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e1c8d700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0099e0 tx=0x7fb6cc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece36dc0daeb41ea server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 1 -- 172.21.15.150:0/3372863038 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00ba30 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: dump: 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: _finish_auth 0 2024-09-18T10:08:51.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 1 -- 172.21.15.150:0/3372863038 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00bb90 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 5 monclient: authenticate success, global_id 4129 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 1 -- 172.21.15.150:0/3372863038 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc011420 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e46f2700 4 set_mon_vals no callback set 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.267+0000 7fb6e148c700 10 monclient: dump: 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.279 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: shutdown 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- 172.21.15.150:0/3372863038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09e3d0 msgr2=0x7fb6dc0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0099e0 tx=0x7fb6cc009250 comp rx=0 tx=0).stop 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- 172.21.15.150:0/3372863038 shutdown_connections 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/3372863038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc0a0980 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- 172.21.15.150:0/3372863038 >> 172.21.15.150:0/3372863038 conn(0x7fb6dc0a5880 msgr2=0x7fb6dc0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:51.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- 172.21.15.150:0/3372863038 shutdown_connections 2024-09-18T10:08:51.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- 172.21.15.150:0/3372863038 wait complete. 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: build_initial_monmap 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: monmap: 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:51.285 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 Processor -- start 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- start start 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: init 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc09cf70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc09e3d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc137490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc137a00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc08f190 con 0x7fb6dc09e3d0 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc09bc80 con 0x7fb6dc09cf70 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0903d0 con 0x7fb6dc137a00 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e46f2700 10 monclient(hunting): authenticate will time out at 813.366269s 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50502/0 (socket says 172.21.15.150:50502) 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 -- 172.21.15.150:0/2750695997 learned_addr learned my addr 172.21.15.150:0/2750695997 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): get_auth_request con 0x7fb6dc09cf70 auth_method 0 2024-09-18T10:08:51.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e2c8f700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc137490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e1c8d700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e2c8f700 10 monclient(hunting): get_auth_request con 0x7fb6dc137a00 auth_method 0 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e2c8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e2c8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e2c8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e1c8d700 10 monclient(hunting): get_auth_request con 0x7fb6dc09e3d0 auth_method 0 2024-09-18T10:08:51.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e1c8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e1c8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e1c8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 -- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 msgr2=0x7fb6dc137490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc137490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 -- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6dc09e3d0 msgr2=0x7fb6dc136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient: _finish_hunting 0 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 monclient: found mon.noname-b 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.271+0000 7fb6e248e700 1 -- 172.21.15.150:0/2750695997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc0094f0 con 0x7fb6dc09cf70 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e248e700 1 -- 172.21.15.150:0/2750695997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 msgr2=0x7fb6dc136a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e248e700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800c5e0 tx=0x7fb6d8007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8005c80 con 0x7fb6dc09cf70 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: dump: 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.290 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: _finish_auth 0 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 5 monclient: authenticate success, global_id 4132 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d8005e00 con 0x7fb6dc09cf70 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _renew_subs 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8007320 con 0x7fb6dc09cf70 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc0900f0 con 0x7fb6dc09cf70 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e0c8b700 4 set_mon_vals no callback set 2024-09-18T10:08:51.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 10 monclient: dump: 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _renew_subs 2024-09-18T10:08:51.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc098090 con 0x7fb6dc09cf70 2024-09-18T10:08:51.293 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.275+0000 7f21e052e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb6d801c440 con 0x7fb6dc09cf70 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fb6d8012910 con 0x7fb6dc09cf70 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.275+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc08e6a0 con 0x7fb6dc09cf70 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.279+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7fb6d8017070 con 0x7fb6dc09cf70 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.279+0000 7fb6d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.279+0000 7fb6d27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:51.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.351+0000 7fb6e46f2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "65c1f281-564d-4628-8323-7b9adf16621f", "id": 4}] 2024-09-18T10:08:51.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.351+0000 7fb6e46f2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "65c1f281-564d-4628-8323-7b9adf16621f", "id": 4}] 2024-09-18T10:08:51.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.351+0000 7fb6e46f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.351+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "65c1f281-564d-4628-8323-7b9adf16621f", "id": 4} v 0) v1 -- 0x7fb6dc090510 con 0x7fb6dc09cf70 2024-09-18T10:08:51.475 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.471+0000 7f21e052e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T10:08:51.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6d27fc700 1 -- 172.21.15.150:0/2750695997 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65c1f281-564d-4628-8323-7b9adf16621f", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb6dc090510 con 0x7fb6dc09cf70 2024-09-18T10:08:51.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "65c1f281-564d-4628-8323-7b9adf16621f", "id": 4}] 2024-09-18T10:08:51.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6d27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:51.539 INFO:teuthology.orchestra.run.smithi150.stdout:4 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 10 monclient: shutdown 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 msgr2=0x7fb6dc136a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb6d800c5e0 tx=0x7fb6d8007f90 comp rx=0 tx=0).stop 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 shutdown_connections 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6dc137a00 0x7fb6dc137490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6dc09e3d0 0x7fb6dc136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 --2- 172.21.15.150:0/2750695997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6dc09cf70 0x7fb6dc136a10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 >> 172.21.15.150:0/2750695997 conn(0x7fb6dc0a5880 msgr2=0x7fb6dc09b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:51.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 shutdown_connections 2024-09-18T10:08:51.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.535+0000 7fb6e46f2700 1 -- 172.21.15.150:0/2750695997 wait complete. 2024-09-18T10:08:51.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd new 6d9435f3-4ce9-40fc-90c2-2432f38deacd 5 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: get_monmap_and_config 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: build_initial_monmap 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: monmap: 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:51.656961+0000 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:51.656961+0000 2024-09-18T10:08:51.659 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:51.660 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.660 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.655+0000 7f21e052e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 Processor -- start 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 -- start start 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: init 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:08:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff514070ef0 0x7ff5140712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-c con 0x7ff514070ef0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff514095f50 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff514071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-b con 0x7ff51409a130 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff514071d40 con 0x7ff514095f50 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51409c9c0 con 0x7ff514070ef0 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51409efd0 con 0x7ff51409a130 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff51a69f700 10 monclient(hunting): authenticate will time out at 813.750866s 2024-09-18T10:08:51.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff514070ef0 0x7ff5140712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff514070ef0 0x7ff5140712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50506/0 (socket says 172.21.15.150:50506) 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 -- 172.21.15.150:0/102377525 learned_addr learned my addr 172.21.15.150:0/102377525 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): get_auth_request con 0x7ff514070ef0 auth_method 0 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff514071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): get_auth_request con 0x7ff51409a130 auth_method 0 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 -- 172.21.15.150:0/102377525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 msgr2=0x7ff514071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff514071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 -- 172.21.15.150:0/102377525 >> v1:172.21.15.142:6789/0 conn(0x7ff514095f50 legacy=0x7ff514136870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient: _finish_hunting 0 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 monclient: found mon.noname-c 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.655+0000 7ff513fff700 1 -- 172.21.15.150:0/102377525 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51409d880 con 0x7ff514070ef0 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff513fff700 1 -- 172.21.15.150:0/102377525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 msgr2=0x7ff5140712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff513fff700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff5140712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff504009f20 tx=0x7ff504009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4de26c2560cbc9d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 1 -- 172.21.15.150:0/102377525 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400e070 con 0x7ff514070ef0 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: dump: 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.673 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: _finish_auth 0 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 5 monclient: authenticate success, global_id 4135 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:08:51.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 1 -- 172.21.15.150:0/102377525 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400bbe0 con 0x7ff514070ef0 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 4 set_mon_vals no callback set 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 1 -- 172.21.15.150:0/102377525 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5040092e0 con 0x7ff514070ef0 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff512ffd700 10 monclient: dump: 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.675 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 10 monclient: shutdown 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 -- 172.21.15.150:0/102377525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 msgr2=0x7ff5140712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff5140712c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff504009f20 tx=0x7ff504009840 comp rx=0 tx=0).stop 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 -- 172.21.15.150:0/102377525 shutdown_connections 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff514071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 --2- 172.21.15.150:0/102377525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff5140712c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 -- 172.21.15.150:0/102377525 >> 172.21.15.150:0/102377525 conn(0x7ff51406a2e0 msgr2=0x7ff5140a26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 -- 172.21.15.150:0/102377525 shutdown_connections 2024-09-18T10:08:51.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.659+0000 7ff51a69f700 1 -- 172.21.15.150:0/102377525 wait complete. 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient: build_initial_monmap 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient: monmap: 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 0 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 0.000000 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:created 0.000000 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 Processor -- start 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 -- start start 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient: init 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-b con 0x7ff514070ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 0x7ff5141420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-a con 0x7ff514095f50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff51408ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): picked mon.noname-c con 0x7ff51409a130 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5140954b0 con 0x7ff514095f50 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51409c000 con 0x7ff514070ef0 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff514090210 con 0x7ff51409a130 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff51a69f700 10 monclient(hunting): authenticate will time out at 813.758004s 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50512/0 (socket says 172.21.15.150:50512) 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 -- 172.21.15.150:0/2163696612 learned_addr learned my addr 172.21.15.150:0/2163696612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): get_auth_request con 0x7ff514070ef0 auth_method 0 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 0x7ff5141420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff51408ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): get_auth_request con 0x7ff51409a130 auth_method 0 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 10 monclient(hunting): get_auth_request con 0x7ff514095f50 auth_method 0 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:51.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff518c3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 -- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 msgr2=0x7ff51408ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff51408ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 -- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 msgr2=0x7ff5141420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 0x7ff5141420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient: _finish_hunting 0 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 monclient: found mon.noname-b 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff513fff700 1 -- 172.21.15.150:0/2163696612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504009ae0 con 0x7ff514070ef0 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.663+0000 7ff5137fe700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 0x7ff5141420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:51.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff513fff700 1 -- 172.21.15.150:0/2163696612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 msgr2=0x7ff514141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff513fff700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff50400b610 tx=0x7ff504017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504012850 con 0x7ff514070ef0 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: dump: 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.682 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: _finish_auth 0 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 5 monclient: authenticate success, global_id 4138 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _renew_subs 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040129b0 con 0x7ff514070ef0 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff514090520 con 0x7ff514070ef0 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504012db0 con 0x7ff514070ef0 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:08:51.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff5127fc700 4 set_mon_vals no callback set 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 10 monclient: dump: 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:epoch 1 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:election_strategy: 1 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _renew_subs 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51408ea60 con 0x7ff514070ef0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ff50401d8c0 con 0x7ff514070ef0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7ff50401db70 con 0x7ff514070ef0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.667+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5140696a0 con 0x7ff514070ef0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.671+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7ff504017070 con 0x7ff514070ef0 2024-09-18T10:08:51.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.671+0000 7ff50bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:51.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.671+0000 7ff50bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:51.733 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.727+0000 7f21e052e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T10:08:51.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.747+0000 7ff51a69f700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d9435f3-4ce9-40fc-90c2-2432f38deacd", "id": 5}] 2024-09-18T10:08:51.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.747+0000 7ff51a69f700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d9435f3-4ce9-40fc-90c2-2432f38deacd", "id": 5}] 2024-09-18T10:08:51.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.747+0000 7ff51a69f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:08:51.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:51.747+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6d9435f3-4ce9-40fc-90c2-2432f38deacd", "id": 5} v 0) v1 -- 0x7ff51408f630 con 0x7ff514070ef0 2024-09-18T10:08:51.908 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.903+0000 7f21e052e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T10:08:51.983 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:51.979+0000 7f21e052e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T10:08:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff50bfff700 1 -- 172.21.15.150:0/2163696612 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d9435f3-4ce9-40fc-90c2-2432f38deacd", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff50401de00 con 0x7ff514070ef0 2024-09-18T10:08:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff50bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d9435f3-4ce9-40fc-90c2-2432f38deacd", "id": 5}] 2024-09-18T10:08:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff50bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:52.160 INFO:teuthology.orchestra.run.smithi150.stdout:5 2024-09-18T10:08:52.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 10 monclient: shutdown 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 msgr2=0x7ff514141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff50400b610 tx=0x7ff504017000 comp rx=0 tx=0).stop 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 shutdown_connections 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff51409a130 0x7ff51408ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff514095f50 0x7ff5141420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 --2- 172.21.15.150:0/2163696612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff514070ef0 0x7ff514141b70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 >> 172.21.15.150:0/2163696612 conn(0x7ff51406a2e0 msgr2=0x7ff51409a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 shutdown_connections 2024-09-18T10:08:52.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-18T10:08:52.155+0000 7ff51a69f700 1 -- 172.21.15.150:0/2163696612 wait complete. 2024-09-18T10:08:52.178 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T10:08:52.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T10:08:52.180 INFO:tasks.ceph.osd.0:Started 2024-09-18T10:08:52.180 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T10:08:52.181 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T10:08:52.183 INFO:tasks.ceph.osd.1:Started 2024-09-18T10:08:52.183 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T10:08:52.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T10:08:52.185 INFO:tasks.ceph.osd.2:Started 2024-09-18T10:08:52.185 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T10:08:52.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T10:08:52.188 INFO:tasks.ceph.osd.3:Started 2024-09-18T10:08:52.188 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-18T10:08:52.188 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-18T10:08:52.190 INFO:tasks.ceph.osd.4:Started 2024-09-18T10:08:52.190 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-18T10:08:52.190 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-18T10:08:52.192 INFO:tasks.ceph.osd.5:Started 2024-09-18T10:08:52.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:08:52.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: get_monmap_and_config 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: build_initial_monmap 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: monmap: 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:52.286228+0000 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:52.286228+0000 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:52.288 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 1 Processor -- start 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 1 -- start start 2024-09-18T10:08:52.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: init 2024-09-18T10:08:52.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:52.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T10:08:52.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f4009d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:08:52.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.281+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-b con 0x7f2f4012f510 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f4012f900 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4009c440 con 0x7f2f4012f900 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4009a6c0 con 0x7f2f4009d870 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4009dc40 con 0x7f2f4012f510 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 10 monclient(hunting): authenticate will time out at 821.212098s 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2f4012f900 0x7f2f40136680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57830/0 (socket says 172.21.15.142:57830) 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3f7fe700 1 -- 172.21.15.142:0/1632985094 learned_addr learned my addr 172.21.15.142:0/1632985094 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48436/0 (socket says 172.21.15.142:48436) 2024-09-18T10:08:52.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): get_auth_request con 0x7f2f4012f510 auth_method 0 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 -- 172.21.15.142:0/1632985094 >> v1:172.21.15.150:6789/0 conn(0x7f2f4009d870 legacy=0x7f2f4012ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 -- 172.21.15.142:0/1632985094 >> v1:172.21.15.142:6789/0 conn(0x7f2f4012f900 legacy=0x7f2f40136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 monclient: found mon.noname-b 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:52.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 -- 172.21.15.142:0/1632985094 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4009a9a0 con 0x7f2f4012f510 2024-09-18T10:08:52.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 -- 172.21.15.142:0/1632985094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 msgr2=0x7f2f40134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:52.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3e7fc700 1 --2- 172.21.15.142:0/1632985094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400bbc0 tx=0x7f2f3400b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ff29698ff34a22 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 1 -- 172.21.15.142:0/1632985094 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34010470 con 0x7f2f4012f510 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: dump: 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: _finish_auth 0 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 5 monclient: authenticate success, global_id 4139 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 1 -- 172.21.15.142:0/1632985094 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f340105d0 con 0x7f2f4012f510 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f457aa700 4 set_mon_vals no callback set 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 1 -- 172.21.15.142:0/1632985094 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34010990 con 0x7f2f4012f510 2024-09-18T10:08:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.285+0000 7f2f3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3dffb700 10 monclient: dump: 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: shutdown 2024-09-18T10:08:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- 172.21.15.142:0/1632985094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 msgr2=0x7f2f40134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 --2- 172.21.15.142:0/1632985094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400bbc0 tx=0x7f2f3400b430 comp rx=0 tx=0).stop 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- 172.21.15.142:0/1632985094 shutdown_connections 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 --2- 172.21.15.142:0/1632985094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4012f510 0x7f2f40134130 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- 172.21.15.142:0/1632985094 >> 172.21.15.142:0/1632985094 conn(0x7f2f40071820 msgr2=0x7f2f400a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- 172.21.15.142:0/1632985094 shutdown_connections 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- 172.21.15.142:0/1632985094 wait complete. 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: build_initial_monmap 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: monmap: 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:08:52.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 Processor -- start 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- start start 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: init 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-c con 0x7f2f4009d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:52.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 0x7f2f4012bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-a con 0x7f2f4012f510 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4012f900 0x7f2f4006e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): picked mon.noname-b con 0x7f2f4012f900 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40126770 con 0x7f2f4012f510 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40126900 con 0x7f2f4012f900 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40127440 con 0x7f2f4009d870 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f457aa700 10 monclient(hunting): authenticate will time out at 821.218451s 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48446/0 (socket says 172.21.15.142:48446) 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 -- 172.21.15.142:0/3774152673 learned_addr learned my addr 172.21.15.142:0/3774152673 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): get_auth_request con 0x7f2f4009d870 auth_method 0 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 0x7f2f4012bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 10 monclient(hunting): get_auth_request con 0x7f2f4012f510 auth_method 0 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3f7fe700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4012f900 0x7f2f4006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3f7fe700 10 monclient(hunting): get_auth_request con 0x7f2f4012f900 auth_method 0 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 -- 172.21.15.142:0/3774152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4012f900 msgr2=0x7f2f4006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4012f900 0x7f2f4006e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 -- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 msgr2=0x7f2f4012bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 0x7f2f4012bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient: _finish_hunting 0 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 monclient: found mon.noname-c 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3effd700 1 -- 172.21.15.142:0/3774152673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3400b6d0 con 0x7f2f4009d870 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.289+0000 7f2f3e7fc700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 0x7f2f4012bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f3effd700 1 -- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 msgr2=0x7f2f4013d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f3effd700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800bc30 tx=0x7f2f38009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800cbc0 con 0x7f2f4009d870 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: dump: 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: _finish_auth 0 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800cd40 con 0x7f2f4009d870 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 5 monclient: authenticate success, global_id 4142 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _renew_subs 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f40128800 con 0x7f2f4009d870 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38014660 con 0x7f2f4009d870 2024-09-18T10:08:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 10 monclient: dump: 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:08:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f3d7fa700 4 set_mon_vals no callback set 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _renew_subs 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4013daa0 con 0x7f2f4009d870 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f2f38014ce0 con 0x7f2f4009d870 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f2f380073e0 con 0x7f2f4009d870 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:52.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.293+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f40128ba0 con 0x7f2f4009d870 2024-09-18T10:08:52.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.301+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f2f38022060 con 0x7f2f4009d870 2024-09-18T10:08:52.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.301+0000 7f2f26ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:52.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.301+0000 7f2f26ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:52.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.365+0000 7f2f457aa700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:52.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.365+0000 7f2f457aa700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:52.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.365+0000 7f2f457aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:08:52.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.365+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f400696a0 con 0x7f2f4009d870 2024-09-18T10:08:52.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.525+0000 7f2f3effd700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800bc30 tx=0x7f2f38009820 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f2f3801d3f0 mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 2024-09-18T10:08:52.538 INFO:tasks.ceph.osd.1.smithi142.stderr:2024-09-18T10:08:52.533+0000 7f2a846b9540 -1 Falling back to public interface 2024-09-18T10:08:52.547 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-18T10:08:52.543+0000 7f9b47b90540 -1 Falling back to public interface 2024-09-18T10:08:52.733 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:52.727+0000 7f21e052e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T10:08:52.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f2f3801d3f0 con 0x7f2f4009d870 2024-09-18T10:08:52.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f26ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:52.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f26ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:52.762 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:08:52.762 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":8,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:08:51.753346+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:08:52.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 10 monclient: shutdown 2024-09-18T10:08:52.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:52.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 msgr2=0x7f2f4013d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:52.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800bc30 tx=0x7f2f38009820 comp rx=0 tx=0).stop 2024-09-18T10:08:52.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f26ffd700 1 -- 172.21.15.142:0/3774152673 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mgrmap(e 2) v1 ==== 42509+0+0 (unknown 0 0 0) 0x7f2f38007780 con 0x7f2f4009d870 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 shutdown_connections 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4012f900 0x7f2f4006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4012f510 0x7f2f4012bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.757+0000 7f2f457aa700 1 --2- 172.21.15.142:0/3774152673 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4009d870 0x7f2f4013d3c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.761+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 >> 172.21.15.142:0/3774152673 conn(0x7f2f40071820 msgr2=0x7f2f400a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.761+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 shutdown_connections 2024-09-18T10:08:52.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:08:52.761+0000 7f2f457aa700 1 -- 172.21.15.142:0/3774152673 wait complete. 2024-09-18T10:08:52.782 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T10:08:52.782 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T10:08:52.803 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-09-18T10:08:52.799+0000 7f3e156f1540 -1 Falling back to public interface 2024-09-18T10:08:52.807 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-09-18T10:08:52.803+0000 7f6997863540 -1 Falling back to public interface 2024-09-18T10:08:52.814 INFO:tasks.ceph.osd.0.smithi142.stderr:2024-09-18T10:08:52.809+0000 7f81f94ef540 -1 Falling back to public interface 2024-09-18T10:08:52.826 INFO:tasks.ceph.osd.2.smithi142.stderr:2024-09-18T10:08:52.821+0000 7f7d37ad0540 -1 Falling back to public interface 2024-09-18T10:08:53.387 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T10:08:53.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.477+0000 7f62d5627700 10 monclient: get_monmap_and_config 2024-09-18T10:08:53.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.477+0000 7f62d5627700 10 monclient: build_initial_monmap 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.477+0000 7f62d5627700 10 monclient: monmap: 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:epoch 0 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:53.482962+0000 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:53.482962+0000 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:08:53.484 INFO:teuthology.misc.health.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:08:53.485 INFO:teuthology.misc.health.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:08:53.485 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 Processor -- start 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 -- start start 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient: init 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62d0070eb0 0x7f62d00a1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-c con 0x7f62d0070eb0 addr v2:172.21.15.150:3300/0 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62d009bcb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 0x7f62d0137650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:53.488 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-b con 0x7f62d0135270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d0071280 con 0x7f62d009bcb0 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d006b6c0 con 0x7f62d0070eb0 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d009eb20 con 0x7f62d0135270 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d5627700 10 monclient(hunting): authenticate will time out at 822.409094s 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cf7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f62d009bcb0 0x7f62d00a1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57840/0 (socket says 172.21.15.142:57840) 2024-09-18T10:08:53.489 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cf7fe700 1 -- 172.21.15.142:0/2330314617 learned_addr learned my addr 172.21.15.142:0/2330314617 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 --2- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 0x7f62d0137650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): get_auth_request con 0x7f62d0135270 auth_method 0 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 1 --2- 172.21.15.142:0/2330314617 >> v2:172.21.15.150:3300/0 conn(0x7f62d0070eb0 0x7f62d00a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): get_auth_request con 0x7f62d0070eb0 auth_method 0 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:53.490 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 -- 172.21.15.142:0/2330314617 >> v2:172.21.15.150:3300/0 conn(0x7f62d0070eb0 msgr2=0x7f62d00a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 --2- 172.21.15.142:0/2330314617 >> v2:172.21.15.150:3300/0 conn(0x7f62d0070eb0 0x7f62d00a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 -- 172.21.15.142:0/2330314617 >> v1:172.21.15.142:6789/0 conn(0x7f62d009bcb0 legacy=0x7f62d00a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient: _finish_hunting 0 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 monclient: found mon.noname-b 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:08:53.491 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.481+0000 7f62d4e26700 1 -- 172.21.15.142:0/2330314617 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d006dad0 con 0x7f62d0135270 2024-09-18T10:08:53.495 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d4e26700 1 -- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 msgr2=0x7f62d0137650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:53.495 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d4e26700 1 --2- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 0x7f62d0137650 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009e90 tx=0x7f62c8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38985f7dd4e9d1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:53.495 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 1 -- 172.21.15.142:0/2330314617 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c800e070 con 0x7f62d0135270 2024-09-18T10:08:53.495 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: dump: 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.496 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 5 monclient: authenticate success, global_id 4163 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 1 -- 172.21.15.142:0/2330314617 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c800be00 con 0x7f62d0135270 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 10 monclient: get_monmap_and_config success 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 4 set_mon_vals no callback set 2024-09-18T10:08:53.497 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 1 -- 172.21.15.142:0/2330314617 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8012660 con 0x7f62d0135270 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62ceffd700 10 monclient: dump: 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:53.498 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 10 monclient: shutdown 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 -- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 msgr2=0x7f62d0137650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 --2- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 0x7f62d0137650 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009e90 tx=0x7f62c8009780 comp rx=0 tx=0).stop 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 -- 172.21.15.142:0/2330314617 shutdown_connections 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 --2- 172.21.15.142:0/2330314617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d0135270 0x7f62d0137650 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 --2- 172.21.15.142:0/2330314617 >> v2:172.21.15.150:3300/0 conn(0x7f62d0070eb0 0x7f62d00a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 -- 172.21.15.142:0/2330314617 >> 172.21.15.142:0/2330314617 conn(0x7f62d00a01f0 msgr2=0x7f62d0067660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 -- 172.21.15.142:0/2330314617 shutdown_connections 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 -- 172.21.15.142:0/2330314617 wait complete. 2024-09-18T10:08:53.499 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 10 monclient: build_initial_monmap 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 10 monclient: monmap: 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:epoch 0 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:last_changed 0.000000 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:created 0.000000 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.485+0000 7f62d5627700 1 Processor -- start 2024-09-18T10:08:53.500 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 -- start start 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient: init 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient: _reopen_session rank -1 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62d0070eb0 0x7f62d011c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-b con 0x7f62d0070eb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 0x7f62d011cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-c con 0x7f62d009bcb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 0x7f62d011d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): picked mon.noname-a con 0x7f62d0135270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d011e8b0 con 0x7f62d0135270 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:53.501 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d009b3c0 con 0x7f62d0070eb0 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): start opening mon connection 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d011e330 con 0x7f62d009bcb0 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): _renew_subs 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d5627700 10 monclient(hunting): authenticate will time out at 822.417290s 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 0x7f62d011cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 0x7f62d011d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 0x7f62d011cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48452/0 (socket says 172.21.15.142:48452) 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 1 -- 172.21.15.142:0/3564824199 learned_addr learned my addr 172.21.15.142:0/3564824199 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): get_auth_request con 0x7f62d009bcb0 auth_method 0 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:53.502 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): get_auth_request con 0x7f62d0135270 auth_method 0 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62d0070eb0 0x7f62d011c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): get_auth_request con 0x7f62d0070eb0 auth_method 0 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:53.503 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient(hunting): handle_auth_done global_id 4173 payload 824 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 -- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 msgr2=0x7f62d011cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 0x7f62d011cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 -- 172.21.15.142:0/3564824199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62d0070eb0 msgr2=0x7f62d011c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62d0070eb0 0x7f62d011c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient: _finish_hunting 0 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 monclient: found mon.noname-a 2024-09-18T10:08:53.504 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:08:53.505 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:08:53.505 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.489+0000 7f62d4e26700 1 -- 172.21.15.142:0/3564824199 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8009a20 con 0x7f62d0135270 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d4e26700 1 -- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 msgr2=0x7f62d011d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d4e26700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 0x7f62d011d2b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009e60 tx=0x7f62c8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8012de0 con 0x7f62d0135270 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: dump: 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:08:53.507 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: _finish_auth 0 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 5 monclient: authenticate success, global_id 4173 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c8021070 con 0x7f62d0135270 2024-09-18T10:08:53.508 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _renew_subs 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8026430 con 0x7f62d0135270 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d006eda0 con 0x7f62d0135270 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62ce7fc700 4 set_mon_vals no callback set 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 10 monclient: dump: 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:08:53.509 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _renew_subs 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d011ef00 con 0x7f62d0135270 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 2) v1 ==== 42509+0+0 (secure 0 0 0) 0x7f62c8023030 con 0x7f62d0135270 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0072ab0 con 0x7f62d0135270 2024-09-18T10:08:53.510 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.493+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f62c80492a0 con 0x7f62d0135270 2024-09-18T10:08:53.511 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.497+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72049 (secure 0 0 0) 0x7f62c801b9b0 con 0x7f62d0135270 2024-09-18T10:08:53.511 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.497+0000 7f62b7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:08:53.511 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.497+0000 7f62b7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:08:53.568 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.561+0000 7f62d5627700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:53.568 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.561+0000 7f62d5627700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:53.568 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.561+0000 7f62d5627700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:08:53.568 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.561+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62d0068670 con 0x7f62d0135270 2024-09-18T10:08:53.569 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62b7fff700 1 -- 172.21.15.142:0/3564824199 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f62c801b9b0 con 0x7f62d0135270 2024-09-18T10:08:53.569 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62b7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:08:53.569 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62b7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:08:53.569 INFO:teuthology.misc.health.smithi142.stdout: 2024-09-18T10:08:53.569 INFO:teuthology.misc.health.smithi142.stdout:{"epoch":8,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:08:51.753346+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:08:53.570 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 10 monclient: shutdown 2024-09-18T10:08:53.570 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 msgr2=0x7f62d011d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 0x7f62d011d2b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009e60 tx=0x7f62c8018000 comp rx=0 tx=0).stop 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 shutdown_connections 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62d0135270 0x7f62d011d2b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62d009bcb0 0x7f62d011cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 --2- 172.21.15.142:0/3564824199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62d0070eb0 0x7f62d011c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 >> 172.21.15.142:0/3564824199 conn(0x7f62d00a01f0 msgr2=0x7f62d0099890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 shutdown_connections 2024-09-18T10:08:53.571 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:08:53.565+0000 7f62d5627700 1 -- 172.21.15.142:0/3564824199 wait complete. 2024-09-18T10:08:53.581 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-18T10:08:53.775 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:53.771+0000 7f21c95a2700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T10:08:53.776 INFO:tasks.ceph.mgr.x.smithi150.stderr:2024-09-18T10:08:53.771+0000 7f21c95a2700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T10:08:56.332 INFO:tasks.ceph.osd.1.smithi142.stderr:2024-09-18T10:08:56.325+0000 7f2a846b9540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:08:56.348 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-18T10:08:56.343+0000 7f9b47b90540 -1 osd.3 0 log_to_monitors true 2024-09-18T10:08:56.384 INFO:tasks.ceph.osd.2.smithi142.stderr:2024-09-18T10:08:56.377+0000 7f7d37ad0540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:08:56.497 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-09-18T10:08:56.491+0000 7f3e156f1540 -1 osd.5 0 log_to_monitors true 2024-09-18T10:08:56.551 INFO:tasks.ceph.osd.0.smithi142.stderr:2024-09-18T10:08:56.545+0000 7f81f94ef540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:08:56.585 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-09-18T10:08:56.579+0000 7f6997863540 -1 osd.4 0 log_to_monitors true 2024-09-18T10:08:57.790 INFO:tasks.ceph.osd.0.smithi142.stderr:2024-09-18T10:08:57.785+0000 7f81f4c8f700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T10:08:57.795 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-09-18T10:08:57.791+0000 7f3e10e91700 -1 osd.5 0 waiting for initial osdmap 2024-09-18T10:08:57.795 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-18T10:08:57.791+0000 7f9b43330700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T10:08:57.795 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-09-18T10:08:57.791+0000 7f6993003700 -1 osd.4 0 waiting for initial osdmap 2024-09-18T10:08:57.796 INFO:tasks.ceph.osd.0.smithi142.stderr:2024-09-18T10:08:57.789+0000 7f81f0237700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:08:57.802 INFO:tasks.ceph.osd.5.smithi150.stderr:2024-09-18T10:08:57.799+0000 7f3e0c439700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:08:57.802 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-18T10:08:57.799+0000 7f9b3e8d8700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:08:57.803 INFO:tasks.ceph.osd.4.smithi150.stderr:2024-09-18T10:08:57.799+0000 7f698e5ab700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:08:58.797 INFO:tasks.ceph.osd.2.smithi142.stderr:2024-09-18T10:08:58.793+0000 7f7d34a95700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T10:08:58.798 INFO:tasks.ceph.osd.1.smithi142.stderr:2024-09-18T10:08:58.793+0000 7f2a7fe59700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T10:08:58.802 INFO:tasks.ceph.osd.2.smithi142.stderr:2024-09-18T10:08:58.797+0000 7f7d2e818700 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:08:58.804 INFO:tasks.ceph.osd.1.smithi142.stderr:2024-09-18T10:08:58.801+0000 7f2a7b401700 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T10:09:00.188 DEBUG:teuthology.orchestra.run.smithi142:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T10:09:00.289 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: get_monmap_and_config 2024-09-18T10:09:00.289 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: build_initial_monmap 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: monmap: 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:epoch 0 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:09:00.288802+0000 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:09:00.288802+0000 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:00.290 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:00.291 INFO:teuthology.misc.health.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:00.291 INFO:teuthology.misc.health.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:00.291 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.292 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 Processor -- start 2024-09-18T10:09:00.292 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 -- start start 2024-09-18T10:09:00.292 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: init 2024-09-18T10:09:00.292 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.292 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde1809d570 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-b con 0x7fde18097160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde181348b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde180996d0 con 0x7fde1809d570 2024-09-18T10:09:00.293 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1809b6a0 con 0x7fde181348b0 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1809d990 con 0x7fde18097160 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde1da43700 10 monclient(hunting): authenticate will time out at 829.214286s 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48482/0 (socket says 172.21.15.142:48482) 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 -- 172.21.15.142:0/1605036957 learned_addr learned my addr 172.21.15.142:0/1605036957 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): get_auth_request con 0x7fde18097160 auth_method 0 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.297 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 -- 172.21.15.142:0/1605036957 >> v1:172.21.15.150:6789/0 conn(0x7fde181348b0 legacy=0x7fde18136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 -- 172.21.15.142:0/1605036957 >> v1:172.21.15.142:6789/0 conn(0x7fde1809d570 legacy=0x7fde18096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 monclient: found mon.noname-b 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.298 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.285+0000 7fde17fff700 1 -- 172.21.15.142:0/1605036957 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1809aac0 con 0x7fde18097160 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde17fff700 1 -- 172.21.15.142:0/1605036957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 msgr2=0x7fde18134370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde17fff700 1 --2- 172.21.15.142:0/1605036957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde080099e0 tx=0x7fde08009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80da1fb9dd4b67d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 1 -- 172.21.15.142:0/1605036957 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800bb50 con 0x7fde18097160 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: dump: 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:09:00.304 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 5 monclient: authenticate success, global_id 4187 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:00.305 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 1 -- 172.21.15.142:0/1605036957 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0800bcb0 con 0x7fde18097160 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 1 -- 172.21.15.142:0/1605036957 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08011650 con 0x7fde18097160 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 4 set_mon_vals no callback set 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde177fe700 10 monclient: dump: 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.306 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 10 monclient: shutdown 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- 172.21.15.142:0/1605036957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 msgr2=0x7fde18134370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 --2- 172.21.15.142:0/1605036957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde080099e0 tx=0x7fde08009250 comp rx=0 tx=0).stop 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- 172.21.15.142:0/1605036957 shutdown_connections 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 --2- 172.21.15.142:0/1605036957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde18097160 0x7fde18134370 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- 172.21.15.142:0/1605036957 >> 172.21.15.142:0/1605036957 conn(0x7fde18071ae0 msgr2=0x7fde180a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- 172.21.15.142:0/1605036957 shutdown_connections 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- 172.21.15.142:0/1605036957 wait complete. 2024-09-18T10:09:00.307 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 10 monclient: build_initial_monmap 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 10 monclient: monmap: 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:epoch 0 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:created 0.000000 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.308 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 Processor -- start 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 1 -- start start 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.289+0000 7fde1da43700 10 monclient: init 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde1809d570 0x7fde18093bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-b con 0x7fde1809d570 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-c con 0x7fde181348b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.309 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde1808ca90 0x7fde1808a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): picked mon.noname-a con 0x7fde1808ca90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde18099ba0 con 0x7fde1808ca90 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180a01d0 con 0x7fde1809d570 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1808e6b0 con 0x7fde181348b0 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient(hunting): authenticate will time out at 829.220482s 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48498/0 (socket says 172.21.15.142:48498) 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 -- 172.21.15.142:0/586940297 learned_addr learned my addr 172.21.15.142:0/586940297 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:00.310 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde1808ca90 0x7fde1808a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): get_auth_request con 0x7fde181348b0 auth_method 0 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde1809d570 0x7fde18093bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): get_auth_request con 0x7fde1809d570 auth_method 0 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): get_auth_request con 0x7fde1808ca90 auth_method 0 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.311 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1d242700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1ca41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 -- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde1809d570 msgr2=0x7fde18093bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.312 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde1809d570 0x7fde18093bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 -- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde1808ca90 msgr2=0x7fde1808a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde1808ca90 0x7fde1808a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 monclient: found mon.noname-c 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 -- 172.21.15.142:0/586940297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080094f0 con 0x7fde181348b0 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 -- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 msgr2=0x7fde1808a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde17fff700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde08009290 tx=0x7fde08016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08011e60 con 0x7fde181348b0 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:00.313 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: dump: 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0801b680 con 0x7fde181348b0 2024-09-18T10:09:00.314 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 5 monclient: authenticate success, global_id 4190 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient: _renew_subs 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1808bec0 con 0x7fde181348b0 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0801ba20 con 0x7fde181348b0 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde14ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde14ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde14ff9700 10 monclient: dump: 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:epoch 1 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.315 INFO:teuthology.misc.health.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr: 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 10 monclient: _renew_subs 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1808d9f0 con 0x7fde181348b0 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.293+0000 7fde16ffd700 4 set_mon_vals no callback set 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 4) v1 ==== 42742+0+0 (secure 0 0 0) 0x7fde0801c450 con 0x7fde181348b0 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.316 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1808dd20 con 0x7fde181348b0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde14ff9700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 0x7fde04037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3978+0+0 (secure 0 0 0) 0x7fde08049a40 con 0x7fde181348b0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1ca41700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 0x7fde04037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.297+0000 7fde1ca41700 10 monclient: get_auth_request con 0x7fde04035a30 auth_method 0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.301+0000 7fde1ca41700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 0x7fde04037ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0039a0 tx=0x7fde0c007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.309+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fde08018580 con 0x7fde181348b0 2024-09-18T10:09:00.317 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.309+0000 7fde14ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.318 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.309+0000 7fde14ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:00.479 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.473+0000 7fde1da43700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:00.479 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.473+0000 7fde1da43700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:00.479 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.473+0000 7fde1da43700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.479 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.473+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde1809b6a0 con 0x7fde181348b0 2024-09-18T10:09:00.480 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde14ff9700 1 -- 172.21.15.142:0/586940297 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+9783 (secure 0 0 0) 0x7fde1809b6a0 con 0x7fde181348b0 2024-09-18T10:09:00.480 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde14ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:00.480 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde14ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:00.481 INFO:teuthology.misc.health.smithi142.stdout: 2024-09-18T10:09:00.481 INFO:teuthology.misc.health.smithi142.stdout:{"epoch":12,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:08:59.791843+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 msgr2=0x7fde04037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 0x7fde04037ee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0039a0 tx=0x7fde0c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 10 monclient: shutdown 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 msgr2=0x7fde1808a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde08009290 tx=0x7fde08016000 comp rx=0 tx=0).stop 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 shutdown_connections 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde04035a30 0x7fde04037ee0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.483 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde1808ca90 0x7fde1808a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde181348b0 0x7fde1808a420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 --2- 172.21.15.142:0/586940297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde1809d570 0x7fde18093bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 >> 172.21.15.142:0/586940297 conn(0x7fde18097160 msgr2=0x7fde1809c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:00.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 shutdown_connections 2024-09-18T10:09:00.484 INFO:teuthology.misc.health.smithi142.stderr:2024-09-18T10:09:00.477+0000 7fde1da43700 1 -- 172.21.15.142:0/586940297 wait complete. 2024-09-18T10:09:00.493 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-18T10:09:00.493 INFO:tasks.ceph:Creating RBD pool 2024-09-18T10:09:00.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T10:09:00.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: get_monmap_and_config 2024-09-18T10:09:00.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: build_initial_monmap 2024-09-18T10:09:00.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: monmap: 2024-09-18T10:09:00.601 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:00.600589+0000 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:00.600589+0000 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:00.602 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 Processor -- start 2024-09-18T10:09:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 -- start start 2024-09-18T10:09:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: init 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02f812f1f0 0x7f02f812f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-c con 0x7f02f812f1f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02f80a11d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-b con 0x7f02f8137330 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f80a1ba0 con 0x7f02f80a11d0 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f8068020 con 0x7f02f812f1f0 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80a6650 con 0x7f02f8137330 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.597+0000 7f02fed71700 10 monclient(hunting): authenticate will time out at 829.527744s 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fdd6f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02f812f1f0 0x7f02f812f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48502/0 (socket says 172.21.15.142:48502) 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 -- 172.21.15.142:0/3437850485 learned_addr learned my addr 172.21.15.142:0/3437850485 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fdd6f700 10 monclient(hunting): get_auth_request con 0x7f02f812f1f0 auth_method 0 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fdd6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fdd6f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fdd6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fd56e700 1 --1- 172.21.15.142:0/3437850485 >> v1:172.21.15.150:6789/0 conn(0x7f02f80a11d0 0x7f02f809cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:36264/0 (socket says 172.21.15.142:36264) 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): get_auth_request con 0x7f02f8137330 auth_method 0 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 -- 172.21.15.142:0/3437850485 >> v2:172.21.15.150:3300/0 conn(0x7f02f812f1f0 msgr2=0x7f02f812f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 --2- 172.21.15.142:0/3437850485 >> v2:172.21.15.150:3300/0 conn(0x7f02f812f1f0 0x7f02f812f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 -- 172.21.15.142:0/3437850485 >> v1:172.21.15.150:6789/0 conn(0x7f02f80a11d0 legacy=0x7f02f809cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 monclient: found mon.noname-b 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 -- 172.21.15.142:0/3437850485 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f80a1830 con 0x7f02f8137330 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 -- 172.21.15.142:0/3437850485 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 msgr2=0x7f02f8139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fe570700 1 --2- 172.21.15.142:0/3437850485 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009f20 tx=0x7f02f0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189bd357da317a55 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 1 -- 172.21.15.142:0/3437850485 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f000e070 con 0x7f02f8137330 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: dump: 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.617 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fed71700 5 monclient: authenticate success, global_id 4193 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 1 -- 172.21.15.142:0/3437850485 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f000bc90 con 0x7f02f8137330 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fed71700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fed71700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:00.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 1 -- 172.21.15.142:0/3437850485 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f00092e0 con 0x7f02f8137330 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fed71700 4 set_mon_vals no callback set 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.601+0000 7f02fcd6d700 10 monclient: dump: 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.619 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: shutdown 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- 172.21.15.142:0/3437850485 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 msgr2=0x7f02f8139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- 172.21.15.142:0/3437850485 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009f20 tx=0x7f02f0009840 comp rx=0 tx=0).stop 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- 172.21.15.142:0/3437850485 shutdown_connections 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- 172.21.15.142:0/3437850485 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f8139710 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- 172.21.15.142:0/3437850485 >> v2:172.21.15.150:3300/0 conn(0x7f02f812f1f0 0x7f02f812f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- 172.21.15.142:0/3437850485 >> 172.21.15.142:0/3437850485 conn(0x7f02f80a5700 msgr2=0x7f02f80a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:00.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- 172.21.15.142:0/3437850485 shutdown_connections 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- 172.21.15.142:0/3437850485 wait complete. 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: build_initial_monmap 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: monmap: 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 Processor -- start 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- start start 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: init 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80a11d0 0x7f02f806f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-b con 0x7f02f80a11d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-a con 0x7f02f812f1f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f812c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): picked mon.noname-c con 0x7f02f8137330 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f809dee0 con 0x7f02f812f1f0 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f806e940 con 0x7f02f80a11d0 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f806b780 con 0x7f02f8137330 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fed71700 10 monclient(hunting): authenticate will time out at 829.533729s 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60360/0 (socket says 172.21.15.142:60360) 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 -- 172.21.15.142:0/3584357420 learned_addr learned my addr 172.21.15.142:0/3584357420 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): get_auth_request con 0x7f02f812f1f0 auth_method 0 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f812c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 10 monclient(hunting): get_auth_request con 0x7f02f8137330 auth_method 0 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fdd6f700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80a11d0 0x7f02f806f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fdd6f700 10 monclient(hunting): get_auth_request con 0x7f02f80a11d0 auth_method 0 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fdd6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fdd6f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fdd6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-18T10:09:00.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 -- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 msgr2=0x7f02f812c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f812c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 -- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80a11d0 msgr2=0x7f02f806f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80a11d0 0x7f02f806f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 monclient: found mon.noname-a 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fd56e700 1 -- 172.21.15.142:0/3584357420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0009ae0 con 0x7f02f812f1f0 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.605+0000 7f02fe570700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f812c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fd56e700 1 -- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 msgr2=0x7f02f806e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fd56e700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c5e0 tx=0x7f02e8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8005c80 con 0x7f02f812f1f0 2024-09-18T10:09:00.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: dump: 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 5 monclient: authenticate success, global_id 4197 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e8005e00 con 0x7f02f812f1f0 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _renew_subs 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f806b410 con 0x7f02f812f1f0 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8007320 con 0x7f02f812f1f0 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 10 monclient: dump: 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.627 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _renew_subs 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02ef7fe700 4 set_mon_vals no callback set 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f812de90 con 0x7f02f812f1f0 2024-09-18T10:09:00.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 4) v1 ==== 42742+0+0 (secure 0 0 0) 0x7f02e801c440 con 0x7f02f812f1f0 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 0x7f02dc037ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3978+0+0 (secure 0 0 0) 0x7f02e8047590 con 0x7f02f812f1f0 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f8072a80 con 0x7f02f812f1f0 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fdd6f700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 0x7f02dc037ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.609+0000 7f02fdd6f700 10 monclient: get_auth_request con 0x7f02dc035a30 auth_method 0 2024-09-18T10:09:00.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.613+0000 7f02fdd6f700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 0x7f02dc037ee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02f4006fd0 tx=0x7f02f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.617+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f02e8017020 con 0x7f02f812f1f0 2024-09-18T10:09:00.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.617+0000 7f02edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:00.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.617+0000 7f02edffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:00.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.789+0000 7f02fed71700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T10:09:00.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.789+0000 7f02fed71700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T10:09:00.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.789+0000 7f02fed71700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:00.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.789+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f02f806b930 con 0x7f02f812f1f0 2024-09-18T10:09:00.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.837+0000 7f02edffb700 1 -- 172.21.15.142:0/3584357420 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f02e801b3b0 con 0x7f02f812f1f0 2024-09-18T10:09:00.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.837+0000 7f02edffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-18T10:09:00.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.837+0000 7f02edffb700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 msgr2=0x7f02dc037ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 0x7f02dc037ee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02f4006fd0 tx=0x7f02f4008040 comp rx=0 tx=0).stop 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 10 monclient: shutdown 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 msgr2=0x7f02f806e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f02e800c5e0 tx=0x7f02e8007f90 comp rx=0 tx=0).stop 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 shutdown_connections 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02dc035a30 0x7f02dc037ee0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8137330 0x7f02f812c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f812f1f0 0x7f02f806e400 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 --2- 172.21.15.142:0/3584357420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80a11d0 0x7f02f806f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 >> 172.21.15.142:0/3584357420 conn(0x7f02f80a5700 msgr2=0x7f02f809e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:00.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 shutdown_connections 2024-09-18T10:09:00.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.841+0000 7f02fed71700 1 -- 172.21.15.142:0/3584357420 wait complete. 2024-09-18T10:09:00.849 INFO:teuthology.orchestra.run.smithi142.stderr:pool 'rbd' created 2024-09-18T10:09:00.860 DEBUG:teuthology.orchestra.run.smithi142:> rbd --cluster ceph pool init rbd 2024-09-18T10:09:00.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: get_monmap_and_config 2024-09-18T10:09:00.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: build_initial_monmap 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: monmap: 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:00.892757+0000 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:00.892757+0000 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:00.894 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 Processor -- start 2024-09-18T10:09:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 -- start start 2024-09-18T10:09:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: init 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-c-legacy con 0x56381ab47510 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 --2- >> v2:172.21.15.150:3300/0 conn(0x56381abe0290 0x56381abe2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-c con 0x56381abe0290 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-a-legacy con 0x56381abe2bb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56381aa69660 con 0x56381abe2bb0 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56381aa427f0 con 0x56381ab47510 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x56381aaa0360 con 0x56381abe0290 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f198b2334c0 10 monclient(hunting): authenticate will time out at 829.819764s 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f1989d69700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x56381abe2bb0 0x56381abe4f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57866/0 (socket says 172.21.15.142:57866) 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.889+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 learned_addr learned my addr 172.21.15.142:0/3749103542 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 1 --2- 172.21.15.142:0/3749103542 >> v2:172.21.15.150:3300/0 conn(0x56381abe0290 0x56381abe2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): get_auth_request con 0x56381abe0290 auth_method 0 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 >> v1:172.21.15.142:6789/0 conn(0x56381abe2bb0 legacy=0x56381abe4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166544978 0 0) 0x56381aa69660 con 0x56381abe2bb0 2024-09-18T10:09:00.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): my global_id is 4200 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1970001e10 con 0x56381abe2bb0 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 1 -- 172.21.15.142:0/3749103542 >> v1:172.21.15.150:6789/0 conn(0x56381ab47510 legacy=0x56381ab4a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650584353 0 0) 0x7f1970001e10 con 0x56381abe2bb0 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 >> v2:172.21.15.150:3300/0 conn(0x56381abe0290 msgr2=0x56381abe2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 --2- 172.21.15.142:0/3749103542 >> v2:172.21.15.150:3300/0 conn(0x56381abe0290 0x56381abe2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 >> v1:172.21.15.150:6789/0 conn(0x56381ab47510 legacy=0x56381ab4a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:09:00.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56381aa4f660 con 0x56381abe2bb0 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f198b2334c0 5 monclient: authenticate success, global_id 4200 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1974001860 con 0x56381abe2bb0 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f198b2334c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 1 --2- 172.21.15.142:0/3749103542 >> v2:172.21.15.150:3300/0 conn(0x56381abe0290 0x56381abe2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: dump: 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 >> v1:172.21.15.142:6789/0 conn(0x56381abe2bb0 legacy=0x56381abe4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1970002420 0x7f1970002830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): picked mon.a con 0x7f1970002420 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19700030b0 0x7f1970003520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): picked mon.b con 0x7f19700030b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x7f197000a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): picked mon.c con 0x7f1970003c60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381aaa0360 con 0x7f1970002420 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197000abc0 con 0x7f19700030b0 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197000ad80 con 0x7f1970003c60 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f197bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f198b2334c0 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1970002420 0x7f1970002830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x7f197000a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): get_auth_request con 0x7f1970002420 auth_method 0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 10 monclient(hunting): get_auth_request con 0x7f1970003c60 auth_method 0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19700030b0 0x7f1970003520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): get_auth_request con 0x7f19700030b0 auth_method 0 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1988d67700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.893+0000 7f1989d69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient(hunting): handle_auth_done global_id 4200 payload 293 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19700030b0 msgr2=0x7f1970003520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19700030b0 0x7f1970003520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1970002420 msgr2=0x7f1970002830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1970002420 0x7f1970002830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 monclient: found mon.c 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970001e10 con 0x7f1970003c60 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989568700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1970002420 0x7f1970002830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 reap_dead start 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 -- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 msgr2=0x7f197000a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f1989d69700 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x7f197000a300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x56381ab30420 tx=0x56381ab47f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2a4ab33a7383ff1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974006850 con 0x7f1970003c60 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: dump: 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197400fe20 con 0x7f1970003c60 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 10 monclient: get_monmap_and_config success 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 1 -- 172.21.15.142:0/3749103542 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974018440 con 0x7f1970003c60 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 4 set_mon_vals no callback set 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f197bfff700 10 monclient: dump: 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.913 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 10 monclient: shutdown 2024-09-18T10:09:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 1 -- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 msgr2=0x7f197000a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x7f197000a300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x56381ab30420 tx=0x56381ab47f90 comp rx=0 tx=0).stop 2024-09-18T10:09:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 1 -- 172.21.15.142:0/3749103542 shutdown_connections 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 1 --2- 172.21.15.142:0/3749103542 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x7f197000a300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.897+0000 7f198b2334c0 1 -- 172.21.15.142:0/3749103542 >> 172.21.15.142:0/3749103542 conn(0x56381ab4d780 msgr2=0x56381ab4db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- 172.21.15.142:0/3749103542 shutdown_connections 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- 172.21.15.142:0/3749103542 wait complete. 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient: build_initial_monmap 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient: monmap: 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:00.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 Processor -- start 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- start start 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient: init 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient: _reopen_session rank -1 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x56381ab466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-c con 0x7f1970003c60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56381ab44ab0 0x56381ab42510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-a con 0x56381ab44ab0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56381ab42a50 0x56381ab42ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): picked mon.noname-b con 0x56381ab42a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381aa34bc0 con 0x56381ab44ab0 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x56381aa60d10 con 0x56381ab42a50 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x56381aa43f40 con 0x7f1970003c60 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): _renew_subs 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f198b2334c0 10 monclient(hunting): authenticate will time out at 829.829761s 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x56381ab466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x56381ab466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:48530/0 (socket says 172.21.15.142:48530) 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 -- 172.21.15.142:0/1529973844 learned_addr learned my addr 172.21.15.142:0/1529973844 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): get_auth_request con 0x7f1970003c60 auth_method 0 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56381ab44ab0 0x56381ab42510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): get_auth_request con 0x56381ab44ab0 auth_method 0 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1988d67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56381ab42a50 msgr2=0x56381ab42ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56381ab42a50 0x56381ab42ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56381ab44ab0 msgr2=0x56381ab42510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x56381ab44ab0 0x56381ab42510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient: _finish_hunting 0 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 monclient: found mon.noname-c 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.901+0000 7f1989568700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56381aa42b80 con 0x7f1970003c60 2024-09-18T10:09:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f1989568700 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 msgr2=0x56381ab466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f1989568700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x56381ab466d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1980001740 tx=0x7f1980001a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1980011660 con 0x7f1970003c60 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: dump: 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 5 monclient: authenticate success, global_id 4202 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _renew_subs 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1980008050 con 0x7f1970003c60 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56381aa425e0 con 0x7f1970003c60 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19800119d0 con 0x7f1970003c60 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f197affd700 4 set_mon_vals no callback set 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 10 monclient: dump: 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _renew_subs 2024-09-18T10:09:00.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56381ab48540 con 0x7f1970003c60 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 4) v1 ==== 42742+0+0 (secure 0 0 0) 0x7f19800103b0 con 0x7f1970003c60 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1968034300 0x7f19680367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4331+0+0 (secure 0 0 0) 0x7f1980046ee0 con 0x7f1970003c60 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x56381ab50260 con 0x7f1970003c60 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f1988d67700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1968034300 0x7f19680367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.905+0000 7f1988d67700 10 monclient: get_auth_request con 0x7f1968034300 auth_method 0 2024-09-18T10:09:00.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:00.909+0000 7f1988d67700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1968034300 0x7f19680367b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f197c005b20 tx=0x7f197c007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:01.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.969+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (secure 0 0 0) 0x7f1980010660 con 0x7f1970003c60 2024-09-18T10:09:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.969+0000 7f19797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-18T10:09:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.969+0000 7f19797fa700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-18T10:09:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.969+0000 7f198b2334c0 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.969+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x56381aa43d30 con 0x7f1970003c60 2024-09-18T10:09:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1980017ea0 con 0x7f1970003c60 2024-09-18T10:09:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f19797fa700 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-18T10:09:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f197affd700 10 monclient: _renew_subs 2024-09-18T10:09:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f197affd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f197affd700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=14}) v3 -- 0x7f1960002990 con 0x7f1970003c60 2024-09-18T10:09:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1020+0+0 (secure 0 0 0) 0x7f1980020300 con 0x7f1970003c60 2024-09-18T10:09:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f198b2334c0 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x56381abf1ba0 0x56381abf3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x56381abf44c0 con 0x56381abf1ba0 2024-09-18T10:09:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f1989d69700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x56381abf1ba0 0x56381abf3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.973+0000 7f1989d69700 10 monclient: get_auth_request con 0x56381abf1ba0 auth_method 0 2024-09-18T10:09:01.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.977+0000 7f1989d69700 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x56381abf1ba0 0x56381abf3f80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:01.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.985+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 <== osd.5 v2:172.21.15.150:6810/21120 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f19740010c0 con 0x56381abf1ba0 2024-09-18T10:09:01.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x56381ac41c60 con 0x56381abf1ba0 2024-09-18T10:09:01.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.989+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 <== osd.5 v2:172.21.15.150:6810/21120 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f19740010c0 con 0x56381abf1ba0 2024-09-18T10:09:01.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.989+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x56381ac40d40 con 0x56381abf1ba0 2024-09-18T10:09:01.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.989+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 <== osd.5 v2:172.21.15.150:6810/21120 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f19740010c0 con 0x56381abf1ba0 2024-09-18T10:09:01.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.989+0000 7f197b7fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:01.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:01.989+0000 7f197b7fe700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f1964001ee0 con 0x7f1970003c60 2024-09-18T10:09:02.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 9 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1980010d40 con 0x7f1970003c60 2024-09-18T10:09:02.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 10 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (secure 0 0 0) 0x7f1980011dc0 con 0x7f1970003c60 2024-09-18T10:09:02.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 10 monclient: _renew_subs 2024-09-18T10:09:02.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:02.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=15}) v3 -- 0x7f196803a9b0 con 0x7f1970003c60 2024-09-18T10:09:02.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 11 ==== osd_map(15..15 src has 1..15) v4 ==== 618+0+0 (secure 0 0 0) 0x7f19800223e0 con 0x7f1970003c60 2024-09-18T10:09:02.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.977+0000 7f197affd700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f1960005450 con 0x56381abf1ba0 2024-09-18T10:09:02.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.985+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 <== osd.5 v2:172.21.15.150:6810/21120 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f19740010c0 con 0x56381abf1ba0 2024-09-18T10:09:02.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.985+0000 7f197b7fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:02.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:02.985+0000 7f197b7fe700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v15) v4 -- 0x7f1964003360 con 0x7f1970003c60 2024-09-18T10:09:03.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.901+0000 7f196ffff700 10 monclient: tick 2024-09-18T10:09:03.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.901+0000 7f196ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:03.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.901+0000 7f196ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:03.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (secure 0 0 0) 0x7f198002dde0 con 0x7f1970003c60 2024-09-18T10:09:03.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f19797fa700 10 monclient: _renew_subs 2024-09-18T10:09:03.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f19797fa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:03.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=16}) v3 -- 0x7f196803a9b0 con 0x7f1970003c60 2024-09-18T10:09:03.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f19797fa700 1 -- 172.21.15.142:0/1529973844 <== mon.2 v2:172.21.15.142:3301/0 13 ==== osd_map(16..16 src has 1..16) v4 ==== 646+0+0 (secure 0 0 0) 0x7f1980020df0 con 0x7f1970003c60 2024-09-18T10:09:03.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.977+0000 7f197b7fe700 1 -- 172.21.15.142:0/1529973844 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f1964004dd0 con 0x56381abf1ba0 2024-09-18T10:09:03.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 <== osd.5 v2:172.21.15.150:6810/21120 5 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f19740010c0 con 0x56381abf1ba0 2024-09-18T10:09:03.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x56381abf1ba0 msgr2=0x56381abf3f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:03.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x56381abf1ba0 0x56381abf3f80 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1968034300 msgr2=0x7f19680367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1968034300 0x7f19680367b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f197c005b20 tx=0x7f197c007040 comp rx=0 tx=0).stop 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 10 monclient: shutdown 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 msgr2=0x56381ab466d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 --2- 172.21.15.142:0/1529973844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1970003c60 0x56381ab466d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1980001740 tx=0x7f1980001a80 comp rx=0 tx=0).stop 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f1989d69700 1 -- 172.21.15.142:0/1529973844 reap_dead start 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 shutdown_connections 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 >> 172.21.15.142:0/1529973844 conn(0x56381ab4d780 msgr2=0x56381ab2e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:03.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 shutdown_connections 2024-09-18T10:09:03.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:03.985+0000 7f198b2334c0 1 -- 172.21.15.142:0/1529973844 wait complete. 2024-09-18T10:09:03.997 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T10:09:03.997 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T10:09:03.997 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T10:09:03.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T10:09:03.998 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T10:09:04.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: get_monmap_and_config 2024-09-18T10:09:04.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: monmap: 2024-09-18T10:09:04.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:04.153678+0000 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:04.153678+0000 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:04.155 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 Processor -- start 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 -- start start 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: init 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T10:09:04.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f009aae0 0x7f44f0096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-b con 0x7f44f009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 0x7f44f009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-c con 0x7f44f00967a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f44f009c320 0x7f44f009c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-a con 0x7f44f009c320 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f0095880 con 0x7f44f009c320 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f009aeb0 con 0x7f44f00967a0 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006f6d0 con 0x7f44f009aae0 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f5193700 10 monclient(hunting): authenticate will time out at 833.079132s 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f44f009c320 0x7f44f009c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f44f009c320 0x7f44f009c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48346/0 (socket says 172.21.15.142:48346) 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 1 -- 172.21.15.142:0/3648153650 learned_addr learned my addr 172.21.15.142:0/3648153650 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): get_auth_request con 0x7f44f009c320 auth_method 0 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f009aae0 0x7f44f0096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): get_auth_request con 0x7f44f009aae0 auth_method 0 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 1 --2- 172.21.15.142:0/3648153650 >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 0x7f44f009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): get_auth_request con 0x7f44f00967a0 auth_method 0 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.149+0000 7f44f4992700 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 -- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f009aae0 msgr2=0x7f44f0096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f009aae0 0x7f44f0096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 -- 172.21.15.142:0/3648153650 >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 msgr2=0x7f44f009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 --2- 172.21.15.142:0/3648153650 >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 0x7f44f009bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 -- 172.21.15.142:0/3648153650 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00955a0 con 0x7f44f009c320 2024-09-18T10:09:04.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44ef7fe700 1 --2- 172.21.15.142:0/3648153650 >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 0x7f44f009bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:04.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 -- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009c320 msgr2=0x7f44f009c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f4992700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009c320 0x7f44f009c780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44f0067590 tx=0x7f44e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715d66261046b583 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 1 -- 172.21.15.142:0/3648153650 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800b850 con 0x7f44f009c320 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: dump: 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.168 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 1 -- 172.21.15.142:0/3648153650 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e800b9b0 con 0x7f44f009c320 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 5 monclient: authenticate success, global_id 4215 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 1 -- 172.21.15.142:0/3648153650 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800bd50 con 0x7f44f009c320 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 4 set_mon_vals no callback set 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44eeffd700 10 monclient: dump: 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 10 monclient: shutdown 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 -- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009c320 msgr2=0x7f44f009c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009c320 0x7f44f009c780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44f0067590 tx=0x7f44e8009250 comp rx=0 tx=0).stop 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 -- 172.21.15.142:0/3648153650 shutdown_connections 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009c320 0x7f44f009c780 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 --2- 172.21.15.142:0/3648153650 >> v2:172.21.15.150:3300/0 conn(0x7f44f00967a0 0x7f44f009bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.153+0000 7f44f5193700 1 --2- 172.21.15.142:0/3648153650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f009aae0 0x7f44f0096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- 172.21.15.142:0/3648153650 >> 172.21.15.142:0/3648153650 conn(0x7f44f00a1fe0 msgr2=0x7f44f0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- 172.21.15.142:0/3648153650 shutdown_connections 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- 172.21.15.142:0/3648153650 wait complete. 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient: monmap: 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:04.171 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 Processor -- start 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- start start 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient: init 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 0x7f44f006df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-c con 0x7f44f00967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 0x7f44f006c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-a con 0x7f44f009aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44f009c320 0x7f44f006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): picked mon.noname-b con 0x7f44f009c320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006dc70 con 0x7f44f009aae0 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f0127300 con 0x7f44f009c320 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f01275e0 con 0x7f44f00967a0 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f5193700 10 monclient(hunting): authenticate will time out at 833.086588s 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 0x7f44f006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 0x7f44f006c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 0x7f44f006df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51102/0 (socket says 172.21.15.142:51102) 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 1 -- 172.21.15.142:0/76345898 learned_addr learned my addr 172.21.15.142:0/76345898 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): get_auth_request con 0x7f44f00967a0 auth_method 0 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): get_auth_request con 0x7f44f009aae0 auth_method 0 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44f009c320 0x7f44f006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): get_auth_request con 0x7f44f009c320 auth_method 0 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44f4992700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 -- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 msgr2=0x7f44f006df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 0x7f44f006df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 -- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44f009c320 msgr2=0x7f44f006c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44f009c320 0x7f44f006c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.157+0000 7f44ef7fe700 1 -- 172.21.15.142:0/76345898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0009a20 con 0x7f44f009aae0 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44ef7fe700 1 -- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 msgr2=0x7f44f006c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44ef7fe700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 0x7f44f006c0b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009fa0 tx=0x7f44e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e000cdb0 con 0x7f44f009aae0 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: dump: 2024-09-18T10:09:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 5 monclient: authenticate success, global_id 4218 2024-09-18T10:09:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e000f040 con 0x7f44f009aae0 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 10 monclient: _renew_subs 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80094f0 con 0x7f44f009aae0 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e00146b0 con 0x7f44f009aae0 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44ee7fc700 4 set_mon_vals no callback set 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 10 monclient: dump: 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.178 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 10 monclient: _renew_subs 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f01302b0 con 0x7f44f009aae0 2024-09-18T10:09:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f44e0014a60 con 0x7f44f009aae0 2024-09-18T10:09:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 0x7f44dc037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.161+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f44e0049000 con 0x7f44f009aae0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44f5193700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44f5193700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44f5193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f01277a0 con 0x7f44f009aae0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44effff700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 0x7f44dc037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44effff700 10 monclient: get_auth_request con 0x7f44dc035af0 auth_method 0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.165+0000 7f44effff700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 0x7f44dc037fa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009e70 tx=0x7f44e40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.169+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f44e0024530 con 0x7f44f009aae0 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.169+0000 7f44d7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.169+0000 7f44d7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:04.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.373+0000 7f44f5193700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-18T10:09:04.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.373+0000 7f44f5193700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T10:09:04.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.373+0000 7f44f5193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.373+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f44f01279d0 con 0x7f44f009aae0 2024-09-18T10:09:04.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.377+0000 7f44d7fff700 1 -- 172.21.15.142:0/76345898 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83404 (secure 0 0 0) 0x7f44e0007e60 con 0x7f44f009aae0 2024-09-18T10:09:04.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.377+0000 7f44d7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-18T10:09:04.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.377+0000 7f44d7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:04.385 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 msgr2=0x7f44dc037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 0x7f44dc037fa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009e70 tx=0x7f44e40091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 10 monclient: shutdown 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 msgr2=0x7f44f006c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 0x7f44f006c0b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009fa0 tx=0x7f44e000b040 comp rx=0 tx=0).stop 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 shutdown_connections 2024-09-18T10:09:04.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44dc035af0 0x7f44dc037fa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44f009c320 0x7f44f006c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f44f009aae0 0x7f44f006c0b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 --2- 172.21.15.142:0/76345898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f44f00967a0 0x7f44f006df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 >> 172.21.15.142:0/76345898 conn(0x7f44f00a1fe0 msgr2=0x7f44f009f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 shutdown_connections 2024-09-18T10:09:04.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.381+0000 7f44f5193700 1 -- 172.21.15.142:0/76345898 wait complete. 2024-09-18T10:09:04.399 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":20862},{"type":"v1","addr":"172.21.15.150:6809","nonce":20862}]},"active_addr":"172.21.15.150:6809/20862","active_change":"2024-09-18T10:08:52.752637+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1731884160}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2433973087}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":570026374}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":987359241}]}]} 2024-09-18T10:09:04.401 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T10:09:04.401 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T10:09:04.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: get_monmap_and_config 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: monmap: 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:04.508174+0000 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:04.508174+0000 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:04.509 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:04.510 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:04.510 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:04.510 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 Processor -- start 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 -- start start 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: init 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T10:09:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14e409a7f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e40993f0 0x7f14e413b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-b con 0x7f14e40993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f14e413bb30 0x7f14e413df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-a con 0x7f14e413bb30 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40997e0 con 0x7f14e409a7f0 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e409abc0 con 0x7f14e413bb30 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e409d950 con 0x7f14e40993f0 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14ea628700 10 monclient(hunting): authenticate will time out at 833.433626s 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f14e413bb30 0x7f14e413df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e40993f0 0x7f14e413b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f14e413bb30 0x7f14e413df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48366/0 (socket says 172.21.15.142:48366) 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 -- 172.21.15.142:0/2597352266 learned_addr learned my addr 172.21.15.142:0/2597352266 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): get_auth_request con 0x7f14e40993f0 auth_method 0 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): get_auth_request con 0x7f14e413bb30 auth_method 0 2024-09-18T10:09:04.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 -- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e40993f0 msgr2=0x7f14e413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 --2- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e40993f0 0x7f14e413b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 -- 172.21.15.142:0/2597352266 >> v1:172.21.15.142:6789/0 conn(0x7f14e409a7f0 legacy=0x7f14e4098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.505+0000 7f14e9e27700 1 -- 172.21.15.142:0/2597352266 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e409db90 con 0x7f14e413bb30 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14e9e27700 1 -- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 msgr2=0x7f14e413df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14e9e27700 1 --2- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e413df10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009f20 tx=0x7f14dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ac1c36e233c6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 1 -- 172.21.15.142:0/2597352266 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc00e070 con 0x7f14e413bb30 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: dump: 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.518 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 5 monclient: authenticate success, global_id 4221 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 1 -- 172.21.15.142:0/2597352266 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc00bbe0 con 0x7f14e413bb30 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 4 set_mon_vals no callback set 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 1 -- 172.21.15.142:0/2597352266 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc0092e0 con 0x7f14e413bb30 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14dbfff700 10 monclient: dump: 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 10 monclient: shutdown 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 msgr2=0x7f14e413df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 --2- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e413df10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009f20 tx=0x7f14dc009840 comp rx=0 tx=0).stop 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- 172.21.15.142:0/2597352266 shutdown_connections 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 --2- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e40993f0 0x7f14e413b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 --2- 172.21.15.142:0/2597352266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e413df10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- 172.21.15.142:0/2597352266 >> 172.21.15.142:0/2597352266 conn(0x7f14e40a1fe0 msgr2=0x7f14e4070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- 172.21.15.142:0/2597352266 shutdown_connections 2024-09-18T10:09:04.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- 172.21.15.142:0/2597352266 wait complete. 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 10 monclient: monmap: 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:04.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 Processor -- start 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 1 -- start start 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.509+0000 7f14ea628700 10 monclient: init 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e40993f0 0x7f14e406c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-b con 0x7f14e40993f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e409a7f0 0x7f14e406c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-c con 0x7f14e409a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): picked mon.noname-a con 0x7f14e413bb30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e406dff0 con 0x7f14e413bb30 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e406f970 con 0x7f14e40993f0 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40a0c10 con 0x7f14e409a7f0 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient(hunting): authenticate will time out at 833.440478s 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e409a7f0 0x7f14e406c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48368/0 (socket says 172.21.15.142:48368) 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 -- 172.21.15.142:0/3779758201 learned_addr learned my addr 172.21.15.142:0/3779758201 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): get_auth_request con 0x7f14e413bb30 auth_method 0 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): get_auth_request con 0x7f14e409a7f0 auth_method 0 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e40993f0 0x7f14e406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): get_auth_request con 0x7f14e40993f0 auth_method 0 2024-09-18T10:09:04.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e8e25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 -- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e409a7f0 msgr2=0x7f14e406c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e409a7f0 0x7f14e406c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 -- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e40993f0 msgr2=0x7f14e406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e40993f0 0x7f14e406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 -- 172.21.15.142:0/3779758201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc009ae0 con 0x7f14e413bb30 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 -- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 msgr2=0x7f14e406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14e9e27700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14e4071c90 tx=0x7f14dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc012a60 con 0x7f14e413bb30 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: dump: 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.530 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 5 monclient: authenticate success, global_id 4224 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc012be0 con 0x7f14e413bb30 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc01d420 con 0x7f14e413bb30 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _renew_subs 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e413b720 con 0x7f14e413bb30 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14d97fa700 10 monclient: dump: 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.532 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14db7fe700 4 set_mon_vals no callback set 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _renew_subs 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.513+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e406da70 con 0x7f14e413bb30 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f14dc01d790 con 0x7f14e413bb30 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14d97fa700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 0x7f14c8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14ea628700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14ea628700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f14dc0497f0 con 0x7f14e413bb30 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14ea628700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e4126870 con 0x7f14e413bb30 2024-09-18T10:09:04.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14e9626700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 0x7f14c8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.517+0000 7f14e9626700 10 monclient: get_auth_request con 0x7f14c8035ac0 auth_method 0 2024-09-18T10:09:04.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.521+0000 7f14e9626700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 0x7f14c8037f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14d4003a30 tx=0x7f14d4007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.521+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f14dc027460 con 0x7f14e413bb30 2024-09-18T10:09:04.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.521+0000 7f14d97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.521+0000 7f14d97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:04.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.685+0000 7f14ea628700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.685+0000 7f14ea628700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.685+0000 7f14ea628700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.685+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14e4092f60 con 0x7f14e413bb30 2024-09-18T10:09:04.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14d97fa700 1 -- 172.21.15.142:0/3779758201 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11452 (secure 0 0 0) 0x7f14dc026bb0 con 0x7f14e413bb30 2024-09-18T10:09:04.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14d97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14d97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:04.694 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:04.694 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":16,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:09:03.973811+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T10:09:00.830395+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.358800+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.344702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:09:04.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 msgr2=0x7f14c8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 0x7f14c8037f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f14d4003a30 tx=0x7f14d4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 10 monclient: shutdown 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 msgr2=0x7f14e406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.689+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f14e4071c90 tx=0x7f14dc017000 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 shutdown_connections 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14c8035ac0 0x7f14c8037f70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14e413bb30 0x7f14e406cb90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14e409a7f0 0x7f14e406c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 --2- 172.21.15.142:0/3779758201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14e40993f0 0x7f14e406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 >> 172.21.15.142:0/3779758201 conn(0x7f14e40a1fe0 msgr2=0x7f14e409fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 shutdown_connections 2024-09-18T10:09:04.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.693+0000 7f14ea628700 1 -- 172.21.15.142:0/3779758201 wait complete. 2024-09-18T10:09:04.708 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T10:09:04.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:09:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: get_monmap_and_config 2024-09-18T10:09:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: monmap: 2024-09-18T10:09:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:04.812454+0000 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:04.812454+0000 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:04.815 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 Processor -- start 2024-09-18T10:09:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 -- start start 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: init 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae8131d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-b con 0x7f0ae80676a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0ae80985c0 0x7f0ae812e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-a con 0x7f0ae80985c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ae8098990 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae806f150 con 0x7f0ae8098990 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae8096af0 con 0x7f0ae80985c0 2024-09-18T10:09:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8067a70 con 0x7f0ae80676a0 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aedd01700 10 monclient(hunting): authenticate will time out at 833.738517s 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0ae80985c0 0x7f0ae812e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0ae80985c0 0x7f0ae812e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48376/0 (socket says 172.21.15.142:48376) 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 -- 172.21.15.142:0/2809388426 learned_addr learned my addr 172.21.15.142:0/2809388426 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): get_auth_request con 0x7f0ae80985c0 auth_method 0 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae8131d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 10 monclient(hunting): get_auth_request con 0x7f0ae80676a0 auth_method 0 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 -- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 msgr2=0x7f0ae8131d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae8131d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 -- 172.21.15.142:0/2809388426 >> v1:172.21.15.142:6789/0 conn(0x7f0ae8098990 legacy=0x7f0ae812ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0adffff700 1 -- 172.21.15.142:0/2809388426 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae809dd80 con 0x7f0ae80985c0 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.809+0000 7f0aeccff700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae8131d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adffff700 1 -- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 msgr2=0x7f0ae812e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adffff700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812e7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ae80a1cc0 tx=0x7f0ad8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d6728c6c9e5d89 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 1 -- 172.21.15.142:0/2809388426 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800ba70 con 0x7f0ae80985c0 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: dump: 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 5 monclient: authenticate success, global_id 4227 2024-09-18T10:09:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 1 -- 172.21.15.142:0/2809388426 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad800bbd0 con 0x7f0ae80985c0 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 4 set_mon_vals no callback set 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 1 -- 172.21.15.142:0/2809388426 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8011420 con 0x7f0ae80985c0 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0adf7fe700 10 monclient: dump: 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 10 monclient: shutdown 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 msgr2=0x7f0ae812e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812e7f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ae80a1cc0 tx=0x7f0ad8009250 comp rx=0 tx=0).stop 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- 172.21.15.142:0/2809388426 shutdown_connections 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812e7f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 --2- 172.21.15.142:0/2809388426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae8131d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- 172.21.15.142:0/2809388426 >> 172.21.15.142:0/2809388426 conn(0x7f0ae8071ae0 msgr2=0x7f0ae80a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- 172.21.15.142:0/2809388426 shutdown_connections 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- 172.21.15.142:0/2809388426 wait complete. 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 10 monclient: build_initial_monmap 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 10 monclient: monmap: 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 Processor -- start 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 1 -- start start 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.813+0000 7f0aedd01700 10 monclient: init 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-c con 0x7f0ae80676a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-a con 0x7f0ae80985c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 0x7f0ae8118670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): picked mon.noname-b con 0x7f0ae8098990 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae811bee0 con 0x7f0ae80985c0 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae809df90 con 0x7f0ae8098990 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8130e90 con 0x7f0ae80676a0 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aedd01700 10 monclient(hunting): authenticate will time out at 833.744618s 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51128/0 (socket says 172.21.15.142:51128) 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48378/0 (socket says 172.21.15.142:48378) 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 1 -- 172.21.15.142:0/3883448742 learned_addr learned my addr 172.21.15.142:0/3883448742 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): get_auth_request con 0x7f0ae80676a0 auth_method 0 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): get_auth_request con 0x7f0ae80985c0 auth_method 0 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 0x7f0ae8118670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 10 monclient(hunting): get_auth_request con 0x7f0ae8098990 auth_method 0 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 -- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 msgr2=0x7f0ae81413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 -- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 msgr2=0x7f0ae8118670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 0x7f0ae8118670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 monclient: found mon.noname-a 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:04.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 -- 172.21.15.142:0/3883448742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad80094f0 con 0x7f0ae80985c0 2024-09-18T10:09:04.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aeccff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:04.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0aed500700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 0x7f0ae8118670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 -- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 msgr2=0x7f0ae812eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.817+0000 7f0adffff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8016000 tx=0x7f0ad8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8006a90 con 0x7f0ae80985c0 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: dump: 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.837 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad801a630 con 0x7f0ae80985c0 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 5 monclient: authenticate success, global_id 4230 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _renew_subs 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae811ad70 con 0x7f0ae80985c0 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:04.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad801a9f0 con 0x7f0ae80985c0 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 10 monclient: dump: 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:04.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _renew_subs 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81421a0 con 0x7f0ae80985c0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adeffd700 4 set_mon_vals no callback set 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0ad802f3f0 con 0x7f0ae80985c0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae8072a80 con 0x7f0ae80985c0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 0x7f0ad4037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f0ad8049d10 con 0x7f0ae80985c0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aeccff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 0x7f0ad4037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.821+0000 7f0aeccff700 10 monclient: get_auth_request con 0x7f0ad4035b10 auth_method 0 2024-09-18T10:09:04.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.825+0000 7f0aeccff700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 0x7f0ad4037fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4006dc0 tx=0x7f0ae4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:04.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.829+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0ad80193b0 con 0x7f0ae80985c0 2024-09-18T10:09:04.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.829+0000 7f0adcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:04.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.829+0000 7f0adcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0aedd01700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0aedd01700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0aedd01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ae811c050 con 0x7f0ae80985c0 2024-09-18T10:09:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0adcff9700 1 -- 172.21.15.142:0/3883448742 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11452 (secure 0 0 0) 0x7f0ad8015030 con 0x7f0ae80985c0 2024-09-18T10:09:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0adcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:04.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.993+0000 7f0adcff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:04.999 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:04.999 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":16,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:09:03.973811+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T10:09:00.830395+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.358800+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.344702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 msgr2=0x7f0ad4037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 0x7f0ad4037fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4006dc0 tx=0x7f0ae4008040 comp rx=0 tx=0).stop 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 10 monclient: shutdown 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 msgr2=0x7f0ae812eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8016000 tx=0x7f0ad8017000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 shutdown_connections 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ad4035b10 0x7f0ad4037fc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ae8098990 0x7f0ae8118670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ae80985c0 0x7f0ae812eae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 --2- 172.21.15.142:0/3883448742 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ae80676a0 0x7f0ae81413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 >> 172.21.15.142:0/3883448742 conn(0x7f0ae8071ae0 msgr2=0x7f0ae80a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 shutdown_connections 2024-09-18T10:09:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:04.997+0000 7f0aedd01700 1 -- 172.21.15.142:0/3883448742 wait complete. 2024-09-18T10:09:05.012 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T10:09:05.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T10:09:05.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T10:09:05.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T10:09:05.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-18T10:09:05.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-18T10:09:05.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: monmap: 2024-09-18T10:09:05.112 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.112 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.108756+0000 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.108756+0000 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 Processor -- start 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 -- start start 2024-09-18T10:09:05.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: init 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88100a1bf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f881009b150 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-b con 0x7f881009b590 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8810098c80 con 0x7f88100a1bf0 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881006fbc0 con 0x7f881009b150 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8810132680 con 0x7f881009b590 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f8815b00700 10 monclient(hunting): authenticate will time out at 834.034328s 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51132/0 (socket says 172.21.15.142:51132) 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 -- 172.21.15.142:0/3662817210 learned_addr learned my addr 172.21.15.142:0/3662817210 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): get_auth_request con 0x7f881009b590 auth_method 0 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 -- 172.21.15.142:0/3662817210 >> v1:172.21.15.150:6789/0 conn(0x7f881009b150 legacy=0x7f88101371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 -- 172.21.15.142:0/3662817210 >> v1:172.21.15.142:6789/0 conn(0x7f88100a1bf0 legacy=0x7f8810131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.105+0000 7f88152ff700 1 -- 172.21.15.142:0/3662817210 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100a1fc0 con 0x7f881009b590 2024-09-18T10:09:05.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f88152ff700 1 -- 172.21.15.142:0/3662817210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 msgr2=0x7f881013a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f88152ff700 1 --2- 172.21.15.142:0/3662817210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8808009f50 tx=0x7f8808009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72bb3cd4fa6eafb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 1 -- 172.21.15.142:0/3662817210 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800e070 con 0x7f881009b590 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: dump: 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 1 -- 172.21.15.142:0/3662817210 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880800bcc0 con 0x7f881009b590 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 1 -- 172.21.15.142:0/3662817210 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808012680 con 0x7f881009b590 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f880f7fe700 10 monclient: dump: 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 5 monclient: authenticate success, global_id 4229 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 4 set_mon_vals no callback set 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 10 monclient: shutdown 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 -- 172.21.15.142:0/3662817210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 msgr2=0x7f881013a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 --2- 172.21.15.142:0/3662817210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8808009f50 tx=0x7f8808009840 comp rx=0 tx=0).stop 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 -- 172.21.15.142:0/3662817210 shutdown_connections 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 --2- 172.21.15.142:0/3662817210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f881013a8d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 -- 172.21.15.142:0/3662817210 >> 172.21.15.142:0/3662817210 conn(0x7f8810071680 msgr2=0x7f88100a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 -- 172.21.15.142:0/3662817210 shutdown_connections 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 1 -- 172.21.15.142:0/3662817210 wait complete. 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.109+0000 7f8815b00700 10 monclient: monmap: 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.122 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 Processor -- start 2024-09-18T10:09:05.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: monmap: 2024-09-18T10:09:05.124 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.124 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.115898+0000 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.115898+0000 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.125 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.126 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.126 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.126 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 -- start start 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient: init 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f881009b150 0x7f8810146170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-a con 0x7f881009b150 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f88101466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-c con 0x7f881009b590 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88100a1bf0 0x7f881006ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): picked mon.noname-b con 0x7f88100a1bf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100a0700 con 0x7f881009b150 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881006e610 con 0x7f88100a1bf0 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881006e330 con 0x7f881009b590 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f88101466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f881009b150 0x7f8810146170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8815b00700 10 monclient(hunting): authenticate will time out at 834.041282s 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f88101466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51138/0 (socket says 172.21.15.142:51138) 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 -- 172.21.15.142:0/2321733463 learned_addr learned my addr 172.21.15.142:0/2321733463 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): get_auth_request con 0x7f881009b590 auth_method 0 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): get_auth_request con 0x7f881009b150 auth_method 0 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f8814afe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 Processor -- start 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f88152ff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88100a1bf0 0x7f881006ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f88152ff700 10 monclient(hunting): get_auth_request con 0x7f88100a1bf0 auth_method 0 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f88152ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f88152ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f88152ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 -- start start 2024-09-18T10:09:05.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: init 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88100a1bf0 msgr2=0x7f881006ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88100a1bf0 0x7f881006ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f881009b150 msgr2=0x7f8810146170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f881009b150 0x7f8810146170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f880ffff700 1 -- 172.21.15.142:0/2321733463 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8808009ae0 con 0x7f881009b590 2024-09-18T10:09:05.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f770006f3b0 0x7f77000989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): picked mon.noname-a con 0x7f770006f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7700098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7700099320 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770009c330 con 0x7f7700098f00 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770009c530 con 0x7f7700099320 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770009b030 con 0x7f770006f3b0 2024-09-18T10:09:05.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7706886700 10 monclient(hunting): authenticate will time out at 834.042570s 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705083700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7700098f00 0x7f7700133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46482/0 (socket says 172.21.15.142:46482) 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705083700 1 -- 172.21.15.142:0/2275702591 learned_addr learned my addr 172.21.15.142:0/2275702591 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 1 --2- 172.21.15.142:0/2275702591 >> v2:172.21.15.142:3300/0 conn(0x7f770006f3b0 0x7f77000989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): get_auth_request con 0x7f770006f3b0 auth_method 0 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 1 -- 172.21.15.142:0/2275702591 >> v1:172.21.15.150:6789/0 conn(0x7f7700099320 legacy=0x7f7700136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 1 -- 172.21.15.142:0/2275702591 >> v1:172.21.15.142:6789/0 conn(0x7f7700098f00 legacy=0x7f7700133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.113+0000 7f7705884700 1 -- 172.21.15.142:0/2275702591 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770009c0f0 con 0x7f770006f3b0 2024-09-18T10:09:05.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880ffff700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 msgr2=0x7f88101466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880ffff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f88101466b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f880000c600 tx=0x7f8800007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88000117a0 con 0x7f881009b590 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: dump: 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.140 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 5 monclient: authenticate success, global_id 4232 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800011920 con 0x7f881009b590 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 10 monclient: _renew_subs 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881006e050 con 0x7f881009b590 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 10 monclient: _renew_subs 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8810090cc0 con 0x7f881009b590 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800011ce0 con 0x7f881009b590 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 10 monclient: dump: 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.143 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7705884700 1 -- 172.21.15.142:0/2275702591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f770006f3b0 msgr2=0x7f77000989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7705884700 1 --2- 172.21.15.142:0/2275702591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f770006f3b0 0x7f77000989c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098e0 tx=0x7f76fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9556461b50d2bb2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8800012630 con 0x7f881009b590 2024-09-18T10:09:05.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 1 -- 172.21.15.142:0/2275702591 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00ba70 con 0x7f770006f3b0 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: dump: 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.146 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 1 -- 172.21.15.142:0/2275702591 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc00bbd0 con 0x7f770006f3b0 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 1 -- 172.21.15.142:0/2275702591 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc011420 con 0x7f770006f3b0 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7704882700 10 monclient: dump: 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.148 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87fc035ac0 0x7f87fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 5 monclient: authenticate success, global_id 4236 2024-09-18T10:09:05.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 4 set_mon_vals no callback set 2024-09-18T10:09:05.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f8800047fa0 con 0x7f881009b590 2024-09-18T10:09:05.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8814afe700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87fc035ac0 0x7f87fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f8814afe700 10 monclient: get_auth_request con 0x7f87fc035ac0 auth_method 0 2024-09-18T10:09:05.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 10 monclient: shutdown 2024-09-18T10:09:05.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 1 -- 172.21.15.142:0/2275702591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f770006f3b0 msgr2=0x7f77000989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.117+0000 7f7706886700 1 --2- 172.21.15.142:0/2275702591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f770006f3b0 0x7f77000989c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098e0 tx=0x7f76fc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f880effd700 4 set_mon_vals no callback set 2024-09-18T10:09:05.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- 172.21.15.142:0/2275702591 shutdown_connections 2024-09-18T10:09:05.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 --2- 172.21.15.142:0/2275702591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f770006f3b0 0x7f77000989c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- 172.21.15.142:0/2275702591 >> 172.21.15.142:0/2275702591 conn(0x7f7700070ef0 msgr2=0x7f77000712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- 172.21.15.142:0/2275702591 shutdown_connections 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- 172.21.15.142:0/2275702591 wait complete. 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient: monmap: 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.153 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.154 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.154 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f8814afe700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87fc035ac0 0x7f87fc037f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8804003a30 tx=0x7f8804007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 Processor -- start 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- start start 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient: init 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f770006f3b0 0x7f7700139910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): picked mon.noname-c con 0x7f770006f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 0x7f7700139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): picked mon.noname-a con 0x7f7700098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7700099320 0x7f7700089b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): picked mon.noname-b con 0x7f7700099320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770013a390 con 0x7f7700098f00 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700097410 con 0x7f7700099320 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770008b6a0 con 0x7f770006f3b0 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706886700 10 monclient(hunting): authenticate will time out at 834.050911s 2024-09-18T10:09:05.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 0x7f7700139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f770006f3b0 0x7f7700139910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f770006f3b0 0x7f7700139910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51148/0 (socket says 172.21.15.142:51148) 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): get_auth_request con 0x7f7700098f00 auth_method 0 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 1 -- 172.21.15.142:0/4255059063 learned_addr learned my addr 172.21.15.142:0/4255059063 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): get_auth_request con 0x7f770006f3b0 auth_method 0 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7700099320 0x7f7700089b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): get_auth_request con 0x7f7700099320 auth_method 0 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7706085700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f770006f3b0 msgr2=0x7f7700139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f770006f3b0 0x7f7700139910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7700099320 msgr2=0x7f7700089b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7700099320 0x7f7700089b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.121+0000 7f7705083700 1 -- 172.21.15.142:0/4255059063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc009420 con 0x7f7700098f00 2024-09-18T10:09:05.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f8815b00700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f87f00015f0 0x7f87f0003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 --> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87f0006ae0 con 0x7f87f00015f0 2024-09-18T10:09:05.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7705083700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 msgr2=0x7f7700139e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7705083700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 0x7f7700139e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76f800cc10 tx=0x7f76f80053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f88152ff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f87f00015f0 0x7f87f0003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f800a9a0 con 0x7f7700098f00 2024-09-18T10:09:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: dump: 2024-09-18T10:09:05.164 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f800ab20 con 0x7f7700098f00 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80070f0 con 0x7f7700098f00 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 10 monclient: dump: 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.166 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f7fff700 4 set_mon_vals no callback set 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 5 monclient: authenticate success, global_id 4239 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 10 monclient: _renew_subs 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770006fa20 con 0x7f7700098f00 2024-09-18T10:09:05.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f88152ff700 10 monclient: get_auth_request con 0x7f87f00015f0 auth_method 0 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 10 monclient: _renew_subs 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770008d1d0 con 0x7f7700098f00 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f76f80074a0 con 0x7f7700098f00 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76e4035ac0 0x7f76e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.125+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f76f80485b0 con 0x7f7700098f00 2024-09-18T10:09:05.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f7705884700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76e4035ac0 0x7f76e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f7705884700 10 monclient: get_auth_request con 0x7f76e4035ac0 auth_method 0 2024-09-18T10:09:05.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f7705884700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76e4035ac0 0x7f76e4037f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0093f0 tx=0x7f76fc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f88152ff700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f87f00015f0 0x7f87f0003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f7706886700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f770008a0a0 0x7f770008a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.129+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f770013c4e0 con 0x7f770008a0a0 2024-09-18T10:09:05.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.133+0000 7f7706085700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f770008a0a0 0x7f770008a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.133+0000 7f7706085700 10 monclient: get_auth_request con 0x7f770008a0a0 auth_method 0 2024-09-18T10:09:05.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.133+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== osd.3 v2:172.21.15.150:6800/21119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f87f0006ae0 con 0x7f87f00015f0 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.137+0000 7f7706085700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f770008a0a0 0x7f770008a470 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.137+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== osd.5 v2:172.21.15.150:6810/21120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f770013c4e0 con 0x7f770008a0a0 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: monmap: 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.172 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.144725+0000 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.144725+0000 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.173 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f629a283700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f629a283700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f629a283700 10 monclient: monmap: 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.145579+0000 2024-09-18T10:09:05.174 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.145579+0000 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.175 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 Processor -- start 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 -- start start 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: init 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21fc1366b0 0x7f21fc136a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-c con 0x7f21fc1366b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-b con 0x7f21fc09db60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 0x7f21fc09c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-a con 0x7f21fc09bed0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc0989c0 con 0x7f21fc09bed0 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc099600 con 0x7f21fc1366b0 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc137500 con 0x7f21fc09db60 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f22033a2700 10 monclient(hunting): authenticate will time out at 834.070412s 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 0x7f21fc09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 0x7f21fc09c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48412/0 (socket says 172.21.15.142:48412) 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 1 -- 172.21.15.142:0/2569486065 learned_addr learned my addr 172.21.15.142:0/2569486065 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): get_auth_request con 0x7f21fc09bed0 auth_method 0 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): get_auth_request con 0x7f21fc09db60 auth_method 0 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220113e700 1 --2- 172.21.15.142:0/2569486065 >> v2:172.21.15.150:3300/0 conn(0x7f21fc1366b0 0x7f21fc136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220193f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 -- 172.21.15.142:0/2569486065 >> v2:172.21.15.150:3300/0 conn(0x7f21fc1366b0 msgr2=0x7f21fc136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 --2- 172.21.15.142:0/2569486065 >> v2:172.21.15.150:3300/0 conn(0x7f21fc1366b0 0x7f21fc136a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 -- 172.21.15.142:0/2569486065 >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 msgr2=0x7f21fc09c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 --2- 172.21.15.142:0/2569486065 >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 0x7f21fc09c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.141+0000 7f220093d700 1 -- 172.21.15.142:0/2569486065 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc09a5b0 con 0x7f21fc09db60 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 Processor -- start 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 -- start start 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient: init 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): picked mon.noname-b con 0x7f629409bed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f629406f230 0x7f629409c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): picked mon.noname-c con 0x7f629406f230 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 0x7f62941386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): picked mon.noname-a con 0x7f629409c7e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6294098eb0 con 0x7f629409c7e0 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629406f600 con 0x7f629406f230 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629406f770 con 0x7f629409bed0 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f629a283700 10 monclient(hunting): authenticate will time out at 834.072679s 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51166/0 (socket says 172.21.15.142:51166) 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 -- 172.21.15.142:0/1072345846 learned_addr learned my addr 172.21.15.142:0/1072345846 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): get_auth_request con 0x7f629409bed0 auth_method 0 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 0x7f62941386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): get_auth_request con 0x7f629409c7e0 auth_method 0 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 -- 172.21.15.142:0/1072345846 >> v2:172.21.15.150:3300/0 conn(0x7f629406f230 msgr2=0x7f629409c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.150:3300/0 conn(0x7f629406f230 0x7f629409c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 -- 172.21.15.142:0/1072345846 >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 msgr2=0x7f62941386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 0x7f62941386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 -- 172.21.15.142:0/1072345846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6294098ad0 con 0x7f629409bed0 2024-09-18T10:09:05.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6298820700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 0x7f62941386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f220093d700 1 -- 172.21.15.142:0/2569486065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 msgr2=0x7f21fc136fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f220093d700 1 --2- 172.21.15.142:0/2569486065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc136fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21fc069d90 tx=0x7f21f80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2b03c673610abe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 1 -- 172.21.15.142:0/2569486065 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8011840 con 0x7f21fc09db60 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: dump: 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 5 monclient: authenticate success, global_id 4238 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 1 -- 172.21.15.142:0/2569486065 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f80119a0 con 0x7f21fc09db60 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 1 -- 172.21.15.142:0/2569486065 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8011d40 con 0x7f21fc09db60 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 4 set_mon_vals no callback set 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f21f3fff700 10 monclient: dump: 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.192 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 10 monclient: shutdown 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 -- 172.21.15.142:0/2569486065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 msgr2=0x7f21fc136fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 --2- 172.21.15.142:0/2569486065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc136fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21fc069d90 tx=0x7f21f80091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 -- 172.21.15.142:0/2569486065 shutdown_connections 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 --2- 172.21.15.142:0/2569486065 >> v2:172.21.15.142:3300/0 conn(0x7f21fc09bed0 0x7f21fc09c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 --2- 172.21.15.142:0/2569486065 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc136fc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 --2- 172.21.15.142:0/2569486065 >> v2:172.21.15.150:3300/0 conn(0x7f21fc1366b0 0x7f21fc136a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 -- 172.21.15.142:0/2569486065 >> 172.21.15.142:0/2569486065 conn(0x7f21fc071820 msgr2=0x7f21fc0a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 -- 172.21.15.142:0/2569486065 shutdown_connections 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 -- 172.21.15.142:0/2569486065 wait complete. 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 10 monclient: monmap: 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.195 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.196 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.196 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 -- 172.21.15.142:0/1072345846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 msgr2=0x7f62940991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f6293fff700 1 --2- 172.21.15.142:0/1072345846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f62940a2010 tx=0x7f628c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b089fe97a547c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.145+0000 7f22033a2700 1 Processor -- start 2024-09-18T10:09:05.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 1 -- 172.21.15.142:0/1072345846 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c00e070 con 0x7f629409bed0 2024-09-18T10:09:05.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 -- start start 2024-09-18T10:09:05.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient: init 2024-09-18T10:09:05.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: dump: 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.199 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 1 -- 172.21.15.142:0/1072345846 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c00bc30 con 0x7f629409bed0 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: monmap: 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.151836+0000 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.151836+0000 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.201 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.202 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.202 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.202 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 1 -- 172.21.15.142:0/1072345846 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0092e0 con 0x7f629409bed0 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 5 monclient: authenticate success, global_id 4241 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 4 set_mon_vals no callback set 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 0x7f21fc145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-a con 0x7f21fc09bed0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6292ffd700 10 monclient: dump: 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.204 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: shutdown 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- 172.21.15.142:0/1072345846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 msgr2=0x7f62940991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- 172.21.15.142:0/1072345846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f62940a2010 tx=0x7f628c009840 comp rx=0 tx=0).stop 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-c con 0x7f21fc09db60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21fc1366b0 0x7f21fc06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): picked mon.noname-b con 0x7f21fc1366b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc1467a0 con 0x7f21fc09bed0 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc06f3e0 con 0x7f21fc1366b0 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc06f100 con 0x7f21fc09db60 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f22033a2700 10 monclient(hunting): authenticate will time out at 834.077003s 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 0x7f21fc145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- 172.21.15.142:0/1072345846 shutdown_connections 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.142:3300/0 conn(0x7f629409c7e0 0x7f62941386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- 172.21.15.142:0/1072345846 >> v2:172.21.15.150:3300/0 conn(0x7f629406f230 0x7f629409c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- 172.21.15.142:0/1072345846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629409bed0 0x7f62940991d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- 172.21.15.142:0/1072345846 >> 172.21.15.142:0/1072345846 conn(0x7f6294071820 msgr2=0x7f62940a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 0x7f21fc145880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48440/0 (socket says 172.21.15.142:48440) 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 -- 172.21.15.142:0/1472944555 learned_addr learned my addr 172.21.15.142:0/1472944555 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): get_auth_request con 0x7f21fc09bed0 auth_method 0 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- 172.21.15.142:0/1072345846 shutdown_connections 2024-09-18T10:09:05.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- 172.21.15.142:0/1072345846 wait complete. 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 10 monclient(hunting): get_auth_request con 0x7f21fc09db60 auth_method 0 2024-09-18T10:09:05.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220193f700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21fc1366b0 0x7f21fc06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 msgr2=0x7f21fc135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21fc1366b0 msgr2=0x7f21fc06bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21fc1366b0 0x7f21fc06bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220113e700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8009420 con 0x7f21fc09bed0 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220193f700 10 monclient: get_auth_request con 0x7f21fc1366b0 auth_method 0 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220193f700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21fc1366b0 0x7f21fc06bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f220093d700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21fc09db60 0x7f21fc135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: monmap: 2024-09-18T10:09:05.212 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.213 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f579f2bc700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f579f2bc700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f579f2bc700 10 monclient: monmap: 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.153040+0000 2024-09-18T10:09:05.214 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.153040+0000 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.215 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 Processor -- start 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 -- start start 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: init 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc46809bed0 0x7fc46809a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-a con 0x7fc46809bed0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc46809c2a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 0x7fc468135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-c con 0x7fc468133870 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4680644b0 con 0x7fc46809c2a0 2024-09-18T10:09:05.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46809d340 con 0x7fc46809bed0 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc468067730 con 0x7fc468133870 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46ff3e700 10 monclient(hunting): authenticate will time out at 834.078478s 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 Processor -- start 2024-09-18T10:09:05.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 0x7fc468135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- start start 2024-09-18T10:09:05.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: init 2024-09-18T10:09:05.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 0x7fc468135c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:40108/0 (socket says 172.21.15.142:40108) 2024-09-18T10:09:05.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 1 -- 172.21.15.142:0/188006209 learned_addr learned my addr 172.21.15.142:0/188006209 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 10 monclient(hunting): get_auth_request con 0x7fc468133870 auth_method 0 2024-09-18T10:09:05.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7fc46e4db700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:05.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 0x7f6294141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): picked mon.noname-c con 0x7f629406f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f629409bed0 0x7f62940943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): picked mon.noname-a con 0x7f629409bed0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629409c7e0 0x7f629408e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): picked mon.noname-b con 0x7f629409c7e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294090080 con 0x7f629409bed0 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629409d550 con 0x7f629409c7e0 2024-09-18T10:09:05.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62940a1110 con 0x7f629406f230 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f629a283700 10 monclient(hunting): authenticate will time out at 834.079657s 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 0x7f6294141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 0x7f6294141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51182/0 (socket says 172.21.15.142:51182) 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 1 -- 172.21.15.142:0/4113037351 learned_addr learned my addr 172.21.15.142:0/4113037351 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): get_auth_request con 0x7f629406f230 auth_method 0 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.149+0000 7f6293fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220113e700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 msgr2=0x7f21fc145880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220113e700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 0x7f21fc145880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00c7d0 tx=0x7f21ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 --2- 172.21.15.142:0/188006209 >> v2:172.21.15.142:3300/0 conn(0x7fc46809bed0 0x7fc46809a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): get_auth_request con 0x7fc46809bed0 auth_method 0 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629409c7e0 msgr2=0x7f629408e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629409c7e0 0x7f629408e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f629409bed0 msgr2=0x7f62940943a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f629409bed0 0x7f62940943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284009420 con 0x7f629406f230 2024-09-18T10:09:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec011810 con 0x7f21fc09bed0 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: dump: 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec011970 con 0x7f21fc09bed0 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 5 monclient: authenticate success, global_id 4248 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 10 monclient: _renew_subs 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc06f6c0 con 0x7f21fc09bed0 2024-09-18T10:09:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-09-18T10:09:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 -- 172.21.15.142:0/188006209 >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 msgr2=0x7fc468135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 --2- 172.21.15.142:0/188006209 >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 0x7fc468135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 -- 172.21.15.142:0/188006209 >> v1:172.21.15.150:6789/0 conn(0x7fc46809c2a0 legacy=0x7fc46809c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 -- 172.21.15.142:0/188006209 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468099ff0 con 0x7fc46809bed0 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 10 monclient: _renew_subs 2024-09-18T10:09:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc09f820 con 0x7f21fc09bed0 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec011d10 con 0x7f21fc09bed0 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f37fe700 4 set_mon_vals no callback set 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: dump: 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.232 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21dc000f20 con 0x7f21fc09bed0 2024-09-18T10:09:05.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 msgr2=0x7f6294141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f6293fff700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 0x7f6294141940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f628c00bfd0 tx=0x7f628c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f21ec010560 con 0x7f21fc09bed0 2024-09-18T10:09:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 Processor -- start 2024-09-18T10:09:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 -- start start 2024-09-18T10:09:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient: init 2024-09-18T10:09:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c007a30 con 0x7f629406f230 2024-09-18T10:09:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21e0035ac0 0x7f21e0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: dump: 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c007bd0 con 0x7f629406f230 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c026460 con 0x7f629406f230 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f627bfff700 10 monclient: dump: 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 5 monclient: authenticate success, global_id 4247 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 10 monclient: _renew_subs 2024-09-18T10:09:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f21ec048dc0 con 0x7f21fc09bed0 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c009ae0 con 0x7f629406f230 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 10 monclient: _renew_subs 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62940906d0 con 0x7f629406f230 2024-09-18T10:09:05.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f62927fc700 4 set_mon_vals no callback set 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7f21e003b940 0x7f21e003dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21e003e440 con 0x7f21e003b940 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21ec049060 con 0x7f21fc09bed0 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f21f17fa700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220093d700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21e0035ac0 0x7f21e0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220093d700 10 monclient: get_auth_request con 0x7f21e0035ac0 auth_method 0 2024-09-18T10:09:05.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 -- 172.21.15.142:0/188006209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809bed0 msgr2=0x7fc46809a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46dcda700 1 --2- 172.21.15.142:0/188006209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809bed0 0x7fc46809a3a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc46809db70 tx=0x7fc4640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4111d86af121874 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220193f700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7f21e003b940 0x7f21e003dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f220193f700 10 monclient: get_auth_request con 0x7f21e003b940 auth_method 0 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46ccd8700 1 -- 172.21.15.142:0/188006209 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400b8c0 con 0x7fc46809bed0 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7fc46ccd8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: dump: 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.243 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 1 -- 172.21.15.142:0/188006209 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46400ba20 con 0x7fc46809bed0 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 1 -- 172.21.15.142:0/188006209 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400bdc0 con 0x7fc46809bed0 2024-09-18T10:09:05.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ccd8700 10 monclient: dump: 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.245 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 5 monclient: authenticate success, global_id 4251 2024-09-18T10:09:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 4 set_mon_vals no callback set 2024-09-18T10:09:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f579809abd0 0x7f579812d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-a con 0x7f579809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5798097b70 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981305b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-b con 0x7f5798097fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579809afa0 con 0x7f5798097b70 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5798098430 con 0x7f579809abd0 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579809b340 con 0x7f5798097fb0 2024-09-18T10:09:05.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579f2bc700 10 monclient(hunting): authenticate will time out at 834.083668s 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.153+0000 7f579c857700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5798097b70 0x7f579812fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46490/0 (socket says 172.21.15.142:46490) 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579c857700 1 -- 172.21.15.142:0/3039560152 learned_addr learned my addr 172.21.15.142:0/3039560152 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 --2- 172.21.15.142:0/3039560152 >> v2:172.21.15.142:3300/0 conn(0x7f579809abd0 0x7f579812d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): get_auth_request con 0x7f579809abd0 auth_method 0 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f629a283700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62940916e0 con 0x7f629406f230 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f628c01d420 con 0x7f629406f230 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): get_auth_request con 0x7f5798097fb0 auth_method 0 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6280035b10 0x7f6280037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-09-18T10:09:05.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: shutdown 2024-09-18T10:09:05.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- 172.21.15.142:0/188006209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809bed0 msgr2=0x7fc46809a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/188006209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809bed0 0x7fc46809a3a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc46809db70 tx=0x7fc4640091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f628c04a010 con 0x7f629406f230 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 -- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 msgr2=0x7f57981305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 -- 172.21.15.142:0/3039560152 >> v1:172.21.15.142:6789/0 conn(0x7f5798097b70 legacy=0x7f579812fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 -- 172.21.15.142:0/3039560152 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579809d840 con 0x7f579809abd0 2024-09-18T10:09:05.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d859700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f628003ba10 0x7f628003de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f628003e510 con 0x7f628003ba10 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f628c04c5a0 con 0x7f629406f230 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f627bfff700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- 172.21.15.142:0/188006209 shutdown_connections 2024-09-18T10:09:05.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/188006209 >> v2:172.21.15.150:3300/0 conn(0x7fc468133870 0x7fc468135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/188006209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809bed0 0x7fc46809a3a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc46809db70 tx=0x7fc4640091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- 172.21.15.142:0/188006209 >> 172.21.15.142:0/188006209 conn(0x7fc468071820 msgr2=0x7fc4680a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- 172.21.15.142:0/188006209 shutdown_connections 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f6298820700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f628003ba10 0x7f628003de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f6298820700 10 monclient: get_auth_request con 0x7f628003ba10 auth_method 0 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f62937fe700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6280035b10 0x7f6280037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f62937fe700 10 monclient: get_auth_request con 0x7f6280035b10 auth_method 0 2024-09-18T10:09:05.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 --> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87f0005ba0 con 0x7f87f00015f0 2024-09-18T10:09:05.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- 172.21.15.142:0/188006209 wait complete. 2024-09-18T10:09:05.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f220093d700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21e0035ac0 0x7f21e0037f70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f21fc06d200 tx=0x7f21f8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f880cff9700 1 -- 172.21.15.142:0/2321733463 <== osd.3 v2:172.21.15.150:6800/21119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f87f0005ba0 con 0x7f87f00015f0 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: monmap: 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.257 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f87f00015f0 msgr2=0x7f87f0003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f87f00015f0 0x7f87f0003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87fc035ac0 msgr2=0x7f87fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87fc035ac0 0x7f87fc037f70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8804003a30 tx=0x7f8804007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 10 monclient: shutdown 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 msgr2=0x7f88101466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 --2- 172.21.15.142:0/2321733463 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f881009b590 0x7f88101466b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f880000c600 tx=0x7f8800007f60 comp rx=0 tx=0).stop 2024-09-18T10:09:05.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 Processor -- start 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f88152ff700 1 -- 172.21.15.142:0/2321733463 reap_dead start 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 shutdown_connections 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 >> 172.21.15.142:0/2321733463 conn(0x7f8810071680 msgr2=0x7f881013a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 shutdown_connections 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f8815b00700 1 -- 172.21.15.142:0/2321733463 wait complete. 2024-09-18T10:09:05.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- start start 2024-09-18T10:09:05.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: init 2024-09-18T10:09:05.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f770008c270 con 0x7f770008a0a0 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809c2a0 0x7fc46813fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-a con 0x7fc46809c2a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc468133870 0x7fc468140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-b con 0x7fc468133870 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 -- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f579809abd0 msgr2=0x7f579812d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 0x7fc468087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7f579d058700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f579809abd0 0x7f579812d0a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57940098e0 tx=0x7f57940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8945b23458953e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): picked mon.noname-c con 0x7fc46808c4e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680992d0 con 0x7fc46809c2a0 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc468097a10 con 0x7fc468133870 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc468089d20 con 0x7fc46808c4e0 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46ff3e700 10 monclient(hunting): authenticate will time out at 834.087906s 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.157+0000 7fc46dcda700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809c2a0 0x7fc46813fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809c2a0 0x7fc46813fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48472/0 (socket says 172.21.15.142:48472) 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 1 -- 172.21.15.142:0/3323525683 learned_addr learned my addr 172.21.15.142:0/3323525683 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 10 monclient(hunting): get_auth_request con 0x7fc46809c2a0 auth_method 0 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46dcda700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 0x7fc468087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 1 -- 172.21.15.142:0/3039560152 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400b850 con 0x7f579809abd0 2024-09-18T10:09:05.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: dump: 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 5 monclient: authenticate success, global_id 4254 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 1 -- 172.21.15.142:0/3039560152 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579400b9b0 con 0x7f579809abd0 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 1 -- 172.21.15.142:0/3039560152 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400bd50 con 0x7f579809abd0 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f578ffff700 10 monclient: dump: 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.268 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): get_auth_request con 0x7fc46808c4e0 auth_method 0 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc468133870 0x7fc468140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): get_auth_request con 0x7fc468133870 auth_method 0 2024-09-18T10:09:05.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 4 set_mon_vals no callback set 2024-09-18T10:09:05.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46d4d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f76f5ffb700 1 -- 172.21.15.142:0/4255059063 <== osd.5 v2:172.21.15.150:6810/21120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f770008c270 con 0x7f770008a0a0 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc468133870 msgr2=0x7fc468140220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc468133870 0x7fc468140220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809c2a0 msgr2=0x7fc46813fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc46809c2a0 0x7fc46813fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc464009420 con 0x7fc46808c4e0 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f220193f700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7f21e003b940 0x7f21e003dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f62937fe700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6280035b10 0x7f6280037fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6288006fd0 tx=0x7f6288006db0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: shutdown 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f579809abd0 msgr2=0x7f579812d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f579809abd0 0x7f579812d0a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f57940098e0 tx=0x7f57940091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- 172.21.15.142:0/3039560152 shutdown_connections 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3039560152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f579809abd0 0x7f579812d0a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- 172.21.15.142:0/3039560152 >> 172.21.15.142:0/3039560152 conn(0x7f5798071820 msgr2=0x7f57980a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- 172.21.15.142:0/3039560152 shutdown_connections 2024-09-18T10:09:05.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- 172.21.15.142:0/3039560152 wait complete. 2024-09-18T10:09:05.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== osd.1 v2:172.21.15.142:6800/20974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f21e003e440 con 0x7f21e003b940 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: monmap: 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.276 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 Processor -- start 2024-09-18T10:09:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- start start 2024-09-18T10:09:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: init 2024-09-18T10:09:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f6298820700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f628003ba10 0x7f628003de40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 0x7f579806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-a con 0x7f5798097b70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981265b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-c con 0x7f5798097fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 msgr2=0x7fc468087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7fc46e4db700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 0x7fc468087fe0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc460007ba0 tx=0x7fc460009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579809abd0 0x7f5798126af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): picked mon.noname-b con 0x7f579809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5798128b00 con 0x7f5798097b70 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579809d340 con 0x7f579809abd0 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579812a990 con 0x7f5798097fb0 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579f2bc700 10 monclient(hunting): authenticate will time out at 834.090879s 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 0x7f579806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 0x7f579806ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48488/0 (socket says 172.21.15.142:48488) 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 -- 172.21.15.142:0/3006010046 learned_addr learned my addr 172.21.15.142:0/3006010046 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): get_auth_request con 0x7f5798097b70 auth_method 0 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579809abd0 0x7f5798126af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): get_auth_request con 0x7f579809abd0 auth_method 0 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d859700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 msgr2=0x7f57981265b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5798097fb0 0x7f57981265b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579809abd0 msgr2=0x7f5798126af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f579809abd0 0x7f5798126af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f579d058700 1 -- 172.21.15.142:0/3006010046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790009a20 con 0x7f5798097b70 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.161+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== osd.0 v2:172.21.15.142:6808/20975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f628003e510 con 0x7f628003ba10 2024-09-18T10:09:05.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000dab0 con 0x7fc46808c4e0 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: dump: 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.284 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.285 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46000f040 con 0x7fc46808c4e0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 5 monclient: authenticate success, global_id 4253 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 10 monclient: _renew_subs 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46808bb60 con 0x7fc46808c4e0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 10 monclient: _renew_subs 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc468089800 con 0x7fc46808c4e0 2024-09-18T10:09:05.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46000de80 con 0x7fc46808c4e0 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45ffff700 4 set_mon_vals no callback set 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: dump: 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.287 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc460003020 con 0x7fc46808c4e0 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc448000f20 con 0x7fc46808c4e0 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc44c035b10 0x7fc44c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f770008a0a0 msgr2=0x7f770008a470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f770008a0a0 0x7f770008a470 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc460047ec0 con 0x7fc46808c4e0 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7fc44c03b990 0x7fc44c03ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc44c03e490 con 0x7fc44c03b990 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc46001e400 con 0x7fc46808c4e0 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc45dffb700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46d4d9700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7fc44c03b990 0x7fc44c03ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46dcda700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc44c035b10 0x7fc44c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46d4d9700 10 monclient: get_auth_request con 0x7fc44c03b990 auth_method 0 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7fc46dcda700 10 monclient: get_auth_request con 0x7fc44c035b10 auth_method 0 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579d058700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 msgr2=0x7f579806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579d058700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 0x7f579806ede0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57940093f0 tx=0x7f5794016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794011ca0 con 0x7f5798097b70 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: dump: 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5794020070 con 0x7f5798097b70 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 5 monclient: authenticate success, global_id 4260 2024-09-18T10:09:05.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 10 monclient: _renew_subs 2024-09-18T10:09:05.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5794009420 con 0x7f5798097b70 2024-09-18T10:09:05.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76e4035ac0 msgr2=0x7f76e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76e4035ac0 0x7f76e4037f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0093f0 tx=0x7f76fc016000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 10 monclient: shutdown 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 msgr2=0x7f7700139e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 --2- 172.21.15.142:0/4255059063 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7700098f00 0x7f7700139e50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76f800cc10 tx=0x7f76f80053c0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706085700 1 -- 172.21.15.142:0/4255059063 reap_dead start 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 shutdown_connections 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 >> 172.21.15.142:0/4255059063 conn(0x7f7700070ef0 msgr2=0x7f7700095860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 shutdown_connections 2024-09-18T10:09:05.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f7706886700 1 -- 172.21.15.142:0/4255059063 wait complete. 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578f7fe700 4 set_mon_vals no callback set 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 10 monclient: _renew_subs 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5798128180 con 0x7f5798097b70 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579401b6a0 con 0x7f5798097b70 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 10 monclient: dump: 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5794022030 con 0x7f5798097b70 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f577c035b10 0x7f577c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.165+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f579404b920 con 0x7f5798097b70 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579c857700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f577c035b10 0x7f577c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579c857700 10 monclient: get_auth_request con 0x7f577c035b10 auth_method 0 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7f5798096260 0x7f5798127030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 --> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f579809d0c0 con 0x7f5798096260 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579d859700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7f5798096260 0x7f5798127030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7fc46dcda700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc44c035b10 0x7fc44c037fc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc4640093f0 tx=0x7fc464016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579d859700 10 monclient: get_auth_request con 0x7f5798096260 auth_method 0 2024-09-18T10:09:05.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7fc46d4d9700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7fc44c03b990 0x7fc44c03ddc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.169+0000 7f579c857700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f577c035b10 0x7f577c037fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5788005030 tx=0x7f5788007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.173+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== osd.2 v2:172.21.15.142:6816/20976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc44c03e490 con 0x7fc44c03b990 2024-09-18T10:09:05.299 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:05.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T10:09:05.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.173+0000 7f579d859700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7f5798096260 0x7f5798127030 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.173+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== osd.4 v2:172.21.15.150:6818/21118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f579809d0c0 con 0x7f5798096260 2024-09-18T10:09:05.301 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:05.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T10:09:05.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.181+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 --> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f62940696a0 con 0x7f628003ba10 2024-09-18T10:09:05.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.181+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 --> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21dc002bd0 con 0x7f21e003b940 2024-09-18T10:09:05.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f627bfff700 1 -- 172.21.15.142:0/4113037351 <== osd.0 v2:172.21.15.142:6808/20975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f62940696a0 con 0x7f628003ba10 2024-09-18T10:09:05.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f21f17fa700 1 -- 172.21.15.142:0/1472944555 <== osd.1 v2:172.21.15.142:6800/20974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f21dc002bd0 con 0x7f21e003b940 2024-09-18T10:09:05.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f628003ba10 msgr2=0x7f628003de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f628003ba10 0x7f628003de40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7f21e003b940 msgr2=0x7f21e003dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7f21e003b940 0x7f21e003dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6280035b10 msgr2=0x7f6280037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6280035b10 0x7f6280037fc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6288006fd0 tx=0x7f6288006db0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 10 monclient: shutdown 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 msgr2=0x7f6294141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 --2- 172.21.15.142:0/4113037351 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f629406f230 0x7f6294141940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f628c00bfd0 tx=0x7f628c018000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f6298820700 1 -- 172.21.15.142:0/4113037351 reap_dead start 2024-09-18T10:09:05.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21e0035ac0 msgr2=0x7f21e0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21e0035ac0 0x7f21e0037f70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f21fc06d200 tx=0x7f21f8017000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 10 monclient: shutdown 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 msgr2=0x7f21fc145880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 --2- 172.21.15.142:0/1472944555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21fc09bed0 0x7f21fc145880 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00c7d0 tx=0x7f21ec008040 comp rx=0 tx=0).stop 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 shutdown_connections 2024-09-18T10:09:05.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 >> 172.21.15.142:0/4113037351 conn(0x7f6294071820 msgr2=0x7f629409f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f220193f700 1 -- 172.21.15.142:0/1472944555 reap_dead start 2024-09-18T10:09:05.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 shutdown_connections 2024-09-18T10:09:05.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 shutdown_connections 2024-09-18T10:09:05.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 >> 172.21.15.142:0/1472944555 conn(0x7f21fc071820 msgr2=0x7f21fc09fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f629a283700 1 -- 172.21.15.142:0/4113037351 wait complete. 2024-09-18T10:09:05.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 shutdown_connections 2024-09-18T10:09:05.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.185+0000 7f22033a2700 1 -- 172.21.15.142:0/1472944555 wait complete. 2024-09-18T10:09:05.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 --> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc448002bd0 con 0x7fc44c03b990 2024-09-18T10:09:05.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc45dffb700 1 -- 172.21.15.142:0/3323525683 <== osd.2 v2:172.21.15.142:6816/20976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc448002bd0 con 0x7fc44c03b990 2024-09-18T10:09:05.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7fc44c03b990 msgr2=0x7fc44c03ddc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7fc44c03b990 0x7fc44c03ddc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc44c035b10 msgr2=0x7fc44c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc44c035b10 0x7fc44c037fc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc4640093f0 tx=0x7fc464016000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 10 monclient: shutdown 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 msgr2=0x7fc468087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 --2- 172.21.15.142:0/3323525683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc46808c4e0 0x7fc468087fe0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc460007ba0 tx=0x7fc460009250 comp rx=0 tx=0).stop 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46e4db700 1 -- 172.21.15.142:0/3323525683 reap_dead start 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 shutdown_connections 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.189+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 >> 172.21.15.142:0/3323525683 conn(0x7fc468071820 msgr2=0x7fc4680966b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.193+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 shutdown_connections 2024-09-18T10:09:05.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.193+0000 7fc46ff3e700 1 -- 172.21.15.142:0/3323525683 wait complete. 2024-09-18T10:09:05.310 INFO:teuthology.orchestra.run.smithi142.stdout:51539607556 2024-09-18T10:09:05.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T10:09:05.311 INFO:teuthology.orchestra.run.smithi142.stdout:47244640260 2024-09-18T10:09:05.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T10:09:05.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.197+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 --> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57981452e0 con 0x7f5798096260 2024-09-18T10:09:05.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f578d7fa700 1 -- 172.21.15.142:0/3006010046 <== osd.4 v2:172.21.15.150:6818/21118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f57981452e0 con 0x7f5798096260 2024-09-18T10:09:05.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7f5798096260 msgr2=0x7f5798127030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7f5798096260 0x7f5798127030 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f577c035b10 msgr2=0x7f577c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f577c035b10 0x7f577c037fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5788005030 tx=0x7f5788007bf0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 10 monclient: shutdown 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 msgr2=0x7f579806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 --2- 172.21.15.142:0/3006010046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5798097b70 0x7f579806ede0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f57940093f0 tx=0x7f5794016000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579d859700 1 -- 172.21.15.142:0/3006010046 reap_dead start 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 shutdown_connections 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 >> 172.21.15.142:0/3006010046 conn(0x7f5798071820 msgr2=0x7f5798095b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 shutdown_connections 2024-09-18T10:09:05.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.201+0000 7f579f2bc700 1 -- 172.21.15.142:0/3006010046 wait complete. 2024-09-18T10:09:05.315 INFO:teuthology.orchestra.run.smithi142.stdout:51539607556 2024-09-18T10:09:05.315 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T10:09:05.316 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:05.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T10:09:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: monmap: 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.461998+0000 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.461998+0000 2024-09-18T10:09:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.464 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 1 Processor -- start 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 1 -- start start 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: init 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3590095d20 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.457+0000 7f35953eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f359009d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-b con 0x7f3590096790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3590134510 0x7f35901368f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-c con 0x7f3590134510 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900976b0 con 0x7f3590095d20 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3590096bd0 con 0x7f3590134510 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900960f0 con 0x7f3590096790 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient(hunting): authenticate will time out at 834.388080s 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f359009d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3590095d20 0x7f359009cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46494/0 (socket says 172.21.15.142:46494) 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358ffff700 1 -- 172.21.15.142:0/3364359145 learned_addr learned my addr 172.21.15.142:0/3364359145 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): get_auth_request con 0x7f3590096790 auth_method 0 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 1 --2- 172.21.15.142:0/3364359145 >> v2:172.21.15.150:3300/0 conn(0x7f3590134510 0x7f35901368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): get_auth_request con 0x7f3590134510 auth_method 0 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 -- 172.21.15.142:0/3364359145 >> v2:172.21.15.150:3300/0 conn(0x7f3590134510 msgr2=0x7f35901368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 --2- 172.21.15.142:0/3364359145 >> v2:172.21.15.150:3300/0 conn(0x7f3590134510 0x7f35901368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 -- 172.21.15.142:0/3364359145 >> v1:172.21.15.142:6789/0 conn(0x7f3590095d20 legacy=0x7f359009cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 -- 172.21.15.142:0/3364359145 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900970f0 con 0x7f3590096790 2024-09-18T10:09:05.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 -- 172.21.15.142:0/3364359145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 msgr2=0x7f359009d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358f7fe700 1 --2- 172.21.15.142:0/3364359145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f359009d490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f35840098e0 tx=0x7f35840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a20ea1f855380b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 1 -- 172.21.15.142:0/3364359145 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358400b850 con 0x7f3590096790 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 10 monclient: dump: 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.474 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 5 monclient: authenticate success, global_id 4256 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 1 -- 172.21.15.142:0/3364359145 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358400b9b0 con 0x7f3590096790 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f358effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.461+0000 7f35953eb700 4 set_mon_vals no callback set 2024-09-18T10:09:05.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: shutdown 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- 172.21.15.142:0/3364359145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 msgr2=0x7f359009d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- 172.21.15.142:0/3364359145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f359009d490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f35840098e0 tx=0x7f35840091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- 172.21.15.142:0/3364359145 shutdown_connections 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- 172.21.15.142:0/3364359145 >> v2:172.21.15.150:3300/0 conn(0x7f3590134510 0x7f35901368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- 172.21.15.142:0/3364359145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f359009d490 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- 172.21.15.142:0/3364359145 >> 172.21.15.142:0/3364359145 conn(0x7f35900a2390 msgr2=0x7f35900a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- 172.21.15.142:0/3364359145 shutdown_connections 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- 172.21.15.142:0/3364359145 wait complete. 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: monmap: 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 Processor -- start 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- start start 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: init 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 0x7f3590141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-b con 0x7f3590095d20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f35901337c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-c con 0x7f3590096790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 0x7f359006bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): picked mon.noname-a con 0x7f3590134510 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359006db00 con 0x7f3590134510 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359006d7e0 con 0x7f3590095d20 2024-09-18T10:09:05.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3590127570 con 0x7f3590096790 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f35953eb700 10 monclient(hunting): authenticate will time out at 834.395356s 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 0x7f359006bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 0x7f359006bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48502/0 (socket says 172.21.15.142:48502) 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 1 -- 172.21.15.142:0/3635021816 learned_addr learned my addr 172.21.15.142:0/3635021816 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): get_auth_request con 0x7f3590134510 auth_method 0 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 0x7f3590141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): get_auth_request con 0x7f3590095d20 auth_method 0 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f3594bea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358f7fe700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f35901337c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358f7fe700 10 monclient(hunting): get_auth_request con 0x7f3590096790 auth_method 0 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f358f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 -- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 msgr2=0x7f35901337c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f35901337c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 -- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 msgr2=0x7f359006bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 0x7f359006bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 -- 172.21.15.142:0/3635021816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584009420 con 0x7f3590095d20 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f1b4df45700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f1b4df45700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.465+0000 7f1b4df45700 10 monclient: monmap: 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.471527+0000 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.471527+0000 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.482 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 -- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 msgr2=0x7f3590141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f358ffff700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 0x7f3590141600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f358000c530 tx=0x7f3580007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580005d40 con 0x7f3590095d20 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 10 monclient: dump: 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.484 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f35953eb700 5 monclient: authenticate success, global_id 4252 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f35953eb700 10 monclient: _renew_subs 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35800070d0 con 0x7f3590095d20 2024-09-18T10:09:05.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 Processor -- start 2024-09-18T10:09:05.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f35953eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 -- start start 2024-09-18T10:09:05.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient: init 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-b con 0x7f1b480a1bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1b4809b150 0x7f1b4809b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-a con 0x7f1b4809b150 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 0x7f1b4809c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-c con 0x7f1b4809bed0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b48098c80 con 0x7f1b4809b150 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b4806fbc0 con 0x7f1b4809bed0 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b480a1fc0 con 0x7f1b480a1bf0 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f1b4df45700 10 monclient(hunting): authenticate will time out at 834.399865s 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.469+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359006e230 con 0x7f3590095d20 2024-09-18T10:09:05.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1b4809b150 0x7f1b4809b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f358e7fc700 4 set_mon_vals no callback set 2024-09-18T10:09:05.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51228/0 (socket says 172.21.15.142:51228) 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 1 -- 172.21.15.142:0/3153899808 learned_addr learned my addr 172.21.15.142:0/3153899808 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): get_auth_request con 0x7f1b480a1bf0 auth_method 0 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): get_auth_request con 0x7f1b4809b150 auth_method 0 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580007490 con 0x7f3590095d20 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 10 monclient: dump: 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 10 monclient: _renew_subs 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359006f240 con 0x7f3590095d20 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 -- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 msgr2=0x7f1b48131f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 -- 172.21.15.142:0/3153899808 >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 msgr2=0x7f1b4809c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4d744700 1 --2- 172.21.15.142:0/3153899808 >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 0x7f1b4809c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 --2- 172.21.15.142:0/3153899808 >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 0x7f1b4809c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 -- 172.21.15.142:0/3153899808 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4809d900 con 0x7f1b4809b150 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4cf43700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:05.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4d744700 10 monclient: get_auth_request con 0x7f1b4809bed0 auth_method 0 2024-09-18T10:09:05.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4d744700 1 --2- 172.21.15.142:0/3153899808 >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 0x7f1b4809c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:05.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f358001b3b0 con 0x7f3590095d20 2024-09-18T10:09:05.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f3580047730 con 0x7f3590095d20 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f358f7fe700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f358f7fe700 10 monclient: get_auth_request con 0x7f357c035af0 auth_method 0 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 -- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b4809b150 msgr2=0x7f1b4809b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b47fff700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b4809b150 0x7f1b4809b560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b48098aa0 tx=0x7f1b3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6061f5befd132fac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359006f560 con 0x7f3590095d20 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 1 -- 172.21.15.142:0/3153899808 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011840 con 0x7f1b4809b150 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: dump: 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 1 -- 172.21.15.142:0/3153899808 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c0119a0 con 0x7f1b4809b150 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 1 -- 172.21.15.142:0/3153899808 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c011d40 con 0x7f1b4809b150 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b477fe700 10 monclient: dump: 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4df45700 5 monclient: authenticate success, global_id 4266 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4df45700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.473+0000 7f1b4df45700 4 set_mon_vals no callback set 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: shutdown 2024-09-18T10:09:05.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b4809b150 msgr2=0x7f1b4809b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b4809b150 0x7f1b4809b560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b48098aa0 tx=0x7f1b3c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- 172.21.15.142:0/3153899808 shutdown_connections 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- 172.21.15.142:0/3153899808 >> v2:172.21.15.150:3300/0 conn(0x7f1b4809bed0 0x7f1b4809c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b4809b150 0x7f1b4809b560 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- 172.21.15.142:0/3153899808 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b480a1bf0 0x7f1b48131f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- 172.21.15.142:0/3153899808 >> 172.21.15.142:0/3153899808 conn(0x7f1b48071680 msgr2=0x7f1b480a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- 172.21.15.142:0/3153899808 shutdown_connections 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- 172.21.15.142:0/3153899808 wait complete. 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: monmap: 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f358f7fe700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f35840093f0 tx=0x7f3584016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 Processor -- start 2024-09-18T10:09:05.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- start start 2024-09-18T10:09:05.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: init 2024-09-18T10:09:05.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4809b150 0x7f1b481414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-b con 0x7f1b4809b150 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 0x7f1b48130f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-c con 0x7f1b4809bed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 0x7f1b480968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): picked mon.noname-a con 0x7f1b480a1bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b480a0670 con 0x7f1b480a1bf0 2024-09-18T10:09:05.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48142310 con 0x7f1b4809b150 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4812a210 con 0x7f1b4809bed0 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4df45700 10 monclient(hunting): authenticate will time out at 834.406665s 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 0x7f1b480968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 0x7f1b480968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48528/0 (socket says 172.21.15.142:48528) 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 1 -- 172.21.15.142:0/559663456 learned_addr learned my addr 172.21.15.142:0/559663456 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): get_auth_request con 0x7f1b480a1bf0 auth_method 0 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 0x7f1b48130f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4809b150 0x7f1b481414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): get_auth_request con 0x7f1b4809bed0 auth_method 0 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4d744700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): get_auth_request con 0x7f1b4809b150 auth_method 0 2024-09-18T10:09:05.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b4cf43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 -- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4809b150 msgr2=0x7f1b481414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4809b150 0x7f1b481414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 -- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 msgr2=0x7f1b480968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 0x7f1b480968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7f1b47fff700 1 -- 172.21.15.142:0/559663456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b380094f0 con 0x7f1b4809bed0 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7fe23bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7fe23bfff700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.477+0000 7fe23bfff700 10 monclient: monmap: 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.483623+0000 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.483623+0000 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.511 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b47fff700 1 -- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 msgr2=0x7f1b48130f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b47fff700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 0x7f1b48130f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c013000 tx=0x7f1b3c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 Processor -- start 2024-09-18T10:09:05.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 -- start start 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00bce0 con 0x7f1b4809bed0 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: dump: 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.514 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 5 monclient: authenticate success, global_id 4265 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _renew_subs 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c009420 con 0x7f1b4809bed0 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _renew_subs 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4812b220 con 0x7f1b4809bed0 2024-09-18T10:09:05.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient: init 2024-09-18T10:09:05.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T10:09:05.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe23c06f3b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe23c0990d0 0x7fe23c134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fe23c0990d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe23c0994f0 0x7fe23c136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fe23c0994f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe23c0994f0 0x7fe23c136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe23c0994f0 0x7fe23c136bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48534/0 (socket says 172.21.15.142:48534) 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23affd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe23c06f3b0 0x7fe23c0989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46504/0 (socket says 172.21.15.142:46504) 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 -- 172.21.15.142:0/876066723 learned_addr learned my addr 172.21.15.142:0/876066723 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c09c330 con 0x7fe23c06f3b0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c09b380 con 0x7fe23c0994f0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c09b030 con 0x7fe23c0990d0 2024-09-18T10:09:05.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23bfff700 10 monclient(hunting): authenticate will time out at 834.410862s 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23a7fc700 1 --2- 172.21.15.142:0/876066723 >> v2:172.21.15.150:3300/0 conn(0x7fe23c0990d0 0x7fe23c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23a7fc700 10 monclient(hunting): get_auth_request con 0x7fe23c0990d0 auth_method 0 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): get_auth_request con 0x7fe23c0994f0 auth_method 0 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3c00be40 con 0x7f1b4809bed0 2024-09-18T10:09:05.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b467fc700 4 set_mon_vals no callback set 2024-09-18T10:09:05.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-18T10:09:05.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 -- 172.21.15.142:0/876066723 >> v2:172.21.15.150:3300/0 conn(0x7fe23c0990d0 msgr2=0x7fe23c134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c01a660 con 0x7f1b4809bed0 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b44ff9700 10 monclient: dump: 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.521 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 --2- 172.21.15.142:0/876066723 >> v2:172.21.15.150:3300/0 conn(0x7fe23c0990d0 0x7fe23c134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 -- 172.21.15.142:0/876066723 >> v1:172.21.15.142:6789/0 conn(0x7fe23c06f3b0 legacy=0x7fe23c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7fe23b7fe700 1 -- 172.21.15.142:0/876066723 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c09c530 con 0x7fe23c0994f0 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 10 monclient: monmap: 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.486713+0000 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.486713+0000 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.523 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b280050b0 con 0x7f1b4809bed0 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 1 Processor -- start 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 1 -- start start 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.481+0000 7f4b27031700 10 monclient: init 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b2009bed0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b2006f230 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c9b0 0x7f4b2013a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-c con 0x7f4b2009c9b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b2006f770 con 0x7f4b2009bed0 2024-09-18T10:09:05.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b20098ad0 con 0x7f4b2006f230 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b20098eb0 con 0x7f4b2009c9b0 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b27031700 10 monclient(hunting): authenticate will time out at 834.412511s 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1b3c01a9d0 con 0x7f1b4809bed0 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b1ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4b2006f230 0x7f4b2009c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:44528/0 (socket says 172.21.15.142:44528) 2024-09-18T10:09:05.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b1ffff700 1 -- 172.21.15.142:0/3572691456 learned_addr learned my addr 172.21.15.142:0/3572691456 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:05.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b44ff9700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 0x7f1b34037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f1b3c049950 con 0x7f1b4809bed0 2024-09-18T10:09:05.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b4cf43700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 0x7f1b34037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b4cf43700 10 monclient: get_auth_request con 0x7f1b34035ac0 auth_method 0 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: monmap: 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.489191+0000 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.489191+0000 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.529 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.530 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.530 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.530 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.530 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23b7fe700 1 -- 172.21.15.142:0/876066723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c0994f0 msgr2=0x7fe23c136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23b7fe700 1 --2- 172.21.15.142:0/876066723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c0994f0 0x7fe23c136bc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe23c098600 tx=0x7fe234009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57019794f7350ec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 1 -- 172.21.15.142:0/876066723 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23400e070 con 0x7fe23c0994f0 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: dump: 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.531 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 5 monclient: authenticate success, global_id 4272 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 1 -- 172.21.15.142:0/876066723 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23400bc50 con 0x7fe23c0994f0 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 4 set_mon_vals no callback set 2024-09-18T10:09:05.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 1 -- 172.21.15.142:0/876066723 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340092e0 con 0x7fe23c0994f0 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe239ffb700 10 monclient: dump: 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fbd73fff700 10 monclient: get_monmap_and_config 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fbd73fff700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fbd73fff700 10 monclient: monmap: 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:05.489599+0000 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:05.489599+0000 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 Processor -- start 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 -- start start 2024-09-18T10:09:05.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: init 2024-09-18T10:09:05.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 10 monclient: shutdown 2024-09-18T10:09:05.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c0994f0 msgr2=0x7fe23c136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 --2- 172.21.15.142:0/876066723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c0994f0 0x7fe23c136bc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe23c098600 tx=0x7fe234009840 comp rx=0 tx=0).stop 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 shutdown_connections 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 --2- 172.21.15.142:0/876066723 >> v2:172.21.15.150:3300/0 conn(0x7fe23c0990d0 0x7fe23c134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 --2- 172.21.15.142:0/876066723 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c0994f0 0x7fe23c136bc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 >> 172.21.15.142:0/876066723 conn(0x7fe23c070ef0 msgr2=0x7fe23c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 shutdown_connections 2024-09-18T10:09:05.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- 172.21.15.142:0/876066723 wait complete. 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a3409abd0 0x7f9a3409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-c con 0x7f9a3409abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a3409db80 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a34097880 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a340958c0 con 0x7f9a3409db80 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a34095ef0 con 0x7f9a34097880 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3406fa60 con 0x7f9a3409abd0 2024-09-18T10:09:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a3a225700 10 monclient(hunting): authenticate will time out at 834.415013s 2024-09-18T10:09:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b1ffff700 1 -- 172.21.15.142:0/3572691456 >> v1:172.21.15.150:6789/0 conn(0x7f4b2006f230 legacy=0x7f4b2009c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a32ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9a3409db80 0x7f9a3409c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46524/0 (socket says 172.21.15.142:46524) 2024-09-18T10:09:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2341574352 learned_addr learned my addr 172.21.15.142:0/2341574352 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 1 --2- 172.21.15.142:0/3572691456 >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c9b0 0x7f4b2013a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 1 --2- 172.21.15.142:0/2341574352 >> v2:172.21.15.150:3300/0 conn(0x7f9a3409abd0 0x7f9a3409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): get_auth_request con 0x7f4b2009c9b0 auth_method 0 2024-09-18T10:09:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): get_auth_request con 0x7f9a3409abd0 auth_method 0 2024-09-18T10:09:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 10 monclient: monmap: 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.542 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f9a337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 1 -- 172.21.15.142:0/3572691456 >> v1:172.21.15.150:6789/0 conn(0x7f4b2006f230 legacy=0x7f4b2009c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 1 -- 172.21.15.142:0/3572691456 >> v1:172.21.15.142:6789/0 conn(0x7f4b2009bed0 legacy=0x7f4b200991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f4b255ce700 1 -- 172.21.15.142:0/3572691456 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2009b940 con 0x7f4b2009c9b0 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 Processor -- start 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 1 -- start start 2024-09-18T10:09:05.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7fe23bfff700 10 monclient: init 2024-09-18T10:09:05.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.485+0000 7f1b4cf43700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 0x7f1b34037f70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b38006d20 tx=0x7f1b38009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 10 monclient(hunting): handle_auth_done global_id 4264 payload 274 2024-09-18T10:09:05.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 -- 172.21.15.142:0/2341574352 >> v1:172.21.15.150:6789/0 conn(0x7f9a34097880 legacy=0x7f9a34137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 -- 172.21.15.142:0/2341574352 >> v1:172.21.15.142:6789/0 conn(0x7f9a3409db80 legacy=0x7f9a3409c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 -- 172.21.15.142:0/2341574352 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3409afa0 con 0x7f9a3409abd0 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 Processor -- start 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 -- start start 2024-09-18T10:09:05.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient: init 2024-09-18T10:09:05.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c06f3b0 0x7fe23c1396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fe23c06f3b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe23c0990d0 0x7fe23c139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fe23c0990d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fe23c0994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c08e170 con 0x7fe23c06f3b0 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c06fa20 con 0x7fe23c0990d0 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c096f50 con 0x7fe23c0994f0 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient(hunting): authenticate will time out at 834.416516s 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51234/0 (socket says 172.21.15.142:51234) 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 -- 172.21.15.142:0/397506489 learned_addr learned my addr 172.21.15.142:0/397506489 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): get_auth_request con 0x7fe23c0994f0 auth_method 0 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd7409c760 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd7409cb30 0x7fbd7409cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbd7409cb30 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:05.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbd74133870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd74098b00 con 0x7fbd7409c760 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd74098d20 con 0x7fbd7409cb30 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7409a100 con 0x7fbd74133870 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd73fff700 10 monclient(hunting): authenticate will time out at 834.416988s 2024-09-18T10:09:05.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51244/0 (socket says 172.21.15.142:51244) 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 -- 172.21.15.142:0/1407411580 learned_addr learned my addr 172.21.15.142:0/1407411580 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): get_auth_request con 0x7fbd74133870 auth_method 0 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 -- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe23c0990d0 msgr2=0x7fe23c139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe23c0990d0 0x7fe23c139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 -- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c06f3b0 msgr2=0x7fe23c1396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c06f3b0 0x7fe23c1396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 -- 172.21.15.142:0/397506489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe234009ae0 con 0x7fe23c0994f0 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 -- 172.21.15.142:0/1407411580 >> v2:172.21.15.150:3300/0 conn(0x7fbd7409cb30 msgr2=0x7fbd7409cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 --2- 172.21.15.142:0/1407411580 >> v2:172.21.15.150:3300/0 conn(0x7fbd7409cb30 0x7fbd7409cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 -- 172.21.15.142:0/1407411580 >> v1:172.21.15.150:6789/0 conn(0x7fbd7409c760 legacy=0x7fbd7409b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 monclient: found mon.noname-b 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 -- 172.21.15.142:0/1407411580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7409a3e0 con 0x7fbd74133870 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b255ce700 1 -- 172.21.15.142:0/3572691456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009c9b0 msgr2=0x7f4b2013a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b255ce700 1 --2- 172.21.15.142:0/3572691456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009c9b0 0x7f4b2013a6c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009960 tx=0x7f4b100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18ce81c55819eeb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3580017080 con 0x7f3590095d20 2024-09-18T10:09:05.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f3577fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f3577fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 -- 172.21.15.142:0/2341574352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a3409abd0 msgr2=0x7f9a3409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a337fe700 1 --2- 172.21.15.142:0/2341574352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a3409abd0 0x7f9a3409bed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a34095140 tx=0x7f9a280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2f7fc7ea1e67cd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/3572691456 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000ba70 con 0x7f4b2009c9b0 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: dump: 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.559 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/3572691456 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1000bbf0 con 0x7f4b2009c9b0 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 5 monclient: authenticate success, global_id 4261 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/3572691456 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10011420 con 0x7f4b2009c9b0 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 4 set_mon_vals no callback set 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b1f7fe700 10 monclient: dump: 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.561 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 1 -- 172.21.15.142:0/2341574352 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2800b9a0 con 0x7f9a3409abd0 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: dump: 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 5 monclient: authenticate success, global_id 4264 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 1 -- 172.21.15.142:0/2341574352 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2800bb00 con 0x7f9a3409abd0 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 4 set_mon_vals no callback set 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 1 -- 172.21.15.142:0/2341574352 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2800bea0 con 0x7f9a3409abd0 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a327fc700 10 monclient: dump: 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 -- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 msgr2=0x7fe23c0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23b7fe700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe234009ab0 tx=0x7fe234017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 10 monclient: shutdown 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 1 -- 172.21.15.142:0/2341574352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a3409abd0 msgr2=0x7f9a3409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2341574352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a3409abd0 0x7f9a3409bed0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9a34095140 tx=0x7f9a280091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 -- 172.21.15.142:0/1407411580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 msgr2=0x7fbd74135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fbd737fe700 1 --2- 172.21.15.142:0/1407411580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009f20 tx=0x7fbd6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0aa341209dbfbca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 10 monclient: shutdown 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 1 -- 172.21.15.142:0/3572691456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009c9b0 msgr2=0x7f4b2013a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7f4b27031700 1 --2- 172.21.15.142:0/3572691456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009c9b0 0x7f4b2013a6c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009960 tx=0x7f4b100091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234012c90 con 0x7fe23c0994f0 2024-09-18T10:09:05.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: dump: 2024-09-18T10:09:05.568 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234012df0 con 0x7fe23c0994f0 2024-09-18T10:09:05.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 5 monclient: authenticate success, global_id 4268 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _renew_subs 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c13a160 con 0x7fe23c0994f0 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23401d670 con 0x7fe23c0994f0 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe222ffd700 10 monclient: dump: 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.570 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _renew_subs 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23c08cf20 con 0x7fe23c0994f0 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.489+0000 7fe2397fa700 4 set_mon_vals no callback set 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1b3c02c350 con 0x7f1b4809bed0 2024-09-18T10:09:05.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f1b44ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f1b44ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- 172.21.15.142:0/2341574352 shutdown_connections 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2341574352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a3409abd0 0x7f9a3409bed0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- 172.21.15.142:0/2341574352 >> 172.21.15.142:0/2341574352 conn(0x7f9a340a14e0 msgr2=0x7f9a340a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- 172.21.15.142:0/2341574352 shutdown_connections 2024-09-18T10:09:05.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 1 -- 172.21.15.142:0/1407411580 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c00e070 con 0x7fbd74133870 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: dump: 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.573 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 1 -- 172.21.15.142:0/1407411580 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c00bc90 con 0x7fbd74133870 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 1 -- 172.21.15.142:0/1407411580 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c012660 con 0x7fbd74133870 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fbd71ffb700 10 monclient: dump: 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- 172.21.15.142:0/2341574352 wait complete. 2024-09-18T10:09:05.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f4b27031700 1 -- 172.21.15.142:0/3572691456 shutdown_connections 2024-09-18T10:09:05.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f4b27031700 1 --2- 172.21.15.142:0/3572691456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009c9b0 0x7f4b2013a6c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f4b27031700 1 -- 172.21.15.142:0/3572691456 >> 172.21.15.142:0/3572691456 conn(0x7f4b20071820 msgr2=0x7f4b200a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23c072a80 con 0x7fe23c0994f0 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe23401daa0 con 0x7fe23c0994f0 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient: monmap: 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe222ffd700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 0x7fe228037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fe234049a80 con 0x7fe23c0994f0 2024-09-18T10:09:05.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23affd700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 0x7fe228037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7fe23affd700 10 monclient: get_auth_request con 0x7fe228035b10 auth_method 0 2024-09-18T10:09:05.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 Processor -- start 2024-09-18T10:09:05.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- start start 2024-09-18T10:09:05.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient: init 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a34097880 0x7f9a34094470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-b con 0x7f9a34097880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-c con 0x7f9a3409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a3409db80 0x7f9a3413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): picked mon.noname-a con 0x7f9a3409db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3408f670 con 0x7f9a3409db80 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34097c90 con 0x7f9a34097880 2024-09-18T10:09:05.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3408e0c0 con 0x7f9a3409abd0 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a3a225700 10 monclient(hunting): authenticate will time out at 834.422841s 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51260/0 (socket says 172.21.15.142:51260) 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2952634553 learned_addr learned my addr 172.21.15.142:0/2952634553 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): get_auth_request con 0x7f9a3409abd0 auth_method 0 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a3409db80 0x7f9a3413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): get_auth_request con 0x7f9a3409db80 auth_method 0 2024-09-18T10:09:05.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a34097880 msgr2=0x7f9a34094470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a34097880 0x7f9a34094470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a3409db80 msgr2=0x7f9a3413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a3409db80 0x7f9a3413fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.493+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2952634553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a28009420 con 0x7f9a3409abd0 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 5 monclient: authenticate success, global_id 4271 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:05.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 4 set_mon_vals no callback set 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fe23affd700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 0x7fe228037fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe22c004fe0 tx=0x7fe22c007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: shutdown 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- 172.21.15.142:0/1407411580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 msgr2=0x7fbd74135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1407411580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009f20 tx=0x7fbd6c009840 comp rx=0 tx=0).stop 2024-09-18T10:09:05.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- 172.21.15.142:0/1407411580 shutdown_connections 2024-09-18T10:09:05.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1407411580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd74133870 0x7fbd74135c50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1407411580 >> v2:172.21.15.150:3300/0 conn(0x7fbd7409cb30 0x7fbd7409cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- 172.21.15.142:0/1407411580 >> 172.21.15.142:0/1407411580 conn(0x7fbd74070ef0 msgr2=0x7fbd740712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- 172.21.15.142:0/3572691456 shutdown_connections 2024-09-18T10:09:05.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- 172.21.15.142:0/3572691456 wait complete. 2024-09-18T10:09:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- 172.21.15.142:0/1407411580 shutdown_connections 2024-09-18T10:09:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- 172.21.15.142:0/1407411580 wait complete. 2024-09-18T10:09:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f9a32ffd700 1 -- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 msgr2=0x7f9a3413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f9a32ffd700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c5e0 tx=0x7f9a24004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient: monmap: 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: build_initial_monmap 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: monmap: 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:05.589 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 Processor -- start 2024-09-18T10:09:05.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- start start 2024-09-18T10:09:05.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient: init 2024-09-18T10:09:05.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 Processor -- start 2024-09-18T10:09:05.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- start start 2024-09-18T10:09:05.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: init 2024-09-18T10:09:05.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:05.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-c con 0x7f4b2006f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009bed0 0x7f4b20095880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-b con 0x7f4b2009bed0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b2009c9b0 0x7f4b2008f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): picked mon.noname-a con 0x7f4b2009c9b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2006f600 con 0x7f4b2009c9b0 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20091300 con 0x7f4b2009bed0 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20096a70 con 0x7f4b2006f230 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b27031700 10 monclient(hunting): authenticate will time out at 834.427040s 2024-09-18T10:09:05.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51266/0 (socket says 172.21.15.142:51266) 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 -- 172.21.15.142:0/3537031224 learned_addr learned my addr 172.21.15.142:0/3537031224 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): get_auth_request con 0x7f4b2006f230 auth_method 0 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009bed0 0x7f4b20095880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request con 0x7f4b2009bed0 auth_method 0 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:05.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 0x7fbd740884a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbd7409c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:05.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-09-18T10:09:05.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 0x7fbd7406e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbd7409cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbd74133870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd74088f20 con 0x7fbd74133870 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7408ebe0 con 0x7fbd7409cb30 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7408eec0 con 0x7fbd7409c760 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:05.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd73fff700 10 monclient(hunting): authenticate will time out at 834.427673s 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 -- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009bed0 msgr2=0x7f4b20095880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009bed0 0x7f4b20095880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 -- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b2009c9b0 msgr2=0x7f4b2008f200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b2009c9b0 0x7f4b2008f200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 monclient: found mon.noname-c 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7f4b24dcd700 1 -- 172.21.15.142:0/3537031224 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10009420 con 0x7f4b2006f230 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48566/0 (socket says 172.21.15.142:48566) 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 1 -- 172.21.15.142:0/1789474183 learned_addr learned my addr 172.21.15.142:0/1789474183 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): get_auth_request con 0x7fbd74133870 auth_method 0 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.497+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 0x7fbd7406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 10 monclient(hunting): get_auth_request con 0x7fbd7409cb30 auth_method 0 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:05.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-18T10:09:05.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe23404be60 con 0x7fe23c0994f0 2024-09-18T10:09:05.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fe222ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fe222ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd72ffd700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 0x7fbd740884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 -- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 msgr2=0x7fbd740884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 0x7fbd740884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 -- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 msgr2=0x7fbd7406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 0x7fbd7406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 monclient: found mon.noname-a 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 -- 172.21.15.142:0/1789474183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c009ae0 con 0x7fbd74133870 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd727fc700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 0x7fbd7406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd72ffd700 10 monclient: get_auth_request con 0x7fbd7409c760 auth_method 0 2024-09-18T10:09:05.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd72ffd700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 0x7fbd740884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24005d40 con 0x7f9a3409abd0 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: dump: 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.604 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a240070d0 con 0x7f9a3409abd0 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 5 monclient: authenticate success, global_id 4274 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 10 monclient: _renew_subs 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a340a0130 con 0x7f9a3409abd0 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24005ea0 con 0x7f9a3409abd0 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 10 monclient: dump: 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 10 monclient: _renew_subs 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a340949b0 con 0x7f9a3409abd0 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a31ffb700 4 set_mon_vals no callback set 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9a2401b3b0 con 0x7f9a3409abd0 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 0x7f9a20037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b24dcd700 1 -- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 msgr2=0x7f4b20096530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b24dcd700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c950 tx=0x7f4b18002760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f9a240477e0 con 0x7f9a3409abd0 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18012840 con 0x7f4b2006f230 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: dump: 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.608 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b180129a0 con 0x7f4b2006f230 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18012d60 con 0x7f4b2006f230 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b1cff9700 10 monclient: dump: 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.610 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a337fe700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 0x7f9a20037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f9a337fe700 10 monclient: get_auth_request con 0x7f9a20035b10 auth_method 0 2024-09-18T10:09:05.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 5 monclient: authenticate success, global_id 4277 2024-09-18T10:09:05.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 10 monclient: _renew_subs 2024-09-18T10:09:05.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2009d550 con 0x7f4b2006f230 2024-09-18T10:09:05.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 10 monclient: _renew_subs 2024-09-18T10:09:05.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b20097c00 con 0x7f4b2006f230 2024-09-18T10:09:05.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 -- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 msgr2=0x7fbd74094710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:05.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd737fe700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b6c0 tx=0x7fbd6c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c012e70 con 0x7fbd74133870 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: dump: 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: _finish_auth 0 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6c01e400 con 0x7fbd74133870 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 5 monclient: authenticate success, global_id 4278 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 10 monclient: _renew_subs 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7408f1a0 con 0x7fbd74133870 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 10 monclient: _renew_subs 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7406f050 con 0x7fbd74133870 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c01e780 con 0x7fbd74133870 2024-09-18T10:09:05.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd70ff9700 4 set_mon_vals no callback set 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.501+0000 7fbd5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd5affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f9a3a225700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f9a3a225700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f9a3a225700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a340696a0 con 0x7f9a3409abd0 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd5affd700 10 monclient: dump: 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:05.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbd6c019030 con 0x7fbd74133870 2024-09-18T10:09:05.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4b1800a630 con 0x7f4b2006f230 2024-09-18T10:09:05.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b1cff9700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 0x7f4b080380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f4b18048790 con 0x7f4b2006f230 2024-09-18T10:09:05.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd5affd700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 0x7fbd60037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:05.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd73fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f9a337fe700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 0x7f9a20037fc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a280093a0 tx=0x7f9a28016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd73fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd73fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd540050b0 con 0x7fbd74133870 2024-09-18T10:09:05.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fbd6c04a000 con 0x7fbd74133870 2024-09-18T10:09:05.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b27031700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b27031700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b27031700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.505+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b20092d50 con 0x7f4b2006f230 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.509+0000 7f4b1effd700 4 set_mon_vals no callback set 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.509+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbd6c026400 con 0x7fbd74133870 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.509+0000 7fbd5affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.509+0000 7fbd5affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.513+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9a24017080 con 0x7f9a3409abd0 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.513+0000 7f9a1b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.513+0000 7f9a1b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.521+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4b18016070 con 0x7f4b2006f230 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.521+0000 7f4b1cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.521+0000 7f4b1cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:05.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.529+0000 7fbd72ffd700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 0x7fbd60037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.529+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 0x7f4b080380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:05.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.529+0000 7f4b1ffff700 10 monclient: get_auth_request con 0x7f4b08035c20 auth_method 0 2024-09-18T10:09:05.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.529+0000 7fbd72ffd700 10 monclient: get_auth_request con 0x7fbd60035b10 auth_method 0 2024-09-18T10:09:05.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.533+0000 7fbd72ffd700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 0x7fbd60037fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd68006fd0 tx=0x7fbd68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.533+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 0x7f4b080380d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b140058d0 tx=0x7f4b14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:05.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1b28005630 con 0x7f1b4809bed0 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b44ff9700 1 -- 172.21.15.142:0/559663456 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b3c015030 con 0x7f1b4809bed0 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b44ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:05.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b44ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.662 INFO:teuthology.orchestra.run.smithi142.stdout:47244640258 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 msgr2=0x7f1b34037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 0x7f1b34037f70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b38006d20 tx=0x7f1b38009250 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 10 monclient: shutdown 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 msgr2=0x7f1b48130f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 0x7f1b48130f80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c013000 tx=0x7f1b3c017000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 shutdown_connections 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b34035ac0 0x7f1b34037f70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b480a1bf0 0x7f1b480968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b4809bed0 0x7f1b48130f80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 --2- 172.21.15.142:0/559663456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4809b150 0x7f1b481414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 >> 172.21.15.142:0/559663456 conn(0x7f1b48071680 msgr2=0x7f1b48134f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 shutdown_connections 2024-09-18T10:09:05.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.657+0000 7f1b4df45700 1 -- 172.21.15.142:0/559663456 wait complete. 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.661+0000 7f35953eb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.661+0000 7f35953eb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.661+0000 7f35953eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.661+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3590072a80 con 0x7f3590095d20 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f3577fff700 1 -- 172.21.15.142:0/3635021816 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f358001a360 con 0x7f3590095d20 2024-09-18T10:09:05.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f3577fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:05.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f3577fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.669 INFO:teuthology.orchestra.run.smithi142.stdout:47244640258 2024-09-18T10:09:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 msgr2=0x7f357c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f35840093f0 tx=0x7f3584016000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 10 monclient: shutdown 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 msgr2=0x7f3590141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 0x7f3590141600 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f358000c530 tx=0x7f3580007f90 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 shutdown_connections 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3590134510 0x7f359006bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3590096790 0x7f35901337c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 --2- 172.21.15.142:0/3635021816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3590095d20 0x7f3590141600 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 >> 172.21.15.142:0/3635021816 conn(0x7f35900a2390 msgr2=0x7f3590070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 shutdown_connections 2024-09-18T10:09:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.665+0000 7f35953eb700 1 -- 172.21.15.142:0/3635021816 wait complete. 2024-09-18T10:09:05.675 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-18T10:09:05.681 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-18T10:09:05.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbd54005360 con 0x7fbd74133870 2024-09-18T10:09:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd5affd700 1 -- 172.21.15.142:0/1789474183 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd6c011b80 con 0x7fbd74133870 2024-09-18T10:09:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd5affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd5affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.764 INFO:teuthology.orchestra.run.smithi142.stdout:47244640258 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 msgr2=0x7fbd60037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 0x7fbd60037fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd68006fd0 tx=0x7fbd68008040 comp rx=0 tx=0).stop 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 10 monclient: shutdown 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 msgr2=0x7fbd74094710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b6c0 tx=0x7fbd6c017000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 shutdown_connections 2024-09-18T10:09:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbd60035b10 0x7fbd60037fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd74133870 0x7fbd74094710 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd7409cb30 0x7fbd7406e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 --2- 172.21.15.142:0/1789474183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd7409c760 0x7fbd740884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 >> 172.21.15.142:0/1789474183 conn(0x7fbd74070ef0 msgr2=0x7fbd741349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 shutdown_connections 2024-09-18T10:09:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.757+0000 7fbd73fff700 1 -- 172.21.15.142:0/1789474183 wait complete. 2024-09-18T10:09:05.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe23bfff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe23bfff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe23bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe23c0717e0 con 0x7fe23c0994f0 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe222ffd700 1 -- 172.21.15.142:0/397506489 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe234011390 con 0x7fe23c0994f0 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe222ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.761+0000 7fe222ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.770 INFO:teuthology.orchestra.run.smithi142.stdout:51539607555 2024-09-18T10:09:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 msgr2=0x7fe228037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 0x7fe228037fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe22c004fe0 tx=0x7fe22c007b50 comp rx=0 tx=0).stop 2024-09-18T10:09:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 10 monclient: shutdown 2024-09-18T10:09:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 msgr2=0x7fe23c0898f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe234009ab0 tx=0x7fe234017000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 shutdown_connections 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe228035b10 0x7fe228037fc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe23c0994f0 0x7fe23c0898f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe23c0990d0 0x7fe23c139c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 --2- 172.21.15.142:0/397506489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe23c06f3b0 0x7fe23c1396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 >> 172.21.15.142:0/397506489 conn(0x7fe23c070ef0 msgr2=0x7fe23c0956d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 shutdown_connections 2024-09-18T10:09:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.765+0000 7fe23bfff700 1 -- 172.21.15.142:0/397506489 wait complete. 2024-09-18T10:09:05.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b27031700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b27031700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b27031700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4b200696a0 con 0x7f4b2006f230 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b1cff9700 1 -- 172.21.15.142:0/3537031224 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4b1802a090 con 0x7f4b2006f230 2024-09-18T10:09:05.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b1cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:05.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.769+0000 7f4b1cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.777 INFO:teuthology.orchestra.run.smithi142.stdout:51539607555 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 msgr2=0x7f4b080380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 0x7f4b080380d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b140058d0 tx=0x7f4b14009040 comp rx=0 tx=0).stop 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 10 monclient: shutdown 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 msgr2=0x7f4b20096530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c950 tx=0x7f4b18002760 comp rx=0 tx=0).stop 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 shutdown_connections 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b08035c20 0x7f4b080380d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b2009c9b0 0x7f4b2008f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009bed0 0x7f4b20095880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 --2- 172.21.15.142:0/3537031224 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2006f230 0x7f4b20096530 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 >> 172.21.15.142:0/3537031224 conn(0x7f4b20071820 msgr2=0x7f4b200972c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 shutdown_connections 2024-09-18T10:09:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.773+0000 7f4b27031700 1 -- 172.21.15.142:0/3537031224 wait complete. 2024-09-18T10:09:05.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-09-18T10:09:05.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-09-18T10:09:05.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:05.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:05.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:05.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9a3408f2e0 con 0x7f9a3409abd0 2024-09-18T10:09:05.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a1b7fe700 1 -- 172.21.15.142:0/2952634553 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a2400f650 con 0x7f9a3409abd0 2024-09-18T10:09:05.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a1b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:05.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a1b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:05.794 INFO:teuthology.orchestra.run.smithi142.stdout:47244640258 2024-09-18T10:09:05.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 msgr2=0x7f9a20037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 0x7f9a20037fc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9a280093a0 tx=0x7f9a28016000 comp rx=0 tx=0).stop 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 10 monclient: shutdown 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 msgr2=0x7f9a3413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c5e0 tx=0x7f9a24004a80 comp rx=0 tx=0).stop 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 shutdown_connections 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a20035b10 0x7f9a20037fc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a3409db80 0x7f9a3413fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a3409abd0 0x7f9a3413f920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 --2- 172.21.15.142:0/2952634553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a34097880 0x7f9a34094470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:05.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 >> 172.21.15.142:0/2952634553 conn(0x7f9a340a14e0 msgr2=0x7f9a34096380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:05.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 shutdown_connections 2024-09-18T10:09:05.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:05.789+0000 7f9a3a225700 1 -- 172.21.15.142:0/2952634553 wait complete. 2024-09-18T10:09:05.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-18T10:09:06.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-18T10:09:06.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T10:09:06.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T10:09:06.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T10:09:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: monmap: 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.788804+0000 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.788804+0000 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.791 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: monmap: 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.788797+0000 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.788797+0000 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 Processor -- start 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 -- start start 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: init 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 Processor -- start 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 -- start start 2024-09-18T10:09:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: init 2024-09-18T10:09:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T10:09:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa28c132360 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa28c06e4b0 0x7fa28c1373d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-c con 0x7fa28c06e4b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa28c137910 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c06e900 con 0x7fa28c132360 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c06db50 con 0x7fa28c137910 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28c06d220 con 0x7fa28c06e4b0 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa2943be700 10 monclient(hunting): authenticate will time out at 835.714532s 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa29215a700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa28c132360 0x7fa28c132730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46536/0 (socket says 172.21.15.142:46536) 2024-09-18T10:09:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa29215a700 1 -- 172.21.15.142:0/88194269 learned_addr learned my addr 172.21.15.142:0/88194269 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cc809cee0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cc809d9c0 0x7f6cc809b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-a con 0x7f6cc809d9c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6cc8134670 0x7f6cc8136a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-c con 0x7f6cc8134670 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc809eaa0 con 0x7f6cc809cee0 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc809a370 con 0x7f6cc809d9c0 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc809b7d0 con 0x7f6cc8134670 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cccea3700 10 monclient(hunting): authenticate will time out at 835.714997s 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa291959700 1 --2- 172.21.15.142:0/88194269 >> v2:172.21.15.150:3300/0 conn(0x7fa28c06e4b0 0x7fa28c1373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa291959700 10 monclient(hunting): get_auth_request con 0x7fa28c06e4b0 auth_method 0 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa291959700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa291959700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T10:09:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cc809d9c0 0x7f6cc809b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cc809d9c0 0x7f6cc809b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48572/0 (socket says 172.21.15.142:48572) 2024-09-18T10:09:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2681312242 learned_addr learned my addr 172.21.15.142:0/2681312242 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): get_auth_request con 0x7f6cc809d9c0 auth_method 0 2024-09-18T10:09:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7fa291959700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2681312242 >> v2:172.21.15.150:3300/0 conn(0x7f6cc8134670 msgr2=0x7f6cc8136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 --2- 172.21.15.142:0/2681312242 >> v2:172.21.15.150:3300/0 conn(0x7f6cc8134670 0x7f6cc8136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2681312242 >> v1:172.21.15.142:6789/0 conn(0x7f6cc809cee0 legacy=0x7f6cc809d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 monclient: found mon.noname-a 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.785+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2681312242 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc809a8b0 con 0x7f6cc809d9c0 2024-09-18T10:09:06.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 1 -- 172.21.15.142:0/88194269 >> v1:172.21.15.142:6789/0 conn(0x7fa28c132360 legacy=0x7fa28c132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535092778 0 0) 0x7fa27c001450 con 0x7fa28c132360 2024-09-18T10:09:06.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): my global_id is 4284 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa278003570 con 0x7fa28c132360 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 1 -- 172.21.15.142:0/88194269 >> v1:172.21.15.150:6789/0 conn(0x7fa28c137910 legacy=0x7fa28c139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300075644 0 0) 0x7fa278003570 con 0x7fa28c132360 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 >> v2:172.21.15.150:3300/0 conn(0x7fa28c06e4b0 msgr2=0x7fa28c1373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 --2- 172.21.15.142:0/88194269 >> v2:172.21.15.150:3300/0 conn(0x7fa28c06e4b0 0x7fa28c1373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 >> v1:172.21.15.150:6789/0 conn(0x7fa28c137910 legacy=0x7fa28c139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c06d500 con 0x7fa28c132360 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fa27c0019f0 con 0x7fa28c132360 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: dump: 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 >> v1:172.21.15.142:6789/0 conn(0x7fa28c132360 legacy=0x7fa28c132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa278003b80 0x7fa278003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): picked mon.b con 0x7fa278003b80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 0x7fa278004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): picked mon.c con 0x7fa278004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2780053c0 0x7fa27800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): picked mon.a con 0x7fa2780053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c06d220 con 0x7fa2780053c0 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27800c320 con 0x7fa278003b80 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27800c4e0 con 0x7fa278004810 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291158700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:09:06.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2681312242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc809d9c0 msgr2=0x7f6cc809b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc5d9b700 1 --2- 172.21.15.142:0/2681312242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc809d9c0 0x7f6cc809b290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6cc806b940 tx=0x7f6cbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313b225e7daf009d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2780053c0 0x7fa27800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): get_auth_request con 0x7fa2780053c0 auth_method 0 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 0x7fa278004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): get_auth_request con 0x7fa278004810 auth_method 0 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 1 -- 172.21.15.142:0/2681312242 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc00ba70 con 0x7f6cc809d9c0 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: dump: 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.809 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 5 monclient: authenticate success, global_id 4281 2024-09-18T10:09:06.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:06.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa278003b80 0x7fa278003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:06.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): get_auth_request con 0x7fa278003b80 auth_method 0 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 1 -- 172.21.15.142:0/2681312242 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc00bbd0 con 0x7f6cc809d9c0 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 1 -- 172.21.15.142:0/2681312242 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc011420 con 0x7f6cc809d9c0 2024-09-18T10:09:06.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 4 set_mon_vals no callback set 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cc559a700 10 monclient: dump: 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient(hunting): handle_auth_done global_id 4284 payload 293 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 -- 172.21.15.142:0/88194269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa278003b80 msgr2=0x7fa278003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa278003b80 0x7fa278003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 -- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2780053c0 msgr2=0x7fa27800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2780053c0 0x7fa27800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 monclient: found mon.c 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 1 -- 172.21.15.142:0/88194269 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa278003570 con 0x7fa278004810 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa291959700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa2943be700 5 monclient: authenticate success, global_id 4284 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa2943be700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2780053c0 0x7fa27800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:06.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7fa29295b700 1 -- 172.21.15.142:0/88194269 reap_dead start 2024-09-18T10:09:06.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 10 monclient: shutdown 2024-09-18T10:09:06.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 -- 172.21.15.142:0/2681312242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc809d9c0 msgr2=0x7f6cc809b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2681312242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc809d9c0 0x7f6cc809b290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6cc806b940 tx=0x7f6cbc009250 comp rx=0 tx=0).stop 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 -- 172.21.15.142:0/2681312242 shutdown_connections 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2681312242 >> v2:172.21.15.150:3300/0 conn(0x7f6cc8134670 0x7f6cc8136a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2681312242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc809d9c0 0x7f6cc809b290 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 -- 172.21.15.142:0/2681312242 >> 172.21.15.142:0/2681312242 conn(0x7f6cc80a24f0 msgr2=0x7f6cc80a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 -- 172.21.15.142:0/2681312242 shutdown_connections 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.789+0000 7f6cccea3700 1 -- 172.21.15.142:0/2681312242 wait complete. 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient: monmap: 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291959700 1 -- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 msgr2=0x7fa278004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291959700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 0x7fa278004c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa288009c00 tx=0x7fa2880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a363b0dd3e7a5c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 Processor -- start 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 -- start start 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient: init 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800e920 con 0x7fa278004810 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: dump: 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28800ea80 con 0x7fa278004810 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 1 -- 172.21.15.142:0/88194269 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800ee40 con 0x7fa278004810 2024-09-18T10:09:06.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa291158700 10 monclient: dump: 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 4 set_mon_vals no callback set 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 0x7f6cc806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-b con 0x7f6cc809cee0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-c con 0x7f6cc809d9c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 10 monclient: shutdown 2024-09-18T10:09:06.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 msgr2=0x7fa278004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8134670 0x7f6cc806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 0x7fa278004c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa288009c00 tx=0x7fa2880091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- 172.21.15.142:0/88194269 shutdown_connections 2024-09-18T10:09:06.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 --2- 172.21.15.142:0/88194269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa278004810 0x7fa278004c80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- 172.21.15.142:0/88194269 >> 172.21.15.142:0/88194269 conn(0x7fa28c0715a0 msgr2=0x7fa28c0a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8134670 0x7f6cc806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51288/0 (socket says 172.21.15.142:51288) 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2647116440 learned_addr learned my addr 172.21.15.142:0/2647116440 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): picked mon.noname-a con 0x7f6cc8134670 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc809bc70 con 0x7f6cc8134670 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc809fd10 con 0x7f6cc809cee0 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc806f960 con 0x7f6cc809d9c0 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cccea3700 10 monclient(hunting): authenticate will time out at 835.722194s 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 0x7f6cc806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): get_auth_request con 0x7f6cc809cee0 auth_method 0 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- 172.21.15.142:0/88194269 shutdown_connections 2024-09-18T10:09:06.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): get_auth_request con 0x7f6cc809d9c0 auth_method 0 2024-09-18T10:09:06.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- 172.21.15.142:0/88194269 wait complete. 2024-09-18T10:09:06.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): get_auth_request con 0x7f6cc8134670 auth_method 0 2024-09-18T10:09:06.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 msgr2=0x7f6cc806d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 0x7f6cc806d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8134670 msgr2=0x7f6cc806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8134670 0x7f6cc806dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0094f0 con 0x7f6cc809d9c0 2024-09-18T10:09:06.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7f6cc659c700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 0x7f6cc806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:06.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 10 monclient: monmap: 2024-09-18T10:09:06.833 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.833 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.833 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 Processor -- start 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 1 -- start start 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.793+0000 7fa2943be700 10 monclient: init 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:09:06.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa28c141d80 0x7fa28c142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-b con 0x7fa28c141d80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c142690 0x7fa28c119db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-a con 0x7fa28c142690 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): picked mon.noname-c con 0x7fa28c11a2f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c06eaf0 con 0x7fa28c142690 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c11b880 con 0x7fa28c141d80 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c0a2f40 con 0x7fa28c11a2f0 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient(hunting): authenticate will time out at 835.724342s 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c142690 0x7fa28c119db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51298/0 (socket says 172.21.15.142:51298) 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 -- 172.21.15.142:0/2765733692 learned_addr learned my addr 172.21.15.142:0/2765733692 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): get_auth_request con 0x7fa28c11a2f0 auth_method 0 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): get_auth_request con 0x7fa28c142690 auth_method 0 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa28c141d80 0x7fa28c142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): get_auth_request con 0x7fa28c141d80 auth_method 0 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa291959700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29215a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 -- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa28c141d80 msgr2=0x7fa28c142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa28c141d80 0x7fa28c142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 -- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c142690 msgr2=0x7fa28c119db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c142690 0x7fa28c119db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 -- 172.21.15.142:0/2765733692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c11ce30 con 0x7fa28c11a2f0 2024-09-18T10:09:06.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cc5d9b700 1 -- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 msgr2=0x7f6cc806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cc5d9b700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cc806add0 tx=0x7f6cbc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc011e80 con 0x7f6cc809d9c0 2024-09-18T10:09:06.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: dump: 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 5 monclient: authenticate success, global_id 4280 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc01f070 con 0x7f6cc809d9c0 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc01b880 con 0x7f6cc809d9c0 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 10 monclient: _renew_subs 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc806ec80 con 0x7f6cc809d9c0 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cc4d99700 4 set_mon_vals no callback set 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 10 monclient: dump: 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.841 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 10 monclient: _renew_subs 2024-09-18T10:09:06.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc8133310 con 0x7f6cc809d9c0 2024-09-18T10:09:06.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6cbc01bc70 con 0x7f6cc809d9c0 2024-09-18T10:09:06.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 -- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 msgr2=0x7fa28c11a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa29295b700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa28c065d90 tx=0x7fa284009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 0x7f6ca8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284005e70 con 0x7fa28c11a2f0 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: dump: 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.845 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 5 monclient: authenticate success, global_id 4283 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28400d8f0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _renew_subs 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c11e0c0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f6cbc021070 con 0x7f6cc809d9c0 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _renew_subs 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c142aa0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400dcb0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa290957700 4 set_mon_vals no callback set 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7fa2827fc700 10 monclient: dump: 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.849 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cc659c700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 0x7f6ca8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.797+0000 7f6cc659c700 10 monclient: get_auth_request con 0x7f6ca8035ac0 auth_method 0 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7f6cccea3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7f6cccea3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7f6cccea3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc81361a0 con 0x7f6cc809d9c0 2024-09-18T10:09:06.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa2840072e0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2943be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2943be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2943be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2740050b0 con 0x7fa28c11a2f0 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2827fc700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 0x7fa270037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fa284048310 con 0x7fa28c11a2f0 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa29215a700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 0x7fa270037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7fa29215a700 10 monclient: get_auth_request con 0x7fa270035ac0 auth_method 0 2024-09-18T10:09:06.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.801+0000 7f6cc659c700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 0x7f6ca8037f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8002d70 tx=0x7f6cb8007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.805+0000 7fa29215a700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 0x7fa270037f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa27c001660 tx=0x7fa27c008f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.809+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6cbc02b020 con 0x7f6cc809d9c0 2024-09-18T10:09:06.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.809+0000 7f6cb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.809+0000 7f6cb67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:06.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.817+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa28401bb40 con 0x7fa28c11a2f0 2024-09-18T10:09:06.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.817+0000 7fa2827fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.817+0000 7fa2827fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:06.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.885+0000 7fb6c32e9700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.885+0000 7fb6c32e9700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.885+0000 7fb6c32e9700 10 monclient: monmap: 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.891443+0000 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.891443+0000 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.893 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 Processor -- start 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 -- start start 2024-09-18T10:09:06.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient: init 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: monmap: 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.892388+0000 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.892388+0000 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.896 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.896 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6bc09c760 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 0x7fb6bc09cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-c con 0x7fb6bc09cb30 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb6bc133870 0x7fb6bc135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-a con 0x7fb6bc133870 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc098b00 con 0x7fb6bc09c760 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc098d20 con 0x7fb6bc133870 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc09a100 con 0x7fb6bc09cb30 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c32e9700 10 monclient(hunting): authenticate will time out at 835.817518s 2024-09-18T10:09:06.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 Processor -- start 2024-09-18T10:09:06.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb6bc133870 0x7fb6bc135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb6bc133870 0x7fb6bc135c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48620/0 (socket says 172.21.15.142:48620) 2024-09-18T10:09:06.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/4211924570 learned_addr learned my addr 172.21.15.142:0/4211924570 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): get_auth_request con 0x7fb6bc133870 auth_method 0 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 -- start start 2024-09-18T10:09:06.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: init 2024-09-18T10:09:06.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c1ae6700 1 --2- 172.21.15.142:0/4211924570 >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 0x7fb6bc09cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/4211924570 >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 msgr2=0x7fb6bc09cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 --2- 172.21.15.142:0/4211924570 >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 0x7fb6bc09cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/4211924570 >> v1:172.21.15.150:6789/0 conn(0x7fb6bc09c760 legacy=0x7fb6bc09b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 monclient: found mon.noname-a 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/4211924570 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc09a3e0 con 0x7fb6bc133870 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c1ae6700 10 monclient: get_auth_request con 0x7fb6bc09cb30 auth_method 0 2024-09-18T10:09:06.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7fb6c1ae6700 1 --2- 172.21.15.142:0/4211924570 >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 0x7fb6bc09cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:06.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:09:06.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f622409cdd0 0x7f622409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-a con 0x7f622409cdd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62240a02e0 0x7f62240a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-c con 0x7f62240a02e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f6224138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-b con 0x7f62240a0c20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622409d1a0 con 0x7f622409cdd0 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f622409e980 con 0x7f62240a02e0 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622409d780 con 0x7f62240a0c20 2024-09-18T10:09:06.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622bdb7700 10 monclient(hunting): authenticate will time out at 835.818739s 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f622409cdd0 0x7f622409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f6224138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f622409cdd0 0x7f622409e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48630/0 (socket says 172.21.15.142:48630) 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 -- 172.21.15.142:0/1614469633 learned_addr learned my addr 172.21.15.142:0/1614469633 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): get_auth_request con 0x7f622409cdd0 auth_method 0 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): get_auth_request con 0x7f62240a0c20 auth_method 0 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f622a354700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 -- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 msgr2=0x7f6224138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 --2- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f6224138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 -- 172.21.15.142:0/1614469633 >> v2:172.21.15.150:3300/0 conn(0x7f62240a02e0 msgr2=0x7f62240a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 --2- 172.21.15.142:0/1614469633 >> v2:172.21.15.150:3300/0 conn(0x7f62240a02e0 0x7f62240a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 monclient: found mon.noname-a 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.889+0000 7f6229b53700 1 -- 172.21.15.142:0/1614469633 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622409dd40 con 0x7f622409cdd0 2024-09-18T10:09:06.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/4211924570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 msgr2=0x7fb6bc135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c2ae8700 1 --2- 172.21.15.142:0/4211924570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc135c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc06fce0 tx=0x7fb6b000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49884f640235931 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 1 -- 172.21.15.142:0/4211924570 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0010470 con 0x7fb6bc133870 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: dump: 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.908 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 1 -- 172.21.15.142:0/4211924570 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b00105d0 con 0x7fb6bc133870 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 5 monclient: authenticate success, global_id 4293 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 4 set_mon_vals no callback set 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 1 -- 172.21.15.142:0/4211924570 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0010970 con 0x7fb6bc133870 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:06.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c12e5700 10 monclient: dump: 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.910 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6229b53700 1 -- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 msgr2=0x7f622409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6229b53700 1 --2- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622409e440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f622406a880 tx=0x7f621c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca969c8ea1a98cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 1 -- 172.21.15.142:0/1614469633 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00e070 con 0x7f622409cdd0 2024-09-18T10:09:06.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 10 monclient: shutdown 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 -- 172.21.15.142:0/4211924570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 msgr2=0x7fb6bc135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/4211924570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc135c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc06fce0 tx=0x7fb6b000b4f0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 -- 172.21.15.142:0/4211924570 shutdown_connections 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/4211924570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc135c50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/4211924570 >> v2:172.21.15.150:3300/0 conn(0x7fb6bc09cb30 0x7fb6bc09cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 -- 172.21.15.142:0/4211924570 >> 172.21.15.142:0/4211924570 conn(0x7fb6bc070ef0 msgr2=0x7fb6bc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: dump: 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 1 -- 172.21.15.142:0/1614469633 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c00bc30 con 0x7f622409cdd0 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 5 monclient: authenticate success, global_id 4296 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 4 set_mon_vals no callback set 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 1 -- 172.21.15.142:0/1614469633 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c0092e0 con 0x7f622409cdd0 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f6228b51700 10 monclient: dump: 2024-09-18T10:09:06.914 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 10 monclient: shutdown 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 -- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 msgr2=0x7f622409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622409e440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f622406a880 tx=0x7f621c009840 comp rx=0 tx=0).stop 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 -- 172.21.15.142:0/4211924570 shutdown_connections 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 -- 172.21.15.142:0/1614469633 shutdown_connections 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f6224138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1614469633 >> v2:172.21.15.150:3300/0 conn(0x7f62240a02e0 0x7f62240a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1614469633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622409e440 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7f622bdb7700 1 -- 172.21.15.142:0/1614469633 >> 172.21.15.142:0/1614469633 conn(0x7f6224071680 msgr2=0x7f62240a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.893+0000 7fb6c32e9700 1 -- 172.21.15.142:0/4211924570 wait complete. 2024-09-18T10:09:06.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient: monmap: 2024-09-18T10:09:06.917 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.917 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 Processor -- start 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 -- start start 2024-09-18T10:09:06.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient: init 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 1 -- 172.21.15.142:0/1614469633 shutdown_connections 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 0x7fb6bc13ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-c con 0x7fb6bc09c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc09cb30 0x7fb6bc12f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-b con 0x7fb6bc09cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): picked mon.noname-a con 0x7fb6bc133870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc130650 con 0x7fb6bc133870 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc126820 con 0x7fb6bc09cb30 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc06f810 con 0x7fb6bc09c760 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c32e9700 10 monclient(hunting): authenticate will time out at 835.825814s 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 0x7fb6bc13ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 1 -- 172.21.15.142:0/1614469633 wait complete. 2024-09-18T10:09:06.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48646/0 (socket says 172.21.15.142:48646) 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 1 -- 172.21.15.142:0/1167495764 learned_addr learned my addr 172.21.15.142:0/1167495764 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): get_auth_request con 0x7fb6bc133870 auth_method 0 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): get_auth_request con 0x7fb6bc09c760 auth_method 0 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc09cb30 0x7fb6bc12f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): get_auth_request con 0x7fb6bc09cb30 auth_method 0 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c1ae6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-18T10:09:06.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 10 monclient: monmap: 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.924 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 -- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc09cb30 msgr2=0x7fb6bc12f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc09cb30 0x7fb6bc12f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 -- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 msgr2=0x7fb6bc06d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c22e7700 1 -- 172.21.15.142:0/1167495764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b000b790 con 0x7fb6bc09c760 2024-09-18T10:09:06.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7fb6c2ae8700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:06.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 1 Processor -- start 2024-09-18T10:09:06.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 1 -- start start 2024-09-18T10:09:06.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.897+0000 7f622bdb7700 10 monclient: init 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-a con 0x7f622409cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62240a02e0 0x7f622408d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-b con 0x7f62240a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f622408d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): picked mon.noname-c con 0x7f62240a0c20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224090540 con 0x7f622409cdd0 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622408e7c0 con 0x7f62240a02e0 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622408ecb0 con 0x7f62240a0c20 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient(hunting): authenticate will time out at 835.828394s 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48652/0 (socket says 172.21.15.142:48652) 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 -- 172.21.15.142:0/1515804358 learned_addr learned my addr 172.21.15.142:0/1515804358 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): get_auth_request con 0x7f622409cdd0 auth_method 0 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62240a02e0 0x7f622408d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): get_auth_request con 0x7f62240a02e0 auth_method 0 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229352700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 -- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 msgr2=0x7f622408d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f622408d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 -- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62240a02e0 msgr2=0x7f622408d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62240a02e0 0x7f622408d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 monclient: found mon.noname-a 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 -- 172.21.15.142:0/1515804358 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c009ae0 con 0x7f622409cdd0 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7fb6c22e7700 1 -- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 msgr2=0x7fb6bc13ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7fb6c22e7700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 0x7fb6bc13ceb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c510 tx=0x7fb6b8007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 -- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 msgr2=0x7f622406f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6229b53700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f621c00b6c0 tx=0x7f621c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c007820 con 0x7f622409cdd0 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: dump: 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.933 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 5 monclient: authenticate success, global_id 4302 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c0079a0 con 0x7f622409cdd0 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c007da0 con 0x7f622409cdd0 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f6219ffb700 10 monclient: dump: 2024-09-18T10:09:06.934 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _renew_subs 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622408f370 con 0x7f622409cdd0 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _renew_subs 2024-09-18T10:09:06.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.901+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62240983a0 con 0x7f622409cdd0 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f621bfff700 4 set_mon_vals no callback set 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f621c01d420 con 0x7f622409cdd0 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f6219ffb700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 0x7f6208037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f621c049e00 con 0x7f622409cdd0 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f6229352700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 0x7f6208037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.905+0000 7f6229352700 10 monclient: get_auth_request con 0x7f6208035ac0 auth_method 0 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800ce00 con 0x7fb6bc09c760 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: dump: 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.937 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b80117a0 con 0x7fb6bc09c760 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 5 monclient: authenticate success, global_id 4289 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _renew_subs 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc06eef0 con 0x7fb6bc09c760 2024-09-18T10:09:06.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _renew_subs 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc13d3f0 con 0x7fb6bc09c760 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8011b60 con 0x7fb6bc09c760 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 10 monclient: dump: 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.939 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c0ae4700 4 set_mon_vals no callback set 2024-09-18T10:09:06.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7f6229352700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 0x7f6208037f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6214009bf0 tx=0x7f62140091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb6b801c460 con 0x7fb6bc09c760 2024-09-18T10:09:06.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 0x7fb6a0037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fb6b8047e20 con 0x7fb6bc09c760 2024-09-18T10:09:06.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c1ae6700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 0x7fb6a0037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c1ae6700 10 monclient: get_auth_request con 0x7fb6a0035b10 auth_method 0 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7f622bdb7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7f622bdb7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7f622bdb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62040050b0 con 0x7f622409cdd0 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.909+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a40050b0 con 0x7fb6bc09c760 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.913+0000 7fb6c1ae6700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 0x7fb6a0037fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc13df00 tx=0x7fb6b4007850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f621c021080 con 0x7f622409cdd0 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7f6219ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7f6219ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb6b80102e0 con 0x7fb6bc09c760 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7fb6ae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.917+0000 7fb6ae7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:06.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: monmap: 2024-09-18T10:09:06.945 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.941035+0000 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.941035+0000 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 Processor -- start 2024-09-18T10:09:06.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 -- start start 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: init 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31a409acc0 0x7f31a409a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f31a409acc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31a409b090 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 0x7f31a409c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f31a409b480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31a4099670 con 0x7f31a409b090 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31a409b8a0 con 0x7f31a409acc0 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a409dd40 con 0x7f31a409b480 2024-09-18T10:09:06.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a359e700 10 monclient(hunting): authenticate will time out at 835.866943s 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a1d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f31a409b090 0x7f31a409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46560/0 (socket says 172.21.15.142:46560) 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a1d9b700 1 -- 172.21.15.142:0/669772266 learned_addr learned my addr 172.21.15.142:0/669772266 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 --2- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 0x7f31a409c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): get_auth_request con 0x7f31a409b480 auth_method 0 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 1 --2- 172.21.15.142:0/669772266 >> v2:172.21.15.150:3300/0 conn(0x7f31a409acc0 0x7f31a409a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): get_auth_request con 0x7f31a409acc0 auth_method 0 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 -- 172.21.15.142:0/669772266 >> v2:172.21.15.150:3300/0 conn(0x7f31a409acc0 msgr2=0x7f31a409a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 --2- 172.21.15.142:0/669772266 >> v2:172.21.15.150:3300/0 conn(0x7f31a409acc0 0x7f31a409a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 -- 172.21.15.142:0/669772266 >> v1:172.21.15.142:6789/0 conn(0x7f31a409b090 legacy=0x7f31a409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 monclient: found mon.noname-b 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.937+0000 7f31a2d9d700 1 -- 172.21.15.142:0/669772266 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4099d70 con 0x7f31a409b480 2024-09-18T10:09:06.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a2d9d700 1 -- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 msgr2=0x7f31a409c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a2d9d700 1 --2- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 0x7f31a409c5e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31a4064650 tx=0x7f31940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98f7f5c51a734265 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 1 -- 172.21.15.142:0/669772266 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194011840 con 0x7f31a409b480 2024-09-18T10:09:06.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: dump: 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.954 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 1 -- 172.21.15.142:0/669772266 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31940119a0 con 0x7f31a409b480 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 1 -- 172.21.15.142:0/669772266 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3194011d40 con 0x7f31a409b480 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a159a700 10 monclient: dump: 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.955 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 5 monclient: authenticate success, global_id 4292 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 4 set_mon_vals no callback set 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 10 monclient: shutdown 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 -- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 msgr2=0x7f31a409c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 --2- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 0x7f31a409c5e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f31a4064650 tx=0x7f31940091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 -- 172.21.15.142:0/669772266 shutdown_connections 2024-09-18T10:09:06.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 --2- 172.21.15.142:0/669772266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409b480 0x7f31a409c5e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 --2- 172.21.15.142:0/669772266 >> v2:172.21.15.150:3300/0 conn(0x7f31a409acc0 0x7f31a409a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 -- 172.21.15.142:0/669772266 >> 172.21.15.142:0/669772266 conn(0x7f31a4071680 msgr2=0x7f31a40a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 -- 172.21.15.142:0/669772266 shutdown_connections 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 1 -- 172.21.15.142:0/669772266 wait complete. 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.941+0000 7f31a359e700 10 monclient: monmap: 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.957 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 Processor -- start 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 -- start start 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient: init 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 0x7f31a41420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-c con 0x7f31a409acc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a409b090 0x7f31a41416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-b con 0x7f31a409b090 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): picked mon.noname-a con 0x7f31a409b480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a40980c0 con 0x7f31a409b480 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4126a60 con 0x7f31a409b090 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a41267a0 con 0x7f31a409acc0 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a359e700 10 monclient(hunting): authenticate will time out at 835.873121s 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48654/0 (socket says 172.21.15.142:48654) 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 1 -- 172.21.15.142:0/2726773792 learned_addr learned my addr 172.21.15.142:0/2726773792 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): get_auth_request con 0x7f31a409b480 auth_method 0 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 0x7f31a41420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): get_auth_request con 0x7f31a409acc0 auth_method 0 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a409b090 0x7f31a41416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): get_auth_request con 0x7f31a409b090 auth_method 0 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 -- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a409b090 msgr2=0x7f31a41416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a409b090 0x7f31a41416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 -- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 msgr2=0x7f31a41455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a259c700 1 -- 172.21.15.142:0/2726773792 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194009420 con 0x7f31a409acc0 2024-09-18T10:09:06.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.945+0000 7f31a2d9d700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:06.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a259c700 1 -- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 msgr2=0x7f31a41420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a259c700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 0x7f31a41420a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f319800ea80 tx=0x7f31980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31980095d0 con 0x7f31a409acc0 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: dump: 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.963 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f319800ca00 con 0x7f31a409acc0 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 5 monclient: authenticate success, global_id 4295 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 10 monclient: _renew_subs 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 1 -- 172.21.15.142:0/2726773792 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a406dcd0 con 0x7f31a409acc0 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319800cdc0 con 0x7f31a409acc0 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31927fc700 10 monclient: dump: 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f3193fff700 4 set_mon_vals no callback set 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 10 monclient: _renew_subs 2024-09-18T10:09:06.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f31a359e700 1 -- 172.21.15.142:0/2726773792 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a406f150 con 0x7f31a409acc0 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f2de7551700 10 monclient: get_monmap_and_config 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.949+0000 7f2de7551700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient: monmap: 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:06.955734+0000 2024-09-18T10:09:06.966 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:06.955734+0000 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:06.967 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3198013680 con 0x7f31a409acc0 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31927fc700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 0x7f3184037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f31980483e0 con 0x7f31a409acc0 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f3183fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f3183fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31a1d9b700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 0x7f3184037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f3183fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a40948a0 con 0x7f31a409acc0 2024-09-18T10:09:06.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31a1d9b700 10 monclient: get_auth_request con 0x7f3184035b10 auth_method 0 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 Processor -- start 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 -- start start 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient: init 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de00999e0 0x7f2de00a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-b con 0x7f2de00999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2de009c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2de0137910 0x7f2de0139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-c con 0x7f2de0137910 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de00a20c0 con 0x7f2de009c410 2024-09-18T10:09:06.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de009a000 con 0x7f2de0137910 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de00a1c80 con 0x7f2de00999e0 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de7551700 10 monclient(hunting): authenticate will time out at 835.882198s 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de5d4e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2de009c410 0x7f2de00a06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46572/0 (socket says 172.21.15.142:46572) 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de5d4e700 1 -- 172.21.15.142:0/872402006 learned_addr learned my addr 172.21.15.142:0/872402006 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 --2- 172.21.15.142:0/872402006 >> v2:172.21.15.150:3300/0 conn(0x7f2de0137910 0x7f2de0139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): get_auth_request con 0x7f2de0137910 auth_method 0 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient(hunting): handle_auth_done global_id 4300 payload 274 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 -- 172.21.15.142:0/872402006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de00999e0 msgr2=0x7f2de00a0180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 --2- 172.21.15.142:0/872402006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de00999e0 0x7f2de00a0180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 -- 172.21.15.142:0/872402006 >> v1:172.21.15.142:6789/0 conn(0x7f2de009c410 legacy=0x7f2de00a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f2de6d50700 1 -- 172.21.15.142:0/872402006 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de009a580 con 0x7f2de0137910 2024-09-18T10:09:06.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.953+0000 7f31a1d9b700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 0x7f3184037fc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31a406e810 tx=0x7f319c0053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de6d50700 1 -- 172.21.15.142:0/872402006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de0137910 msgr2=0x7f2de0139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de6d50700 1 --2- 172.21.15.142:0/872402006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de0137910 0x7f2de0139cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009910 tx=0x7f2dd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fca3ffdaf358db96 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 1 -- 172.21.15.142:0/872402006 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd000ba30 con 0x7f2de0137910 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 10 monclient: dump: 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 1 -- 172.21.15.142:0/872402006 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd000bbb0 con 0x7f2de0137910 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de554d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 5 monclient: authenticate success, global_id 4300 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 4 set_mon_vals no callback set 2024-09-18T10:09:06.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 10 monclient: shutdown 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 -- 172.21.15.142:0/872402006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de0137910 msgr2=0x7f2de0139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 --2- 172.21.15.142:0/872402006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de0137910 0x7f2de0139cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0009910 tx=0x7f2dd00091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 -- 172.21.15.142:0/872402006 shutdown_connections 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 --2- 172.21.15.142:0/872402006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de0137910 0x7f2de0139cf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 --2- 172.21.15.142:0/872402006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de00999e0 0x7f2de00a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 -- 172.21.15.142:0/872402006 >> 172.21.15.142:0/872402006 conn(0x7f2de00a5790 msgr2=0x7f2de00a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 -- 172.21.15.142:0/872402006 shutdown_connections 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 1 -- 172.21.15.142:0/872402006 wait complete. 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 10 monclient: build_initial_monmap 2024-09-18T10:09:06.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.957+0000 7f2de7551700 10 monclient: monmap: 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:06.980 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f31a40948a0 con 0x7f31a409acc0 2024-09-18T10:09:06.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f31927fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f31927fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 Processor -- start 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 -- start start 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient: init 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 0x7f2de006b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-a con 0x7f2de00999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de009c410 0x7f2de006bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-b con 0x7f2de009c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:06.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): picked mon.noname-c con 0x7f2de0137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01369c0 con 0x7f2de00999e0 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de006e7f0 con 0x7f2de009c410 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de006c2f0 con 0x7f2de0137910 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de7551700 10 monclient(hunting): authenticate will time out at 835.889349s 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51340/0 (socket says 172.21.15.142:51340) 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 -- 172.21.15.142:0/3302799911 learned_addr learned my addr 172.21.15.142:0/3302799911 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): get_auth_request con 0x7f2de0137910 auth_method 0 2024-09-18T10:09:06.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de009c410 0x7f2de006bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): get_auth_request con 0x7f2de009c410 auth_method 0 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de654f700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 0x7f2de006b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:06.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de5d4e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 -- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de009c410 msgr2=0x7f2de006bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de009c410 0x7f2de006bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 -- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 msgr2=0x7f2de006b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 0x7f2de006b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient: _finish_hunting 0 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 monclient: found mon.noname-c 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de6d50700 1 -- 172.21.15.142:0/3302799911 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0009420 con 0x7f2de0137910 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de654f700 10 monclient: get_auth_request con 0x7f2de00999e0 auth_method 0 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.961+0000 7f2de654f700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 0x7f2de006b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:06.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de6d50700 1 -- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 msgr2=0x7f2de0141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de6d50700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2de00a56f0 tx=0x7f2dd0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd001c880 con 0x7f2de0137910 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: dump: 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.989 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 5 monclient: authenticate success, global_id 4298 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 10 monclient: _renew_subs 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de006e510 con 0x7f2de0137910 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 10 monclient: _renew_subs 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de006f980 con 0x7f2de0137910 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd001c9e0 con 0x7f2de0137910 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:06.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd001cd80 con 0x7f2de0137910 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.965+0000 7f2dd67fc700 10 monclient: dump: 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:06.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2dd001a660 con 0x7f2de0137910 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de4d4c700 4 set_mon_vals no callback set 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2dd67fc700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 0x7f2dc4037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f2dd004a8e0 con 0x7f2de0137910 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de654f700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 0x7f2dc4037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de654f700 10 monclient: get_auth_request con 0x7f2dc4035b10 auth_method 0 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de654f700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 0x7f2dc4037fc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800c330 tx=0x7f2dd80074d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de7551700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de7551700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de7551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.969+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80050b0 con 0x7f2de0137910 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.981+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2dd002eb20 con 0x7f2de0137910 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.981+0000 7f2dd67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:06.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:06.981+0000 7f2dd67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cccea3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cccea3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cccea3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6cc80696a0 con 0x7f6cc809d9c0 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cb67fc700 1 -- 172.21.15.142:0/2647116440 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6cbc037090 con 0x7f6cc809d9c0 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-18T10:09:07.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.061+0000 7f6cb67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.068 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 msgr2=0x7f6ca8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 0x7f6ca8037f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8002d70 tx=0x7f6cb8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 10 monclient: shutdown 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 msgr2=0x7f6cc806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6cc806add0 tx=0x7f6cbc016000 comp rx=0 tx=0).stop 2024-09-18T10:09:07.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 shutdown_connections 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ca8035ac0 0x7f6ca8037f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cc8134670 0x7f6cc806dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cc809d9c0 0x7f6cc806d800 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 --2- 172.21.15.142:0/2647116440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cc809cee0 0x7f6cc806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 >> 172.21.15.142:0/2647116440 conn(0x7f6cc80a24f0 msgr2=0x7f6cc8133c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 shutdown_connections 2024-09-18T10:09:07.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.065+0000 7f6cccea3700 1 -- 172.21.15.142:0/2647116440 wait complete. 2024-09-18T10:09:07.085 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-18T10:09:07.085 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa274005360 con 0x7fa28c11a2f0 2024-09-18T10:09:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2827fc700 1 -- 172.21.15.142:0/2765733692 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa28401b360 con 0x7fa28c11a2f0 2024-09-18T10:09:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2827fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-18T10:09:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2827fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.097 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:07.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 msgr2=0x7fa270037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 0x7fa270037f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa27c001660 tx=0x7fa27c008f90 comp rx=0 tx=0).stop 2024-09-18T10:09:07.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 10 monclient: shutdown 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 msgr2=0x7fa28c11a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa28c065d90 tx=0x7fa284009040 comp rx=0 tx=0).stop 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 shutdown_connections 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa270035ac0 0x7fa270037f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c11a2f0 0x7fa28c11a770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c142690 0x7fa28c119db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 --2- 172.21.15.142:0/2765733692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa28c141d80 0x7fa28c142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 >> 172.21.15.142:0/2765733692 conn(0x7fa28c0715a0 msgr2=0x7fa28c0a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 shutdown_connections 2024-09-18T10:09:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.093+0000 7fa2943be700 1 -- 172.21.15.142:0/2765733692 wait complete. 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6204005360 con 0x7f622409cdd0 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f6219ffb700 1 -- 172.21.15.142:0/1515804358 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f621c02b020 con 0x7f622409cdd0 2024-09-18T10:09:07.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f6219ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-18T10:09:07.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f6219ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.111 INFO:teuthology.orchestra.run.smithi142.stdout:47244640260 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 msgr2=0x7f6208037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 0x7f6208037f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6214009bf0 tx=0x7f62140091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 10 monclient: shutdown 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 msgr2=0x7f622406f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.105+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f621c00b6c0 tx=0x7f621c018000 comp rx=0 tx=0).stop 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 shutdown_connections 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6208035ac0 0x7f6208037f70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62240a0c20 0x7f622408d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62240a02e0 0x7f622408d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 --2- 172.21.15.142:0/1515804358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f622409cdd0 0x7f622406f610 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 >> 172.21.15.142:0/1515804358 conn(0x7f6224071680 msgr2=0x7f62240a4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 shutdown_connections 2024-09-18T10:09:07.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-18T10:09:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.109+0000 7f622bdb7700 1 -- 172.21.15.142:0/1515804358 wait complete. 2024-09-18T10:09:07.114 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-18T10:09:07.122 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.125+0000 7f3183fff700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:07.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.125+0000 7f3183fff700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:07.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.125+0000 7f3183fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.125+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f31a40948a0 con 0x7f31a409acc0 2024-09-18T10:09:07.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f31927fc700 1 -- 172.21.15.142:0/2726773792 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f31a40948a0 con 0x7f31a409acc0 2024-09-18T10:09:07.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f31927fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-18T10:09:07.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f31927fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.134 INFO:teuthology.orchestra.run.smithi142.stdout:51539607557 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 msgr2=0x7f3184037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 0x7f3184037fc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31a406e810 tx=0x7f319c0053c0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 10 monclient: shutdown 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 msgr2=0x7f31a41420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 0x7f31a41420a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f319800ea80 tx=0x7f31980091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 shutdown_connections 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3184035b10 0x7f3184037fc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31a409b480 0x7f31a41455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31a409b090 0x7f31a41416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 --2- 172.21.15.142:0/2726773792 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a409acc0 0x7f31a41420a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 >> 172.21.15.142:0/2726773792 conn(0x7f31a4071680 msgr2=0x7f31a40a08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 shutdown_connections 2024-09-18T10:09:07.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.129+0000 7f3183fff700 1 -- 172.21.15.142:0/2726773792 wait complete. 2024-09-18T10:09:07.144 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607557 for osd.1 2024-09-18T10:09:07.144 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6c32e9700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6c32e9700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6c32e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb6a4005360 con 0x7fb6bc09c760 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6ae7fc700 1 -- 172.21.15.142:0/1167495764 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6b8002090 con 0x7fb6bc09c760 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6ae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.157+0000 7fb6ae7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.164 INFO:teuthology.orchestra.run.smithi142.stdout:51539607557 2024-09-18T10:09:07.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 msgr2=0x7fb6a0037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 0x7fb6a0037fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc13df00 tx=0x7fb6b4007850 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 10 monclient: shutdown 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 msgr2=0x7fb6bc13ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 0x7fb6bc13ceb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800c510 tx=0x7fb6b8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 shutdown_connections 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb6a0035b10 0x7fb6a0037fc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6bc133870 0x7fb6bc06d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6bc09cb30 0x7fb6bc12f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 --2- 172.21.15.142:0/1167495764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb6bc09c760 0x7fb6bc13ceb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 >> 172.21.15.142:0/1167495764 conn(0x7fb6bc070ef0 msgr2=0x7fb6bc132a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 shutdown_connections 2024-09-18T10:09:07.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.161+0000 7fb6c32e9700 1 -- 172.21.15.142:0/1167495764 wait complete. 2024-09-18T10:09:07.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607557 for osd.2 2024-09-18T10:09:07.177 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2de7551700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:07.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2de7551700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:07.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2de7551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2dc8005360 con 0x7f2de0137910 2024-09-18T10:09:07.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2dd67fc700 1 -- 172.21.15.142:0/3302799911 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dd0023b80 con 0x7f2de0137910 2024-09-18T10:09:07.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2dd67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-18T10:09:07.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.189+0000 7f2dd67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:07.196 INFO:teuthology.orchestra.run.smithi142.stdout:47244640259 2024-09-18T10:09:07.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 msgr2=0x7f2dc4037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 0x7f2dc4037fc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800c330 tx=0x7f2dd80074d0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 10 monclient: shutdown 2024-09-18T10:09:07.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 msgr2=0x7f2de0141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2de00a56f0 tx=0x7f2dd0016000 comp rx=0 tx=0).stop 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 shutdown_connections 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2dc4035b10 0x7f2dc4037fc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2de0137910 0x7f2de0141290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de009c410 0x7f2de006bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 --2- 172.21.15.142:0/3302799911 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2de00999e0 0x7f2de006b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 >> 172.21.15.142:0/3302799911 conn(0x7f2de00a5790 msgr2=0x7f2de00a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 shutdown_connections 2024-09-18T10:09:07.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.193+0000 7f2de7551700 1 -- 172.21.15.142:0/3302799911 wait complete. 2024-09-18T10:09:07.208 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-18T10:09:07.208 DEBUG:teuthology.parallel:result is None 2024-09-18T10:09:07.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T10:09:07.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:09:07.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: get_monmap_and_config 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: monmap: 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:07.352031+0000 2024-09-18T10:09:07.353 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:07.352031+0000 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:07.354 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 Processor -- start 2024-09-18T10:09:07.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 -- start start 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: init 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-b con 0x7fc55c132040 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc55c0a1f30 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc55c137370 0x7fc55c139750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-a con 0x7fc55c137370 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:07.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c09dcf0 con 0x7fc55c0a1f30 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c06fc50 con 0x7fc55c137370 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c132950 con 0x7fc55c132040 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc56337b700 10 monclient(hunting): authenticate will time out at 836.277420s 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc55c137370 0x7fc55c139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51356/0 (socket says 172.21.15.142:51356) 2024-09-18T10:09:07.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 1 -- 172.21.15.142:0/2596134196 learned_addr learned my addr 172.21.15.142:0/2596134196 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): get_auth_request con 0x7fc55c132040 auth_method 0 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): get_auth_request con 0x7fc55c137370 auth_method 0 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 -- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 msgr2=0x7fc55c132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 -- 172.21.15.142:0/2596134196 >> v1:172.21.15.150:6789/0 conn(0x7fc55c0a1f30 legacy=0x7fc55c09f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 monclient: found mon.noname-a 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561918700 1 -- 172.21.15.142:0/2596134196 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c132b80 con 0x7fc55c137370 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.349+0000 7fc561117700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc561918700 1 -- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 msgr2=0x7fc55c139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc561918700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c139750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5540098e0 tx=0x7fc5540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7030e5b774e6d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 1 -- 172.21.15.142:0/2596134196 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400b850 con 0x7fc55c137370 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: dump: 2024-09-18T10:09:07.363 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 5 monclient: authenticate success, global_id 4308 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 1 -- 172.21.15.142:0/2596134196 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400b9b0 con 0x7fc55c137370 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 4 set_mon_vals no callback set 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 1 -- 172.21.15.142:0/2596134196 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400bd50 con 0x7fc55c137370 2024-09-18T10:09:07.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc553fff700 10 monclient: dump: 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.366 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 10 monclient: shutdown 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 -- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 msgr2=0x7fc55c139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c139750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5540098e0 tx=0x7fc5540091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 -- 172.21.15.142:0/2596134196 shutdown_connections 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c139750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 --2- 172.21.15.142:0/2596134196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 -- 172.21.15.142:0/2596134196 >> 172.21.15.142:0/2596134196 conn(0x7fc55c071680 msgr2=0x7fc55c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 -- 172.21.15.142:0/2596134196 shutdown_connections 2024-09-18T10:09:07.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 1 -- 172.21.15.142:0/2596134196 wait complete. 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.353+0000 7fc56337b700 10 monclient: monmap: 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:07.373 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 Processor -- start 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 -- start start 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient: init 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc55c0a1f30 0x7fc55c09b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-b con 0x7fc55c0a1f30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-c con 0x7fc55c132040 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c09bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): picked mon.noname-a con 0x7fc55c137370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:07.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c09d1c0 con 0x7fc55c137370 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c12a950 con 0x7fc55c0a1f30 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c136d30 con 0x7fc55c132040 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc56337b700 10 monclient(hunting): authenticate will time out at 836.285849s 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51368/0 (socket says 172.21.15.142:51368) 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 -- 172.21.15.142:0/635749030 learned_addr learned my addr 172.21.15.142:0/635749030 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): get_auth_request con 0x7fc55c132040 auth_method 0 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c09bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc55c0a1f30 0x7fc55c09b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): get_auth_request con 0x7fc55c0a1f30 auth_method 0 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): get_auth_request con 0x7fc55c137370 auth_method 0 2024-09-18T10:09:07.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561117700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 -- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc55c0a1f30 msgr2=0x7fc55c09b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc55c0a1f30 0x7fc55c09b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 -- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 msgr2=0x7fc55c09bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c09bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 monclient: found mon.noname-c 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc560916700 1 -- 172.21.15.142:0/635749030 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0094f0 con 0x7fc55c132040 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.357+0000 7fc561918700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc560916700 1 -- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 msgr2=0x7fc55c09b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc560916700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc55800bc60 tx=0x7fc558009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800c9f0 con 0x7fc55c132040 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:07.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: dump: 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 5 monclient: authenticate success, global_id 4304 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _renew_subs 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55800cb50 con 0x7fc55c132040 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc554009420 con 0x7fc55c132040 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558014410 con 0x7fc55c132040 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5537fe700 4 set_mon_vals no callback set 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 10 monclient: dump: 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.380 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _renew_subs 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c12e960 con 0x7fc55c132040 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc558014840 con 0x7fc55c132040 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 0x7fc548037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c0696a0 con 0x7fc55c132040 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fc5580481b0 con 0x7fc55c132040 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc561117700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 0x7fc548037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.361+0000 7fc561117700 10 monclient: get_auth_request con 0x7fc548035ac0 auth_method 0 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.365+0000 7fc561117700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 0x7fc548037f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc54c006d20 tx=0x7fc54c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.369+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc5580073e0 con 0x7fc55c132040 2024-09-18T10:09:07.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.369+0000 7fc5517fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.369+0000 7fc5517fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:07.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.529+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc55c0678a0 con 0x7fc548035ac0 2024-09-18T10:09:07.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.533+0000 7fc5517fa700 1 -- 172.21.15.142:0/635749030 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41009 (secure 0 0 0) 0x7fc55c0678a0 con 0x7fc548035ac0 2024-09-18T10:09:07.539 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 msgr2=0x7fc548037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 0x7fc548037f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc54c006d20 tx=0x7fc54c009250 comp rx=0 tx=0).stop 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 10 monclient: shutdown 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 msgr2=0x7fc55c09b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc55800bc60 tx=0x7fc558009820 comp rx=0 tx=0).stop 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 shutdown_connections 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc548035ac0 0x7fc548037f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc55c137370 0x7fc55c09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc55c132040 0x7fc55c09b6d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 --2- 172.21.15.142:0/635749030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc55c0a1f30 0x7fc55c09b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 >> 172.21.15.142:0/635749030 conn(0x7fc55c071680 msgr2=0x7fc55c138ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 shutdown_connections 2024-09-18T10:09:07.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.537+0000 7fc56337b700 1 -- 172.21.15.142:0/635749030 wait complete. 2024-09-18T10:09:07.543 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:09:07.553 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-18T10:09:06.764349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786356,"kb_used_data":1936,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444684,"statfs":{"total":579820584960,"available":577991356416,"internally_reserved":0,"allocated":1982464,"data_stored":1338028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.784974"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982814+0000","last_change":"2024-09-18T10:09:03.983242+0000","last_active":"2024-09-18T10:09:03.982814+0000","last_peered":"2024-09-18T10:09:03.982814+0000","last_clean":"2024-09-18T10:09:03.982814+0000","last_became_active":"2024-09-18T10:09:02.313264+0000","last_became_peered":"2024-09-18T10:09:02.313264+0000","last_unstale":"2024-09-18T10:09:03.982814+0000","last_undegraded":"2024-09-18T10:09:03.982814+0000","last_fullsized":"2024-09-18T10:09:03.982814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:35:49.571437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980698+0000","last_change":"2024-09-18T10:09:03.981143+0000","last_active":"2024-09-18T10:09:03.980698+0000","last_peered":"2024-09-18T10:09:03.980698+0000","last_clean":"2024-09-18T10:09:03.980698+0000","last_became_active":"2024-09-18T10:09:01.977398+0000","last_became_peered":"2024-09-18T10:09:01.977398+0000","last_unstale":"2024-09-18T10:09:03.980698+0000","last_undegraded":"2024-09-18T10:09:03.980698+0000","last_fullsized":"2024-09-18T10:09:03.980698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:55:31.629074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982451+0000","last_change":"2024-09-18T10:09:03.982833+0000","last_active":"2024-09-18T10:09:03.982451+0000","last_peered":"2024-09-18T10:09:03.982451+0000","last_clean":"2024-09-18T10:09:03.982451+0000","last_became_active":"2024-09-18T10:09:01.975781+0000","last_became_peered":"2024-09-18T10:09:01.975781+0000","last_unstale":"2024-09-18T10:09:03.982451+0000","last_undegraded":"2024-09-18T10:09:03.982451+0000","last_fullsized":"2024-09-18T10:09:03.982451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:07:07.146280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980775+0000","last_change":"2024-09-18T10:09:03.981141+0000","last_active":"2024-09-18T10:09:03.980775+0000","last_peered":"2024-09-18T10:09:03.980775+0000","last_clean":"2024-09-18T10:09:03.980775+0000","last_became_active":"2024-09-18T10:09:01.982185+0000","last_became_peered":"2024-09-18T10:09:01.982185+0000","last_unstale":"2024-09-18T10:09:03.980775+0000","last_undegraded":"2024-09-18T10:09:03.980775+0000","last_fullsized":"2024-09-18T10:09:03.980775+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:12:33.081514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.988120+0000","last_change":"2024-09-18T10:09:03.988120+0000","last_active":"2024-09-18T10:09:03.988120+0000","last_peered":"2024-09-18T10:09:03.988120+0000","last_clean":"2024-09-18T10:09:03.988120+0000","last_became_active":"2024-09-18T10:09:01.980916+0000","last_became_peered":"2024-09-18T10:09:01.980916+0000","last_unstale":"2024-09-18T10:09:03.988120+0000","last_undegraded":"2024-09-18T10:09:03.988120+0000","last_fullsized":"2024-09-18T10:09:03.988120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:51.358013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:04.292190+0000","last_change":"2024-09-18T10:09:04.292594+0000","last_active":"2024-09-18T10:09:04.292190+0000","last_peered":"2024-09-18T10:09:04.292190+0000","last_clean":"2024-09-18T10:09:04.292190+0000","last_became_active":"2024-09-18T10:09:01.985425+0000","last_became_peered":"2024-09-18T10:09:01.985425+0000","last_unstale":"2024-09-18T10:09:04.292190+0000","last_undegraded":"2024-09-18T10:09:04.292190+0000","last_fullsized":"2024-09-18T10:09:04.292190+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:20.190675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982341+0000","last_change":"2024-09-18T10:09:03.982693+0000","last_active":"2024-09-18T10:09:03.982341+0000","last_peered":"2024-09-18T10:09:03.982341+0000","last_clean":"2024-09-18T10:09:03.982341+0000","last_became_active":"2024-09-18T10:09:01.976391+0000","last_became_peered":"2024-09-18T10:09:01.976391+0000","last_unstale":"2024-09-18T10:09:03.982341+0000","last_undegraded":"2024-09-18T10:09:03.982341+0000","last_fullsized":"2024-09-18T10:09:03.982341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:46:46.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.987814+0000","last_change":"2024-09-18T10:09:03.988400+0000","last_active":"2024-09-18T10:09:03.987814+0000","last_peered":"2024-09-18T10:09:03.987814+0000","last_clean":"2024-09-18T10:09:03.987814+0000","last_became_active":"2024-09-18T10:09:01.983714+0000","last_became_peered":"2024-09-18T10:09:01.983714+0000","last_unstale":"2024-09-18T10:09:03.987814+0000","last_undegraded":"2024-09-18T10:09:03.987814+0000","last_fullsized":"2024-09-18T10:09:03.987814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:44.111047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":101,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T10:09:02.980246+0000","last_change":"2024-09-18T10:09:00.850110+0000","last_active":"2024-09-18T10:09:02.980246+0000","last_peered":"2024-09-18T10:09:02.980246+0000","last_clean":"2024-09-18T10:09:02.980246+0000","last_became_active":"2024-09-18T10:09:00.849769+0000","last_became_peered":"2024-09-18T10:09:00.849769+0000","last_unstale":"2024-09-18T10:09:02.980246+0000","last_undegraded":"2024-09-18T10:09:02.980246+0000","last_fullsized":"2024-09-18T10:09:02.980246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:02.351581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":131072,"data_stored":38499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:09:07.555 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: get_monmap_and_config 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: monmap: 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:07.657306+0000 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:07.657306+0000 2024-09-18T10:09:07.658 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:07.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 Processor -- start 2024-09-18T10:09:07.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 -- start start 2024-09-18T10:09:07.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: init 2024-09-18T10:09:07.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 0x7fd54c0a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): picked mon.noname-a con 0x7fd54c0a5f90 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): picked mon.noname-b con 0x7fd54c0990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd54c09e360 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c09d980 con 0x7fd54c09e360 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c09d780 con 0x7fd54c0a5f90 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c09fc70 con 0x7fd54c0990b0 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd553510700 10 monclient(hunting): authenticate will time out at 836.583195s 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 0x7fd54c0a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51382/0 (socket says 172.21.15.142:51382) 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 -- 172.21.15.142:0/2140474961 learned_addr learned my addr 172.21.15.142:0/2140474961 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): get_auth_request con 0x7fd54c0990b0 auth_method 0 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): get_auth_request con 0x7fd54c0a5f90 auth_method 0 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 -- 172.21.15.142:0/2140474961 >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 msgr2=0x7fd54c0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 --2- 172.21.15.142:0/2140474961 >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 0x7fd54c0a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 -- 172.21.15.142:0/2140474961 >> v1:172.21.15.150:6789/0 conn(0x7fd54c09e360 legacy=0x7fd54c138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 monclient: found mon.noname-b 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd550aab700 1 -- 172.21.15.142:0/2140474961 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0a6360 con 0x7fd54c0990b0 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.652+0000 7fd5512ac700 1 --2- 172.21.15.142:0/2140474961 >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 0x7fd54c0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd550aab700 1 -- 172.21.15.142:0/2140474961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 msgr2=0x7fd54c0a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd550aab700 1 --2- 172.21.15.142:0/2140474961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5400099b0 tx=0x7fd540009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2b364735455c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 1 -- 172.21.15.142:0/2140474961 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd540009ea0 con 0x7fd54c0990b0 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: dump: 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.668 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 5 monclient: authenticate success, global_id 4307 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 1 -- 172.21.15.142:0/2140474961 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54000ba70 con 0x7fd54c0990b0 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 4 set_mon_vals no callback set 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 1 -- 172.21.15.142:0/2140474961 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54000be30 con 0x7fd54c0990b0 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd54bfff700 10 monclient: dump: 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.669 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 10 monclient: shutdown 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 -- 172.21.15.142:0/2140474961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 msgr2=0x7fd54c0a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 --2- 172.21.15.142:0/2140474961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5400099b0 tx=0x7fd540009250 comp rx=0 tx=0).stop 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 -- 172.21.15.142:0/2140474961 shutdown_connections 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 --2- 172.21.15.142:0/2140474961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0990b0 0x7fd54c0a07b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 --2- 172.21.15.142:0/2140474961 >> v2:172.21.15.142:3300/0 conn(0x7fd54c0a5f90 0x7fd54c0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.656+0000 7fd553510700 1 -- 172.21.15.142:0/2140474961 >> 172.21.15.142:0/2140474961 conn(0x7fd54c071680 msgr2=0x7fd54c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- 172.21.15.142:0/2140474961 shutdown_connections 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- 172.21.15.142:0/2140474961 wait complete. 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient: monmap: 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:07.677 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 Processor -- start 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- start start 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient: init 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): picked mon.noname-a con 0x7fd54c0990b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c09e360 0x7fd54c08db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): picked mon.noname-b con 0x7fd54c09e360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:07.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0a5f90 0x7fd54c08e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): picked mon.noname-c con 0x7fd54c0a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c09c120 con 0x7fd54c0990b0 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c090670 con 0x7fd54c09e360 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c08ece0 con 0x7fd54c0a5f90 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd553510700 10 monclient(hunting): authenticate will time out at 836.591350s 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48708/0 (socket says 172.21.15.142:48708) 2024-09-18T10:09:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 -- 172.21.15.142:0/49766328 learned_addr learned my addr 172.21.15.142:0/49766328 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): get_auth_request con 0x7fd54c0990b0 auth_method 0 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd551aad700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0a5f90 0x7fd54c08e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd550aab700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c09e360 0x7fd54c08db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd551aad700 10 monclient(hunting): get_auth_request con 0x7fd54c0a5f90 auth_method 0 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd551aad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd551aad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd551aad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd550aab700 10 monclient(hunting): get_auth_request con 0x7fd54c09e360 auth_method 0 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd550aab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd550aab700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd550aab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 -- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0a5f90 msgr2=0x7fd54c08e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0a5f90 0x7fd54c08e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 -- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c09e360 msgr2=0x7fd54c08db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c09e360 0x7fd54c08db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 monclient: found mon.noname-a 2024-09-18T10:09:07.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.660+0000 7fd5512ac700 1 -- 172.21.15.142:0/49766328 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd544009a20 con 0x7fd54c0990b0 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5512ac700 1 -- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 msgr2=0x7fd54c08d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5512ac700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c170 tx=0x7fd54400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544014480 con 0x7fd54c0990b0 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: dump: 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 5 monclient: authenticate success, global_id 4317 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _renew_subs 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54400f040 con 0x7fd54c0990b0 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5400094f0 con 0x7fd54c0990b0 2024-09-18T10:09:07.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5440147f0 con 0x7fd54c0990b0 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd5497fa700 10 monclient: dump: 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd54b7fe700 4 set_mon_vals no callback set 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _renew_subs 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c0903b0 con 0x7fd54c0990b0 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.664+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c06d3f0 con 0x7fd54c0990b0 2024-09-18T10:09:07.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd544014b60 con 0x7fd54c0990b0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd5497fa700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 0x7fd538037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7fd5440491c0 con 0x7fd54c0990b0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd550aab700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 0x7fd538037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd550aab700 10 monclient: get_auth_request con 0x7fd538035ac0 auth_method 0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.668+0000 7fd550aab700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 0x7fd538037f70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5400099b0 tx=0x7fd540016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.676+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd544024300 con 0x7fd54c0990b0 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.676+0000 7fd5497fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.676+0000 7fd5497fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:07.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.836+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd54c0696a0 con 0x7fd538035ac0 2024-09-18T10:09:07.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.840+0000 7fd5497fa700 1 -- 172.21.15.142:0/49766328 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41009 (secure 0 0 0) 0x7fd54c0696a0 con 0x7fd538035ac0 2024-09-18T10:09:07.848 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:07.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 msgr2=0x7fd538037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 0x7fd538037f70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd5400099b0 tx=0x7fd540016000 comp rx=0 tx=0).stop 2024-09-18T10:09:07.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 10 monclient: shutdown 2024-09-18T10:09:07.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 msgr2=0x7fd54c08d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd54400c170 tx=0x7fd54400b040 comp rx=0 tx=0).stop 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 shutdown_connections 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd538035ac0 0x7fd538037f70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd54c0a5f90 0x7fd54c08e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd54c09e360 0x7fd54c08db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 --2- 172.21.15.142:0/49766328 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd54c0990b0 0x7fd54c08d5f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 >> 172.21.15.142:0/49766328 conn(0x7fd54c071680 msgr2=0x7fd54c09f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 shutdown_connections 2024-09-18T10:09:07.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.844+0000 7fd553510700 1 -- 172.21.15.142:0/49766328 wait complete. 2024-09-18T10:09:07.852 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:09:07.863 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-18T10:09:06.764349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786356,"kb_used_data":1936,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444684,"statfs":{"total":579820584960,"available":577991356416,"internally_reserved":0,"allocated":1982464,"data_stored":1338028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.784974"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982814+0000","last_change":"2024-09-18T10:09:03.983242+0000","last_active":"2024-09-18T10:09:03.982814+0000","last_peered":"2024-09-18T10:09:03.982814+0000","last_clean":"2024-09-18T10:09:03.982814+0000","last_became_active":"2024-09-18T10:09:02.313264+0000","last_became_peered":"2024-09-18T10:09:02.313264+0000","last_unstale":"2024-09-18T10:09:03.982814+0000","last_undegraded":"2024-09-18T10:09:03.982814+0000","last_fullsized":"2024-09-18T10:09:03.982814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:35:49.571437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980698+0000","last_change":"2024-09-18T10:09:03.981143+0000","last_active":"2024-09-18T10:09:03.980698+0000","last_peered":"2024-09-18T10:09:03.980698+0000","last_clean":"2024-09-18T10:09:03.980698+0000","last_became_active":"2024-09-18T10:09:01.977398+0000","last_became_peered":"2024-09-18T10:09:01.977398+0000","last_unstale":"2024-09-18T10:09:03.980698+0000","last_undegraded":"2024-09-18T10:09:03.980698+0000","last_fullsized":"2024-09-18T10:09:03.980698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:55:31.629074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982451+0000","last_change":"2024-09-18T10:09:03.982833+0000","last_active":"2024-09-18T10:09:03.982451+0000","last_peered":"2024-09-18T10:09:03.982451+0000","last_clean":"2024-09-18T10:09:03.982451+0000","last_became_active":"2024-09-18T10:09:01.975781+0000","last_became_peered":"2024-09-18T10:09:01.975781+0000","last_unstale":"2024-09-18T10:09:03.982451+0000","last_undegraded":"2024-09-18T10:09:03.982451+0000","last_fullsized":"2024-09-18T10:09:03.982451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:07:07.146280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980775+0000","last_change":"2024-09-18T10:09:03.981141+0000","last_active":"2024-09-18T10:09:03.980775+0000","last_peered":"2024-09-18T10:09:03.980775+0000","last_clean":"2024-09-18T10:09:03.980775+0000","last_became_active":"2024-09-18T10:09:01.982185+0000","last_became_peered":"2024-09-18T10:09:01.982185+0000","last_unstale":"2024-09-18T10:09:03.980775+0000","last_undegraded":"2024-09-18T10:09:03.980775+0000","last_fullsized":"2024-09-18T10:09:03.980775+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:12:33.081514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.988120+0000","last_change":"2024-09-18T10:09:03.988120+0000","last_active":"2024-09-18T10:09:03.988120+0000","last_peered":"2024-09-18T10:09:03.988120+0000","last_clean":"2024-09-18T10:09:03.988120+0000","last_became_active":"2024-09-18T10:09:01.980916+0000","last_became_peered":"2024-09-18T10:09:01.980916+0000","last_unstale":"2024-09-18T10:09:03.988120+0000","last_undegraded":"2024-09-18T10:09:03.988120+0000","last_fullsized":"2024-09-18T10:09:03.988120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:51.358013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:04.292190+0000","last_change":"2024-09-18T10:09:04.292594+0000","last_active":"2024-09-18T10:09:04.292190+0000","last_peered":"2024-09-18T10:09:04.292190+0000","last_clean":"2024-09-18T10:09:04.292190+0000","last_became_active":"2024-09-18T10:09:01.985425+0000","last_became_peered":"2024-09-18T10:09:01.985425+0000","last_unstale":"2024-09-18T10:09:04.292190+0000","last_undegraded":"2024-09-18T10:09:04.292190+0000","last_fullsized":"2024-09-18T10:09:04.292190+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:20.190675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982341+0000","last_change":"2024-09-18T10:09:03.982693+0000","last_active":"2024-09-18T10:09:03.982341+0000","last_peered":"2024-09-18T10:09:03.982341+0000","last_clean":"2024-09-18T10:09:03.982341+0000","last_became_active":"2024-09-18T10:09:01.976391+0000","last_became_peered":"2024-09-18T10:09:01.976391+0000","last_unstale":"2024-09-18T10:09:03.982341+0000","last_undegraded":"2024-09-18T10:09:03.982341+0000","last_fullsized":"2024-09-18T10:09:03.982341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:46:46.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.987814+0000","last_change":"2024-09-18T10:09:03.988400+0000","last_active":"2024-09-18T10:09:03.987814+0000","last_peered":"2024-09-18T10:09:03.987814+0000","last_clean":"2024-09-18T10:09:03.987814+0000","last_became_active":"2024-09-18T10:09:01.983714+0000","last_became_peered":"2024-09-18T10:09:01.983714+0000","last_unstale":"2024-09-18T10:09:03.987814+0000","last_undegraded":"2024-09-18T10:09:03.987814+0000","last_fullsized":"2024-09-18T10:09:03.987814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:44.111047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":101,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T10:09:02.980246+0000","last_change":"2024-09-18T10:09:00.850110+0000","last_active":"2024-09-18T10:09:02.980246+0000","last_peered":"2024-09-18T10:09:02.980246+0000","last_clean":"2024-09-18T10:09:02.980246+0000","last_became_active":"2024-09-18T10:09:00.849769+0000","last_became_peered":"2024-09-18T10:09:00.849769+0000","last_unstale":"2024-09-18T10:09:02.980246+0000","last_undegraded":"2024-09-18T10:09:02.980246+0000","last_fullsized":"2024-09-18T10:09:02.980246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:02.351581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":131072,"data_stored":38499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:09:07.864 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T10:09:07.864 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T10:09:07.864 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T10:09:07.864 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T10:09:07.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: get_monmap_and_config 2024-09-18T10:09:07.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: monmap: 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:07.963876+0000 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:07.963876+0000 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:07.965 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 Processor -- start 2024-09-18T10:09:07.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 -- start start 2024-09-18T10:09:07.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: init 2024-09-18T10:09:07.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:09:07.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b340a2520 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:07.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b3409e090 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:07.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b34135420 0x7f3b34137800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-a con 0x7f3b34135420 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:07.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3406c590 con 0x7f3b3409e090 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b340a28f0 con 0x7f3b340a2520 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3406b960 con 0x7f3b34135420 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b38b52700 10 monclient(hunting): authenticate will time out at 836.891091s 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b32ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3b3409e090 0x7f3b3406b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46578/0 (socket says 172.21.15.142:46578) 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b32ffd700 1 -- 172.21.15.142:0/3770252664 learned_addr learned my addr 172.21.15.142:0/3770252664 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 1 --2- 172.21.15.142:0/3770252664 >> v2:172.21.15.142:3300/0 conn(0x7f3b34135420 0x7f3b34137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): get_auth_request con 0x7f3b34135420 auth_method 0 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 1 -- 172.21.15.142:0/3770252664 >> v1:172.21.15.150:6789/0 conn(0x7f3b340a2520 legacy=0x7f3b3406ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 1 -- 172.21.15.142:0/3770252664 >> v1:172.21.15.142:6789/0 conn(0x7f3b3409e090 legacy=0x7f3b3406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 1 monclient: found mon.noname-a 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.960+0000 7f3b33fff700 1 -- 172.21.15.142:0/3770252664 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3409e490 con 0x7f3b34135420 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b33fff700 1 -- 172.21.15.142:0/3770252664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 msgr2=0x7f3b34137800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b33fff700 1 --2- 172.21.15.142:0/3770252664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b34137800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406a810 tx=0x7f3b2c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc37cf5ef743634f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 1 -- 172.21.15.142:0/3770252664 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c00e070 con 0x7f3b34135420 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: dump: 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.973 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 5 monclient: authenticate success, global_id 4320 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 1 -- 172.21.15.142:0/3770252664 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c012430 con 0x7f3b34135420 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:07.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 4 set_mon_vals no callback set 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 1 -- 172.21.15.142:0/3770252664 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0127d0 con 0x7f3b34135420 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b327fc700 10 monclient: dump: 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.975 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 10 monclient: shutdown 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 -- 172.21.15.142:0/3770252664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 msgr2=0x7f3b34137800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3770252664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b34137800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406a810 tx=0x7f3b2c009780 comp rx=0 tx=0).stop 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 -- 172.21.15.142:0/3770252664 shutdown_connections 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3770252664 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b34137800 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 -- 172.21.15.142:0/3770252664 >> 172.21.15.142:0/3770252664 conn(0x7f3b340715d0 msgr2=0x7f3b340a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 -- 172.21.15.142:0/3770252664 shutdown_connections 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.964+0000 7f3b38b52700 1 -- 172.21.15.142:0/3770252664 wait complete. 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient: build_initial_monmap 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient: monmap: 2024-09-18T10:09:07.976 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:07.977 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 Processor -- start 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 -- start start 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient: init 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3409e090 0x7f3b3409a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-b con 0x7f3b3409e090 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-c con 0x7f3b340a2520 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b3409cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): picked mon.noname-a con 0x7f3b34135420 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34119000 con 0x7f3b34135420 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3406bda0 con 0x7f3b3409e090 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3409afb0 con 0x7f3b340a2520 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b38b52700 10 monclient(hunting): authenticate will time out at 836.898376s 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b3409cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51400/0 (socket says 172.21.15.142:51400) 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 1 -- 172.21.15.142:0/3321834574 learned_addr learned my addr 172.21.15.142:0/3321834574 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): get_auth_request con 0x7f3b340a2520 auth_method 0 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): get_auth_request con 0x7f3b34135420 auth_method 0 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3409e090 0x7f3b3409a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): get_auth_request con 0x7f3b3409e090 auth_method 0 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 -- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 msgr2=0x7f3b3409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 -- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3409e090 msgr2=0x7f3b3409a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3409e090 0x7f3b3409a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:07.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 monclient: found mon.noname-a 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b33fff700 1 -- 172.21.15.142:0/3321834574 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c009a20 con 0x7f3b34135420 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.968+0000 7f3b32ffd700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b33fff700 1 -- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 msgr2=0x7f3b3409cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b33fff700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b3409cb40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c015000 tx=0x7f3b2c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c026430 con 0x7f3b34135420 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: dump: 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.986 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 5 monclient: authenticate success, global_id 4323 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c021070 con 0x7f3b34135420 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:07.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c0267e0 con 0x7f3b34135420 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _renew_subs 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b34118910 con 0x7f3b34135420 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b31ffb700 4 set_mon_vals no callback set 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 10 monclient: dump: 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:07.988 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _renew_subs 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b34118430 con 0x7f3b34135420 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3404c430 con 0x7f3b34135420 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.972+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3b2c023030 con 0x7f3b34135420 2024-09-18T10:09:07.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.976+0000 7f3b1b7fe700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 0x7f3b20037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.976+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4398+0+0 (secure 0 0 0) 0x7f3b2c04a420 con 0x7f3b34135420 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.976+0000 7f3b337fe700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 0x7f3b20037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.976+0000 7f3b337fe700 10 monclient: get_auth_request con 0x7f3b20035ac0 auth_method 0 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.976+0000 7f3b337fe700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 0x7f3b20037f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b24003a30 tx=0x7f3b24007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.984+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3b2c02f440 con 0x7f3b34135420 2024-09-18T10:09:07.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.984+0000 7f3b1b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:07.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:07.984+0000 7f3b1b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b38b52700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-18T10:09:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b38b52700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-18T10:09:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b38b52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3b3404c430 con 0x7f3b34135420 2024-09-18T10:09:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b1b7fe700 1 -- 172.21.15.142:0/3321834574 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3b2c01b3a0 con 0x7f3b34135420 2024-09-18T10:09:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b1b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-18T10:09:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.184+0000 7f3b1b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:08.192 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:08.193 INFO:teuthology.orchestra.run.smithi142.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T10:09:08.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 msgr2=0x7f3b20037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 0x7f3b20037f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b24003a30 tx=0x7f3b24007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 10 monclient: shutdown 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 msgr2=0x7f3b3409cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b3409cb40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c015000 tx=0x7f3b2c018000 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 shutdown_connections 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b20035ac0 0x7f3b20037f70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b34135420 0x7f3b3409cb40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b340a2520 0x7f3b3409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 --2- 172.21.15.142:0/3321834574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b3409e090 0x7f3b3409a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 >> 172.21.15.142:0/3321834574 conn(0x7f3b340715d0 msgr2=0x7f3b340a0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 shutdown_connections 2024-09-18T10:09:08.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.188+0000 7f3b38b52700 1 -- 172.21.15.142:0/3321834574 wait complete. 2024-09-18T10:09:08.205 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T10:09:08.205 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-18T10:09:08.218 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-18T10:09:08.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.312+0000 7ff676617700 10 monclient: get_monmap_and_config 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.312+0000 7ff676617700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.312+0000 7ff676617700 10 monclient: monmap: 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:08.319340+0000 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:08.319340+0000 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.320 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:08.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 Processor -- start 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 -- start start 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient: init 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff67009aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): picked mon.noname-b con 0x7ff670132570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff670134e90 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67009aeb0 con 0x7ff670134e90 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67006f6d0 con 0x7ff67009aae0 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700955a0 con 0x7ff670132570 2024-09-18T10:09:08.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient(hunting): authenticate will time out at 837.244772s 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51410/0 (socket says 172.21.15.142:51410) 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 -- 172.21.15.142:0/1797355631 learned_addr learned my addr 172.21.15.142:0/1797355631 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): get_auth_request con 0x7ff670132570 auth_method 0 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-09-18T10:09:08.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 -- 172.21.15.142:0/1797355631 >> v1:172.21.15.150:6789/0 conn(0x7ff67009aae0 legacy=0x7ff670096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 -- 172.21.15.142:0/1797355631 >> v1:172.21.15.142:6789/0 conn(0x7ff670134e90 legacy=0x7ff670137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 monclient: found mon.noname-b 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 -- 172.21.15.142:0/1797355631 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670095b60 con 0x7ff670132570 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 -- 172.21.15.142:0/1797355631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 msgr2=0x7ff670134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff674e14700 1 --2- 172.21.15.142:0/1797355631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6600099e0 tx=0x7ff660009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e3aa9de41e2567a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 1 -- 172.21.15.142:0/1797355631 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66000bb50 con 0x7ff670132570 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: dump: 2024-09-18T10:09:08.327 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 5 monclient: authenticate success, global_id 4316 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:08.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 1 -- 172.21.15.142:0/1797355631 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66000bcb0 con 0x7ff670132570 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff676617700 4 set_mon_vals no callback set 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 1 -- 172.21.15.142:0/1797355631 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff660011670 con 0x7ff670132570 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.316+0000 7ff667fff700 10 monclient: dump: 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.329 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: shutdown 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- 172.21.15.142:0/1797355631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 msgr2=0x7ff670134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 --2- 172.21.15.142:0/1797355631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6600099e0 tx=0x7ff660009250 comp rx=0 tx=0).stop 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- 172.21.15.142:0/1797355631 shutdown_connections 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 --2- 172.21.15.142:0/1797355631 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670132570 0x7ff670134950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- 172.21.15.142:0/1797355631 >> 172.21.15.142:0/1797355631 conn(0x7ff6700a1fe0 msgr2=0x7ff670070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- 172.21.15.142:0/1797355631 shutdown_connections 2024-09-18T10:09:08.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- 172.21.15.142:0/1797355631 wait complete. 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: monmap: 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 Processor -- start 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- start start 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: init 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): picked mon.noname-c con 0x7ff67009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670132570 0x7ff67006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): picked mon.noname-b con 0x7ff670132570 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 0x7ff67006d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): picked mon.noname-a con 0x7ff670134e90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67006f8d0 con 0x7ff670134e90 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff670127500 con 0x7ff670132570 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff670127220 con 0x7ff67009aae0 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff676617700 10 monclient(hunting): authenticate will time out at 837.250989s 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51414/0 (socket says 172.21.15.142:51414) 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 1 -- 172.21.15.142:0/2823160626 learned_addr learned my addr 172.21.15.142:0/2823160626 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 0x7ff67006d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): get_auth_request con 0x7ff67009aae0 auth_method 0 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): get_auth_request con 0x7ff670134e90 auth_method 0 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff674e14700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670132570 0x7ff67006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff674e14700 10 monclient(hunting): get_auth_request con 0x7ff670132570 auth_method 0 2024-09-18T10:09:08.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff674e14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff674e14700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff674e14700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 -- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 msgr2=0x7ff67006cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 -- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670132570 msgr2=0x7ff67006d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670132570 0x7ff67006d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 monclient: found mon.noname-a 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675e16700 1 -- 172.21.15.142:0/2823160626 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6600094f0 con 0x7ff670134e90 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.320+0000 7ff675615700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff675e16700 1 -- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 msgr2=0x7ff67006d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff675e16700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 0x7ff67006d6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c810 tx=0x7ff668002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668012840 con 0x7ff670134e90 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: dump: 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.339 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6680129a0 con 0x7ff670134e90 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 5 monclient: authenticate success, global_id 4326 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _renew_subs 2024-09-18T10:09:08.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670127ac0 con 0x7ff670134e90 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668012d60 con 0x7ff670134e90 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 10 monclient: dump: 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.341 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _renew_subs 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6677fe700 4 set_mon_vals no callback set 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff670126570 con 0x7ff670134e90 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff668011590 con 0x7ff670134e90 2024-09-18T10:09:08.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff670072a80 con 0x7ff670134e90 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 0x7ff654037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.324+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7ff668047f20 con 0x7ff670134e90 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.328+0000 7ff675615700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 0x7ff654037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.328+0000 7ff675615700 10 monclient: get_auth_request con 0x7ff654035ac0 auth_method 0 2024-09-18T10:09:08.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.328+0000 7ff675615700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 0x7ff654037f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff66c007fd0 tx=0x7ff66c007f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.332+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff668019910 con 0x7ff670134e90 2024-09-18T10:09:08.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.332+0000 7ff6657fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.332+0000 7ff6657fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:08.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.496+0000 7ff676617700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:08.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.496+0000 7ff676617700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:08.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.496+0000 7ff676617700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.496+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff670127640 con 0x7ff670134e90 2024-09-18T10:09:08.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff6657fa700 1 -- 172.21.15.142:0/2823160626 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+11409 (secure 0 0 0) 0x7ff668016020 con 0x7ff670134e90 2024-09-18T10:09:08.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff6657fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:09:08.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff6657fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:08.505 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:08.505 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":17,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:09:08.096885+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:08:51.753346+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T10:09:00.830395+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.358800+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.344702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 msgr2=0x7ff654037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 0x7ff654037f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff66c007fd0 tx=0x7ff66c007f60 comp rx=0 tx=0).stop 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 10 monclient: shutdown 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 msgr2=0x7ff67006d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 0x7ff67006d6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c810 tx=0x7ff668002620 comp rx=0 tx=0).stop 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 shutdown_connections 2024-09-18T10:09:08.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff654035ac0 0x7ff654037f70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670134e90 0x7ff67006d6e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670132570 0x7ff67006d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 --2- 172.21.15.142:0/2823160626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff67009aae0 0x7ff67006cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 >> 172.21.15.142:0/2823160626 conn(0x7ff6700a1fe0 msgr2=0x7ff6701342a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 shutdown_connections 2024-09-18T10:09:08.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.500+0000 7ff676617700 1 -- 172.21.15.142:0/2823160626 wait complete. 2024-09-18T10:09:08.518 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-18T10:08:58.836373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-18T10:09:00.830395+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-18T10:09:08.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-18T10:09:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:09:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: monmap: 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:08.627921+0000 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:08.627921+0000 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:08.629 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 Processor -- start 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 -- start start 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: init 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab6009c4b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fab60131560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 0x7fab60136260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fab60133e80 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6009c880 con 0x7fab6009c4b0 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6009b100 con 0x7fab60133e80 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6009b870 con 0x7fab60131560 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5f59e700 10 monclient(hunting): authenticate will time out at 837.553769s 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51418/0 (socket says 172.21.15.142:51418) 2024-09-18T10:09:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5e59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fab6009c4b0 0x7fab60099af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46584/0 (socket says 172.21.15.142:46584) 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 -- 172.21.15.142:0/1786551023 learned_addr learned my addr 172.21.15.142:0/1786551023 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): get_auth_request con 0x7fab60131560 auth_method 0 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5ed9d700 1 --2- 172.21.15.142:0/1786551023 >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 0x7fab60136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 -- 172.21.15.142:0/1786551023 >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 msgr2=0x7fab60136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 --2- 172.21.15.142:0/1786551023 >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 0x7fab60136260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 -- 172.21.15.142:0/1786551023 >> v1:172.21.15.142:6789/0 conn(0x7fab6009c4b0 legacy=0x7fab60099af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 monclient: found mon.noname-b 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5dd9b700 1 -- 172.21.15.142:0/1786551023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6009e950 con 0x7fab60131560 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5ed9d700 10 monclient: get_auth_request con 0x7fab60133e80 auth_method 0 2024-09-18T10:09:08.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.624+0000 7fab5ed9d700 1 --2- 172.21.15.142:0/1786551023 >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 0x7fab60136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 -- 172.21.15.142:0/1786551023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 msgr2=0x7fab60133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 --2- 172.21.15.142:0/1786551023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab480099b0 tx=0x7fab48009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcaeb8e0331fc540 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 1 -- 172.21.15.142:0/1786551023 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800bb50 con 0x7fab60131560 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: dump: 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 5 monclient: authenticate success, global_id 4322 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 1 -- 172.21.15.142:0/1786551023 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4800bcb0 con 0x7fab60131560 2024-09-18T10:09:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 4 set_mon_vals no callback set 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 1 -- 172.21.15.142:0/1786551023 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48011670 con 0x7fab60131560 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5d59a700 10 monclient: dump: 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: shutdown 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- 172.21.15.142:0/1786551023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 msgr2=0x7fab60133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- 172.21.15.142:0/1786551023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fab480099b0 tx=0x7fab48009250 comp rx=0 tx=0).stop 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- 172.21.15.142:0/1786551023 shutdown_connections 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- 172.21.15.142:0/1786551023 >> v2:172.21.15.150:3300/0 conn(0x7fab60133e80 0x7fab60136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- 172.21.15.142:0/1786551023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60131560 0x7fab60133940 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- 172.21.15.142:0/1786551023 >> 172.21.15.142:0/1786551023 conn(0x7fab600a2400 msgr2=0x7fab600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- 172.21.15.142:0/1786551023 shutdown_connections 2024-09-18T10:09:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- 172.21.15.142:0/1786551023 wait complete. 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: monmap: 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 Processor -- start 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- start start 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: init 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009c4b0 0x7fab60145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fab6009c4b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 0x7fab60146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fab60133e80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fab6006ba50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006d0a0 con 0x7fab6006ba50 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab600992e0 con 0x7fab6009c4b0 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006ec80 con 0x7fab60133e80 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5f59e700 10 monclient(hunting): authenticate will time out at 837.558888s 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48744/0 (socket says 172.21.15.142:48744) 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 -- 172.21.15.142:0/3856296507 learned_addr learned my addr 172.21.15.142:0/3856296507 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 10 monclient(hunting): get_auth_request con 0x7fab6006ba50 auth_method 0 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 0x7fab60146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): get_auth_request con 0x7fab60133e80 auth_method 0 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009c4b0 0x7fab60145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 msgr2=0x7fab6006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 msgr2=0x7fab6006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): get_auth_request con 0x7fab6009c4b0 auth_method 0 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5ed9d700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009c4b0 msgr2=0x7fab60145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009c4b0 0x7fab60145ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 msgr2=0x7fab6006be90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 monclient: found mon.noname-c 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.628+0000 7fab5dd9b700 1 -- 172.21.15.142:0/3856296507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480094f0 con 0x7fab60133e80 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5dd9b700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 msgr2=0x7fab60146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5dd9b700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 0x7fab60146220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab48015000 tx=0x7fab48016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48006a90 con 0x7fab60133e80 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: dump: 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 5 monclient: authenticate success, global_id 4325 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4801b470 con 0x7fab60133e80 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 10 monclient: _renew_subs 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4801b830 con 0x7fab60133e80 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6006daa0 con 0x7fab60133e80 2024-09-18T10:09:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab57fff700 4 set_mon_vals no callback set 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab567fc700 10 monclient: dump: 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 10 monclient: _renew_subs 2024-09-18T10:09:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab60096070 con 0x7fab60133e80 2024-09-18T10:09:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.632+0000 7fab5f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab5f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60072a80 con 0x7fab60133e80 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fab4801bc20 con 0x7fab60133e80 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab567fc700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 0x7fab44037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fab480496b0 con 0x7fab60133e80 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab5e59c700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 0x7fab44037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.636+0000 7fab5e59c700 10 monclient: get_auth_request con 0x7fab44035ac0 auth_method 0 2024-09-18T10:09:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.640+0000 7fab5e59c700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 0x7fab44037f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab50006e80 tx=0x7fab500053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.644+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fab48023950 con 0x7fab60133e80 2024-09-18T10:09:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.644+0000 7fab567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.644+0000 7fab567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.804+0000 7fab5f59e700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T10:09:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.804+0000 7fab5f59e700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.804+0000 7fab5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.804+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab6006da40 con 0x7fab60133e80 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.804+0000 7fab567fc700 1 -- 172.21.15.142:0/3856296507 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab4801a3d0 con 0x7fab60133e80 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:08.812 INFO:teuthology.orchestra.run.smithi142.stdout:pg_num: 1 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 msgr2=0x7fab44037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 0x7fab44037f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab50006e80 tx=0x7fab500053c0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 10 monclient: shutdown 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 msgr2=0x7fab60146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 0x7fab60146220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab48015000 tx=0x7fab48016000 comp rx=0 tx=0).stop 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 shutdown_connections 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab44035ac0 0x7fab44037f70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab6006ba50 0x7fab6006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab60133e80 0x7fab60146220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 --2- 172.21.15.142:0/3856296507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009c4b0 0x7fab60145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 >> 172.21.15.142:0/3856296507 conn(0x7fab600a2400 msgr2=0x7fab60070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 shutdown_connections 2024-09-18T10:09:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.808+0000 7fab5f59e700 1 -- 172.21.15.142:0/3856296507 wait complete. 2024-09-18T10:09:08.826 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T10:09:08.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:09:08.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: monmap: 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:08.937806+0000 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:08.937806+0000 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:08.939 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 Processor -- start 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 -- start start 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: init 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d8098010 0x7fb0d8096150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb0d8098010 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0d8096690 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb0d809c5e0 0x7fb0d8137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb0d809c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d806fae0 con 0x7fb0d8096690 2024-09-18T10:09:08.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d8098470 con 0x7fb0d8098010 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d809b500 con 0x7fb0d809c5e0 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d7fff700 10 monclient(hunting): authenticate will time out at 837.863221s 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d8098010 0x7fb0d8096150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d8098010 0x7fb0d8096150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48754/0 (socket says 172.21.15.142:48754) 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/150365522 learned_addr learned my addr 172.21.15.142:0/150365522 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb0d8098010 auth_method 0 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/150365522 >> v2:172.21.15.150:3300/0 conn(0x7fb0d809c5e0 0x7fb0d8137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): get_auth_request con 0x7fb0d809c5e0 auth_method 0 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.932+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_done global_id 4332 payload 274 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/150365522 >> v2:172.21.15.150:3300/0 conn(0x7fb0d809c5e0 msgr2=0x7fb0d8137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/150365522 >> v2:172.21.15.150:3300/0 conn(0x7fb0d809c5e0 0x7fb0d8137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/150365522 >> v1:172.21.15.150:6789/0 conn(0x7fb0d8096690 legacy=0x7fb0d809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/150365522 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d809b740 con 0x7fb0d8098010 2024-09-18T10:09:08.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/150365522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d8098010 msgr2=0x7fb0d8096150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/150365522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d8098010 0x7fb0d8096150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80676a0 tx=0x7fb0d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a5b8ee3d8aea2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 1 -- 172.21.15.142:0/150365522 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000e070 con 0x7fb0d8098010 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: dump: 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.952 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 5 monclient: authenticate success, global_id 4332 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 1 -- 172.21.15.142:0/150365522 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d000bc30 con 0x7fb0d8098010 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 4 set_mon_vals no callback set 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 1 -- 172.21.15.142:0/150365522 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00092e0 con 0x7fb0d8098010 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:08.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d5ffb700 10 monclient: dump: 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 10 monclient: shutdown 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:08.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 -- 172.21.15.142:0/150365522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d8098010 msgr2=0x7fb0d8096150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/150365522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d8098010 0x7fb0d8096150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80676a0 tx=0x7fb0d0009840 comp rx=0 tx=0).stop 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 -- 172.21.15.142:0/150365522 shutdown_connections 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/150365522 >> v2:172.21.15.150:3300/0 conn(0x7fb0d809c5e0 0x7fb0d8137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/150365522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d8098010 0x7fb0d8096150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 -- 172.21.15.142:0/150365522 >> 172.21.15.142:0/150365522 conn(0x7fb0d8070ef0 msgr2=0x7fb0d80712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 -- 172.21.15.142:0/150365522 shutdown_connections 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.936+0000 7fb0d7fff700 1 -- 172.21.15.142:0/150365522 wait complete. 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient: monmap: 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:08.955 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 Processor -- start 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 -- start start 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient: init 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0d8141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb0d8096690 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0d8141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb0d8098010 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb0d809c5e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d80919f0 con 0x7fb0d809c5e0 2024-09-18T10:09:08.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d809d270 con 0x7fb0d8098010 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d8090410 con 0x7fb0d8096690 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d7fff700 10 monclient(hunting): authenticate will time out at 837.869583s 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0d8141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0d8141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51440/0 (socket says 172.21.15.142:51440) 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/304472680 learned_addr learned my addr 172.21.15.142:0/304472680 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb0d8096690 auth_method 0 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0d8141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): get_auth_request con 0x7fb0d809c5e0 auth_method 0 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): get_auth_request con 0x7fb0d8098010 auth_method 0 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 msgr2=0x7fb0d8141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0d8141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 msgr2=0x7fb0d8141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0d8141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 monclient: found mon.noname-a 2024-09-18T10:09:08.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.940+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0009ae0 con 0x7fb0d809c5e0 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 msgr2=0x7fb0d808de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e950 tx=0x7fb0c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 msgr2=0x7fb0d808de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 msgr2=0x7fb0d808de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e950 tx=0x7fb0c80091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800e950 tx=0x7fb0c80091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 msgr2=0x7fb0d808de80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d809c5e0 0x7fb0d808de80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 0x7fb0b8005930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): picked mon.noname-a con 0x7fb0b80034f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0b80084a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): picked mon.noname-c con 0x7fb0d8096690 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0b800aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): picked mon.noname-b con 0x7fb0d8098010 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b800b860 con 0x7fb0b80034f0 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b800b9a0 con 0x7fb0d8098010 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b800bb60 con 0x7fb0d8096690 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0ceffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 0x7fb0b8005930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb0b80034f0 auth_method 0 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d67fc700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0b80084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d67fc700 10 monclient(hunting): get_auth_request con 0x7fb0d8096690 auth_method 0 2024-09-18T10:09:08.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d67fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0b800aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient(hunting): handle_auth_done global_id 4335 payload 293 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 msgr2=0x7fb0b80084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0b80084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 msgr2=0x7fb0b800aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0b800aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b800bde0 con 0x7fb0b80034f0 2024-09-18T10:09:08.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 10 monclient: get_auth_request con 0x7fb0d8098010 auth_method 0 2024-09-18T10:09:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d77fe700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0b800aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 msgr2=0x7fb0b8005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.944+0000 7fb0d6ffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 0x7fb0b8005930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000b6c0 tx=0x7fb0d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00092e0 con 0x7fb0b80034f0 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: dump: 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0000f40 con 0x7fb0b80034f0 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 5 monclient: authenticate success, global_id 4335 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _renew_subs 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d808fd50 con 0x7fb0b80034f0 2024-09-18T10:09:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0012640 con 0x7fb0b80034f0 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d57fa700 4 set_mon_vals no callback set 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 10 monclient: dump: 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _renew_subs 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d808f840 con 0x7fb0b80034f0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb0d0024070 con 0x7fb0b80034f0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d8072a80 con 0x7fb0b80034f0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 0x7fb0b803ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fb0d00490e0 con 0x7fb0b80034f0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d67fc700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 0x7fb0b803ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.948+0000 7fb0d67fc700 10 monclient: get_auth_request con 0x7fb0d809c5e0 auth_method 0 2024-09-18T10:09:08.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.952+0000 7fb0d67fc700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 0x7fb0b803ffc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0007c80 tx=0x7fb0c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:08.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.956+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb0d001e400 con 0x7fb0b80034f0 2024-09-18T10:09:08.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.956+0000 7fb0ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:08.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:08.956+0000 7fb0ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.116+0000 7fb0d7fff700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T10:09:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.116+0000 7fb0d7fff700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T10:09:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.116+0000 7fb0d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.116+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb0d80916e0 con 0x7fb0b80034f0 2024-09-18T10:09:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0ceffd700 1 -- 172.21.15.142:0/304472680 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb0d002db30 con 0x7fb0b80034f0 2024-09-18T10:09:09.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0ceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-18T10:09:09.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0ceffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:09.125 INFO:teuthology.orchestra.run.smithi142.stdout:pg_num: 8 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 msgr2=0x7fb0b803ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 0x7fb0b803ffc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0007c80 tx=0x7fb0c0009040 comp rx=0 tx=0).stop 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 10 monclient: shutdown 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 msgr2=0x7fb0b8005930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 0x7fb0b8005930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb0d000b6c0 tx=0x7fb0d0017000 comp rx=0 tx=0).stop 2024-09-18T10:09:09.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 shutdown_connections 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0d809c5e0 0x7fb0b803ffc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d8098010 0x7fb0b800aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d8096690 0x7fb0b80084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 --2- 172.21.15.142:0/304472680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0b80034f0 0x7fb0b8005930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 >> 172.21.15.142:0/304472680 conn(0x7fb0d8070ef0 msgr2=0x7fb0d8087fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 shutdown_connections 2024-09-18T10:09:09.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.120+0000 7fb0d7fff700 1 -- 172.21.15.142:0/304472680 wait complete. 2024-09-18T10:09:09.142 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-18T10:09:09.142 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:09:09.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: get_monmap_and_config 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: monmap: 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:09.240957+0000 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:09.240957+0000 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:09.242 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:09.243 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:09.243 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 Processor -- start 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 -- start start 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: init 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a80a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-b con 0x7ff0a80a6260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0a809e280 0x7ff0a80a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-c con 0x7ff0a809e280 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:09.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0a80990b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a809d8a0 con 0x7ff0a80990b0 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a809df60 con 0x7ff0a809e280 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a809fb20 con 0x7ff0a80a6260 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0aed12700 10 monclient(hunting): authenticate will time out at 838.166501s 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a80a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0ad2af700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff0a80990b0 0x7ff0a8138a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46588/0 (socket says 172.21.15.142:46588) 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0ad2af700 1 -- 172.21.15.142:0/3543871699 learned_addr learned my addr 172.21.15.142:0/3543871699 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): get_auth_request con 0x7ff0a80a6260 auth_method 0 2024-09-18T10:09:09.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 1 --2- 172.21.15.142:0/3543871699 >> v2:172.21.15.150:3300/0 conn(0x7ff0a809e280 0x7ff0a80a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): get_auth_request con 0x7ff0a809e280 auth_method 0 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-18T10:09:09.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 -- 172.21.15.142:0/3543871699 >> v2:172.21.15.150:3300/0 conn(0x7ff0a809e280 msgr2=0x7ff0a80a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 --2- 172.21.15.142:0/3543871699 >> v2:172.21.15.150:3300/0 conn(0x7ff0a809e280 0x7ff0a80a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 -- 172.21.15.142:0/3543871699 >> v1:172.21.15.142:6789/0 conn(0x7ff0a80990b0 legacy=0x7ff0a8138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 monclient: found mon.noname-b 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:09.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.236+0000 7ff0acaae700 1 -- 172.21.15.142:0/3543871699 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80a6630 con 0x7ff0a80a6260 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0acaae700 1 -- 172.21.15.142:0/3543871699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 msgr2=0x7ff0a80a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0acaae700 1 --2- 172.21.15.142:0/3543871699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a80a0270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098e0 tx=0x7ff09c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36491bd49a6083e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 1 -- 172.21.15.142:0/3543871699 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00b850 con 0x7ff0a80a6260 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: dump: 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.252 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 5 monclient: authenticate success, global_id 4331 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 1 -- 172.21.15.142:0/3543871699 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c00b9b0 con 0x7ff0a80a6260 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 4 set_mon_vals no callback set 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 1 -- 172.21.15.142:0/3543871699 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c00bd50 con 0x7ff0a80a6260 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0a77fe700 10 monclient: dump: 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 10 monclient: shutdown 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 -- 172.21.15.142:0/3543871699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 msgr2=0x7ff0a80a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 --2- 172.21.15.142:0/3543871699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a80a0270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098e0 tx=0x7ff09c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 -- 172.21.15.142:0/3543871699 shutdown_connections 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 --2- 172.21.15.142:0/3543871699 >> v2:172.21.15.150:3300/0 conn(0x7ff0a809e280 0x7ff0a80a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 --2- 172.21.15.142:0/3543871699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a80a0270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 -- 172.21.15.142:0/3543871699 >> 172.21.15.142:0/3543871699 conn(0x7ff0a8071680 msgr2=0x7ff0a80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 -- 172.21.15.142:0/3543871699 shutdown_connections 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.240+0000 7ff0aed12700 1 -- 172.21.15.142:0/3543871699 wait complete. 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient: monmap: 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:09.255 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 Processor -- start 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 -- start start 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient: init 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-a con 0x7ff0a80990b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:09.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a809e280 0x7ff0a808d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-b con 0x7ff0a809e280 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a808da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): picked mon.noname-c con 0x7ff0a80a6260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a80a4110 con 0x7ff0a80990b0 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a808f4a0 con 0x7ff0a809e280 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a808ede0 con 0x7ff0a80a6260 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0aed12700 10 monclient(hunting): authenticate will time out at 838.173274s 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a808da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a808da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51474/0 (socket says 172.21.15.142:51474) 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48792/0 (socket says 172.21.15.142:48792) 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 1 -- 172.21.15.142:0/1875807524 learned_addr learned my addr 172.21.15.142:0/1875807524 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): get_auth_request con 0x7ff0a80a6260 auth_method 0 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): get_auth_request con 0x7ff0a80990b0 auth_method 0 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a809e280 0x7ff0a808d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0ad2af700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): get_auth_request con 0x7ff0a809e280 auth_method 0 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 -- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 msgr2=0x7ff0a808da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a808da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 -- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a809e280 msgr2=0x7ff0a808d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a809e280 0x7ff0a808d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 monclient: found mon.noname-a 2024-09-18T10:09:09.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.244+0000 7ff0acaae700 1 -- 172.21.15.142:0/1875807524 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c009420 con 0x7ff0a80990b0 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0acaae700 1 -- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 msgr2=0x7ff0a808cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0acaae700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8066280 tx=0x7ff09c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c011d50 con 0x7ff0a80990b0 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: dump: 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.263 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 5 monclient: authenticate success, global_id 4338 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _renew_subs 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c01c460 con 0x7ff0a80990b0 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a8090400 con 0x7ff0a80990b0 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c01c820 con 0x7ff0a80990b0 2024-09-18T10:09:09.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a6ffd700 4 set_mon_vals no callback set 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 10 monclient: dump: 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.265 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _renew_subs 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a8091690 con 0x7ff0a80990b0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a8090ac0 con 0x7ff0a80990b0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff09c01cc10 con 0x7ff0a80990b0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 0x7ff094037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7ff09c04a420 con 0x7ff0a80990b0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a7fff700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 0x7ff094037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.248+0000 7ff0a7fff700 10 monclient: get_auth_request con 0x7ff094035ac0 auth_method 0 2024-09-18T10:09:09.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.252+0000 7ff0a7fff700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 0x7ff094037f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff098009fd0 tx=0x7ff09800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.256+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff09c010980 con 0x7ff0a80990b0 2024-09-18T10:09:09.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.256+0000 7ff0a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.256+0000 7ff0a4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:09.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.416+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0a8090ac0 con 0x7ff094035ac0 2024-09-18T10:09:09.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0a4ff9700 1 -- 172.21.15.142:0/1875807524 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41009 (secure 0 0 0) 0x7ff0a8090ac0 con 0x7ff094035ac0 2024-09-18T10:09:09.425 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:09.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 msgr2=0x7ff094037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 0x7ff094037f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff098009fd0 tx=0x7ff09800b040 comp rx=0 tx=0).stop 2024-09-18T10:09:09.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 10 monclient: shutdown 2024-09-18T10:09:09.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 msgr2=0x7ff0a808cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8066280 tx=0x7ff09c016000 comp rx=0 tx=0).stop 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 shutdown_connections 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff094035ac0 0x7ff094037f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0a80a6260 0x7ff0a808da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0a809e280 0x7ff0a808d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 --2- 172.21.15.142:0/1875807524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0a80990b0 0x7ff0a808cfc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 >> 172.21.15.142:0/1875807524 conn(0x7ff0a8071680 msgr2=0x7ff0a809e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 shutdown_connections 2024-09-18T10:09:09.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.420+0000 7ff0aed12700 1 -- 172.21.15.142:0/1875807524 wait complete. 2024-09-18T10:09:09.429 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:09:09.439 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T10:09:08.764991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786356,"kb_used_data":1936,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444684,"statfs":{"total":579820584960,"available":577991356416,"internally_reserved":0,"allocated":1982464,"data_stored":1338028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982814+0000","last_change":"2024-09-18T10:09:03.983242+0000","last_active":"2024-09-18T10:09:03.982814+0000","last_peered":"2024-09-18T10:09:03.982814+0000","last_clean":"2024-09-18T10:09:03.982814+0000","last_became_active":"2024-09-18T10:09:02.313264+0000","last_became_peered":"2024-09-18T10:09:02.313264+0000","last_unstale":"2024-09-18T10:09:03.982814+0000","last_undegraded":"2024-09-18T10:09:03.982814+0000","last_fullsized":"2024-09-18T10:09:03.982814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:35:49.571437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980698+0000","last_change":"2024-09-18T10:09:03.981143+0000","last_active":"2024-09-18T10:09:03.980698+0000","last_peered":"2024-09-18T10:09:03.980698+0000","last_clean":"2024-09-18T10:09:03.980698+0000","last_became_active":"2024-09-18T10:09:01.977398+0000","last_became_peered":"2024-09-18T10:09:01.977398+0000","last_unstale":"2024-09-18T10:09:03.980698+0000","last_undegraded":"2024-09-18T10:09:03.980698+0000","last_fullsized":"2024-09-18T10:09:03.980698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:55:31.629074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982451+0000","last_change":"2024-09-18T10:09:03.982833+0000","last_active":"2024-09-18T10:09:03.982451+0000","last_peered":"2024-09-18T10:09:03.982451+0000","last_clean":"2024-09-18T10:09:03.982451+0000","last_became_active":"2024-09-18T10:09:01.975781+0000","last_became_peered":"2024-09-18T10:09:01.975781+0000","last_unstale":"2024-09-18T10:09:03.982451+0000","last_undegraded":"2024-09-18T10:09:03.982451+0000","last_fullsized":"2024-09-18T10:09:03.982451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:07:07.146280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980775+0000","last_change":"2024-09-18T10:09:03.981141+0000","last_active":"2024-09-18T10:09:03.980775+0000","last_peered":"2024-09-18T10:09:03.980775+0000","last_clean":"2024-09-18T10:09:03.980775+0000","last_became_active":"2024-09-18T10:09:01.982185+0000","last_became_peered":"2024-09-18T10:09:01.982185+0000","last_unstale":"2024-09-18T10:09:03.980775+0000","last_undegraded":"2024-09-18T10:09:03.980775+0000","last_fullsized":"2024-09-18T10:09:03.980775+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:12:33.081514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.988120+0000","last_change":"2024-09-18T10:09:03.988120+0000","last_active":"2024-09-18T10:09:03.988120+0000","last_peered":"2024-09-18T10:09:03.988120+0000","last_clean":"2024-09-18T10:09:03.988120+0000","last_became_active":"2024-09-18T10:09:01.980916+0000","last_became_peered":"2024-09-18T10:09:01.980916+0000","last_unstale":"2024-09-18T10:09:03.988120+0000","last_undegraded":"2024-09-18T10:09:03.988120+0000","last_fullsized":"2024-09-18T10:09:03.988120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:51.358013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:04.292190+0000","last_change":"2024-09-18T10:09:04.292594+0000","last_active":"2024-09-18T10:09:04.292190+0000","last_peered":"2024-09-18T10:09:04.292190+0000","last_clean":"2024-09-18T10:09:04.292190+0000","last_became_active":"2024-09-18T10:09:01.985425+0000","last_became_peered":"2024-09-18T10:09:01.985425+0000","last_unstale":"2024-09-18T10:09:04.292190+0000","last_undegraded":"2024-09-18T10:09:04.292190+0000","last_fullsized":"2024-09-18T10:09:04.292190+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:20.190675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982341+0000","last_change":"2024-09-18T10:09:03.982693+0000","last_active":"2024-09-18T10:09:03.982341+0000","last_peered":"2024-09-18T10:09:03.982341+0000","last_clean":"2024-09-18T10:09:03.982341+0000","last_became_active":"2024-09-18T10:09:01.976391+0000","last_became_peered":"2024-09-18T10:09:01.976391+0000","last_unstale":"2024-09-18T10:09:03.982341+0000","last_undegraded":"2024-09-18T10:09:03.982341+0000","last_fullsized":"2024-09-18T10:09:03.982341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:46:46.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.987814+0000","last_change":"2024-09-18T10:09:03.988400+0000","last_active":"2024-09-18T10:09:03.987814+0000","last_peered":"2024-09-18T10:09:03.987814+0000","last_clean":"2024-09-18T10:09:03.987814+0000","last_became_active":"2024-09-18T10:09:01.983714+0000","last_became_peered":"2024-09-18T10:09:01.983714+0000","last_unstale":"2024-09-18T10:09:03.987814+0000","last_undegraded":"2024-09-18T10:09:03.987814+0000","last_fullsized":"2024-09-18T10:09:03.987814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:44.111047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":101,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T10:09:02.980246+0000","last_change":"2024-09-18T10:09:00.850110+0000","last_active":"2024-09-18T10:09:02.980246+0000","last_peered":"2024-09-18T10:09:02.980246+0000","last_clean":"2024-09-18T10:09:02.980246+0000","last_became_active":"2024-09-18T10:09:00.849769+0000","last_became_peered":"2024-09-18T10:09:00.849769+0000","last_unstale":"2024-09-18T10:09:02.980246+0000","last_undegraded":"2024-09-18T10:09:02.980246+0000","last_fullsized":"2024-09-18T10:09:02.980246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:02.351581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":131072,"data_stored":38499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:09:09.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.536+0000 7f3dd7370700 10 monclient: get_monmap_and_config 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.536+0000 7f3dd7370700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.536+0000 7f3dd7370700 10 monclient: monmap: 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:09.543161+0000 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:09.543161+0000 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:09.544 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:09.545 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:09.545 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:09.545 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:09.545 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 Processor -- start 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 -- start start 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient: init 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-b con 0x7f3dd00a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 0x7f3dd009f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-c con 0x7f3dd012f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3dd00a02d0 0x7f3dd01320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-a con 0x7f3dd00a02d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:09.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd009d990 con 0x7f3dd00a02d0 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd009dc70 con 0x7f3dd012f810 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00a20c0 con 0x7f3dd00a1cf0 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd7370700 10 monclient(hunting): authenticate will time out at 838.469287s 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3dd00a02d0 0x7f3dd01320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51484/0 (socket says 172.21.15.142:51484) 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 1 -- 172.21.15.142:0/2794022371 learned_addr learned my addr 172.21.15.142:0/2794022371 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:09.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): get_auth_request con 0x7f3dd00a02d0 auth_method 0 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): get_auth_request con 0x7f3dd00a1cf0 auth_method 0 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 1 --2- 172.21.15.142:0/2794022371 >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 0x7f3dd009f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): get_auth_request con 0x7f3dd012f810 auth_method 0 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2024-09-18T10:09:09.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 msgr2=0x7f3dd012f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/2794022371 >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 msgr2=0x7f3dd009f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/2794022371 >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 0x7f3dd009f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 monclient: found mon.noname-a 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/2794022371 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd009f9c0 con 0x7f3dd00a02d0 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd5b6d700 1 --2- 172.21.15.142:0/2794022371 >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 0x7f3dd009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:09.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.540+0000 7f3dd636e700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 msgr2=0x7f3dd01320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd01320e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00a6930 tx=0x7f3dc8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e7e6b30e19af84 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 1 -- 172.21.15.142:0/2794022371 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc800e070 con 0x7f3dd00a02d0 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: dump: 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 1 -- 172.21.15.142:0/2794022371 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc800bcc0 con 0x7f3dd00a02d0 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 5 monclient: authenticate success, global_id 4341 2024-09-18T10:09:09.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 1 -- 172.21.15.142:0/2794022371 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8012680 con 0x7f3dd00a02d0 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd536c700 10 monclient: dump: 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.558 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 4 set_mon_vals no callback set 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 10 monclient: shutdown 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 -- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 msgr2=0x7f3dd01320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd01320e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00a6930 tx=0x7f3dc8009780 comp rx=0 tx=0).stop 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 -- 172.21.15.142:0/2794022371 shutdown_connections 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd01320e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 --2- 172.21.15.142:0/2794022371 >> v2:172.21.15.150:3300/0 conn(0x7f3dd012f810 0x7f3dd009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 --2- 172.21.15.142:0/2794022371 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd00a1cf0 0x7f3dd012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 -- 172.21.15.142:0/2794022371 >> 172.21.15.142:0/2794022371 conn(0x7f3dd0071820 msgr2=0x7f3dd00a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 -- 172.21.15.142:0/2794022371 shutdown_connections 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 1 -- 172.21.15.142:0/2794022371 wait complete. 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.544+0000 7f3dd7370700 10 monclient: monmap: 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:09.560 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 Processor -- start 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 -- start start 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient: init 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-a con 0x7f3dd00a02d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd00a1cf0 0x7f3dd009ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-b con 0x7f3dd00a1cf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): picked mon.noname-c con 0x7f3dd012f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd009cdc0 con 0x7f3dd00a02d0 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00a5110 con 0x7f3dd00a1cf0 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd006c6b0 con 0x7f3dd012f810 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd7370700 10 monclient(hunting): authenticate will time out at 838.477807s 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48814/0 (socket says 172.21.15.142:48814) 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51494/0 (socket says 172.21.15.142:51494) 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 1 -- 172.21.15.142:0/929246944 learned_addr learned my addr 172.21.15.142:0/929246944 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): get_auth_request con 0x7f3dd00a02d0 auth_method 0 2024-09-18T10:09:09.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): get_auth_request con 0x7f3dd012f810 auth_method 0 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd00a1cf0 0x7f3dd009ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): get_auth_request con 0x7f3dd00a1cf0 auth_method 0 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd5b6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd00a1cf0 msgr2=0x7f3dd009ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd00a1cf0 0x7f3dd009ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 msgr2=0x7f3dd006f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 monclient: found mon.noname-c 2024-09-18T10:09:09.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:09.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/929246944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0094f0 con 0x7f3dd012f810 2024-09-18T10:09:09.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.548+0000 7f3dd636e700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd6b6f700 1 -- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 msgr2=0x7f3dd009b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd6b6f700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00677a0 tx=0x7f3dc8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8012e90 con 0x7f3dd012f810 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: dump: 2024-09-18T10:09:09.567 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc8027470 con 0x7f3dd012f810 2024-09-18T10:09:09.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 5 monclient: authenticate success, global_id 4340 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _renew_subs 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8027810 con 0x7f3dd012f810 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd4b6b700 4 set_mon_vals no callback set 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc0009420 con 0x7f3dd012f810 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 10 monclient: dump: 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _renew_subs 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc8009a20 con 0x7f3dd012f810 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3dc8024070 con 0x7f3dd012f810 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd0072a80 con 0x7f3dd012f810 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 0x7f3db4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3dc804a7a0 con 0x7f3dd012f810 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd636e700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 0x7f3db4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.552+0000 7f3dd636e700 10 monclient: get_auth_request con 0x7f3db4035ac0 auth_method 0 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.556+0000 7f3dd636e700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 0x7f3db4037f70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006e80 tx=0x7f3dcc009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.560+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3dc8011970 con 0x7f3dd012f810 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.560+0000 7f3dc67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.560+0000 7f3dc67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:09.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.720+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3dd006c8f0 con 0x7f3db4035ac0 2024-09-18T10:09:09.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.724+0000 7f3dc67fc700 1 -- 172.21.15.142:0/929246944 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41009 (secure 0 0 0) 0x7f3dcc014440 con 0x7f3db4035ac0 2024-09-18T10:09:09.731 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:09:09.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 msgr2=0x7f3db4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 0x7f3db4037f70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc006e80 tx=0x7f3dcc009250 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 10 monclient: shutdown 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 msgr2=0x7f3dd009b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00677a0 tx=0x7f3dc8019000 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 shutdown_connections 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3db4035ac0 0x7f3db4037f70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3dd012f810 0x7f3dd009b1d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd00a1cf0 0x7f3dd009ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 --2- 172.21.15.142:0/929246944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3dd00a02d0 0x7f3dd006f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 >> 172.21.15.142:0/929246944 conn(0x7f3dd0071820 msgr2=0x7f3dd009e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 shutdown_connections 2024-09-18T10:09:09.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.728+0000 7f3dd7370700 1 -- 172.21.15.142:0/929246944 wait complete. 2024-09-18T10:09:09.735 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:09:09.746 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T10:09:08.764991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1786356,"kb_used_data":1936,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564444684,"statfs":{"total":579820584960,"available":577991356416,"internally_reserved":0,"allocated":1982464,"data_stored":1338028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.785616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982814+0000","last_change":"2024-09-18T10:09:03.983242+0000","last_active":"2024-09-18T10:09:03.982814+0000","last_peered":"2024-09-18T10:09:03.982814+0000","last_clean":"2024-09-18T10:09:03.982814+0000","last_became_active":"2024-09-18T10:09:02.313264+0000","last_became_peered":"2024-09-18T10:09:02.313264+0000","last_unstale":"2024-09-18T10:09:03.982814+0000","last_undegraded":"2024-09-18T10:09:03.982814+0000","last_fullsized":"2024-09-18T10:09:03.982814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:35:49.571437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980698+0000","last_change":"2024-09-18T10:09:03.981143+0000","last_active":"2024-09-18T10:09:03.980698+0000","last_peered":"2024-09-18T10:09:03.980698+0000","last_clean":"2024-09-18T10:09:03.980698+0000","last_became_active":"2024-09-18T10:09:01.977398+0000","last_became_peered":"2024-09-18T10:09:01.977398+0000","last_unstale":"2024-09-18T10:09:03.980698+0000","last_undegraded":"2024-09-18T10:09:03.980698+0000","last_fullsized":"2024-09-18T10:09:03.980698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:55:31.629074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982451+0000","last_change":"2024-09-18T10:09:03.982833+0000","last_active":"2024-09-18T10:09:03.982451+0000","last_peered":"2024-09-18T10:09:03.982451+0000","last_clean":"2024-09-18T10:09:03.982451+0000","last_became_active":"2024-09-18T10:09:01.975781+0000","last_became_peered":"2024-09-18T10:09:01.975781+0000","last_unstale":"2024-09-18T10:09:03.982451+0000","last_undegraded":"2024-09-18T10:09:03.982451+0000","last_fullsized":"2024-09-18T10:09:03.982451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:07:07.146280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.980775+0000","last_change":"2024-09-18T10:09:03.981141+0000","last_active":"2024-09-18T10:09:03.980775+0000","last_peered":"2024-09-18T10:09:03.980775+0000","last_clean":"2024-09-18T10:09:03.980775+0000","last_became_active":"2024-09-18T10:09:01.982185+0000","last_became_peered":"2024-09-18T10:09:01.982185+0000","last_unstale":"2024-09-18T10:09:03.980775+0000","last_undegraded":"2024-09-18T10:09:03.980775+0000","last_fullsized":"2024-09-18T10:09:03.980775+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:12:33.081514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.988120+0000","last_change":"2024-09-18T10:09:03.988120+0000","last_active":"2024-09-18T10:09:03.988120+0000","last_peered":"2024-09-18T10:09:03.988120+0000","last_clean":"2024-09-18T10:09:03.988120+0000","last_became_active":"2024-09-18T10:09:01.980916+0000","last_became_peered":"2024-09-18T10:09:01.980916+0000","last_unstale":"2024-09-18T10:09:03.988120+0000","last_undegraded":"2024-09-18T10:09:03.988120+0000","last_fullsized":"2024-09-18T10:09:03.988120+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:51.358013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:04.292190+0000","last_change":"2024-09-18T10:09:04.292594+0000","last_active":"2024-09-18T10:09:04.292190+0000","last_peered":"2024-09-18T10:09:04.292190+0000","last_clean":"2024-09-18T10:09:04.292190+0000","last_became_active":"2024-09-18T10:09:01.985425+0000","last_became_peered":"2024-09-18T10:09:01.985425+0000","last_unstale":"2024-09-18T10:09:04.292190+0000","last_undegraded":"2024-09-18T10:09:04.292190+0000","last_fullsized":"2024-09-18T10:09:04.292190+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:20.190675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.982341+0000","last_change":"2024-09-18T10:09:03.982693+0000","last_active":"2024-09-18T10:09:03.982341+0000","last_peered":"2024-09-18T10:09:03.982341+0000","last_clean":"2024-09-18T10:09:03.982341+0000","last_became_active":"2024-09-18T10:09:01.976391+0000","last_became_peered":"2024-09-18T10:09:01.976391+0000","last_unstale":"2024-09-18T10:09:03.982341+0000","last_undegraded":"2024-09-18T10:09:03.982341+0000","last_fullsized":"2024-09-18T10:09:03.982341+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:46:46.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T10:09:03.987814+0000","last_change":"2024-09-18T10:09:03.988400+0000","last_active":"2024-09-18T10:09:03.987814+0000","last_peered":"2024-09-18T10:09:03.987814+0000","last_clean":"2024-09-18T10:09:03.987814+0000","last_became_active":"2024-09-18T10:09:01.983714+0000","last_became_peered":"2024-09-18T10:09:01.983714+0000","last_unstale":"2024-09-18T10:09:03.987814+0000","last_undegraded":"2024-09-18T10:09:03.987814+0000","last_fullsized":"2024-09-18T10:09:03.987814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:32:44.111047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":101,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T10:09:02.980246+0000","last_change":"2024-09-18T10:09:00.850110+0000","last_active":"2024-09-18T10:09:02.980246+0000","last_peered":"2024-09-18T10:09:02.980246+0000","last_clean":"2024-09-18T10:09:02.980246+0000","last_became_active":"2024-09-18T10:09:00.849769+0000","last_became_peered":"2024-09-18T10:09:00.849769+0000","last_unstale":"2024-09-18T10:09:02.980246+0000","last_undegraded":"2024-09-18T10:09:02.980246+0000","last_fullsized":"2024-09-18T10:09:02.980246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:58:02.351581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297512,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074328,"statfs":{"total":96636764160,"available":96332111872,"internally_reserved":0,"allocated":155648,"data_stored":48327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297516,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074324,"statfs":{"total":96636764160,"available":96332107776,"internally_reserved":0,"allocated":159744,"data_stored":48346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":131072,"data_stored":38499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":688128,"data_stored":577255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.238}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:09:09.747 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-18T10:09:09.748 DEBUG:tasks.mon_thrash:Yielding 2024-09-18T10:09:09.748 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T10:09:09.761 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:09:09.761 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T10:09:09.762 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T10:09:09.762 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-18T10:09:09.762 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726654149, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-09-18T10:09:09.762 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:09:09.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:09:09.770 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T10:09:09.770 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T10:09:09.820 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T10:09:09.820 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T10:09:09.827 INFO:tasks.workunit:timeout=3h 2024-09-18T10:09:09.827 INFO:tasks.workunit:cleanup=True 2024-09-18T10:09:09.828 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:09:09.835 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: get_monmap_and_config 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: monmap: 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:09.868357+0000 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:09.868357+0000 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:09.870 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 Processor -- start 2024-09-18T10:09:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 -- start start 2024-09-18T10:09:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: init 2024-09-18T10:09:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e3c09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e3c132660 0x7f1e3c134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-a con 0x7f1e3c132660 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e3c134f80 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3c09dc20 con 0x7f1e3c134f80 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3c09afa0 con 0x7f1e3c09abd0 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3c095cc0 con 0x7f1e3c132660 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e42b6d700 10 monclient(hunting): authenticate will time out at 838.793799s 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e3c132660 0x7f1e3c134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e3c132660 0x7f1e3c134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48826/0 (socket says 172.21.15.142:48826) 2024-09-18T10:09:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 -- 172.21.15.142:0/4210940338 learned_addr learned my addr 172.21.15.142:0/4210940338 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): get_auth_request con 0x7f1e3c132660 auth_method 0 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 -- 172.21.15.142:0/4210940338 >> v1:172.21.15.150:6789/0 conn(0x7f1e3c09abd0 legacy=0x7f1e3c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 -- 172.21.15.142:0/4210940338 >> v1:172.21.15.142:6789/0 conn(0x7f1e3c134f80 legacy=0x7f1e3c137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 monclient: found mon.noname-a 2024-09-18T10:09:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.864+0000 7f1e4136a700 1 -- 172.21.15.142:0/4210940338 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c09b470 con 0x7f1e3c132660 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e4136a700 1 -- 172.21.15.142:0/4210940338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 msgr2=0x7f1e3c134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e4136a700 1 --2- 172.21.15.142:0/4210940338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c134a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009f20 tx=0x7f1e2c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3690acc34e3ab648 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 1 -- 172.21.15.142:0/4210940338 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c00e070 con 0x7f1e3c132660 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: dump: 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 5 monclient: authenticate success, global_id 4347 2024-09-18T10:09:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 1 -- 172.21.15.142:0/4210940338 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c00bc90 con 0x7f1e3c132660 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 4 set_mon_vals no callback set 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 1 -- 172.21.15.142:0/4210940338 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c012680 con 0x7f1e3c132660 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e40b69700 10 monclient: dump: 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 10 monclient: shutdown 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 -- 172.21.15.142:0/4210940338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 msgr2=0x7f1e3c134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/4210940338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c134a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009f20 tx=0x7f1e2c009840 comp rx=0 tx=0).stop 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 -- 172.21.15.142:0/4210940338 shutdown_connections 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/4210940338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c134a40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 -- 172.21.15.142:0/4210940338 >> 172.21.15.142:0/4210940338 conn(0x7f1e3c071ae0 msgr2=0x7f1e3c0a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 -- 172.21.15.142:0/4210940338 shutdown_connections 2024-09-18T10:09:09.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 1 -- 172.21.15.142:0/4210940338 wait complete. 2024-09-18T10:09:09.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 10 monclient: build_initial_monmap 2024-09-18T10:09:09.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.868+0000 7f1e42b6d700 10 monclient: monmap: 2024-09-18T10:09:09.882 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:09.882 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:09.882 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 Processor -- start 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 -- start start 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient: init 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:09.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e3c09abd0 0x7f1e3c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-c con 0x7f1e3c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-a con 0x7f1e3c132660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e3c134f80 0x7f1e3c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): picked mon.noname-b con 0x7f1e3c134f80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c127460 con 0x7f1e3c132660 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c126620 con 0x7f1e3c134f80 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c09f520 con 0x7f1e3c09abd0 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e42b6d700 10 monclient(hunting): authenticate will time out at 838.801454s 2024-09-18T10:09:09.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e3c09abd0 0x7f1e3c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48832/0 (socket says 172.21.15.142:48832) 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): get_auth_request con 0x7f1e3c09abd0 auth_method 0 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 -- 172.21.15.142:0/1227794050 learned_addr learned my addr 172.21.15.142:0/1227794050 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): get_auth_request con 0x7f1e3c132660 auth_method 0 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e3c134f80 0x7f1e3c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e41b6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): get_auth_request con 0x7f1e3c134f80 auth_method 0 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:09.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4236c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 -- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e3c09abd0 msgr2=0x7f1e3c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e3c09abd0 0x7f1e3c06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 -- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e3c134f80 msgr2=0x7f1e3c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e3c134f80 0x7f1e3c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient: _finish_hunting 0 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 monclient: found mon.noname-a 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.872+0000 7f1e4136a700 1 -- 172.21.15.142:0/1227794050 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c009ae0 con 0x7f1e3c132660 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e4136a700 1 -- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 msgr2=0x7f1e3c06cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:09.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e4136a700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c09ddb0 tx=0x7f1e2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c0126c0 con 0x7f1e3c132660 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: dump: 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.888 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 5 monclient: authenticate success, global_id 4350 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c012c40 con 0x7f1e3c132660 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _renew_subs 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c1286d0 con 0x7f1e3c132660 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c01d420 con 0x7f1e3c132660 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:09.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 10 monclient: dump: 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e3bfff700 4 set_mon_vals no callback set 2024-09-18T10:09:09.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _renew_subs 2024-09-18T10:09:09.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c06e5a0 con 0x7f1e3c132660 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1e2c01d7d0 con 0x7f1e3c132660 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c072a80 con 0x7f1e3c132660 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 0x7f1e28037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f1e2c049860 con 0x7f1e3c132660 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e41b6b700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 0x7f1e28037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.876+0000 7f1e41b6b700 10 monclient: get_auth_request con 0x7f1e28035ac0 auth_method 0 2024-09-18T10:09:09.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.880+0000 7f1e41b6b700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 0x7f1e28037f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e300039a0 tx=0x7f1e30007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:09.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.888+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1e2c0076c0 con 0x7f1e3c132660 2024-09-18T10:09:09.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.888+0000 7f1e39ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:09.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:09.888+0000 7f1e39ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:10.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e42b6d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:10.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e42b6d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:10.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e42b6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1e3c127f40 con 0x7f1e3c132660 2024-09-18T10:09:10.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e39ffb700 1 -- 172.21.15.142:0/1227794050 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1e3c127f40 con 0x7f1e3c132660 2024-09-18T10:09:10.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e39ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:10.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.108+0000 7f1e39ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:10.115 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:10.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 msgr2=0x7f1e28037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 0x7f1e28037f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1e300039a0 tx=0x7f1e30007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 10 monclient: shutdown 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 msgr2=0x7f1e3c06cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c09ddb0 tx=0x7f1e2c017000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 shutdown_connections 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e28035ac0 0x7f1e28037f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e3c134f80 0x7f1e3c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e3c132660 0x7f1e3c06cea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 --2- 172.21.15.142:0/1227794050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e3c09abd0 0x7f1e3c06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 >> 172.21.15.142:0/1227794050 conn(0x7f1e3c071ae0 msgr2=0x7f1e3c09f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 shutdown_connections 2024-09-18T10:09:10.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.112+0000 7f1e42b6d700 1 -- 172.21.15.142:0/1227794050 wait complete. 2024-09-18T10:09:10.129 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:09:10.129 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:09:10.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:09:10.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.228+0000 7f3b61963700 10 monclient: get_monmap_and_config 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.228+0000 7f3b61963700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.228+0000 7f3b61963700 10 monclient: monmap: 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:10.235407+0000 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:10.235407+0000 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:10.237 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 Processor -- start 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 -- start start 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient: init 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b5c09d870 0x7f3b5c0954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c09d870 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b5c0989c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b5c0994d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c069c70 con 0x7f3b5c0994d0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c09dc40 con 0x7f3b5c0989c0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c06f9d0 con 0x7f3b5c09d870 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b61963700 10 monclient(hunting): authenticate will time out at 839.161517s 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b5c09d870 0x7f3b5c0954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b5c09d870 0x7f3b5c0954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48836/0 (socket says 172.21.15.142:48836) 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 -- 172.21.15.142:0/1992846493 learned_addr learned my addr 172.21.15.142:0/1992846493 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): get_auth_request con 0x7f3b5c09d870 auth_method 0 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 -- 172.21.15.142:0/1992846493 >> v1:172.21.15.150:6789/0 conn(0x7f3b5c0989c0 legacy=0x7f3b5c098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 -- 172.21.15.142:0/1992846493 >> v1:172.21.15.142:6789/0 conn(0x7f3b5c0994d0 legacy=0x7f3b5c136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.232+0000 7f3b5affd700 1 -- 172.21.15.142:0/1992846493 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c06fc20 con 0x7f3b5c09d870 2024-09-18T10:09:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b5affd700 1 -- 172.21.15.142:0/1992846493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c09d870 msgr2=0x7f3b5c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b5affd700 1 --2- 172.21.15.142:0/1992846493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c09d870 0x7f3b5c0954b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00bc50 tx=0x7f3b4c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553e2116eee5b3af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 1 -- 172.21.15.142:0/1992846493 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00de00 con 0x7f3b5c09d870 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: dump: 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 5 monclient: authenticate success, global_id 4353 2024-09-18T10:09:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 1 -- 172.21.15.142:0/1992846493 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c010680 con 0x7f3b5c09d870 2024-09-18T10:09:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 4 set_mon_vals no callback set 2024-09-18T10:09:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 1 -- 172.21.15.142:0/1992846493 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c010a40 con 0x7f3b5c09d870 2024-09-18T10:09:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b59ffb700 10 monclient: dump: 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: shutdown 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- 172.21.15.142:0/1992846493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c09d870 msgr2=0x7f3b5c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 --2- 172.21.15.142:0/1992846493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c09d870 0x7f3b5c0954b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00bc50 tx=0x7f3b4c00b4f0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- 172.21.15.142:0/1992846493 shutdown_connections 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 --2- 172.21.15.142:0/1992846493 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c09d870 0x7f3b5c0954b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- 172.21.15.142:0/1992846493 >> 172.21.15.142:0/1992846493 conn(0x7f3b5c0a14e0 msgr2=0x7f3b5c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- 172.21.15.142:0/1992846493 shutdown_connections 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- 172.21.15.142:0/1992846493 wait complete. 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: monmap: 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:10.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 Processor -- start 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- start start 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: init 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:10.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b5c0989c0 0x7f3b5c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-c con 0x7f3b5c0989c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 0x7f3b5c094300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-a con 0x7f3b5c0994d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b5c090630 0x7f3b5c08df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): picked mon.noname-b con 0x7f3b5c090630 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c091ae0 con 0x7f3b5c0994d0 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c09e950 con 0x7f3b5c090630 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c091520 con 0x7f3b5c0989c0 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b61963700 10 monclient(hunting): authenticate will time out at 839.167860s 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b5affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b5c0989c0 0x7f3b5c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.236+0000 7f3b5a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 0x7f3b5c094300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b5c0989c0 0x7f3b5c141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51518/0 (socket says 172.21.15.142:51518) 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 1 -- 172.21.15.142:0/1721638104 learned_addr learned my addr 172.21.15.142:0/1721638104 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): get_auth_request con 0x7f3b5c0989c0 auth_method 0 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): get_auth_request con 0x7f3b5c0994d0 auth_method 0 2024-09-18T10:09:10.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5b7fe700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b5c090630 0x7f3b5c08df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b5c0989c0 msgr2=0x7f3b5c141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b5c0989c0 0x7f3b5c141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b5c090630 msgr2=0x7f3b5c08df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b5c090630 0x7f3b5c08df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c00b790 con 0x7f3b5c0994d0 2024-09-18T10:09:10.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5b7fe700 10 monclient: get_auth_request con 0x7f3b5c090630 auth_method 0 2024-09-18T10:09:10.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5b7fe700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b5c090630 0x7f3b5c08df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 msgr2=0x7f3b5c094300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b5a7fc700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 0x7f3b5c094300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c510 tx=0x7f3b50007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b500117a0 con 0x7f3b5c0994d0 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: dump: 2024-09-18T10:09:10.263 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.264 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 5 monclient: authenticate success, global_id 4356 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50011920 con 0x7f3b5c0994d0 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50011ce0 con 0x7f3b5c0994d0 2024-09-18T10:09:10.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 10 monclient: _renew_subs 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c091800 con 0x7f3b5c0994d0 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b597fa700 4 set_mon_vals no callback set 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b42ffd700 10 monclient: dump: 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.266 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.267 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 10 monclient: _renew_subs 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.240+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c141f80 con 0x7f3b5c0994d0 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b61963700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b61963700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 0x7f3b5c08e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3b50012630 con 0x7f3b5c0994d0 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c064950 con 0x7f3b5c094960 2024-09-18T10:09:10.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 0x7f3b5c08e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient: get_auth_request con 0x7f3b5c094960 auth_method 0 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b42ffd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b48035b10 0x7f3b48037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3b50047f20 con 0x7f3b5c0994d0 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5b7fe700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b48035b10 0x7f3b48037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5affd700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b5c0696a0 con 0x7f3b5c094960 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.244+0000 7f3b5b7fe700 10 monclient: get_auth_request con 0x7f3b48035b10 auth_method 0 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b5affd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 msgr2=0x7f3b5c08e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b5affd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 0x7f3b5c08e450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0a24d0 tx=0x7f3b4c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c0189a0 con 0x7f3b5c094960 2024-09-18T10:09:10.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 10 monclient: dump: 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b5b7fe700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b48035b10 0x7f3b48037fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3b54007fd0 tx=0x7f3b54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3b4c018b00 con 0x7f3b5c094960 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 msgr2=0x7f3b5c08e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 0x7f3b5c08e450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0a24d0 tx=0x7f3b4c01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 msgr2=0x7f3b5c08e450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.248+0000 7f3b42ffd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c094960 0x7f3b5c08e450 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0a24d0 tx=0x7f3b4c01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b61963700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b61963700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 0x7f3b5c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b61963700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c091990 con 0x7f3b5c08e990 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 0x7f3b5c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient: get_auth_request con 0x7f3b5c08e990 auth_method 0 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 1 -- 172.21.15.142:0/1721638104 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b5c08ed90 con 0x7f3b5c08e990 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 msgr2=0x7f3b5c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b5affd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 0x7f3b5c06d410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00e010 tx=0x7f3b4c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c02e960 con 0x7f3b5c08e990 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 10 monclient: dump: 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f3b4c02b390 con 0x7f3b5c08e990 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 msgr2=0x7f3b5c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.260+0000 7f3b42ffd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 0x7f3b5c06d410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00e010 tx=0x7f3b4c021000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b42ffd700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 msgr2=0x7f3b5c06d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b42ffd700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c08e990 0x7f3b5c06d410 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00e010 tx=0x7f3b4c021000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:10.277 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 8, 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 22, 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.278 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.280 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.281 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.282 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:10.282 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:10.282 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:10.282 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:10.282 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.283 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:10.286 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.288 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.289 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.290 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:10.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:10.292 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b48035b10 msgr2=0x7f3b48037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b48035b10 0x7f3b48037fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3b54007fd0 tx=0x7f3b54009040 comp rx=0 tx=0).stop 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 10 monclient: shutdown 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 msgr2=0x7f3b5c094300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 0x7f3b5c094300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c510 tx=0x7f3b50007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b5b7fe700 1 -- 172.21.15.142:0/1721638104 reap_dead start 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 shutdown_connections 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 --2- 172.21.15.142:0/1721638104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c0994d0 0x7f3b5c094300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 >> 172.21.15.142:0/1721638104 conn(0x7f3b5c0a14e0 msgr2=0x7f3b5c09f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 shutdown_connections 2024-09-18T10:09:10.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.264+0000 7f3b61963700 1 -- 172.21.15.142:0/1721638104 wait complete. 2024-09-18T10:09:10.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:10.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:09:10.296 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:09:10.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.396+0000 7f142eac8700 10 monclient: get_monmap_and_config 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.396+0000 7f142eac8700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.396+0000 7f142eac8700 10 monclient: monmap: 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:10.402998+0000 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:10.402998+0000 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:10.404 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:10.405 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:10.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.396+0000 7f142eac8700 1 Processor -- start 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 1 -- start start 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient: init 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f142809abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:10.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f142809b740 0x7f1428134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-a con 0x7f142809b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1428134bb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142809dc90 con 0x7f142809abd0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142809afa0 con 0x7f1428134bb0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142809b500 con 0x7f142809b740 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142eac8700 10 monclient(hunting): authenticate will time out at 839.328622s 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f142809b740 0x7f1428134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142dac6700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f142809abd0 0x7f14280960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46616/0 (socket says 172.21.15.142:46616) 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142dac6700 1 -- 172.21.15.142:0/183397502 learned_addr learned my addr 172.21.15.142:0/183397502 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): get_auth_request con 0x7f142809b740 auth_method 0 2024-09-18T10:09:10.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 -- 172.21.15.142:0/183397502 >> v1:172.21.15.150:6789/0 conn(0x7f1428134bb0 legacy=0x7f1428136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 -- 172.21.15.142:0/183397502 >> v1:172.21.15.142:6789/0 conn(0x7f142809abd0 legacy=0x7f14280960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 -- 172.21.15.142:0/183397502 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142806f810 con 0x7f142809b740 2024-09-18T10:09:10.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 -- 172.21.15.142:0/183397502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 msgr2=0x7f1428134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142d2c5700 1 --2- 172.21.15.142:0/183397502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428134670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1428095730 tx=0x7f14240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca6d7f5c8bdbe90 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.400+0000 7f142cac4700 1 -- 172.21.15.142:0/183397502 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400b9a0 con 0x7f142809b740 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: dump: 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.418 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 5 monclient: authenticate success, global_id 4365 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 1 -- 172.21.15.142:0/183397502 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142400bb00 con 0x7f142809b740 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 1 -- 172.21.15.142:0/183397502 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400bea0 con 0x7f142809b740 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 4 set_mon_vals no callback set 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142cac4700 10 monclient: dump: 2024-09-18T10:09:10.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: shutdown 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- 172.21.15.142:0/183397502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 msgr2=0x7f1428134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 --2- 172.21.15.142:0/183397502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428134670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1428095730 tx=0x7f14240091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- 172.21.15.142:0/183397502 shutdown_connections 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 --2- 172.21.15.142:0/183397502 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428134670 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- 172.21.15.142:0/183397502 >> 172.21.15.142:0/183397502 conn(0x7f1428070ef0 msgr2=0x7f14280712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- 172.21.15.142:0/183397502 shutdown_connections 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- 172.21.15.142:0/183397502 wait complete. 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: monmap: 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.421 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 Processor -- start 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- start start 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: init 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f142809abd0 0x7f1428087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-b con 0x7f142809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-a con 0x7f142809b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:10.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428134bb0 0x7f1428088a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): picked mon.noname-c con 0x7f1428134bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142809fa00 con 0x7f142809b740 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428089820 con 0x7f142809abd0 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1428089f70 con 0x7f1428134bb0 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142eac8700 10 monclient(hunting): authenticate will time out at 839.335071s 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48876/0 (socket says 172.21.15.142:48876) 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 -- 172.21.15.142:0/2265256638 learned_addr learned my addr 172.21.15.142:0/2265256638 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): get_auth_request con 0x7f142809b740 auth_method 0 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428134bb0 0x7f1428088a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): get_auth_request con 0x7f1428134bb0 auth_method 0 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142dac6700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f142809abd0 0x7f1428087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142dac6700 10 monclient(hunting): get_auth_request con 0x7f142809abd0 auth_method 0 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142dac6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142dac6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142dac6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428134bb0 msgr2=0x7f1428088a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428134bb0 0x7f1428088a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f142809abd0 msgr2=0x7f1428087fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f142809abd0 0x7f1428087fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142d2c5700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424009420 con 0x7f142809b740 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.404+0000 7f142e2c7700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428134bb0 0x7f1428088a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142d2c5700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 msgr2=0x7f1428088520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142d2c5700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1428071df0 tx=0x7f1424016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424006af0 con 0x7f142809b740 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: dump: 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 5 monclient: authenticate success, global_id 4368 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 monclient: _renew_subs 2024-09-18T10:09:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1424020070 con 0x7f142809b740 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142809d240 con 0x7f142809b740 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142401b6a0 con 0x7f142809b740 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 monclient: _renew_subs 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142808ae70 con 0x7f142809b740 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f141b7fe700 4 set_mon_vals no callback set 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f1419ffb700 10 monclient: dump: 2024-09-18T10:09:10.428 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 0x7f142806b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.408+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142808b690 con 0x7f1428088fa0 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 0x7f142806b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient: get_auth_request con 0x7f1428088fa0 auth_method 0 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1424022030 con 0x7f142809b740 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f140c035ac0 0x7f140c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-18T10:09:10.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f142808ae70 con 0x7f1428088fa0 2024-09-18T10:09:10.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f142404a090 con 0x7f142809b740 2024-09-18T10:09:10.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142e2c7700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f140c035ac0 0x7f140c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142e2c7700 10 monclient: get_auth_request con 0x7f140c035ac0 auth_method 0 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 msgr2=0x7f142806b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142dac6700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 0x7f142806b370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f141c00e740 tx=0x7f141c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c007170 con 0x7f1428088fa0 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 10 monclient: dump: 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.433 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f141c0072d0 con 0x7f1428088fa0 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 msgr2=0x7f142806b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 0x7f142806b370 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f141c00e740 tx=0x7f141c009040 comp rx=0 tx=0).stop 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 msgr2=0x7f142806b370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f1419ffb700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1428088fa0 0x7f142806b370 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f141c00e740 tx=0x7f141c009040 comp rx=0 tx=0).stop 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.412+0000 7f142e2c7700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f140c035ac0 0x7f140c037f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1420007fd0 tx=0x7f1420002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142eac8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142eac8700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 0x7f142806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142eac8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14280648a0 con 0x7f142806b8b0 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 0x7f142806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient: get_auth_request con 0x7f142806b8b0 auth_method 0 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-18T10:09:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.424+0000 7f142dac6700 1 -- 172.21.15.142:0/2265256638 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1428072a80 con 0x7f142806b8b0 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142dac6700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 msgr2=0x7f142806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142dac6700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 0x7f142806bcb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f141c017000 tx=0x7f141c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c0208a0 con 0x7f142806b8b0 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 10 monclient: dump: 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f141c01c400 con 0x7f142806b8b0 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 msgr2=0x7f142806bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 0x7f142806bcb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f141c017000 tx=0x7f141c01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 msgr2=0x7f142806bcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f1419ffb700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f142806b8b0 0x7f142806bcb0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f141c017000 tx=0x7f141c01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:10.439 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 8, 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 22, 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.440 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.441 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:10.442 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:10.443 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:10.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:10.446 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.449 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.450 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.452 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:10.452 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:10.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f140c035ac0 msgr2=0x7f140c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f140c035ac0 0x7f140c037f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1420007fd0 tx=0x7f1420002620 comp rx=0 tx=0).stop 2024-09-18T10:09:10.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 10 monclient: shutdown 2024-09-18T10:09:10.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 msgr2=0x7f1428088520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142e2c7700 1 -- 172.21.15.142:0/2265256638 reap_dead start 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1428071df0 tx=0x7f1424016000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 shutdown_connections 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 --2- 172.21.15.142:0/2265256638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f142809b740 0x7f1428088520 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 >> 172.21.15.142:0/2265256638 conn(0x7f1428070ef0 msgr2=0x7f142809bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 shutdown_connections 2024-09-18T10:09:10.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.428+0000 7f142eac8700 1 -- 172.21.15.142:0/2265256638 wait complete. 2024-09-18T10:09:10.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:10.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:09:10.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:09:10.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 10 monclient: get_monmap_and_config 2024-09-18T10:09:10.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 10 monclient: monmap: 2024-09-18T10:09:10.567 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:10.566623+0000 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:10.566623+0000 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:10.568 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 1 Processor -- start 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 1 -- start start 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.560+0000 7fb3c1ce0700 10 monclient: init 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-b con 0x7fb3bc09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3bc09b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3bc09c5e0 0x7fb3bc137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-c con 0x7fb3bc09c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc09dc90 con 0x7fb3bc09b740 2024-09-18T10:09:10.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc09b500 con 0x7fb3bc09c5e0 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc09afa0 con 0x7fb3bc09abd0 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient(hunting): authenticate will time out at 839.492153s 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b3fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb3bc09b740 0x7fb3bc09bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46626/0 (socket says 172.21.15.142:46626) 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): get_auth_request con 0x7fb3bc09abd0 auth_method 0 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b3fff700 1 -- 172.21.15.142:0/2405949976 learned_addr learned my addr 172.21.15.142:0/2405949976 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c14df700 1 --2- 172.21.15.142:0/2405949976 >> v2:172.21.15.150:3300/0 conn(0x7fb3bc09c5e0 0x7fb3bc137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c14df700 10 monclient(hunting): get_auth_request con 0x7fb3bc09c5e0 auth_method 0 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c14df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c14df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c14df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 -- 172.21.15.142:0/2405949976 >> v2:172.21.15.150:3300/0 conn(0x7fb3bc09c5e0 msgr2=0x7fb3bc137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 --2- 172.21.15.142:0/2405949976 >> v2:172.21.15.150:3300/0 conn(0x7fb3bc09c5e0 0x7fb3bc137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 -- 172.21.15.142:0/2405949976 >> v1:172.21.15.142:6789/0 conn(0x7fb3bc09b740 legacy=0x7fb3bc09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 monclient: found mon.noname-b 2024-09-18T10:09:10.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:10.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 -- 172.21.15.142:0/2405949976 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc06f810 con 0x7fb3bc09abd0 2024-09-18T10:09:10.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 -- 172.21.15.142:0/2405949976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 msgr2=0x7fb3bc0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c0cde700 1 --2- 172.21.15.142:0/2405949976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc0960e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80098e0 tx=0x7fb3b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f63ec4474f159e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 1 -- 172.21.15.142:0/2405949976 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8009e80 con 0x7fb3bc09abd0 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: dump: 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.578 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 5 monclient: authenticate success, global_id 4358 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 1 -- 172.21.15.142:0/2405949976 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b800ba70 con 0x7fb3bc09abd0 2024-09-18T10:09:10.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3c1ce0700 4 set_mon_vals no callback set 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 1 -- 172.21.15.142:0/2405949976 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b800bdf0 con 0x7fb3bc09abd0 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.564+0000 7fb3b37fe700 10 monclient: dump: 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: shutdown 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/2405949976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 msgr2=0x7fb3bc0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/2405949976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc0960e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3b80098e0 tx=0x7fb3b80091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/2405949976 shutdown_connections 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/2405949976 >> v2:172.21.15.150:3300/0 conn(0x7fb3bc09c5e0 0x7fb3bc137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/2405949976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc0960e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/2405949976 >> 172.21.15.142:0/2405949976 conn(0x7fb3bc070ef0 msgr2=0x7fb3bc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/2405949976 shutdown_connections 2024-09-18T10:09:10.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/2405949976 wait complete. 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: monmap: 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 Processor -- start 2024-09-18T10:09:10.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- start start 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: init 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc13f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-c con 0x7fb3bc09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc09b740 0x7fb3bc13feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-b con 0x7fb3bc09b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3bc09c5e0 0x7fb3bc08ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): picked mon.noname-a con 0x7fb3bc09c5e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:10.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc09d240 con 0x7fb3bc09c5e0 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc09fe20 con 0x7fb3bc09b740 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc08db00 con 0x7fb3bc09abd0 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c1ce0700 10 monclient(hunting): authenticate will time out at 839.498490s 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc13f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc13f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51556/0 (socket says 172.21.15.142:51556) 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 -- 172.21.15.142:0/3770487114 learned_addr learned my addr 172.21.15.142:0/3770487114 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): get_auth_request con 0x7fb3bc09abd0 auth_method 0 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3bc09c5e0 0x7fb3bc08ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): get_auth_request con 0x7fb3bc09c5e0 auth_method 0 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc09b740 0x7fb3bc13feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 10 monclient(hunting): get_auth_request con 0x7fb3bc09b740 auth_method 0 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c14df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc09b740 msgr2=0x7fb3bc13feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc09b740 0x7fb3bc13feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3bc09c5e0 msgr2=0x7fb3bc08ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3bc09c5e0 0x7fb3bc08ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 monclient: found mon.noname-c 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3c0cde700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8009420 con 0x7fb3bc09abd0 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.568+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc09b740 0x7fb3bc13feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c0cde700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 msgr2=0x7fb3bc13f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c0cde700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc13f970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc071df0 tx=0x7fb3b8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8000c90 con 0x7fb3bc09abd0 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: dump: 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.591 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 5 monclient: authenticate success, global_id 4361 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b8020070 con 0x7fb3bc09abd0 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 monclient: _renew_subs 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc08f6d0 con 0x7fb3bc09abd0 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b801b450 con 0x7fb3bc09abd0 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 10 monclient: dump: 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.593 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 monclient: _renew_subs 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc08d5f0 con 0x7fb3bc09abd0 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b2ffd700 4 set_mon_vals no callback set 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 0x7fb3bc08c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0648a0 con 0x7fb3bc08bf50 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 0x7fb3bc08c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient: get_auth_request con 0x7fb3bc08bf50 auth_method 0 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.572+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb3b8022030 con 0x7fb3bc09abd0 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3a8035ac0 0x7fb3a8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b3fff700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3bc072a80 con 0x7fb3bc08bf50 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fb3b8049f00 con 0x7fb3bc09abd0 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3c14df700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3a8035ac0 0x7fb3a8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3c14df700 10 monclient: get_auth_request con 0x7fb3a8035ac0 auth_method 0 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b3fff700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 msgr2=0x7fb3bc08c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 0x7fb3bc08c320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e740 tx=0x7fb3ac009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac005d40 con 0x7fb3bc08bf50 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 10 monclient: dump: 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.596 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3c14df700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3a8035ac0 0x7fb3a8037f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4007640 tx=0x7fb3b4009850 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb3ac0070d0 con 0x7fb3bc08bf50 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 msgr2=0x7fb3bc08c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 0x7fb3bc08c320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00e740 tx=0x7fb3ac009040 comp rx=0 tx=0).stop 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 msgr2=0x7fb3bc08c320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.576+0000 7fb3b0ff9700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08bf50 0x7fb3bc08c320 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3c1ce0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 0x7fb3bc087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3c1ce0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc08f2b0 con 0x7fb3bc08c860 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 0x7fb3bc087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient: get_auth_request con 0x7fb3bc08c860 auth_method 0 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.588+0000 7fb3b3fff700 1 -- 172.21.15.142:0/3770487114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3bc08e1c0 con 0x7fb3bc08c860 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b3fff700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 msgr2=0x7fb3bc087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b3fff700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 0x7fb3bc087fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac017000 tx=0x7fb3ac01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac020940 con 0x7fb3bc08c860 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 10 monclient: dump: 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.600 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fb3ac020aa0 con 0x7fb3bc08c860 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 msgr2=0x7fb3bc087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 0x7fb3bc087fe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac017000 tx=0x7fb3ac01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 msgr2=0x7fb3bc087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:10.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3b0ff9700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3bc08c860 0x7fb3bc087fe0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac017000 tx=0x7fb3ac01a000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 8, 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.602 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 22, 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.603 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.604 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:10.605 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:10.606 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:10.608 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:10.612 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.613 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:10.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:10.615 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:10.616 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3a8035ac0 msgr2=0x7fb3a8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3a8035ac0 0x7fb3a8037f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4007640 tx=0x7fb3b4009850 comp rx=0 tx=0).stop 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 10 monclient: shutdown 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 msgr2=0x7fb3bc13f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 --2- 172.21.15.142:0/3770487114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3bc09abd0 0x7fb3bc13f970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc071df0 tx=0x7fb3b8016000 comp rx=0 tx=0).stop 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c14df700 1 -- 172.21.15.142:0/3770487114 reap_dead start 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c14df700 1 -- 172.21.15.142:0/3770487114 reap_dead start 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 shutdown_connections 2024-09-18T10:09:10.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 >> 172.21.15.142:0/3770487114 conn(0x7fb3bc070ef0 msgr2=0x7fb3bc09de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 shutdown_connections 2024-09-18T10:09:10.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.592+0000 7fb3c1ce0700 1 -- 172.21.15.142:0/3770487114 wait complete. 2024-09-18T10:09:10.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:10.619 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:09:10.619 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:09:10.619 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:09:10.620 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:09:10.620 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:09:10.620 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:09:10.639 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:09:10.639 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:09:10.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 10 monclient: get_monmap_and_config 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 10 monclient: monmap: 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.743 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:10.742608+0000 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:10.742608+0000 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:10.744 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 1 Processor -- start 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 1 -- start start 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.736+0000 7f7709b5b700 10 monclient: init 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 0x7f77040a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-b con 0x7f77040a0270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 0x7f77040a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-c con 0x7f770409cf70 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7704099120 0x7f77040a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-a con 0x7f7704099120 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770406a7a0 con 0x7f7704099120 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770409d4d0 con 0x7f770409cf70 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770409fd50 con 0x7f77040a0270 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7709b5b700 10 monclient(hunting): authenticate will time out at 839.668268s 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7702ffd700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 msgr2=0x7f77040a0b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7702ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 0x7f77040a0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7704099120 0x7f77040a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 0x7f77040a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7704099120 0x7f77040a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48900/0 (socket says 172.21.15.142:48900) 2024-09-18T10:09:10.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 0x7f77040a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51570/0 (socket says 172.21.15.142:51570) 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 -- 172.21.15.142:0/1685690938 learned_addr learned my addr 172.21.15.142:0/1685690938 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): get_auth_request con 0x7f7704099120 auth_method 0 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): get_auth_request con 0x7f77040a0270 auth_method 0 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 -- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 msgr2=0x7f77040a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 --2- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 0x7f77040a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 -- 172.21.15.142:0/1685690938 >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 msgr2=0x7f77040a0b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 --2- 172.21.15.142:0/1685690938 >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 0x7f77040a0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 -- 172.21.15.142:0/1685690938 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770409e040 con 0x7f7704099120 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 -- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 msgr2=0x7f77040a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.740+0000 7f7703fff700 1 --2- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f77040a10c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7704071ed0 tx=0x7f76fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18dbafff1a90b7dc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 1 -- 172.21.15.142:0/1685690938 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00e070 con 0x7f7704099120 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: dump: 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.754 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 1 -- 172.21.15.142:0/1685690938 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc00bc30 con 0x7f7704099120 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 1 -- 172.21.15.142:0/1685690938 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc0092e0 con 0x7f7704099120 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77027fc700 10 monclient: dump: 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 5 monclient: authenticate success, global_id 4374 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:10.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 4 set_mon_vals no callback set 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: shutdown 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 msgr2=0x7f77040a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f77040a10c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7704071ed0 tx=0x7f76fc009840 comp rx=0 tx=0).stop 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- 172.21.15.142:0/1685690938 shutdown_connections 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f77040a10c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1685690938 >> v2:172.21.15.150:3300/0 conn(0x7f770409cf70 0x7f77040a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1685690938 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77040a0270 0x7f77040a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- 172.21.15.142:0/1685690938 >> 172.21.15.142:0/1685690938 conn(0x7f7704071820 msgr2=0x7f77040a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- 172.21.15.142:0/1685690938 shutdown_connections 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- 172.21.15.142:0/1685690938 wait complete. 2024-09-18T10:09:10.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: build_initial_monmap 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: monmap: 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 Processor -- start 2024-09-18T10:09:10.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- start start 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: init 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-a con 0x7f7704099120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 0x7f7704137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-b con 0x7f77040a0270 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77041384a0 0x7f770408cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): picked mon.noname-c con 0x7f77041384a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77040986e0 con 0x7f7704099120 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770409bd40 con 0x7f77040a0270 2024-09-18T10:09:10.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770408f470 con 0x7f77041384a0 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7709b5b700 10 monclient(hunting): authenticate will time out at 839.675304s 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48916/0 (socket says 172.21.15.142:48916) 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7702ffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 msgr2=0x7f7704137f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 -- 172.21.15.142:0/1755626161 learned_addr learned my addr 172.21.15.142:0/1755626161 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): get_auth_request con 0x7f7704099120 auth_method 0 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77041384a0 0x7f770408cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7702ffd700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 0x7f7704137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): get_auth_request con 0x7f77041384a0 auth_method 0 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f7703fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-18T10:09:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 -- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77041384a0 msgr2=0x7f770408cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77041384a0 0x7f770408cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.744+0000 7f77037fe700 1 -- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 msgr2=0x7f7704137f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 0x7f7704137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 1 monclient: found mon.noname-a 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 1 -- 172.21.15.142:0/1755626161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc009ae0 con 0x7f7704099120 2024-09-18T10:09:10.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 1 -- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 msgr2=0x7f7704137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:10.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f77037fe700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76f800ecb0 tx=0x7f76f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f800cab0 con 0x7f7704099120 2024-09-18T10:09:10.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: dump: 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:10.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 5 monclient: authenticate success, global_id 4377 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f800cc10 con 0x7f7704099120 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 10 monclient: _renew_subs 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77040902c0 con 0x7f7704099120 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8013470 con 0x7f7704099120 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 10 monclient: dump: 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:10.766 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 10 monclient: _renew_subs 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770408e7d0 con 0x7f7704099120 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f7701ffb700 4 set_mon_vals no callback set 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.748+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f76f8013860 con 0x7f7704099120 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f76eb7fe700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 0x7f76f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:10.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7709b5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7709b5b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7709b5b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f76f80484c0 con 0x7f7704099120 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770408f000 con 0x7f7704099120 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7702ffd700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 0x7f76f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.752+0000 7f7702ffd700 10 monclient: get_auth_request con 0x7f76f0035a70 auth_method 0 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.756+0000 7f7702ffd700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 0x7f76f0037f20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f4006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.760+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f76f8023880 con 0x7f7704099120 2024-09-18T10:09:10.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.760+0000 7f76eb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:10.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.760+0000 7f76eb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:10.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f7709b5b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:10.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f7709b5b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:10.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f7709b5b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:10.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f77040696a0 con 0x7f7704099120 2024-09-18T10:09:10.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f76eb7fe700 1 -- 172.21.15.142:0/1755626161 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f76f802a620 con 0x7f7704099120 2024-09-18T10:09:10.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f76eb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:10.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.984+0000 7f76eb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:10.992 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:10.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 msgr2=0x7f76f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 0x7f76f0037f20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f4006e10 comp rx=0 tx=0).stop 2024-09-18T10:09:10.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 10 monclient: shutdown 2024-09-18T10:09:10.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 msgr2=0x7f7704137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76f800ecb0 tx=0x7f76f80091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 shutdown_connections 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f76f0035a70 0x7f76f0037f20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77041384a0 0x7f770408cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77040a0270 0x7f7704137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 --2- 172.21.15.142:0/1755626161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7704099120 0x7f7704137a20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 >> 172.21.15.142:0/1755626161 conn(0x7f7704071820 msgr2=0x7f770409b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 shutdown_connections 2024-09-18T10:09:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:10.988+0000 7f7709b5b700 1 -- 172.21.15.142:0/1755626161 wait complete. 2024-09-18T10:09:14.007 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 10 monclient: get_monmap_and_config 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 10 monclient: build_initial_monmap 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 10 monclient: monmap: 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:14.118067+0000 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:14.118067+0000 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:14.120 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 1 Processor -- start 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 1 -- start start 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.112+0000 7f4b34e95700 10 monclient: init 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a1cf0 0x7f4b3012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-b con 0x7f4b300a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b3012f810 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4b300a02d0 0x7f4b30139420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-a con 0x7f4b300a02d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3009fb90 con 0x7f4b3012f810 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3009d990 con 0x7f4b300a02d0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3009dc70 con 0x7f4b300a1cf0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient(hunting): authenticate will time out at 843.044182s 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4b300a02d0 0x7f4b30139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2effd700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f4b3012f810 legacy=0x7f4b3009f480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4b300a02d0 0x7f4b30139420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43114/0 (socket says 172.21.15.142:43114) 2024-09-18T10:09:14.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a1cf0 0x7f4b3012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 -- 172.21.15.142:0/2239322999 learned_addr learned my addr 172.21.15.142:0/2239322999 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): get_auth_request con 0x7f4b300a02d0 auth_method 0 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f4b300a1cf0 auth_method 0 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:14.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 -- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a1cf0 msgr2=0x7f4b3012f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 --2- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a1cf0 0x7f4b3012f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 -- 172.21.15.142:0/2239322999 >> v1:172.21.15.150:6789/0 conn(0x7f4b3012f810 legacy=0x7f4b3009f480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 monclient: found mon.noname-a 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 -- 172.21.15.142:0/2239322999 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b300a20c0 con 0x7f4b300a02d0 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 -- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a02d0 msgr2=0x7f4b30139420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2ffff700 1 --2- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a02d0 0x7f4b30139420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009f20 tx=0x7f4b28009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2053c8e2b8049d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 1 -- 172.21.15.142:0/2239322999 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800e070 con 0x7f4b300a02d0 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: dump: 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.130 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 5 monclient: authenticate success, global_id 4380 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 1 -- 172.21.15.142:0/2239322999 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800bbe0 con 0x7f4b300a02d0 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:14.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b34e95700 4 set_mon_vals no callback set 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 1 -- 172.21.15.142:0/2239322999 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b280092e0 con 0x7f4b300a02d0 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.116+0000 7f4b2e7fc700 10 monclient: dump: 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:14.132 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: shutdown 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a02d0 msgr2=0x7f4b30139420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a02d0 0x7f4b30139420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009f20 tx=0x7f4b28009840 comp rx=0 tx=0).stop 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- 172.21.15.142:0/2239322999 shutdown_connections 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a02d0 0x7f4b30139420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- 172.21.15.142:0/2239322999 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a1cf0 0x7f4b3012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- 172.21.15.142:0/2239322999 >> 172.21.15.142:0/2239322999 conn(0x7f4b30071820 msgr2=0x7f4b300a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:14.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- 172.21.15.142:0/2239322999 shutdown_connections 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- 172.21.15.142:0/2239322999 wait complete. 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: build_initial_monmap 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: monmap: 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:14.134 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 Processor -- start 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- start start 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: init 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-c con 0x7f4b300a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-a con 0x7f4b300a1cf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 0x7f4b3006b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): picked mon.noname-b con 0x7f4b3012f810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3009cdc0 con 0x7f4b300a1cf0 2024-09-18T10:09:14.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3012ee20 con 0x7f4b3012f810 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3006ec00 con 0x7f4b300a02d0 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b34e95700 10 monclient(hunting): authenticate will time out at 843.050802s 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43118/0 (socket says 172.21.15.142:43118) 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55482/0 (socket says 172.21.15.142:55482) 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 -- 172.21.15.142:0/3392312400 learned_addr learned my addr 172.21.15.142:0/3392312400 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2ffff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 msgr2=0x7f4b3006b540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:14.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): get_auth_request con 0x7f4b300a1cf0 auth_method 0 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f4b300a02d0 auth_method 0 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2ffff700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 0x7f4b3006b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:14.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 -- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 msgr2=0x7f4b30138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 -- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 msgr2=0x7f4b3006b540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 0x7f4b3006b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 monclient: found mon.noname-a 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2effd700 1 -- 172.21.15.142:0/3392312400 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b28009ae0 con 0x7f4b300a1cf0 2024-09-18T10:09:14.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.120+0000 7f4b2f7fe700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:14.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b2effd700 1 -- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 msgr2=0x7f4b3006b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:14.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b2effd700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800b840 tx=0x7f4b180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:14.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18003dd0 con 0x7f4b300a1cf0 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: dump: 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:14.141 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 5 monclient: authenticate success, global_id 4383 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 10 monclient: _renew_subs 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18006150 con 0x7f4b300a1cf0 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3006e6a0 con 0x7f4b300a1cf0 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18006510 con 0x7f4b300a1cf0 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 10 monclient: dump: 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:14.142 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 10 monclient: _renew_subs 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3006d280 con 0x7f4b300a1cf0 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b2dffb700 4 set_mon_vals no callback set 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4b1801b3b0 con 0x7f4b300a1cf0 2024-09-18T10:09:14.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.124+0000 7f4b277fe700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 0x7f4b10037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f4b18047810 con 0x7f4b300a1cf0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b2f7fe700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 0x7f4b10037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b2f7fe700 10 monclient: get_auth_request con 0x7f4b10035a70 auth_method 0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b34e95700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b34e95700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b34e95700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3006cc10 con 0x7f4b300a1cf0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.128+0000 7f4b2f7fe700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 0x7f4b10037f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b20011480 tx=0x7f4b200091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.136+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4b18017080 con 0x7f4b300a1cf0 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.136+0000 7f4b277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:14.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.136+0000 7f4b277fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:14.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b30072a80 con 0x7f4b300a1cf0 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b277fe700 1 -- 172.21.15.142:0/3392312400 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4b1800e860 con 0x7f4b300a1cf0 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:14.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b277fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:14.367 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:14.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 msgr2=0x7f4b10037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 0x7f4b10037f20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b20011480 tx=0x7f4b200091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.360+0000 7f4b34e95700 10 monclient: shutdown 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 msgr2=0x7f4b3006b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800b840 tx=0x7f4b180091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 shutdown_connections 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b10035a70 0x7f4b10037f20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b3012f810 0x7f4b3006b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b300a1cf0 0x7f4b3006b000 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 --2- 172.21.15.142:0/3392312400 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b300a02d0 0x7f4b30138650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 >> 172.21.15.142:0/3392312400 conn(0x7f4b30071820 msgr2=0x7f4b30138f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 shutdown_connections 2024-09-18T10:09:14.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:14.364+0000 7f4b34e95700 1 -- 172.21.15.142:0/3392312400 wait complete. 2024-09-18T10:09:17.382 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 10 monclient: get_monmap_and_config 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 10 monclient: build_initial_monmap 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 10 monclient: monmap: 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:17.490110+0000 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:17.490110+0000 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:17.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 1 Processor -- start 2024-09-18T10:09:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 1 -- start start 2024-09-18T10:09:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.484+0000 7f1feb59e700 10 monclient: init 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1fec0a2e70 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fec070ef0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fec071a00 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec09dc50 con 0x7f1fec070ef0 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec09d700 con 0x7f1fec071a00 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec09d330 con 0x7f1fec0a2e70 2024-09-18T10:09:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1feb59e700 10 monclient(hunting): authenticate will time out at 846.416713s 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fead9d700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fec137620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fea59c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 msgr2=0x7f1fec09e3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fea59c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fe9d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1fec070ef0 0x7f1fec0712f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46122/0 (socket says 172.21.15.142:46122) 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 learned_addr learned my addr 172.21.15.142:0/708783111 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.488+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.142:6789/0 conn(0x7f1fec070ef0 legacy=0x7f1fec0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:17.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.692+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fec137620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:17.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.692+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 msgr2=0x7f1fec09e3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:17.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:17.692+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:18.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.096+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fec137620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:18.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.096+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 msgr2=0x7f1fec09e3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:18.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.096+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:18.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.900+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fec137620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:18.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.900+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 msgr2=0x7f1fec09e3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:18.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:18.900+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:20.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): tick 2024-09-18T10:09:20.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:09:20.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:09:20.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 msgr2=0x7f1fec09e3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:20.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.150:3300/0 conn(0x7f1fec0a2e70 0x7f1fec09e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fec137620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.142:6789/0 conn(0x7f1fec070ef0 legacy=0x7f1fec0712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient: _add_conns ranks=[2,3] 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f1fcc003780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc006200 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec09d330 con 0x7f1fcc006200 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc008d70 con 0x7f1fcc003780 2024-09-18T10:09:20.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request con 0x7f1fcc006200 auth_method 0 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:20.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:20.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:20.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:20.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:20.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:20.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.488+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:20.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:20.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:20.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:20.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:20.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request con 0x7f1fcc006200 auth_method 0 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:20.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:20.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:20.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:20.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:20.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:20.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:20.692+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:21.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request con 0x7f1fcc006200 auth_method 0 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:21.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:21.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:21.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:21.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:21.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:21.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.096+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:21.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request con 0x7f1fcc006200 auth_method 0 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:21.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:21.900+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:23.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): tick 2024-09-18T10:09:23.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:09:23.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:09:23.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:23.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 msgr2=0x7f1fcc0086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc006200 0x7f1fcc0086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f1fcc003780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 reap_dead start 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fec071a00 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fec070ef0 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec0a3240 con 0x7f1fec070ef0 2024-09-18T10:09:23.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec09d700 con 0x7f1fec071a00 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec09d330 con 0x7f1fcc003780 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fcc00b180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:23.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:23.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:23.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:23.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.492+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.142:6789/0 conn(0x7f1fec070ef0 legacy=0x7f1fcc00e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:23.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fcc00b180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:23.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:23.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:23.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:23.696+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fcc00b180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:24.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:24.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:24.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:24.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:24.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.100+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:24.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fcc00b180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:24.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:24.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:24.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:24.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:24.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:24.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:24.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:24.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:24.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:24.904+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:25.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436672218 0 0) 0x7f1fec0a3240 con 0x7f1fec070ef0 2024-09-18T10:09:25.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 10 monclient(hunting): my global_id is 4386 2024-09-18T10:09:25.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd8003570 con 0x7f1fec070ef0 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 669071703 0 0) 0x7f1fd8003570 con 0x7f1fec070ef0 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fcc005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fcc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.150:6789/0 conn(0x7f1fec071a00 legacy=0x7f1fcc00b180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.196+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc007750 con 0x7f1fec070ef0 2024-09-18T10:09:25.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1fe4004200 con 0x7f1fec070ef0 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: dump: 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.207 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 >> v1:172.21.15.142:6789/0 conn(0x7f1fec070ef0 legacy=0x7f1fcc00e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd8003a90 0x7f1fd8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): picked mon.b con 0x7f1fd8003a90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:25.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fd8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): picked mon.c con 0x7f1fcc003780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8004d50 0x7f1fd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): picked mon.a con 0x7f1fd8004d50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec09d330 con 0x7f1fd8004d50 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd800bc20 con 0x7f1fd8003a90 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd800bde0 con 0x7f1fcc003780 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe959a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fd8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8004d50 0x7f1fd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fea59c700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd8003a90 msgr2=0x7f1fd8003e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): get_auth_request con 0x7f1fd8004d50 auth_method 0 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fea59c700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd8003a90 0x7f1fd8003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient(hunting): handle_auth_done global_id 4386 payload 293 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd8003a90 msgr2=0x7f1fd8003e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd8003a90 0x7f1fd8003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8004d50 msgr2=0x7f1fd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8004d50 0x7f1fd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 monclient: found mon.c 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8003570 con 0x7f1fcc003780 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fe9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1feb59e700 5 monclient: authenticate success, global_id 4386 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1feb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:25.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.200+0000 7f1fead9d700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8004d50 0x7f1fd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe9d9b700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fd8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe9d9b700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fd8004810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4001be0 tx=0x7f1fd4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6856641c5e34f75c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4015480 con 0x7f1fcc003780 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: dump: 2024-09-18T10:09:25.214 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd4015600 con 0x7f1fcc003780 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 1 -- 172.21.15.142:0/708783111 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40157b0 con 0x7f1fcc003780 2024-09-18T10:09:25.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1fe959a700 10 monclient: dump: 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1feb59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:25.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.204+0000 7f1feb59e700 4 set_mon_vals no callback set 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 10 monclient: shutdown 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 msgr2=0x7f1fd8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 --2- 172.21.15.142:0/708783111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc003780 0x7f1fd8004810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4001be0 tx=0x7f1fd4009040 comp rx=0 tx=0).stop 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1fead9d700 1 -- 172.21.15.142:0/708783111 reap_dead start 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- 172.21.15.142:0/708783111 shutdown_connections 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- 172.21.15.142:0/708783111 >> 172.21.15.142:0/708783111 conn(0x7f1fec06a2e0 msgr2=0x7f1fec0a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- 172.21.15.142:0/708783111 shutdown_connections 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- 172.21.15.142:0/708783111 wait complete. 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 10 monclient: build_initial_monmap 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 10 monclient: monmap: 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:25.217 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:25.218 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 Processor -- start 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 1 -- start start 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.208+0000 7f1feb59e700 10 monclient: init 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:25.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 0x7f1fec13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc003780 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 0x7f1fec13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f1fec13fc10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f1fec08aba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0952e0 con 0x7f1fcc003780 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec098be0 con 0x7f1fec13fc10 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec140520 con 0x7f1fec08aba0 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient(hunting): authenticate will time out at 854.140239s 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe9d9b700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 msgr2=0x7f1fec13ffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe9d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 0x7f1fec13ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 0x7f1fec13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50194/0 (socket says 172.21.15.142:50194) 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 -- 172.21.15.142:0/328790712 learned_addr learned my addr 172.21.15.142:0/328790712 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): get_auth_request con 0x7f1fec08aba0 auth_method 0 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 0x7f1fec13f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59640/0 (socket says 172.21.15.142:59640) 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): get_auth_request con 0x7f1fcc003780 auth_method 0 2024-09-18T10:09:25.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fea59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 -- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 msgr2=0x7f1fec13ffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 0x7f1fec13ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 -- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 msgr2=0x7f1fec13f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 0x7f1fec13f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 monclient: found mon.noname-c 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 -- 172.21.15.142:0/328790712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec08c5d0 con 0x7f1fec08aba0 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 -- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 msgr2=0x7f1fec08afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fead9d700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc008d10 tx=0x7f1fe4007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40116d0 con 0x7f1fec08aba0 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: dump: 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.227 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4011870 con 0x7f1fec08aba0 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 5 monclient: authenticate success, global_id 4376 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4011c30 con 0x7f1fec08aba0 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _renew_subs 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe8d99700 4 set_mon_vals no callback set 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec08caa0 con 0x7f1fec08aba0 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1fe27fc700 10 monclient: dump: 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.229 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _renew_subs 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.212+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec08e0c0 con 0x7f1fec08aba0 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1feb59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1feb59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1feb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec08d190 con 0x7f1fec08aba0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1fe401e3b0 con 0x7f1fec08aba0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1fe27fc700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 0x7f1fcc038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f1fe4047e10 con 0x7f1fec08aba0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1fea59c700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 0x7f1fcc038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.216+0000 7f1fea59c700 10 monclient: get_auth_request con 0x7f1fcc036740 auth_method 0 2024-09-18T10:09:25.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.220+0000 7f1fea59c700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 0x7f1fcc038bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0039c0 tx=0x7f1fdc017040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.220+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1fe4010640 con 0x7f1fec08aba0 2024-09-18T10:09:25.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.220+0000 7f1fe27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.220+0000 7f1fe27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:25.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:25.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:25.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1fec0648f0 con 0x7f1fec08aba0 2024-09-18T10:09:25.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1fe27fc700 1 -- 172.21.15.142:0/328790712 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f1fe401a030 con 0x7f1fec08aba0 2024-09-18T10:09:25.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1fe27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:25.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1fe27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:25.451 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:25.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 msgr2=0x7f1fcc038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 0x7f1fcc038bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0039c0 tx=0x7f1fdc017040 comp rx=0 tx=0).stop 2024-09-18T10:09:25.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 10 monclient: shutdown 2024-09-18T10:09:25.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 msgr2=0x7f1fec08afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.444+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc008d10 tx=0x7f1fe4007a20 comp rx=0 tx=0).stop 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 shutdown_connections 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fcc036740 0x7f1fcc038bf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fec08aba0 0x7f1fec08afd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec13fc10 0x7f1fec13ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 --2- 172.21.15.142:0/328790712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc003780 0x7f1fec13f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 >> 172.21.15.142:0/328790712 conn(0x7f1fec06a2e0 msgr2=0x7f1fec0976e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 shutdown_connections 2024-09-18T10:09:25.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.448+0000 7f1feb59e700 1 -- 172.21.15.142:0/328790712 wait complete. 2024-09-18T10:09:25.464 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:09:25.464 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:09:25.464 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: get_monmap_and_config 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: build_initial_monmap 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: monmap: 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:25.569270+0000 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:25.569270+0000 2024-09-18T10:09:25.570 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:25.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 Processor -- start 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 -- start start 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: init 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf9009abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf9009b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 0x7fdf90138170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-b con 0x7fdf90135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9009dc90 con 0x7fdf9009b740 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9009afa0 con 0x7fdf9009abd0 2024-09-18T10:09:25.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9009b500 con 0x7fdf90135d90 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95a64700 10 monclient(hunting): authenticate will time out at 854.495009s 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf94a62700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fdf9009abd0 legacy=0x7fdf900960e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf8ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdf9009b740 0x7fdf90134670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:35120/0 (socket says 172.21.15.142:35120) 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf8ffff700 1 -- 172.21.15.142:0/2353519069 learned_addr learned my addr 172.21.15.142:0/2353519069 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 1 --2- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 0x7fdf90138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): get_auth_request con 0x7fdf90135d90 auth_method 0 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 1 -- 172.21.15.142:0/2353519069 >> v1:172.21.15.150:6789/0 conn(0x7fdf9009abd0 legacy=0x7fdf900960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 1 -- 172.21.15.142:0/2353519069 >> v1:172.21.15.142:6789/0 conn(0x7fdf9009b740 legacy=0x7fdf90134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 1 monclient: found mon.noname-b 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.564+0000 7fdf95263700 1 -- 172.21.15.142:0/2353519069 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9006f810 con 0x7fdf90135d90 2024-09-18T10:09:25.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95263700 1 -- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 msgr2=0x7fdf90138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95263700 1 --2- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 0x7fdf90138170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009e90 tx=0x7fdf88009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=612ba9846e5f3812 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 1 -- 172.21.15.142:0/2353519069 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800e070 con 0x7fdf90135d90 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: dump: 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.584 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 5 monclient: authenticate success, global_id 4379 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 1 -- 172.21.15.142:0/2353519069 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88012430 con 0x7fdf90135d90 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 4 set_mon_vals no callback set 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 1 -- 172.21.15.142:0/2353519069 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880127f0 con 0x7fdf90135d90 2024-09-18T10:09:25.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf8f7fe700 10 monclient: dump: 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 10 monclient: shutdown 2024-09-18T10:09:25.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 -- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 msgr2=0x7fdf90138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 --2- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 0x7fdf90138170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009e90 tx=0x7fdf88009780 comp rx=0 tx=0).stop 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 -- 172.21.15.142:0/2353519069 shutdown_connections 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 --2- 172.21.15.142:0/2353519069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf90135d90 0x7fdf90138170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 -- 172.21.15.142:0/2353519069 >> 172.21.15.142:0/2353519069 conn(0x7fdf90070ef0 msgr2=0x7fdf900712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 -- 172.21.15.142:0/2353519069 shutdown_connections 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 1 -- 172.21.15.142:0/2353519069 wait complete. 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 10 monclient: build_initial_monmap 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.568+0000 7fdf95a64700 10 monclient: monmap: 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:25.587 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 Processor -- start 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 -- start start 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient: init 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 0x7fdf9008dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-a con 0x7fdf9009abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 0x7fdf9008ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-c con 0x7fdf9009b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf90135d90 0x7fdf9008b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): picked mon.noname-b con 0x7fdf90135d90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9008e500 con 0x7fdf9009abd0 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9008e220 con 0x7fdf90135d90 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9008e7e0 con 0x7fdf9009b740 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95a64700 10 monclient(hunting): authenticate will time out at 854.500725s 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 0x7fdf9008dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95263700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf90135d90 msgr2=0x7fdf9008b3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf95263700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf90135d90 0x7fdf9008b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 0x7fdf9008ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 0x7fdf9008dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59650/0 (socket says 172.21.15.142:59650) 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 1 -- 172.21.15.142:0/1280935157 learned_addr learned my addr 172.21.15.142:0/1280935157 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): get_auth_request con 0x7fdf9009abd0 auth_method 0 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): get_auth_request con 0x7fdf9009b740 auth_method 0 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf90135d90 msgr2=0x7fdf9008b3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf90135d90 0x7fdf9008b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 msgr2=0x7fdf9008dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 0x7fdf9008dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 monclient: found mon.noname-c 2024-09-18T10:09:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88009a20 con 0x7fdf9009b740 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf94a62700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9009abd0 0x7fdf9008dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 msgr2=0x7fdf9008ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.572+0000 7fdf8ffff700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 0x7fdf9008ae90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800c530 tx=0x7fdf78007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78005d40 con 0x7fdf9009b740 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: dump: 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.592 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 5 monclient: authenticate success, global_id 4382 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf780070d0 con 0x7fdf9009b740 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 monclient: _renew_subs 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78007490 con 0x7fdf9009b740 2024-09-18T10:09:25.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9008eac0 con 0x7fdf9009b740 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 10 monclient: dump: 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.594 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8e7fc700 4 set_mon_vals no callback set 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 monclient: _renew_subs 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9008f140 con 0x7fdf9009b740 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdf7801b3b0 con 0x7fdf9009b740 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 0x7fdf90087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900648a0 con 0x7fdf9008b910 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdf70035ac0 0x7fdf70037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fdf780476a0 con 0x7fdf9009b740 2024-09-18T10:09:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 0x7fdf90087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient: get_auth_request con 0x7fdf9008b910 auth_method 0 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95263700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdf70035ac0 0x7fdf70037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf95263700 10 monclient: get_auth_request con 0x7fdf70035ac0 auth_method 0 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-09-18T10:09:25.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.576+0000 7fdf94a62700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf9008e390 con 0x7fdf9008b910 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf94a62700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 msgr2=0x7fdf90087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf94a62700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 0x7fdf90087fe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006610 tx=0x7fdf80009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000ed10 con 0x7fdf9008b910 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 10 monclient: dump: 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.600 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf95263700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdf70035ac0 0x7fdf70037f70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b020 tx=0x7fdf88018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdf8000ee70 con 0x7fdf9008b910 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 msgr2=0x7fdf90087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 0x7fdf90087fe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006610 tx=0x7fdf80009220 comp rx=0 tx=0).stop 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 msgr2=0x7fdf90087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:25.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.580+0000 7fdf8cff9700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf9008b910 0x7fdf90087fe0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdf80006610 tx=0x7fdf80009220 comp rx=0 tx=0).stop 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf95a64700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf95a64700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 0x7fdf90088920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf95a64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9008ec50 con 0x7fdf90088520 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 0x7fdf90088920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient: get_auth_request con 0x7fdf90088520 auth_method 0 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-18T10:09:25.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.588+0000 7fdf94a62700 1 -- 172.21.15.142:0/1280935157 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdf9008c7b0 con 0x7fdf90088520 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf94a62700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 msgr2=0x7fdf90088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf94a62700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 0x7fdf90088920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf80017000 tx=0x7fdf80015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80025070 con 0x7fdf90088520 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 10 monclient: dump: 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.603 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fdf80020ae0 con 0x7fdf90088520 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 msgr2=0x7fdf90088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 0x7fdf90088920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf80017000 tx=0x7fdf80015000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 msgr2=0x7fdf90088920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf8cff9700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf90088520 0x7fdf90088920 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.604 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 12, 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:25.605 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.606 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:25.607 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:25.608 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:25.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:25.610 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:25.611 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.612 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:25.614 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.615 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.617 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.617 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:25.617 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:25.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdf70035ac0 msgr2=0x7fdf70037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdf70035ac0 0x7fdf70037f70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800b020 tx=0x7fdf88018000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 10 monclient: shutdown 2024-09-18T10:09:25.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 msgr2=0x7fdf9008ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 --2- 172.21.15.142:0/1280935157 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf9009b740 0x7fdf9008ae90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800c530 tx=0x7fdf78007f90 comp rx=0 tx=0).stop 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95263700 1 -- 172.21.15.142:0/1280935157 reap_dead start 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95263700 1 -- 172.21.15.142:0/1280935157 reap_dead start 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 shutdown_connections 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 >> 172.21.15.142:0/1280935157 conn(0x7fdf90070ef0 msgr2=0x7fdf9009de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 shutdown_connections 2024-09-18T10:09:25.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.592+0000 7fdf95a64700 1 -- 172.21.15.142:0/1280935157 wait complete. 2024-09-18T10:09:25.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:09:25.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:09:25.620 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: get_monmap_and_config 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: build_initial_monmap 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: monmap: 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:25.732104+0000 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:25.732104+0000 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:25.733 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:25.734 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 Processor -- start 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 -- start start 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: init 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd25c09c760 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd25c09cb30 0x7fd25c09cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-a con 0x7fd25c09cb30 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 0x7fd25c135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-c con 0x7fd25c133870 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25c098b00 con 0x7fd25c09c760 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25c098d20 con 0x7fd25c09cb30 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25c09a100 con 0x7fd25c133870 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd26339a700 10 monclient(hunting): authenticate will time out at 854.657973s 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd262b99700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 msgr2=0x7fd25c135c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd262b99700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 0x7fd25c135c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd25c09cb30 0x7fd25c09cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd262398700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd25c09c760 0x7fd25c09b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:35134/0 (socket says 172.21.15.142:35134) 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd262398700 1 -- 172.21.15.142:0/4016991354 learned_addr learned my addr 172.21.15.142:0/4016991354 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): get_auth_request con 0x7fd25c09cb30 auth_method 0 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 -- 172.21.15.142:0/4016991354 >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 msgr2=0x7fd25c135c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 --2- 172.21.15.142:0/4016991354 >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 0x7fd25c135c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 -- 172.21.15.142:0/4016991354 >> v1:172.21.15.142:6789/0 conn(0x7fd25c09c760 legacy=0x7fd25c09b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 monclient: found mon.noname-a 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.728+0000 7fd261b97700 1 -- 172.21.15.142:0/4016991354 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c09a3e0 con 0x7fd25c09cb30 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261b97700 1 -- 172.21.15.142:0/4016991354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c09cb30 msgr2=0x7fd25c09cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261b97700 1 --2- 172.21.15.142:0/4016991354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c09cb30 0x7fd25c09cf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd25c06fd10 tx=0x7fd25800a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7be25e96a7d35f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 1 -- 172.21.15.142:0/4016991354 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800cc60 con 0x7fd25c09cb30 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: dump: 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 1 -- 172.21.15.142:0/4016991354 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800cdc0 con 0x7fd25c09cb30 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 5 monclient: authenticate success, global_id 4401 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 4 set_mon_vals no callback set 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 1 -- 172.21.15.142:0/4016991354 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258018640 con 0x7fd25c09cb30 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd261396700 10 monclient: dump: 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.746 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 10 monclient: shutdown 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 -- 172.21.15.142:0/4016991354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c09cb30 msgr2=0x7fd25c09cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 --2- 172.21.15.142:0/4016991354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c09cb30 0x7fd25c09cf90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd25c06fd10 tx=0x7fd25800a330 comp rx=0 tx=0).stop 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 -- 172.21.15.142:0/4016991354 shutdown_connections 2024-09-18T10:09:25.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 --2- 172.21.15.142:0/4016991354 >> v2:172.21.15.150:3300/0 conn(0x7fd25c133870 0x7fd25c135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 --2- 172.21.15.142:0/4016991354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c09cb30 0x7fd25c09cf90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 -- 172.21.15.142:0/4016991354 >> 172.21.15.142:0/4016991354 conn(0x7fd25c070ef0 msgr2=0x7fd25c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 -- 172.21.15.142:0/4016991354 shutdown_connections 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 1 -- 172.21.15.142:0/4016991354 wait complete. 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 10 monclient: build_initial_monmap 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.732+0000 7fd26339a700 10 monclient: monmap: 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:25.748 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 Processor -- start 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 -- start start 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient: init 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 0x7fd25c12e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-c con 0x7fd25c09c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd25c09cb30 0x7fd25c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-b con 0x7fd25c09cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c133870 0x7fd25c06df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): picked mon.noname-a con 0x7fd25c133870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:25.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c126680 con 0x7fd25c133870 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c12d690 con 0x7fd25c09cb30 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c128e50 con 0x7fd25c09c760 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd26339a700 10 monclient(hunting): authenticate will time out at 854.665933s 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd261b97700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd25c09cb30 msgr2=0x7fd25c06d9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 0x7fd25c12e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd261b97700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd25c09cb30 0x7fd25c06d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 0x7fd25c12e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50216/0 (socket says 172.21.15.142:50216) 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 -- 172.21.15.142:0/1395388566 learned_addr learned my addr 172.21.15.142:0/1395388566 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): get_auth_request con 0x7fd25c09c760 auth_method 0 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c133870 0x7fd25c06df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): get_auth_request con 0x7fd25c133870 auth_method 0 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262b99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-18T10:09:25.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd25c09cb30 msgr2=0x7fd25c06d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd25c09cb30 0x7fd25c06d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c133870 msgr2=0x7fd25c06df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd25c133870 0x7fd25c06df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient: _finish_hunting 0 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 monclient: found mon.noname-c 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.736+0000 7fd262398700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25800a5d0 con 0x7fd25c09c760 2024-09-18T10:09:25.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd262398700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 msgr2=0x7fd25c12e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd262398700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 0x7fd25c12e080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c5e0 tx=0x7fd24c004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c005d40 con 0x7fd25c09c760 2024-09-18T10:09:25.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: dump: 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 5 monclient: authenticate success, global_id 4385 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24c0070d0 con 0x7fd25c09c760 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 monclient: _renew_subs 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c126df0 con 0x7fd25c09c760 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24c007280 con 0x7fd25c09c760 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 10 monclient: dump: 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 monclient: _renew_subs 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd260b95700 4 set_mon_vals no callback set 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c128190 con 0x7fd25c09c760 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 0x7fd25c06e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c0648a0 con 0x7fd25c127230 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd24c017020 con 0x7fd25c09c760 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 0x7fd25c06e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient: get_auth_request con 0x7fd25c127230 auth_method 0 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd240035a70 0x7fd240037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd24c04b610 con 0x7fd25c09c760 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd261b97700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd25c128600 con 0x7fd25c127230 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd262b99700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd240035a70 0x7fd240037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.740+0000 7fd262b99700 10 monclient: get_auth_request con 0x7fd240035a70 auth_method 0 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd261b97700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 msgr2=0x7fd25c06e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd261b97700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 0x7fd25c06e440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd25c06fd10 tx=0x7fd258019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258010600 con 0x7fd25c127230 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 10 monclient: dump: 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.761 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd262b99700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd240035a70 0x7fd240037f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd25c128540 tx=0x7fd25400d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd258010760 con 0x7fd25c127230 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 msgr2=0x7fd25c06e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 0x7fd25c06e440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd25c06fd10 tx=0x7fd258019000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 msgr2=0x7fd25c06e440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:25.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.744+0000 7fd2527fc700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c127230 0x7fd25c06e440 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd25c06fd10 tx=0x7fd258019000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.756+0000 7fd26339a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.756+0000 7fd26339a700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 0x7fd25c1321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.756+0000 7fd26339a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.756+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c072a80 con 0x7fd25c131dd0 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 0x7fd25c1321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient: get_auth_request con 0x7fd25c131dd0 auth_method 0 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:25.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:25.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:25.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:25.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-18T10:09:25.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 1 -- 172.21.15.142:0/1395388566 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd25c06d700 con 0x7fd25c131dd0 2024-09-18T10:09:25.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 msgr2=0x7fd25c1321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:25.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd261b97700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 0x7fd25c1321d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd25c1288e0 tx=0x7fd258020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258025e50 con 0x7fd25c131dd0 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 10 monclient: dump: 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:25.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fd258023470 con 0x7fd25c131dd0 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 msgr2=0x7fd25c1321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 0x7fd25c1321d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd25c1288e0 tx=0x7fd258020000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 msgr2=0x7fd25c1321d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd2527fc700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c131dd0 0x7fd25c1321d0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd25c1288e0 tx=0x7fd258020000 comp rx=0 tx=0).stop 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 12, 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:25.770 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.771 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:25.773 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:25.774 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.775 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:25.776 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:25.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:25.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:25.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:25.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:25.783 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:25.783 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:25.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd240035a70 msgr2=0x7fd240037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd240035a70 0x7fd240037f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd25c128540 tx=0x7fd25400d040 comp rx=0 tx=0).stop 2024-09-18T10:09:25.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 10 monclient: shutdown 2024-09-18T10:09:25.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:25.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 msgr2=0x7fd25c12e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 1 --2- 172.21.15.142:0/1395388566 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd25c09c760 0x7fd25c12e080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00c5e0 tx=0x7fd24c004a80 comp rx=0 tx=0).stop 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd262b99700 1 -- 172.21.15.142:0/1395388566 reap_dead start 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd262b99700 1 -- 172.21.15.142:0/1395388566 reap_dead start 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.760+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 shutdown_connections 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.764+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 >> 172.21.15.142:0/1395388566 conn(0x7fd25c070ef0 msgr2=0x7fd25c09f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.764+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 shutdown_connections 2024-09-18T10:09:25.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:25.764+0000 7fd26339a700 1 -- 172.21.15.142:0/1395388566 wait complete. 2024-09-18T10:09:25.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:09:25.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:09:45.786 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:09:45.787 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:09:45.787 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:09:45.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:09:45.790 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:09:45.790 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:09:45.790 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:45.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: get_monmap_and_config 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: build_initial_monmap 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: monmap: 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:45.899937+0000 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:45.899937+0000 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:45.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:45.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 Processor -- start 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 -- start start 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: init 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-b con 0x7f97500a3ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 0x7f975006ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-a con 0x7f975006ebc0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f975009de80 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975006bbc0 con 0x7f975009de80 2024-09-18T10:09:45.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975006d120 con 0x7f975006ebc0 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975006d690 con 0x7f97500a3ba0 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f9754c2d700 10 monclient(hunting): authenticate will time out at 874.825755s 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34822/0 (socket says 172.21.15.142:34822) 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 -- 172.21.15.142:0/1499566621 learned_addr learned my addr 172.21.15.142:0/1499566621 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): get_auth_request con 0x7f97500a3ba0 auth_method 0 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:45.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 1 --2- 172.21.15.142:0/1499566621 >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 0x7f975006ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 10 monclient(hunting): get_auth_request con 0x7f975006ebc0 auth_method 0 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:45.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient(hunting): handle_auth_done global_id 4394 payload 274 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 -- 172.21.15.142:0/1499566621 >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 msgr2=0x7f975006ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 --2- 172.21.15.142:0/1499566621 >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 0x7f975006ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 -- 172.21.15.142:0/1499566621 >> v1:172.21.15.142:6789/0 conn(0x7f975009de80 legacy=0x7f975009e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974f7fe700 1 -- 172.21.15.142:0/1499566621 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97500a3f70 con 0x7f97500a3ba0 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.896+0000 7f974effd700 1 --2- 172.21.15.142:0/1499566621 >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 0x7f975006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974f7fe700 1 -- 172.21.15.142:0/1499566621 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 msgr2=0x7f9750130190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974f7fe700 1 --2- 172.21.15.142:0/1499566621 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9740009f20 tx=0x7f9740009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61345a59e392e01 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:45.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 1 -- 172.21.15.142:0/1499566621 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974000e070 con 0x7f97500a3ba0 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: dump: 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:45.910 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 1 -- 172.21.15.142:0/1499566621 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974000bc90 con 0x7f97500a3ba0 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 5 monclient: authenticate success, global_id 4394 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 1 -- 172.21.15.142:0/1499566621 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97400092e0 con 0x7f97500a3ba0 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 4 set_mon_vals no callback set 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:45.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f974e7fc700 10 monclient: dump: 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:45.912 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 10 monclient: shutdown 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- 172.21.15.142:0/1499566621 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 msgr2=0x7f9750130190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1499566621 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9740009f20 tx=0x7f9740009840 comp rx=0 tx=0).stop 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- 172.21.15.142:0/1499566621 shutdown_connections 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1499566621 >> v2:172.21.15.142:3300/0 conn(0x7f975006ebc0 0x7f975006ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1499566621 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97500a3ba0 0x7f9750130190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:45.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- 172.21.15.142:0/1499566621 >> 172.21.15.142:0/1499566621 conn(0x7f9750071680 msgr2=0x7f97500a3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- 172.21.15.142:0/1499566621 shutdown_connections 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- 172.21.15.142:0/1499566621 wait complete. 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 10 monclient: build_initial_monmap 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 10 monclient: monmap: 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 Processor -- start 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 1 -- start start 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.900+0000 7f9754c2d700 10 monclient: init 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-c con 0x7f975006ebc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-b con 0x7f975009de80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 0x7f9750119af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): picked mon.noname-a con 0x7f97500a3ba0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750132db0 con 0x7f97500a3ba0 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975011c4c0 con 0x7f975009de80 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975011c1e0 con 0x7f975006ebc0 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient(hunting): authenticate will time out at 874.832314s 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34826/0 (socket says 172.21.15.142:34826) 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 -- 172.21.15.142:0/1131573690 learned_addr learned my addr 172.21.15.142:0/1131573690 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): get_auth_request con 0x7f975006ebc0 auth_method 0 2024-09-18T10:09:45.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 0x7f9750119af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 10 monclient(hunting): get_auth_request con 0x7f975009de80 auth_method 0 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 10 monclient(hunting): get_auth_request con 0x7f97500a3ba0 auth_method 0 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 msgr2=0x7f975012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 msgr2=0x7f975012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 msgr2=0x7f975012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974effd700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 msgr2=0x7f9750119af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 0x7f9750119af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 -- 172.21.15.142:0/1131573690 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9740009ae0 con 0x7f975006ebc0 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974ffff700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 0x7f9750119af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 msgr2=0x7f975009c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974f7fe700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f974000b6c0 tx=0x7f9740017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740012b00 con 0x7f975006ebc0 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: dump: 2024-09-18T10:09:45.923 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 5 monclient: authenticate success, global_id 4397 2024-09-18T10:09:45.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740012ca0 con 0x7f975006ebc0 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _renew_subs 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975011bf00 con 0x7f975006ebc0 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974001d670 con 0x7f975006ebc0 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f97477fe700 10 monclient: dump: 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:45.925 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f974dffb700 4 set_mon_vals no callback set 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _renew_subs 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.904+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975011b1b0 con 0x7f975006ebc0 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f9754c2d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f9754c2d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f9754c2d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:45.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975009d200 con 0x7f975006ebc0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f974001daa0 con 0x7f975006ebc0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f97477fe700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 0x7f9734037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9740049630 con 0x7f975006ebc0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f974effd700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 0x7f9734037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.908+0000 7f974effd700 10 monclient: get_auth_request con 0x7f9734035ac0 auth_method 0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.912+0000 7f974effd700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 0x7f9734037f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9738006f00 tx=0x7f9738008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.916+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f974001b310 con 0x7f975006ebc0 2024-09-18T10:09:45.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.916+0000 7f97477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:45.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:45.916+0000 7f97477fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:46.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f9754c2d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:46.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f9754c2d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:46.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f9754c2d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:46.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9750072a80 con 0x7f975006ebc0 2024-09-18T10:09:46.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f97477fe700 1 -- 172.21.15.142:0/1131573690 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9740007320 con 0x7f975006ebc0 2024-09-18T10:09:46.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f97477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:46.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.136+0000 7f97477fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:46.144 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 msgr2=0x7f9734037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 0x7f9734037f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9738006f00 tx=0x7f9738008040 comp rx=0 tx=0).stop 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 10 monclient: shutdown 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 msgr2=0x7f975009c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:46.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f974000b6c0 tx=0x7f9740017000 comp rx=0 tx=0).stop 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 shutdown_connections 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9734035ac0 0x7f9734037f70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97500a3ba0 0x7f9750119af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f975009de80 0x7f975012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 --2- 172.21.15.142:0/1131573690 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f975006ebc0 0x7f975009c7f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 >> 172.21.15.142:0/1131573690 conn(0x7f9750071680 msgr2=0x7f9750132fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 shutdown_connections 2024-09-18T10:09:46.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:46.140+0000 7f9754c2d700 1 -- 172.21.15.142:0/1131573690 wait complete. 2024-09-18T10:09:49.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: get_monmap_and_config 2024-09-18T10:09:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: build_initial_monmap 2024-09-18T10:09:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: monmap: 2024-09-18T10:09:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:49.281532+0000 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:49.281532+0000 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:49.283 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:49.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 Processor -- start 2024-09-18T10:09:49.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 -- start start 2024-09-18T10:09:49.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: init 2024-09-18T10:09:49.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:49.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:09:49.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:49.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2009d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:49.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b200989c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:49.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:49.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-a con 0x7f3b200994a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:49.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b20069c70 con 0x7f3b200989c0 2024-09-18T10:09:49.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:49.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2006f9d0 con 0x7f3b200994a0 2024-09-18T10:09:49.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2009dc40 con 0x7f3b2009d870 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b24f41700 10 monclient(hunting): authenticate will time out at 878.207026s 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34842/0 (socket says 172.21.15.142:34842) 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1dd9b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3b200989c0 0x7f3b20098d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:34440/0 (socket says 172.21.15.142:34440) 2024-09-18T10:09:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51834/0 (socket says 172.21.15.142:51834) 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 learned_addr learned my addr 172.21.15.142:0/726994073 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b2009d870 auth_method 0 2024-09-18T10:09:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.276+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:49.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.280+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.150:6789/0 conn(0x7f3b200989c0 legacy=0x7f3b20098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b2009d870 auth_method 0 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:49.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:49.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.480+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b2009d870 auth_method 0 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:49.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:49.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:49.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:49.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:49.884+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:50.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:50.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b2009d870 auth_method 0 2024-09-18T10:09:50.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:50.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:50.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:50.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:50.688+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:52.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): tick 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 monclient(hunting): continuing hunt 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 msgr2=0x7f3b200954b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2009d870 0x7f3b200954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b20136680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b20136680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.150:6789/0 conn(0x7f3b200989c0 legacy=0x7f3b20098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient: _add_conns ranks=[4,0] 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f3b000037c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b00006300 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:52.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2006fc20 con 0x7f3b00006300 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2009dc40 con 0x7f3b000037c0 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b0ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.276+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:52.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.280+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:52.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.280+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.142:6789/0 conn(0x7f3b00006300 legacy=0x7f3b000087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:52.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:52.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:52.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:52.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:52.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:52.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:52.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:52.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:53.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:53.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:53.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:53.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:55.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): tick 2024-09-18T10:09:55.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 monclient(hunting): continuing hunt 2024-09-18T10:09:55.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.142:6789/0 conn(0x7f3b00006300 legacy=0x7f3b000087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 reap_dead start 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f3b200994a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f3b000037c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b0000a760 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:55.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0000b120 con 0x7f3b0000a760 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2009dc40 con 0x7f3b200994a0 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 1 -- 172.21.15.142:0/726994073 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0000acb0 con 0x7f3b000037c0 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b0ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:09:55.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.280+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.150:6789/0 conn(0x7f3b0000a760 legacy=0x7f3b000087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:55.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:55.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.484+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:09:55.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:55.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:55.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:09:56.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request con 0x7f3b000037c0 auth_method 0 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b200994a0 auth_method 0 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:56.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:09:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:09:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:09:56.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.692+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:09:56.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476628727 0 0) 0x7f3b0000b120 con 0x7f3b0000a760 2024-09-18T10:09:56.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:56.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): my global_id is 14098 2024-09-18T10:09:56.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b08003570 con 0x7f3b0000a760 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208835855 0 0) 0x7f3b08003570 con 0x7f3b0000a760 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 msgr2=0x7f3b00005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.150:3300/0 conn(0x7f3b000037c0 0x7f3b00005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 msgr2=0x7f3b0000a1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 --2- 172.21.15.142:0/726994073 >> v2:172.21.15.142:3300/0 conn(0x7f3b200994a0 0x7f3b0000a1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: _finish_hunting 0 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:09:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0000ae70 con 0x7f3b0000a760 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: _finish_auth 0 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b24f41700 5 monclient: authenticate success, global_id 14098 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f3b180099f0 con 0x7f3b0000a760 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b24f41700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: dump: 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.885 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 >> v1:172.21.15.150:6789/0 conn(0x7f3b0000a760 legacy=0x7f3b000087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 0x7f3b08003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): picked mon.a con 0x7f3b08003ab0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:56.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004810 0x7f3b08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): picked mon.b con 0x7f3b08004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b080052e0 0x7f3b0800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): picked mon.c con 0x7f3b080052e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2009dc40 con 0x7f3b08003ab0 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0000acb0 con 0x7f3b08004810 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0800c2b0 con 0x7f3b080052e0 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1d59a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b24f41700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:56.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 0x7f3b08003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b080052e0 0x7f3b0800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b08003ab0 auth_method 0 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b080052e0 auth_method 0 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004810 0x7f3b08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_done global_id 14098 payload 293 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b080052e0 msgr2=0x7f3b0800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b080052e0 0x7f3b0800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004810 msgr2=0x7f3b08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004810 0x7f3b08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 monclient: found mon.a 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08003570 con 0x7f3b08003ab0 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 10 monclient: _finish_auth 0 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1dd9b700 10 monclient: get_auth_request con 0x7f3b08004810 auth_method 0 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08004810 0x7f3b08004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b080052e0 0x7f3b0800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.876+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/726994073 reap_dead start 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.880+0000 7f3b1e59c700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 msgr2=0x7f3b08003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.880+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 0x7f3b08003ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006fd0 tx=0x7f3b14008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb25d032727bd9d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b140108e0 con 0x7f3b08003ab0 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: dump: 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1401e040 con 0x7f3b08003ab0 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 4 set_mon_vals no callback set 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 1 -- 172.21.15.142:0/726994073 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14018da0 con 0x7f3b08003ab0 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1d59a700 10 monclient: dump: 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: shutdown 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 msgr2=0x7f3b08003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 0x7f3b08003ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006fd0 tx=0x7f3b14008040 comp rx=0 tx=0).stop 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- 172.21.15.142:0/726994073 shutdown_connections 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 --2- 172.21.15.142:0/726994073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b08003ab0 0x7f3b08003ec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- 172.21.15.142:0/726994073 >> 172.21.15.142:0/726994073 conn(0x7f3b200a14e0 msgr2=0x7f3b200a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- 172.21.15.142:0/726994073 shutdown_connections 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- 172.21.15.142:0/726994073 wait complete. 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: build_initial_monmap 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: monmap: 2024-09-18T10:09:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 Processor -- start 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- start start 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: init 2024-09-18T10:09:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-b con 0x7f3b08003ab0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2013dc60 0x7f3b2013e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-c con 0x7f3b2013dc60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): picked mon.noname-a con 0x7f3b20089be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b20097b80 con 0x7f3b20089be0 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2013e570 con 0x7f3b08003ab0 2024-09-18T10:09:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2008bc80 con 0x7f3b2013dc60 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b24f41700 10 monclient(hunting): authenticate will time out at 885.815810s 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.884+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2013dc60 0x7f3b2013e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52880/0 (socket says 172.21.15.142:52880) 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:44668/0 (socket says 172.21.15.142:44668) 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 1 -- 172.21.15.142:0/4166494778 learned_addr learned my addr 172.21.15.142:0/4166494778 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request con 0x7f3b20089be0 auth_method 0 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request con 0x7f3b08003ab0 auth_method 0 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request con 0x7f3b2013dc60 auth_method 0 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient(hunting): handle_auth_done global_id 14101 payload 824 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2013dc60 msgr2=0x7f3b2013e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2013dc60 0x7f3b2013e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 msgr2=0x7f3b2008a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 monclient: found mon.noname-b 2024-09-18T10:09:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/4166494778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2009ff20 con 0x7f3b08003ab0 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1ed9d700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 -- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 msgr2=0x7f3b2013d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1e59c700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b08004790 tx=0x7f3b14029000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1400a040 con 0x7f3b08003ab0 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: dump: 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1401c070 con 0x7f3b08003ab0 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 5 monclient: authenticate success, global_id 14101 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 10 monclient: _renew_subs 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2008e550 con 0x7f3b08003ab0 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14024e20 con 0x7f3b08003ab0 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b0e7fc700 10 monclient: dump: 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:56.908 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b1cd99700 4 set_mon_vals no callback set 2024-09-18T10:09:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 10 monclient: _renew_subs 2024-09-18T10:09:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.888+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2008b770 con 0x7f3b08003ab0 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3b1402e400 con 0x7f3b08003ab0 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b0e7fc700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 0x7f3b0003b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3b14000e40 con 0x7f3b08003ab0 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b24f41700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b24f41700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b24f41700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 0x7f3b0003b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b200696a0 con 0x7f3b08003ab0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b1dd9b700 10 monclient: get_auth_request con 0x7f3b00038d40 auth_method 0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.892+0000 7f3b1dd9b700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 0x7f3b0003b1f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b2008b370 tx=0x7f3b10007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.900+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3b14032070 con 0x7f3b08003ab0 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.900+0000 7f3b0e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:56.900+0000 7f3b0e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:57.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b24f41700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:09:57.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b24f41700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:57.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b24f41700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:57.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3b2009fa40 con 0x7f3b08003ab0 2024-09-18T10:09:57.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b0e7fc700 1 -- 172.21.15.142:0/4166494778 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f3b14043620 con 0x7f3b08003ab0 2024-09-18T10:09:57.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b0e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:09:57.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.132+0000 7f3b0e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:57.140 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 msgr2=0x7f3b0003b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 0x7f3b0003b1f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3b2008b370 tx=0x7f3b10007f90 comp rx=0 tx=0).stop 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 10 monclient: shutdown 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 msgr2=0x7f3b2013d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b08004790 tx=0x7f3b14029000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 shutdown_connections 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b00038d40 0x7f3b0003b1f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b20089be0 0x7f3b2008a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2013dc60 0x7f3b2013e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 --2- 172.21.15.142:0/4166494778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b08003ab0 0x7f3b2013d720 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 >> 172.21.15.142:0/4166494778 conn(0x7f3b200a14e0 msgr2=0x7f3b20095c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 shutdown_connections 2024-09-18T10:09:57.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.136+0000 7f3b24f41700 1 -- 172.21.15.142:0/4166494778 wait complete. 2024-09-18T10:09:57.154 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:09:57.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:09:57.154 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.252+0000 7f7c69f18700 10 monclient: get_monmap_and_config 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.252+0000 7f7c69f18700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.252+0000 7f7c69f18700 10 monclient: monmap: 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:57.259451+0000 2024-09-18T10:09:57.260 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:57.259451+0000 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:57.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 Processor -- start 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 -- start start 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient: init 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T10:09:57.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c6409abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c6409b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 0x7f7c64138170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-b con 0x7f7c64135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6409dc90 con 0x7f7c6409b740 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6409afa0 con 0x7f7c6409abd0 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6409b500 con 0x7f7c64135d90 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69f18700 10 monclient(hunting): authenticate will time out at 886.184935s 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c5bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7c6409b740 0x7f7c64134670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:34810/0 (socket says 172.21.15.142:34810) 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 0x7f7c64138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c5bfff700 1 -- 172.21.15.142:0/3933198762 learned_addr learned my addr 172.21.15.142:0/3933198762 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): get_auth_request con 0x7f7c64135d90 auth_method 0 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 1 -- 172.21.15.142:0/3933198762 >> v1:172.21.15.150:6789/0 conn(0x7f7c6409abd0 legacy=0x7f7c640960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 1 -- 172.21.15.142:0/3933198762 >> v1:172.21.15.142:6789/0 conn(0x7f7c6409b740 legacy=0x7f7c64134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 1 monclient: found mon.noname-b 2024-09-18T10:09:57.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.256+0000 7f7c69717700 1 -- 172.21.15.142:0/3933198762 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6406f810 con 0x7f7c64135d90 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69717700 1 -- 172.21.15.142:0/3933198762 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 msgr2=0x7f7c64138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69717700 1 --2- 172.21.15.142:0/3933198762 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 0x7f7c64138170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009e90 tx=0x7f7c5c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f4bd93afc17b676 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 1 -- 172.21.15.142:0/3933198762 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c00e070 con 0x7f7c64135d90 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:57.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: dump: 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 5 monclient: authenticate success, global_id 4403 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 1 -- 172.21.15.142:0/3933198762 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c012430 con 0x7f7c64135d90 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 4 set_mon_vals no callback set 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 1 -- 172.21.15.142:0/3933198762 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0127f0 con 0x7f7c64135d90 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c5b7fe700 10 monclient: dump: 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: shutdown 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- 172.21.15.142:0/3933198762 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 msgr2=0x7f7c64138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 --2- 172.21.15.142:0/3933198762 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 0x7f7c64138170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009e90 tx=0x7f7c5c009780 comp rx=0 tx=0).stop 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- 172.21.15.142:0/3933198762 shutdown_connections 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 --2- 172.21.15.142:0/3933198762 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c64135d90 0x7f7c64138170 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- 172.21.15.142:0/3933198762 >> 172.21.15.142:0/3933198762 conn(0x7f7c64070ef0 msgr2=0x7f7c640712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- 172.21.15.142:0/3933198762 shutdown_connections 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- 172.21.15.142:0/3933198762 wait complete. 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: monmap: 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:57.274 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 Processor -- start 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- start start 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: init 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6409abd0 0x7f7c6408b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-b con 0x7f7c6409abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 0x7f7c640942c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-c con 0x7f7c6409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 0x7f7c64088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): picked mon.noname-a con 0x7f7c64135d90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6408cd80 con 0x7f7c64135d90 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6409d240 con 0x7f7c6409abd0 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6409f3e0 con 0x7f7c6409b740 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.260+0000 7f7c69f18700 10 monclient(hunting): authenticate will time out at 886.191882s 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 0x7f7c64088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 0x7f7c640942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6409abd0 0x7f7c6408b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 0x7f7c64088fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52882/0 (socket says 172.21.15.142:52882) 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): get_auth_request con 0x7f7c6409b740 auth_method 0 2024-09-18T10:09:57.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 1 -- 172.21.15.142:0/494989006 learned_addr learned my addr 172.21.15.142:0/494989006 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): get_auth_request con 0x7f7c6409abd0 auth_method 0 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): get_auth_request con 0x7f7c64135d90 auth_method 0 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6409abd0 msgr2=0x7f7c6408b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6409abd0 0x7f7c6408b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 msgr2=0x7f7c64088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 0x7f7c64088fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c009a20 con 0x7f7c6409b740 2024-09-18T10:09:57.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69717700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c64135d90 0x7f7c64088fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:57.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 msgr2=0x7f7c640942c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5bfff700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 0x7f7c640942c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c530 tx=0x7f7c54007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54005d40 con 0x7f7c6409b740 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: dump: 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.283 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c540070d0 con 0x7f7c6409b740 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 5 monclient: authenticate success, global_id 4406 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 10 monclient: _renew_subs 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6408c430 con 0x7f7c6409b740 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54007490 con 0x7f7c6409b740 2024-09-18T10:09:57.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c58ff9700 10 monclient: dump: 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 10 monclient: _renew_subs 2024-09-18T10:09:57.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c5affd700 4 set_mon_vals no callback set 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.264+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6408b1d0 con 0x7f7c6409b740 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69f18700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69f18700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 0x7f7c640898c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c640648a0 con 0x7f7c640894f0 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 0x7f7c640898c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient: get_auth_request con 0x7f7c640894f0 auth_method 0 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c5bfff700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 0x7f7c640942c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c530 tx=0x7f7c54007f90 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f7c54047680 osd_map(17..17 src has 1..17) v4 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7c5401b3b0 con 0x7f7c6409b740 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c68f16700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c64089e00 con 0x7f7c640894f0 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c58ff9700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c50035b10 0x7f7c50037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69717700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c50035b10 0x7f7c50037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.268+0000 7f7c69717700 10 monclient: get_auth_request con 0x7f7c50035b10 auth_method 0 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c68f16700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 msgr2=0x7f7c640898c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c68f16700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 0x7f7c640898c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60014890 con 0x7f7c640894f0 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 10 monclient: dump: 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.288 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c69717700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c50035b10 0x7f7c50037fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009de0 tx=0x7f7c5c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7c600149f0 con 0x7f7c640894f0 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 msgr2=0x7f7c640898c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 0x7f7c640898c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 msgr2=0x7f7c640898c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.272+0000 7f7c58ff9700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c640894f0 0x7f7c640898c0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60007ea0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c69f18700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:57.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c69f18700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 0x7f7c6406c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c69f18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6408c0c0 con 0x7f7c6406c370 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 0x7f7c6406c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient: get_auth_request con 0x7f7c6406c370 auth_method 0 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.280+0000 7f7c68f16700 1 -- 172.21.15.142:0/494989006 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7c64072a80 con 0x7f7c6406c370 2024-09-18T10:09:57.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c68f16700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 msgr2=0x7f7c6406c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c68f16700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 0x7f7c6406c770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6001f810 con 0x7f7c6406c370 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 10 monclient: dump: 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.293 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f7c60024070 con 0x7f7c6406c370 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 msgr2=0x7f7c6406c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 0x7f7c6406c770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60019000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 msgr2=0x7f7c6406c770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c6406c370 0x7f7c6406c770 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000f7b0 tx=0x7f7c60019000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.295 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.296 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.297 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:57.298 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.299 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.300 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.301 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:57.302 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.304 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.306 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:57.307 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c50035b10 msgr2=0x7f7c50037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c50035b10 0x7f7c50037fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009de0 tx=0x7f7c5c018000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 10 monclient: shutdown 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 msgr2=0x7f7c640942c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 --2- 172.21.15.142:0/494989006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c6409b740 0x7f7c640942c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c530 tx=0x7f7c54007f90 comp rx=0 tx=0).stop 2024-09-18T10:09:57.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69717700 1 -- 172.21.15.142:0/494989006 reap_dead start 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c58ff9700 1 -- 172.21.15.142:0/494989006 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7f7c54047680 con 0x7f7c6409b740 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69717700 1 -- 172.21.15.142:0/494989006 reap_dead start 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 shutdown_connections 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 >> 172.21.15.142:0/494989006 conn(0x7f7c64070ef0 msgr2=0x7f7c6409de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 shutdown_connections 2024-09-18T10:09:57.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.284+0000 7f7c69f18700 1 -- 172.21.15.142:0/494989006 wait complete. 2024-09-18T10:09:57.310 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:57.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:09:57.311 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.412+0000 7fd2fc8b9700 10 monclient: get_monmap_and_config 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.412+0000 7fd2fc8b9700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.412+0000 7fd2fc8b9700 10 monclient: monmap: 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:57.419365+0000 2024-09-18T10:09:57.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:57.419365+0000 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:57.421 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 Processor -- start 2024-09-18T10:09:57.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 -- start start 2024-09-18T10:09:57.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient: init 2024-09-18T10:09:57.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2f80999e0 0x7fd2f80a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2f80999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2f809c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 0x7fd2f8139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2f8137910 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80a20c0 con 0x7fd2f809c410 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f809a000 con 0x7fd2f8137910 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80a1c80 con 0x7fd2f80999e0 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2fc8b9700 10 monclient(hunting): authenticate will time out at 886.345037s 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 0x7fd2f8139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2f80999e0 0x7fd2f80a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 0x7fd2f8139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52922/0 (socket says 172.21.15.142:52922) 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 1 -- 172.21.15.142:0/130441063 learned_addr learned my addr 172.21.15.142:0/130441063 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request con 0x7fd2f80999e0 auth_method 0 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 10 monclient(hunting): get_auth_request con 0x7fd2f8137910 auth_method 0 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_done global_id 14107 payload 274 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 -- 172.21.15.142:0/130441063 >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 msgr2=0x7fd2f8139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/130441063 >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 0x7fd2f8139cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 -- 172.21.15.142:0/130441063 >> v1:172.21.15.142:6789/0 conn(0x7fd2f809c410 legacy=0x7fd2f80a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:57.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.416+0000 7fd2f77fe700 1 -- 172.21.15.142:0/130441063 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f809a580 con 0x7fd2f80999e0 2024-09-18T10:09:57.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f77fe700 1 -- 172.21.15.142:0/130441063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 msgr2=0x7fd2f80a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/130441063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f80a0180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8069c60 tx=0x7fd2e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f53d52f549047f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 1 -- 172.21.15.142:0/130441063 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000b8c0 con 0x7fd2f80999e0 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: dump: 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.435 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 5 monclient: authenticate success, global_id 14107 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 1 -- 172.21.15.142:0/130441063 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000ba20 con 0x7fd2f80999e0 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:57.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 4 set_mon_vals no callback set 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 1 -- 172.21.15.142:0/130441063 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e000bdc0 con 0x7fd2f80999e0 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2f67fc700 10 monclient: dump: 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.437 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 10 monclient: shutdown 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/130441063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 msgr2=0x7fd2f80a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/130441063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f80a0180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8069c60 tx=0x7fd2e00091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/130441063 shutdown_connections 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/130441063 >> v2:172.21.15.142:3300/0 conn(0x7fd2f8137910 0x7fd2f8139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/130441063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f80a0180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/130441063 >> 172.21.15.142:0/130441063 conn(0x7fd2f80a5790 msgr2=0x7fd2f80a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/130441063 shutdown_connections 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/130441063 wait complete. 2024-09-18T10:09:57.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 10 monclient: monmap: 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.420+0000 7fd2fc8b9700 1 Processor -- start 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 -- start start 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient: init 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f812c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-b con 0x7fd2f80999e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2f809c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f8137910 0x7fd2f812ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2f8137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:57.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f812f9a0 con 0x7fd2f809c410 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f812fed0 con 0x7fd2f80999e0 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f809f210 con 0x7fd2f8137910 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2fc8b9700 10 monclient(hunting): authenticate will time out at 886.352683s 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f8137910 0x7fd2f812ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52938/0 (socket says 172.21.15.142:52938) 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 -- 172.21.15.142:0/1363930707 learned_addr learned my addr 172.21.15.142:0/1363930707 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): get_auth_request con 0x7fd2f809c410 auth_method 0 2024-09-18T10:09:57.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f812c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request con 0x7fd2f80999e0 auth_method 0 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): get_auth_request con 0x7fd2f8137910 auth_method 0 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f8137910 msgr2=0x7fd2f812ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f8137910 0x7fd2f812ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 msgr2=0x7fd2f812c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f812c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0009420 con 0x7fd2f809c410 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2f80999e0 0x7fd2f812c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 msgr2=0x7fd2f812c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2f6ffd700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c950 tx=0x7fd2e8002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.424+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8012840 con 0x7fd2f809c410 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: dump: 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.444 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e80129c0 con 0x7fd2f809c410 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 5 monclient: authenticate success, global_id 4428 2024-09-18T10:09:57.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 monclient: _renew_subs 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f8130850 con 0x7fd2f809c410 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8012d80 con 0x7fd2f809c410 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 10 monclient: dump: 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.446 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f57fa700 4 set_mon_vals no callback set 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 monclient: _renew_subs 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f81375d0 con 0x7fd2f809c410 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 0x7fd2f806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f812fbd0 con 0x7fd2f812d3f0 2024-09-18T10:09:57.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd2e8011590 con 0x7fd2f809c410 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 0x7fd2f806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient: get_auth_request con 0x7fd2f812d3f0 auth_method 0 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2d8035ac0 0x7fd2d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd2e80485f0 con 0x7fd2f809c410 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-18T10:09:57.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f77fe700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f812f6d0 con 0x7fd2f812d3f0 2024-09-18T10:09:57.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f7fff700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2d8035ac0 0x7fd2d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.428+0000 7fd2f7fff700 10 monclient: get_auth_request con 0x7fd2d8035ac0 auth_method 0 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2f77fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 msgr2=0x7fd2f806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 0x7fd2f806dad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8069c60 tx=0x7fd2e0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2f7fff700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2d8035ac0 0x7fd2d8037f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0006fd0 tx=0x7fd2f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0022070 con 0x7fd2f812d3f0 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 10 monclient: dump: 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.450 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd2e001a650 con 0x7fd2f812d3f0 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 msgr2=0x7fd2f806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 0x7fd2f806dad0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8069c60 tx=0x7fd2e0016000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 msgr2=0x7fd2f806dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.432+0000 7fd2ef7fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f812d3f0 0x7fd2f806dad0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2fc8b9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 0x7fd2f806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2fc8b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80647f0 con 0x7fd2f806e010 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 0x7fd2f806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient: get_auth_request con 0x7fd2f806e010 auth_method 0 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-09-18T10:09:57.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.444+0000 7fd2f77fe700 1 -- 172.21.15.142:0/1363930707 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd2f8072a80 con 0x7fd2f806e010 2024-09-18T10:09:57.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2f77fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 msgr2=0x7fd2f806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2f77fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 0x7fd2f806e410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8064340 tx=0x7fd2e0022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e001b650 con 0x7fd2f806e010 2024-09-18T10:09:57.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 10 monclient: dump: 2024-09-18T10:09:57.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fd2e001a440 con 0x7fd2f806e010 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 msgr2=0x7fd2f806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 0x7fd2f806e410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8064340 tx=0x7fd2e0022000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 msgr2=0x7fd2f806e410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2ef7fe700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2f806e010 0x7fd2f806e410 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8064340 tx=0x7fd2e0022000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.459 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:57.459 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:57.459 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:57.459 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:09:57.459 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.460 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.461 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.463 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.463 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:57.464 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.465 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.466 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.467 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:57.468 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.469 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.470 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.471 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.472 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.473 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.474 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:57.475 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.476 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.477 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.477 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:57.477 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2d8035ac0 msgr2=0x7fd2d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2d8035ac0 0x7fd2d8037f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0006fd0 tx=0x7fd2f0008040 comp rx=0 tx=0).stop 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 10 monclient: shutdown 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 msgr2=0x7fd2f812c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c950 tx=0x7fd2e8002760 comp rx=0 tx=0).stop 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2f7fff700 1 -- 172.21.15.142:0/1363930707 reap_dead start 2024-09-18T10:09:57.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 shutdown_connections 2024-09-18T10:09:57.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 --2- 172.21.15.142:0/1363930707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2f809c410 0x7fd2f812c970 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 >> 172.21.15.142:0/1363930707 conn(0x7fd2f80a5790 msgr2=0x7fd2f809de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 shutdown_connections 2024-09-18T10:09:57.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.448+0000 7fd2fc8b9700 1 -- 172.21.15.142:0/1363930707 wait complete. 2024-09-18T10:09:57.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:57.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:09:57.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: monmap: 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:57.589432+0000 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:57.589432+0000 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.590 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:57.591 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 Processor -- start 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 -- start start 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: init 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c7c06f3b0 0x7f3c7c0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3c7c06f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c7c098f00 addr v1:172.21.15.150:6789/0 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c7c099320 0x7f3c7c136d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3c7c099320 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c09c330 con 0x7f3c7c098f00 2024-09-18T10:09:57.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c09b030 con 0x7f3c7c06f3b0 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c09c530 con 0x7f3c7c099320 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7bfff700 10 monclient(hunting): authenticate will time out at 886.515275s 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c7c099320 0x7f3c7c136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c7c099320 0x7f3c7c136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:44698/0 (socket says 172.21.15.142:44698) 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c7c06f3b0 0x7f3c7c0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/64105505 learned_addr learned my addr 172.21.15.142:0/64105505 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): get_auth_request con 0x7f3c7c099320 auth_method 0 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7a7fc700 1 --1- 172.21.15.142:0/64105505 >> v1:172.21.15.150:6789/0 conn(0x7f3c7c098f00 0x7f3c7c133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:58756/0 (socket says 172.21.15.142:58756) 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): get_auth_request con 0x7f3c7c06f3b0 auth_method 0 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.584+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_done global_id 14113 payload 274 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/64105505 >> v2:172.21.15.142:3300/0 conn(0x7f3c7c06f3b0 msgr2=0x7f3c7c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 --2- 172.21.15.142:0/64105505 >> v2:172.21.15.142:3300/0 conn(0x7f3c7c06f3b0 0x7f3c7c0989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/64105505 >> v1:172.21.15.150:6789/0 conn(0x7f3c7c098f00 legacy=0x7f3c7c133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:57.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/64105505 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c09c0f0 con 0x7f3c7c099320 2024-09-18T10:09:57.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/64105505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c099320 msgr2=0x7f3c7c136d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7b7fe700 1 --2- 172.21.15.142:0/64105505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c099320 0x7f3c7c136d90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c09a7f0 tx=0x7f3c7400b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d664c1b38bdd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 1 -- 172.21.15.142:0/64105505 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7400ddb0 con 0x7f3c7c099320 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: dump: 2024-09-18T10:09:57.607 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.608 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 1 -- 172.21.15.142:0/64105505 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c74010470 con 0x7f3c7c099320 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 5 monclient: authenticate success, global_id 14113 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:57.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 1 -- 172.21.15.142:0/64105505 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74010810 con 0x7f3c7c099320 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 4 set_mon_vals no callback set 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c79ffb700 10 monclient: dump: 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.610 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 10 monclient: shutdown 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 1 -- 172.21.15.142:0/64105505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c099320 msgr2=0x7f3c7c136d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.588+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/64105505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c099320 0x7f3c7c136d90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c09a7f0 tx=0x7f3c7400b4f0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- 172.21.15.142:0/64105505 shutdown_connections 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/64105505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c099320 0x7f3c7c136d90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/64105505 >> v2:172.21.15.142:3300/0 conn(0x7f3c7c06f3b0 0x7f3c7c0989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- 172.21.15.142:0/64105505 >> 172.21.15.142:0/64105505 conn(0x7f3c7c070ef0 msgr2=0x7f3c7c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- 172.21.15.142:0/64105505 shutdown_connections 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- 172.21.15.142:0/64105505 wait complete. 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient: monmap: 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:57.613 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 Processor -- start 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- start start 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient: init 2024-09-18T10:09:57.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06f3b0 0x7f3c7c089600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f3c7c06f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c7c098f00 0x7f3c7c089b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f3c7c098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 0x7f3c7c08a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f3c7c099320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c08b6a0 con 0x7f3c7c098f00 2024-09-18T10:09:57.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c097020 con 0x7f3c7c06f3b0 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c06fa20 con 0x7f3c7c099320 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7bfff700 10 monclient(hunting): authenticate will time out at 886.523270s 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c7c098f00 0x7f3c7c089b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 0x7f3c7c08a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c7c098f00 0x7f3c7c089b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52942/0 (socket says 172.21.15.142:52942) 2024-09-18T10:09:57.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06f3b0 0x7f3c7c089600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 1 -- 172.21.15.142:0/2585456693 learned_addr learned my addr 172.21.15.142:0/2585456693 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): get_auth_request con 0x7f3c7c098f00 auth_method 0 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): get_auth_request con 0x7f3c7c099320 auth_method 0 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): get_auth_request con 0x7f3c7c06f3b0 auth_method 0 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06f3b0 msgr2=0x7f3c7c089600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06f3b0 0x7f3c7c089600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c7c098f00 msgr2=0x7f3c7c089b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c7c098f00 0x7f3c7c089b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.592+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7400b790 con 0x7f3c7c099320 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 msgr2=0x7f3c7c08a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7b7fe700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 0x7f3c7c08a080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c74017000 tx=0x7f3c7401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74006a90 con 0x7f3c7c099320 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: dump: 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.620 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 5 monclient: authenticate success, global_id 4418 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c74018840 con 0x7f3c7c099320 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 monclient: _renew_subs 2024-09-18T10:09:57.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c08cc10 con 0x7f3c7c099320 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74018c00 con 0x7f3c7c099320 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 10 monclient: dump: 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.622 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c797fa700 4 set_mon_vals no callback set 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 monclient: _renew_subs 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c08bd20 con 0x7f3c7c099320 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3c7400f3d0 con 0x7f3c7c099320 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 0x7f3c7c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.596+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0648a0 con 0x7f3c7c08a5c0 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c68035ac0 0x7f3c68037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 0x7f3c7c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient: get_auth_request con 0x7f3c7c08a5c0 auth_method 0 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3c74014070 con 0x7f3c7c099320 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7a7fc700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c68035ac0 0x7f3c68037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7a7fc700 10 monclient: get_auth_request con 0x7f3c68035ac0 auth_method 0 2024-09-18T10:09:57.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 10 monclient(hunting): handle_auth_done global_id 14119 payload 824 2024-09-18T10:09:57.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c7c08bf70 con 0x7f3c7c08a5c0 2024-09-18T10:09:57.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 msgr2=0x7f3c7c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7affd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 0x7f3c7c06c370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c70011c20 tx=0x7f3c700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c7a7fc700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c68035ac0 0x7f3c68037f70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c005b90 tx=0x7f3c6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000fe40 con 0x7f3c7c08a5c0 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 10 monclient: dump: 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.628 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3c70014440 con 0x7f3c7c08a5c0 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 msgr2=0x7f3c7c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 0x7f3c7c06c370 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c70011c20 tx=0x7f3c700091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 msgr2=0x7f3c7c06c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.600+0000 7f3c62ffd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c08a5c0 0x7f3c7c06c370 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c70011c20 tx=0x7f3c700091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:09:57.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 0x7f3c7c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c094c20 con 0x7f3c7c06c8b0 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 0x7f3c7c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient: get_auth_request con 0x7f3c7c06c8b0 auth_method 0 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 10 monclient(hunting): handle_auth_done global_id 14122 payload 824 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.612+0000 7f3c7affd700 1 -- 172.21.15.142:0/2585456693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c7c072a80 con 0x7f3c7c06c8b0 2024-09-18T10:09:57.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7affd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 msgr2=0x7f3c7c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7affd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 0x7f3c7c06ccb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009940 tx=0x7f3c7001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c700273f0 con 0x7f3c7c06c8b0 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 10 monclient: dump: 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.631 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f3c70023040 con 0x7f3c7c06c8b0 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 msgr2=0x7f3c7c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 0x7f3c7c06ccb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009940 tx=0x7f3c7001c000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 msgr2=0x7f3c7c06ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:09:57.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c62ffd700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c06c8b0 0x7f3c7c06ccb0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009940 tx=0x7f3c7001c000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:09:57.633 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:09:57.634 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.638 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.638 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.638 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.638 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.638 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:09:57.639 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.640 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:09:57.641 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:09:57.642 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.643 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:09:57.644 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:09:57.645 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:09:57.647 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:09:57.648 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:09:57.649 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:09:57.650 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:09:57.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c68035ac0 msgr2=0x7f3c68037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c68035ac0 0x7f3c68037f70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c005b90 tx=0x7f3c6c009040 comp rx=0 tx=0).stop 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 10 monclient: shutdown 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 msgr2=0x7f3c7c08a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 --2- 172.21.15.142:0/2585456693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c7c099320 0x7f3c7c08a080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c74017000 tx=0x7f3c7401a000 comp rx=0 tx=0).stop 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 reap_dead start 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7b7fe700 1 -- 172.21.15.142:0/2585456693 reap_dead start 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 shutdown_connections 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 >> 172.21.15.142:0/2585456693 conn(0x7f3c7c070ef0 msgr2=0x7f3c7c0a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 shutdown_connections 2024-09-18T10:09:57.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.616+0000 7f3c7bfff700 1 -- 172.21.15.142:0/2585456693 wait complete. 2024-09-18T10:09:57.653 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:09:57.653 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:09:57.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:09:57.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: get_monmap_and_config 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: monmap: 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:57.765035+0000 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:57.765035+0000 2024-09-18T10:09:57.766 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:57.767 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 Processor -- start 2024-09-18T10:09:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 -- start start 2024-09-18T10:09:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: init 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8aa809a7f0 0x7f8aa8098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-a con 0x7f8aa809a7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8aa8099220 0x7f8aa809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-c con 0x7f8aa8099220 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8aa809c320 addr v1:172.21.15.142:6789/0 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa809abc0 con 0x7f8aa809c320 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa809d950 con 0x7f8aa809a7f0 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa809db90 con 0x7f8aa8099220 2024-09-18T10:09:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aaefa4700 10 monclient(hunting): authenticate will time out at 886.690544s 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8aa8099220 0x7f8aa809bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8aa809a7f0 0x7f8aa8098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8aa8099220 0x7f8aa809bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:44736/0 (socket says 172.21.15.142:44736) 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 -- 172.21.15.142:0/3666276539 learned_addr learned my addr 172.21.15.142:0/3666276539 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): get_auth_request con 0x7f8aa8099220 auth_method 0 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): get_auth_request con 0x7f8aa809a7f0 auth_method 0 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_done global_id 14125 payload 274 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 -- 172.21.15.142:0/3666276539 >> v2:172.21.15.142:3300/0 conn(0x7f8aa809a7f0 msgr2=0x7f8aa8098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/3666276539 >> v2:172.21.15.142:3300/0 conn(0x7f8aa809a7f0 0x7f8aa8098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 -- 172.21.15.142:0/3666276539 >> v1:172.21.15.142:6789/0 conn(0x7f8aa809c320 legacy=0x7f8aa813d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.760+0000 7f8aad7a1700 1 -- 172.21.15.142:0/3666276539 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8099a30 con 0x7f8aa8099220 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aad7a1700 1 -- 172.21.15.142:0/3666276539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa8099220 msgr2=0x7f8aa809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/3666276539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa8099220 0x7f8aa809bde0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809d6b0 tx=0x7f8aa4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=933e35bfe57f2c14 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 1 -- 172.21.15.142:0/3666276539 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4009e70 con 0x7f8aa8099220 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: dump: 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 1 -- 172.21.15.142:0/3666276539 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa400ba70 con 0x7f8aa8099220 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 1 -- 172.21.15.142:0/3666276539 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400be10 con 0x7f8aa8099220 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 5 monclient: authenticate success, global_id 14125 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 4 set_mon_vals no callback set 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aacfa0700 10 monclient: dump: 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.777 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 10 monclient: shutdown 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 -- 172.21.15.142:0/3666276539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa8099220 msgr2=0x7f8aa809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/3666276539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa8099220 0x7f8aa809bde0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809d6b0 tx=0x7f8aa4009250 comp rx=0 tx=0).stop 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 -- 172.21.15.142:0/3666276539 shutdown_connections 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/3666276539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa8099220 0x7f8aa809bde0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/3666276539 >> v2:172.21.15.142:3300/0 conn(0x7f8aa809a7f0 0x7f8aa8098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 -- 172.21.15.142:0/3666276539 >> 172.21.15.142:0/3666276539 conn(0x7f8aa80a1fe0 msgr2=0x7f8aa8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 -- 172.21.15.142:0/3666276539 shutdown_connections 2024-09-18T10:09:57.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.764+0000 7f8aaefa4700 1 -- 172.21.15.142:0/3666276539 wait complete. 2024-09-18T10:09:57.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient: build_initial_monmap 2024-09-18T10:09:57.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient: monmap: 2024-09-18T10:09:57.783 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:57.783 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 Processor -- start 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 -- start start 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient: init 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:57.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-a con 0x7f8aa8099220 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-c con 0x7f8aa809a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa809c320 0x7f8aa806b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): picked mon.noname-b con 0x7f8aa809c320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806f970 con 0x7f8aa8099220 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8097540 con 0x7f8aa809c320 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806d8f0 con 0x7f8aa809a7f0 2024-09-18T10:09:57.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aaefa4700 10 monclient(hunting): authenticate will time out at 886.698866s 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52962/0 (socket says 172.21.15.142:52962) 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): get_auth_request con 0x7f8aa809a7f0 auth_method 0 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 1 -- 172.21.15.142:0/371743329 learned_addr learned my addr 172.21.15.142:0/371743329 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): get_auth_request con 0x7f8aa8099220 auth_method 0 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa809c320 0x7f8aa806b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): get_auth_request con 0x7f8aa809c320 auth_method 0 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aae7a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-09-18T10:09:57.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 -- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa809c320 msgr2=0x7f8aa806b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa809c320 0x7f8aa806b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 -- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 msgr2=0x7f8aa8145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient: _finish_hunting 0 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 monclient: found mon.noname-c 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aad7a1700 1 -- 172.21.15.142:0/371743329 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa40094f0 con 0x7f8aa809a7f0 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.768+0000 7f8aadfa2700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aad7a1700 1 -- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 msgr2=0x7f8aa8138950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4002000 tx=0x7f8aa4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4024440 con 0x7f8aa809a7f0 2024-09-18T10:09:57.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: dump: 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: _finish_auth 0 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 5 monclient: authenticate success, global_id 4421 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa40245a0 con 0x7f8aa809a7f0 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4024960 con 0x7f8aa809a7f0 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _renew_subs 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa806d230 con 0x7f8aa809a7f0 2024-09-18T10:09:57.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9ffff700 4 set_mon_vals no callback set 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 10 monclient: dump: 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:57.792 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _renew_subs 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa806cd20 con 0x7f8aa809a7f0 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa8072a80 con 0x7f8aa809a7f0 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8aad7a1700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4002000 tx=0x7f8aa4016000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f8aa404a020 osd_map(17..17 src has 1..17) v4 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.772+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8aa401a430 con 0x7f8aa809a7f0 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.776+0000 7f8a9dffb700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 0x7f8a8c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:57.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.776+0000 7f8aadfa2700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 0x7f8a8c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:57.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.776+0000 7f8aadfa2700 10 monclient: get_auth_request con 0x7f8a8c035ac0 auth_method 0 2024-09-18T10:09:57.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.776+0000 7f8aadfa2700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 0x7f8a8c037f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009fd0 tx=0x7f8a980091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:57.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:57.780+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f8aa404a020 con 0x7f8aa809a7f0 2024-09-18T10:09:58.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.604+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8aa402c750 con 0x7f8aa809a7f0 2024-09-18T10:09:58.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.604+0000 7f8a9dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:09:58.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.604+0000 7f8a9dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:09:58.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8aaefa4700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:09:58.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8aaefa4700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:09:58.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8aaefa4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:09:58.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8aa8072a80 con 0x7f8aa809a7f0 2024-09-18T10:09:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8a9dffb700 1 -- 172.21.15.142:0/371743329 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f8aa402c390 con 0x7f8aa809a7f0 2024-09-18T10:09:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8a9dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:09:58.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.828+0000 7f8a9dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:09:58.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=10,kv=8,logm=73} crc {auth=3382279806,config=1228700967,health=3288728195,kv=4261215878,logm=611386610}) 2024-09-18T10:09:58.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=6,config=1,health=12,kv=8,logm=73} crc {auth=2735820302,config=1228700967,health=861809967,kv=4261215878,logm=3692278889}) 2024-09-18T10:09:58.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=10,kv=8,logm=73} crc {auth=3382279806,config=1228700967,health=3288728195,kv=4261215878,logm=611386610}) 2024-09-18T10:09:58.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=12,kv=8,logm=70} crc {auth=3382279806,config=3140514009,health=861809967,kv=4261215878,logm=739549281}) 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623872545}) 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2868466120}) 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623872545}) 2024-09-18T10:09:58.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973381983}) 2024-09-18T10:09:58.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695849837}) 2024-09-18T10:09:58.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4279314452}) 2024-09-18T10:09:58.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695849837}) 2024-09-18T10:09:58.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468717903}) 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 msgr2=0x7f8a8c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 0x7f8a8c037f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009fd0 tx=0x7f8a980091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 10 monclient: shutdown 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 msgr2=0x7f8aa8138950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4002000 tx=0x7f8aa4016000 comp rx=0 tx=0).stop 2024-09-18T10:09:58.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 shutdown_connections 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a8c035ac0 0x7f8a8c037f70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8aa809c320 0x7f8aa806b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8aa809a7f0 0x7f8aa8138950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 --2- 172.21.15.142:0/371743329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8aa8099220 0x7f8aa8145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 >> 172.21.15.142:0/371743329 conn(0x7f8aa80a1fe0 msgr2=0x7f8aa809f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 shutdown_connections 2024-09-18T10:09:58.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:58.836+0000 7f8aaefa4700 1 -- 172.21.15.142:0/371743329 wait complete. 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=2152828210,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1632291871,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=4071715846}) 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=36} crc {logm=446727575,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=766905293,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=2927750135,osd_snap=2673976383,osdmap=3307915459}) 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=2152828210,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1632291871,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=4071715846}) 2024-09-18T10:09:58.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:09:58.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=32} crc {logm=1245605721,mdsmap=1531151339,mgr=2679965908,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1632291871,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=2927750135,osd_snap=454132730,osdmap=3963922778}) 2024-09-18T10:09:58.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:09:59.858 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:09:59.858 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: get_monmap_and_config 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: build_initial_monmap 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: monmap: 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:09:59.971921+0000 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:09:59.971921+0000 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:59.973 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 Processor -- start 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 -- start start 2024-09-18T10:09:59.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: init 2024-09-18T10:09:59.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:59.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:09:59.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-b con 0x7f6bc40a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bc412f810 0x7f6bc409f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-c con 0x7f6bc412f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 0x7f6bc41320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-a con 0x7f6bc40a02d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc409d990 con 0x7f6bc40a02d0 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc409dc70 con 0x7f6bc412f810 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40a20c0 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:59.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc9032700 10 monclient(hunting): authenticate will time out at 888.897410s 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bc412f810 0x7f6bc409f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 0x7f6bc41320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53712/0 (socket says 172.21.15.142:53712) 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bc412f810 0x7f6bc409f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:44754/0 (socket says 172.21.15.142:44754) 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 1 -- 172.21.15.142:0/4176779524 learned_addr learned my addr 172.21.15.142:0/4176779524 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): get_auth_request con 0x7f6bc40a1cf0 auth_method 0 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): get_auth_request con 0x7f6bc412f810 auth_method 0 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): get_auth_request con 0x7f6bc40a02d0 auth_method 0 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_done global_id 14131 payload 274 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 -- 172.21.15.142:0/4176779524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 msgr2=0x7f6bc412f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 -- 172.21.15.142:0/4176779524 >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 msgr2=0x7f6bc41320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/4176779524 >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 0x7f6bc41320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 monclient: found mon.noname-c 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc37fe700 1 -- 172.21.15.142:0/4176779524 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc409f9c0 con 0x7f6bc412f810 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc3fff700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:59.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.968+0000 7f6bc8831700 1 --2- 172.21.15.142:0/4176779524 >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 0x7f6bc41320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc37fe700 1 -- 172.21.15.142:0/4176779524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc412f810 msgr2=0x7f6bc409f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc412f810 0x7f6bc409f480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a6930 tx=0x7f6bac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fe4e1a697a37ab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 1 -- 172.21.15.142:0/4176779524 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac011840 con 0x7f6bc412f810 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: dump: 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:59.984 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 5 monclient: authenticate success, global_id 14131 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 1 -- 172.21.15.142:0/4176779524 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac0119c0 con 0x7f6bc412f810 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 10 monclient: get_monmap_and_config success 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 4 set_mon_vals no callback set 2024-09-18T10:09:59.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 1 -- 172.21.15.142:0/4176779524 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac011d80 con 0x7f6bc412f810 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc2ffd700 10 monclient: dump: 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 10 monclient: shutdown 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 -- 172.21.15.142:0/4176779524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc412f810 msgr2=0x7f6bc409f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc412f810 0x7f6bc409f480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40a6930 tx=0x7f6bac0091a0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 -- 172.21.15.142:0/4176779524 shutdown_connections 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 --2- 172.21.15.142:0/4176779524 >> v2:172.21.15.142:3300/0 conn(0x7f6bc40a02d0 0x7f6bc41320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc412f810 0x7f6bc409f480 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 --2- 172.21.15.142:0/4176779524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc40a1cf0 0x7f6bc412f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 -- 172.21.15.142:0/4176779524 >> 172.21.15.142:0/4176779524 conn(0x7f6bc4071820 msgr2=0x7f6bc40a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 -- 172.21.15.142:0/4176779524 shutdown_connections 2024-09-18T10:09:59.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 1 -- 172.21.15.142:0/4176779524 wait complete. 2024-09-18T10:09:59.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 10 monclient: build_initial_monmap 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.972+0000 7f6bc9032700 10 monclient: monmap: 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 Processor -- start 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 -- start start 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient: init 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient: _reopen_session rank -1 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 0x7f6bc409b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-a con 0x7f6bc40a02d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 0x7f6bc409b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-b con 0x7f6bc40a1cf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc412f810 0x7f6bc409bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): picked mon.noname-c con 0x7f6bc412f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc406ba80 con 0x7f6bc40a02d0 2024-09-18T10:09:59.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc406d340 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): start opening mon connection 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40a5160 con 0x7f6bc412f810 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): _renew_subs 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc9032700 10 monclient(hunting): authenticate will time out at 888.905682s 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 0x7f6bc409b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc412f810 0x7f6bc409bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 0x7f6bc409b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52972/0 (socket says 172.21.15.142:52972) 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 0x7f6bc409b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 1 -- 172.21.15.142:0/2176334671 learned_addr learned my addr 172.21.15.142:0/2176334671 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): get_auth_request con 0x7f6bc40a02d0 auth_method 0 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): get_auth_request con 0x7f6bc412f810 auth_method 0 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): get_auth_request con 0x7f6bc40a1cf0 auth_method 0 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc8831700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient(hunting): handle_auth_done global_id 14134 payload 824 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 -- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc412f810 msgr2=0x7f6bc409bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc412f810 0x7f6bc409bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 -- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 msgr2=0x7f6bc409b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 0x7f6bc409b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 monclient: found mon.noname-b 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:09:59.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.976+0000 7f6bc37fe700 1 -- 172.21.15.142:0/2176334671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc009a20 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc37fe700 1 -- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 msgr2=0x7f6bc409b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc37fe700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 0x7f6bc409b5a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009fd0 tx=0x7f6bac017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac01f070 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: dump: 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:59.997 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: _finish_auth 0 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 5 monclient: authenticate success, global_id 14134 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac010c40 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac01a420 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _renew_subs 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb40094f0 con 0x7f6bc40a1cf0 2024-09-18T10:09:59.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc27fc700 4 set_mon_vals no callback set 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 10 monclient: dump: 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:09:59.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _renew_subs 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac009420 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc4072a80 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6bac01a7d0 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 0x7f6ba4037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.980+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f6bac048fe0 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.984+0000 7f6bc3fff700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 0x7f6ba4037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.984+0000 7f6bc3fff700 10 monclient: get_auth_request con 0x7f6ba4035af0 auth_method 0 2024-09-18T10:10:00.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.984+0000 7f6bc3fff700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 0x7f6ba4037fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4006e80 tx=0x7f6bb4009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.996+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6bac01aa80 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.996+0000 7f6bbbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:09:59.996+0000 7f6bbbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:00.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bc9032700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:00.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bc9032700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:00.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bc9032700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6bc40a1a90 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bbbfff700 1 -- 172.21.15.142:0/2176334671 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f6bac015070 con 0x7f6bc40a1cf0 2024-09-18T10:10:00.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bbbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:00.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.220+0000 7f6bbbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:00.227 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:00.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 msgr2=0x7f6ba4037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 0x7f6ba4037fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4006e80 tx=0x7f6bb4009250 comp rx=0 tx=0).stop 2024-09-18T10:10:00.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 10 monclient: shutdown 2024-09-18T10:10:00.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 msgr2=0x7f6bc409b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 0x7f6bc409b5a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009fd0 tx=0x7f6bac017000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 shutdown_connections 2024-09-18T10:10:00.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ba4035af0 0x7f6ba4037fa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bc412f810 0x7f6bc409bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bc40a1cf0 0x7f6bc409b5a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 --2- 172.21.15.142:0/2176334671 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bc40a02d0 0x7f6bc409b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 >> 172.21.15.142:0/2176334671 conn(0x7f6bc4071820 msgr2=0x7f6bc412ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 shutdown_connections 2024-09-18T10:10:00.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.224+0000 7f6bc9032700 1 -- 172.21.15.142:0/2176334671 wait complete. 2024-09-18T10:10:00.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:10:00.246 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:10:00.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:10:00.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: monmap: 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.359 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:00.356101+0000 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:00.356101+0000 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:00.360 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 Processor -- start 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 -- start start 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: init 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7118098140 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f711809b5b0 0x7f71180719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): picked mon.noname-a con 0x7f711809b5b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7118099060 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711809ab60 con 0x7f7118099060 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711809a880 con 0x7f7118098140 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711809ae40 con 0x7f711809b5b0 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f711759e700 10 monclient(hunting): authenticate will time out at 889.281568s 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f711809b5b0 0x7f71180719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f711809b5b0 0x7f71180719a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52986/0 (socket says 172.21.15.142:52986) 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 1 -- 172.21.15.142:0/3378514646 learned_addr learned my addr 172.21.15.142:0/3378514646 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): get_auth_request con 0x7f711809b5b0 auth_method 0 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.352+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7116d9d700 1 -- 172.21.15.142:0/3378514646 >> v1:172.21.15.142:6789/0 conn(0x7f7118099060 legacy=0x7f7118136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711659c700 1 -- 172.21.15.142:0/3378514646 >> v1:172.21.15.150:6789/0 conn(0x7f7118098140 legacy=0x7f7118098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 -- 172.21.15.142:0/3378514646 >> v1:172.21.15.150:6789/0 conn(0x7f7118098140 legacy=0x7f7118098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 -- 172.21.15.142:0/3378514646 >> v1:172.21.15.142:6789/0 conn(0x7f7118099060 legacy=0x7f7118136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 -- 172.21.15.142:0/3378514646 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711806fb20 con 0x7f711809b5b0 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 -- 172.21.15.142:0/3378514646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711809b5b0 msgr2=0x7f71180719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f7115d9b700 1 --2- 172.21.15.142:0/3378514646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711809b5b0 0x7f71180719a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f710800b0d0 tx=0x7f7108009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f086d2408880cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 1 -- 172.21.15.142:0/3378514646 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710800e070 con 0x7f711809b5b0 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:00.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: dump: 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.369 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 1 -- 172.21.15.142:0/3378514646 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710800bd90 con 0x7f711809b5b0 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711759e700 5 monclient: authenticate success, global_id 4449 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 1 -- 172.21.15.142:0/3378514646 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108007650 con 0x7f711809b5b0 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711759e700 4 set_mon_vals no callback set 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.356+0000 7f711559a700 10 monclient: dump: 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: shutdown 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- 172.21.15.142:0/3378514646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711809b5b0 msgr2=0x7f71180719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 --2- 172.21.15.142:0/3378514646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711809b5b0 0x7f71180719a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f710800b0d0 tx=0x7f7108009840 comp rx=0 tx=0).stop 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- 172.21.15.142:0/3378514646 shutdown_connections 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 --2- 172.21.15.142:0/3378514646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711809b5b0 0x7f71180719a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- 172.21.15.142:0/3378514646 >> 172.21.15.142:0/3378514646 conn(0x7f71180a1ec0 msgr2=0x7f7118070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- 172.21.15.142:0/3378514646 shutdown_connections 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- 172.21.15.142:0/3378514646 wait complete. 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: monmap: 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.372 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 Processor -- start 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- start start 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: init 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 0x7f71181416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): picked mon.noname-b con 0x7f7118098140 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:00.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): picked mon.noname-a con 0x7f7118099060 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711809b5b0 0x7f7118142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): picked mon.noname-c con 0x7f711809b5b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7118091510 con 0x7f7118099060 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7118091a40 con 0x7f7118098140 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f711806bf50 con 0x7f711809b5b0 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711759e700 10 monclient(hunting): authenticate will time out at 889.289979s 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52996/0 (socket says 172.21.15.142:52996) 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 -- 172.21.15.142:0/1163471148 learned_addr learned my addr 172.21.15.142:0/1163471148 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): get_auth_request con 0x7f7118099060 auth_method 0 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 0x7f71181416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 10 monclient(hunting): get_auth_request con 0x7f7118098140 auth_method 0 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711809b5b0 0x7f7118142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 msgr2=0x7f71181416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 msgr2=0x7f71181416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): get_auth_request con 0x7f711809b5b0 auth_method 0 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 0x7f71181416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f711659c700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 0x7f71181416f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711809b5b0 msgr2=0x7f7118142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711809b5b0 0x7f7118142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 msgr2=0x7f71181416f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7118098140 0x7f71181416f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.360+0000 7f7115d9b700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7108009ae0 con 0x7f7118099060 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7115d9b700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 msgr2=0x7f7118141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7115d9b700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f710800b820 tx=0x7f7108018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080070d0 con 0x7f7118099060 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: dump: 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.381 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7108026460 con 0x7f7118099060 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 5 monclient: authenticate success, global_id 4452 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 monclient: _renew_subs 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7118099bd0 con 0x7f7118099060 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108026800 con 0x7f7118099060 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 10 monclient: dump: 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.383 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 monclient: _renew_subs 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7114d99700 4 set_mon_vals no callback set 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711809fd60 con 0x7f7118099060 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7115d9b700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f710800b820 tx=0x7f7108018000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f710804a390 osd_map(17..17 src has 1..17) v4 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f710801d420 con 0x7f7118099060 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f710e7fc700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f70f8035ac0 0x7f70f8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 0x7f71180883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71180648a0 con 0x7f7118087fe0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 0x7f71180883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient: get_auth_request con 0x7f7118087fe0 auth_method 0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711659c700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f70f8035ac0 0x7f70f8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f711659c700 10 monclient: get_auth_request con 0x7f70f8035ac0 auth_method 0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.364+0000 7f7116d9d700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71180917f0 con 0x7f7118087fe0 2024-09-18T10:10:00.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f711659c700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f70f8035ac0 0x7f70f8037f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7100007b50 tx=0x7f7100009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f7116d9d700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 msgr2=0x7f71180883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f7116d9d700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 0x7f71180883b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71100074f0 con 0x7f7118087fe0 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 10 monclient: dump: 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.386 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7110007650 con 0x7f7118087fe0 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 msgr2=0x7f71180883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 0x7f71180883b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110009040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 msgr2=0x7f71180883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.368+0000 7f710e7fc700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118087fe0 0x7f71180883b0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110009040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f711759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:00.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f711759e700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 0x7f711806a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f711759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7118094f20 con 0x7f71180888f0 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 0x7f711806a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient: get_auth_request con 0x7f71180888f0 auth_method 0 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 1 -- 172.21.15.142:0/1163471148 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7118072a80 con 0x7f71180888f0 2024-09-18T10:10:00.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 msgr2=0x7f711806a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f7116d9d700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 0x7f711806a410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7110013c60 con 0x7f71180888f0 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f710e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f710e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.380+0000 7f710e7fc700 10 monclient: dump: 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.391 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f7110013dc0 con 0x7f71180888f0 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 msgr2=0x7f711806a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 0x7f711806a410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110015000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 msgr2=0x7f711806a410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71180888f0 0x7f711806a410 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f711000f7b0 tx=0x7f7110015000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:00.392 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:00.393 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.394 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:00.395 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.396 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.397 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.399 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:00.401 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.402 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.404 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:00.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:00.406 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f70f8035ac0 msgr2=0x7f70f8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f70f8035ac0 0x7f70f8037f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7100007b50 tx=0x7f7100009040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 10 monclient: shutdown 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 msgr2=0x7f7118141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f7116d9d700 1 -- 172.21.15.142:0/1163471148 reap_dead start 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f710800b820 tx=0x7f7108018000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f710e7fc700 1 -- 172.21.15.142:0/1163471148 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (unknown 0 0 0) 0x7f710804a390 con 0x7f7118099060 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 shutdown_connections 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 --2- 172.21.15.142:0/1163471148 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7118099060 0x7f7118141c30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 >> 172.21.15.142:0/1163471148 conn(0x7f71180a1ec0 msgr2=0x7f711809c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 shutdown_connections 2024-09-18T10:10:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.384+0000 7f711759e700 1 -- 172.21.15.142:0/1163471148 wait complete. 2024-09-18T10:10:00.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:00.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:10:00.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.504+0000 7f359f8c8700 10 monclient: get_monmap_and_config 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.504+0000 7f359f8c8700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.504+0000 7f359f8c8700 10 monclient: monmap: 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.512 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:00.511258+0000 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:00.511258+0000 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:00.513 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 Processor -- start 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 -- start start 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient: init 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f359809abd0 0x7f359809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-a con 0x7f359809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-b con 0x7f359809db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f359809c950 0x7f3598096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-c con 0x7f359809c950 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359806fa60 con 0x7f359809abd0 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980958c0 con 0x7f359809c950 2024-09-18T10:10:00.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3598095ef0 con 0x7f359809db80 2024-09-18T10:10:00.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359f8c8700 10 monclient(hunting): authenticate will time out at 889.436760s 2024-09-18T10:10:00.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f359809abd0 0x7f359809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53736/0 (socket says 172.21.15.142:53736) 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 1 -- 172.21.15.142:0/3060633010 learned_addr learned my addr 172.21.15.142:0/3060633010 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): get_auth_request con 0x7f359809abd0 auth_method 0 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): get_auth_request con 0x7f359809db80 auth_method 0 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 1 --2- 172.21.15.142:0/3060633010 >> v2:172.21.15.150:3300/0 conn(0x7f359809c950 0x7f3598096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): get_auth_request con 0x7f359809c950 auth_method 0 2024-09-18T10:10:00.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 -- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 msgr2=0x7f359809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 -- 172.21.15.142:0/3060633010 >> v2:172.21.15.150:3300/0 conn(0x7f359809c950 msgr2=0x7f3598096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 --2- 172.21.15.142:0/3060633010 >> v2:172.21.15.150:3300/0 conn(0x7f359809c950 0x7f3598096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 monclient: found mon.noname-a 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359d664700 1 -- 172.21.15.142:0/3060633010 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359809afa0 con 0x7f359809abd0 2024-09-18T10:10:00.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.508+0000 7f359ce63700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 -- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809abd0 msgr2=0x7f359809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809abd0 0x7f359809bed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f359806fdb0 tx=0x7f35940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bd0cf85c4b5ddb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 1 -- 172.21.15.142:0/3060633010 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594011840 con 0x7f359809abd0 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: dump: 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.529 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 5 monclient: authenticate success, global_id 4461 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 1 -- 172.21.15.142:0/3060633010 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35940119c0 con 0x7f359809abd0 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 4 set_mon_vals no callback set 2024-09-18T10:10:00.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 1 -- 172.21.15.142:0/3060633010 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594011d80 con 0x7f359809abd0 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f358ffff700 10 monclient: dump: 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.531 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: shutdown 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809abd0 msgr2=0x7f359809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809abd0 0x7f359809bed0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f359806fdb0 tx=0x7f35940091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- 172.21.15.142:0/3060633010 shutdown_connections 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- 172.21.15.142:0/3060633010 >> v2:172.21.15.150:3300/0 conn(0x7f359809c950 0x7f3598096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809db80 0x7f359809c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- 172.21.15.142:0/3060633010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809abd0 0x7f359809bed0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- 172.21.15.142:0/3060633010 >> 172.21.15.142:0/3060633010 conn(0x7f35980a14e0 msgr2=0x7f35980a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- 172.21.15.142:0/3060633010 shutdown_connections 2024-09-18T10:10:00.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- 172.21.15.142:0/3060633010 wait complete. 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: monmap: 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:00.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 Processor -- start 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- start start 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: init 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 0x7f3598089810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-b con 0x7f359809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 0x7f3598089d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-c con 0x7f359809c950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 0x7f359808a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): picked mon.noname-a con 0x7f359809db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359808b8b0 con 0x7f359809db80 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598097c90 con 0x7f359809abd0 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359809a2d0 con 0x7f359809c950 2024-09-18T10:10:00.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359f8c8700 10 monclient(hunting): authenticate will time out at 889.443030s 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 0x7f359808a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 0x7f3598089d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 0x7f359808a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53048/0 (socket says 172.21.15.142:53048) 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 0x7f3598089d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53744/0 (socket says 172.21.15.142:53744) 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 1 -- 172.21.15.142:0/660833385 learned_addr learned my addr 172.21.15.142:0/660833385 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): get_auth_request con 0x7f359809db80 auth_method 0 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): get_auth_request con 0x7f359809c950 auth_method 0 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 0x7f3598089810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): get_auth_request con 0x7f359809abd0 auth_method 0 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient(hunting): handle_auth_done global_id 14143 payload 824 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 msgr2=0x7f3598089d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 0x7f3598089d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 msgr2=0x7f359808a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 0x7f359808a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 monclient: found mon.noname-b 2024-09-18T10:10:00.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359d664700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35880094f0 con 0x7f359809abd0 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359ce63700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359809c950 0x7f3598089d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.512+0000 7f359de65700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f359809db80 0x7f359808a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359d664700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 msgr2=0x7f3598089810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359d664700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 0x7f3598089810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3594009fd0 tx=0x7f3594017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359401f070 con 0x7f359809abd0 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: dump: 2024-09-18T10:10:00.540 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 5 monclient: authenticate success, global_id 14143 2024-09-18T10:10:00.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3594010820 con 0x7f359809abd0 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3594010be0 con 0x7f359809abd0 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 monclient: _renew_subs 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3594009420 con 0x7f359809abd0 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358f7fe700 4 set_mon_vals no callback set 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f358d7fa700 10 monclient: dump: 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.542 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 monclient: _renew_subs 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359808cdf0 con 0x7f359809abd0 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 0x7f359802ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.516+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3598064950 con 0x7f359808cfe0 2024-09-18T10:10:00.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 0x7f359802ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient: get_auth_request con 0x7f359808cfe0 auth_method 0 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3594023070 con 0x7f359809abd0 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f359808b430 con 0x7f359808cfe0 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f3594048e60 con 0x7f359809abd0 2024-09-18T10:10:00.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359de65700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359de65700 10 monclient: get_auth_request con 0x7f357c035af0 auth_method 0 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 msgr2=0x7f359802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f359ce63700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 0x7f359802ae80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f3588009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588014860 con 0x7f359808cfe0 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 10 monclient: dump: 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.545 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f35880149c0 con 0x7f359808cfe0 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:00.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 msgr2=0x7f359802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.520+0000 7f358d7fa700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 0x7f359802ae80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f3588009250 comp rx=0 tx=0).stop 2024-09-18T10:10:00.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.524+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 msgr2=0x7f359802ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.524+0000 7f358d7fa700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808cfe0 0x7f359802ae80 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f3588009250 comp rx=0 tx=0).stop 2024-09-18T10:10:00.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.524+0000 7f359de65700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35900099f0 tx=0x7f359000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359f8c8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359f8c8700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 0x7f359805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359f8c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359806db60 con 0x7f359808a7d0 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 0x7f359805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient: get_auth_request con 0x7f359808a7d0 auth_method 0 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-09-18T10:10:00.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.532+0000 7f359ce63700 1 -- 172.21.15.142:0/660833385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f35980696a0 con 0x7f359808a7d0 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359ce63700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 msgr2=0x7f359805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359ce63700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 0x7f359805e890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f358801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588014e00 con 0x7f359808a7d0 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 10 monclient: dump: 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.551 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f358801c800 con 0x7f359808a7d0 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 msgr2=0x7f359805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 0x7f359805e890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f358801d000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 msgr2=0x7f359805e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f358d7fa700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f359808a7d0 0x7f359805e890 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3588006fd0 tx=0x7f358801d000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:10:00.553 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.554 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.555 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:00.556 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.557 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:00.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.559 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:00.560 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.562 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.563 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.564 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:10:00.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.566 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:00.567 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.567 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.567 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.567 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:00.567 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:00.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 msgr2=0x7f357c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f357c035af0 0x7f357c037fa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f35900099f0 tx=0x7f359000b040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 10 monclient: shutdown 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 msgr2=0x7f3598089810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 --2- 172.21.15.142:0/660833385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f359809abd0 0x7f3598089810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3594009fd0 tx=0x7f3594017000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359de65700 1 -- 172.21.15.142:0/660833385 reap_dead start 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359de65700 1 -- 172.21.15.142:0/660833385 reap_dead start 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 shutdown_connections 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 >> 172.21.15.142:0/660833385 conn(0x7f35980a14e0 msgr2=0x7f35980a0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 shutdown_connections 2024-09-18T10:10:00.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.536+0000 7f359f8c8700 1 -- 172.21.15.142:0/660833385 wait complete. 2024-09-18T10:10:00.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:00.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:10:00.570 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:10:00.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.676+0000 7f6c3ac34700 10 monclient: get_monmap_and_config 2024-09-18T10:10:00.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.676+0000 7f6c3ac34700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.676+0000 7f6c3ac34700 10 monclient: monmap: 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:00.682958+0000 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:00.682958+0000 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:00.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.676+0000 7f6c3ac34700 1 Processor -- start 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 -- start start 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient: init 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-b con 0x7f6c3409c2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c340a0110 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c340a0bf0 0x7f6c341398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-c con 0x7f6c340a0bf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34064380 con 0x7f6c340a0110 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3409a560 con 0x7f6c340a0bf0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3409ab90 con 0x7f6c3409c2c0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3ac34700 10 monclient(hunting): authenticate will time out at 889.608925s 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c340a0bf0 0x7f6c341398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39431700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6c340a0110 0x7f6c340a04e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:58782/0 (socket says 172.21.15.142:58782) 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53778/0 (socket says 172.21.15.142:53778) 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39431700 1 -- 172.21.15.142:0/735474820 learned_addr learned my addr 172.21.15.142:0/735474820 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): get_auth_request con 0x7f6c3409c2c0 auth_method 0 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): get_auth_request con 0x7f6c340a0bf0 auth_method 0 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient(hunting): handle_auth_done global_id 14146 payload 274 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 -- 172.21.15.142:0/735474820 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 msgr2=0x7f6c340999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 -- 172.21.15.142:0/735474820 >> v1:172.21.15.150:6789/0 conn(0x7f6c340a0110 legacy=0x7f6c340a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 monclient: found mon.noname-c 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c3a433700 1 -- 172.21.15.142:0/735474820 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3409c690 con 0x7f6c340a0bf0 2024-09-18T10:10:00.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.680+0000 7f6c39c32700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3a433700 1 -- 172.21.15.142:0/735474820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 msgr2=0x7f6c341398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3a433700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c341398b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e90 tx=0x7f6c2c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e55a22c6ebdd9feb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 1 -- 172.21.15.142:0/735474820 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00e070 con 0x7f6c340a0bf0 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: dump: 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.695 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 5 monclient: authenticate success, global_id 14146 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 1 -- 172.21.15.142:0/735474820 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c00be00 con 0x7f6c340a0bf0 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 4 set_mon_vals no callback set 2024-09-18T10:10:00.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 1 -- 172.21.15.142:0/735474820 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c012680 con 0x7f6c340a0bf0 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c38c30700 10 monclient: dump: 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: shutdown 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- 172.21.15.142:0/735474820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 msgr2=0x7f6c341398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c341398b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e90 tx=0x7f6c2c009780 comp rx=0 tx=0).stop 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- 172.21.15.142:0/735474820 shutdown_connections 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c341398b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/735474820 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c340999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- 172.21.15.142:0/735474820 >> 172.21.15.142:0/735474820 conn(0x7f6c340a5720 msgr2=0x7f6c340a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- 172.21.15.142:0/735474820 shutdown_connections 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- 172.21.15.142:0/735474820 wait complete. 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: monmap: 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.698 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:00.699 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 Processor -- start 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 -- start start 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: init 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c3408df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-c con 0x7f6c3409c2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.684+0000 7f6c3ac34700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c340a0110 0x7f6c3408e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-a con 0x7f6c340a0110 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c3408e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): picked mon.noname-b con 0x7f6c340a0bf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3409f180 con 0x7f6c340a0110 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34091890 con 0x7f6c340a0bf0 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340915b0 con 0x7f6c3409c2c0 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient(hunting): authenticate will time out at 889.616067s 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c3408df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c3408df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53788/0 (socket says 172.21.15.142:53788) 2024-09-18T10:10:00.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 -- 172.21.15.142:0/3338064610 learned_addr learned my addr 172.21.15.142:0/3338064610 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c340a0110 0x7f6c3408e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): get_auth_request con 0x7f6c3409c2c0 auth_method 0 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): get_auth_request con 0x7f6c340a0110 auth_method 0 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c3408e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): get_auth_request con 0x7f6c340a0bf0 auth_method 0 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 msgr2=0x7f6c3408e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c340a0bf0 0x7f6c3408e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c340a0110 msgr2=0x7f6c3408e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c340a0110 0x7f6c3408e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 monclient: found mon.noname-c 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24009420 con 0x7f6c3409c2c0 2024-09-18T10:10:00.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39431700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c340a0110 0x7f6c3408e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 msgr2=0x7f6c3408df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c39c32700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c3408df30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400ec80 tx=0x7f6c240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2400ca60 con 0x7f6c3409c2c0 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: dump: 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.710 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2400cbc0 con 0x7f6c3409c2c0 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 5 monclient: authenticate success, global_id 4448 2024-09-18T10:10:00.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient: _renew_subs 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c009a20 con 0x7f6c3409c2c0 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24013470 con 0x7f6c3409c2c0 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c29ffb700 10 monclient: dump: 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.712 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient: _renew_subs 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c34099410 con 0x7f6c3409c2c0 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.688+0000 7f6c2bfff700 4 set_mon_vals no callback set 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6c240137e0 con 0x7f6c3409c2c0 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c29ffb700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c18035ac0 0x7f6c18037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3ac34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f6c24048560 con 0x7f6c3409c2c0 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 0x7f6c3406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34091470 con 0x7f6c3408eef0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c39431700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c18035ac0 0x7f6c18037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c39431700 10 monclient: get_auth_request con 0x7f6c18035ac0 auth_method 0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 0x7f6c3406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient: get_auth_request con 0x7f6c3408eef0 auth_method 0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 10 monclient(hunting): handle_auth_done global_id 14152 payload 824 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.692+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c34091a00 con 0x7f6c3408eef0 2024-09-18T10:10:00.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c39431700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c18035ac0 0x7f6c18037f70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c3409fd00 tx=0x7f6c30007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 msgr2=0x7f6c3406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c3a433700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 0x7f6c3406ca30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e90 tx=0x7f6c2c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c01e640 con 0x7f6c3408eef0 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 10 monclient: dump: 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.715 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6c2c01e7a0 con 0x7f6c3408eef0 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 msgr2=0x7f6c3406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 0x7f6c3406ca30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e90 tx=0x7f6c2c018000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 msgr2=0x7f6c3406ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.696+0000 7f6c29ffb700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3408eef0 0x7f6c3406ca30 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e90 tx=0x7f6c2c018000 comp rx=0 tx=0).stop 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3ac34700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 0x7f6c3406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3ac34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34064750 con 0x7f6c3406cf70 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 0x7f6c3406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient: get_auth_request con 0x7f6c3406cf70 auth_method 0 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 10 monclient(hunting): handle_auth_done global_id 14155 payload 824 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.708+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c34090480 con 0x7f6c3406cf70 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 msgr2=0x7f6c3406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3a433700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 0x7f6c3406d370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e60 tx=0x7f6c2c021040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c038070 con 0x7f6c3406cf70 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 10 monclient: dump: 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f6c2c01dcb0 con 0x7f6c3406cf70 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 msgr2=0x7f6c3406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 0x7f6c3406d370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e60 tx=0x7f6c2c021040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 msgr2=0x7f6c3406d370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c29ffb700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c3406cf70 0x7f6c3406d370 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009e60 tx=0x7f6c2c021040 comp rx=0 tx=0).stop 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 26, 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:00.721 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.722 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.723 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:00.724 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:00.725 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.726 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.727 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.728 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.729 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.730 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.731 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:00.732 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:00.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c18035ac0 msgr2=0x7f6c18037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c18035ac0 0x7f6c18037f70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c3409fd00 tx=0x7f6c30007f90 comp rx=0 tx=0).stop 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 10 monclient: shutdown 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 msgr2=0x7f6c3408df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 --2- 172.21.15.142:0/3338064610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c3409c2c0 0x7f6c3408df30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400ec80 tx=0x7f6c240091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 reap_dead start 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3a433700 1 -- 172.21.15.142:0/3338064610 reap_dead start 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 shutdown_connections 2024-09-18T10:10:00.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 >> 172.21.15.142:0/3338064610 conn(0x7f6c340a5720 msgr2=0x7f6c3409ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 shutdown_connections 2024-09-18T10:10:00.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.712+0000 7f6c3ac34700 1 -- 172.21.15.142:0/3338064610 wait complete. 2024-09-18T10:10:00.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:00.736 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:10:00.736 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:10:00.736 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:10:00.736 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:10:00.736 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:10:00.736 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:10:00.754 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:10:00.754 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:10:00.754 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:00.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.860+0000 7f28ef59e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:00.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.860+0000 7f28ef59e700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.860+0000 7f28ef59e700 10 monclient: monmap: 2024-09-18T10:10:00.867 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.867 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:00.866781+0000 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:00.866781+0000 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:00.868 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 Processor -- start 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 -- start start 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient: init 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28f00a1cf0 0x7f28f012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f28f00a1cf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:00.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f28f012f810 0x7f28f009f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f28f012f810 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28f00a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f009d990 con 0x7f28f00a02d0 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f009dc70 con 0x7f28f012f810 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00a20c0 con 0x7f28f00a1cf0 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ef59e700 10 monclient(hunting): authenticate will time out at 889.792861s 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f28f012f810 0x7f28f009f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f28f012f810 0x7f28f009f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53092/0 (socket says 172.21.15.142:53092) 2024-09-18T10:10:00.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 -- 172.21.15.142:0/495645430 learned_addr learned my addr 172.21.15.142:0/495645430 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): get_auth_request con 0x7f28f012f810 auth_method 0 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 1 --2- 172.21.15.142:0/495645430 >> v2:172.21.15.150:3300/0 conn(0x7f28f00a1cf0 0x7f28f012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): get_auth_request con 0x7f28f00a1cf0 auth_method 0 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 -- 172.21.15.142:0/495645430 >> v2:172.21.15.150:3300/0 conn(0x7f28f00a1cf0 msgr2=0x7f28f012f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 --2- 172.21.15.142:0/495645430 >> v2:172.21.15.150:3300/0 conn(0x7f28f00a1cf0 0x7f28f012f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 -- 172.21.15.142:0/495645430 >> v1:172.21.15.150:6789/0 conn(0x7f28f00a02d0 legacy=0x7f28f01320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.864+0000 7f28e5d9b700 1 -- 172.21.15.142:0/495645430 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f009f9c0 con 0x7f28f012f810 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28e5d9b700 1 -- 172.21.15.142:0/495645430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 msgr2=0x7f28f009f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28e5d9b700 1 --2- 172.21.15.142:0/495645430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009f480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f28f0071e60 tx=0x7f28d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167cd035dea02a54 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 1 -- 172.21.15.142:0/495645430 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8009e70 con 0x7f28f012f810 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: dump: 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.879 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 1 -- 172.21.15.142:0/495645430 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d800ba70 con 0x7f28f012f810 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 1 -- 172.21.15.142:0/495645430 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d800be30 con 0x7f28f012f810 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 5 monclient: authenticate success, global_id 4470 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 4 set_mon_vals no callback set 2024-09-18T10:10:00.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28edd9b700 10 monclient: dump: 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 10 monclient: shutdown 2024-09-18T10:10:00.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- 172.21.15.142:0/495645430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 msgr2=0x7f28f009f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 --2- 172.21.15.142:0/495645430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009f480 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f28f0071e60 tx=0x7f28d8009250 comp rx=0 tx=0).stop 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- 172.21.15.142:0/495645430 shutdown_connections 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 --2- 172.21.15.142:0/495645430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009f480 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 --2- 172.21.15.142:0/495645430 >> v2:172.21.15.150:3300/0 conn(0x7f28f00a1cf0 0x7f28f012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- 172.21.15.142:0/495645430 >> 172.21.15.142:0/495645430 conn(0x7f28f0071820 msgr2=0x7f28f00a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- 172.21.15.142:0/495645430 shutdown_connections 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- 172.21.15.142:0/495645430 wait complete. 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 10 monclient: build_initial_monmap 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 10 monclient: monmap: 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:00.882 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 Processor -- start 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 1 -- start start 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.868+0000 7f28ef59e700 10 monclient: init 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:00.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 0x7f28f009b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-b con 0x7f28f00a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 0x7f28f009b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-c con 0x7f28f00a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): picked mon.noname-a con 0x7f28f012f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f009ce30 con 0x7f28f012f810 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f006bca0 con 0x7f28f00a02d0 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f012eee0 con 0x7f28f00a1cf0 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ef59e700 10 monclient(hunting): authenticate will time out at 889.800338s 2024-09-18T10:10:00.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28e5d9b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 msgr2=0x7f28f009b790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28e5d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 0x7f28f009b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 0x7f28f009b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53094/0 (socket says 172.21.15.142:53094) 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 -- 172.21.15.142:0/3211703442 learned_addr learned my addr 172.21.15.142:0/3211703442 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): get_auth_request con 0x7f28f012f810 auth_method 0 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): get_auth_request con 0x7f28f00a02d0 auth_method 0 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:00.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 -- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 msgr2=0x7f28f009b790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 0x7f28f009b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 -- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 msgr2=0x7f28f009b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 0x7f28f009b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 monclient: found mon.noname-a 2024-09-18T10:10:00.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:00.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 -- 172.21.15.142:0/3211703442 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80094f0 con 0x7f28f012f810 2024-09-18T10:10:00.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28ee59c700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 0x7f28f009b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:00.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 -- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 msgr2=0x7f28f009bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:00.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28eed9d700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c930 tx=0x7f28e80026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.872+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8012840 con 0x7f28f012f810 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: dump: 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 5 monclient: authenticate success, global_id 4473 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e80129c0 con 0x7f28f012f810 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8012d80 con 0x7f28f012f810 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 10 monclient: dump: 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:00.890 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _renew_subs 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ed59a700 4 set_mon_vals no callback set 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f006d260 con 0x7f28f012f810 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _renew_subs 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f006ae90 con 0x7f28f012f810 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f28e800a5c0 con 0x7f28f012f810 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f0072a80 con 0x7f28f012f810 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 0x7f28d0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f28e8048270 con 0x7f28f012f810 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ee59c700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 0x7f28d0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.876+0000 7f28ee59c700 10 monclient: get_auth_request con 0x7f28d0035a70 auth_method 0 2024-09-18T10:10:00.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.880+0000 7f28ee59c700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 0x7f28d0037f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f28e0011480 tx=0x7f28e00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:00.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.884+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f28e8010d50 con 0x7f28f012f810 2024-09-18T10:10:00.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.884+0000 7f28e6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:00.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:00.884+0000 7f28e6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:01.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28ef59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:01.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28ef59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:01.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28ef59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:01.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28f006ba80 con 0x7f28f012f810 2024-09-18T10:10:01.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28e6ffd700 1 -- 172.21.15.142:0/3211703442 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f28e8010750 con 0x7f28f012f810 2024-09-18T10:10:01.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28e6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:01.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.112+0000 7f28e6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:01.119 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:01.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 msgr2=0x7f28d0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:01.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 0x7f28d0037f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f28e0011480 tx=0x7f28e00091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 10 monclient: shutdown 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 msgr2=0x7f28f009bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c930 tx=0x7f28e80026e0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 shutdown_connections 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28d0035a70 0x7f28d0037f20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f012f810 0x7f28f009bcd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28f00a1cf0 0x7f28f009b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 --2- 172.21.15.142:0/3211703442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28f00a02d0 0x7f28f009b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:01.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 >> 172.21.15.142:0/3211703442 conn(0x7f28f0071820 msgr2=0x7f28f0132040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:01.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 shutdown_connections 2024-09-18T10:10:01.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:01.116+0000 7f28ef59e700 1 -- 172.21.15.142:0/3211703442 wait complete. 2024-09-18T10:10:01.790 INFO:tasks.workunit.client.0.smithi150.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T10:10:01.790 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.791 INFO:tasks.workunit.client.0.smithi150.stderr: git switch -c 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr:Or undo this operation with: 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr: git switch - 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:01.792 INFO:tasks.workunit.client.0.smithi150.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T10:10:01.796 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T10:10:01.825 INFO:tasks.workunit.client.0.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-18T10:10:01.828 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T10:10:01.828 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-18T10:10:02.659 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-18T10:10:02.741 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-18T10:10:02.793 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-18T10:10:02.795 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T10:10:02.795 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-18T10:10:02.857 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-18T10:10:02.862 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-18T10:10:02.862 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T10:10:02.869 INFO:tasks.workunit:Running workunits matching mon/pool_ops.sh on client.0... 2024-09-18T10:10:02.870 INFO:tasks.workunit:Running workunit mon/pool_ops.sh... 2024-09-18T10:10:02.870 DEBUG:teuthology.orchestra.run.smithi150:workunit test mon/pool_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh 2024-09-18T10:10:02.926 INFO:tasks.workunit.client.0.smithi150.stderr:+ TEST_POOL=testpool1234 2024-09-18T10:10:02.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create testpool1234 8 --autoscale-mode off 2024-09-18T10:10:03.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: get_monmap_and_config 2024-09-18T10:10:03.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: build_initial_monmap 2024-09-18T10:10:03.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: monmap: 2024-09-18T10:10:03.036 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:03.034581+0000 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:03.034581+0000 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:03.037 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 1 Processor -- start 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 1 -- start start 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: init 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.030+0000 7f458bf76700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45840a0290 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:03.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 0x7f458409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-b con 0x7f45840a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4584137a20 0x7f4584139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-c con 0x7f4584137a20 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458409e360 con 0x7f45840a0290 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458409ef80 con 0x7f4584137a20 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840a1fb0 con 0x7f45840a0d70 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient(hunting): authenticate will time out at 885.128163s 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4584137a20 0x7f4584139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4589511700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 msgr2=0x7f458409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:03.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4589511700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 0x7f458409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4584137a20 0x7f4584139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49252/0 (socket says 172.21.15.150:49252) 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4589d12700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f45840a0290 0x7f45840a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:43612/0 (socket says 172.21.15.150:43612) 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 -- 172.21.15.150:0/3872899737 learned_addr learned my addr 172.21.15.150:0/3872899737 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): get_auth_request con 0x7f4584137a20 auth_method 0 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient(hunting): handle_auth_done global_id 14164 payload 274 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 -- 172.21.15.150:0/3872899737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 msgr2=0x7f458409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:03.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 --2- 172.21.15.150:0/3872899737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 0x7f458409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 -- 172.21.15.150:0/3872899737 >> v1:172.21.15.150:6789/0 conn(0x7f45840a0290 legacy=0x7f45840a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient: _finish_hunting 0 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 monclient: found mon.noname-c 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 -- 172.21.15.150:0/3872899737 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458409da50 con 0x7f4584137a20 2024-09-18T10:10:03.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 -- 172.21.15.150:0/3872899737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4584137a20 msgr2=0x7f4584139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458a513700 1 --2- 172.21.15.150:0/3872899737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4584137a20 0x7f4584139e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f457c0099b0 tx=0x7f457c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a912e0518c674 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 1 -- 172.21.15.150:0/3872899737 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c00bb50 con 0x7f4584137a20 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: dump: 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:03.044 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: _finish_auth 0 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 1 -- 172.21.15.150:0/3872899737 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c00bcb0 con 0x7f4584137a20 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 5 monclient: authenticate success, global_id 14164 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 1 -- 172.21.15.150:0/3872899737 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c011670 con 0x7f4584137a20 2024-09-18T10:10:03.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f458bf76700 4 set_mon_vals no callback set 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.034+0000 7f4588d10700 10 monclient: dump: 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: shutdown 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- 172.21.15.150:0/3872899737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4584137a20 msgr2=0x7f4584139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 --2- 172.21.15.150:0/3872899737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4584137a20 0x7f4584139e00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f457c0099b0 tx=0x7f457c009250 comp rx=0 tx=0).stop 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- 172.21.15.150:0/3872899737 shutdown_connections 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 --2- 172.21.15.150:0/3872899737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0d70 0x7f458409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 --2- 172.21.15.150:0/3872899737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4584137a20 0x7f4584139e00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- 172.21.15.150:0/3872899737 >> 172.21.15.150:0/3872899737 conn(0x7f45840a58a0 msgr2=0x7f45840a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- 172.21.15.150:0/3872899737 shutdown_connections 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- 172.21.15.150:0/3872899737 wait complete. 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: build_initial_monmap 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: monmap: 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:03.047 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:03.048 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 Processor -- start 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 -- start start 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: init 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:10:03.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.038+0000 7f458bf76700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 0x7f458412b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-c con 0x7f45840a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-b con 0x7f45840a0d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4584137a20 0x7f458412bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): picked mon.noname-a con 0x7f4584137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458409d3f0 con 0x7f4584137a20 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458409bd60 con 0x7f45840a0d70 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458412e6a0 con 0x7f45840a0290 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:03.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient(hunting): authenticate will time out at 885.136147s 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589d12700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 msgr2=0x7f458412b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589d12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 0x7f458412b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49268/0 (socket says 172.21.15.150:49268) 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 -- 172.21.15.150:0/3350293262 learned_addr learned my addr 172.21.15.150:0/3350293262 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): get_auth_request con 0x7f45840a0d70 auth_method 0 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4584137a20 0x7f458412bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:03.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): get_auth_request con 0x7f4584137a20 auth_method 0 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458a513700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient(hunting): handle_auth_done global_id 14167 payload 824 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 -- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 msgr2=0x7f458412b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 0x7f458412b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 -- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4584137a20 msgr2=0x7f458412bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4584137a20 0x7f458412bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient: _finish_hunting 0 2024-09-18T10:10:03.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 monclient: found mon.noname-b 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0094f0 con 0x7f45840a0d70 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 -- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 msgr2=0x7f458412ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4589511700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f458000b7e0 tx=0x7f4580009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580003e20 con 0x7f45840a0d70 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: dump: 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:03.056 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580006150 con 0x7f45840a0d70 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 5 monclient: authenticate success, global_id 14167 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient: _renew_subs 2024-09-18T10:10:03.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458412ce70 con 0x7f45840a0d70 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580006510 con 0x7f45840a0d70 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f457bfff700 4 set_mon_vals no callback set 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f4579ffb700 10 monclient: dump: 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:03.058 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient: _renew_subs 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.042+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458412da40 con 0x7f45840a0d70 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f45800103c0 con 0x7f45840a0d70 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f458bf76700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f458bf76700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f458bf76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458409d6c0 con 0x7f45840a0d70 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f4579ffb700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 0x7f4570037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:03.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f45800479a0 con 0x7f45840a0d70 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f4589d12700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 0x7f4570037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.046+0000 7f4589d12700 10 monclient: get_auth_request con 0x7f4570035af0 auth_method 0 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.050+0000 7f4589d12700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 0x7f4570037fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4574004bf0 tx=0x7f4574005060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.054+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f458001b570 con 0x7f45840a0d70 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.054+0000 7f4579ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:03.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.054+0000 7f4579ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:03.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.218+0000 7f458bf76700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:03.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.218+0000 7f458bf76700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:03.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.218+0000 7f458bf76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:03.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:03.218+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f4584063e80 con 0x7f45840a0d70 2024-09-18T10:10:04.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:04.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: get_monmap_and_config 2024-09-18T10:10:04.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: build_initial_monmap 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: monmap: 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:04.247996+0000 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:04.247996+0000 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:04.249 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:04.250 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 Processor -- start 2024-09-18T10:10:04.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 -- start start 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: init 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 0x7fd79406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-b con 0x7fd7940a3e60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd79406cab0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd79406ce80 0x7fd794130620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-c con 0x7fd79406ce80 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7940a0070 con 0x7fd79406cab0 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7940a4230 con 0x7fd79406ce80 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:04.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd794096f80 con 0x7fd7940a3e60 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd79a283700 10 monclient(hunting): authenticate will time out at 893.173597s 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799281700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 msgr2=0x7fd79406c570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799281700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 0x7fd79406c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd798a80700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd79406cab0 0x7fd79412ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49972/0 (socket says 172.21.15.142:49972) 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd798a80700 1 -- 172.21.15.142:0/2900436996 learned_addr learned my addr 172.21.15.142:0/2900436996 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 --2- 172.21.15.142:0/2900436996 >> v2:172.21.15.150:3300/0 conn(0x7fd79406ce80 0x7fd794130620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): get_auth_request con 0x7fd79406ce80 auth_method 0 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:04.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient(hunting): handle_auth_done global_id 14170 payload 274 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 -- 172.21.15.142:0/2900436996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 msgr2=0x7fd79406c570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 --2- 172.21.15.142:0/2900436996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 0x7fd79406c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 -- 172.21.15.142:0/2900436996 >> v1:172.21.15.142:6789/0 conn(0x7fd79406cab0 legacy=0x7fd79412ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient: _finish_hunting 0 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 monclient: found mon.noname-c 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:04.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.244+0000 7fd799a82700 1 -- 172.21.15.142:0/2900436996 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794097260 con 0x7fd79406ce80 2024-09-18T10:10:04.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd799a82700 1 -- 172.21.15.142:0/2900436996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 msgr2=0x7fd794130620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:04.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd799a82700 1 --2- 172.21.15.142:0/2900436996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794130620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0098e0 tx=0x7fd78c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cecf00a935301079 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:04.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 1 -- 172.21.15.142:0/2900436996 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c00ba70 con 0x7fd79406ce80 2024-09-18T10:10:04.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: dump: 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 5 monclient: authenticate success, global_id 14170 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 1 -- 172.21.15.142:0/2900436996 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c00bbf0 con 0x7fd79406ce80 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:04.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 4 set_mon_vals no callback set 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 1 -- 172.21.15.142:0/2900436996 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c011420 con 0x7fd79406ce80 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd793fff700 10 monclient: dump: 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:04.262 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 10 monclient: shutdown 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 -- 172.21.15.142:0/2900436996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 msgr2=0x7fd794130620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 --2- 172.21.15.142:0/2900436996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794130620 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0098e0 tx=0x7fd78c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 -- 172.21.15.142:0/2900436996 shutdown_connections 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 --2- 172.21.15.142:0/2900436996 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794130620 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 --2- 172.21.15.142:0/2900436996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7940a3e60 0x7fd79406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 -- 172.21.15.142:0/2900436996 >> 172.21.15.142:0/2900436996 conn(0x7fd794071400 msgr2=0x7fd7940a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 -- 172.21.15.142:0/2900436996 shutdown_connections 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 1 -- 172.21.15.142:0/2900436996 wait complete. 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 10 monclient: build_initial_monmap 2024-09-18T10:10:04.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.248+0000 7fd79a283700 10 monclient: monmap: 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 Processor -- start 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 -- start start 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient: init 2024-09-18T10:10:04.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 0x7fd7941413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-c con 0x7fd79406cab0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794133190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-b con 0x7fd79406ce80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): picked mon.noname-a con 0x7fd7940a3e60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79406ed40 con 0x7fd7940a3e60 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd794098480 con 0x7fd79406ce80 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:04.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd794121000 con 0x7fd79406cab0 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd79a283700 10 monclient(hunting): authenticate will time out at 893.181184s 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799281700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 msgr2=0x7fd7941413a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799281700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 0x7fd7941413a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41128/0 (socket says 172.21.15.142:41128) 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 -- 172.21.15.142:0/3392459472 learned_addr learned my addr 172.21.15.142:0/3392459472 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): get_auth_request con 0x7fd7940a3e60 auth_method 0 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794133190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:04.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): get_auth_request con 0x7fd79406ce80 auth_method 0 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd798a80700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 -- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 msgr2=0x7fd7941413a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 0x7fd7941413a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 -- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 msgr2=0x7fd794133190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:04.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794133190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient: _finish_hunting 0 2024-09-18T10:10:04.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 monclient: found mon.noname-a 2024-09-18T10:10:04.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:04.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:04.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.252+0000 7fd799a82700 1 -- 172.21.15.142:0/3392459472 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c009420 con 0x7fd7940a3e60 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd799a82700 1 -- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 msgr2=0x7fd7941336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd799a82700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd7940a45a0 tx=0x7fd78c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c01b450 con 0x7fd7940a3e60 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: dump: 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.270 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 5 monclient: authenticate success, global_id 4479 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _renew_subs 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c020070 con 0x7fd7940a3e60 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794122b90 con 0x7fd7940a3e60 2024-09-18T10:10:04.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd792ffd700 4 set_mon_vals no callback set 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _renew_subs 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7941225d0 con 0x7fd7940a3e60 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c01b840 con 0x7fd7940a3e60 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 10 monclient: dump: 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:04.272 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd78c022030 con 0x7fd7940a3e60 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 0x7fd774037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fd78c04a220 con 0x7fd7940a3e60 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7941227c0 con 0x7fd7940a3e60 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd799281700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 0x7fd774037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:04.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.256+0000 7fd799281700 10 monclient: get_auth_request con 0x7fd774035a70 auth_method 0 2024-09-18T10:10:04.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.260+0000 7fd799281700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 0x7fd774037f20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd77c003d50 tx=0x7fd77c009310 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:04.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.264+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd78c01a3d0 con 0x7fd7940a3e60 2024-09-18T10:10:04.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.264+0000 7fd7917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:04.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.264+0000 7fd7917fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd7941227c0 con 0x7fd7940a3e60 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd7917fa700 1 -- 172.21.15.142:0/3392459472 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd78c023c80 con 0x7fd7940a3e60 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd7917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:04.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd7917fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:04.498 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 msgr2=0x7fd774037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 0x7fd774037f20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd77c003d50 tx=0x7fd77c009310 comp rx=0 tx=0).stop 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 10 monclient: shutdown 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 msgr2=0x7fd7941336d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd7940a45a0 tx=0x7fd78c016000 comp rx=0 tx=0).stop 2024-09-18T10:10:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 shutdown_connections 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd774035a70 0x7fd774037f20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7940a3e60 0x7fd7941336d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79406ce80 0x7fd794133190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 --2- 172.21.15.142:0/3392459472 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd79406cab0 0x7fd7941413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 >> 172.21.15.142:0/3392459472 conn(0x7fd794071400 msgr2=0x7fd7940a22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 shutdown_connections 2024-09-18T10:10:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:04.492+0000 7fd79a283700 1 -- 172.21.15.142:0/3392459472 wait complete. 2024-09-18T10:10:06.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:06.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:06.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:06.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:06.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:06.042+0000 7f4578ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:07.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:07.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:07.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:07.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:07.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:07.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: get_monmap_and_config 2024-09-18T10:10:07.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: build_initial_monmap 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: monmap: 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:07.625462+0000 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:07.625462+0000 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:07.627 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:07.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 Processor -- start 2024-09-18T10:10:07.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 -- start start 2024-09-18T10:10:07.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: init 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab8c09a7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-b con 0x7fab8c0993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab8c13bb30 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c0997e0 con 0x7fab8c13bb30 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c09abc0 con 0x7fab8c09a7f0 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:07.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c09d950 con 0x7fab8c0993f0 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab90d66700 10 monclient(hunting): authenticate will time out at 896.550973s 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab8affd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab8c13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab8affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab8bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fab8c13bb30 0x7fab8c13df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49988/0 (socket says 172.21.15.142:49988) 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.620+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 learned_addr learned my addr 172.21.15.142:0/4251480641 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.624+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.142:6789/0 conn(0x7fab8c13bb30 legacy=0x7fab8c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:07.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.624+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.150:6789/0 conn(0x7fab8c09a7f0 legacy=0x7fab8c098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:07.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.824+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab8c13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:07.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:07.824+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:08.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:08.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:08.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:08.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:08.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:08.228+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab8c13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:08.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:08.228+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:09.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:09.032+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab8c13b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:09.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:09.032+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:10:09.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:09.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:09.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:09.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:10.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:10.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:10.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:10.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): tick 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab8c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab8c13b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.150:6789/0 conn(0x7fab8c09a7f0 legacy=0x7fab8c098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.142:6789/0 conn(0x7fab8c13bb30 legacy=0x7fab8c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient: _add_conns ranks=[3,4] 2024-09-18T10:10:10.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): picked mon.noname-a con 0x7fab6c003860 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): picked mon.noname-c con 0x7fab6c0063c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8c09d950 con 0x7fab6c003860 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6c008f10 con 0x7fab6c0063c0 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab897fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request con 0x7fab6c003860 auth_method 0 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:10.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab6c0063c0 auth_method 0 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:10.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:10.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.623+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:10.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:10.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request con 0x7fab6c003860 auth_method 0 2024-09-18T10:10:10.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:10.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:10.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab6c0063c0 auth_method 0 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:10.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:10.827+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:11.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:11.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:11.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:11.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request con 0x7fab6c003860 auth_method 0 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:11.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab6c0063c0 auth_method 0 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:11.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:11.231+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:12.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request con 0x7fab6c003860 auth_method 0 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab6c0063c0 auth_method 0 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:12.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:12.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:12.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:12.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:10:12.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:12.035+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:10:12.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:12.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:12.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:12.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:13.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:13.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:13.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:13.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): tick 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 msgr2=0x7fab6c008850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.150:3300/0 conn(0x7fab6c0063c0 0x7fab6c008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 msgr2=0x7fab6c005cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c003860 0x7fab6c005cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 reap_dead start 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab6c00b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:13.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): picked mon.noname-b con 0x7fab8c0993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab6c003cc0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): picked mon.noname-a con 0x7fab6c007f60 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c09db90 con 0x7fab6c003cc0 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8c09d950 con 0x7fab6c007f60 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab6c00b330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 1 -- 172.21.15.142:0/4251480641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c008f10 con 0x7fab8c0993f0 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab897fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab6c00b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:13.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 10 monclient(hunting): get_auth_request con 0x7fab6c007f60 auth_method 0 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:13.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.623+0000 7fab8affd700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.142:6789/0 conn(0x7fab6c003cc0 legacy=0x7fab6c006840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:13.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab6c00b330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:13.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab6c00b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:13.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:13.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 10 monclient(hunting): get_auth_request con 0x7fab6c007f60 auth_method 0 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:13.827+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:14.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:14.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:14.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:14.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:14.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab6c00b330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:14.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab6c00b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 10 monclient(hunting): get_auth_request con 0x7fab6c007f60 auth_method 0 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:14.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.231+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:14.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089275128 0 0) 0x7fab8c09db90 con 0x7fab6c003cc0 2024-09-18T10:10:14.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:14.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:14.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): my global_id is 4482 2024-09-18T10:10:14.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78003570 con 0x7fab6c003cc0 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513844665 0 0) 0x7fab78003570 con 0x7fab6c003cc0 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab6c00b330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab6c00b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 msgr2=0x7fab6c0083b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 --2- 172.21.15.142:0/4251480641 >> v2:172.21.15.142:3300/0 conn(0x7fab6c007f60 0x7fab6c0083b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c0089e0 con 0x7fab6c003cc0 2024-09-18T10:10:14.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fab7c007a60 con 0x7fab6c003cc0 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab90d66700 5 monclient: authenticate success, global_id 4482 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab90d66700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: dump: 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.955 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 >> v1:172.21.15.142:6789/0 conn(0x7fab6c003cc0 legacy=0x7fab6c006840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 0x7fab78003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): picked mon.a con 0x7fab78003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab78004810 0x7fab78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): picked mon.b con 0x7fab78004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:14.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab78005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): picked mon.c con 0x7fab8c0993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c09d950 con 0x7fab78003a60 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c008f10 con 0x7fab78004810 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7800bca0 con 0x7fab8c0993f0 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab90d66700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab78005290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 0x7fab78003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:14.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8bfff700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab78005290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request con 0x7fab78003a60 auth_method 0 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.943+0000 7fab8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab78004810 0x7fab78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab78004810 auth_method 0 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:10:14.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient(hunting): handle_auth_done global_id 4482 payload 293 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 msgr2=0x7fab78005290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c0993f0 0x7fab78005290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab78004810 msgr2=0x7fab78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab78004810 0x7fab78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 monclient: found mon.a 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab78003570 con 0x7fab78003a60 2024-09-18T10:10:14.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:14.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:14.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 msgr2=0x7fab78003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:14.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.947+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 0x7fab78003e70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab780057f0 tx=0x7fab74009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1ed80149754e8f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:14.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.955+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab740015f0 con 0x7fab78003a60 2024-09-18T10:10:14.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.955+0000 7fab8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:14.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.955+0000 7fab8a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.955+0000 7fab8a7fc700 10 monclient: dump: 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.963 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:14.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:14.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:14.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.955+0000 7fab90d66700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:14.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab74012a20 con 0x7fab78003a60 2024-09-18T10:10:14.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:14.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 1 -- 172.21.15.142:0/4251480641 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74012dc0 con 0x7fab78003a60 2024-09-18T10:10:14.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:14.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 4 set_mon_vals no callback set 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8a7fc700 10 monclient: dump: 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: shutdown 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 msgr2=0x7fab78003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 --2- 172.21.15.142:0/4251480641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab78003a60 0x7fab78003e70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab780057f0 tx=0x7fab74009040 comp rx=0 tx=0).stop 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8bfff700 1 -- 172.21.15.142:0/4251480641 reap_dead start 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- 172.21.15.142:0/4251480641 shutdown_connections 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- 172.21.15.142:0/4251480641 >> 172.21.15.142:0/4251480641 conn(0x7fab8c0a1fe0 msgr2=0x7fab8c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- 172.21.15.142:0/4251480641 shutdown_connections 2024-09-18T10:10:14.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- 172.21.15.142:0/4251480641 wait complete. 2024-09-18T10:10:14.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: build_initial_monmap 2024-09-18T10:10:14.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: monmap: 2024-09-18T10:10:14.969 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 Processor -- start 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- start start 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: init 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:14.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab8c06a2b0 0x7fab8c06e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-a con 0x7fab8c06a2b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-b con 0x7fab8c06a680 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 0x7fab8c138a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): picked mon.noname-c con 0x7fab8c1385e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c06f970 con 0x7fab8c06a2b0 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c06d8c0 con 0x7fab8c06a680 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c06eb10 con 0x7fab8c1385e0 2024-09-18T10:10:14.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab90d66700 10 monclient(hunting): authenticate will time out at 903.891184s 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8bfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 msgr2=0x7fab8c138a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 0x7fab8c138a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:46652/0 (socket says 172.21.15.142:46652) 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 1 -- 172.21.15.142:0/1082135951 learned_addr learned my addr 172.21.15.142:0/1082135951 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): get_auth_request con 0x7fab8c06a680 auth_method 0 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:14.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.959+0000 7fab8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 10 monclient(hunting): handle_auth_done global_id 14176 payload 824 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 -- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 msgr2=0x7fab8c138a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 0x7fab8c138a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 -- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab8c06a2b0 msgr2=0x7fab8c06e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab8c06a2b0 0x7fab8c06e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 10 monclient: _finish_hunting 0 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 monclient: found mon.noname-b 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:14.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:14.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 -- 172.21.15.142:0/1082135951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c06bfc0 con 0x7fab8c06a680 2024-09-18T10:10:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 -- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 msgr2=0x7fab8c06aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab8affd700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00f730 tx=0x7fab7c0125d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:14.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c017040 con 0x7fab8c06a680 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: dump: 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:14.976 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 5 monclient: authenticate success, global_id 14176 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c010620 con 0x7fab8c06a680 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 10 monclient: _renew_subs 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c06bab0 con 0x7fab8c06a680 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c010a00 con 0x7fab8c06a680 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:14.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab837fe700 10 monclient: dump: 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 10 monclient: _renew_subs 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c06c680 con 0x7fab8c06a680 2024-09-18T10:10:14.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.963+0000 7fab89ffb700 4 set_mon_vals no callback set 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fab7c025460 con 0x7fab8c06a680 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab837fe700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 0x7fab6c039260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fab7c049ef0 con 0x7fab8c06a680 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 0x7fab6c039260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab8b7fe700 10 monclient: get_auth_request con 0x7fab6c036db0 auth_method 0 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab90d66700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab90d66700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab90d66700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:14.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.967+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c072a80 con 0x7fab8c06a680 2024-09-18T10:10:14.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.971+0000 7fab8b7fe700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 0x7fab6c039260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab6c006790 tx=0x7fab7401e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:14.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.983+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fab7c023b40 con 0x7fab8c06a680 2024-09-18T10:10:14.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.983+0000 7fab837fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:14.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:14.983+0000 7fab837fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:15.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:15.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:15.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:15.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:15.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab90d66700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:15.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab90d66700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:15.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab90d66700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:15.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab8c09d730 con 0x7fab8c06a680 2024-09-18T10:10:15.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab837fe700 1 -- 172.21.15.142:0/1082135951 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fab7c023360 con 0x7fab8c06a680 2024-09-18T10:10:15.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab837fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:15.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.211+0000 7fab837fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:15.218 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:15.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 msgr2=0x7fab6c039260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 0x7fab6c039260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fab6c006790 tx=0x7fab7401e000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 10 monclient: shutdown 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 msgr2=0x7fab8c06aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00f730 tx=0x7fab7c0125d0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 shutdown_connections 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab6c036db0 0x7fab6c039260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab8c1385e0 0x7fab8c138a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab8c06a680 0x7fab8c06aa50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 --2- 172.21.15.142:0/1082135951 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab8c06a2b0 0x7fab8c06e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 >> 172.21.15.142:0/1082135951 conn(0x7fab8c0a1fe0 msgr2=0x7fab8c09f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 shutdown_connections 2024-09-18T10:10:15.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.215+0000 7fab90d66700 1 -- 172.21.15.142:0/1082135951 wait complete. 2024-09-18T10:10:15.238 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:10:15.239 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:10:15.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.331+0000 7f919d496700 10 monclient: get_monmap_and_config 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.331+0000 7f919d496700 10 monclient: build_initial_monmap 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.331+0000 7f919d496700 10 monclient: monmap: 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:15.339540+0000 2024-09-18T10:10:15.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:15.339540+0000 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:15.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 Processor -- start 2024-09-18T10:10:15.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 -- start start 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient: init 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): picked mon.noname-b con 0x7f919809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f919809db80 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9198097880 0x7f9198137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): picked mon.noname-c con 0x7f9198097880 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980958c0 con 0x7f919809db80 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919806fa60 con 0x7f9198097880 2024-09-18T10:10:15.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9198095ef0 con 0x7f919809abd0 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f919d496700 10 monclient(hunting): authenticate will time out at 904.265765s 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f9196ffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 msgr2=0x7f919809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f9196ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9198097880 0x7f9198137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91967fc700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f919809db80 0x7f919809c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:40536/0 (socket says 172.21.15.142:40536) 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91967fc700 1 -- 172.21.15.142:0/2465610244 learned_addr learned my addr 172.21.15.142:0/2465610244 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): get_auth_request con 0x7f9198097880 auth_method 0 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient(hunting): handle_auth_done global_id 14179 payload 274 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 -- 172.21.15.142:0/2465610244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 msgr2=0x7f919809bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 --2- 172.21.15.142:0/2465610244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 -- 172.21.15.142:0/2465610244 >> v1:172.21.15.150:6789/0 conn(0x7f919809db80 legacy=0x7f919809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:15.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.335+0000 7f91977fe700 1 -- 172.21.15.142:0/2465610244 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919809afa0 con 0x7f9198097880 2024-09-18T10:10:15.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f91977fe700 1 -- 172.21.15.142:0/2465610244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 msgr2=0x7f9198137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f91977fe700 1 --2- 172.21.15.142:0/2465610244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f9198137690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f919809cf20 tx=0x7f91900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a880fcf6b6c6f68 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 1 -- 172.21.15.142:0/2465610244 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000ba30 con 0x7f9198097880 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: dump: 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.351 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 1 -- 172.21.15.142:0/2465610244 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919000bbb0 con 0x7f9198097880 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 5 monclient: authenticate success, global_id 14179 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 1 -- 172.21.15.142:0/2465610244 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190011420 con 0x7f9198097880 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 4 set_mon_vals no callback set 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:15.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f9195ffb700 10 monclient: dump: 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 10 monclient: shutdown 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:15.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- 172.21.15.142:0/2465610244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 msgr2=0x7f9198137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 --2- 172.21.15.142:0/2465610244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f9198137690 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f919809cf20 tx=0x7f91900091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- 172.21.15.142:0/2465610244 shutdown_connections 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 --2- 172.21.15.142:0/2465610244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f9198137690 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 --2- 172.21.15.142:0/2465610244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- 172.21.15.142:0/2465610244 >> 172.21.15.142:0/2465610244 conn(0x7f91980a14e0 msgr2=0x7f91980a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- 172.21.15.142:0/2465610244 shutdown_connections 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- 172.21.15.142:0/2465610244 wait complete. 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 10 monclient: build_initial_monmap 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 10 monclient: monmap: 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:15.354 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 Processor -- start 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 1 -- start start 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.339+0000 7f919d496700 10 monclient: init 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f919813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): picked mon.noname-b con 0x7f9198097880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:15.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): picked mon.noname-c con 0x7f919809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 0x7f919808bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): picked mon.noname-a con 0x7f919809db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198097c90 con 0x7f919809db80 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919809fbc0 con 0x7f9198097880 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919808db60 con 0x7f919809abd0 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91967fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 msgr2=0x7f919813feb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient(hunting): authenticate will time out at 904.272494s 2024-09-18T10:10:15.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91967fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919813feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 0x7f919808bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 0x7f919808bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36642/0 (socket says 172.21.15.142:36642) 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 learned_addr learned my addr 172.21.15.142:0/3249864204 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): get_auth_request con 0x7f919809db80 auth_method 0 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f919813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): get_auth_request con 0x7f9198097880 auth_method 0 2024-09-18T10:10:15.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient(hunting): handle_auth_done global_id 4485 payload 824 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 msgr2=0x7f919813feb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f919809abd0 0x7f919813feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 msgr2=0x7f919813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9198097880 0x7f919813f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:15.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9190009420 con 0x7f919809db80 2024-09-18T10:10:15.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 msgr2=0x7f919808bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f91977fe700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 0x7f919808bba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f91980a2530 tx=0x7f9190017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919001f070 con 0x7f919809db80 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 10 monclient: dump: 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 10 monclient: _finish_auth 0 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 5 monclient: authenticate success, global_id 4485 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _renew_subs 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919808e220 con 0x7f919809db80 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _renew_subs 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919808d650 con 0x7f919809db80 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190006b10 con 0x7f919809db80 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.343+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190011e00 con 0x7f919809db80 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 10 monclient: dump: 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f91957fa700 4 set_mon_vals no callback set 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f919d496700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:15.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f919d496700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 0x7f917c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c007cd0 con 0x7f917c002860 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 0x7f917c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient: get_auth_request con 0x7f917c002860 auth_method 0 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 10 monclient(hunting): handle_auth_done global_id 4488 payload 824 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9190023070 con 0x7f919809db80 2024-09-18T10:10:15.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f917c007e90 con 0x7f917c002860 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9178035a70 0x7f9178037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7f9190049620 con 0x7f919809db80 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f91967fc700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9178035a70 0x7f9178037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f91967fc700 10 monclient: get_auth_request con 0x7f9178035a70 auth_method 0 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 msgr2=0x7f917c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.347+0000 7f9196ffd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 0x7f917c004cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f91800039d0 tx=0x7f9180009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180012940 con 0x7f917c002860 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 10 monclient: dump: 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.366 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9180012aa0 con 0x7f917c002860 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 msgr2=0x7f917c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 0x7f917c004cf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f91800039d0 tx=0x7f9180009120 comp rx=0 tx=0).stop 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 msgr2=0x7f917c004cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:15.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f918effd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c002860 0x7f917c004cf0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.351+0000 7f91967fc700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9178035a70 0x7f9178037f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9188004be0 tx=0x7f9188005190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f919d496700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f919d496700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 0x7f917c007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f919d496700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0077a0 con 0x7f917c006cc0 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 0x7f917c007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient: get_auth_request con 0x7f917c006cc0 auth_method 0 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 10 monclient(hunting): handle_auth_done global_id 4491 payload 824 2024-09-18T10:10:15.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.363+0000 7f9196ffd700 1 -- 172.21.15.142:0/3249864204 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f917c007960 con 0x7f917c006cc0 2024-09-18T10:10:15.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f9196ffd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 msgr2=0x7f917c007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f9196ffd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 0x7f917c007140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9180017000 tx=0x7f9180018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918001cde0 con 0x7f917c006cc0 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 10 monclient: dump: 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.373 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f9180012730 con 0x7f917c006cc0 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 msgr2=0x7f917c007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 0x7f917c007140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9180017000 tx=0x7f9180018000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 msgr2=0x7f917c007140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:15.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f918effd700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f917c006cc0 0x7f917c007140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 30, 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:15.375 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.376 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:15.378 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.379 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:15.383 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.384 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.385 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.386 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:15.387 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:15.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9178035a70 msgr2=0x7f9178037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9178035a70 0x7f9178037f20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9188004be0 tx=0x7f9188005190 comp rx=0 tx=0).stop 2024-09-18T10:10:15.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 10 monclient: shutdown 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 msgr2=0x7f919808bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 --2- 172.21.15.142:0/3249864204 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f919809db80 0x7f919808bba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f91980a2530 tx=0x7f9190017000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 reap_dead start 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f91977fe700 1 -- 172.21.15.142:0/3249864204 reap_dead start 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 shutdown_connections 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 >> 172.21.15.142:0/3249864204 conn(0x7f91980a14e0 msgr2=0x7f9198096330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 shutdown_connections 2024-09-18T10:10:15.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.367+0000 7f919d496700 1 -- 172.21.15.142:0/3249864204 wait complete. 2024-09-18T10:10:15.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:10:15.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:10:15.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 10 monclient: get_monmap_and_config 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 10 monclient: build_initial_monmap 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 10 monclient: monmap: 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:15.494414+0000 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:15.494414+0000 2024-09-18T10:10:15.495 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:15.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 1 Processor -- start 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 1 -- start start 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.487+0000 7fe5ad076700 10 monclient: init 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5a809c760 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 0x7fe5a809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a809cb30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe5a8133870 0x7fe5a8135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a8133870 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8098b00 con 0x7fe5a809c760 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a8098d20 con 0x7fe5a8133870 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a809a100 con 0x7fe5a809cb30 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ad076700 10 monclient(hunting): authenticate will time out at 904.420707s 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5a77fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 msgr2=0x7fe5a809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5a77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 0x7fe5a809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5a7fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe5a809c760 0x7fe5a809b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:40538/0 (socket says 172.21.15.142:40538) 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5a7fff700 1 -- 172.21.15.142:0/429025005 learned_addr learned my addr 172.21.15.142:0/429025005 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 --2- 172.21.15.142:0/429025005 >> v2:172.21.15.150:3300/0 conn(0x7fe5a8133870 0x7fe5a8135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): get_auth_request con 0x7fe5a8133870 auth_method 0 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient(hunting): handle_auth_done global_id 14185 payload 274 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 -- 172.21.15.142:0/429025005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 msgr2=0x7fe5a809cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 --2- 172.21.15.142:0/429025005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 0x7fe5a809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 -- 172.21.15.142:0/429025005 >> v1:172.21.15.150:6789/0 conn(0x7fe5a809c760 legacy=0x7fe5a809b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient: _finish_hunting 0 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 monclient: found mon.noname-c 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:15.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:15.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.491+0000 7fe5ac875700 1 -- 172.21.15.142:0/429025005 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a809a3e0 con 0x7fe5a8133870 2024-09-18T10:10:15.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ac875700 1 -- 172.21.15.142:0/429025005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8133870 msgr2=0x7fe5a8135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ac875700 1 --2- 172.21.15.142:0/429025005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8133870 0x7fe5a8135c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009e60 tx=0x7fe5a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dffc79cef176795 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 1 -- 172.21.15.142:0/429025005 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000e070 con 0x7fe5a8133870 2024-09-18T10:10:15.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: dump: 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 5 monclient: authenticate success, global_id 14185 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 1 -- 172.21.15.142:0/429025005 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0012430 con 0x7fe5a8133870 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 4 set_mon_vals no callback set 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 1 -- 172.21.15.142:0/429025005 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a00127d0 con 0x7fe5a8133870 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5a6ffd700 10 monclient: dump: 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.507 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 10 monclient: shutdown 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- 172.21.15.142:0/429025005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8133870 msgr2=0x7fe5a8135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 --2- 172.21.15.142:0/429025005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8133870 0x7fe5a8135c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009e60 tx=0x7fe5a0009780 comp rx=0 tx=0).stop 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- 172.21.15.142:0/429025005 shutdown_connections 2024-09-18T10:10:15.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 --2- 172.21.15.142:0/429025005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a809cb30 0x7fe5a809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 --2- 172.21.15.142:0/429025005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8133870 0x7fe5a8135c50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- 172.21.15.142:0/429025005 >> 172.21.15.142:0/429025005 conn(0x7fe5a8070ef0 msgr2=0x7fe5a80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- 172.21.15.142:0/429025005 shutdown_connections 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- 172.21.15.142:0/429025005 wait complete. 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 10 monclient: build_initial_monmap 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 10 monclient: monmap: 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:15.509 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 Processor -- start 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 1 -- start start 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.495+0000 7fe5ad076700 10 monclient: init 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 0x7fe5a806c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-a con 0x7fe5a809c760 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:15.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a809cb30 0x7fe5a806c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-b con 0x7fe5a809cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a8133870 0x7fe5a806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): picked mon.noname-c con 0x7fe5a8133870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8088af0 con 0x7fe5a809c760 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80a0830 con 0x7fe5a809cb30 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a806f810 con 0x7fe5a8133870 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient(hunting): authenticate will time out at 904.428228s 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 0x7fe5a806c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ac875700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a8133870 msgr2=0x7fe5a806cc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ac875700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a8133870 0x7fe5a806cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 0x7fe5a806c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36662/0 (socket says 172.21.15.142:36662) 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 -- 172.21.15.142:0/1238448498 learned_addr learned my addr 172.21.15.142:0/1238448498 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): get_auth_request con 0x7fe5a809c760 auth_method 0 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a809cb30 0x7fe5a806c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): get_auth_request con 0x7fe5a809cb30 auth_method 0 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a8133870 msgr2=0x7fe5a806cc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5a8133870 0x7fe5a806cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a809cb30 msgr2=0x7fe5a806c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a809cb30 0x7fe5a806c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:15.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 monclient: found mon.noname-a 2024-09-18T10:10:15.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:15.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a0009a20 con 0x7fe5a809c760 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 msgr2=0x7fe5a806c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a7fff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 0x7fe5a806c1d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe59800c5e0 tx=0x7fe598004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598005d40 con 0x7fe5a809c760 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: dump: 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.517 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: _finish_auth 0 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5980070d0 con 0x7fe5a809c760 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 5 monclient: authenticate success, global_id 4494 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _renew_subs 2024-09-18T10:10:15.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a80884a0 con 0x7fe5a809c760 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598007490 con 0x7fe5a809c760 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe58ffff700 10 monclient: dump: 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.519 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5a5ffb700 4 set_mon_vals no callback set 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _renew_subs 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.499+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a806dab0 con 0x7fe5a809c760 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ad076700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ad076700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 0x7fe5a8135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a806dcd0 con 0x7fe5a806d190 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 0x7fe5a8135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe59801b3b0 con 0x7fe5a809c760 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient: get_auth_request con 0x7fe5a806d190 auth_method 0 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe58ffff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe594035a70 0x7fe594037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4370+0+0 (secure 0 0 0) 0x7fe598047750 con 0x7fe5a809c760 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_done global_id 14191 payload 824 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5a77fe700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5a806e7e0 con 0x7fe5a806d190 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ac875700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe594035a70 0x7fe594037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.503+0000 7fe5ac875700 10 monclient: get_auth_request con 0x7fe594035a70 auth_method 0 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe5a77fe700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 msgr2=0x7fe5a8135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe5a77fe700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 0x7fe5a8135c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0039d0 tx=0x7fe59c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c012940 con 0x7fe5a806d190 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 10 monclient: dump: 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.522 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe5ac875700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe594035a70 0x7fe594037f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009e60 tx=0x7fe5a0018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe59c012aa0 con 0x7fe5a806d190 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 msgr2=0x7fe5a8135c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 0x7fe5a8135c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0039d0 tx=0x7fe59c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 msgr2=0x7fe5a8135c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:15.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.507+0000 7fe58ffff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a806d190 0x7fe5a8135c50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:15.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5ad076700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5ad076700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 0x7fe5a8136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5ad076700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80648a0 con 0x7fe5a8136190 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 0x7fe5a8136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient: get_auth_request con 0x7fe5a8136190 auth_method 0 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 10 monclient(hunting): handle_auth_done global_id 14194 payload 824 2024-09-18T10:10:15.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.519+0000 7fe5a77fe700 1 -- 172.21.15.142:0/1238448498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe5a8072a80 con 0x7fe5a8136190 2024-09-18T10:10:15.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5a77fe700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 msgr2=0x7fe5a8136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:15.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5a77fe700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 0x7fe5a8136590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe59c017000 tx=0x7fe59c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:15.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c01cde0 con 0x7fe5a8136190 2024-09-18T10:10:15.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:15.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 10 monclient: dump: 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:15.530 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fe59c012730 con 0x7fe5a8136190 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 msgr2=0x7fe5a8136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 0x7fe5a8136590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe59c017000 tx=0x7fe59c015000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 msgr2=0x7fe5a8136590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:15.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe58ffff700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a8136190 0x7fe5a8136590 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe59c017000 tx=0x7fe59c015000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 30, 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:15.532 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.533 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:15.534 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:15.535 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.536 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:15.539 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:15.540 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.542 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.543 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:15.544 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:15.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe594035a70 msgr2=0x7fe594037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe594035a70 0x7fe594037f20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009e60 tx=0x7fe5a0018000 comp rx=0 tx=0).stop 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 10 monclient: shutdown 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 msgr2=0x7fe5a806c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 --2- 172.21.15.142:0/1238448498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5a809c760 0x7fe5a806c1d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe59800c5e0 tx=0x7fe598004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ac875700 1 -- 172.21.15.142:0/1238448498 reap_dead start 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ac875700 1 -- 172.21.15.142:0/1238448498 reap_dead start 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 shutdown_connections 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 >> 172.21.15.142:0/1238448498 conn(0x7fe5a8070ef0 msgr2=0x7fe5a809f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 shutdown_connections 2024-09-18T10:10:15.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:15.523+0000 7fe5ad076700 1 -- 172.21.15.142:0/1238448498 wait complete. 2024-09-18T10:10:15.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:10:15.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:10:16.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.042+0000 7f4578ff9700 10 monclient: tick 2024-09-18T10:10:16.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.042+0000 7f4578ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:16.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.042+0000 7f4578ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:16.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.162+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' created v18) v1 ==== 152+0+0 (secure 0 0 0) 0x7f4580017030 con 0x7f45840a0d70 2024-09-18T10:10:16.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.166+0000 7f4579ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:16.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.166+0000 7f4579ffb700 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' created 2024-09-18T10:10:16.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f458bf76700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:16.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f458bf76700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:16.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f458bf76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"} v 0) v1 -- 0x7f4584067b10 con 0x7f45840a0d70 2024-09-18T10:10:16.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f4579ffb700 1 -- 172.21.15.150:0/3350293262 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}]=0 pool 'testpool1234' already exists v18) v1 ==== 159+0+0 (secure 0 0 0) 0x7f4580017020 con 0x7f45840a0d70 2024-09-18T10:10:16.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f4579ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "testpool1234", "pg_num": 8, "autoscale_mode": "off"}] 2024-09-18T10:10:16.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.270+0000 7f4579ffb700 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' already exists 2024-09-18T10:10:16.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 msgr2=0x7f4570037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 0x7f4570037fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4574004bf0 tx=0x7f4574005060 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 10 monclient: shutdown 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 msgr2=0x7f458412ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f458000b7e0 tx=0x7f4580009220 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 shutdown_connections 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4570035af0 0x7f4570037fa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4584137a20 0x7f458412bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45840a0d70 0x7f458412ba20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 --2- 172.21.15.150:0/3350293262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45840a0290 0x7f458412b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 >> 172.21.15.150:0/3350293262 conn(0x7f45840a58a0 msgr2=0x7f45841371a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:16.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 shutdown_connections 2024-09-18T10:10:16.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.274+0000 7f458bf76700 1 -- 172.21.15.150:0/3350293262 wait complete. 2024-09-18T10:10:16.280 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'testpool1234' already exists 2024-09-18T10:10:16.289 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_min 2 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.382+0000 7f6da8146700 10 monclient: get_monmap_and_config 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.382+0000 7f6da8146700 10 monclient: build_initial_monmap 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.382+0000 7f6da8146700 10 monclient: monmap: 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:16.387296+0000 2024-09-18T10:10:16.388 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:16.387296+0000 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:16.389 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 Processor -- start 2024-09-18T10:10:16.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 -- start start 2024-09-18T10:10:16.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient: init 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6da00a5fb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 0x7f6da0137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-b con 0x7f6da00a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6da00a0dc0 0x7f6da0139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-a con 0x7f6da00a0dc0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da00a6380 con 0x7f6da00a5fb0 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da00a6770 con 0x7f6da00a0dc0 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da009e660 con 0x7f6da00a09a0 2024-09-18T10:10:16.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da8146700 10 monclient(hunting): authenticate will time out at 898.480850s 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da5ee2700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6da00a5fb0 0x7f6da00a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:60812/0 (socket says 172.21.15.150:60812) 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da56e1700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 msgr2=0x7f6da0137a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da5ee2700 1 -- 172.21.15.150:0/3832518016 learned_addr learned my addr 172.21.15.150:0/3832518016 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da56e1700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 0x7f6da0137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 --2- 172.21.15.150:0/3832518016 >> v2:172.21.15.142:3300/0 conn(0x7f6da00a0dc0 0x7f6da0139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): get_auth_request con 0x7f6da00a0dc0 auth_method 0 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:16.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient(hunting): handle_auth_done global_id 4497 payload 274 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 -- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 msgr2=0x7f6da0137a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 0x7f6da0137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 -- 172.21.15.150:0/3832518016 >> v1:172.21.15.150:6789/0 conn(0x7f6da00a5fb0 legacy=0x7f6da00a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient: _finish_hunting 0 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 monclient: found mon.noname-a 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:16.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.386+0000 7f6da66e3700 1 -- 172.21.15.150:0/3832518016 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da009e920 con 0x7f6da00a0dc0 2024-09-18T10:10:16.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da66e3700 1 -- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a0dc0 msgr2=0x7f6da0139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:16.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da66e3700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a0dc0 0x7f6da0139f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6d980098e0 tx=0x7f6d980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b00832a3148f05 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:16.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 1 -- 172.21.15.150:0/3832518016 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9800ba70 con 0x7f6da00a0dc0 2024-09-18T10:10:16.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:16.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: dump: 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: _finish_auth 0 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:16.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 1 -- 172.21.15.150:0/3832518016 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9800bbf0 con 0x7f6da00a0dc0 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 1 -- 172.21.15.150:0/3832518016 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98011420 con 0x7f6da00a0dc0 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da4ee0700 10 monclient: dump: 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:16.400 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 5 monclient: authenticate success, global_id 4497 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 4 set_mon_vals no callback set 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: shutdown 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a0dc0 msgr2=0x7f6da0139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a0dc0 0x7f6da0139f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6d980098e0 tx=0x7f6d980091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- 172.21.15.150:0/3832518016 shutdown_connections 2024-09-18T10:10:16.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a0dc0 0x7f6da0139f70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 --2- 172.21.15.150:0/3832518016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a09a0 0x7f6da0137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- 172.21.15.150:0/3832518016 >> 172.21.15.150:0/3832518016 conn(0x7f6da0071ae0 msgr2=0x7f6da00a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- 172.21.15.150:0/3832518016 shutdown_connections 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- 172.21.15.150:0/3832518016 wait complete. 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: build_initial_monmap 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: monmap: 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.402 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 Processor -- start 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 -- start start 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: init 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a09a0 0x7f6da008de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-a con 0x7f6da00a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.390+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-b con 0x7f6da00a0dc0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 0x7f6da008e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): picked mon.noname-c con 0x7f6da00a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da00987f0 con 0x7f6da00a09a0 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da008fd20 con 0x7f6da00a0dc0 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da009bf80 con 0x7f6da00a5fb0 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient(hunting): authenticate will time out at 898.488067s 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da66e3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 msgr2=0x7f6da008e8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da66e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 0x7f6da008e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:16.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47778/0 (socket says 172.21.15.150:47778) 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 -- 172.21.15.150:0/4217378029 learned_addr learned my addr 172.21.15.150:0/4217378029 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): get_auth_request con 0x7f6da00a0dc0 auth_method 0 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a09a0 0x7f6da008de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): get_auth_request con 0x7f6da00a09a0 auth_method 0 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:16.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da5ee2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient(hunting): handle_auth_done global_id 14197 payload 824 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 -- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 msgr2=0x7f6da008e8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 0x7f6da008e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 -- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a09a0 msgr2=0x7f6da008de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a09a0 0x7f6da008de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient: _finish_hunting 0 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 monclient: found mon.noname-b 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98009420 con 0x7f6da00a0dc0 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 -- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 msgr2=0x7f6da008e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da56e1700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400b7e0 tx=0x7f6d94009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94003e20 con 0x7f6da00a0dc0 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: dump: 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:16.409 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d94006150 con 0x7f6da00a0dc0 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 5 monclient: authenticate success, global_id 14197 2024-09-18T10:10:16.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient: _renew_subs 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da0091260 con 0x7f6da00a0dc0 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d94006510 con 0x7f6da00a0dc0 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9dffb700 10 monclient: dump: 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:16.411 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6d9ffff700 4 set_mon_vals no callback set 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient: _renew_subs 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.394+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da0099410 con 0x7f6da00a0dc0 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6d940103c0 con 0x7f6da00a0dc0 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6d9dffb700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 0x7f6d8c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f6d94047ae0 con 0x7f6da00a0dc0 2024-09-18T10:10:16.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da8146700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da8146700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da8146700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da0063ed0 con 0x7f6da00a0dc0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da5ee2700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 0x7f6d8c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.398+0000 7f6da5ee2700 10 monclient: get_auth_request con 0x7f6d8c035a70 auth_method 0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.402+0000 7f6da5ee2700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 0x7f6d8c037f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d90006e80 tx=0x7f6d900053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.406+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6d94017030 con 0x7f6da00a0dc0 2024-09-18T10:10:16.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.406+0000 7f6d9dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:16.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.406+0000 7f6d9dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:16.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.570+0000 7f6da8146700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:16.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.570+0000 7f6da8146700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:16.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.570+0000 7f6da8146700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:16.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:16.570+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f6da0067b10 con 0x7f6da00a0dc0 2024-09-18T10:10:17.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.174+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v19) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6d94017030 con 0x7f6da00a0dc0 2024-09-18T10:10:17.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.174+0000 7f6d9dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:17.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.174+0000 7f6d9dffb700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 2 2024-09-18T10:10:17.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.274+0000 7f6da8146700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:17.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.274+0000 7f6da8146700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:17.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.274+0000 7f6da8146700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:17.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:17.274+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"} v 0) v1 -- 0x7f6da008ee00 con 0x7f6da00a0dc0 2024-09-18T10:10:18.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.174+0000 7f6d9dffb700 1 -- 172.21.15.150:0/4217378029 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}]=0 set pool 3 pg_num_min to 2 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6d94017020 con 0x7f6da00a0dc0 2024-09-18T10:10:18.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.174+0000 7f6d9dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "2"}] 2024-09-18T10:10:18.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.174+0000 7f6d9dffb700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 2 2024-09-18T10:10:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 msgr2=0x7f6d8c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 0x7f6d8c037f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d90006e80 tx=0x7f6d900053c0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 10 monclient: shutdown 2024-09-18T10:10:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:18.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 msgr2=0x7f6da008e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400b7e0 tx=0x7f6d94009220 comp rx=0 tx=0).stop 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 shutdown_connections 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6d8c035a70 0x7f6d8c037f20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6da00a5fb0 0x7f6da008e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6da00a0dc0 0x7f6da008e380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 --2- 172.21.15.150:0/4217378029 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6da00a09a0 0x7f6da008de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 >> 172.21.15.150:0/4217378029 conn(0x7f6da0071ae0 msgr2=0x7f6da009b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 shutdown_connections 2024-09-18T10:10:18.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.178+0000 7f6da8146700 1 -- 172.21.15.150:0/4217378029 wait complete. 2024-09-18T10:10:18.186 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 3 pg_num_min to 2 2024-09-18T10:10:18.198 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-18T10:10:18.199 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 2 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.298+0000 7fe4c4b59700 10 monclient: get_monmap_and_config 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.298+0000 7fe4c4b59700 10 monclient: build_initial_monmap 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.298+0000 7fe4c4b59700 10 monclient: monmap: 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:18.303995+0000 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:18.303995+0000 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:18.305 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:18.306 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:18.306 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:18.306 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:18.306 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 Processor -- start 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 -- start start 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient: init 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 0x7fe4c009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-b con 0x7fe4c00a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe4c00a04f0 0x7fe4c00a0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-a con 0x7fe4c00a04f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4c01369a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c009dae0 con 0x7fe4c01369a0 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c00a21d0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c009ddc0 con 0x7fe4c00a0120 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4c4b59700 10 monclient(hunting): authenticate will time out at 900.397768s 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4be59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 msgr2=0x7fe4c009e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4be59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 0x7fe4c009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bed9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe4c01369a0 0x7fe4c0138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:60818/0 (socket says 172.21.15.150:60818) 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bed9d700 1 -- 172.21.15.150:0/3937555525 learned_addr learned my addr 172.21.15.150:0/3937555525 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/3937555525 >> v2:172.21.15.142:3300/0 conn(0x7fe4c00a04f0 0x7fe4c00a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): get_auth_request con 0x7fe4c00a04f0 auth_method 0 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_done global_id 4503 payload 274 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 msgr2=0x7fe4c009e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 0x7fe4c009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/3937555525 >> v1:172.21.15.150:6789/0 conn(0x7fe4c01369a0 legacy=0x7fe4c0138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.302+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/3937555525 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c00a1590 con 0x7fe4c00a04f0 2024-09-18T10:10:18.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 msgr2=0x7fe4c00a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c00a0920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00098e0 tx=0x7fe4b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcde420794406e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 1 -- 172.21.15.150:0/3937555525 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000ba70 con 0x7fe4c00a04f0 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: dump: 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.314 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: _finish_auth 0 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 1 -- 172.21.15.150:0/3937555525 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b000bbf0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 5 monclient: authenticate success, global_id 4503 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:18.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 4 set_mon_vals no callback set 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 1 -- 172.21.15.150:0/3937555525 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0011420 con 0x7fe4c00a04f0 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4bd59a700 10 monclient: dump: 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.316 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 10 monclient: shutdown 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 -- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 msgr2=0x7fe4c00a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c00a0920 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00098e0 tx=0x7fe4b00091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 -- 172.21.15.150:0/3937555525 shutdown_connections 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c00a0920 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/3937555525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c00a0120 0x7fe4c009e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 -- 172.21.15.150:0/3937555525 >> 172.21.15.150:0/3937555525 conn(0x7fe4c00a5830 msgr2=0x7fe4c00a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 -- 172.21.15.150:0/3937555525 shutdown_connections 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.306+0000 7fe4c4b59700 1 -- 172.21.15.150:0/3937555525 wait complete. 2024-09-18T10:10:18.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient: build_initial_monmap 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient: monmap: 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 Processor -- start 2024-09-18T10:10:18.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 -- start start 2024-09-18T10:10:18.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient: init 2024-09-18T10:10:18.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:18.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:18.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-b con 0x7fe4c00a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:18.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c0142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-a con 0x7fe4c00a04f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 0x7fe4c01428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): picked mon.noname-c con 0x7fe4c01369a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c012eba0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c009d720 con 0x7fe4c00a0120 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c009c010 con 0x7fe4c01369a0 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4c4b59700 10 monclient(hunting): authenticate will time out at 900.405922s 2024-09-18T10:10:18.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bed9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 msgr2=0x7fe4c01428a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 0x7fe4c01428a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c0142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47794/0 (socket says 172.21.15.150:47794) 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 1 -- 172.21.15.150:0/1850362632 learned_addr learned my addr 172.21.15.150:0/1850362632 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): get_auth_request con 0x7fe4c00a0120 auth_method 0 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): get_auth_request con 0x7fe4c00a04f0 auth_method 0 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 msgr2=0x7fe4c01428a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:18.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 0x7fe4c01428a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 msgr2=0x7fe4c0141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0009420 con 0x7fe4c00a04f0 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.310+0000 7fe4be59c700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4bdd9b700 1 -- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 msgr2=0x7fe4c0142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4bdd9b700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c0142360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00093f0 tx=0x7fe4b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0011e30 con 0x7fe4c00a04f0 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:18.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: dump: 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:18.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0020070 con 0x7fe4c00a04f0 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 5 monclient: authenticate success, global_id 4506 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _renew_subs 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c0130db0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b001b680 con 0x7fe4c00a04f0 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4bcd99700 4 set_mon_vals no callback set 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 10 monclient: dump: 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.328 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _renew_subs 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c012e6d0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe4b0022030 con 0x7fe4c00a04f0 2024-09-18T10:10:18.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c009d8b0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 0x7fe4a0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.314+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7fe4b004a180 con 0x7fe4c00a04f0 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.318+0000 7fe4be59c700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 0x7fe4a0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.318+0000 7fe4be59c700 10 monclient: get_auth_request con 0x7fe4a0035a70 auth_method 0 2024-09-18T10:10:18.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.318+0000 7fe4be59c700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 0x7fe4a0037f20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8003fd0 tx=0x7fe4a8009220 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.330+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe4b001ba70 con 0x7fe4c00a04f0 2024-09-18T10:10:18.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.330+0000 7fe4b67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.330+0000 7fe4b67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:18.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4c4b59700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4c4b59700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4c4b59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fe4c009d8b0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7fe4b002cb70 con 0x7fe4c00a04f0 2024-09-18T10:10:18.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4b67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.498+0000 7fe4b67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:18.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4c4b59700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4c4b59700 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4c4b59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fe4c009d8b0 con 0x7fe4c00a04f0 2024-09-18T10:10:18.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4b67fc700 1 -- 172.21.15.150:0/1850362632 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=0 v20) v1 ==== 105+0+14 (secure 0 0 0) 0x7fe4b001ba70 con 0x7fe4c00a04f0 2024-09-18T10:10:18.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4b67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:18.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.590+0000 7fe4b67fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:18.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 msgr2=0x7fe4a0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 0x7fe4a0037f20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8003fd0 tx=0x7fe4a8009220 comp rx=0 tx=0).stop 2024-09-18T10:10:18.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 10 monclient: shutdown 2024-09-18T10:10:18.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:18.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 msgr2=0x7fe4c0142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c0142360 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00093f0 tx=0x7fe4b0016000 comp rx=0 tx=0).stop 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 shutdown_connections 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe4a0035a70 0x7fe4a0037f20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4c01369a0 0x7fe4c01428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4c00a04f0 0x7fe4c0142360 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 --2- 172.21.15.150:0/1850362632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4c00a0120 0x7fe4c0141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 >> 172.21.15.150:0/1850362632 conn(0x7fe4c00a5830 msgr2=0x7fe4c009abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 shutdown_connections 2024-09-18T10:10:18.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.594+0000 7fe4c4b59700 1 -- 172.21.15.150:0/1850362632 wait complete. 2024-09-18T10:10:18.608 INFO:tasks.workunit.client.0.smithi150.stdout:pg_num_min: 2 2024-09-18T10:10:18.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_max 33 2024-09-18T10:10:18.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 10 monclient: get_monmap_and_config 2024-09-18T10:10:18.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 10 monclient: build_initial_monmap 2024-09-18T10:10:18.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 10 monclient: monmap: 2024-09-18T10:10:18.711 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:18.711 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:18.710480+0000 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:18.710480+0000 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:18.712 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 1 Processor -- start 2024-09-18T10:10:18.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 1 -- start start 2024-09-18T10:10:18.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.706+0000 7f0b0bc16700 10 monclient: init 2024-09-18T10:10:18.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b040a0290 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b040a0d70 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b0409ea40 0x7f0b0413b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-a con 0x7f0b0409ea40 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0409e360 con 0x7f0b040a0d70 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0409da50 con 0x7f0b040a0290 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b040a1fb0 con 0x7f0b0409ea40 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0bc16700 10 monclient(hunting): authenticate will time out at 900.804380s 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b099b2700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b040a0290 0x7f0b040a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:60826/0 (socket says 172.21.15.150:60826) 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b099b2700 1 -- 172.21.15.150:0/3196098979 learned_addr learned my addr 172.21.15.150:0/3196098979 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:18.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 --2- 172.21.15.150:0/3196098979 >> v2:172.21.15.142:3300/0 conn(0x7f0b0409ea40 0x7f0b0413b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): get_auth_request con 0x7f0b0409ea40 auth_method 0 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_done global_id 4509 payload 274 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/3196098979 >> v1:172.21.15.150:6789/0 conn(0x7f0b040a0290 legacy=0x7f0b040a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/3196098979 >> v1:172.21.15.142:6789/0 conn(0x7f0b040a0d70 legacy=0x7f0b04137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient: _finish_hunting 0 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 monclient: found mon.noname-a 2024-09-18T10:10:18.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:18.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:18.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/3196098979 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0409dd30 con 0x7f0b0409ea40 2024-09-18T10:10:18.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/3196098979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b0409ea40 msgr2=0x7f0b0413b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:18.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.710+0000 7f0b0a1b3700 1 --2- 172.21.15.150:0/3196098979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b0409ea40 0x7f0b0413b150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009e60 tx=0x7f0afc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b8c3fda1b728d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 1 -- 172.21.15.150:0/3196098979 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00e070 con 0x7f0b0409ea40 2024-09-18T10:10:18.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: dump: 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.719 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: _finish_auth 0 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 5 monclient: authenticate success, global_id 4509 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 1 -- 172.21.15.150:0/3196098979 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc012430 con 0x7f0b0409ea40 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 4 set_mon_vals no callback set 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 1 -- 172.21.15.150:0/3196098979 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0127d0 con 0x7f0b0409ea40 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:18.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b089b0700 10 monclient: dump: 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: shutdown 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:18.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- 172.21.15.150:0/3196098979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b0409ea40 msgr2=0x7f0b0413b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/3196098979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b0409ea40 0x7f0b0413b150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009e60 tx=0x7f0afc009780 comp rx=0 tx=0).stop 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- 172.21.15.150:0/3196098979 shutdown_connections 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/3196098979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b0409ea40 0x7f0b0413b150 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- 172.21.15.150:0/3196098979 >> 172.21.15.150:0/3196098979 conn(0x7f0b040a58a0 msgr2=0x7f0b040a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- 172.21.15.150:0/3196098979 shutdown_connections 2024-09-18T10:10:18.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- 172.21.15.150:0/3196098979 wait complete. 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: build_initial_monmap 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: monmap: 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:18.725 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 Processor -- start 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- start start 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: init 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 0x7f0b04141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-c con 0x7f0b0409ea40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-a con 0x7f0b040a0290 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): picked mon.noname-b con 0x7f0b040a0d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:18.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0409d3f0 con 0x7f0b040a0290 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0409c130 con 0x7f0b040a0d70 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04092940 con 0x7f0b0409ea40 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0bc16700 10 monclient(hunting): authenticate will time out at 900.811506s 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b099b2700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 msgr2=0x7f0b04141d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b099b2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 0x7f0b04141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:51054/0 (socket says 172.21.15.150:51054) 2024-09-18T10:10:18.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47802/0 (socket says 172.21.15.150:47802) 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 1 -- 172.21.15.150:0/2659569422 learned_addr learned my addr 172.21.15.150:0/2659569422 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 10 monclient(hunting): get_auth_request con 0x7f0b040a0290 auth_method 0 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b091b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 10 monclient(hunting): get_auth_request con 0x7f0b040a0d70 auth_method 0 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.714+0000 7f0b0a1b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b091b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b091b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b091b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient(hunting): handle_auth_done global_id 14203 payload 824 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 msgr2=0x7f0b04141d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 0x7f0b04141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 msgr2=0x7f0b04142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient: _finish_hunting 0 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 monclient: found mon.noname-b 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:18.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc009a20 con 0x7f0b040a0d70 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b091b1700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 -- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 msgr2=0x7f0b0408e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0a1b3700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b04064fb0 tx=0x7f0afc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc026430 con 0x7f0b040a0d70 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: dump: 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.730 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc021070 con 0x7f0b040a0d70 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 5 monclient: authenticate success, global_id 14203 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 10 monclient: _renew_subs 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b04091a40 con 0x7f0b040a0d70 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc026820 con 0x7f0b040a0d70 2024-09-18T10:10:18.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0af9ffb700 10 monclient: dump: 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:18.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0afbfff700 4 set_mon_vals no callback set 2024-09-18T10:10:18.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 10 monclient: _renew_subs 2024-09-18T10:10:18.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:18.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.718+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b040913e0 con 0x7f0b040a0d70 2024-09-18T10:10:18.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b0bc16700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b0bc16700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b0bc16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:18.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b04091c80 con 0x7f0b040a0d70 2024-09-18T10:10:18.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0afc023030 con 0x7f0b040a0d70 2024-09-18T10:10:18.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0af9ffb700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 0x7f0ae8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:18.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f0afc04a980 con 0x7f0b040a0d70 2024-09-18T10:10:18.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b099b2700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 0x7f0ae8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:18.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.722+0000 7f0b099b2700 10 monclient: get_auth_request con 0x7f0ae8035ac0 auth_method 0 2024-09-18T10:10:18.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.726+0000 7f0b099b2700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 0x7f0ae8037f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b00004860 tx=0x7f0b00007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:18.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.734+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0afc011b70 con 0x7f0b040a0d70 2024-09-18T10:10:18.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.734+0000 7f0af9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:18.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.734+0000 7f0af9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:18.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.898+0000 7f0b0bc16700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:18.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.898+0000 7f0b0bc16700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:18.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.898+0000 7f0b0bc16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:18.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:18.898+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f0b04091c80 con 0x7f0b040a0d70 2024-09-18T10:10:19.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.194+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v21) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0afc011390 con 0x7f0b040a0d70 2024-09-18T10:10:19.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.194+0000 7f0af9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:19.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.194+0000 7f0af9ffb700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 33 2024-09-18T10:10:19.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.298+0000 7f0b0bc16700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:19.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.298+0000 7f0b0bc16700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:19.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.298+0000 7f0b0bc16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:19.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:19.298+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"} v 0) v1 -- 0x7f0b04091c80 con 0x7f0b040a0d70 2024-09-18T10:10:20.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.198+0000 7f0af9ffb700 1 -- 172.21.15.150:0/2659569422 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}]=0 set pool 3 pg_num_max to 33 v22) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0afc011570 con 0x7f0b040a0d70 2024-09-18T10:10:20.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.198+0000 7f0af9ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "33"}] 2024-09-18T10:10:20.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.198+0000 7f0af9ffb700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 33 2024-09-18T10:10:20.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 msgr2=0x7f0ae8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 0x7f0ae8037f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b00004860 tx=0x7f0b00007c10 comp rx=0 tx=0).stop 2024-09-18T10:10:20.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 10 monclient: shutdown 2024-09-18T10:10:20.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:20.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 msgr2=0x7f0b0408e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b04064fb0 tx=0x7f0afc018000 comp rx=0 tx=0).stop 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 shutdown_connections 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ae8035ac0 0x7f0ae8037f70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b040a0d70 0x7f0b0408e1d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b040a0290 0x7f0b04142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 --2- 172.21.15.150:0/2659569422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b0409ea40 0x7f0b04141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 >> 172.21.15.150:0/2659569422 conn(0x7f0b040a58a0 msgr2=0x7f0b0409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 shutdown_connections 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.202+0000 7f0b0bc16700 1 -- 172.21.15.150:0/2659569422 wait complete. 2024-09-18T10:10:20.207 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 3 pg_num_max to 33 2024-09-18T10:10:20.223 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-18T10:10:20.223 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 33 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 10 monclient: get_monmap_and_config 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 10 monclient: build_initial_monmap 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 10 monclient: monmap: 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:20.326489+0000 2024-09-18T10:10:20.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:20.326489+0000 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:20.328 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 1 Processor -- start 2024-09-18T10:10:20.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 1 -- start start 2024-09-18T10:10:20.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.322+0000 7fb9cef00700 10 monclient: init 2024-09-18T10:10:20.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:20.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T10:10:20.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb9c8096a70 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb9c8096e40 0x7fb9c8097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-a con 0x7fb9c8096e40 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9c8097780 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80a2c00 con 0x7fb9c8097780 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9ccc9c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb9c8096a70 0x7fb9c80998e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47772/0 (socket says 172.21.15.150:47772) 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c80985a0 con 0x7fb9c8096a70 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2401425615 learned_addr learned my addr 172.21.15.150:0/2401425615 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c8098f60 con 0x7fb9c8096e40 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9cef00700 10 monclient(hunting): authenticate will time out at 902.421717s 2024-09-18T10:10:20.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 1 --2- 172.21.15.150:0/2401425615 >> v2:172.21.15.142:3300/0 conn(0x7fb9c8096e40 0x7fb9c8097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): get_auth_request con 0x7fb9c8096e40 auth_method 0 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_done global_id 4515 payload 274 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 1 -- 172.21.15.150:0/2401425615 >> v1:172.21.15.150:6789/0 conn(0x7fb9c8096a70 legacy=0x7fb9c80998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 1 -- 172.21.15.150:0/2401425615 >> v1:172.21.15.142:6789/0 conn(0x7fb9c8097780 legacy=0x7fb9c8136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:20.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 1 monclient: found mon.noname-a 2024-09-18T10:10:20.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:20.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:20.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.326+0000 7fb9c7fff700 1 -- 172.21.15.150:0/2401425615 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80712d0 con 0x7fb9c8096e40 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c7fff700 1 -- 172.21.15.150:0/2401425615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 msgr2=0x7fb9c8097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c7fff700 1 --2- 172.21.15.150:0/2401425615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c8097240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0098e0 tx=0x7fb9bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82776fb0d4b1f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 1 -- 172.21.15.150:0/2401425615 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00ba70 con 0x7fb9c8096e40 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: dump: 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.336 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 5 monclient: authenticate success, global_id 4515 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 1 -- 172.21.15.150:0/2401425615 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc00bbd0 con 0x7fb9c8096e40 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:20.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 1 -- 172.21.15.150:0/2401425615 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc011420 con 0x7fb9c8096e40 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 4 set_mon_vals no callback set 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9c77fe700 10 monclient: dump: 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.338 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 10 monclient: shutdown 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 msgr2=0x7fb9c8097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2401425615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c8097240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0098e0 tx=0x7fb9bc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 shutdown_connections 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2401425615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c8097240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 >> 172.21.15.150:0/2401425615 conn(0x7fb9c8068430 msgr2=0x7fb9c80a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 shutdown_connections 2024-09-18T10:10:20.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.330+0000 7fb9cef00700 1 -- 172.21.15.150:0/2401425615 wait complete. 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient: build_initial_monmap 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient: monmap: 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:20.344 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 Processor -- start 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 -- start start 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient: init 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-b con 0x7fb9c8096a70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:20.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c811d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-a con 0x7fb9c8096e40 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 0x7fb9c811d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): picked mon.noname-c con 0x7fb9c8097780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8097cb0 con 0x7fb9c8096e40 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80a0c30 con 0x7fb9c8096a70 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c811f220 con 0x7fb9c8097780 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cef00700 10 monclient(hunting): authenticate will time out at 902.430505s 2024-09-18T10:10:20.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45808/0 (socket says 172.21.15.150:45808) 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2706352448 learned_addr learned my addr 172.21.15.150:0/2706352448 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): get_auth_request con 0x7fb9c8096a70 auth_method 0 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cd49d700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 msgr2=0x7fb9c811d710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9cd49d700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 0x7fb9c811d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c811d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): get_auth_request con 0x7fb9c8096e40 auth_method 0 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient(hunting): handle_auth_done global_id 14206 payload 824 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 msgr2=0x7fb9c811d710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 0x7fb9c811d710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 msgr2=0x7fb9c811d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c811d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient: _finish_hunting 0 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 monclient: found mon.noname-b 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.334+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc009420 con 0x7fb9c8096a70 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9ccc9c700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 msgr2=0x7fb9c81378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9ccc9c700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c610 tx=0x7fb9b8004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8005d40 con 0x7fb9c8096a70 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: dump: 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.349 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 5 monclient: authenticate success, global_id 14206 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _renew_subs 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b80070d0 con 0x7fb9c8096a70 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c81207a0 con 0x7fb9c8096a70 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8007490 con 0x7fb9c8096a70 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 10 monclient: dump: 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.351 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c6ffd700 4 set_mon_vals no callback set 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _renew_subs 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c8136cd0 con 0x7fb9c8096a70 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb9b801b3b0 con 0x7fb9c8096a70 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 0x7fb9b4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c81209c0 con 0x7fb9c8096a70 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7fb9b80478d0 con 0x7fb9c8096a70 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c7fff700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 0x7fb9b4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.338+0000 7fb9c7fff700 10 monclient: get_auth_request con 0x7fb9b4035a70 auth_method 0 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.342+0000 7fb9c7fff700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 0x7fb9b4037f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0030c0 tx=0x7fb9bc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.346+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb9b801a360 con 0x7fb9c8096a70 2024-09-18T10:10:20.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.346+0000 7fb9c4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.346+0000 7fb9c4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:20.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.510+0000 7fb9cef00700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.510+0000 7fb9cef00700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.510+0000 7fb9cef00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.510+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7fb9c806fc90 con 0x7fb9c8096a70 2024-09-18T10:10:20.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.510+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7fb9b8017030 con 0x7fb9c8096a70 2024-09-18T10:10:20.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.514+0000 7fb9c4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.514+0000 7fb9c4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9cef00700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9cef00700 10 monclient: _send_command 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9cef00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7fb9c8063ed0 con 0x7fb9c8096a70 2024-09-18T10:10:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9c4ff9700 1 -- 172.21.15.150:0/2706352448 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=0 v22) v1 ==== 105+0+15 (secure 0 0 0) 0x7fb9b8017020 con 0x7fb9c8096a70 2024-09-18T10:10:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9c4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.606+0000 7fb9c4ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 msgr2=0x7fb9b4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 0x7fb9b4037f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0030c0 tx=0x7fb9bc016000 comp rx=0 tx=0).stop 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 10 monclient: shutdown 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 msgr2=0x7fb9c81378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800c610 tx=0x7fb9b8004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:20.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 shutdown_connections 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb9b4035a70 0x7fb9b4037f20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9c8097780 0x7fb9c811d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9c8096e40 0x7fb9c811d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 --2- 172.21.15.150:0/2706352448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9c8096a70 0x7fb9c81378a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 >> 172.21.15.150:0/2706352448 conn(0x7fb9c8068430 msgr2=0x7fb9c806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 shutdown_connections 2024-09-18T10:10:20.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.610+0000 7fb9cef00700 1 -- 172.21.15.150:0/2706352448 wait complete. 2024-09-18T10:10:20.627 INFO:tasks.workunit.client.0.smithi150.stdout:pg_num_max: 33 2024-09-18T10:10:20.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_min 9 2024-09-18T10:10:20.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:20.628 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_min 9 2024-09-18T10:10:20.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:20.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 10 monclient: build_initial_monmap 2024-09-18T10:10:20.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 10 monclient: monmap: 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:20.729831+0000 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:20.729831+0000 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:20.731 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 1 Processor -- start 2024-09-18T10:10:20.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 1 -- start start 2024-09-18T10:10:20.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.726+0000 7f2982a0e700 10 monclient: init 2024-09-18T10:10:20.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:20.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:10:20.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f297c09c7c0 0x7f297c0978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-a con 0x7f297c09c7c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f297c09cb90 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c1399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-b con 0x7f297c137600 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297c071630 con 0x7f297c09cb90 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297c096ba0 con 0x7f297c09c7c0 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c09d6a0 con 0x7f297c137600 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2982a0e700 10 monclient(hunting): authenticate will time out at 902.824727s 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2980fab700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 msgr2=0x7f297c1399e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:20.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f2980fab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c1399e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f297c09c7c0 0x7f297c0978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f297c09c7c0 0x7f297c0978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36510/0 (socket says 172.21.15.150:36510) 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 -- 172.21.15.150:0/2021229677 learned_addr learned my addr 172.21.15.150:0/2021229677 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): get_auth_request con 0x7f297c09c7c0 auth_method 0 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient(hunting): handle_auth_done global_id 4521 payload 274 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 -- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 msgr2=0x7f297c1399e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:20.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 --2- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c1399e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 -- 172.21.15.150:0/2021229677 >> v1:172.21.15.142:6789/0 conn(0x7f297c09cb90 legacy=0x7f297c09cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 monclient: found mon.noname-a 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:20.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.730+0000 7f297bfff700 1 -- 172.21.15.150:0/2021229677 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c09e180 con 0x7f297c09c7c0 2024-09-18T10:10:20.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 -- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09c7c0 msgr2=0x7f297c0978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 --2- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09c7c0 0x7f297c0978d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f296c0099b0 tx=0x7f296c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912e36989d4e320b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 1 -- 172.21.15.150:0/2021229677 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c00bb50 con 0x7f297c09c7c0 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: dump: 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.739 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 5 monclient: authenticate success, global_id 4521 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 1 -- 172.21.15.150:0/2021229677 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c00bcb0 con 0x7f297c09c7c0 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 4 set_mon_vals no callback set 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 1 -- 172.21.15.150:0/2021229677 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c011670 con 0x7f297c09c7c0 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297b7fe700 10 monclient: dump: 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: shutdown 2024-09-18T10:10:20.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09c7c0 msgr2=0x7f297c0978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09c7c0 0x7f297c0978d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f296c0099b0 tx=0x7f296c009250 comp rx=0 tx=0).stop 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- 172.21.15.150:0/2021229677 shutdown_connections 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c1399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- 172.21.15.150:0/2021229677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09c7c0 0x7f297c0978d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- 172.21.15.150:0/2021229677 >> 172.21.15.150:0/2021229677 conn(0x7f297c068590 msgr2=0x7f297c0a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- 172.21.15.150:0/2021229677 shutdown_connections 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- 172.21.15.150:0/2021229677 wait complete. 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: build_initial_monmap 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: monmap: 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:20.742 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 Processor -- start 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- start start 2024-09-18T10:10:20.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: init 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c09c7c0 0x7f297c12ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-b con 0x7f297c09c7c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09cb90 0x7f297c134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-a con 0x7f297c09cb90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): picked mon.noname-c con 0x7f297c137600 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c119190 con 0x7f297c09cb90 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c09e450 con 0x7f297c09c7c0 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c119b70 con 0x7f297c137600 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2982a0e700 10 monclient(hunting): authenticate will time out at 902.830652s 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2980fab700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 msgr2=0x7f297c134e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c09c7c0 0x7f297c12ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f2980fab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c09c7c0 0x7f297c12ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45824/0 (socket says 172.21.15.150:45824) 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 -- 172.21.15.150:0/17371857 learned_addr learned my addr 172.21.15.150:0/17371857 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:20.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): get_auth_request con 0x7f297c09c7c0 auth_method 0 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09cb90 0x7f297c134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): get_auth_request con 0x7f297c09cb90 auth_method 0 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:20.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f29737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient(hunting): handle_auth_done global_id 14209 payload 824 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 -- 172.21.15.150:0/17371857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 msgr2=0x7f297c134e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f297c137600 0x7f297c134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 -- 172.21.15.150:0/17371857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09cb90 msgr2=0x7f297c134930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f297c09cb90 0x7f297c134930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 monclient: found mon.noname-b 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.734+0000 7f297bfff700 1 -- 172.21.15.150:0/17371857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0094f0 con 0x7f297c09c7c0 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f297bfff700 1 -- 172.21.15.150:0/17371857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c09c7c0 msgr2=0x7f297c12ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f297bfff700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f297c09c7c0 0x7f297c12ffd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f296c0117e0 tx=0x7f296c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c01b470 con 0x7f297c09c7c0 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: dump: 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 5 monclient: authenticate success, global_id 14209 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c01b5d0 con 0x7f297c09c7c0 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _renew_subs 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 1 -- 172.21.15.150:0/17371857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c139150 con 0x7f297c09c7c0 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c01b990 con 0x7f297c09c7c0 2024-09-18T10:10:20.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 10 monclient: dump: 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:20.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _renew_subs 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f297affd700 4 set_mon_vals no callback set 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 1 -- 172.21.15.150:0/17371857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c136640 con 0x7f297c09c7c0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f296c01bd40 con 0x7f297c09c7c0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2982a0e700 1 -- 172.21.15.150:0/17371857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c12f490 con 0x7f297c09c7c0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.738+0000 7f2978ff9700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f295c035aa0 0x7f295c037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.742+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f296c049e40 con 0x7f297c09c7c0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.742+0000 7f29737fe700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f295c035aa0 0x7f295c037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:20.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.742+0000 7f29737fe700 10 monclient: get_auth_request con 0x7f295c035aa0 auth_method 0 2024-09-18T10:10:20.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.742+0000 7f29737fe700 1 --2- 172.21.15.150:0/17371857 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f295c035aa0 0x7f295c037f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f297c136940 tx=0x7f2964007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:20.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.746+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f296c018950 con 0x7f297c09c7c0 2024-09-18T10:10:20.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.746+0000 7f2978ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:20.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.746+0000 7f2978ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:20.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.910+0000 7f2982a0e700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T10:10:20.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.910+0000 7f2982a0e700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T10:10:20.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.910+0000 7f2982a0e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:20.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.910+0000 7f2982a0e700 1 -- 172.21.15.150:0/17371857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"} v 0) v1 -- 0x7f297c119600 con 0x7f297c09c7c0 2024-09-18T10:10:20.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.914+0000 7f2978ff9700 1 -- 172.21.15.150:0/17371857 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}]=-22 specified pg_num_min 9 > pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7f296c02d350 con 0x7f297c09c7c0 2024-09-18T10:10:20.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.914+0000 7f2978ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "9"}] 2024-09-18T10:10:20.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:20.914+0000 7f2978ff9700 10 monclient: _finish_command 2 = mon:22 specified pg_num_min 9 > pg_num 8 2024-09-18T10:10:20.919 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: specified pg_num_min 9 > pg_num 8 2024-09-18T10:10:20.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:20.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set testpool1234 pg_num_max 7 2024-09-18T10:10:20.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:20.923 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_max 7 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 10 monclient: get_monmap_and_config 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 10 monclient: monmap: 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:21.026495+0000 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:21.026495+0000 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:21.028 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 1 Processor -- start 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 1 -- start start 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.022+0000 7f113a0d5700 10 monclient: init 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f113409d000 0x7f11340a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-a con 0x7f113409d000 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f113409e780 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f11341379b0 0x7f1134139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-c con 0x7f11341379b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340699a0 con 0x7f113409e780 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340a1640 con 0x7f113409d000 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113409e3d0 con 0x7f11341379b0 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient(hunting): authenticate will time out at 903.120234s 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f11341379b0 0x7f1134139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f11341379b0 0x7f1134139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45830/0 (socket says 172.21.15.150:45830) 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 -- 172.21.15.150:0/2808544162 learned_addr learned my addr 172.21.15.150:0/2808544162 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): get_auth_request con 0x7f11341379b0 auth_method 0 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 1 --2- 172.21.15.150:0/2808544162 >> v2:172.21.15.142:3300/0 conn(0x7f113409d000 0x7f11340a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): get_auth_request con 0x7f113409d000 auth_method 0 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient(hunting): handle_auth_done global_id 14212 payload 274 2024-09-18T10:10:21.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 -- 172.21.15.150:0/2808544162 >> v2:172.21.15.142:3300/0 conn(0x7f113409d000 msgr2=0x7f11340a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 --2- 172.21.15.150:0/2808544162 >> v2:172.21.15.142:3300/0 conn(0x7f113409d000 0x7f11340a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 -- 172.21.15.150:0/2808544162 >> v1:172.21.15.142:6789/0 conn(0x7f113409e780 legacy=0x7f11340a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 monclient: found mon.noname-c 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 -- 172.21.15.150:0/2808544162 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113409ebf0 con 0x7f11341379b0 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 -- 172.21.15.150:0/2808544162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341379b0 msgr2=0x7f1134139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f1133fff700 1 --2- 172.21.15.150:0/2808544162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341379b0 0x7f1134139d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f113409d9c0 tx=0x7f112c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a199d2906d394093 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 1 -- 172.21.15.150:0/2808544162 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00e070 con 0x7f11341379b0 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: dump: 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 5 monclient: authenticate success, global_id 14212 2024-09-18T10:10:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 1 -- 172.21.15.150:0/2808544162 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00bc30 con 0x7f11341379b0 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f113a0d5700 4 set_mon_vals no callback set 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 1 -- 172.21.15.150:0/2808544162 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0092e0 con 0x7f11341379b0 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.026+0000 7f11327fc700 10 monclient: dump: 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: shutdown 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- 172.21.15.150:0/2808544162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341379b0 msgr2=0x7f1134139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- 172.21.15.150:0/2808544162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341379b0 0x7f1134139d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f113409d9c0 tx=0x7f112c009840 comp rx=0 tx=0).stop 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- 172.21.15.150:0/2808544162 shutdown_connections 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- 172.21.15.150:0/2808544162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11341379b0 0x7f1134139d90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- 172.21.15.150:0/2808544162 >> v2:172.21.15.142:3300/0 conn(0x7f113409d000 0x7f11340a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- 172.21.15.150:0/2808544162 >> 172.21.15.150:0/2808544162 conn(0x7f11340a5830 msgr2=0x7f11340a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- 172.21.15.150:0/2808544162 shutdown_connections 2024-09-18T10:10:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- 172.21.15.150:0/2808544162 wait complete. 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: monmap: 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 Processor -- start 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- start start 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: init 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f113409d000 0x7f113412c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-a con 0x7f113409d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113409e780 0x7f113412ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-b con 0x7f113409e780 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11341379b0 0x7f113412cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): picked mon.noname-c con 0x7f11341379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113412e0d0 con 0x7f113409d000 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113412fa90 con 0x7f113409e780 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113412e850 con 0x7f11341379b0 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f113a0d5700 10 monclient(hunting): authenticate will time out at 903.125696s 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1133fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11341379b0 msgr2=0x7f113412cf40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113409e780 0x7f113412ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1133fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11341379b0 0x7f113412cf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113409e780 0x7f113412ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45832/0 (socket says 172.21.15.150:45832) 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 -- 172.21.15.150:0/1822172283 learned_addr learned my addr 172.21.15.150:0/1822172283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): get_auth_request con 0x7f113409e780 auth_method 0 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f11337fe700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f113409d000 0x7f113412c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f11337fe700 10 monclient(hunting): get_auth_request con 0x7f113409d000 auth_method 0 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f11337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f11337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f11337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient(hunting): handle_auth_done global_id 14215 payload 824 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 -- 172.21.15.150:0/1822172283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11341379b0 msgr2=0x7f113412cf40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f11341379b0 0x7f113412cf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 -- 172.21.15.150:0/1822172283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f113409d000 msgr2=0x7f113412c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f113409d000 0x7f113412c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 monclient: found mon.noname-b 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.030+0000 7f1132ffd700 1 -- 172.21.15.150:0/1822172283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009ae0 con 0x7f113409e780 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f1132ffd700 1 -- 172.21.15.150:0/1822172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113409e780 msgr2=0x7f113412ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f1132ffd700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f113409e780 0x7f113412ca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f112400b5e0 tx=0x7f1124009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112400f720 con 0x7f113409e780 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: dump: 2024-09-18T10:10:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 5 monclient: authenticate success, global_id 14215 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _renew_subs 2024-09-18T10:10:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 1 -- 172.21.15.150:0/1822172283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113412f7b0 con 0x7f113409e780 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _renew_subs 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 1 -- 172.21.15.150:0/1822172283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1134130930 con 0x7f113409e780 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112400f8a0 con 0x7f113409e780 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112400fc60 con 0x7f113409e780 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 10 monclient: dump: 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f1131ffb700 4 set_mon_vals no callback set 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f113a0d5700 1 -- 172.21.15.150:0/1822172283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11140050b0 con 0x7f113409e780 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f112401a400 con 0x7f113409e780 2024-09-18T10:10:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1120035a70 0x7f1120037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f1124047ea0 con 0x7f113409e780 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f11337fe700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1120035a70 0x7f1120037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.034+0000 7f11337fe700 10 monclient: get_auth_request con 0x7f1120035a70 auth_method 0 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.038+0000 7f11337fe700 1 --2- 172.21.15.150:0/1822172283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1120035a70 0x7f1120037f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1128009ca0 tx=0x7f1128008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.046+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1124017880 con 0x7f113409e780 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.046+0000 7f111b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.046+0000 7f111b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.210+0000 7f113a0d5700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T10:10:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.210+0000 7f113a0d5700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T10:10:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.214+0000 7f113a0d5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.214+0000 7f113a0d5700 1 -- 172.21.15.150:0/1822172283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"} v 0) v1 -- 0x7f1114005360 con 0x7f113409e780 2024-09-18T10:10:21.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.214+0000 7f111b7fe700 1 -- 172.21.15.150:0/1822172283 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}]=-22 specified pg_num_max 7 < pg_num 8 v22) v1 ==== 150+0+0 (secure 0 0 0) 0x7f11240172b0 con 0x7f113409e780 2024-09-18T10:10:21.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.214+0000 7f111b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "7"}] 2024-09-18T10:10:21.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.214+0000 7f111b7fe700 10 monclient: _finish_command 2 = mon:22 specified pg_num_max 7 < pg_num 8 2024-09-18T10:10:21.220 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: specified pg_num_max 7 < pg_num 8 2024-09-18T10:10:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 1 2024-09-18T10:10:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:21.224 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num 1 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.310+0000 7f67443fd700 10 monclient: get_monmap_and_config 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.310+0000 7f67443fd700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.310+0000 7f67443fd700 10 monclient: monmap: 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:21.315289+0000 2024-09-18T10:10:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:21.315289+0000 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:21.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 Processor -- start 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 -- start start 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient: init 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f673c06b5a0 0x7f673c1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-a con 0x7f673c06b5a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f673c132700 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 0x7f673c09ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-b con 0x7f673c132b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c06aac0 con 0x7f673c132700 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c06b970 con 0x7f673c06b5a0 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c09fb10 con 0x7f673c132b40 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f67443fd700 10 monclient(hunting): authenticate will time out at 903.408813s 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f674299a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 msgr2=0x7f673c09ee80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f674299a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 0x7f673c09ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f673c06b5a0 0x7f673c1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f673c06b5a0 0x7f673c1321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36536/0 (socket says 172.21.15.150:36536) 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 -- 172.21.15.150:0/2083251122 learned_addr learned my addr 172.21.15.150:0/2083251122 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): get_auth_request con 0x7f673c06b5a0 auth_method 0 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6741998700 1 --1- 172.21.15.150:0/2083251122 >> v1:172.21.15.142:6789/0 conn(0x7f673c132700 0x7f673c09e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:49862/0 (socket says 172.21.15.150:49862) 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient(hunting): handle_auth_done global_id 4533 payload 274 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 -- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 msgr2=0x7f673c09ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 --2- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 0x7f673c09ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 -- 172.21.15.150:0/2083251122 >> v1:172.21.15.142:6789/0 conn(0x7f673c132700 legacy=0x7f673c09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 monclient: found mon.noname-a 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.314+0000 7f6742199700 1 -- 172.21.15.150:0/2083251122 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c06a7e0 con 0x7f673c06b5a0 2024-09-18T10:10:21.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6742199700 1 -- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 msgr2=0x7f673c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6742199700 1 --2- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c1321c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6738009f20 tx=0x7f6738009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6584ef726168b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 1 -- 172.21.15.150:0/2083251122 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673800e070 con 0x7f673c06b5a0 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: dump: 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.330 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 1 -- 172.21.15.150:0/2083251122 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673800bc90 con 0x7f673c06b5a0 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 5 monclient: authenticate success, global_id 4533 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 1 -- 172.21.15.150:0/2083251122 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6738012660 con 0x7f673c06b5a0 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 4 set_mon_vals no callback set 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f6741197700 10 monclient: dump: 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 10 monclient: shutdown 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 msgr2=0x7f673c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 --2- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c1321c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6738009f20 tx=0x7f6738009840 comp rx=0 tx=0).stop 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- 172.21.15.150:0/2083251122 shutdown_connections 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 --2- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132b40 0x7f673c09ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 --2- 172.21.15.150:0/2083251122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c1321c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- 172.21.15.150:0/2083251122 >> 172.21.15.150:0/2083251122 conn(0x7f673c071b60 msgr2=0x7f673c0a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- 172.21.15.150:0/2083251122 shutdown_connections 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- 172.21.15.150:0/2083251122 wait complete. 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 10 monclient: monmap: 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 Processor -- start 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 1 -- start start 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.318+0000 7f67443fd700 10 monclient: init 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-a con 0x7f673c06b5a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132700 0x7f673c1315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-c con 0x7f673c132700 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673c132b40 0x7f673c131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): picked mon.noname-b con 0x7f673c132b40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c142910 con 0x7f673c06b5a0 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c11cc50 con 0x7f673c132b40 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c069b30 con 0x7f673c132700 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient(hunting): authenticate will time out at 903.416289s 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6741998700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132700 msgr2=0x7f673c1315d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6741998700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132700 0x7f673c1315d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673c132b40 0x7f673c131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673c132b40 0x7f673c131b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45840/0 (socket says 172.21.15.150:45840) 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 -- 172.21.15.150:0/1446330167 learned_addr learned my addr 172.21.15.150:0/1446330167 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): get_auth_request con 0x7f673c132b40 auth_method 0 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): get_auth_request con 0x7f673c06b5a0 auth_method 0 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6742199700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient(hunting): handle_auth_done global_id 14218 payload 824 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 -- 172.21.15.150:0/1446330167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132700 msgr2=0x7f673c1315d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673c132700 0x7f673c1315d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 -- 172.21.15.150:0/1446330167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 msgr2=0x7f673c141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f673c06b5a0 0x7f673c141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 monclient: found mon.noname-b 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 -- 172.21.15.150:0/1446330167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6738009ae0 con 0x7f673c132b40 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 -- 172.21.15.150:0/1446330167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673c132b40 msgr2=0x7f673c131b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f674299a700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673c132b40 0x7f673c131b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f673400b5d0 tx=0x7f6734009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734003dd0 con 0x7f673c132b40 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: dump: 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 5 monclient: authenticate success, global_id 14218 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6734006150 con 0x7f673c132b40 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _renew_subs 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 -- 172.21.15.150:0/1446330167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c0a31e0 con 0x7f673c132b40 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734006510 con 0x7f673c132b40 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67327fc700 10 monclient: dump: 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.340 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f6740996700 4 set_mon_vals no callback set 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _renew_subs 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.322+0000 7f67443fd700 1 -- 172.21.15.150:0/1446330167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c11d790 con 0x7f673c132b40 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67443fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67443fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67443fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67443fd700 1 -- 172.21.15.150:0/1446330167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c063ed0 con 0x7f673c132b40 2024-09-18T10:10:21.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f673401b3b0 con 0x7f673c132b40 2024-09-18T10:10:21.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67327fc700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6720035a70 0x7f6720037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f6734047890 con 0x7f673c132b40 2024-09-18T10:10:21.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f6742199700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6720035a70 0x7f6720037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.326+0000 7f6742199700 10 monclient: get_auth_request con 0x7f6720035a70 auth_method 0 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.330+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.330+0000 7f6742199700 1 --2- 172.21.15.150:0/1446330167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6720035a70 0x7f6720037f20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6738009f20 tx=0x7f6738017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.334+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6734019850 con 0x7f673c132b40 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.334+0000 7f67327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.334+0000 7f67327fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:21.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67443fd700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T10:10:21.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67443fd700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T10:10:21.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67443fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67443fd700 1 -- 172.21.15.150:0/1446330167 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"} v 0) v1 -- 0x7f673c09f3a0 con 0x7f673c132b40 2024-09-18T10:10:21.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67327fc700 1 -- 172.21.15.150:0/1446330167 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}]=-22 specified pg_num 1 < pg_num_min 2 v22) v1 ==== 146+0+0 (secure 0 0 0) 0x7f6734017070 con 0x7f673c132b40 2024-09-18T10:10:21.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "1"}] 2024-09-18T10:10:21.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.494+0000 7f67327fc700 10 monclient: _finish_command 2 = mon:22 specified pg_num 1 < pg_num_min 2 2024-09-18T10:10:21.501 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: specified pg_num 1 < pg_num_min 2 2024-09-18T10:10:21.504 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:21.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set testpool1234 pg_num 44 2024-09-18T10:10:21.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:21.505 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num 44 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 10 monclient: get_monmap_and_config 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 10 monclient: monmap: 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:21.602004+0000 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:21.602004+0000 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:21.603 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:21.604 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 1 Processor -- start 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 1 -- start start 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.598+0000 7f9457e30700 10 monclient: init 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94500a0290 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94500a0d70 0x7f945009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-c con 0x7f94500a0d70 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-b con 0x7f9450137a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945009e360 con 0x7f94500a0290 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945009ef80 con 0x7f94500a0d70 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94500a1fb0 con 0x7f9450137a20 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient(hunting): authenticate will time out at 903.696251s 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94563cd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 msgr2=0x7f9450139e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94563cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94500a0d70 0x7f945009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94500a0d70 0x7f945009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45854/0 (socket says 172.21.15.150:45854) 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 -- 172.21.15.150:0/1786805103 learned_addr learned my addr 172.21.15.150:0/1786805103 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): get_auth_request con 0x7f94500a0d70 auth_method 0 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient(hunting): handle_auth_done global_id 14221 payload 274 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 -- 172.21.15.150:0/1786805103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 msgr2=0x7f9450139e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 --2- 172.21.15.150:0/1786805103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 -- 172.21.15.150:0/1786805103 >> v1:172.21.15.142:6789/0 conn(0x7f94500a0290 legacy=0x7f94500a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 monclient: found mon.noname-c 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 -- 172.21.15.150:0/1786805103 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945009da50 con 0x7f94500a0d70 2024-09-18T10:10:21.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 -- 172.21.15.150:0/1786805103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0d70 msgr2=0x7f945009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f94553cb700 1 --2- 172.21.15.150:0/1786805103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0d70 0x7f945009ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94400099b0 tx=0x7f9440009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb2830b82a967a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 1 -- 172.21.15.150:0/1786805103 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944000baf0 con 0x7f94500a0d70 2024-09-18T10:10:21.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: dump: 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.611 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 5 monclient: authenticate success, global_id 14221 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 1 -- 172.21.15.150:0/1786805103 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944000bc50 con 0x7f94500a0d70 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9457e30700 4 set_mon_vals no callback set 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 1 -- 172.21.15.150:0/1786805103 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440011420 con 0x7f94500a0d70 2024-09-18T10:10:21.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.602+0000 7f9454bca700 10 monclient: dump: 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.613 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: shutdown 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- 172.21.15.150:0/1786805103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0d70 msgr2=0x7f945009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- 172.21.15.150:0/1786805103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0d70 0x7f945009ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f94400099b0 tx=0x7f9440009250 comp rx=0 tx=0).stop 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- 172.21.15.150:0/1786805103 shutdown_connections 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- 172.21.15.150:0/1786805103 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- 172.21.15.150:0/1786805103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0d70 0x7f945009ea40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- 172.21.15.150:0/1786805103 >> 172.21.15.150:0/1786805103 conn(0x7f94500a58a0 msgr2=0x7f94500a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- 172.21.15.150:0/1786805103 shutdown_connections 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- 172.21.15.150:0/1786805103 wait complete. 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: monmap: 2024-09-18T10:10:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 Processor -- start 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- start start 2024-09-18T10:10:21.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: init 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0290 0x7f9450098700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-b con 0x7f94500a0290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450098c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-c con 0x7f9450137a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f945012e790 0x7f945012c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): picked mon.noname-a con 0x7f945012e790 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94501300c0 con 0x7f945012e790 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945012ee80 con 0x7f94500a0290 2024-09-18T10:10:21.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945012fde0 con 0x7f9450137a20 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9457e30700 10 monclient(hunting): authenticate will time out at 903.702465s 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94553cb700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 msgr2=0x7f9450098c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0290 0x7f9450098700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94553cb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450098c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0290 0x7f9450098700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45856/0 (socket says 172.21.15.150:45856) 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 -- 172.21.15.150:0/4276516213 learned_addr learned my addr 172.21.15.150:0/4276516213 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): get_auth_request con 0x7f94500a0290 auth_method 0 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f945012e790 0x7f945012c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 10 monclient(hunting): get_auth_request con 0x7f945012e790 auth_method 0 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient(hunting): handle_auth_done global_id 14224 payload 824 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 -- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 msgr2=0x7f9450098c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9450137a20 0x7f9450098c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 -- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f945012e790 msgr2=0x7f945012c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f945012e790 0x7f945012c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 monclient: found mon.noname-b 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f9455bcc700 1 -- 172.21.15.150:0/4276516213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94400094f0 con 0x7f94500a0290 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.606+0000 7f94563cd700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f945012e790 0x7f945012c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9455bcc700 1 -- 172.21.15.150:0/4276516213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0290 msgr2=0x7f9450098700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9455bcc700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94500a0290 0x7f9450098700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c5d0 tx=0x7f944c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c0117a0 con 0x7f94500a0290 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: dump: 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.622 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c011900 con 0x7f94500a0290 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 5 monclient: authenticate success, global_id 14224 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _renew_subs 2024-09-18T10:10:21.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 1 -- 172.21.15.150:0/4276516213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94501303a0 con 0x7f94500a0290 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c011cc0 con 0x7f94500a0290 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 10 monclient: dump: 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.624 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9447fff700 4 set_mon_vals no callback set 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _renew_subs 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 1 -- 172.21.15.150:0/4276516213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9450099180 con 0x7f94500a0290 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9457e30700 1 -- 172.21.15.150:0/4276516213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9450130560 con 0x7f94500a0290 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.610+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f944c012630 con 0x7f94500a0290 2024-09-18T10:10:21.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.614+0000 7f9445ffb700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9434035a70 0x7f9434037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.614+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f944c0481c0 con 0x7f94500a0290 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.614+0000 7f94553cb700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9434035a70 0x7f9434037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.614+0000 7f94553cb700 10 monclient: get_auth_request con 0x7f9434035a70 auth_method 0 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.614+0000 7f94553cb700 1 --2- 172.21.15.150:0/4276516213 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9434035a70 0x7f9434037f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9440015000 tx=0x7f9440016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.618+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f944c0128e0 con 0x7f94500a0290 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.618+0000 7f9445ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.618+0000 7f9445ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:21.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9457e30700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T10:10:21.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9457e30700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T10:10:21.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9457e30700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9457e30700 1 -- 172.21.15.150:0/4276516213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"} v 0) v1 -- 0x7f945009d6c0 con 0x7f94500a0290 2024-09-18T10:10:21.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9445ffb700 1 -- 172.21.15.150:0/4276516213 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}]=-22 specified pg_num 44 < pg_num_max 33 v22) v1 ==== 149+0+0 (secure 0 0 0) 0x7f944c0102e0 con 0x7f94500a0290 2024-09-18T10:10:21.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9445ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num", "val": "44"}] 2024-09-18T10:10:21.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.778+0000 7f9445ffb700 10 monclient: _finish_command 2 = mon:22 specified pg_num 44 < pg_num_max 33 2024-09-18T10:10:21.785 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: specified pg_num 44 < pg_num_max 33 2024-09-18T10:10:21.788 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:21.788 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_min 0 2024-09-18T10:10:21.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 10 monclient: get_monmap_and_config 2024-09-18T10:10:21.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 10 monclient: monmap: 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:21.889780+0000 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:21.889780+0000 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:21.891 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:21.892 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 1 Processor -- start 2024-09-18T10:10:21.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 1 -- start start 2024-09-18T10:10:21.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.886+0000 7f7e402c9700 10 monclient: init 2024-09-18T10:10:21.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e380989e0 0x7f7e38099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-a con 0x7f7e380989e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7e3809b8f0 0x7f7e3809bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-c con 0x7f7e3809b8f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 0x7f7e3809cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-b con 0x7f7e3809c7c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e38096d40 con 0x7f7e380989e0 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e38097cd0 con 0x7f7e3809b8f0 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3809c230 con 0x7f7e3809c7c0 2024-09-18T10:10:21.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient(hunting): authenticate will time out at 903.984724s 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7e3809b8f0 0x7f7e3809bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e866700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 msgr2=0x7f7e3809cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7e3809b8f0 0x7f7e3809bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45868/0 (socket says 172.21.15.150:45868) 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 -- 172.21.15.150:0/2717840876 learned_addr learned my addr 172.21.15.150:0/2717840876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): get_auth_request con 0x7f7e3809b8f0 auth_method 0 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e866700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 0x7f7e3809cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 1 -- 172.21.15.150:0/2717840876 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:10:21.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e065700 1 --2- 172.21.15.150:0/2717840876 >> v2:172.21.15.142:3300/0 conn(0x7f7e380989e0 0x7f7e38099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e065700 10 monclient(hunting): get_auth_request con 0x7f7e380989e0 auth_method 0 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e065700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e065700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3e065700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient(hunting): handle_auth_done global_id 14227 payload 274 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 -- 172.21.15.150:0/2717840876 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 msgr2=0x7f7e3809cbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 0x7f7e3809cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 -- 172.21.15.150:0/2717840876 >> v2:172.21.15.142:3300/0 conn(0x7f7e380989e0 msgr2=0x7f7e38099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 --2- 172.21.15.150:0/2717840876 >> v2:172.21.15.142:3300/0 conn(0x7f7e380989e0 0x7f7e38099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 monclient: found mon.noname-c 2024-09-18T10:10:21.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 -- 172.21.15.150:0/2717840876 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e380979f0 con 0x7f7e3809b8f0 2024-09-18T10:10:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 -- 172.21.15.150:0/2717840876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 msgr2=0x7f7e3809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d864700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e3809bcf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7e380a2d20 tx=0x7f7e2800b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74df4b2d0c058222 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 1 -- 172.21.15.150:0/2717840876 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800dd20 con 0x7f7e3809b8f0 2024-09-18T10:10:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: dump: 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 5 monclient: authenticate success, global_id 14227 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 1 -- 172.21.15.150:0/2717840876 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2800de80 con 0x7f7e3809b8f0 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e402c9700 4 set_mon_vals no callback set 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 1 -- 172.21.15.150:0/2717840876 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28017670 con 0x7f7e3809b8f0 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.890+0000 7f7e3d063700 10 monclient: dump: 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.903 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: shutdown 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- 172.21.15.150:0/2717840876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 msgr2=0x7f7e3809bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e3809bcf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7e380a2d20 tx=0x7f7e2800b4e0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- 172.21.15.150:0/2717840876 shutdown_connections 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e3809c7c0 0x7f7e3809cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- 172.21.15.150:0/2717840876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e3809bcf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- 172.21.15.150:0/2717840876 >> v2:172.21.15.142:3300/0 conn(0x7f7e380989e0 0x7f7e38099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- 172.21.15.150:0/2717840876 >> 172.21.15.150:0/2717840876 conn(0x7f7e38068590 msgr2=0x7f7e380a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- 172.21.15.150:0/2717840876 shutdown_connections 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- 172.21.15.150:0/2717840876 wait complete. 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: build_initial_monmap 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: monmap: 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:21.905 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 Processor -- start 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- start start 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: init 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:10:21.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 0x7f7e3811c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-c con 0x7f7e380989e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-b con 0x7f7e3809b8f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e3809c7c0 0x7f7e3813d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): picked mon.noname-a con 0x7f7e3809c7c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3806f120 con 0x7f7e3809c7c0 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3809b190 con 0x7f7e3809b8f0 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3811e8f0 con 0x7f7e380989e0 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:21.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e402c9700 10 monclient(hunting): authenticate will time out at 903.991719s 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3e065700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 msgr2=0x7f7e3811c400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3e065700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 0x7f7e3811c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45878/0 (socket says 172.21.15.150:45878) 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 1 -- 172.21.15.150:0/3640255784 learned_addr learned my addr 172.21.15.150:0/3640255784 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 10 monclient(hunting): get_auth_request con 0x7f7e3809b8f0 auth_method 0 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.894+0000 7f7e3d864700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e3809c7c0 0x7f7e3813d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): get_auth_request con 0x7f7e3809c7c0 auth_method 0 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:21.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3e866700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient(hunting): handle_auth_done global_id 14230 payload 824 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 -- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 msgr2=0x7f7e3811c400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 0x7f7e3811c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 -- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e3809c7c0 msgr2=0x7f7e3813d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e3809c7c0 0x7f7e3813d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:21.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient: _finish_hunting 0 2024-09-18T10:10:21.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 monclient: found mon.noname-b 2024-09-18T10:10:21.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:21.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2800b780 con 0x7f7e3809b8f0 2024-09-18T10:10:21.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 -- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 msgr2=0x7f7e38139850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:21.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e3d864700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800bfd0 tx=0x7f7e2801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2800dd20 con 0x7f7e3809b8f0 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: dump: 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.913 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 5 monclient: authenticate success, global_id 14230 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e28017630 con 0x7f7e3809b8f0 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 10 monclient: _renew_subs 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3811ebd0 con 0x7f7e3809b8f0 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e28017c30 con 0x7f7e3809b8f0 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2ffff700 4 set_mon_vals no callback set 2024-09-18T10:10:21.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e2e7fc700 10 monclient: dump: 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 10 monclient: _renew_subs 2024-09-18T10:10:21.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.898+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3813ade0 con 0x7f7e3809b8f0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7e28015070 con 0x7f7e3809b8f0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e2e7fc700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 0x7f7e1c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4725+0+0 (secure 0 0 0) 0x7f7e2804b120 con 0x7f7e3809b8f0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e3e065700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 0x7f7e1c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e3e065700 10 monclient: get_auth_request con 0x7f7e1c035a70 auth_method 0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e402c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e402c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e402c9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.902+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3811ed60 con 0x7f7e3809b8f0 2024-09-18T10:10:21.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.906+0000 7f7e3e065700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 0x7f7e1c037f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006fd0 tx=0x7f7e34009530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:21.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.910+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7e28006c70 con 0x7f7e3809b8f0 2024-09-18T10:10:21.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.910+0000 7f7e2e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:21.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:21.910+0000 7f7e2e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:22.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.078+0000 7f7e402c9700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:22.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.078+0000 7f7e402c9700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:22.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.078+0000 7f7e402c9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:22.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.078+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f7e38097fb0 con 0x7f7e3809b8f0 2024-09-18T10:10:22.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.214+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v23) v1 ==== 143+0+0 (secure 0 0 0) 0x7f7e28027030 con 0x7f7e3809b8f0 2024-09-18T10:10:22.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.214+0000 7f7e2e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:22.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.214+0000 7f7e2e7fc700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_min to 0 2024-09-18T10:10:22.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.318+0000 7f7e402c9700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:22.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.318+0000 7f7e402c9700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:22.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.318+0000 7f7e402c9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:22.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:22.318+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"} v 0) v1 -- 0x7f7e38096ab0 con 0x7f7e3809b8f0 2024-09-18T10:10:23.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.230+0000 7f7e2e7fc700 1 -- 172.21.15.150:0/3640255784 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}]=0 set pool 3 pg_num_min to 0 v24) v1 ==== 143+0+0 (secure 0 0 0) 0x7f7e28027020 con 0x7f7e3809b8f0 2024-09-18T10:10:23.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.230+0000 7f7e2e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_min", "val": "0"}] 2024-09-18T10:10:23.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.230+0000 7f7e2e7fc700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_min to 0 2024-09-18T10:10:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 msgr2=0x7f7e1c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 0x7f7e1c037f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006fd0 tx=0x7f7e34009530 comp rx=0 tx=0).stop 2024-09-18T10:10:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 10 monclient: shutdown 2024-09-18T10:10:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 msgr2=0x7f7e38139850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800bfd0 tx=0x7f7e2801a000 comp rx=0 tx=0).stop 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 shutdown_connections 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e1c035a70 0x7f7e1c037f20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e3809c7c0 0x7f7e3813d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e3809b8f0 0x7f7e38139850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 --2- 172.21.15.150:0/3640255784 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e380989e0 0x7f7e3811c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 >> 172.21.15.150:0/3640255784 conn(0x7f7e38068590 msgr2=0x7f7e380a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 shutdown_connections 2024-09-18T10:10:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.234+0000 7f7e402c9700 1 -- 172.21.15.150:0/3640255784 wait complete. 2024-09-18T10:10:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 3 pg_num_min to 0 2024-09-18T10:10:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_min 2024-09-18T10:10:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool get testpool1234 pg_num_min 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: get_monmap_and_config 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: build_initial_monmap 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: monmap: 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.361 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:23.360517+0000 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:23.360517+0000 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:23.362 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 Processor -- start 2024-09-18T10:10:23.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 -- start start 2024-09-18T10:10:23.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: init 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe75c0a6090 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe75c09e460 0x7fe75c09d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-a con 0x7fe75c09e460 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-b con 0x7fe75c0a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c0a6460 con 0x7fe75c0a6090 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c09fd00 con 0x7fe75c09e460 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0a6850 con 0x7fe75c0a0a80 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe7622b9700 10 monclient(hunting): authenticate will time out at 905.453989s 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe760856700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 msgr2=0x7fe75c139b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe760856700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c139b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe75c09e460 0x7fe75c09d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe75c09e460 0x7fe75c09d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36584/0 (socket says 172.21.15.150:36584) 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 -- 172.21.15.150:0/2046562212 learned_addr learned my addr 172.21.15.150:0/2046562212 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): get_auth_request con 0x7fe75c09e460 auth_method 0 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 -- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 msgr2=0x7fe75c139b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 --2- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c139b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 -- 172.21.15.150:0/2046562212 >> v1:172.21.15.142:6789/0 conn(0x7fe75c0a6090 legacy=0x7fe75c0a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.358+0000 7fe75b7fe700 1 -- 172.21.15.150:0/2046562212 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c09d7a0 con 0x7fe75c09e460 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75b7fe700 1 -- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 msgr2=0x7fe75c09d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75b7fe700 1 --2- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c09d0f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7440099b0 tx=0x7fe744009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242505c1e4a656ce server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 1 -- 172.21.15.150:0/2046562212 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74400bb50 con 0x7fe75c09e460 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: dump: 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.375 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: _finish_auth 0 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 1 -- 172.21.15.150:0/2046562212 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74400bcb0 con 0x7fe75c09e460 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 5 monclient: authenticate success, global_id 4548 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:23.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 1 -- 172.21.15.150:0/2046562212 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744011670 con 0x7fe75c09e460 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 4 set_mon_vals no callback set 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe75affd700 10 monclient: dump: 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.377 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 10 monclient: shutdown 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 -- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 msgr2=0x7fe75c09d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 --2- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c09d0f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7440099b0 tx=0x7fe744009250 comp rx=0 tx=0).stop 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 -- 172.21.15.150:0/2046562212 shutdown_connections 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 --2- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 --2- 172.21.15.150:0/2046562212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c09d0f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 -- 172.21.15.150:0/2046562212 >> 172.21.15.150:0/2046562212 conn(0x7fe75c071830 msgr2=0x7fe75c0a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 -- 172.21.15.150:0/2046562212 shutdown_connections 2024-09-18T10:10:23.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 1 -- 172.21.15.150:0/2046562212 wait complete. 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 10 monclient: build_initial_monmap 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.362+0000 7fe7622b9700 10 monclient: monmap: 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 Processor -- start 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 -- start start 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient: init 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-a con 0x7fe75c09e460 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c1422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-c con 0x7fe75c0a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:23.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe75c0a6090 0x7fe75c08d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): picked mon.noname-b con 0x7fe75c0a6090 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c0a4dd0 con 0x7fe75c09e460 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c090c40 con 0x7fe75c0a6090 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c08ebc0 con 0x7fe75c0a0a80 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe7622b9700 10 monclient(hunting): authenticate will time out at 905.461957s 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 msgr2=0x7fe75c1422a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c1422a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe75c0a6090 0x7fe75c08d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe75c0a6090 0x7fe75c08d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45894/0 (socket says 172.21.15.150:45894) 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 -- 172.21.15.150:0/550529055 learned_addr learned my addr 172.21.15.150:0/550529055 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): get_auth_request con 0x7fe75c0a6090 auth_method 0 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): get_auth_request con 0x7fe75c09e460 auth_method 0 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient(hunting): handle_auth_done global_id 14233 payload 824 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 -- 172.21.15.150:0/550529055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 msgr2=0x7fe75c1422a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe75c0a0a80 0x7fe75c1422a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 -- 172.21.15.150:0/550529055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 msgr2=0x7fe75c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient: _finish_hunting 0 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 monclient: found mon.noname-b 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:23.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe760856700 1 -- 172.21.15.150:0/550529055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440094f0 con 0x7fe75c0a6090 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.366+0000 7fe75bfff700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe75c09e460 0x7fe75c141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe760856700 1 -- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe75c0a6090 msgr2=0x7fe75c08d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe760856700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe75c0a6090 0x7fe75c08d0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe75400bf80 tx=0x7fe754009730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75400ce50 con 0x7fe75c0a6090 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: dump: 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.384 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 5 monclient: authenticate success, global_id 14233 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe754016430 con 0x7fe75c0a6090 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _renew_subs 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 1 -- 172.21.15.150:0/550529055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c08f790 con 0x7fe75c0a6090 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7540165e0 con 0x7fe75c0a6090 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 10 monclient: dump: 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.386 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe75a7fc700 4 set_mon_vals no callback set 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _renew_subs 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 1 -- 172.21.15.150:0/550529055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c1427e0 con 0x7fe75c0a6090 2024-09-18T10:10:23.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe754012020 con 0x7fe75c0a6090 2024-09-18T10:10:23.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe7622b9700 1 -- 172.21.15.150:0/550529055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c09da80 con 0x7fe75c0a6090 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 0x7fe73c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7fe7540480a0 con 0x7fe75c0a6090 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.370+0000 7fe75bfff700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 0x7fe73c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.374+0000 7fe75bfff700 10 monclient: get_auth_request con 0x7fe73c035a70 auth_method 0 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.374+0000 7fe75bfff700 1 -- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 msgr2=0x7fe73c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.374+0000 7fe75bfff700 1 -- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 msgr2=0x7fe73c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.374+0000 7fe75bfff700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 0x7fe73c037f20 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fe74c002e00 tx=0x7fe74c007ea0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.374+0000 7fe75bfff700 1 --2- 172.21.15.150:0/550529055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe73c035a70 0x7fe73c037f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.382+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe7540153e0 con 0x7fe75c0a6090 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.382+0000 7fe753fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.382+0000 7fe753fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:23.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe7622b9700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:23.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe7622b9700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:23.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe7622b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe7622b9700 1 -- 172.21.15.150:0/550529055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"} v 0) v1 -- 0x7fe75c063ed0 con 0x7fe75c0a6090 2024-09-18T10:10:23.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe753fff700 1 -- 172.21.15.150:0/550529055 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}]=-2 option 'pg_num_min' is not set on pool 'testpool1234' v24) v1 ==== 158+0+0 (secure 0 0 0) 0x7fe75401c2b0 con 0x7fe75c0a6090 2024-09-18T10:10:23.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe753fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_min"}] 2024-09-18T10:10:23.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.542+0000 7fe753fff700 10 monclient: _finish_command 2 = mon:2 option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-18T10:10:23.548 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOENT: option 'pg_num_min' is not set on pool 'testpool1234' 2024-09-18T10:10:23.552 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:23.552 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set testpool1234 pg_num_max 0 2024-09-18T10:10:23.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.646+0000 7f6c739b7700 10 monclient: get_monmap_and_config 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.646+0000 7f6c739b7700 10 monclient: build_initial_monmap 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.646+0000 7f6c739b7700 10 monclient: monmap: 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:23.650774+0000 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:23.650774+0000 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:23.652 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:23.653 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:23.653 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 Processor -- start 2024-09-18T10:10:23.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 -- start start 2024-09-18T10:10:23.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient: init 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c6c0710a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c6c097180 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6c6c1355e0 0x7f6c6c1379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-a con 0x7f6c6c1355e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c071470 con 0x7f6c6c097180 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c09dad0 con 0x7f6c6c0710a0 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c099320 con 0x7f6c6c1355e0 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71753700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6c6c0710a0 0x7f6c6c096a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47788/0 (socket says 172.21.15.150:47788) 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71753700 1 -- 172.21.15.150:0/385022975 learned_addr learned my addr 172.21.15.150:0/385022975 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:23.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c739b7700 10 monclient(hunting): authenticate will time out at 905.745792s 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 1 --2- 172.21.15.150:0/385022975 >> v2:172.21.15.142:3300/0 conn(0x7f6c6c1355e0 0x7f6c6c1379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): get_auth_request con 0x7f6c6c1355e0 auth_method 0 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient(hunting): handle_auth_done global_id 4554 payload 274 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 1 -- 172.21.15.150:0/385022975 >> v1:172.21.15.150:6789/0 conn(0x7f6c6c0710a0 legacy=0x7f6c6c096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 1 -- 172.21.15.150:0/385022975 >> v1:172.21.15.142:6789/0 conn(0x7f6c6c097180 legacy=0x7f6c6c0975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient: _finish_hunting 0 2024-09-18T10:10:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 1 monclient: found mon.noname-a 2024-09-18T10:10:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.650+0000 7f6c71f54700 1 -- 172.21.15.150:0/385022975 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c097cb0 con 0x7f6c6c1355e0 2024-09-18T10:10:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c71f54700 1 -- 172.21.15.150:0/385022975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c1355e0 msgr2=0x7f6c6c1379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c71f54700 1 --2- 172.21.15.150:0/385022975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c1379c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009e60 tx=0x7f6c64009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45cbe8778511f74 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 1 -- 172.21.15.150:0/385022975 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6400e070 con 0x7f6c6c1355e0 2024-09-18T10:10:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: dump: 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 5 monclient: authenticate success, global_id 4554 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 1 -- 172.21.15.150:0/385022975 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64012430 con 0x7f6c6c1355e0 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 1 -- 172.21.15.150:0/385022975 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c640127d0 con 0x7f6c6c1355e0 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 4 set_mon_vals no callback set 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c63fff700 10 monclient: dump: 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 10 monclient: shutdown 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 -- 172.21.15.150:0/385022975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c1355e0 msgr2=0x7f6c6c1379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 --2- 172.21.15.150:0/385022975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c1379c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009e60 tx=0x7f6c64009780 comp rx=0 tx=0).stop 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 -- 172.21.15.150:0/385022975 shutdown_connections 2024-09-18T10:10:23.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 --2- 172.21.15.150:0/385022975 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c1379c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 -- 172.21.15.150:0/385022975 >> 172.21.15.150:0/385022975 conn(0x7f6c6c068590 msgr2=0x7f6c6c0a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:23.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 -- 172.21.15.150:0/385022975 shutdown_connections 2024-09-18T10:10:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 1 -- 172.21.15.150:0/385022975 wait complete. 2024-09-18T10:10:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.654+0000 7f6c739b7700 10 monclient: build_initial_monmap 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient: monmap: 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 Processor -- start 2024-09-18T10:10:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 -- start start 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient: init 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 0x7f6c6c1349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-c con 0x7f6c6c0710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 0x7f6c6c134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-a con 0x7f6c6c097180 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): picked mon.noname-b con 0x7f6c6c1355e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0a04d0 con 0x7f6c6c097180 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c132eb0 con 0x7f6c6c1355e0 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c097f50 con 0x7f6c6c0710a0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c739b7700 10 monclient(hunting): authenticate will time out at 905.753957s 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71753700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 msgr2=0x7f6c6c1349a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71753700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 0x7f6c6c1349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45898/0 (socket says 172.21.15.150:45898) 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 -- 172.21.15.150:0/4291843684 learned_addr learned my addr 172.21.15.150:0/4291843684 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): get_auth_request con 0x7f6c6c1355e0 auth_method 0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 0x7f6c6c134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): get_auth_request con 0x7f6c6c097180 auth_method 0 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient(hunting): handle_auth_done global_id 14236 payload 824 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 -- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 msgr2=0x7f6c6c1349a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 0x7f6c6c1349a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 -- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 msgr2=0x7f6c6c134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 0x7f6c6c134ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient: _finish_hunting 0 2024-09-18T10:10:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 monclient: found mon.noname-b 2024-09-18T10:10:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c71f54700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64009a20 con 0x7f6c6c1355e0 2024-09-18T10:10:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.658+0000 7f6c70f52700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 0x7f6c6c134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c71f54700 1 -- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 msgr2=0x7f6c6c13f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c71f54700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400b6c0 tx=0x7f6c64018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64026430 con 0x7f6c6c1355e0 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: dump: 2024-09-18T10:10:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64021070 con 0x7f6c6c1355e0 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 5 monclient: authenticate success, global_id 14236 2024-09-18T10:10:23.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _renew_subs 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c1310a0 con 0x7f6c6c1355e0 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64026820 con 0x7f6c6c1355e0 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 10 monclient: dump: 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c62ffd700 4 set_mon_vals no callback set 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _renew_subs 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c13f8a0 con 0x7f6c6c1355e0 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6c64023030 con 0x7f6c6c1355e0 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 0x7f6c50037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4709+0+0 (secure 0 0 0) 0x7f6c6404ab90 con 0x7f6c6c1355e0 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c71753700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 0x7f6c50037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c098370 con 0x7f6c6c1355e0 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.662+0000 7f6c71753700 10 monclient: get_auth_request con 0x7f6c50035a70 auth_method 0 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.666+0000 7f6c71753700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 0x7f6c50037f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c68004860 tx=0x7f6c68007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.670+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6c6401c420 con 0x7f6c6c1355e0 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.670+0000 7f6c617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.670+0000 7f6c617fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:23.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.838+0000 7f6c739b7700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:23.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.838+0000 7f6c739b7700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:23.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.838+0000 7f6c739b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:23.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:23.838+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f6c6c131c90 con 0x7f6c6c1355e0 2024-09-18T10:10:24.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.238+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v25) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6c64011390 con 0x7f6c6c1355e0 2024-09-18T10:10:24.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.238+0000 7f6c617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:24.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.238+0000 7f6c617fa700 10 monclient: _finish_command 2 = system:0 set pool 3 pg_num_max to 0 2024-09-18T10:10:24.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.342+0000 7f6c739b7700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:24.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.342+0000 7f6c739b7700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:24.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.342+0000 7f6c739b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:24.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:24.342+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"} v 0) v1 -- 0x7f6c6c131c90 con 0x7f6c6c1355e0 2024-09-18T10:10:25.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.470+0000 7f6c617fa700 1 -- 172.21.15.150:0/4291843684 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}]=0 set pool 3 pg_num_max to 0 v26) v1 ==== 143+0+0 (secure 0 0 0) 0x7f6c64011570 con 0x7f6c6c1355e0 2024-09-18T10:10:25.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.470+0000 7f6c617fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "testpool1234", "var": "pg_num_max", "val": "0"}] 2024-09-18T10:10:25.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.470+0000 7f6c617fa700 10 monclient: _finish_command 3 = system:0 set pool 3 pg_num_max to 0 2024-09-18T10:10:25.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 msgr2=0x7f6c50037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 0x7f6c50037f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c68004860 tx=0x7f6c68007c10 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 10 monclient: shutdown 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 msgr2=0x7f6c6c13f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400b6c0 tx=0x7f6c64018000 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 shutdown_connections 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6c50035a70 0x7f6c50037f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c6c1355e0 0x7f6c6c13f360 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c6c097180 0x7f6c6c134ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 --2- 172.21.15.150:0/4291843684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c6c0710a0 0x7f6c6c1349a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 >> 172.21.15.150:0/4291843684 conn(0x7f6c6c068590 msgr2=0x7f6c6c09e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 shutdown_connections 2024-09-18T10:10:25.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.478+0000 7f6c739b7700 1 -- 172.21.15.150:0/4291843684 wait complete. 2024-09-18T10:10:25.483 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 3 pg_num_max to 0 2024-09-18T10:10:25.494 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool get testpool1234 pg_num_max 2024-09-18T10:10:25.494 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:25.494 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool get testpool1234 pg_num_max 2024-09-18T10:10:25.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: get_monmap_and_config 2024-09-18T10:10:25.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: build_initial_monmap 2024-09-18T10:10:25.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: monmap: 2024-09-18T10:10:25.593 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:25.593 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:25.592408+0000 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:25.592408+0000 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:25.594 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 Processor -- start 2024-09-18T10:10:25.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 -- start start 2024-09-18T10:10:25.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: init 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5fa0132420 0x7f5fa006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-a con 0x7f5fa0132420 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5fa01327f0 0x7f5fa0132c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-c con 0x7f5fa01327f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 0x7f5fa0067f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-b con 0x7f5fa0067b10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa006c040 con 0x7f5fa0132420 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa006e620 con 0x7f5fa01327f0 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa006d390 con 0x7f5fa0067b10 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa6416700 10 monclient(hunting): authenticate will time out at 907.686018s 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa49b3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 msgr2=0x7f5fa0067f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5fa49b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 0x7f5fa0067f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5fa01327f0 0x7f5fa0132c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5fa01327f0 0x7f5fa0132c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45914/0 (socket says 172.21.15.150:45914) 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1504239010 learned_addr learned my addr 172.21.15.150:0/1504239010 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): get_auth_request con 0x7f5fa01327f0 auth_method 0 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 1 --2- 172.21.15.150:0/1504239010 >> v2:172.21.15.142:3300/0 conn(0x7f5fa0132420 0x7f5fa006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): get_auth_request con 0x7f5fa0132420 auth_method 0 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_done global_id 14239 payload 274 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1504239010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 msgr2=0x7f5fa0067f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1504239010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 0x7f5fa0067f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1504239010 >> v2:172.21.15.142:3300/0 conn(0x7f5fa0132420 msgr2=0x7f5fa006ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1504239010 >> v2:172.21.15.142:3300/0 conn(0x7f5fa0132420 0x7f5fa006ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.590+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1504239010 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa006d670 con 0x7f5fa01327f0 2024-09-18T10:10:25.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1504239010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa01327f0 msgr2=0x7f5fa0132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:25.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1504239010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa01327f0 0x7f5fa0132c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00a3830 tx=0x7f5f90009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53756bf1362f2484 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 1 -- 172.21.15.150:0/1504239010 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90009e70 con 0x7f5fa01327f0 2024-09-18T10:10:25.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: dump: 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: _finish_auth 0 2024-09-18T10:10:25.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 5 monclient: authenticate success, global_id 14239 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 1 -- 172.21.15.150:0/1504239010 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9000ba70 con 0x7f5fa01327f0 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 4 set_mon_vals no callback set 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 1 -- 172.21.15.150:0/1504239010 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000be30 con 0x7f5fa01327f0 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5f9effd700 10 monclient: dump: 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.611 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 10 monclient: shutdown 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 -- 172.21.15.150:0/1504239010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa01327f0 msgr2=0x7f5fa0132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 --2- 172.21.15.150:0/1504239010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa01327f0 0x7f5fa0132c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00a3830 tx=0x7f5f90009250 comp rx=0 tx=0).stop 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 -- 172.21.15.150:0/1504239010 shutdown_connections 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 --2- 172.21.15.150:0/1504239010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa0067b10 0x7f5fa0067f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 --2- 172.21.15.150:0/1504239010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa01327f0 0x7f5fa0132c70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 --2- 172.21.15.150:0/1504239010 >> v2:172.21.15.142:3300/0 conn(0x7f5fa0132420 0x7f5fa006ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 -- 172.21.15.150:0/1504239010 >> 172.21.15.150:0/1504239010 conn(0x7f5fa00a31e0 msgr2=0x7f5fa009f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 -- 172.21.15.150:0/1504239010 shutdown_connections 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 -- 172.21.15.150:0/1504239010 wait complete. 2024-09-18T10:10:25.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 10 monclient: build_initial_monmap 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 10 monclient: monmap: 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.594+0000 7f5fa6416700 1 Processor -- start 2024-09-18T10:10:25.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 -- start start 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient: init 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0067b10 0x7f5fa0141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-a con 0x7f5fa0067b10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa0132420 0x7f5fa01312f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-b con 0x7f5fa0132420 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa01327f0 0x7f5fa011adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): picked mon.noname-c con 0x7f5fa01327f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:25.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa011e410 con 0x7f5fa0067b10 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0142530 con 0x7f5fa0132420 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa011ce20 con 0x7f5fa01327f0 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa6416700 10 monclient(hunting): authenticate will time out at 907.693378s 2024-09-18T10:10:25.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa49b3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa01327f0 msgr2=0x7f5fa011adf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa0132420 0x7f5fa01312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5fa49b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa01327f0 0x7f5fa011adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa0132420 0x7f5fa01312f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45928/0 (socket says 172.21.15.150:45928) 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1020716112 learned_addr learned my addr 172.21.15.150:0/1020716112 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0067b10 0x7f5fa0141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): get_auth_request con 0x7f5fa0132420 auth_method 0 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): get_auth_request con 0x7f5fa0067b10 auth_method 0 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient(hunting): handle_auth_done global_id 14242 payload 824 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1020716112 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa01327f0 msgr2=0x7f5fa011adf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa01327f0 0x7f5fa011adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1020716112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0067b10 msgr2=0x7f5fa0141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0067b10 0x7f5fa0141650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.598+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1020716112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900094f0 con 0x7f5fa0132420 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f9f7fe700 1 -- 172.21.15.150:0/1020716112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa0132420 msgr2=0x7f5fa01312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f9f7fe700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fa0132420 0x7f5fa01312f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5f90006fd0 tx=0x7f5f90017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9001f070 con 0x7f5fa0132420 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: dump: 2024-09-18T10:10:25.619 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 5 monclient: authenticate success, global_id 14242 2024-09-18T10:10:25.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90009e70 con 0x7f5fa0132420 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000bbe0 con 0x7f5fa0132420 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _renew_subs 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 1 -- 172.21.15.150:0/1020716112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa006c6e0 con 0x7f5fa0132420 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f9dffb700 4 set_mon_vals no callback set 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 10 monclient: dump: 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.621 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _renew_subs 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 1 -- 172.21.15.150:0/1020716112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa011d4b0 con 0x7f5fa0132420 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5fa6416700 1 -- 172.21.15.150:0/1020716112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa0063ed0 con 0x7f5fa0132420 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5f90023070 con 0x7f5fa0132420 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f80035ac0 0x7f5f80037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7f5f90049590 con 0x7f5fa0132420 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f9ffff700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f80035ac0 0x7f5f80037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.602+0000 7f5f9ffff700 10 monclient: get_auth_request con 0x7f5f80035ac0 auth_method 0 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.606+0000 7f5f9ffff700 1 --2- 172.21.15.150:0/1020716112 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f80035ac0 0x7f5f80037f70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009d90 tx=0x7f5f880091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.610+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5f9002d590 con 0x7f5fa0132420 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.610+0000 7f5f97fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.610+0000 7f5f97fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:25.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.766+0000 7f5fa6416700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.766+0000 7f5fa6416700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.770+0000 7f5fa6416700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.770+0000 7f5fa6416700 1 -- 172.21.15.150:0/1020716112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"} v 0) v1 -- 0x7f5fa00008d0 con 0x7f5fa0132420 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.770+0000 7f5f97fff700 1 -- 172.21.15.150:0/1020716112 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}]=-2 option 'pg_num_max' is not set on pool 'testpool1234' v26) v1 ==== 158+0+0 (secure 0 0 0) 0x7f5f9002a020 con 0x7f5fa0132420 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.770+0000 7f5f97fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "testpool1234", "var": "pg_num_max"}] 2024-09-18T10:10:25.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.770+0000 7f5f97fff700 10 monclient: _finish_command 2 = mon:2 option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-18T10:10:25.775 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOENT: option 'pg_num_max' is not set on pool 'testpool1234' 2024-09-18T10:10:25.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:25.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete testpool1234 testpool1234 --yes-i-really-really-mean-it 2024-09-18T10:10:25.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:25.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: build_initial_monmap 2024-09-18T10:10:25.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: monmap: 2024-09-18T10:10:25.873 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:25.872620+0000 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:25.872620+0000 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:25.874 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:25.875 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 Processor -- start 2024-09-18T10:10:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 -- start start 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: init 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 0x7fb0ac0a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac0a05e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0ac09e800 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac09d230 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0a1430 con 0x7fb0ac09e800 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ac068b30 con 0x7fb0ac09d230 2024-09-18T10:10:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac09e3b0 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0b159e700 10 monclient(hunting): authenticate will time out at 907.966548s 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0aaffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 msgr2=0x7fb0ac0a09b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 0x7fb0ac0a09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36634/0 (socket says 172.21.15.150:36634) 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/396719830 learned_addr learned my addr 172.21.15.150:0/396719830 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb0ac09d230 auth_method 0 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_done global_id 4566 payload 274 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 msgr2=0x7fb0ac0a09b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 --2- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 0x7fb0ac0a09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/396719830 >> v1:172.21.15.142:6789/0 conn(0x7fb0ac09e800 legacy=0x7fb0ac136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.870+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/396719830 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac09e060 con 0x7fb0ac09d230 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 msgr2=0x7fb0ac0a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0ab7fe700 1 --2- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009f20 tx=0x7fb0a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54c01364c9a0469 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 1 -- 172.21.15.150:0/396719830 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400e070 con 0x7fb0ac09d230 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: dump: 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 1 -- 172.21.15.150:0/396719830 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a400bc90 con 0x7fb0ac09d230 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 1 -- 172.21.15.150:0/396719830 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4012660 con 0x7fb0ac09d230 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 5 monclient: authenticate success, global_id 4566 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 4 set_mon_vals no callback set 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0a9ffb700 10 monclient: dump: 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 10 monclient: shutdown 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 -- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 msgr2=0x7fb0ac0a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 --2- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009f20 tx=0x7fb0a4009840 comp rx=0 tx=0).stop 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 -- 172.21.15.150:0/396719830 shutdown_connections 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 --2- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac0a0ef0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 --2- 172.21.15.150:0/396719830 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac0a05e0 0x7fb0ac0a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 -- 172.21.15.150:0/396719830 >> 172.21.15.150:0/396719830 conn(0x7fb0ac0a5a40 msgr2=0x7fb0ac0a5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 -- 172.21.15.150:0/396719830 shutdown_connections 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.874+0000 7fb0b159e700 1 -- 172.21.15.150:0/396719830 wait complete. 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient: build_initial_monmap 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient: monmap: 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 Processor -- start 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 -- start start 2024-09-18T10:10:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient: init 2024-09-18T10:10:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac09d230 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 0x7fb0ac142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-c con 0x7fb0ac09e800 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac0a05e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac08fb40 con 0x7fb0ac09d230 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0900c0 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac0a5050 con 0x7fb0ac09e800 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0b159e700 10 monclient(hunting): authenticate will time out at 907.975324s 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aa7fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 msgr2=0x7fb0ac142020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aa7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 0x7fb0ac142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45942/0 (socket says 172.21.15.150:45942) 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/3080945144 learned_addr learned my addr 172.21.15.150:0/3080945144 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb0ac0a05e0 auth_method 0 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36644/0 (socket says 172.21.15.150:36644) 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): get_auth_request con 0x7fb0ac09d230 auth_method 0 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.878+0000 7fb0aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 10 monclient(hunting): handle_auth_done global_id 14245 payload 824 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 msgr2=0x7fb0ac142020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 0x7fb0ac142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 msgr2=0x7fb0ac141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4009ae0 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 -- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 msgr2=0x7fb0ac142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0ab7fe700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400b6c0 tx=0x7fb0a4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4012c60 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:25.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: dump: 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:25.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 5 monclient: authenticate success, global_id 14245 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4012dc0 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 10 monclient: _renew_subs 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a401e680 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0a97fa700 4 set_mon_vals no callback set 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac090710 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb092ffd700 10 monclient: dump: 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:25.897 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 10 monclient: _renew_subs 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.882+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac091950 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb0a401e9f0 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb092ffd700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 0x7fb098037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0b159e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0b159e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0b159e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac0a5590 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4693+0+0 (secure 0 0 0) 0x7fb0a4049e00 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0aaffd700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 0x7fb098037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0aaffd700 10 monclient: get_auth_request con 0x7fb098035a70 auth_method 0 2024-09-18T10:10:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.886+0000 7fb0aaffd700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 0x7fb098037f20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0003980 tx=0x7fb0a00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:25.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.894+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb0a4007550 con 0x7fb0ac0a05e0 2024-09-18T10:10:25.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.894+0000 7fb092ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:25.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:25.894+0000 7fb092ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:26.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.058+0000 7fb0b159e700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.058+0000 7fb0b159e700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.058+0000 7fb0b159e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.058+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0ac063ed0 con 0x7fb0ac0a05e0 2024-09-18T10:10:26.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.474+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' removed v27) v1 ==== 176+0+0 (secure 0 0 0) 0x7fb0a4027bb0 con 0x7fb0ac0a05e0 2024-09-18T10:10:26.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.474+0000 7fb092ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.474+0000 7fb092ffd700 10 monclient: _finish_command 2 = system:0 pool 'testpool1234' removed 2024-09-18T10:10:26.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.578+0000 7fb0b159e700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.578+0000 7fb0b159e700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.578+0000 7fb0b159e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.578+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb0ac063ed0 con 0x7fb0ac0a05e0 2024-09-18T10:10:26.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb092ffd700 1 -- 172.21.15.150:0/3080945144 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}]=0 pool 'testpool1234' does not exist v27) v1 ==== 183+0+0 (secure 0 0 0) 0x7fb0a4027d90 con 0x7fb0ac0a05e0 2024-09-18T10:10:26.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb092ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "testpool1234", "pool2": "testpool1234", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:26.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb092ffd700 10 monclient: _finish_command 3 = system:0 pool 'testpool1234' does not exist 2024-09-18T10:10:26.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 msgr2=0x7fb098037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:26.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 0x7fb098037f20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0003980 tx=0x7fb0a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 10 monclient: shutdown 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 msgr2=0x7fb0ac142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400b6c0 tx=0x7fb0a4017000 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 shutdown_connections 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb098035a70 0x7fb098037f20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0ac0a05e0 0x7fb0ac142560 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0ac09e800 0x7fb0ac142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 --2- 172.21.15.150:0/3080945144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0ac09d230 0x7fb0ac141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 >> 172.21.15.150:0/3080945144 conn(0x7fb0ac0a5a40 msgr2=0x7fb0ac09a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:26.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 shutdown_connections 2024-09-18T10:10:26.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.582+0000 7fb0b159e700 1 -- 172.21.15.150:0/3080945144 wait complete. 2024-09-18T10:10:26.589 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'testpool1234' does not exist 2024-09-18T10:10:26.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-18T10:10:26.597 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:26.598 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create foo 123 123 invalid foo-profile foo-rule 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 10 monclient: get_monmap_and_config 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 10 monclient: build_initial_monmap 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 10 monclient: monmap: 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:26.702140+0000 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:26.702140+0000 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:26.703 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 1 Processor -- start 2024-09-18T10:10:26.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 1 -- start start 2024-09-18T10:10:26.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.698+0000 7f8213f20700 10 monclient: init 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f820c071240 0x7f820c096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-a con 0x7f820c071240 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f820c09e1e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f820c0994f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:26.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c098cf0 con 0x7f820c09e1e0 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820c09df80 con 0x7f820c0994f0 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c09e640 con 0x7f820c071240 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8213f20700 10 monclient(hunting): authenticate will time out at 908.796929s 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f82124bd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f820c0994f0 0x7f820c1364d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47792/0 (socket says 172.21.15.150:47792) 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f82124bd700 1 -- 172.21.15.150:0/3205168605 learned_addr learned my addr 172.21.15.150:0/3205168605 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 1 --2- 172.21.15.150:0/3205168605 >> v2:172.21.15.142:3300/0 conn(0x7f820c071240 0x7f820c096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): get_auth_request con 0x7f820c071240 auth_method 0 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:26.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 1 -- 172.21.15.150:0/3205168605 >> v1:172.21.15.150:6789/0 conn(0x7f820c0994f0 legacy=0x7f820c1364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 1 -- 172.21.15.150:0/3205168605 >> v1:172.21.15.142:6789/0 conn(0x7f820c09e1e0 legacy=0x7f820c096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 1 monclient: found mon.noname-a 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:26.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.702+0000 7f8211cbc700 1 -- 172.21.15.150:0/3205168605 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c098a10 con 0x7f820c071240 2024-09-18T10:10:26.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8211cbc700 1 -- 172.21.15.150:0/3205168605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c071240 msgr2=0x7f820c096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:26.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8211cbc700 1 --2- 172.21.15.150:0/3205168605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c071240 0x7f820c096a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82080098e0 tx=0x7f82080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b0ff69abdfdc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:26.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 1 -- 172.21.15.150:0/3205168605 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820800ba70 con 0x7f820c071240 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: dump: 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:26.712 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: _finish_auth 0 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 5 monclient: authenticate success, global_id 4572 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 1 -- 172.21.15.150:0/3205168605 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820800bbd0 con 0x7f820c071240 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 4 set_mon_vals no callback set 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 1 -- 172.21.15.150:0/3205168605 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208011420 con 0x7f820c071240 2024-09-18T10:10:26.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8210cba700 10 monclient: dump: 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:26.714 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 10 monclient: shutdown 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 -- 172.21.15.150:0/3205168605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c071240 msgr2=0x7f820c096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 --2- 172.21.15.150:0/3205168605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c071240 0x7f820c096a70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f82080098e0 tx=0x7f82080091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 -- 172.21.15.150:0/3205168605 shutdown_connections 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 --2- 172.21.15.150:0/3205168605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c071240 0x7f820c096a70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 -- 172.21.15.150:0/3205168605 >> 172.21.15.150:0/3205168605 conn(0x7f820c0a1cc0 msgr2=0x7f820c0a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 -- 172.21.15.150:0/3205168605 shutdown_connections 2024-09-18T10:10:26.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 1 -- 172.21.15.150:0/3205168605 wait complete. 2024-09-18T10:10:26.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 10 monclient: build_initial_monmap 2024-09-18T10:10:26.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.706+0000 7f8213f20700 10 monclient: monmap: 2024-09-18T10:10:26.720 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:26.720 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:26.720 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 Processor -- start 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 -- start start 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient: init 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:26.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c071240 0x7f820c09c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-b con 0x7f820c071240 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0994f0 0x7f820c11e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-c con 0x7f820c0994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c09e1e0 0x7f820c11e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): picked mon.noname-a con 0x7f820c09e1e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c136dc0 con 0x7f820c09e1e0 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c137710 con 0x7f820c071240 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c0980f0 con 0x7f820c0994f0 2024-09-18T10:10:26.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8213f20700 10 monclient(hunting): authenticate will time out at 908.805819s 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c071240 0x7f820c09c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82114bb700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0994f0 msgr2=0x7f820c11e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82114bb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0994f0 0x7f820c11e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c071240 0x7f820c09c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45958/0 (socket says 172.21.15.150:45958) 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 -- 172.21.15.150:0/4159242587 learned_addr learned my addr 172.21.15.150:0/4159242587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): get_auth_request con 0x7f820c071240 auth_method 0 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c09e1e0 0x7f820c11e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:26.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): get_auth_request con 0x7f820c09e1e0 auth_method 0 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f82124bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient(hunting): handle_auth_done global_id 14248 payload 824 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 -- 172.21.15.150:0/4159242587 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0994f0 msgr2=0x7f820c11e190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0994f0 0x7f820c11e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 -- 172.21.15.150:0/4159242587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c09e1e0 msgr2=0x7f820c11e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:26.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c09e1e0 0x7f820c11e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 monclient: found mon.noname-b 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.710+0000 7f8211cbc700 1 -- 172.21.15.150:0/4159242587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8208009420 con 0x7f820c071240 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8211cbc700 1 -- 172.21.15.150:0/4159242587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c071240 msgr2=0x7f820c09c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8211cbc700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c071240 0x7f820c09c6b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82080030c0 tx=0x7f8208016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820801b680 con 0x7f820c071240 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: dump: 2024-09-18T10:10:26.725 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820801b7e0 con 0x7f820c071240 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:26.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820801bba0 con 0x7f820c071240 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 5 monclient: authenticate success, global_id 14248 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _renew_subs 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fffff700 4 set_mon_vals no callback set 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 1 -- 172.21.15.150:0/4159242587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c09b590 con 0x7f820c071240 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 10 monclient: dump: 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:26.727 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _renew_subs 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 1 -- 172.21.15.150:0/4159242587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c09d150 con 0x7f820c071240 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f8213f20700 1 -- 172.21.15.150:0/4159242587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c120b00 con 0x7f820c071240 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f82080253f0 con 0x7f820c071240 2024-09-18T10:10:26.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f0035ac0 0x7f81f0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:26.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8208049e60 con 0x7f820c071240 2024-09-18T10:10:26.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f82114bb700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f0035ac0 0x7f81f0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:26.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.714+0000 7f82114bb700 10 monclient: get_auth_request con 0x7f81f0035ac0 auth_method 0 2024-09-18T10:10:26.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.718+0000 7f82114bb700 1 --2- 172.21.15.150:0/4159242587 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f0035ac0 0x7f81f0037f70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8200007fd0 tx=0x7f8200007a70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:26.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.726+0000 7f81fdffb700 1 -- 172.21.15.150:0/4159242587 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f820804d030 con 0x7f820c071240 2024-09-18T10:10:26.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.726+0000 7f81fdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:26.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.726+0000 7f81fdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:26.897 INFO:tasks.workunit.client.0.smithi150.stderr:foo-rule not valid: foo-rule doesn't represent a float 2024-09-18T10:10:26.897 INFO:tasks.workunit.client.0.smithi150.stderr:Invalid command: unused arguments: ['foo-rule'] 2024-09-18T10:10:26.897 INFO:tasks.workunit.client.0.smithi150.stderr:osd pool create [] [] [] [] [] [] [] [] [] [] [--bulk] [] [] : create pool 2024-09-18T10:10:26.897 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: invalid command 2024-09-18T10:10:26.899 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:26.900 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create foo 123 123 replicated 2024-09-18T10:10:26.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.990+0000 7f6585ec4700 10 monclient: get_monmap_and_config 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.990+0000 7f6585ec4700 10 monclient: build_initial_monmap 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.990+0000 7f6585ec4700 10 monclient: monmap: 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:26.995930+0000 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:26.995930+0000 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:26.998 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:26.998 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:26.998 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 Processor -- start 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 -- start start 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient: init 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f658006b5a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:26.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f65801374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-b con 0x7f65801328d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6580132ce0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658006aac0 con 0x7f658006b5a0 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658006b970 con 0x7f6580132ce0 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658009fb10 con 0x7f65801328d0 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f6585ec4700 10 monclient(hunting): authenticate will time out at 909.089441s 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f657effd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 msgr2=0x7f65801374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f657ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6580132ce0 0x7f6580139a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47798/0 (socket says 172.21.15.150:47798) 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 learned_addr learned my addr 172.21.15.150:0/2088734665 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:27.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f657effd700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f65801374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.994+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 >> v1:172.21.15.150:6789/0 conn(0x7f6580132ce0 legacy=0x7f6580139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 1 -- 172.21.15.150:0/2088734665 >> v1:172.21.15.142:6789/0 conn(0x7f658006b5a0 legacy=0x7f65801321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338678896 0 0) 0x7f658006b970 con 0x7f6580132ce0 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): my global_id is 14251 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568003570 con 0x7f6580132ce0 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622944314 0 0) 0x7f658006aac0 con 0x7f658006b5a0 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): my global_id is 4578 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658006b970 con 0x7f658006b5a0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123619145 0 0) 0x7f6568003570 con 0x7f6580132ce0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 msgr2=0x7f65801374f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f65801374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 >> v1:172.21.15.142:6789/0 conn(0x7f658006b5a0 legacy=0x7f65801321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658006a7e0 con 0x7f6580132ce0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:27.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f6578001be0 con 0x7f6580132ce0 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f6585ec4700 5 monclient: authenticate success, global_id 14251 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f6585ec4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: dump: 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.006 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 >> v1:172.21.15.150:6789/0 conn(0x7f6580132ce0 legacy=0x7f6580139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6568003bf0 0x7f6568003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): picked mon.a con 0x7f6568003bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f6568004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): picked mon.c con 0x7f65801328d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f65680052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): picked mon.b con 0x7f6568004e70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658009fb10 con 0x7f6568003bf0 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656800bdb0 con 0x7f6568004e70 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656800bf70 con 0x7f65801328d0 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657effd700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 msgr2=0x7f6568004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f65680052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657effd700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f6568004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:27.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): get_auth_request con 0x7f6568004e70 auth_method 0 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f6585ec4700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6568003bf0 0x7f6568003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 10 monclient(hunting): get_auth_request con 0x7f6568003bf0 auth_method 0 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient(hunting): handle_auth_done global_id 14251 payload 293 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 msgr2=0x7f6568004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801328d0 0x7f6568004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6568003bf0 msgr2=0x7f6568003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6568003bf0 0x7f6568003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 monclient: found mon.b 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568003570 con 0x7f6568004e70 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 10 monclient: _finish_auth 0 2024-09-18T10:10:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 msgr2=0x7f65680052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:26.998+0000 7f657ffff700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f65680052e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6578005a20 tx=0x7f6578005130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efdab716c0ed79d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578005b20 con 0x7f6568004e70 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: dump: 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657800d870 con 0x7f6568004e70 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 1 -- 172.21.15.150:0/2088734665 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657800dc10 con 0x7f6568004e70 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 4 set_mon_vals no callback set 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657e7fc700 10 monclient: dump: 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: shutdown 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 msgr2=0x7f65680052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 --2- 172.21.15.150:0/2088734665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f65680052e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6578005a20 tx=0x7f6578005130 comp rx=0 tx=0).stop 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657ffff700 1 -- 172.21.15.150:0/2088734665 reap_dead start 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- 172.21.15.150:0/2088734665 shutdown_connections 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- 172.21.15.150:0/2088734665 >> 172.21.15.150:0/2088734665 conn(0x7f6580071b60 msgr2=0x7f65800a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- 172.21.15.150:0/2088734665 shutdown_connections 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- 172.21.15.150:0/2088734665 wait complete. 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: build_initial_monmap 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: monmap: 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 Processor -- start 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- start start 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: init 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-b con 0x7f6568004e70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f658011d430 0x7f6580131270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-a con 0x7f658011d430 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 0x7f658011aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): picked mon.noname-c con 0x7f65801317b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658011c8f0 con 0x7f658011d430 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65800a3040 con 0x7f6568004e70 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658011e670 con 0x7f65801317b0 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f6585ec4700 10 monclient(hunting): authenticate will time out at 909.099844s 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.002+0000 7f657ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 msgr2=0x7f658011aee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 0x7f658011aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45974/0 (socket says 172.21.15.150:45974) 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 -- 172.21.15.150:0/641331245 learned_addr learned my addr 172.21.15.150:0/641331245 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): get_auth_request con 0x7f6568004e70 auth_method 0 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f658011d430 0x7f6580131270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): get_auth_request con 0x7f658011d430 auth_method 0 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient(hunting): handle_auth_done global_id 14254 payload 824 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 -- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 msgr2=0x7f658011aee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 0x7f658011aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 -- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f658011d430 msgr2=0x7f6580131270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f658011d430 0x7f6580131270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658011e390 con 0x7f6568004e70 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 -- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 msgr2=0x7f6580141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657f7fe700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f657400e7b0 tx=0x7f6574009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65740074f0 con 0x7f6568004e70 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: dump: 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574010040 con 0x7f6568004e70 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 5 monclient: authenticate success, global_id 14254 2024-09-18T10:10:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574005a60 con 0x7f6568004e70 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f65677fe700 10 monclient: dump: 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.026 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 10 monclient: _renew_subs 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f657dffb700 4 set_mon_vals no callback set 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6580142490 con 0x7f6568004e70 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 10 monclient: _renew_subs 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.006+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658011b420 con 0x7f6568004e70 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f6585ec4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f6585ec4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f6585ec4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6580141d70 con 0x7f6568004e70 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f657401d3b0 con 0x7f6568004e70 2024-09-18T10:10:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f65677fe700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 0x7f656c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f65740475e0 con 0x7f6568004e70 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f657effd700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 0x7f656c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.010+0000 7f657effd700 10 monclient: get_auth_request con 0x7f656c035a70 auth_method 0 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.014+0000 7f657effd700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 0x7f656c037f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6570003980 tx=0x7f65700091d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.018+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6574005e10 con 0x7f6568004e70 2024-09-18T10:10:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.018+0000 7f65677fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.018+0000 7f65677fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:27.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.186+0000 7f6585ec4700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.186+0000 7f6585ec4700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.186+0000 7f6585ec4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.186+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f6580069ba0 con 0x7f6568004e70 2024-09-18T10:10:27.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.498+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' created v28) v1 ==== 154+0+0 (secure 0 0 0) 0x7f6580069ba0 con 0x7f6568004e70 2024-09-18T10:10:27.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.498+0000 7f65677fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.498+0000 7f65677fe700 10 monclient: _finish_command 2 = system:0 pool 'foo' created 2024-09-18T10:10:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f6585ec4700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f6585ec4700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f6585ec4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"} v 0) v1 -- 0x7f6580069ba0 con 0x7f6568004e70 2024-09-18T10:10:27.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f65677fe700 1 -- 172.21.15.150:0/641331245 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}]=0 pool 'foo' already exists v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6580069ba0 con 0x7f6568004e70 2024-09-18T10:10:27.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f65677fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123, "pgp_num": 123, "pool_type": "replicated"}] 2024-09-18T10:10:27.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.610+0000 7f65677fe700 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 msgr2=0x7f656c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 0x7f656c037f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6570003980 tx=0x7f65700091d0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 10 monclient: shutdown 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 msgr2=0x7f6580141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f657400e7b0 tx=0x7f6574009040 comp rx=0 tx=0).stop 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 shutdown_connections 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f656c035a70 0x7f656c037f20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65801317b0 0x7f658011aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f658011d430 0x7f6580131270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 --2- 172.21.15.150:0/641331245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6568004e70 0x7f6580141690 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 >> 172.21.15.150:0/641331245 conn(0x7f6580071b60 msgr2=0x7f65800a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:27.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 shutdown_connections 2024-09-18T10:10:27.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.614+0000 7f6585ec4700 1 -- 172.21.15.150:0/641331245 wait complete. 2024-09-18T10:10:27.620 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foo' already exists 2024-09-18T10:10:27.629 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create fooo 123 123 erasure default 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.730+0000 7f794484f700 10 monclient: get_monmap_and_config 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.730+0000 7f794484f700 10 monclient: build_initial_monmap 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.730+0000 7f794484f700 10 monclient: monmap: 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:27.735301+0000 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:27.735301+0000 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:27.736 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:27.737 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 Processor -- start 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 -- start start 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient: init 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f794009cf90 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f794009e480 0x7f79400a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): picked mon.noname-a con 0x7f794009e480 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79400990a0 0x7f7940139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): picked mon.noname-c con 0x7f79400990a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794009e060 con 0x7f794009cf90 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79400a66c0 con 0x7f794009e480 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794009d750 con 0x7f79400990a0 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f794484f700 10 monclient(hunting): authenticate will time out at 909.828797s 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79400990a0 0x7f7940139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79400990a0 0x7f7940139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45982/0 (socket says 172.21.15.150:45982) 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 -- 172.21.15.150:0/2409194556 learned_addr learned my addr 172.21.15.150:0/2409194556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): get_auth_request con 0x7f79400990a0 auth_method 0 2024-09-18T10:10:27.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 1 --2- 172.21.15.150:0/2409194556 >> v2:172.21.15.142:3300/0 conn(0x7f794009e480 0x7f79400a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): get_auth_request con 0x7f794009e480 auth_method 0 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient(hunting): handle_auth_done global_id 14257 payload 274 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 -- 172.21.15.150:0/2409194556 >> v2:172.21.15.142:3300/0 conn(0x7f794009e480 msgr2=0x7f79400a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 --2- 172.21.15.150:0/2409194556 >> v2:172.21.15.142:3300/0 conn(0x7f794009e480 0x7f79400a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 -- 172.21.15.150:0/2409194556 >> v1:172.21.15.142:6789/0 conn(0x7f794009cf90 legacy=0x7f79400a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.734+0000 7f793ed9d700 1 -- 172.21.15.150:0/2409194556 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794009d360 con 0x7f79400990a0 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 1 -- 172.21.15.150:0/2409194556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79400990a0 msgr2=0x7f7940139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 1 --2- 172.21.15.150:0/2409194556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79400990a0 0x7f7940139a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f794006fe50 tx=0x7f7938009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9cb57bd22b0d595 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 1 -- 172.21.15.150:0/2409194556 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793800e070 con 0x7f79400990a0 2024-09-18T10:10:27.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: dump: 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: _finish_auth 0 2024-09-18T10:10:27.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 1 -- 172.21.15.150:0/2409194556 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793800bc30 con 0x7f79400990a0 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 5 monclient: authenticate success, global_id 14257 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 1 -- 172.21.15.150:0/2409194556 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380092e0 con 0x7f79400990a0 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 4 set_mon_vals no callback set 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793d59a700 10 monclient: dump: 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.747 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: shutdown 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- 172.21.15.150:0/2409194556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79400990a0 msgr2=0x7f7940139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- 172.21.15.150:0/2409194556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79400990a0 0x7f7940139a30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f794006fe50 tx=0x7f7938009840 comp rx=0 tx=0).stop 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- 172.21.15.150:0/2409194556 shutdown_connections 2024-09-18T10:10:27.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- 172.21.15.150:0/2409194556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79400990a0 0x7f7940139a30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- 172.21.15.150:0/2409194556 >> v2:172.21.15.142:3300/0 conn(0x7f794009e480 0x7f79400a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- 172.21.15.150:0/2409194556 >> 172.21.15.150:0/2409194556 conn(0x7f79400a58a0 msgr2=0x7f79400a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:27.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- 172.21.15.150:0/2409194556 shutdown_connections 2024-09-18T10:10:27.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- 172.21.15.150:0/2409194556 wait complete. 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: build_initial_monmap 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: monmap: 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:27.755 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 Processor -- start 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- start start 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: init 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 0x7f7940097ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): picked mon.noname-c con 0x7f79400990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 0x7f7940097210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): picked mon.noname-a con 0x7f794009cf90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): picked mon.noname-b con 0x7f794009e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940097750 con 0x7f794009cf90 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794008e970 con 0x7f794009e480 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7940090780 con 0x7f79400990a0 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f794484f700 10 monclient(hunting): authenticate will time out at 909.835539s 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793e59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 msgr2=0x7f7940097ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 0x7f7940097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45996/0 (socket says 172.21.15.150:45996) 2024-09-18T10:10:27.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 1 -- 172.21.15.150:0/2969583743 learned_addr learned my addr 172.21.15.150:0/2969583743 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 10 monclient(hunting): get_auth_request con 0x7f794009e480 auth_method 0 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793dd9b700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 0x7f7940097210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793dd9b700 10 monclient(hunting): get_auth_request con 0x7f794009cf90 auth_method 0 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.738+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:27.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 10 monclient(hunting): handle_auth_done global_id 14260 payload 824 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 -- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 msgr2=0x7f7940097ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 0x7f7940097ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 -- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 msgr2=0x7f7940097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 0x7f7940097210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 monclient: found mon.noname-b 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938009ae0 con 0x7f794009e480 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793dd9b700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 0x7f7940097210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 -- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 msgr2=0x7f794008d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:27.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793ed9d700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f794009d890 tx=0x7f7938017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938012ae0 con 0x7f794009e480 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: dump: 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.760 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 5 monclient: authenticate success, global_id 14260 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938012c80 con 0x7f794009e480 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793801d420 con 0x7f794009e480 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 10 monclient: _renew_subs 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f793cd99700 4 set_mon_vals no callback set 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794008ede0 con 0x7f794009e480 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f792e7fc700 10 monclient: dump: 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:27.762 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 10 monclient: _renew_subs 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.742+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794008f260 con 0x7f794009e480 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f794484f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f794484f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f794484f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940090940 con 0x7f794009e480 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f793801d890 con 0x7f794009e480 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f792e7fc700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 0x7f7920037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4684+0+0 (secure 0 0 0) 0x7f7938049920 con 0x7f794009e480 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f793e59c700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 0x7f7920037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:27.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.746+0000 7f793e59c700 10 monclient: get_auth_request con 0x7f7920035a70 auth_method 0 2024-09-18T10:10:27.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.750+0000 7f793e59c700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 0x7f7920037f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7934007fd0 tx=0x7f7934007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:27.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.758+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7938007b00 con 0x7f794009e480 2024-09-18T10:10:27.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.758+0000 7f792e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:27.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.758+0000 7f792e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:27.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.922+0000 7f794484f700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:27.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.922+0000 7f794484f700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:27.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.922+0000 7f794484f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:27.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:27.922+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f794008f450 con 0x7f794009e480 2024-09-18T10:10:29.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.510+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' created v30) v1 ==== 188+0+0 (secure 0 0 0) 0x7f7938007320 con 0x7f794009e480 2024-09-18T10:10:29.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.510+0000 7f792e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:29.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.510+0000 7f792e7fc700 10 monclient: _finish_command 2 = system:0 pool 'fooo' created 2024-09-18T10:10:29.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f794484f700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:29.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f794484f700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:29.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f794484f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"} v 0) v1 -- 0x7f79400a0ee0 con 0x7f794009e480 2024-09-18T10:10:29.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f792e7fc700 1 -- 172.21.15.150:0/2969583743 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}]=0 pool 'fooo' already exists v30) v1 ==== 195+0+0 (secure 0 0 0) 0x7f7938007500 con 0x7f794009e480 2024-09-18T10:10:29.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f792e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooo", "pg_num": 123, "pgp_num": 123, "pool_type": "erasure", "erasure_code_profile": "default"}] 2024-09-18T10:10:29.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.626+0000 7f792e7fc700 10 monclient: _finish_command 3 = system:0 pool 'fooo' already exists 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 msgr2=0x7f7920037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 0x7f7920037f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7934007fd0 tx=0x7f7934007b00 comp rx=0 tx=0).stop 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 10 monclient: shutdown 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 msgr2=0x7f794008d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:29.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f794009d890 tx=0x7f7938017000 comp rx=0 tx=0).stop 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 shutdown_connections 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7920035a70 0x7f7920037f20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f794009e480 0x7f794008d3e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f794009cf90 0x7f7940097210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 --2- 172.21.15.150:0/2969583743 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79400990a0 0x7f7940097ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 >> 172.21.15.150:0/2969583743 conn(0x7f79400a58a0 msgr2=0x7f79400a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 shutdown_connections 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.630+0000 7f794484f700 1 -- 172.21.15.150:0/2969583743 wait complete. 2024-09-18T10:10:29.635 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fooo' already exists 2024-09-18T10:10:29.650 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create foooo 123 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: build_initial_monmap 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: monmap: 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:29.753257+0000 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:29.753257+0000 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:29.754 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:29.755 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 Processor -- start 2024-09-18T10:10:29.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 -- start start 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: init 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b600955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-b con 0x7f8b6009d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b60098ac0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b600995a0 0x7f8b60136780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-c con 0x7f8b600995a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6009b0a0 con 0x7f8b60098ac0 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b60099a00 con 0x7f8b600995a0 2024-09-18T10:10:29.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6009ab50 con 0x7f8b6009d970 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6858e700 10 monclient(hunting): authenticate will time out at 911.846835s 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6632a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 msgr2=0x7f8b600955b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b6632a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b600955b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b600995a0 0x7f8b60136780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b600995a0 0x7f8b60136780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39738/0 (socket says 172.21.15.150:39738) 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 -- 172.21.15.150:0/201600214 learned_addr learned my addr 172.21.15.150:0/201600214 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): get_auth_request con 0x7f8b600995a0 auth_method 0 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:29.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient(hunting): handle_auth_done global_id 14263 payload 274 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 -- 172.21.15.150:0/201600214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 msgr2=0x7f8b600955b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 --2- 172.21.15.150:0/201600214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b600955b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 -- 172.21.15.150:0/201600214 >> v1:172.21.15.142:6789/0 conn(0x7f8b60098ac0 legacy=0x7f8b60098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 monclient: found mon.noname-c 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.750+0000 7f8b66b2b700 1 -- 172.21.15.150:0/201600214 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6009ae30 con 0x7f8b600995a0 2024-09-18T10:10:29.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b66b2b700 1 -- 172.21.15.150:0/201600214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b600995a0 msgr2=0x7f8b60136780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:29.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b66b2b700 1 --2- 172.21.15.150:0/201600214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b600995a0 0x7f8b60136780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8b580099b0 tx=0x7f8b58009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9981cc8dcb34340d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 1 -- 172.21.15.150:0/201600214 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800bb50 con 0x7f8b600995a0 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: dump: 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:29.763 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: _finish_auth 0 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 1 -- 172.21.15.150:0/201600214 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5800bcb0 con 0x7f8b600995a0 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 1 -- 172.21.15.150:0/201600214 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58011650 con 0x7f8b600995a0 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 5 monclient: authenticate success, global_id 14263 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 4 set_mon_vals no callback set 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:29.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b65328700 10 monclient: dump: 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 10 monclient: shutdown 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:29.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 -- 172.21.15.150:0/201600214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b600995a0 msgr2=0x7f8b60136780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 --2- 172.21.15.150:0/201600214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b600995a0 0x7f8b60136780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8b580099b0 tx=0x7f8b58009250 comp rx=0 tx=0).stop 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 -- 172.21.15.150:0/201600214 shutdown_connections 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 --2- 172.21.15.150:0/201600214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b600995a0 0x7f8b60136780 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 --2- 172.21.15.150:0/201600214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b600955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 -- 172.21.15.150:0/201600214 >> 172.21.15.150:0/201600214 conn(0x7f8b600a15e0 msgr2=0x7f8b600a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 -- 172.21.15.150:0/201600214 shutdown_connections 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.754+0000 7f8b6858e700 1 -- 172.21.15.150:0/201600214 wait complete. 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient: build_initial_monmap 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient: monmap: 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:29.766 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 Processor -- start 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 -- start start 2024-09-18T10:10:29.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient: init 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-b con 0x7f8b60098ac0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b600995a0 0x7f8b6008ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-a con 0x7f8b600995a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b6008d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): picked mon.noname-c con 0x7f8b6009d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:29.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6009d360 con 0x7f8b600995a0 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b60098450 con 0x7f8b60098ac0 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600978c0 con 0x7f8b6009d970 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6858e700 10 monclient(hunting): authenticate will time out at 911.855118s 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39742/0 (socket says 172.21.15.150:39742) 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b66b2b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 msgr2=0x7f8b6008d3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 -- 172.21.15.150:0/3980596638 learned_addr learned my addr 172.21.15.150:0/3980596638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): get_auth_request con 0x7f8b60098ac0 auth_method 0 2024-09-18T10:10:29.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b66b2b700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b6008d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b65b29700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b600995a0 0x7f8b6008ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b65b29700 10 monclient(hunting): get_auth_request con 0x7f8b600995a0 auth_method 0 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b65b29700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b65b29700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b65b29700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient(hunting): handle_auth_done global_id 14266 payload 824 2024-09-18T10:10:29.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 -- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 msgr2=0x7f8b6008d3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b6008d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 -- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b600995a0 msgr2=0x7f8b6008ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b600995a0 0x7f8b6008ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient: _finish_hunting 0 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 monclient: found mon.noname-b 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.758+0000 7f8b6632a700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b580094f0 con 0x7f8b60098ac0 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6632a700 1 -- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 msgr2=0x7f8b6009c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6632a700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000b810 tx=0x7f8b50009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6632a700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000b810 tx=0x7f8b50009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f8b50015070 mon_map magic: 0 v1 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50015070 con 0x7f8b60098ac0 2024-09-18T10:10:29.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 10 monclient: dump: 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:29.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50003e20 con 0x7f8b60098ac0 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 5 monclient: authenticate success, global_id 14266 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _renew_subs 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6008e060 con 0x7f8b60098ac0 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b64b27700 4 set_mon_vals no callback set 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _renew_subs 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6008e480 con 0x7f8b60098ac0 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:29.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.762+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b60063ed0 con 0x7f8b60098ac0 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b500063a0 con 0x7f8b60098ac0 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 10 monclient: dump: 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:29.778 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8b50006750 con 0x7f8b60098ac0 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b567fc700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 0x7f8b44037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b65b29700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 0x7f8b44037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b65b29700 10 monclient: get_auth_request con 0x7f8b44035ac0 auth_method 0 2024-09-18T10:10:29.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.766+0000 7f8b65b29700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 0x7f8b44037f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007fd0 tx=0x7f8b5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:29.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:29.774+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f8b500480e0 con 0x7f8b60098ac0 2024-09-18T10:10:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.386+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8b5001a410 con 0x7f8b60098ac0 2024-09-18T10:10:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.386+0000 7f8b567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:30.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.386+0000 7f8b567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.594+0000 7f8b6858e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.594+0000 7f8b6858e700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.594+0000 7f8b6858e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:30.594+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f8b6008f440 con 0x7f8b60098ac0 2024-09-18T10:10:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.534+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' created v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8b50018310 con 0x7f8b60098ac0 2024-09-18T10:10:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.534+0000 7f8b567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.534+0000 7f8b567fc700 10 monclient: _finish_command 2 = system:0 pool 'foooo' created 2024-09-18T10:10:31.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b6858e700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:31.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b6858e700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:31.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b6858e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foooo", "pg_num": 123} v 0) v1 -- 0x7f8b600008d0 con 0x7f8b60098ac0 2024-09-18T10:10:31.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b567fc700 1 -- 172.21.15.150:0/3980596638 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}]=0 pool 'foooo' already exists v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8b500184f0 con 0x7f8b60098ac0 2024-09-18T10:10:31.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b567fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foooo", "pg_num": 123}] 2024-09-18T10:10:31.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.650+0000 7f8b567fc700 10 monclient: _finish_command 3 = system:0 pool 'foooo' already exists 2024-09-18T10:10:31.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 msgr2=0x7f8b44037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:31.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 0x7f8b44037f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007fd0 tx=0x7f8b5c009040 comp rx=0 tx=0).stop 2024-09-18T10:10:31.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 10 monclient: shutdown 2024-09-18T10:10:31.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:31.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 msgr2=0x7f8b6009c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000b810 tx=0x7f8b50009220 comp rx=0 tx=0).stop 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 shutdown_connections 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8b44035ac0 0x7f8b44037f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8b6009d970 0x7f8b6008d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b600995a0 0x7f8b6008ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 --2- 172.21.15.150:0/3980596638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b60098ac0 0x7f8b6009c880 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 >> 172.21.15.150:0/3980596638 conn(0x7f8b600a15e0 msgr2=0x7f8b600a0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 shutdown_connections 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.654+0000 7f8b6858e700 1 -- 172.21.15.150:0/3980596638 wait complete. 2024-09-18T10:10:31.662 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foooo' already exists 2024-09-18T10:10:31.667 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create foo 123 2024-09-18T10:10:31.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: get_monmap_and_config 2024-09-18T10:10:31.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: build_initial_monmap 2024-09-18T10:10:31.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: monmap: 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:31.765746+0000 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:31.765746+0000 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:31.767 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 Processor -- start 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 -- start start 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: init 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd49c09c930 0x7fd49c0977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-a con 0x7fd49c09c930 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:31.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd49c09cd00 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd49c137770 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c098f00 con 0x7fd49c137770 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c097d00 con 0x7fd49c09cd00 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c097030 con 0x7fd49c09c930 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a3f71700 10 monclient(hunting): authenticate will time out at 913.859396s 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a150c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd49c09cd00 0x7fd49c09d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:43018/0 (socket says 172.21.15.150:43018) 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a150c700 1 -- 172.21.15.150:0/4010536991 learned_addr learned my addr 172.21.15.150:0/4010536991 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:31.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a1d0d700 1 --2- 172.21.15.150:0/4010536991 >> v2:172.21.15.142:3300/0 conn(0x7fd49c09c930 0x7fd49c0977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a1d0d700 10 monclient(hunting): get_auth_request con 0x7fd49c09c930 auth_method 0 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a1d0d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a1d0d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.762+0000 7fd4a1d0d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/4010536991 >> v1:172.21.15.150:6789/0 conn(0x7fd49c09cd00 legacy=0x7fd49c09d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/4010536991 >> v1:172.21.15.142:6789/0 conn(0x7fd49c137770 legacy=0x7fd49c139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 monclient: found mon.noname-a 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:31.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:31.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/4010536991 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c06f730 con 0x7fd49c09c930 2024-09-18T10:10:31.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/4010536991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09c930 msgr2=0x7fd49c0977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:31.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a1d0d700 1 --2- 172.21.15.150:0/4010536991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09c930 0x7fd49c0977c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd4900098e0 tx=0x7fd4900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc0711da784757d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:31.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 1 -- 172.21.15.150:0/4010536991 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49000ba70 con 0x7fd49c09c930 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: dump: 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:31.775 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: _finish_auth 0 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a3f71700 5 monclient: authenticate success, global_id 4593 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 1 -- 172.21.15.150:0/4010536991 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49000bbd0 con 0x7fd49c09c930 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a3f71700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a3f71700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a3f71700 4 set_mon_vals no callback set 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 1 -- 172.21.15.150:0/4010536991 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490011420 con 0x7fd49c09c930 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:31.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.766+0000 7fd4a0d0b700 10 monclient: dump: 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: shutdown 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:31.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- 172.21.15.150:0/4010536991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09c930 msgr2=0x7fd49c0977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/4010536991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09c930 0x7fd49c0977c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd4900098e0 tx=0x7fd4900091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- 172.21.15.150:0/4010536991 shutdown_connections 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/4010536991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09c930 0x7fd49c0977c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- 172.21.15.150:0/4010536991 >> 172.21.15.150:0/4010536991 conn(0x7fd49c068360 msgr2=0x7fd49c0a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- 172.21.15.150:0/4010536991 shutdown_connections 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- 172.21.15.150:0/4010536991 wait complete. 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: build_initial_monmap 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: monmap: 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:31.778 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 Processor -- start 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- start start 2024-09-18T10:10:31.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: init 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-b con 0x7fd49c09c930 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09cd00 0x7fd49c142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-a con 0x7fd49c09cd00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 0x7fd49c11f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): picked mon.noname-c con 0x7fd49c137770 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:31.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c1226c0 con 0x7fd49c09cd00 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c122bf0 con 0x7fd49c09c930 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c09e3f0 con 0x7fd49c137770 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a3f71700 10 monclient(hunting): authenticate will time out at 913.866942s 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a250e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 msgr2=0x7fd49c11f100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39744/0 (socket says 172.21.15.150:39744) 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/2512489123 learned_addr learned my addr 172.21.15.150:0/2512489123 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): get_auth_request con 0x7fd49c09c930 auth_method 0 2024-09-18T10:10:31.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a250e700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 0x7fd49c11f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09cd00 0x7fd49c142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): get_auth_request con 0x7fd49c09cd00 auth_method 0 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:31.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a150c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient(hunting): handle_auth_done global_id 14269 payload 824 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 msgr2=0x7fd49c11f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 0x7fd49c11f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09cd00 msgr2=0x7fd49c142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09cd00 0x7fd49c142350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 monclient: found mon.noname-b 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.770+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490009420 con 0x7fd49c09c930 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.774+0000 7fd4a1d0d700 1 -- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 msgr2=0x7fd49c141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:31.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.774+0000 7fd4a1d0d700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd49c097eb0 tx=0x7fd490016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.774+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490011420 con 0x7fd49c09c930 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: dump: 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:31.787 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49001b680 con 0x7fd49c09c930 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49001ba00 con 0x7fd49c09c930 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 10 monclient: dump: 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:31.788 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48f7fe700 4 set_mon_vals no callback set 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 5 monclient: authenticate success, global_id 14269 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _renew_subs 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c09bae0 con 0x7fd49c09c930 2024-09-18T10:10:31.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _renew_subs 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c121660 con 0x7fd49c09c930 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c1229a0 con 0x7fd49c09c930 2024-09-18T10:10:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd49001c450 con 0x7fd49c09c930 2024-09-18T10:10:31.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 0x7fd480037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:31.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fd49004a160 con 0x7fd49c09c930 2024-09-18T10:10:31.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a150c700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 0x7fd480037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:31.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.778+0000 7fd4a150c700 10 monclient: get_auth_request con 0x7fd480035ac0 auth_method 0 2024-09-18T10:10:31.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.782+0000 7fd4a150c700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 0x7fd480037f70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd49c136ef0 tx=0x7fd498007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:31.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.790+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd490018950 con 0x7fd49c09c930 2024-09-18T10:10:31.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.790+0000 7fd48dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:31.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.790+0000 7fd48dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:31.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.954+0000 7fd4a3f71700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:31.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.954+0000 7fd4a3f71700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:31.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.954+0000 7fd4a3f71700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:31.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.954+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fd49c1229a0 con 0x7fd49c09c930 2024-09-18T10:10:31.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.958+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd49002d350 con 0x7fd49c09c930 2024-09-18T10:10:31.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.958+0000 7fd48dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:31.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:31.958+0000 7fd48dffb700 10 monclient: _finish_command 2 = system:0 pool 'foo' already exists 2024-09-18T10:10:32.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.054+0000 7fd4a3f71700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:32.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.054+0000 7fd4a3f71700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:32.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.054+0000 7fd4a3f71700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.054+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "foo", "pg_num": 123} v 0) v1 -- 0x7fd49c1229a0 con 0x7fd49c09c930 2024-09-18T10:10:32.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd48dffb700 1 -- 172.21.15.150:0/2512489123 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}]=0 pool 'foo' already exists v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd490018350 con 0x7fd49c09c930 2024-09-18T10:10:32.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd48dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "foo", "pg_num": 123}] 2024-09-18T10:10:32.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd48dffb700 10 monclient: _finish_command 3 = system:0 pool 'foo' already exists 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 msgr2=0x7fd480037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 0x7fd480037f70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd49c136ef0 tx=0x7fd498007f90 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 10 monclient: shutdown 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 msgr2=0x7fd49c141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.058+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd49c097eb0 tx=0x7fd490016000 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 shutdown_connections 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd480035ac0 0x7fd480037f70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c137770 0x7fd49c11f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c09cd00 0x7fd49c142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 --2- 172.21.15.150:0/2512489123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c09c930 0x7fd49c141e10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 >> 172.21.15.150:0/2512489123 conn(0x7fd49c068360 msgr2=0x7fd49c09a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:32.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 shutdown_connections 2024-09-18T10:10:32.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.062+0000 7fd4a3f71700 1 -- 172.21.15.150:0/2512489123 wait complete. 2024-09-18T10:10:32.066 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foo' already exists 2024-09-18T10:10:32.075 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set foo size 1 --yes-i-really-mean-it 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: get_monmap_and_config 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: build_initial_monmap 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: monmap: 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:32.173571+0000 2024-09-18T10:10:32.174 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:32.173571+0000 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:32.175 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 Processor -- start 2024-09-18T10:10:32.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 -- start start 2024-09-18T10:10:32.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: init 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b780a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-b con 0x7f0b780a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b780a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b780a0bc0 0x7f0b7813a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-c con 0x7f0b780a0bc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780a6380 con 0x7f0b780a07d0 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780a6770 con 0x7f0b780a0bc0 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:32.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7809e660 con 0x7f0b780a5fb0 2024-09-18T10:10:32.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:32.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7fce2700 10 monclient(hunting): authenticate will time out at 914.267446s 2024-09-18T10:10:32.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b780a0bc0 0x7f0b7813a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:32.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7da7e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 msgr2=0x7f0b780a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:32.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7da7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b780a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b780a0bc0 0x7f0b7813a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39748/0 (socket says 172.21.15.150:39748) 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 1 -- 172.21.15.150:0/4075624275 learned_addr learned my addr 172.21.15.150:0/4075624275 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): get_auth_request con 0x7f0b780a0bc0 auth_method 0 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:32.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 10 monclient(hunting): handle_auth_done global_id 14272 payload 274 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 1 -- 172.21.15.150:0/4075624275 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 msgr2=0x7f0b780a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.170+0000 7f0b7e27f700 1 --2- 172.21.15.150:0/4075624275 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b780a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 1 -- 172.21.15.150:0/4075624275 >> v1:172.21.15.142:6789/0 conn(0x7f0b780a07d0 legacy=0x7f0b78136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 10 monclient: _finish_hunting 0 2024-09-18T10:10:32.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 1 monclient: found mon.noname-c 2024-09-18T10:10:32.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:32.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 1 -- 172.21.15.150:0/4075624275 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7809e920 con 0x7f0b780a0bc0 2024-09-18T10:10:32.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 1 -- 172.21.15.150:0/4075624275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a0bc0 msgr2=0x7f0b7813a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:32.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7e27f700 1 --2- 172.21.15.150:0/4075624275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a0bc0 0x7f0b7813a140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009f20 tx=0x7f0b70009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ad3891dc051c2e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 1 -- 172.21.15.150:0/4075624275 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000e070 con 0x7f0b780a0bc0 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: dump: 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: _finish_auth 0 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 1 -- 172.21.15.150:0/4075624275 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7000bc90 con 0x7f0b780a0bc0 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 5 monclient: authenticate success, global_id 14272 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:32.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 4 set_mon_vals no callback set 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 1 -- 172.21.15.150:0/4075624275 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70012680 con 0x7f0b780a0bc0 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7ca7c700 10 monclient: dump: 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:32.191 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 10 monclient: shutdown 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 -- 172.21.15.150:0/4075624275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a0bc0 msgr2=0x7f0b7813a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/4075624275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a0bc0 0x7f0b7813a140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009f20 tx=0x7f0b70009840 comp rx=0 tx=0).stop 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 -- 172.21.15.150:0/4075624275 shutdown_connections 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/4075624275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a0bc0 0x7f0b7813a140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/4075624275 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b780a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 -- 172.21.15.150:0/4075624275 >> 172.21.15.150:0/4075624275 conn(0x7f0b78071ae0 msgr2=0x7f0b780a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 -- 172.21.15.150:0/4075624275 shutdown_connections 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.174+0000 7f0b7fce2700 1 -- 172.21.15.150:0/4075624275 wait complete. 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient: build_initial_monmap 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient: monmap: 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:32.193 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 Processor -- start 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 -- start start 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient: init 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:32.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-b con 0x7f0b780a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 0x7f0b7808e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-a con 0x7f0b780a0bc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b7808e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): picked mon.noname-c con 0x7f0b780a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78092540 con 0x7f0b780a0bc0 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78091620 con 0x7f0b780a07d0 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78091900 con 0x7f0b780a5fb0 2024-09-18T10:10:32.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7fce2700 10 monclient(hunting): authenticate will time out at 914.274351s 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7e27f700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 msgr2=0x7f0b7808e980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39762/0 (socket says 172.21.15.150:39762) 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 -- 172.21.15.150:0/1268424332 learned_addr learned my addr 172.21.15.150:0/1268424332 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): get_auth_request con 0x7f0b780a07d0 auth_method 0 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7e27f700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b7808e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:32.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 0x7f0b7808e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 10 monclient(hunting): get_auth_request con 0x7f0b780a0bc0 auth_method 0 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient(hunting): handle_auth_done global_id 14275 payload 824 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 -- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 msgr2=0x7f0b7808e980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b7808e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 -- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 msgr2=0x7f0b7808e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 0x7f0b7808e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient: _finish_hunting 0 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 monclient: found mon.noname-b 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7da7e700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70009ae0 con 0x7f0b780a07d0 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.178+0000 7f0b7d27d700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 0x7f0b7808e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:32.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7da7e700 1 -- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 msgr2=0x7f0b7808df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7da7e700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b7e0 tx=0x7f0b74009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7da7e700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b7e0 tx=0x7f0b74009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f0b74015070 mon_map magic: 0 v1 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74015070 con 0x7f0b780a07d0 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: dump: 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:32.199 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 5 monclient: authenticate success, global_id 14275 2024-09-18T10:10:32.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74003e20 con 0x7f0b780a07d0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _renew_subs 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b78091340 con 0x7f0b780a07d0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6ffff700 4 set_mon_vals no callback set 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _renew_subs 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b78099480 con 0x7f0b780a07d0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74006380 con 0x7f0b780a07d0 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:32.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 10 monclient: dump: 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:32.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b74006730 con 0x7f0b780a07d0 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7806e3c0 con 0x7f0b780a07d0 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.182+0000 7f0b6dffb700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 0x7f0b5c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.186+0000 7f0b7d27d700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 0x7f0b5c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:32.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.186+0000 7f0b7d27d700 10 monclient: get_auth_request con 0x7f0b5c035ac0 auth_method 0 2024-09-18T10:10:32.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.186+0000 7f0b7d27d700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 0x7f0b5c037f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0b68006fd0 tx=0x7f0b68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:32.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.194+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f0b74048190 con 0x7f0b780a07d0 2024-09-18T10:10:32.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.806+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0b74018af0 con 0x7f0b780a07d0 2024-09-18T10:10:32.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.806+0000 7f0b6dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:32.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.806+0000 7f0b6dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:32.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.974+0000 7f0b7fce2700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:32.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.974+0000 7f0b7fce2700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:32.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.974+0000 7f0b7fce2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:32.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:32.974+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0b7809d8c0 con 0x7f0b780a07d0 2024-09-18T10:10:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.562+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v34) v1 ==== 152+0+0 (secure 0 0 0) 0x7f0b74018310 con 0x7f0b780a07d0 2024-09-18T10:10:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.562+0000 7f0b6dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.562+0000 7f0b6dffb700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 1 2024-09-18T10:10:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.690+0000 7f0b7fce2700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.690+0000 7f0b7fce2700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.690+0000 7f0b7fce2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:33.690+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0b78063ed0 con 0x7f0b780a07d0 2024-09-18T10:10:34.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.570+0000 7f0b6dffb700 1 -- 172.21.15.150:0/1268424332 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}]=0 set pool 4 size to 1 v35) v1 ==== 152+0+0 (secure 0 0 0) 0x7f0b740184f0 con 0x7f0b780a07d0 2024-09-18T10:10:34.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.570+0000 7f0b6dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "1", "yes_i_really_mean_it": true}] 2024-09-18T10:10:34.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.570+0000 7f0b6dffb700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 1 2024-09-18T10:10:34.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.582+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 msgr2=0x7f0b5c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 0x7f0b5c037f70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0b68006fd0 tx=0x7f0b68008040 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 10 monclient: shutdown 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 msgr2=0x7f0b7808df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400b7e0 tx=0x7f0b74009220 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 shutdown_connections 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b5c035ac0 0x7f0b5c037f70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b780a5fb0 0x7f0b7808e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b780a0bc0 0x7f0b7808e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 --2- 172.21.15.150:0/1268424332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b780a07d0 0x7f0b7808df00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 >> 172.21.15.150:0/1268424332 conn(0x7f0b78071ae0 msgr2=0x7f0b7809a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 shutdown_connections 2024-09-18T10:10:34.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:34.586+0000 7f0b7fce2700 1 -- 172.21.15.150:0/1268424332 wait complete. 2024-09-18T10:10:34.600 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 4 size to 1 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_config_value foo min_size 1 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ local pool_name config_opt expected_val val 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ pool_name=foo 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ config_opt=min_size 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_val=1 2024-09-18T10:10:34.609 INFO:tasks.workunit.client.0.smithi150.stderr:++ get_config_value_or_die foo min_size 2024-09-18T10:10:34.610 INFO:tasks.workunit.client.0.smithi150.stderr:++ local pool_name config_opt raw val 2024-09-18T10:10:34.610 INFO:tasks.workunit.client.0.smithi150.stderr:++ pool_name=foo 2024-09-18T10:10:34.610 INFO:tasks.workunit.client.0.smithi150.stderr:++ config_opt=min_size 2024-09-18T10:10:34.610 INFO:tasks.workunit.client.0.smithi150.stderr:+++ ceph osd pool get foo min_size 2024-09-18T10:10:35.003 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw='min_size: 1' 2024-09-18T10:10:35.003 INFO:tasks.workunit.client.0.smithi150.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T10:10:35.004 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size: 1 2024-09-18T10:10:35.004 INFO:tasks.workunit.client.0.smithi150.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T10:10:35.006 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw=min_size:1 2024-09-18T10:10:35.007 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size:1 2024-09-18T10:10:35.007 INFO:tasks.workunit.client.0.smithi150.stderr:+++ cut -f2 -d: 2024-09-18T10:10:35.009 INFO:tasks.workunit.client.0.smithi150.stderr:++ val=1 2024-09-18T10:10:35.010 INFO:tasks.workunit.client.0.smithi150.stderr:++ echo 1 2024-09-18T10:10:35.010 INFO:tasks.workunit.client.0.smithi150.stderr:++ return 0 2024-09-18T10:10:35.010 INFO:tasks.workunit.client.0.smithi150.stderr:+ val=1 2024-09-18T10:10:35.010 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 1 != \1 ]] 2024-09-18T10:10:35.010 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set foo size 4 2024-09-18T10:10:35.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: get_monmap_and_config 2024-09-18T10:10:35.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: build_initial_monmap 2024-09-18T10:10:35.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: monmap: 2024-09-18T10:10:35.110 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:35.109730+0000 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:35.109730+0000 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:35.111 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 Processor -- start 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 -- start start 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: init 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c1409b2b0 0x7f6c1409b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-c con 0x7f6c1409b2b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c140955b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c14098180 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c140985a0 con 0x7f6c140955b0 2024-09-18T10:10:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1409aab0 con 0x7f6c14098180 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1409a7d0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c1a9dd700 10 monclient(hunting): authenticate will time out at 917.203302s 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c13fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c1409b2b0 0x7f6c1409b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c18f7a700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6c14098180 0x7f6c14135980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:43020/0 (socket says 172.21.15.150:43020) 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c18f7a700 1 -- 172.21.15.150:0/3275441450 learned_addr learned my addr 172.21.15.150:0/3275441450 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c13fff700 10 monclient(hunting): get_auth_request con 0x7f6c1409b2b0 auth_method 0 2024-09-18T10:10:35.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.106+0000 7f6c13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c18f7a700 1 -- 172.21.15.150:0/3275441450 >> v1:172.21.15.150:6789/0 conn(0x7f6c14098180 legacy=0x7f6c14135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient(hunting): handle_auth_done global_id 14284 payload 274 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 1 -- 172.21.15.150:0/3275441450 >> v1:172.21.15.150:6789/0 conn(0x7f6c14098180 legacy=0x7f6c14135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 1 -- 172.21.15.150:0/3275441450 >> v1:172.21.15.142:6789/0 conn(0x7f6c140955b0 legacy=0x7f6c14097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:35.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c12ffd700 1 -- 172.21.15.150:0/3275441450 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550037723 0 0) 0x7f6c1409aab0 con 0x7f6c14098180 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c137fe700 1 -- 172.21.15.150:0/3275441450 >> v1:172.21.15.142:6789/0 conn(0x7f6c140955b0 legacy=0x7f6c14097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 1 monclient: found mon.noname-c 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c13fff700 1 -- 172.21.15.150:0/3275441450 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140992f0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.110+0000 7f6c12ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 1 -- 172.21.15.150:0/3275441450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 msgr2=0x7f6c1409b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 1 --2- 172.21.15.150:0/3275441450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1409b680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009a10 tx=0x7f6c040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae5ac9cadb83f5b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 1 -- 172.21.15.150:0/3275441450 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400bb60 con 0x7f6c1409b2b0 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: dump: 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:35.121 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 5 monclient: authenticate success, global_id 14284 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 1 -- 172.21.15.150:0/3275441450 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0400bcc0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 4 set_mon_vals no callback set 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 1 -- 172.21.15.150:0/3275441450 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04011650 con 0x7f6c1409b2b0 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c12ffd700 10 monclient: dump: 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.123 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: shutdown 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/3275441450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 msgr2=0x7f6c1409b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/3275441450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1409b680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009a10 tx=0x7f6c040091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/3275441450 shutdown_connections 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/3275441450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1409b680 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/3275441450 >> 172.21.15.150:0/3275441450 conn(0x7f6c140a15e0 msgr2=0x7f6c140a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/3275441450 shutdown_connections 2024-09-18T10:10:35.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/3275441450 wait complete. 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: build_initial_monmap 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: monmap: 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 Processor -- start 2024-09-18T10:10:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- start start 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: init 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c140955b0 0x7f6c14141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-a con 0x7f6c140955b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 0x7f6c14094520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-c con 0x7f6c14098180 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): picked mon.noname-b con 0x7f6c1409b2b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14090190 con 0x7f6c140955b0 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14091740 con 0x7f6c1409b2b0 2024-09-18T10:10:35.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c140907e0 con 0x7f6c14098180 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c1a9dd700 10 monclient(hunting): authenticate will time out at 917.210991s 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c137fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 msgr2=0x7f6c14094520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 0x7f6c14094520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39812/0 (socket says 172.21.15.150:39812) 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 -- 172.21.15.150:0/1293136474 learned_addr learned my addr 172.21.15.150:0/1293136474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): get_auth_request con 0x7f6c1409b2b0 auth_method 0 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c140955b0 0x7f6c14141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): get_auth_request con 0x7f6c140955b0 auth_method 0 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c13fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient(hunting): handle_auth_done global_id 14290 payload 824 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 -- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 msgr2=0x7f6c14094520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 0x7f6c14094520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 -- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c140955b0 msgr2=0x7f6c14141b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c140955b0 0x7f6c14141b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient: _finish_hunting 0 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 monclient: found mon.noname-b 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.114+0000 7f6c18f7a700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04009420 con 0x7f6c1409b2b0 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c18f7a700 1 -- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 msgr2=0x7f6c1408e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c18f7a700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00cf60 tx=0x7f6c0c001c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00ab70 con 0x7f6c1409b2b0 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: dump: 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: _finish_auth 0 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 5 monclient: authenticate success, global_id 14290 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c00acf0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 10 monclient: _renew_subs 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c007340 con 0x7f6c1409b2b0 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1406fc00 con 0x7f6c1409b2b0 2024-09-18T10:10:35.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c11ffb700 4 set_mon_vals no callback set 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 10 monclient: dump: 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:35.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 10 monclient: _renew_subs 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c14142090 con 0x7f6c1409b2b0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.118+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6c0c007770 con 0x7f6c1409b2b0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c0bfff700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 0x7f6bf4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f6c0c048ac0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c13fff700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 0x7f6bf4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c13fff700 10 monclient: get_auth_request con 0x7f6bf4035a70 auth_method 0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c1a9dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c1a9dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c1a9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c140990d0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.122+0000 7f6c13fff700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 0x7f6bf4037f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c04017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.134+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6c0c00ae60 con 0x7f6c1409b2b0 2024-09-18T10:10:35.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.134+0000 7f6c0bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.134+0000 7f6c0bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:35.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.294+0000 7f6c1a9dd700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:35.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.294+0000 7f6c1a9dd700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:35.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.294+0000 7f6c1a9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.294+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f6c14063ed0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.548 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:10:35.548 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:10:35.548 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:10:35.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:10:35.551 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:10:35.551 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:10:35.551 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:35.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.598+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6c0c02a0a0 con 0x7f6c1409b2b0 2024-09-18T10:10:35.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.598+0000 7f6c0bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:35.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.598+0000 7f6c0bfff700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 4 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: get_monmap_and_config 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: build_initial_monmap 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: monmap: 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:35.680494+0000 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:35.680494+0000 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.681 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:35.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:35.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:35.682 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:35.682 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:35.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 Processor -- start 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 -- start start 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: init 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe700a5f90 0x7fbe700a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-a con 0x7fbe700a5f90 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe700990b0 0x7fbe700a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-c con 0x7fbe700990b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe7009e360 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7009d980 con 0x7fbe7009e360 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7009d780 con 0x7fbe700a5f90 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7009fc70 con 0x7fbe700990b0 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe75759700 10 monclient(hunting): authenticate will time out at 924.605939s 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe700a5f90 0x7fbe700a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe700a5f90 0x7fbe700a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38178/0 (socket says 172.21.15.142:38178) 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 -- 172.21.15.142:0/28287916 learned_addr learned my addr 172.21.15.142:0/28287916 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): get_auth_request con 0x7fbe700a5f90 auth_method 0 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6e7fc700 1 --2- 172.21.15.142:0/28287916 >> v2:172.21.15.150:3300/0 conn(0x7fbe700990b0 0x7fbe700a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6e7fc700 10 monclient(hunting): get_auth_request con 0x7fbe700990b0 auth_method 0 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient(hunting): handle_auth_done global_id 4611 payload 274 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 -- 172.21.15.142:0/28287916 >> v2:172.21.15.150:3300/0 conn(0x7fbe700990b0 msgr2=0x7fbe700a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 --2- 172.21.15.142:0/28287916 >> v2:172.21.15.150:3300/0 conn(0x7fbe700990b0 0x7fbe700a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 -- 172.21.15.142:0/28287916 >> v1:172.21.15.150:6789/0 conn(0x7fbe7009e360 legacy=0x7fbe70138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient: _finish_hunting 0 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 monclient: found mon.noname-a 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.679+0000 7fbe6effd700 1 -- 172.21.15.142:0/28287916 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700a6360 con 0x7fbe700a5f90 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 1 -- 172.21.15.142:0/28287916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe700a5f90 msgr2=0x7fbe700a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 1 --2- 172.21.15.142:0/28287916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe700a5f90 0x7fbe700a0270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe70099720 tx=0x7fbe68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d1b81dadd6a3ee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 1 -- 172.21.15.142:0/28287916 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6800e070 con 0x7fbe700a5f90 2024-09-18T10:10:35.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: dump: 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 5 monclient: authenticate success, global_id 4611 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 1 -- 172.21.15.142:0/28287916 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6800bc60 con 0x7fbe700a5f90 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 1 -- 172.21.15.142:0/28287916 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe680092e0 con 0x7fbe700a5f90 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 4 set_mon_vals no callback set 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6dffb700 10 monclient: dump: 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.691 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: shutdown 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- 172.21.15.142:0/28287916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe700a5f90 msgr2=0x7fbe700a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- 172.21.15.142:0/28287916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe700a5f90 0x7fbe700a0270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe70099720 tx=0x7fbe68009840 comp rx=0 tx=0).stop 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- 172.21.15.142:0/28287916 shutdown_connections 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- 172.21.15.142:0/28287916 >> v2:172.21.15.150:3300/0 conn(0x7fbe700990b0 0x7fbe700a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- 172.21.15.142:0/28287916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe700a5f90 0x7fbe700a0270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- 172.21.15.142:0/28287916 >> 172.21.15.142:0/28287916 conn(0x7fbe70071680 msgr2=0x7fbe700a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- 172.21.15.142:0/28287916 shutdown_connections 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- 172.21.15.142:0/28287916 wait complete. 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: build_initial_monmap 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: monmap: 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.693 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 Processor -- start 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- start start 2024-09-18T10:10:35.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: init 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe700990b0 0x7fbe7008cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-b con 0x7fbe700990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-a con 0x7fbe7009e360 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): picked mon.noname-c con 0x7fbe700a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe700a40c0 con 0x7fbe7009e360 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7008ed60 con 0x7fbe700990b0 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70090380 con 0x7fbe700a5f90 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe75759700 10 monclient(hunting): authenticate will time out at 924.610874s 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38182/0 (socket says 172.21.15.142:38182) 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 -- 172.21.15.142:0/178151243 learned_addr learned my addr 172.21.15.142:0/178151243 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): get_auth_request con 0x7fbe7009e360 auth_method 0 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 10 monclient(hunting): get_auth_request con 0x7fbe700a5f90 auth_method 0 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe700990b0 0x7fbe7008cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): get_auth_request con 0x7fbe700990b0 auth_method 0 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 msgr2=0x7fbe7008da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 msgr2=0x7fbe7008da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:35.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6f7fe700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient(hunting): handle_auth_done global_id 4614 payload 824 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 msgr2=0x7fbe7008da50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe700990b0 msgr2=0x7fbe7008cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe700990b0 0x7fbe7008cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:35.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 monclient: found mon.noname-a 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.683+0000 7fbe6e7fc700 1 -- 172.21.15.142:0/178151243 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68009ae0 con 0x7fbe7009e360 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe6e7fc700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 msgr2=0x7fbe7008d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe6e7fc700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000d890 tx=0x7fbe60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60012800 con 0x7fbe7009e360 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: dump: 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.702 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 5 monclient: authenticate success, global_id 4614 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _renew_subs 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70090660 con 0x7fbe7009e360 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe60012960 con 0x7fbe7009e360 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe6d7fa700 4 set_mon_vals no callback set 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _renew_subs 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70091580 con 0x7fbe7009e360 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60007470 con 0x7fbe7009e360 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 10 monclient: dump: 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:35.704 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbe6001e460 con 0x7fbe7009e360 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.687+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70090a60 con 0x7fbe7009e360 2024-09-18T10:10:35.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.691+0000 7fbe66ffd700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 0x7fbe50037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.691+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fbe60048d80 con 0x7fbe7009e360 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.691+0000 7fbe6effd700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 0x7fbe50037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.691+0000 7fbe6effd700 10 monclient: get_auth_request con 0x7fbe50035ac0 auth_method 0 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.691+0000 7fbe6effd700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 0x7fbe50037f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe70099720 tx=0x7fbe68018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.699+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbe6001cb40 con 0x7fbe7009e360 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.699+0000 7fbe66ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:35.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.699+0000 7fbe66ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:35.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.718+0000 7f6c1a9dd700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:35.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.718+0000 7f6c1a9dd700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:35.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.718+0000 7f6c1a9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:35.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:35.718+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"} v 0) v1 -- 0x7f6c1409eb00 con 0x7f6c1409b2b0 2024-09-18T10:10:35.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:35.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:35.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:35.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbe700696a0 con 0x7fbe7009e360 2024-09-18T10:10:35.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe66ffd700 1 -- 172.21.15.142:0/178151243 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fbe6001c360 con 0x7fbe7009e360 2024-09-18T10:10:35.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe66ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:35.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe66ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:35.926 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:35.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 msgr2=0x7fbe50037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 0x7fbe50037f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe70099720 tx=0x7fbe68018000 comp rx=0 tx=0).stop 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 10 monclient: shutdown 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 msgr2=0x7fbe7008d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.923+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000d890 tx=0x7fbe60008040 comp rx=0 tx=0).stop 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 shutdown_connections 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe50035ac0 0x7fbe50037f70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe700a5f90 0x7fbe7008da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7009e360 0x7fbe7008d510 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 --2- 172.21.15.142:0/178151243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe700990b0 0x7fbe7008cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:35.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 >> 172.21.15.142:0/178151243 conn(0x7fbe70071680 msgr2=0x7fbe7009e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:35.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 shutdown_connections 2024-09-18T10:10:35.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:35.927+0000 7fbe75759700 1 -- 172.21.15.142:0/178151243 wait complete. 2024-09-18T10:10:36.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.594+0000 7f6c0bfff700 1 -- 172.21.15.150:0/1293136474 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}]=0 set pool 4 size to 4 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6c0c010af0 con 0x7f6c1409b2b0 2024-09-18T10:10:36.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.594+0000 7f6c0bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "4"}] 2024-09-18T10:10:36.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.594+0000 7f6c0bfff700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 4 2024-09-18T10:10:36.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 msgr2=0x7f6bf4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:36.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 0x7f6bf4037f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c04017000 comp rx=0 tx=0).stop 2024-09-18T10:10:36.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 10 monclient: shutdown 2024-09-18T10:10:36.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 msgr2=0x7f6c1408e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c00cf60 tx=0x7f6c0c001c30 comp rx=0 tx=0).stop 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 shutdown_connections 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf4035a70 0x7f6bf4037f20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c1409b2b0 0x7f6c1408e270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c14098180 0x7f6c14094520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 --2- 172.21.15.150:0/1293136474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c140955b0 0x7f6c14141b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 >> 172.21.15.150:0/1293136474 conn(0x7f6c140a15e0 msgr2=0x7f6c1409fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 shutdown_connections 2024-09-18T10:10:36.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:36.618+0000 7f6c1a9dd700 1 -- 172.21.15.150:0/1293136474 wait complete. 2024-09-18T10:10:36.627 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 4 size to 4 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_config_value foo min_size 2 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ local pool_name config_opt expected_val val 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ pool_name=foo 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ config_opt=min_size 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_val=2 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:++ get_config_value_or_die foo min_size 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:++ local pool_name config_opt raw val 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:++ pool_name=foo 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:++ config_opt=min_size 2024-09-18T10:10:36.644 INFO:tasks.workunit.client.0.smithi150.stderr:+++ ceph osd pool get foo min_size 2024-09-18T10:10:37.062 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw='min_size: 2' 2024-09-18T10:10:37.062 INFO:tasks.workunit.client.0.smithi150.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T10:10:37.063 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size: 2 2024-09-18T10:10:37.063 INFO:tasks.workunit.client.0.smithi150.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T10:10:37.066 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw=min_size:2 2024-09-18T10:10:37.066 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size:2 2024-09-18T10:10:37.067 INFO:tasks.workunit.client.0.smithi150.stderr:+++ cut -f2 -d: 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ val=2 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ echo 2 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:++ return 0 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ val=2 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 2 != \2 ]] 2024-09-18T10:10:37.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set foo size 10 2024-09-18T10:10:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: get_monmap_and_config 2024-09-18T10:10:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: build_initial_monmap 2024-09-18T10:10:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: monmap: 2024-09-18T10:10:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:37.169767+0000 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:37.169767+0000 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:37.171 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 Processor -- start 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 -- start start 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: init 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc8ec09d000 0x7fc8ec0a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-a con 0x7fc8ec09d000 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8ec09e780 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8ec1379b0 0x7fc8ec139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-c con 0x7fc8ec1379b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec0699a0 con 0x7fc8ec09e780 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0a1640 con 0x7fc8ec09d000 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec09e3d0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8f1343700 10 monclient(hunting): authenticate will time out at 919.263342s 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8ec1379b0 0x7fc8ec139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8ec1379b0 0x7fc8ec139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39836/0 (socket says 172.21.15.150:39836) 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/4036029279 learned_addr learned my addr 172.21.15.150:0/4036029279 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): get_auth_request con 0x7fc8ec1379b0 auth_method 0 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eaffd700 1 --2- 172.21.15.150:0/4036029279 >> v2:172.21.15.142:3300/0 conn(0x7fc8ec09d000 0x7fc8ec0a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eaffd700 10 monclient(hunting): get_auth_request con 0x7fc8ec09d000 auth_method 0 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.166+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_done global_id 14308 payload 274 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/4036029279 >> v2:172.21.15.142:3300/0 conn(0x7fc8ec09d000 msgr2=0x7fc8ec0a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 --2- 172.21.15.150:0/4036029279 >> v2:172.21.15.142:3300/0 conn(0x7fc8ec09d000 0x7fc8ec0a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/4036029279 >> v1:172.21.15.142:6789/0 conn(0x7fc8ec09e780 legacy=0x7fc8ec0a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/4036029279 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec09ebf0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/4036029279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 msgr2=0x7fc8ec139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8eb7fe700 1 --2- 172.21.15.150:0/4036029279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec139d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec09d9c0 tx=0x7fc8e4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8c4359650275250 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 1 -- 172.21.15.150:0/4036029279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400e070 con 0x7fc8ec1379b0 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: dump: 2024-09-18T10:10:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8f1343700 5 monclient: authenticate success, global_id 14308 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 1 -- 172.21.15.150:0/4036029279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400bc30 con 0x7fc8ec1379b0 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8f1343700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8f1343700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 1 -- 172.21.15.150:0/4036029279 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40092e0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8f1343700 4 set_mon_vals no callback set 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.170+0000 7fc8e9ffb700 10 monclient: dump: 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: shutdown 2024-09-18T10:10:37.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- 172.21.15.150:0/4036029279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 msgr2=0x7fc8ec139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- 172.21.15.150:0/4036029279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec139d90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec09d9c0 tx=0x7fc8e4009840 comp rx=0 tx=0).stop 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- 172.21.15.150:0/4036029279 shutdown_connections 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- 172.21.15.150:0/4036029279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec139d90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- 172.21.15.150:0/4036029279 >> v2:172.21.15.142:3300/0 conn(0x7fc8ec09d000 0x7fc8ec0a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- 172.21.15.150:0/4036029279 >> 172.21.15.150:0/4036029279 conn(0x7fc8ec0a5830 msgr2=0x7fc8ec0a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- 172.21.15.150:0/4036029279 shutdown_connections 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- 172.21.15.150:0/4036029279 wait complete. 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: build_initial_monmap 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: monmap: 2024-09-18T10:10:37.185 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 Processor -- start 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- start start 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: init 2024-09-18T10:10:37.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8ec09d000 0x7fc8ec12c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-a con 0x7fc8ec09d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-c con 0x7fc8ec09e780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): picked mon.noname-b con 0x7fc8ec1379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:37.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec12e0d0 con 0x7fc8ec09d000 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec12fa90 con 0x7fc8ec1379b0 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec12e850 con 0x7fc8ec09e780 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8f1343700 10 monclient(hunting): authenticate will time out at 919.270363s 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39850/0 (socket says 172.21.15.150:39850) 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/3638379158 learned_addr learned my addr 172.21.15.150:0/3638379158 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): get_auth_request con 0x7fc8ec1379b0 auth_method 0 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:37.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8ec09d000 0x7fc8ec12c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): get_auth_request con 0x7fc8ec09d000 auth_method 0 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:37.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 10 monclient(hunting): get_auth_request con 0x7fc8ec09e780 auth_method 0 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient(hunting): handle_auth_done global_id 14311 payload 824 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 msgr2=0x7fc8ec12ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 msgr2=0x7fc8ec12ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 msgr2=0x7fc8ec12ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:37.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8ea7fc700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8ec09d000 msgr2=0x7fc8ec12c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8ec09d000 0x7fc8ec12c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.174+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009ae0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8eb7fe700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 msgr2=0x7fc8ec12cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:37.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8eb7fe700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b6c0 tx=0x7fc8e4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:37.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4012ae0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:37.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:37.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: dump: 2024-09-18T10:10:37.194 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:37.194 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:37.194 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.194 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:37.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 5 monclient: authenticate success, global_id 14311 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _renew_subs 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4012c40 con 0x7fc8ec1379b0 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec12f7b0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e401d420 con 0x7fc8ec1379b0 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 10 monclient: dump: 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:37.197 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8e97fa700 4 set_mon_vals no callback set 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _renew_subs 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec12fe20 con 0x7fc8ec1379b0 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc8e401d850 con 0x7fc8ec1379b0 2024-09-18T10:10:37.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.178+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec06e3e0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.182+0000 7fc8d2ffd700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 0x7fc8d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.182+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc8e4049c90 con 0x7fc8ec1379b0 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.182+0000 7fc8eaffd700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 0x7fc8d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.182+0000 7fc8eaffd700 10 monclient: get_auth_request con 0x7fc8d8035ac0 auth_method 0 2024-09-18T10:10:37.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.182+0000 7fc8eaffd700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 0x7fc8d8037f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0009d10 tx=0x7fc8e00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:37.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.190+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc8e4007500 con 0x7fc8ec1379b0 2024-09-18T10:10:37.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.190+0000 7fc8d2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:37.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.190+0000 7fc8d2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:37.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.350+0000 7fc8f1343700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:37.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.350+0000 7fc8f1343700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:37.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.350+0000 7fc8f1343700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.350+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7fc8ec0a49c0 con 0x7fc8ec1379b0 2024-09-18T10:10:37.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.614+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v38) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc8e4007320 con 0x7fc8ec1379b0 2024-09-18T10:10:37.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.614+0000 7fc8d2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:37.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.614+0000 7fc8d2ffd700 10 monclient: _finish_command 2 = system:0 set pool 4 size to 10 2024-09-18T10:10:37.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.758+0000 7fc8f1343700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:37.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.758+0000 7fc8f1343700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:37.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.758+0000 7fc8f1343700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:37.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:37.758+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"} v 0) v1 -- 0x7fc8ec0a49c0 con 0x7fc8ec1379b0 2024-09-18T10:10:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.614+0000 7fc8d2ffd700 1 -- 172.21.15.150:0/3638379158 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}]=0 set pool 4 size to 10 v39) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc8e4007500 con 0x7fc8ec1379b0 2024-09-18T10:10:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.614+0000 7fc8d2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "10"}] 2024-09-18T10:10:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.614+0000 7fc8d2ffd700 10 monclient: _finish_command 3 = system:0 set pool 4 size to 10 2024-09-18T10:10:38.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 msgr2=0x7fc8d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:38.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 0x7fc8d8037f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0009d10 tx=0x7fc8e00091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 10 monclient: shutdown 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 msgr2=0x7fc8ec12cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.626+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b6c0 tx=0x7fc8e4017000 comp rx=0 tx=0).stop 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 shutdown_connections 2024-09-18T10:10:38.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc8d8035ac0 0x7fc8d8037f70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8ec1379b0 0x7fc8ec12cf40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc8ec09e780 0x7fc8ec12ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 --2- 172.21.15.150:0/3638379158 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc8ec09d000 0x7fc8ec12c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 >> 172.21.15.150:0/3638379158 conn(0x7fc8ec0a5830 msgr2=0x7fc8ec09b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 shutdown_connections 2024-09-18T10:10:38.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:38.630+0000 7fc8f1343700 1 -- 172.21.15.150:0/3638379158 wait complete. 2024-09-18T10:10:38.639 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 4 size to 10 2024-09-18T10:10:38.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_config_value foo min_size 5 2024-09-18T10:10:38.655 INFO:tasks.workunit.client.0.smithi150.stderr:+ local pool_name config_opt expected_val val 2024-09-18T10:10:38.655 INFO:tasks.workunit.client.0.smithi150.stderr:+ pool_name=foo 2024-09-18T10:10:38.655 INFO:tasks.workunit.client.0.smithi150.stderr:+ config_opt=min_size 2024-09-18T10:10:38.655 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_val=5 2024-09-18T10:10:38.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ get_config_value_or_die foo min_size 2024-09-18T10:10:38.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ local pool_name config_opt raw val 2024-09-18T10:10:38.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ pool_name=foo 2024-09-18T10:10:38.660 INFO:tasks.workunit.client.0.smithi150.stderr:++ config_opt=min_size 2024-09-18T10:10:38.661 INFO:tasks.workunit.client.0.smithi150.stderr:+++ ceph osd pool get foo min_size 2024-09-18T10:10:38.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:39.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.043+0000 7fd6ceaca700 10 monclient: get_monmap_and_config 2024-09-18T10:10:39.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.043+0000 7fd6ceaca700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.043+0000 7fd6ceaca700 10 monclient: monmap: 2024-09-18T10:10:39.047 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:39.047 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:39.046824+0000 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:39.046824+0000 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:39.048 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 Processor -- start 2024-09-18T10:10:39.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 -- start start 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient: init 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd6c80a1cf0 0x7fd6c812f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-a con 0x7fd6c80a1cf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6c812f810 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6c80a02d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c809fb90 con 0x7fd6c80a02d0 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c809d990 con 0x7fd6c812f810 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c809dc70 con 0x7fd6c80a1cf0 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ceaca700 10 monclient(hunting): authenticate will time out at 927.972980s 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cd2c7700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd6c812f810 0x7fd6c809f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:45730/0 (socket says 172.21.15.142:45730) 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6ce2c9700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd6c80a02d0 0x7fd6c8139420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55994/0 (socket says 172.21.15.142:55994) 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd6c80a1cf0 0x7fd6c812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/3618605693 learned_addr learned my addr 172.21.15.142:0/3618605693 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): get_auth_request con 0x7fd6c80a1cf0 auth_method 0 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_done global_id 4629 payload 274 2024-09-18T10:10:39.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 1 -- 172.21.15.142:0/3618605693 >> v1:172.21.15.150:6789/0 conn(0x7fd6c812f810 legacy=0x7fd6c809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 1 -- 172.21.15.142:0/3618605693 >> v1:172.21.15.142:6789/0 conn(0x7fd6c80a02d0 legacy=0x7fd6c8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 1 monclient: found mon.noname-a 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.047+0000 7fd6cdac8700 1 -- 172.21.15.142:0/3618605693 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c80a20c0 con 0x7fd6c80a1cf0 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6cdac8700 1 -- 172.21.15.142:0/3618605693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c80a1cf0 msgr2=0x7fd6c812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6cdac8700 1 --2- 172.21.15.142:0/3618605693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c80a1cf0 0x7fd6c812f030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40098e0 tx=0x7fd6c40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8984b984ab4ec667 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 1 -- 172.21.15.142:0/3618605693 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c400ba70 con 0x7fd6c80a1cf0 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: dump: 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.055 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 5 monclient: authenticate success, global_id 4629 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 1 -- 172.21.15.142:0/3618605693 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c400bbd0 con 0x7fd6c80a1cf0 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:39.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 4 set_mon_vals no callback set 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 1 -- 172.21.15.142:0/3618605693 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4011420 con 0x7fd6c80a1cf0 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ccac6700 10 monclient: dump: 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.057 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 10 monclient: shutdown 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 -- 172.21.15.142:0/3618605693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c80a1cf0 msgr2=0x7fd6c812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/3618605693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c80a1cf0 0x7fd6c812f030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40098e0 tx=0x7fd6c40091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 -- 172.21.15.142:0/3618605693 shutdown_connections 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/3618605693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c80a1cf0 0x7fd6c812f030 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 -- 172.21.15.142:0/3618605693 >> 172.21.15.142:0/3618605693 conn(0x7fd6c8071820 msgr2=0x7fd6c80a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 -- 172.21.15.142:0/3618605693 shutdown_connections 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 -- 172.21.15.142:0/3618605693 wait complete. 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 10 monclient: monmap: 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:39.059 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.051+0000 7fd6ceaca700 1 Processor -- start 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 -- start start 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient: init 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6c80a02d0 0x7fd6c806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-b con 0x7fd6c80a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-a con 0x7fd6c812f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6c806be10 0x7fd6c806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): picked mon.noname-c con 0x7fd6c806be10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c806d840 con 0x7fd6c812f810 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c806c7f0 con 0x7fd6c80a02d0 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c809cdf0 con 0x7fd6c806be10 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ceaca700 10 monclient(hunting): authenticate will time out at 927.981291s 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38246/0 (socket says 172.21.15.142:38246) 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/4294952846 learned_addr learned my addr 172.21.15.142:0/4294952846 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): get_auth_request con 0x7fd6c812f810 auth_method 0 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ce2c9700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6c806be10 0x7fd6c806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6c80a02d0 0x7fd6c806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): get_auth_request con 0x7fd6c80a02d0 auth_method 0 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ce2c9700 10 monclient(hunting): get_auth_request con 0x7fd6c806be10 auth_method 0 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ce2c9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ce2c9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6ce2c9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cdac8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6c806be10 msgr2=0x7fd6c806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6c806be10 0x7fd6c806c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6c80a02d0 msgr2=0x7fd6c806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6c80a02d0 0x7fd6c806b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 monclient: found mon.noname-a 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.055+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/4294952846 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4009420 con 0x7fd6c812f810 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6cd2c7700 1 -- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 msgr2=0x7fd6c806b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6cd2c7700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c5e0 tx=0x7fd6b8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8005d40 con 0x7fd6c812f810 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: dump: 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.068 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b80070d0 con 0x7fd6c812f810 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 5 monclient: authenticate success, global_id 4632 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _renew_subs 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c806ec10 con 0x7fd6c812f810 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8007490 con 0x7fd6c812f810 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 10 monclient: dump: 2024-09-18T10:10:39.069 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bffff700 4 set_mon_vals no callback set 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _renew_subs 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c806d9d0 con 0x7fd6c812f810 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd6b801b3b0 con 0x7fd6c812f810 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c806eda0 con 0x7fd6c812f810 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 0x7fd6ac037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fd6b8047c10 con 0x7fd6c812f810 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6cdac8700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 0x7fd6ac037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.059+0000 7fd6cdac8700 10 monclient: get_auth_request con 0x7fd6ac035ac0 auth_method 0 2024-09-18T10:10:39.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.063+0000 7fd6cdac8700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 0x7fd6ac037f70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8071e30 tx=0x7fd6c4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.067+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd6b800f820 con 0x7fd6c812f810 2024-09-18T10:10:39.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.067+0000 7fd6bdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.067+0000 7fd6bdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:39.094 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw='min_size: 5' 2024-09-18T10:10:39.095 INFO:tasks.workunit.client.0.smithi150.stderr:++ [[ 0 -ne 0 ]] 2024-09-18T10:10:39.095 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size: 5 2024-09-18T10:10:39.095 INFO:tasks.workunit.client.0.smithi150.stderr:+++ sed -e 's/[{} "]//g' 2024-09-18T10:10:39.097 INFO:tasks.workunit.client.0.smithi150.stderr:++ raw=min_size:5 2024-09-18T10:10:39.098 INFO:tasks.workunit.client.0.smithi150.stderr:+++ echo min_size:5 2024-09-18T10:10:39.098 INFO:tasks.workunit.client.0.smithi150.stderr:+++ cut -f2 -d: 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:++ val=5 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:++ echo 5 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:++ return 0 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:+ val=5 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 5 != \5 ]] 2024-09-18T10:10:39.101 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set foo size 0 2024-09-18T10:10:39.102 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:39.102 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set foo size 0 2024-09-18T10:10:39.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 10 monclient: get_monmap_and_config 2024-09-18T10:10:39.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 10 monclient: monmap: 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:39.190815+0000 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:39.190815+0000 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:39.193 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:39.193 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 1 Processor -- start 2024-09-18T10:10:39.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 1 -- start start 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.186+0000 7f0ee88de700 10 monclient: init 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-b con 0x7f0ee406f160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0ee409cc90 0x7f0ee409d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-c con 0x7f0ee409cc90 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ee4132560 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee409e170 con 0x7f0ee4132560 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee4071250 con 0x7f0ee409cc90 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee409d5a0 con 0x7f0ee406f160 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee88de700 10 monclient(hunting): authenticate will time out at 921.284273s 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee2d9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0ee4132560 0x7f0ee4132960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38198/0 (socket says 172.21.15.150:38198) 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/2628672276 learned_addr learned my addr 172.21.15.150:0/2628672276 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/2628672276 >> v2:172.21.15.150:3300/0 conn(0x7f0ee409cc90 0x7f0ee409d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): get_auth_request con 0x7f0ee409cc90 auth_method 0 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 10 monclient(hunting): get_auth_request con 0x7f0ee406f160 auth_method 0 2024-09-18T10:10:39.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 1 -- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 msgr2=0x7f0ee409c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 1 -- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 msgr2=0x7f0ee409c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee259c700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/2628672276 >> v1:172.21.15.150:6789/0 conn(0x7f0ee4132560 legacy=0x7f0ee4132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient(hunting): handle_auth_done global_id 14323 payload 274 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 msgr2=0x7f0ee409c750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/2628672276 >> v1:172.21.15.150:6789/0 conn(0x7f0ee4132560 legacy=0x7f0ee4132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 monclient: found mon.noname-c 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.190+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/2628672276 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4098ab0 con 0x7f0ee409cc90 2024-09-18T10:10:39.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/2628672276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee409cc90 msgr2=0x7f0ee409d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee409cc90 0x7f0ee409d060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009960 tx=0x7f0ed80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee150eb198afc70 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 1 -- 172.21.15.150:0/2628672276 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800ba70 con 0x7f0ee409cc90 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: dump: 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.204 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 5 monclient: authenticate success, global_id 14323 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 1 -- 172.21.15.150:0/2628672276 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed800bbf0 con 0x7f0ee409cc90 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 4 set_mon_vals no callback set 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 1 -- 172.21.15.150:0/2628672276 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8011420 con 0x7f0ee409cc90 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee159a700 10 monclient: dump: 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.206 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.207 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.207 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.207 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.207 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.207 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 10 monclient: shutdown 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 -- 172.21.15.150:0/2628672276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee409cc90 msgr2=0x7f0ee409d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee409cc90 0x7f0ee409d060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009960 tx=0x7f0ed80091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 -- 172.21.15.150:0/2628672276 shutdown_connections 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee409cc90 0x7f0ee409d060 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 --2- 172.21.15.150:0/2628672276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee406f160 0x7f0ee409c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 -- 172.21.15.150:0/2628672276 >> 172.21.15.150:0/2628672276 conn(0x7f0ee40681c0 msgr2=0x7f0ee40a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 -- 172.21.15.150:0/2628672276 shutdown_connections 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 -- 172.21.15.150:0/2628672276 wait complete. 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 10 monclient: monmap: 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:39.209 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:39.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:39.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:39.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:39.211 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.194+0000 7f0ee88de700 1 Processor -- start 2024-09-18T10:10:39.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 -- start start 2024-09-18T10:10:39.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient: init 2024-09-18T10:10:39.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:39.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ee406f160 0x7f0ee4141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-a con 0x7f0ee406f160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:39.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 0x7f0ee4131720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-c con 0x7f0ee4132560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee4121990 0x7f0ee411f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): picked mon.noname-b con 0x7f0ee4121990 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee40a1600 con 0x7f0ee406f160 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4120e30 con 0x7f0ee4121990 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee4122880 con 0x7f0ee4132560 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee88de700 10 monclient(hunting): authenticate will time out at 921.292822s 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee4121990 0x7f0ee411f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee4121990 0x7f0ee411f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47238/0 (socket says 172.21.15.150:47238) 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/1032820979 learned_addr learned my addr 172.21.15.150:0/1032820979 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): get_auth_request con 0x7f0ee4121990 auth_method 0 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 0x7f0ee4131720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 10 monclient(hunting): get_auth_request con 0x7f0ee4132560 auth_method 0 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ee406f160 0x7f0ee4141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): get_auth_request con 0x7f0ee406f160 auth_method 0 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 msgr2=0x7f0ee4131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 1 -- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 msgr2=0x7f0ee4131720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 0x7f0ee4131720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee1d9b700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 0x7f0ee4131720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:39.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient(hunting): handle_auth_done global_id 14329 payload 824 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 msgr2=0x7f0ee4131720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ee4132560 0x7f0ee4131720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ee406f160 msgr2=0x7f0ee4141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ee406f160 0x7f0ee4141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 monclient: found mon.noname-b 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.198+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/1032820979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8009420 con 0x7f0ee4121990 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee2d9d700 1 -- 172.21.15.150:0/1032820979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee4121990 msgr2=0x7f0ee411f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee2d9d700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ee4121990 0x7f0ee411f2c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00cf60 tx=0x7f0edc001c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc00ab40 con 0x7f0ee4121990 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: dump: 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.219 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 5 monclient: authenticate success, global_id 14329 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _renew_subs 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc00aca0 con 0x7f0ee4121990 2024-09-18T10:10:39.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 1 -- 172.21.15.150:0/1032820979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee4122b60 con 0x7f0ee4121990 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc007340 con 0x7f0ee4121990 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed3fff700 4 set_mon_vals no callback set 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 10 monclient: dump: 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.221 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _renew_subs 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 1 -- 172.21.15.150:0/1032820979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee4142000 con 0x7f0ee4121990 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0edc0076f0 con 0x7f0ee4121990 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee88de700 1 -- 172.21.15.150:0/1032820979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee409ca70 con 0x7f0ee4121990 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ec4035ac0 0x7f0ec4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f0edc048a50 con 0x7f0ee4121990 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee259c700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ec4035ac0 0x7f0ec4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.202+0000 7f0ee259c700 10 monclient: get_auth_request con 0x7f0ec4035ac0 auth_method 0 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.206+0000 7f0ee259c700 1 --2- 172.21.15.150:0/1032820979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ec4035ac0 0x7f0ec4037f70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4006f30 tx=0x7f0ed4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.214+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0edc01a590 con 0x7f0ee4121990 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.214+0000 7f0ed27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.214+0000 7f0ed27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6ceaca700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6ceaca700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6ceaca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd6c8072a80 con 0x7fd6c812f810 2024-09-18T10:10:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6bdffb700 1 -- 172.21.15.142:0/4294952846 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fd6b8017020 con 0x7fd6c812f810 2024-09-18T10:10:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6bdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.295+0000 7fd6bdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:39.298 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 msgr2=0x7fd6ac037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 0x7fd6ac037f70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8071e30 tx=0x7fd6c4016000 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 10 monclient: shutdown 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 msgr2=0x7fd6c806b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800c5e0 tx=0x7fd6b8004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 shutdown_connections 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd6ac035ac0 0x7fd6ac037f70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6c806be10 0x7fd6c806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6c812f810 0x7fd6c806b8d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 --2- 172.21.15.142:0/4294952846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6c80a02d0 0x7fd6c806b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 >> 172.21.15.142:0/4294952846 conn(0x7fd6c8071820 msgr2=0x7fd6c813a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 shutdown_connections 2024-09-18T10:10:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:39.299+0000 7fd6ceaca700 1 -- 172.21.15.142:0/4294952846 wait complete. 2024-09-18T10:10:39.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.358+0000 7f0ee88de700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T10:10:39.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.358+0000 7f0ee88de700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T10:10:39.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.358+0000 7f0ee88de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.358+0000 7f0ee88de700 1 -- 172.21.15.150:0/1032820979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"} v 0) v1 -- 0x7f0ee4063ed0 con 0x7f0ee4121990 2024-09-18T10:10:39.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.362+0000 7f0ed27fc700 1 -- 172.21.15.150:0/1032820979 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}]=-22 pool size must be between 1 and 10 v39) v1 ==== 136+0+0 (secure 0 0 0) 0x7f0edc010ab0 con 0x7f0ee4121990 2024-09-18T10:10:39.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.362+0000 7f0ed27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "0"}] 2024-09-18T10:10:39.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.362+0000 7f0ed27fc700 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-18T10:10:39.366 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-18T10:10:39.370 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:39.370 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool set foo size 20 2024-09-18T10:10:39.370 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:39.370 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set foo size 20 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: get_monmap_and_config 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: monmap: 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:39.473103+0000 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:39.473103+0000 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.474 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.475 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:39.475 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:39.475 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:39.475 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:39.476 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:39.476 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 Processor -- start 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 -- start start 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: init 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66140989e0 0x7f6614099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): picked mon.noname-a con 0x7f66140989e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f661409b8f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): picked mon.noname-b con 0x7f66141338c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6614097cd0 con 0x7f661409b8f0 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6614096d40 con 0x7f66140989e0 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66140979f0 con 0x7f66141338c0 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f661a867700 10 monclient(hunting): authenticate will time out at 921.567515s 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f66137fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f661409b8f0 0x7f661409bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38210/0 (socket says 172.21.15.150:38210) 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f66137fe700 1 -- 172.21.15.150:0/2070441555 learned_addr learned my addr 172.21.15.150:0/2070441555 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:39.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6613fff700 1 --2- 172.21.15.150:0/2070441555 >> v2:172.21.15.142:3300/0 conn(0x7f66140989e0 0x7f6614099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6613fff700 10 monclient(hunting): get_auth_request con 0x7f66140989e0 auth_method 0 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6613fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6613fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6613fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.470+0000 7f6618e04700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 10 monclient(hunting): get_auth_request con 0x7f66141338c0 auth_method 0 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 1 -- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 msgr2=0x7f6614135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 1 -- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 msgr2=0x7f6614135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6618e04700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient(hunting): handle_auth_done global_id 4638 payload 274 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 -- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 msgr2=0x7f6614135ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 -- 172.21.15.150:0/2070441555 >> v1:172.21.15.150:6789/0 conn(0x7f661409b8f0 legacy=0x7f661409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:39.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 monclient: found mon.noname-a 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 -- 172.21.15.150:0/2070441555 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614097fb0 con 0x7f66140989e0 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 -- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66140989e0 msgr2=0x7f6614099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6613fff700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66140989e0 0x7f6614099320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f66040098e0 tx=0x7f66040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a5b2a04c508b69 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 1 -- 172.21.15.150:0/2070441555 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604009e80 con 0x7f66140989e0 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: dump: 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.489 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f661a867700 5 monclient: authenticate success, global_id 4638 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f661a867700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 1 -- 172.21.15.150:0/2070441555 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660400ba70 con 0x7f66140989e0 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f661a867700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f661a867700 4 set_mon_vals no callback set 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 1 -- 172.21.15.150:0/2070441555 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660400bdf0 con 0x7f66140989e0 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.474+0000 7f6612ffd700 10 monclient: dump: 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.490 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: shutdown 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66140989e0 msgr2=0x7f6614099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66140989e0 0x7f6614099320 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f66040098e0 tx=0x7f66040091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- 172.21.15.150:0/2070441555 shutdown_connections 2024-09-18T10:10:39.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66141338c0 0x7f6614135ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- 172.21.15.150:0/2070441555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66140989e0 0x7f6614099320 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- 172.21.15.150:0/2070441555 >> 172.21.15.150:0/2070441555 conn(0x7f6614068590 msgr2=0x7f66140a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- 172.21.15.150:0/2070441555 shutdown_connections 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- 172.21.15.150:0/2070441555 wait complete. 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: monmap: 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:39.492 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 Processor -- start 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- start start 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: init 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66140989e0 0x7f661413e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): picked mon.noname-b con 0x7f66140989e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 0x7f6614132cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): picked mon.noname-c con 0x7f661409b8f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66141338c0 0x7f6614118890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): picked mon.noname-a con 0x7f66141338c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66140a08f0 con 0x7f66141338c0 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661411a2f0 con 0x7f66140989e0 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661411bd90 con 0x7f661409b8f0 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f661a867700 10 monclient(hunting): authenticate will time out at 921.574487s 2024-09-18T10:10:39.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66140989e0 0x7f661413e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66140989e0 0x7f661413e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47248/0 (socket says 172.21.15.150:47248) 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 -- 172.21.15.150:0/3641503893 learned_addr learned my addr 172.21.15.150:0/3641503893 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): get_auth_request con 0x7f66140989e0 auth_method 0 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66141338c0 0x7f6614118890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 0x7f6614132cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): get_auth_request con 0x7f66141338c0 auth_method 0 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 10 monclient(hunting): get_auth_request con 0x7f661409b8f0 auth_method 0 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6618e04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 1 -- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 msgr2=0x7f6614132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 1 -- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 msgr2=0x7f6614132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient(hunting): handle_auth_done global_id 14332 payload 824 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 0x7f6614132cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:39.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f66137fe700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 0x7f6614132cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 -- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 msgr2=0x7f6614132cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f661409b8f0 0x7f6614132cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 -- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66141338c0 msgr2=0x7f6614118890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66141338c0 0x7f6614118890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 monclient: found mon.noname-b 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.478+0000 7f6613fff700 1 -- 172.21.15.150:0/3641503893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604009420 con 0x7f66140989e0 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f6613fff700 1 -- 172.21.15.150:0/3641503893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66140989e0 msgr2=0x7f661413e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f6613fff700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66140989e0 0x7f661413e5e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f66040030c0 tx=0x7f6604016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604006af0 con 0x7f66140989e0 2024-09-18T10:10:39.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: dump: 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6604020070 con 0x7f66140989e0 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 5 monclient: authenticate success, global_id 14332 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _renew_subs 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 1 -- 172.21.15.150:0/3641503893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661411c070 con 0x7f66140989e0 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660401b6a0 con 0x7f66140989e0 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 10 monclient: dump: 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.499 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f66127fc700 4 set_mon_vals no callback set 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _renew_subs 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 1 -- 172.21.15.150:0/3641503893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661413f390 con 0x7f66140989e0 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6604022030 con 0x7f66140989e0 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65f4035ac0 0x7f65f4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f660404a3a0 con 0x7f66140989e0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f661a867700 1 -- 172.21.15.150:0/3641503893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661411bed0 con 0x7f66140989e0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.482+0000 7f66137fe700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65f4035ac0 0x7f65f4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.486+0000 7f66137fe700 10 monclient: get_auth_request con 0x7f65f4035ac0 auth_method 0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.486+0000 7f66137fe700 1 --2- 172.21.15.150:0/3641503893 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65f4035ac0 0x7f65f4037f70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f65fc007fd0 tx=0x7f65fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.490+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f661411bed0 con 0x7f66140989e0 2024-09-18T10:10:39.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.490+0000 7f660bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.490+0000 7f660bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:39.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f661a867700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T10:10:39.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f661a867700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T10:10:39.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f661a867700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f661a867700 1 -- 172.21.15.150:0/3641503893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"} v 0) v1 -- 0x7f661411bed0 con 0x7f66140989e0 2024-09-18T10:10:39.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f660bfff700 1 -- 172.21.15.150:0/3641503893 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}]=-22 pool size must be between 1 and 10 v39) v1 ==== 137+0+0 (secure 0 0 0) 0x7f661411bed0 con 0x7f66140989e0 2024-09-18T10:10:39.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f660bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "foo", "var": "size", "val": "20"}] 2024-09-18T10:10:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.646+0000 7f660bfff700 10 monclient: _finish_command 2 = mon:22 pool size must be between 1 and 10 2024-09-18T10:10:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: pool size must be between 1 and 10 2024-09-18T10:10:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool delete foo 2024-09-18T10:10:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 10 monclient: get_monmap_and_config 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 10 monclient: monmap: 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:39.742317+0000 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:39.742317+0000 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:39.743 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 1 Processor -- start 2024-09-18T10:10:39.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 1 -- start start 2024-09-18T10:10:39.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.738+0000 7fc24fea2700 10 monclient: init 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc2480a2760 0x7fc24809c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-a con 0x7fc2480a2760 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc24809cd00 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc24809d100 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24809daa0 con 0x7fc24809d100 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2480999e0 con 0x7fc24809cd00 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480a2b30 con 0x7fc2480a2760 2024-09-18T10:10:39.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24fea2700 10 monclient(hunting): authenticate will time out at 921.837104s 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24d43d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc24809cd00 0x7fc2481345d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38216/0 (socket says 172.21.15.150:38216) 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24d43d700 1 -- 172.21.15.150:0/564163271 learned_addr learned my addr 172.21.15.150:0/564163271 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 1 --2- 172.21.15.150:0/564163271 >> v2:172.21.15.142:3300/0 conn(0x7fc2480a2760 0x7fc24809c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): get_auth_request con 0x7fc2480a2760 auth_method 0 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_done global_id 4644 payload 274 2024-09-18T10:10:39.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 1 -- 172.21.15.150:0/564163271 >> v1:172.21.15.150:6789/0 conn(0x7fc24809cd00 legacy=0x7fc2481345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 1 -- 172.21.15.150:0/564163271 >> v1:172.21.15.142:6789/0 conn(0x7fc24809d100 legacy=0x7fc248137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 1 monclient: found mon.noname-a 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:39.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.742+0000 7fc24dc3e700 1 -- 172.21.15.150:0/564163271 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248068a10 con 0x7fc2480a2760 2024-09-18T10:10:39.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24dc3e700 1 -- 172.21.15.150:0/564163271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2480a2760 msgr2=0x7fc24809c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24dc3e700 1 --2- 172.21.15.150:0/564163271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2480a2760 0x7fc24809c7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2440098e0 tx=0x7fc2440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc986de9046e4c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 1 -- 172.21.15.150:0/564163271 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24400ba70 con 0x7fc2480a2760 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: dump: 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.755 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 5 monclient: authenticate success, global_id 4644 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 1 -- 172.21.15.150:0/564163271 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24400bbd0 con 0x7fc2480a2760 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 1 -- 172.21.15.150:0/564163271 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244011420 con 0x7fc2480a2760 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 4 set_mon_vals no callback set 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24cc3c700 10 monclient: dump: 2024-09-18T10:10:39.756 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 10 monclient: shutdown 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- 172.21.15.150:0/564163271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2480a2760 msgr2=0x7fc24809c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 --2- 172.21.15.150:0/564163271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2480a2760 0x7fc24809c7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2440098e0 tx=0x7fc2440091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- 172.21.15.150:0/564163271 shutdown_connections 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 --2- 172.21.15.150:0/564163271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2480a2760 0x7fc24809c7c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- 172.21.15.150:0/564163271 >> 172.21.15.150:0/564163271 conn(0x7fc248068220 msgr2=0x7fc2480a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- 172.21.15.150:0/564163271 shutdown_connections 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- 172.21.15.150:0/564163271 wait complete. 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 10 monclient: build_initial_monmap 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 10 monclient: monmap: 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.758 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 Processor -- start 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 1 -- start start 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.746+0000 7fc24fea2700 10 monclient: init 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc24809cd00 0x7fc24813c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-a con 0x7fc24809cd00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 0x7fc24813b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-c con 0x7fc24809d100 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:39.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2480a2760 0x7fc24813f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): picked mon.noname-b con 0x7fc2480a2760 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248121050 con 0x7fc24809cd00 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248120a90 con 0x7fc2480a2760 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24811e520 con 0x7fc24809d100 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient(hunting): authenticate will time out at 921.844339s 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2480a2760 0x7fc24813f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2480a2760 0x7fc24813f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47258/0 (socket says 172.21.15.150:47258) 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 -- 172.21.15.150:0/712298128 learned_addr learned my addr 172.21.15.150:0/712298128 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:39.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): get_auth_request con 0x7fc2480a2760 auth_method 0 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc24809cd00 0x7fc24813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 0x7fc24813b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): get_auth_request con 0x7fc24809cd00 auth_method 0 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 10 monclient(hunting): get_auth_request con 0x7fc24809d100 auth_method 0 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24dc3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 1 -- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 msgr2=0x7fc24813b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 1 -- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 msgr2=0x7fc24813b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient(hunting): handle_auth_done global_id 14335 payload 824 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 0x7fc24813b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 -- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 msgr2=0x7fc24813b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24d43d700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 0x7fc24813b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:39.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc24809d100 0x7fc24813b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 -- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc24809cd00 msgr2=0x7fc24813c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc24809cd00 0x7fc24813c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient: _finish_hunting 0 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 monclient: found mon.noname-b 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 -- 172.21.15.150:0/712298128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc244009420 con 0x7fc2480a2760 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 -- 172.21.15.150:0/712298128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2480a2760 msgr2=0x7fc24813f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24e43f700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2480a2760 0x7fc24813f5d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc24000c930 tx=0x7fc2400026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240012840 con 0x7fc2480a2760 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:39.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: dump: 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:39.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 5 monclient: authenticate success, global_id 14335 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2400129c0 con 0x7fc2480a2760 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _renew_subs 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 -- 172.21.15.150:0/712298128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2481207b0 con 0x7fc2480a2760 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240012d80 con 0x7fc2480a2760 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23dffb700 10 monclient: dump: 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.767 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _renew_subs 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc24fea2700 1 -- 172.21.15.150:0/712298128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc248120e20 con 0x7fc2480a2760 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.750+0000 7fc23f7fe700 4 set_mon_vals no callback set 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc24000a5c0 con 0x7fc2480a2760 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24fea2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24fea2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24fea2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24fea2700 1 -- 172.21.15.150:0/712298128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc248099700 con 0x7fc2480a2760 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc23dffb700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc22c035ac0 0x7fc22c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fc2400486b0 con 0x7fc2480a2760 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24dc3e700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc22c035ac0 0x7fc22c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.754+0000 7fc24dc3e700 10 monclient: get_auth_request con 0x7fc22c035ac0 auth_method 0 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.758+0000 7fc24dc3e700 1 --2- 172.21.15.150:0/712298128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc22c035ac0 0x7fc22c037f70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc2440030c0 tx=0x7fc244016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.762+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc240010d50 con 0x7fc2480a2760 2024-09-18T10:10:39.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.762+0000 7fc23dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:39.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.762+0000 7fc23dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:39.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.914+0000 7fc24fea2700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T10:10:39.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.914+0000 7fc24fea2700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T10:10:39.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.914+0000 7fc24fea2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:39.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.914+0000 7fc24fea2700 1 -- 172.21.15.150:0/712298128 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo"} v 0) v1 -- 0x7fc248063ed0 con 0x7fc2480a2760 2024-09-18T10:10:39.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.918+0000 7fc23dffb700 1 -- 172.21.15.150:0/712298128 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v39) v1 ==== 274+0+0 (secure 0 0 0) 0x7fc240010750 con 0x7fc2480a2760 2024-09-18T10:10:39.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.918+0000 7fc23dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo"}] 2024-09-18T10:10:39.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:39.918+0000 7fc23dffb700 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T10:10:39.923 INFO:tasks.workunit.client.0.smithi150.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T10:10:39.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:39.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool delete foo foo 2024-09-18T10:10:39.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:39.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo foo 2024-09-18T10:10:40.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: get_monmap_and_config 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: monmap: 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:40.021232+0000 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:40.021232+0000 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:40.022 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:40.023 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:40.023 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:40.023 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 Processor -- start 2024-09-18T10:10:40.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 -- start start 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: init 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f17fc0a0120 0x7f17fc09e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): picked mon.noname-a con 0x7f17fc0a0120 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17fc0a04f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 0x7f17fc139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): picked mon.noname-c con 0x7f17fc1379b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17fc09dae0 con 0x7f17fc0a04f0 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc0a21d0 con 0x7f17fc0a0120 2024-09-18T10:10:40.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc09ddc0 con 0x7f17fc1379b0 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f1802449700 10 monclient(hunting): authenticate will time out at 922.115189s 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 0x7f17fc139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fb7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f17fc0a04f0 0x7f17fc0a0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38218/0 (socket says 172.21.15.150:38218) 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fb7fe700 1 -- 172.21.15.150:0/2170402600 learned_addr learned my addr 172.21.15.150:0/2170402600 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 0x7f17fc139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47266/0 (socket says 172.21.15.150:47266) 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 10 monclient(hunting): get_auth_request con 0x7f17fc1379b0 auth_method 0 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f18009e6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 --2- 172.21.15.150:0/2170402600 >> v2:172.21.15.142:3300/0 conn(0x7f17fc0a0120 0x7f17fc09e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): get_auth_request con 0x7f17fc0a0120 auth_method 0 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient(hunting): handle_auth_done global_id 4650 payload 274 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 -- 172.21.15.150:0/2170402600 >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 msgr2=0x7f17fc139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 --2- 172.21.15.150:0/2170402600 >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 0x7f17fc139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 -- 172.21.15.150:0/2170402600 >> v1:172.21.15.150:6789/0 conn(0x7f17fc0a04f0 legacy=0x7f17fc0a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 monclient: found mon.noname-a 2024-09-18T10:10:40.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:40.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.018+0000 7f17fbfff700 1 -- 172.21.15.150:0/2170402600 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc0a1590 con 0x7f17fc0a0120 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17fbfff700 1 -- 172.21.15.150:0/2170402600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 msgr2=0x7f17fc09e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17fbfff700 1 --2- 172.21.15.150:0/2170402600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc09e7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0a36e0 tx=0x7f17ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc159b38ecafd798 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 1 -- 172.21.15.150:0/2170402600 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00b8c0 con 0x7f17fc0a0120 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:40.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: dump: 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 5 monclient: authenticate success, global_id 4650 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 1 -- 172.21.15.150:0/2170402600 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ec00ba20 con 0x7f17fc0a0120 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 1 -- 172.21.15.150:0/2170402600 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00bdc0 con 0x7f17fc0a0120 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f17faffd700 10 monclient: dump: 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.035 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 4 set_mon_vals no callback set 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 10 monclient: shutdown 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 -- 172.21.15.150:0/2170402600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 msgr2=0x7f17fc09e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 --2- 172.21.15.150:0/2170402600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc09e7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0a36e0 tx=0x7f17ec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 -- 172.21.15.150:0/2170402600 shutdown_connections 2024-09-18T10:10:40.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 --2- 172.21.15.150:0/2170402600 >> v2:172.21.15.150:3300/0 conn(0x7f17fc1379b0 0x7f17fc139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 --2- 172.21.15.150:0/2170402600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc09e7a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 -- 172.21.15.150:0/2170402600 >> 172.21.15.150:0/2170402600 conn(0x7f17fc0a5830 msgr2=0x7f17fc0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.022+0000 7f1802449700 1 -- 172.21.15.150:0/2170402600 shutdown_connections 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 -- 172.21.15.150:0/2170402600 wait complete. 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient: monmap: 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:40.037 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 Processor -- start 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 -- start start 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient: init 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc1305d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): picked mon.noname-a con 0x7f17fc0a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 0x7f17fc12c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): picked mon.noname-c con 0x7f17fc0a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17fc1379b0 0x7f17fc12ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): picked mon.noname-b con 0x7f17fc1379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc12d9a0 con 0x7f17fc0a0120 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc1301b0 con 0x7f17fc1379b0 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc12f9d0 con 0x7f17fc0a04f0 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f1802449700 10 monclient(hunting): authenticate will time out at 922.122333s 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17fc1379b0 0x7f17fc12ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17fc1379b0 0x7f17fc12ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47274/0 (socket says 172.21.15.150:47274) 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 -- 172.21.15.150:0/448082937 learned_addr learned my addr 172.21.15.150:0/448082937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc1305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): get_auth_request con 0x7f17fc1379b0 auth_method 0 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): get_auth_request con 0x7f17fc0a0120 auth_method 0 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 0x7f17fc12c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 10 monclient(hunting): get_auth_request con 0x7f17fc0a04f0 auth_method 0 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 1 -- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 msgr2=0x7f17fc12c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:40.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 1 -- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 msgr2=0x7f17fc12c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 0x7f17fc12c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fb7fe700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 0x7f17fc12c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient(hunting): handle_auth_done global_id 14341 payload 824 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 -- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 msgr2=0x7f17fc12c4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f17fc0a04f0 0x7f17fc12c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 -- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 msgr2=0x7f17fc1305d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc1305d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 monclient: found mon.noname-b 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f18009e6700 1 -- 172.21.15.150:0/448082937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec009420 con 0x7f17fc1379b0 2024-09-18T10:10:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.026+0000 7f17fbfff700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17fc0a0120 0x7f17fc1305d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f18009e6700 1 -- 172.21.15.150:0/448082937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17fc1379b0 msgr2=0x7f17fc12ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f18009e6700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17fc1379b0 0x7f17fc12ca00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009fd0 tx=0x7f17f4009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f400eeb0 con 0x7f17fc1379b0 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: dump: 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.045 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f4014420 con 0x7f17fc1379b0 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f40147e0 con 0x7f17fc1379b0 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 5 monclient: authenticate success, global_id 14341 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17fa7fc700 4 set_mon_vals no callback set 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 10 monclient: dump: 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _renew_subs 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 1 -- 172.21.15.150:0/448082937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc12fed0 con 0x7f17fc1379b0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _renew_subs 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 1 -- 172.21.15.150:0/448082937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc12f470 con 0x7f17fc1379b0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f1802449700 1 -- 172.21.15.150:0/448082937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc12f690 con 0x7f17fc1379b0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.030+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f17f4014b90 con 0x7f17fc1379b0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.034+0000 7f17e3fff700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f17e8035ac0 0x7f17e8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.034+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f17f4048df0 con 0x7f17fc1379b0 2024-09-18T10:10:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.034+0000 7f17fbfff700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f17e8035ac0 0x7f17e8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.034+0000 7f17fbfff700 10 monclient: get_auth_request con 0x7f17e8035ac0 auth_method 0 2024-09-18T10:10:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.034+0000 7f17fbfff700 1 --2- 172.21.15.150:0/448082937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f17e8035ac0 0x7f17e8037f70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0030c0 tx=0x7f17ec016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.042+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f17f40122e0 con 0x7f17fc1379b0 2024-09-18T10:10:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.042+0000 7f17e3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.042+0000 7f17e3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:40.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.202+0000 7f1802449700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T10:10:40.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.202+0000 7f1802449700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T10:10:40.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.202+0000 7f1802449700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.202+0000 7f1802449700 1 -- 172.21.15.150:0/448082937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"} v 0) v1 -- 0x7f17fc12fcd0 con 0x7f17fc1379b0 2024-09-18T10:10:40.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.206+0000 7f17e3fff700 1 -- 172.21.15.150:0/448082937 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}]=-1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. v39) v1 ==== 290+0+0 (secure 0 0 0) 0x7f17f401bca0 con 0x7f17fc1379b0 2024-09-18T10:10:40.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.206+0000 7f17e3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo"}] 2024-09-18T10:10:40.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.206+0000 7f17e3fff700 10 monclient: _finish_command 2 = mon:1 WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T10:10:40.211 INFO:tasks.workunit.client.0.smithi150.stderr:Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool foo. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. 2024-09-18T10:10:40.214 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:40.214 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool delete foo foo --force 2024-09-18T10:10:40.214 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:40.214 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo foo --force 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: get_monmap_and_config 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: monmap: 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:40.313407+0000 2024-09-18T10:10:40.314 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:40.313407+0000 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:40.315 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 Processor -- start 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 -- start start 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: init 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f641c06b5a0 0x7f641c1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-a con 0x7f641c06b5a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f641c132700 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-b con 0x7f641c132b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c06aac0 con 0x7f641c132700 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641c06b970 con 0x7f641c06b5a0 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c09fb10 con 0x7f641c132b40 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f6421bf3700 10 monclient(hunting): authenticate will time out at 922.407005s 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641affd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f641c132700 0x7f641c09e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38230/0 (socket says 172.21.15.150:38230) 2024-09-18T10:10:40.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641affd700 1 -- 172.21.15.150:0/436171605 learned_addr learned my addr 172.21.15.150:0/436171605 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 --2- 172.21.15.150:0/436171605 >> v2:172.21.15.142:3300/0 conn(0x7f641c06b5a0 0x7f641c1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): get_auth_request con 0x7f641c06b5a0 auth_method 0 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 10 monclient(hunting): get_auth_request con 0x7f641c132b40 auth_method 0 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 1 -- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 msgr2=0x7f641c09ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 1 -- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 msgr2=0x7f641c09ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641bfff700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient(hunting): handle_auth_done global_id 4656 payload 274 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 -- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 msgr2=0x7f641c09ee80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 -- 172.21.15.150:0/436171605 >> v1:172.21.15.150:6789/0 conn(0x7f641c132700 legacy=0x7f641c09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:40.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.310+0000 7f641b7fe700 1 -- 172.21.15.150:0/436171605 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c06a7e0 con 0x7f641c06b5a0 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641b7fe700 1 -- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c06b5a0 msgr2=0x7f641c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641b7fe700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c06b5a0 0x7f641c1321c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f640c0098e0 tx=0x7f640c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c693ff0122fb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 1 -- 172.21.15.150:0/436171605 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c009e80 con 0x7f641c06b5a0 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: dump: 2024-09-18T10:10:40.328 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 5 monclient: authenticate success, global_id 4656 2024-09-18T10:10:40.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 1 -- 172.21.15.150:0/436171605 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00ba70 con 0x7f641c06b5a0 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 4 set_mon_vals no callback set 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 1 -- 172.21.15.150:0/436171605 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00bdf0 con 0x7f641c06b5a0 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f641a7fc700 10 monclient: dump: 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.330 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 10 monclient: shutdown 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 -- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c06b5a0 msgr2=0x7f641c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c06b5a0 0x7f641c1321c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f640c0098e0 tx=0x7f640c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 -- 172.21.15.150:0/436171605 shutdown_connections 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c132b40 0x7f641c09ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 --2- 172.21.15.150:0/436171605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c06b5a0 0x7f641c1321c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 -- 172.21.15.150:0/436171605 >> 172.21.15.150:0/436171605 conn(0x7f641c071b60 msgr2=0x7f641c0a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 -- 172.21.15.150:0/436171605 shutdown_connections 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.314+0000 7f6421bf3700 1 -- 172.21.15.150:0/436171605 wait complete. 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient: monmap: 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:40.332 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 Processor -- start 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 -- start start 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient: init 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 0x7f641c131980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-c con 0x7f641c06b5a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c132700 0x7f641c11b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-a con 0x7f641c132700 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:40.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c132b40 0x7f641c11b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): picked mon.noname-b con 0x7f641c132b40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c069b30 con 0x7f641c132700 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0a3270 con 0x7f641c132b40 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c11ee80 con 0x7f641c06b5a0 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f6421bf3700 10 monclient(hunting): authenticate will time out at 922.414032s 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c132b40 0x7f641c11b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c132b40 0x7f641c11b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47280/0 (socket says 172.21.15.150:47280) 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 -- 172.21.15.150:0/800462610 learned_addr learned my addr 172.21.15.150:0/800462610 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): get_auth_request con 0x7f641c132b40 auth_method 0 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c132700 0x7f641c11b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): get_auth_request con 0x7f641c132700 auth_method 0 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 0x7f641c131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 10 monclient(hunting): get_auth_request con 0x7f641c06b5a0 auth_method 0 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 1 -- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 msgr2=0x7f641c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 1 -- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 msgr2=0x7f641c131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 0x7f641c131980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641b7fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 0x7f641c131980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient(hunting): handle_auth_done global_id 14344 payload 824 2024-09-18T10:10:40.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 -- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 msgr2=0x7f641c131980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f641c06b5a0 0x7f641c131980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 -- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c132700 msgr2=0x7f641c11b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f641c132700 0x7f641c11b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 monclient: found mon.noname-b 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.318+0000 7f641bfff700 1 -- 172.21.15.150:0/800462610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009420 con 0x7f641c132b40 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f641bfff700 1 -- 172.21.15.150:0/800462610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c132b40 msgr2=0x7f641c11b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f641bfff700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c132b40 0x7f641c11b7b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6414007ef0 tx=0x7f6414009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641400ee50 con 0x7f641c132b40 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: dump: 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 5 monclient: authenticate success, global_id 14344 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _renew_subs 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 1 -- 172.21.15.150:0/800462610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c11d310 con 0x7f641c132b40 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6414014420 con 0x7f641c132b40 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64140147e0 con 0x7f641c132b40 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 10 monclient: dump: 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.339 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _renew_subs 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6419ffb700 4 set_mon_vals no callback set 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 1 -- 172.21.15.150:0/800462610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c130d90 con 0x7f641c132b40 2024-09-18T10:10:40.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6414014c50 con 0x7f641c132b40 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f6421bf3700 1 -- 172.21.15.150:0/800462610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c0a3d60 con 0x7f641c132b40 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6408035ac0 0x7f6408037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f6414048dc0 con 0x7f641c132b40 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f641b7fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6408035ac0 0x7f6408037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.322+0000 7f641b7fe700 10 monclient: get_auth_request con 0x7f6408035ac0 auth_method 0 2024-09-18T10:10:40.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.326+0000 7f641b7fe700 1 --2- 172.21.15.150:0/800462610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6408035ac0 0x7f6408037f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f640c0030c0 tx=0x7f640c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.334+0000 7f64037fe700 1 -- 172.21.15.150:0/800462610 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f641401ecb0 con 0x7f641c132b40 2024-09-18T10:10:40.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.334+0000 7f64037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.334+0000 7f64037fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:40.499 INFO:tasks.workunit.client.0.smithi150.stderr:--force not valid: --force not one of 'true', 'false' 2024-09-18T10:10:40.499 INFO:tasks.workunit.client.0.smithi150.stderr:Invalid command: unused arguments: ['--force'] 2024-09-18T10:10:40.499 INFO:tasks.workunit.client.0.smithi150.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T10:10:40.499 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: invalid command 2024-09-18T10:10:40.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:40.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-18T10:10:40.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:40.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo fooo --yes-i-really-mean-it 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: get_monmap_and_config 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: monmap: 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:40.593450+0000 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:40.593450+0000 2024-09-18T10:10:40.594 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:40.595 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 Processor -- start 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 -- start start 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: init 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f473c09bcb0 0x7f473c09d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-c con 0x7f473c09bcb0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:40.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c067b10 0x7f473c067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-b con 0x7f473c067b10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f473c068420 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c09e3b0 con 0x7f473c068420 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c09ec50 con 0x7f473c09bcb0 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c09daa0 con 0x7f473c067b10 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47428b5700 10 monclient(hunting): authenticate will time out at 922.687735s 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f473c09bcb0 0x7f473c09d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f473c09bcb0 0x7f473c09d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47288/0 (socket says 172.21.15.150:47288) 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 1 -- 172.21.15.150:0/2628498122 learned_addr learned my addr 172.21.15.150:0/2628498122 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): get_auth_request con 0x7f473c09bcb0 auth_method 0 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.590+0000 7f47410b2700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c067b10 0x7f473c067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47410b2700 10 monclient(hunting): get_auth_request con 0x7f473c067b10 auth_method 0 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47410b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47410b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47410b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 10 monclient(hunting): handle_auth_done global_id 14347 payload 274 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 -- 172.21.15.150:0/2628498122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c067b10 msgr2=0x7f473c067ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c067b10 0x7f473c067ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 -- 172.21.15.150:0/2628498122 >> v1:172.21.15.142:6789/0 conn(0x7f473c068420 legacy=0x7f473c1365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 monclient: found mon.noname-c 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 -- 172.21.15.150:0/2628498122 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0688c0 con 0x7f473c09bcb0 2024-09-18T10:10:40.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 -- 172.21.15.150:0/2628498122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 msgr2=0x7f473c09d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47418b3700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c09d020 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f47380099b0 tx=0x7f4738009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7773b3593c3a6725 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 1 -- 172.21.15.150:0/2628498122 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473800ba70 con 0x7f473c09bcb0 2024-09-18T10:10:40.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: dump: 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 5 monclient: authenticate success, global_id 14347 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 1 -- 172.21.15.150:0/2628498122 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473800bbd0 con 0x7f473c09bcb0 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 4 set_mon_vals no callback set 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 1 -- 172.21.15.150:0/2628498122 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738011420 con 0x7f473c09bcb0 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47408b1700 10 monclient: dump: 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.609 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 10 monclient: shutdown 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 -- 172.21.15.150:0/2628498122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 msgr2=0x7f473c09d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c09d020 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f47380099b0 tx=0x7f4738009250 comp rx=0 tx=0).stop 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 -- 172.21.15.150:0/2628498122 shutdown_connections 2024-09-18T10:10:40.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c067b10 0x7f473c067ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 --2- 172.21.15.150:0/2628498122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c09d020 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 -- 172.21.15.150:0/2628498122 >> 172.21.15.150:0/2628498122 conn(0x7f473c0a2420 msgr2=0x7f473c0a2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 -- 172.21.15.150:0/2628498122 shutdown_connections 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.594+0000 7f47428b5700 1 -- 172.21.15.150:0/2628498122 wait complete. 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient: monmap: 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 Processor -- start 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 -- start start 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient: init 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f473c067b10 0x7f473c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-a con 0x7f473c067b10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:40.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 0x7f473c095500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-c con 0x7f473c068420 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c08c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): picked mon.noname-b con 0x7f473c09bcb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c09ea20 con 0x7f473c067b10 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0a18e0 con 0x7f473c09bcb0 2024-09-18T10:10:40.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c140570 con 0x7f473c068420 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47428b5700 10 monclient(hunting): authenticate will time out at 922.693528s 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c08c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c08c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47300/0 (socket says 172.21.15.150:47300) 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 -- 172.21.15.150:0/87349370 learned_addr learned my addr 172.21.15.150:0/87349370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): get_auth_request con 0x7f473c09bcb0 auth_method 0 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f473c067b10 0x7f473c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 0x7f473c095500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): get_auth_request con 0x7f473c067b10 auth_method 0 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 10 monclient(hunting): get_auth_request con 0x7f473c068420 auth_method 0 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47418b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 1 -- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 msgr2=0x7f473c095500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 1 -- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 msgr2=0x7f473c095500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 0x7f473c095500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47410b2700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 0x7f473c095500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient(hunting): handle_auth_done global_id 14350 payload 824 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 -- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 msgr2=0x7f473c095500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:40.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f473c068420 0x7f473c095500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 -- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f473c067b10 msgr2=0x7f473c13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f473c067b10 0x7f473c13f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 monclient: found mon.noname-b 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.598+0000 7f47420b4700 1 -- 172.21.15.150:0/87349370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47380094f0 con 0x7f473c09bcb0 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47420b4700 1 -- 172.21.15.150:0/87349370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 msgr2=0x7f473c08c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47420b4700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f473c09bcb0 0x7f473c08c0d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f473400c930 tx=0x7f47340026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734012840 con 0x7f473c09bcb0 2024-09-18T10:10:40.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: dump: 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.618 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 5 monclient: authenticate success, global_id 14350 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47340129c0 con 0x7f473c09bcb0 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _renew_subs 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 1 -- 172.21.15.150:0/87349370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c08dc30 con 0x7f473c09bcb0 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734012d80 con 0x7f473c09bcb0 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 10 monclient: dump: 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.620 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4733fff700 4 set_mon_vals no callback set 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _renew_subs 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 1 -- 172.21.15.150:0/87349370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c08f7a0 con 0x7f473c09bcb0 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f473400a5c0 con 0x7f473c09bcb0 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47428b5700 1 -- 172.21.15.150:0/87349370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c08fe60 con 0x7f473c09bcb0 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4720035ac0 0x7f4720037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f4734048470 con 0x7f473c09bcb0 2024-09-18T10:10:40.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47418b3700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4720035ac0 0x7f4720037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.602+0000 7f47418b3700 10 monclient: get_auth_request con 0x7f4720035ac0 auth_method 0 2024-09-18T10:10:40.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.606+0000 7f47418b3700 1 --2- 172.21.15.150:0/87349370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4720035ac0 0x7f4720037f70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f47380099b0 tx=0x7f4738016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.610+0000 7f4731ffb700 1 -- 172.21.15.150:0/87349370 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f473c08fe60 con 0x7f473c09bcb0 2024-09-18T10:10:40.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.610+0000 7f4731ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.610+0000 7f4731ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:40.777 INFO:tasks.workunit.client.0.smithi150.stderr:--yes-i-really-mean-it not valid: --yes-i-really-mean-it not one of 'true', 'false' 2024-09-18T10:10:40.778 INFO:tasks.workunit.client.0.smithi150.stderr:Invalid command: unused arguments: ['--yes-i-really-mean-it'] 2024-09-18T10:10:40.778 INFO:tasks.workunit.client.0.smithi150.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T10:10:40.778 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: invalid command 2024-09-18T10:10:40.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:40.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-18T10:10:40.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:10:40.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo --yes-i-really-mean-it foo 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 10 monclient: get_monmap_and_config 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 10 monclient: monmap: 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:40.878473+0000 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:40.878473+0000 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.884 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 1 Processor -- start 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 1 -- start start 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.874+0000 7fbfad9dd700 10 monclient: init 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfa80a0290 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-b con 0x7fbfa80a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbfa8137a20 0x7fbfa8139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-a con 0x7fbfa8137a20 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa809e360 con 0x7fbfa80a0290 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa809ef80 con 0x7fbfa8137a20 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa80a1fb0 con 0x7fbfa80a0d70 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfad9dd700 10 monclient(hunting): authenticate will time out at 922.972440s 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbfa8137a20 0x7fbfa8139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:48232/0 (socket says 172.21.15.150:48232) 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 -- 172.21.15.150:0/735563488 learned_addr learned my addr 172.21.15.150:0/735563488 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 10 monclient(hunting): get_auth_request con 0x7fbfa80a0d70 auth_method 0 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): get_auth_request con 0x7fbfa8137a20 auth_method 0 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 -- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 msgr2=0x7fbfa809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 -- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 msgr2=0x7fbfa809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa67fc700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_done global_id 4665 payload 274 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 -- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 msgr2=0x7fbfa809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 -- 172.21.15.150:0/735563488 >> v1:172.21.15.142:6789/0 conn(0x7fbfa80a0290 legacy=0x7fbfa80a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:40.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.878+0000 7fbfa77fe700 1 -- 172.21.15.150:0/735563488 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa809da50 con 0x7fbfa8137a20 2024-09-18T10:10:40.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa77fe700 1 -- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa8137a20 msgr2=0x7fbfa8139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa77fe700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa8137a20 0x7fbfa8139e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009e60 tx=0x7fbfa0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d623afa829504c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 1 -- 172.21.15.150:0/735563488 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa000e070 con 0x7fbfa8137a20 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: dump: 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.896 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 1 -- 172.21.15.150:0/735563488 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa000be40 con 0x7fbfa8137a20 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 1 -- 172.21.15.150:0/735563488 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0012660 con 0x7fbfa8137a20 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfa5ffb700 10 monclient: dump: 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.897 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 5 monclient: authenticate success, global_id 4665 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 4 set_mon_vals no callback set 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: shutdown 2024-09-18T10:10:40.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa8137a20 msgr2=0x7fbfa8139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa8137a20 0x7fbfa8139e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009e60 tx=0x7fbfa0009780 comp rx=0 tx=0).stop 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- 172.21.15.150:0/735563488 shutdown_connections 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0d70 0x7fbfa809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- 172.21.15.150:0/735563488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa8137a20 0x7fbfa8139e00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- 172.21.15.150:0/735563488 >> 172.21.15.150:0/735563488 conn(0x7fbfa80a58a0 msgr2=0x7fbfa80a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- 172.21.15.150:0/735563488 shutdown_connections 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- 172.21.15.150:0/735563488 wait complete. 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: build_initial_monmap 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: monmap: 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:40.899 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 Processor -- start 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- start start 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: init 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:40.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 0x7fbfa812b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-c con 0x7fbfa80a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa80a0d70 0x7fbfa812b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-a con 0x7fbfa80a0d70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfa8137a20 0x7fbfa812bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): picked mon.noname-b con 0x7fbfa8137a20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa809d3f0 con 0x7fbfa80a0d70 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa809bce0 con 0x7fbfa8137a20 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa812e4a0 con 0x7fbfa80a0290 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:40.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.882+0000 7fbfad9dd700 10 monclient(hunting): authenticate will time out at 922.979934s 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfa8137a20 0x7fbfa812bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfa8137a20 0x7fbfa812bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47306/0 (socket says 172.21.15.150:47306) 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 -- 172.21.15.150:0/3938338589 learned_addr learned my addr 172.21.15.150:0/3938338589 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): get_auth_request con 0x7fbfa8137a20 auth_method 0 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa80a0d70 0x7fbfa812b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): get_auth_request con 0x7fbfa80a0d70 auth_method 0 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 0x7fbfa812b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 10 monclient(hunting): get_auth_request con 0x7fbfa80a0290 auth_method 0 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient(hunting): handle_auth_done global_id 14353 payload 824 2024-09-18T10:10:40.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 -- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 msgr2=0x7fbfa812b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 1 -- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 msgr2=0x7fbfa812b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 1 -- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 msgr2=0x7fbfa812b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 0x7fbfa812b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa6ffd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 0x7fbfa812b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfa80a0290 0x7fbfa812b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 -- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa80a0d70 msgr2=0x7fbfa812b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa80a0d70 0x7fbfa812b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 -- 172.21.15.150:0/3938338589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0009a20 con 0x7fbfa8137a20 2024-09-18T10:10:40.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 -- 172.21.15.150:0/3938338589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfa8137a20 msgr2=0x7fbfa812bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa77fe700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfa8137a20 0x7fbfa812bc70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0003000 tx=0x7fbfa0018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0026430 con 0x7fbfa8137a20 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: dump: 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.905 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: _finish_auth 0 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 5 monclient: authenticate success, global_id 14353 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa0021070 con 0x7fbfa8137a20 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0026800 con 0x7fbfa8137a20 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 10 monclient: _renew_subs 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 1 -- 172.21.15.150:0/3938338589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa812cd30 con 0x7fbfa8137a20 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfa57fa700 4 set_mon_vals no callback set 2024-09-18T10:10:40.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbf8effd700 10 monclient: dump: 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:40.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 10 monclient: _renew_subs 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.886+0000 7fbfad9dd700 1 -- 172.21.15.150:0/3938338589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa812d900 con 0x7fbfa8137a20 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfad9dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfad9dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfad9dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfad9dd700 1 -- 172.21.15.150:0/3938338589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa812ed00 con 0x7fbfa8137a20 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbfa0023030 con 0x7fbfa8137a20 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbf8effd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf94035af0 0x7fbf94037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7fbfa004ac40 con 0x7fbfa8137a20 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfa6ffd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf94035af0 0x7fbf94037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:40.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.890+0000 7fbfa6ffd700 10 monclient: get_auth_request con 0x7fbf94035af0 auth_method 0 2024-09-18T10:10:40.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.894+0000 7fbfa6ffd700 1 --2- 172.21.15.150:0/3938338589 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf94035af0 0x7fbf94037fa0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbf98006df0 tx=0x7fbf98008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:40.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.898+0000 7fbf8effd700 1 -- 172.21.15.150:0/3938338589 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbfa0011b70 con 0x7fbfa8137a20 2024-09-18T10:10:40.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.898+0000 7fbf8effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:40.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:40.898+0000 7fbf8effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:41.065 INFO:tasks.workunit.client.0.smithi150.stderr:foo not valid: foo not one of 'true', 'false' 2024-09-18T10:10:41.066 INFO:tasks.workunit.client.0.smithi150.stderr:Invalid command: unused arguments: ['foo'] 2024-09-18T10:10:41.066 INFO:tasks.workunit.client.0.smithi150.stderr:osd pool delete [] [--yes-i-really-really-mean-it] [--yes-i-really-really-mean-it-not-faking] : delete pool 2024-09-18T10:10:41.066 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: invalid command 2024-09-18T10:10:41.068 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:10:41.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foooo foooo --yes-i-really-really-mean-it 2024-09-18T10:10:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 10 monclient: get_monmap_and_config 2024-09-18T10:10:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 10 monclient: build_initial_monmap 2024-09-18T10:10:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 10 monclient: monmap: 2024-09-18T10:10:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:41.170660+0000 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:41.170660+0000 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:41.172 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 1 Processor -- start 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 1 -- start start 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.166+0000 7f878f411700 10 monclient: init 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87880989e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f8788099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): picked mon.noname-b con 0x7f878809c000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87881338c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8788097cd0 con 0x7f87880989e0 2024-09-18T10:10:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87880979f0 con 0x7f87881338c0 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8788096d40 con 0x7f878809c000 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878f411700 10 monclient(hunting): authenticate will time out at 923.265391s 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878d9ae700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f87881338c0 0x7f8788135ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38242/0 (socket says 172.21.15.150:38242) 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878d9ae700 1 -- 172.21.15.150:0/512181339 learned_addr learned my addr 172.21.15.150:0/512181339 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f8788099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 10 monclient(hunting): get_auth_request con 0x7f878809c000 auth_method 0 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 msgr2=0x7f8788099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 msgr2=0x7f8788099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f8788099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878c9ac700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f8788099320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878d9ae700 1 -- 172.21.15.150:0/512181339 >> v1:172.21.15.150:6789/0 conn(0x7f87881338c0 legacy=0x7f8788135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f878d1ad700 1 -- 172.21.15.150:0/512181339 >> v1:172.21.15.142:6789/0 conn(0x7f87880989e0 legacy=0x7f878809b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.170+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880367680 0 0) 0x7f87880979f0 con 0x7f87881338c0 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): my global_id is 14356 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8774003570 con 0x7f87881338c0 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590626209 0 0) 0x7f8788097cd0 con 0x7f87880989e0 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): my global_id is 4671 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87880979f0 con 0x7f87880989e0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892304562 0 0) 0x7f87880979f0 con 0x7f87880989e0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 msgr2=0x7f8788099320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f8788099320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 >> v1:172.21.15.150:6789/0 conn(0x7f87881338c0 legacy=0x7f8788135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788097fb0 con 0x7f87880989e0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: _finish_auth 0 2024-09-18T10:10:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f8784001890 con 0x7f87880989e0 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: dump: 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.184 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 >> v1:172.21.15.142:6789/0 conn(0x7f87880989e0 legacy=0x7f878809b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 0x7f87740040f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): picked mon.b con 0x7f8774003d20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8774004810 0x7f8774004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): picked mon.a con 0x7f8774004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f877400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): picked mon.c con 0x7f878809c000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780001960 con 0x7f8774004810 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740056b0 con 0x7f8774003d20 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877400bff0 con 0x7f878809c000 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f877ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 0x7f87740040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): get_auth_request con 0x7f8774003d20 auth_method 0 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8774004810 0x7f8774004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): get_auth_request con 0x7f8774004810 auth_method 0 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d9ae700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f877400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d9ae700 10 monclient(hunting): get_auth_request con 0x7f878809c000 auth_method 0 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d9ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d9ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d9ae700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient(hunting): handle_auth_done global_id 4671 payload 293 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 msgr2=0x7f877400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878809c000 0x7f877400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8774004810 msgr2=0x7f8774004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8774004810 0x7f8774004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient: _finish_hunting 0 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 monclient: found mon.b 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 1 -- 172.21.15.150:0/512181339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880979f0 con 0x7f8774003d20 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 10 monclient: _finish_auth 0 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878d1ad700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:41.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878f411700 5 monclient: authenticate success, global_id 4671 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.174+0000 7f878f411700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 msgr2=0x7f87740040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 0x7f87740040f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f878400d500 tx=0x7f8784008aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=899ae4248986f8be server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784008db0 con 0x7f8774003d20 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 10 monclient: dump: 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.189 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878400ddf0 con 0x7f8774003d20 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 4 set_mon_vals no callback set 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 0x7f87740040f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f878400d500 tx=0x7f8784008aa0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f8784013650 mon_map magic: 0 v1 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: shutdown 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:41.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 msgr2=0x7f87740040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 --2- 172.21.15.150:0/512181339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774003d20 0x7f87740040f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f878400d500 tx=0x7f8784008aa0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 1 -- 172.21.15.150:0/512181339 reap_dead start 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 1 -- 172.21.15.150:0/512181339 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f8784013650 con 0x7f8774003d20 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f877ffff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- 172.21.15.150:0/512181339 shutdown_connections 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- 172.21.15.150:0/512181339 >> 172.21.15.150:0/512181339 conn(0x7f8788068590 msgr2=0x7f87880a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- 172.21.15.150:0/512181339 shutdown_connections 2024-09-18T10:10:41.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- 172.21.15.150:0/512181339 wait complete. 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: build_initial_monmap 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: monmap: 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:41.196 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 Processor -- start 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- start start 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: init 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): picked mon.noname-c con 0x7f878811ae00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:41.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 0x7f878811bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): picked mon.noname-a con 0x7f878811b710 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): picked mon.noname-b con 0x7f8788145d90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8788132bb0 con 0x7f878811b710 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878811c0c0 con 0x7f8788145d90 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8788122bc0 con 0x7f878811ae00 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878f411700 10 monclient(hunting): authenticate will time out at 923.275578s 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 0x7f878811bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47334/0 (socket says 172.21.15.150:47334) 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 0x7f878811bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:34770/0 (socket says 172.21.15.150:34770) 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 1 -- 172.21.15.150:0/2408027793 learned_addr learned my addr 172.21.15.150:0/2408027793 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 10 monclient(hunting): get_auth_request con 0x7f8788145d90 auth_method 0 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d9ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 10 monclient(hunting): get_auth_request con 0x7f878811b710 auth_method 0 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878c9ac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 10 monclient(hunting): get_auth_request con 0x7f878811ae00 auth_method 0 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.178+0000 7f878d1ad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d1ad700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 msgr2=0x7f878811b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d1ad700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 msgr2=0x7f878811b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d1ad700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878c9ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:41.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d1ad700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient(hunting): handle_auth_done global_id 14359 payload 824 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 msgr2=0x7f878811b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 msgr2=0x7f878811bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 0x7f878811bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient: _finish_hunting 0 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 monclient: found mon.noname-b 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788122e10 con 0x7f8788145d90 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 msgr2=0x7f8788148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878d9ae700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8788067360 tx=0x7f8780004bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780017070 con 0x7f8788145d90 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: dump: 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.202 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 5 monclient: authenticate success, global_id 14359 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8780007b10 con 0x7f8788145d90 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 10 monclient: _renew_subs 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8788121430 con 0x7f8788145d90 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800127c0 con 0x7f8788145d90 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:41.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877d7fa700 10 monclient: dump: 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f877effd700 4 set_mon_vals no callback set 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 10 monclient: _renew_subs 2024-09-18T10:10:41.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.182+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878811c6c0 con 0x7f8788145d90 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8780012b30 con 0x7f8788145d90 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878f411700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878f411700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878f411700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8788133630 con 0x7f8788145d90 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f877d7fa700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 0x7f876c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5470+0+0 (secure 0 0 0) 0x7f8780048780 con 0x7f8788145d90 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878d1ad700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 0x7f876c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.186+0000 7f878d1ad700 10 monclient: get_auth_request con 0x7f876c035ac0 auth_method 0 2024-09-18T10:10:41.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.190+0000 7f878d1ad700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 0x7f876c037f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f878804d2e0 tx=0x7f8784018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.198+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f87800198c0 con 0x7f8788145d90 2024-09-18T10:10:41.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.198+0000 7f877d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.198+0000 7f877d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:41.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.362+0000 7f878f411700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.362+0000 7f878f411700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.362+0000 7f878f411700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.362+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8788098db0 con 0x7f8788145d90 2024-09-18T10:10:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.642+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' removed v40) v1 ==== 155+0+0 (secure 0 0 0) 0x7f87800112e0 con 0x7f8788145d90 2024-09-18T10:10:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.642+0000 7f877d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.642+0000 7f877d7fa700 10 monclient: _finish_command 2 = system:0 pool 'foooo' removed 2024-09-18T10:10:41.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f878f411700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f878f411700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f878f411700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8788096ab0 con 0x7f8788145d90 2024-09-18T10:10:41.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f877d7fa700 1 -- 172.21.15.150:0/2408027793 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}]=0 pool 'foooo' does not exist v40) v1 ==== 162+0+0 (secure 0 0 0) 0x7f87800192c0 con 0x7f8788145d90 2024-09-18T10:10:41.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f877d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foooo", "pool2": "foooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:41.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.754+0000 7f877d7fa700 10 monclient: _finish_command 3 = system:0 pool 'foooo' does not exist 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 msgr2=0x7f876c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 0x7f876c037f70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f878804d2e0 tx=0x7f8784018000 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 10 monclient: shutdown 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 msgr2=0x7f8788148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8788067360 tx=0x7f8780004bd0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 shutdown_connections 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f876c035ac0 0x7f876c037f70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8788145d90 0x7f8788148170 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f878811b710 0x7f878811bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 --2- 172.21.15.150:0/2408027793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f878811ae00 0x7f878811b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 >> 172.21.15.150:0/2408027793 conn(0x7f8788068590 msgr2=0x7f87880a0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:41.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 shutdown_connections 2024-09-18T10:10:41.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.758+0000 7f878f411700 1 -- 172.21.15.150:0/2408027793 wait complete. 2024-09-18T10:10:41.764 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foooo' does not exist 2024-09-18T10:10:41.771 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: get_monmap_and_config 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: build_initial_monmap 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: monmap: 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:41.870223+0000 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:41.870223+0000 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.871 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:41.872 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:41.872 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:41.872 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:41.872 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:41.872 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 Processor -- start 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 -- start start 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: init 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1a3809d000 0x7f1a380a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-a con 0x7f1a3809d000 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a3809e780 0x7f1a380a0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-c con 0x7f1a3809e780 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a380a0ba0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a380699a0 con 0x7f1a380a0ba0 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380a0ff0 con 0x7f1a3809d000 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380a1640 con 0x7f1a3809e780 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3f043700 10 monclient(hunting): authenticate will time out at 923.963771s 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3d5e0700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1a380a0ba0 0x7f1a381389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38246/0 (socket says 172.21.15.150:38246) 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.866+0000 7f1a3d5e0700 1 -- 172.21.15.150:0/3175410842 learned_addr learned my addr 172.21.15.150:0/3175410842 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a37fff700 1 --2- 172.21.15.150:0/3175410842 >> v2:172.21.15.150:3300/0 conn(0x7f1a3809e780 0x7f1a380a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/3175410842 >> v2:172.21.15.142:3300/0 conn(0x7f1a3809d000 0x7f1a380a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): get_auth_request con 0x7f1a3809d000 auth_method 0 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a37fff700 10 monclient(hunting): get_auth_request con 0x7f1a3809e780 auth_method 0 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a37fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:41.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_done global_id 4677 payload 274 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 -- 172.21.15.150:0/3175410842 >> v2:172.21.15.150:3300/0 conn(0x7f1a3809e780 msgr2=0x7f1a380a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/3175410842 >> v2:172.21.15.150:3300/0 conn(0x7f1a3809e780 0x7f1a380a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 -- 172.21.15.150:0/3175410842 >> v1:172.21.15.150:6789/0 conn(0x7f1a380a0ba0 legacy=0x7f1a381389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient: _finish_hunting 0 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 monclient: found mon.noname-a 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:41.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 -- 172.21.15.150:0/3175410842 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3809e3d0 con 0x7f1a3809d000 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 -- 172.21.15.150:0/3175410842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809d000 msgr2=0x7f1a380a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/3175410842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809d000 0x7f1a380a0120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1a380698a0 tx=0x7f1a2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547b90f5a6e6e4b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 1 -- 172.21.15.150:0/3175410842 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00b8c0 con 0x7f1a3809d000 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:41.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: dump: 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:41.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 1 -- 172.21.15.150:0/3175410842 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c00ba20 con 0x7f1a3809d000 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 1 -- 172.21.15.150:0/3175410842 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00bdc0 con 0x7f1a3809d000 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a377fe700 10 monclient: dump: 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.883 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3f043700 5 monclient: authenticate success, global_id 4677 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3f043700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.870+0000 7f1a3f043700 4 set_mon_vals no callback set 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: shutdown 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- 172.21.15.150:0/3175410842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809d000 msgr2=0x7f1a380a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- 172.21.15.150:0/3175410842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809d000 0x7f1a380a0120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1a380698a0 tx=0x7f1a2c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- 172.21.15.150:0/3175410842 shutdown_connections 2024-09-18T10:10:41.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- 172.21.15.150:0/3175410842 >> v2:172.21.15.150:3300/0 conn(0x7f1a3809e780 0x7f1a380a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- 172.21.15.150:0/3175410842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809d000 0x7f1a380a0120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- 172.21.15.150:0/3175410842 >> 172.21.15.150:0/3175410842 conn(0x7f1a380a5830 msgr2=0x7f1a380a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- 172.21.15.150:0/3175410842 shutdown_connections 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- 172.21.15.150:0/3175410842 wait complete. 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: build_initial_monmap 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: monmap: 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.885 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 Processor -- start 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- start start 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: init 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-b con 0x7f1a3809d000 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 0x7f1a381421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-a con 0x7f1a3809e780 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:41.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a380a0ba0 0x7f1a3808df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): picked mon.noname-c con 0x7f1a380a0ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3809c270 con 0x7f1a3809e780 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38091b40 con 0x7f1a3809d000 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38092780 con 0x7f1a380a0ba0 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3f043700 10 monclient(hunting): authenticate will time out at 923.970847s 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47350/0 (socket says 172.21.15.150:47350) 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 -- 172.21.15.150:0/2226794341 learned_addr learned my addr 172.21.15.150:0/2226794341 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): get_auth_request con 0x7f1a3809d000 auth_method 0 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 0x7f1a381421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3d5e0700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a380a0ba0 0x7f1a3808df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 10 monclient(hunting): get_auth_request con 0x7f1a3809e780 auth_method 0 2024-09-18T10:10:41.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3d5e0700 10 monclient(hunting): get_auth_request con 0x7f1a380a0ba0 auth_method 0 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3d5e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3d5e0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3d5e0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient(hunting): handle_auth_done global_id 14365 payload 824 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 -- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a380a0ba0 msgr2=0x7f1a3808df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a380a0ba0 0x7f1a3808df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 -- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 msgr2=0x7f1a381421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 0x7f1a381421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:41.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient: _finish_hunting 0 2024-09-18T10:10:41.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 monclient: found mon.noname-b 2024-09-18T10:10:41.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:41.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a3cddf700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c009420 con 0x7f1a3809d000 2024-09-18T10:10:41.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.874+0000 7f1a37fff700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 0x7f1a381421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:41.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3cddf700 1 -- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 msgr2=0x7f1a38141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:41.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3cddf700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1a3806aa60 tx=0x7f1a2c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c006af0 con 0x7f1a3809d000 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: dump: 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.893 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c020070 con 0x7f1a3809d000 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 5 monclient: authenticate success, global_id 14365 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _renew_subs 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a38099490 con 0x7f1a3809d000 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c01b6a0 con 0x7f1a3809d000 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a36ffd700 4 set_mon_vals no callback set 2024-09-18T10:10:41.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 10 monclient: dump: 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _renew_subs 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a38091630 con 0x7f1a3809d000 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1a2c022030 con 0x7f1a3809d000 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.878+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.882+0000 7f1a34ff9700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 0x7f1a1c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.882+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f1a2c04a2c0 con 0x7f1a3809d000 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.882+0000 7f1a37fff700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 0x7f1a1c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.882+0000 7f1a37fff700 10 monclient: get_auth_request con 0x7f1a1c035ac0 auth_method 0 2024-09-18T10:10:41.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.882+0000 7f1a37fff700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 0x7f1a1c037f70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:41.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.890+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:41.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.890+0000 7f1a34ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:41.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:41.890+0000 7f1a34ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:42.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.050+0000 7f1a3f043700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.050+0000 7f1a3f043700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.050+0000 7f1a3f043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:42.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.050+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:42.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:42.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: get_monmap_and_config 2024-09-18T10:10:42.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: monmap: 2024-09-18T10:10:42.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:42.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:42.425404+0000 2024-09-18T10:10:42.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:42.425404+0000 2024-09-18T10:10:42.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.427 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:42.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 Processor -- start 2024-09-18T10:10:42.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 -- start start 2024-09-18T10:10:42.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: init 2024-09-18T10:10:42.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T10:10:42.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7f8009d920 0x7f7f8006f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-a con 0x7f7f8009d920 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:42.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 0x7f7f80096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-c con 0x7f7f800953c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:42.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f80095800 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:42.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f8009b0b0 con 0x7f7f80095800 2024-09-18T10:10:42.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8006fba0 con 0x7f7f8009d920 2024-09-18T10:10:42.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f8009a7e0 con 0x7f7f800953c0 2024-09-18T10:10:42.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f863b0700 10 monclient(hunting): authenticate will time out at 931.351431s 2024-09-18T10:10:42.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f85baf700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7f80095800 0x7f7f800971f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60330/0 (socket says 172.21.15.142:60330) 2024-09-18T10:10:42.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7f8009d920 0x7f7f8006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f85baf700 1 -- 172.21.15.142:0/2373541604 learned_addr learned my addr 172.21.15.142:0/2373541604 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:42.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): get_auth_request con 0x7f7f8009d920 auth_method 0 2024-09-18T10:10:42.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f84bad700 1 --2- 172.21.15.142:0/2373541604 >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 0x7f7f80096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f853ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f84bad700 10 monclient(hunting): get_auth_request con 0x7f7f800953c0 auth_method 0 2024-09-18T10:10:42.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f84bad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f84bad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.423+0000 7f7f84bad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 10 monclient(hunting): handle_auth_done global_id 4683 payload 274 2024-09-18T10:10:42.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 -- 172.21.15.142:0/2373541604 >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 msgr2=0x7f7f80096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:42.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2373541604 >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 0x7f7f80096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 -- 172.21.15.142:0/2373541604 >> v1:172.21.15.142:6789/0 conn(0x7f7f80095800 legacy=0x7f7f800971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:42.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 monclient: found mon.noname-a 2024-09-18T10:10:42.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 -- 172.21.15.142:0/2373541604 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f80097900 con 0x7f7f8009d920 2024-09-18T10:10:42.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f84bad700 1 --2- 172.21.15.142:0/2373541604 >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 0x7f7f80096cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 -- 172.21.15.142:0/2373541604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f8009d920 msgr2=0x7f7f8006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2373541604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f8009d920 0x7f7f8006f660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f80065250 tx=0x7f7f7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf00f50933ad825c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 1 -- 172.21.15.142:0/2373541604 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00b8c0 con 0x7f7f8009d920 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:42.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: dump: 2024-09-18T10:10:42.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.456 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.457 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.457 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.457 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.457 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.457 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 5 monclient: authenticate success, global_id 4683 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 1 -- 172.21.15.142:0/2373541604 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c00ba20 con 0x7f7f8009d920 2024-09-18T10:10:42.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:42.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 4 set_mon_vals no callback set 2024-09-18T10:10:42.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 1 -- 172.21.15.142:0/2373541604 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00bdc0 con 0x7f7f8009d920 2024-09-18T10:10:42.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f77fff700 10 monclient: dump: 2024-09-18T10:10:42.460 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.460 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.460 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.462 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 10 monclient: shutdown 2024-09-18T10:10:42.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 -- 172.21.15.142:0/2373541604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f8009d920 msgr2=0x7f7f8006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2373541604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f8009d920 0x7f7f8006f660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f80065250 tx=0x7f7f7c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 -- 172.21.15.142:0/2373541604 shutdown_connections 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2373541604 >> v2:172.21.15.150:3300/0 conn(0x7f7f800953c0 0x7f7f80096cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2373541604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f8009d920 0x7f7f8006f660 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 -- 172.21.15.142:0/2373541604 >> 172.21.15.142:0/2373541604 conn(0x7f7f800a1fe0 msgr2=0x7f7f80070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.427+0000 7f7f863b0700 1 -- 172.21.15.142:0/2373541604 shutdown_connections 2024-09-18T10:10:42.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 -- 172.21.15.142:0/2373541604 wait complete. 2024-09-18T10:10:42.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient: monmap: 2024-09-18T10:10:42.464 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:42.464 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:42.464 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:42.465 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:42.465 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.465 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.465 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:42.465 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 Processor -- start 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 -- start start 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient: init 2024-09-18T10:10:42.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-c con 0x7f7f800953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-a con 0x7f7f80095800 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:42.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f8009d920 0x7f7f80136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): picked mon.noname-b con 0x7f7f8009d920 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:42.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008c880 con 0x7f7f80095800 2024-09-18T10:10:42.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008e0a0 con 0x7f7f8009d920 2024-09-18T10:10:42.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8008c370 con 0x7f7f800953c0 2024-09-18T10:10:42.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f863b0700 10 monclient(hunting): authenticate will time out at 931.359561s 2024-09-18T10:10:42.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55528/0 (socket says 172.21.15.142:55528) 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36196/0 (socket says 172.21.15.142:36196) 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 1 -- 172.21.15.142:0/2031074108 learned_addr learned my addr 172.21.15.142:0/2031074108 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 10 monclient(hunting): get_auth_request con 0x7f7f800953c0 auth_method 0 2024-09-18T10:10:42.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f853ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 10 monclient(hunting): get_auth_request con 0x7f7f80095800 auth_method 0 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f84bad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f85baf700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f8009d920 0x7f7f80136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f85baf700 10 monclient(hunting): get_auth_request con 0x7f7f8009d920 auth_method 0 2024-09-18T10:10:42.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f85baf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f85baf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.431+0000 7f7f85baf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f853ae700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 msgr2=0x7f7f801355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f853ae700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 msgr2=0x7f7f801355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:42.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f85baf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f85baf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f85baf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient(hunting): handle_auth_done global_id 4686 payload 824 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 msgr2=0x7f7f801355b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f8009d920 msgr2=0x7f7f80136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f8009d920 0x7f7f80136030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 monclient: found mon.noname-a 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 -- 172.21.15.142:0/2031074108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c009420 con 0x7f7f80095800 2024-09-18T10:10:42.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 msgr2=0x7f7f80135af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f84bad700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f70006fd0 tx=0x7f7f70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70013620 con 0x7f7f80095800 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: dump: 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.476 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f70013780 con 0x7f7f80095800 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 5 monclient: authenticate success, global_id 4686 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 10 monclient: _renew_subs 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f80094ec0 con 0x7f7f80095800 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70013b20 con 0x7f7f80095800 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f757fa700 10 monclient: dump: 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 10 monclient: _renew_subs 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f777fe700 4 set_mon_vals no callback set 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.435+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8008ced0 con 0x7f7f80095800 2024-09-18T10:10:42.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f863b0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f863b0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f863b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8008e480 con 0x7f7f80095800 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7f70012370 con 0x7f7f80095800 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f757fa700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 0x7f7f64037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5126+0+0 (secure 0 0 0) 0x7f7f700480b0 con 0x7f7f80095800 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 0x7f7f64037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.439+0000 7f7f853ae700 10 monclient: get_auth_request con 0x7f7f64035ac0 auth_method 0 2024-09-18T10:10:42.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.443+0000 7f7f853ae700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 0x7f7f64037f70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0093a0 tx=0x7f7f7c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.447+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7f70078670 con 0x7f7f80095800 2024-09-18T10:10:42.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.447+0000 7f7f757fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.447+0000 7f7f757fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:42.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f863b0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:42.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f863b0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:42.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f863b0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7f80094e60 con 0x7f7f80095800 2024-09-18T10:10:42.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f757fa700 1 -- 172.21.15.142:0/2031074108 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7f70020030 con 0x7f7f80095800 2024-09-18T10:10:42.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f757fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:42.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.683+0000 7f7f757fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:42.688 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:42.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 msgr2=0x7f7f64037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 0x7f7f64037f70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c0093a0 tx=0x7f7f7c016000 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 10 monclient: shutdown 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 msgr2=0x7f7f80135af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7f70006fd0 tx=0x7f7f70008040 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 shutdown_connections 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f64035ac0 0x7f7f64037f70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f8009d920 0x7f7f80136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80095800 0x7f7f80135af0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 --2- 172.21.15.142:0/2031074108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f800953c0 0x7f7f801355b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 >> 172.21.15.142:0/2031074108 conn(0x7f7f800a1fe0 msgr2=0x7f7f800987d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 shutdown_connections 2024-09-18T10:10:42.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.687+0000 7f7f863b0700 1 -- 172.21.15.142:0/2031074108 wait complete. 2024-09-18T10:10:42.702 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:10:42.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:10:42.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:10:42.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.702+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' removed v41) v1 ==== 152+0+0 (secure 0 0 0) 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:42.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.702+0000 7f1a34ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.702+0000 7f1a34ff9700 10 monclient: _finish_command 2 = system:0 pool 'fooo' removed 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.799+0000 7f2245d6c700 10 monclient: get_monmap_and_config 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.799+0000 7f2245d6c700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.799+0000 7f2245d6c700 10 monclient: monmap: 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:42.803585+0000 2024-09-18T10:10:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:42.803585+0000 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:42.806 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:42.806 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 Processor -- start 2024-09-18T10:10:42.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 -- start start 2024-09-18T10:10:42.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient: init 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f224009e350 0x7f224009e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-c con 0x7f224009e350 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-b con 0x7f224009fd00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f224006e3b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224009f1a0 con 0x7f224006e3b0 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224006d7c0 con 0x7f224009e350 2024-09-18T10:10:42.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224009fac0 con 0x7f224009fd00 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2245d6c700 10 monclient(hunting): authenticate will time out at 931.730690s 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f224006e3b0 0x7f2240130120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60340/0 (socket says 172.21.15.142:60340) 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223ffff700 1 -- 172.21.15.142:0/1923229330 learned_addr learned my addr 172.21.15.142:0/1923229330 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 10 monclient(hunting): get_auth_request con 0x7f224009fd00 auth_method 0 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 --2- 172.21.15.142:0/1923229330 >> v2:172.21.15.150:3300/0 conn(0x7f224009e350 0x7f224009e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): get_auth_request con 0x7f224009e350 auth_method 0 2024-09-18T10:10:42.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 1 -- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 msgr2=0x7f224009ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 1 -- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 msgr2=0x7f224009ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f2236ffd700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient(hunting): handle_auth_done global_id 14374 payload 274 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 -- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 msgr2=0x7f224009ec60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 -- 172.21.15.142:0/1923229330 >> v1:172.21.15.142:6789/0 conn(0x7f224006e3b0 legacy=0x7f2240130120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:42.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:42.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:42.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.803+0000 7f223f7fe700 1 -- 172.21.15.142:0/1923229330 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224006d200 con 0x7f224009e350 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223f7fe700 1 -- 172.21.15.142:0/1923229330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009e350 msgr2=0x7f224009e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223f7fe700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009e350 0x7f224009e720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f22400a3890 tx=0x7f22300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac0835ddedf2e9d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 1 -- 172.21.15.142:0/1923229330 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223000b8c0 con 0x7f224009e350 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: dump: 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 5 monclient: authenticate success, global_id 14374 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:42.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 1 -- 172.21.15.142:0/1923229330 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223000ba20 con 0x7f224009e350 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 4 set_mon_vals no callback set 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 1 -- 172.21.15.142:0/1923229330 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223000bdc0 con 0x7f224009e350 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f223effd700 10 monclient: dump: 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 10 monclient: shutdown 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 -- 172.21.15.142:0/1923229330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009e350 msgr2=0x7f224009e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009e350 0x7f224009e720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f22400a3890 tx=0x7f22300091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 -- 172.21.15.142:0/1923229330 shutdown_connections 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224009fd00 0x7f224009ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 --2- 172.21.15.142:0/1923229330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009e350 0x7f224009e720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 -- 172.21.15.142:0/1923229330 >> 172.21.15.142:0/1923229330 conn(0x7f2240071290 msgr2=0x7f2240071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 -- 172.21.15.142:0/1923229330 shutdown_connections 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.807+0000 7f2245d6c700 1 -- 172.21.15.142:0/1923229330 wait complete. 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient: monmap: 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:42.821 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 Processor -- start 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 -- start start 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient: init 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-c con 0x7f224006e3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 0x7f224011bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-a con 0x7f224009e350 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009fd00 0x7f224011c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): picked mon.noname-b con 0x7f224009fd00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224006e8a0 con 0x7f224009e350 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240133150 con 0x7f224009fd00 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224011d7e0 con 0x7f224006e3b0 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2245d6c700 10 monclient(hunting): authenticate will time out at 931.739074s 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 0x7f224011bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55540/0 (socket says 172.21.15.142:55540) 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 learned_addr learned my addr 172.21.15.142:0/2754731720 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 10 monclient(hunting): get_auth_request con 0x7f224006e3b0 auth_method 0 2024-09-18T10:10:42.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 0x7f224011bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36212/0 (socket says 172.21.15.142:36212) 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): get_auth_request con 0x7f224009e350 auth_method 0 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009fd00 0x7f224011c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): get_auth_request con 0x7f224009fd00 auth_method 0 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 msgr2=0x7f2240132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:42.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 msgr2=0x7f2240132180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient(hunting): handle_auth_done global_id 4689 payload 824 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 msgr2=0x7f2240132180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224006e3b0 0x7f2240132180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009fd00 msgr2=0x7f224011c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009fd00 0x7f224011c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 monclient: found mon.noname-a 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f2236ffd700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2230009420 con 0x7f224009e350 2024-09-18T10:10:42.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.811+0000 7f223ffff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f224009fd00 0x7f224011c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:42.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2236ffd700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 msgr2=0x7f224011bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2236ffd700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 0x7f224011bc90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f222800d860 tx=0x7f2228008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228012800 con 0x7f224009e350 2024-09-18T10:10:42.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: dump: 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.830 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 5 monclient: authenticate success, global_id 4689 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2228012980 con 0x7f224009e350 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 monclient: _renew_subs 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2240140460 con 0x7f224009e350 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228012d40 con 0x7f224009e350 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 10 monclient: dump: 2024-09-18T10:10:42.831 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f223e7fc700 4 set_mon_vals no callback set 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 monclient: _renew_subs 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224011e3b0 con 0x7f224009e350 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 0x7f224011c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240064930 con 0x7f2240134a60 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.815+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2228014640 con 0x7f224009e350 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 0x7f224011c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient: get_auth_request con 0x7f2240134a60 auth_method 0 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2220035af0 0x7f2220037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4776+0+0 (secure 0 0 0) 0x7f22280486f0 con 0x7f224009e350 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 10 monclient(hunting): handle_auth_done global_id 4692 payload 824 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22400696a0 con 0x7f2240134a60 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223ffff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2220035af0 0x7f2220037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223ffff700 10 monclient: get_auth_request con 0x7f2220035af0 auth_method 0 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 msgr2=0x7f224011c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 0x7f224011c710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f22300091e0 tx=0x7f2230017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230020070 con 0x7f2240134a60 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 10 monclient: dump: 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.834 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f223ffff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2220035af0 0x7f2220037fa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f223800d6a0 tx=0x7f2238009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2230025470 con 0x7f2240134a60 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 msgr2=0x7f224011c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 0x7f224011c710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f22300091e0 tx=0x7f2230017000 comp rx=0 tx=0).stop 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 msgr2=0x7f224011c710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:42.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.819+0000 7f2237fff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2240134a60 0x7f224011c710 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f22300091e0 tx=0x7f2230017000 comp rx=0 tx=0).stop 2024-09-18T10:10:42.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f2245d6c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f2245d6c700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 0x7f224011cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f2245d6c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240067990 con 0x7f22401326c0 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 0x7f224011cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient: get_auth_request con 0x7f22401326c0 auth_method 0 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 10 monclient(hunting): handle_auth_done global_id 4695 payload 824 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.831+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f224011f720 con 0x7f22401326c0 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f223f7fe700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 msgr2=0x7f224011cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f223f7fe700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 0x7f224011cc50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2230024000 tx=0x7f223001d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223001abc0 con 0x7f22401326c0 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 10 monclient: dump: 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.839 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f22300303a0 con 0x7f22401326c0 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 msgr2=0x7f224011cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 0x7f224011cc50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2230024000 tx=0x7f223001d070 comp rx=0 tx=0).stop 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 msgr2=0x7f224011cc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:42.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2237fff700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22401326c0 0x7f224011cc50 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2230024000 tx=0x7f223001d070 comp rx=0 tx=0).stop 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:10:42.841 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:42.842 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:42.843 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:42.844 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:42.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.846 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:42.847 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.848 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:42.849 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:42.850 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:42.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.852 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:42.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:42.854 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:42.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:42.854 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:42.854 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2220035af0 msgr2=0x7f2220037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2220035af0 0x7f2220037fa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f223800d6a0 tx=0x7f2238009040 comp rx=0 tx=0).stop 2024-09-18T10:10:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 10 monclient: shutdown 2024-09-18T10:10:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 msgr2=0x7f224011bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 --2- 172.21.15.142:0/2754731720 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f224009e350 0x7f224011bc90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f222800d860 tx=0x7f2228008040 comp rx=0 tx=0).stop 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f223ffff700 1 -- 172.21.15.142:0/2754731720 reap_dead start 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f223ffff700 1 -- 172.21.15.142:0/2754731720 reap_dead start 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 shutdown_connections 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 >> 172.21.15.142:0/2754731720 conn(0x7f2240071290 msgr2=0x7f2240133360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 shutdown_connections 2024-09-18T10:10:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.835+0000 7f2245d6c700 1 -- 172.21.15.142:0/2754731720 wait complete. 2024-09-18T10:10:42.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:42.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:10:42.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a3f043700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a3f043700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a3f043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a34ff9700 1 -- 172.21.15.150:0/2226794341 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v41) v1 ==== 159+0+0 (secure 0 0 0) 0x7f1a3808fb10 con 0x7f1a3809d000 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a34ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.842+0000 7f1a34ff9700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T10:10:42.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 msgr2=0x7f1a1c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 0x7f1a1c037f70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a24006fd0 tx=0x7f1a24008040 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 10 monclient: shutdown 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 msgr2=0x7f1a38141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1a3806aa60 tx=0x7f1a2c016000 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 shutdown_connections 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a1c035ac0 0x7f1a1c037f70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a380a0ba0 0x7f1a3808df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a3809e780 0x7f1a381421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 --2- 172.21.15.150:0/2226794341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3809d000 0x7f1a38141ca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 >> 172.21.15.150:0/2226794341 conn(0x7f1a380a5830 msgr2=0x7f1a3809ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 shutdown_connections 2024-09-18T10:10:42.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.846+0000 7f1a3f043700 1 -- 172.21.15.150:0/2226794341 wait complete. 2024-09-18T10:10:42.861 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fooo' does not exist 2024-09-18T10:10:42.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T10:10:42.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.950+0000 7f5305ed5700 10 monclient: get_monmap_and_config 2024-09-18T10:10:42.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.950+0000 7f5305ed5700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.950+0000 7f5305ed5700 10 monclient: monmap: 2024-09-18T10:10:42.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:42.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:42.954640+0000 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:42.954640+0000 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:42.956 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:42.957 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:42.957 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:42.957 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:42.957 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.950+0000 7f5305ed5700 1 Processor -- start 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 -- start start 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient: init 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T10:10:42.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53000989e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-b con 0x7f530009c000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f53001338c0 0x7f5300135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-a con 0x7f53001338c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5300097cd0 con 0x7f53000989e0 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5300096d40 con 0x7f53001338c0 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53000979f0 con 0x7f530009c000 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f5305ed5700 10 monclient(hunting): authenticate will time out at 925.049381s 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52ff7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f53000989e0 0x7f530009b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:54626/0 (socket says 172.21.15.150:54626) 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52ff7fe700 1 -- 172.21.15.150:0/3201557731 learned_addr learned my addr 172.21.15.150:0/3201557731 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 10 monclient(hunting): get_auth_request con 0x7f530009c000 auth_method 0 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 --2- 172.21.15.150:0/3201557731 >> v2:172.21.15.142:3300/0 conn(0x7f53001338c0 0x7f5300135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 1 -- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 1 -- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52f6ffd700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): get_auth_request con 0x7f53001338c0 auth_method 0 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient(hunting): handle_auth_done global_id 4698 payload 274 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 -- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300099320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:42.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 -- 172.21.15.150:0/3201557731 >> v1:172.21.15.142:6789/0 conn(0x7f53000989e0 legacy=0x7f530009b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 monclient: found mon.noname-a 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.954+0000 7f52fffff700 1 -- 172.21.15.150:0/3201557731 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300097fb0 con 0x7f53001338c0 2024-09-18T10:10:42.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52fffff700 1 -- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 msgr2=0x7f5300135ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52fffff700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300135ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009e60 tx=0x7f52f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ed5f52d867cfd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 1 -- 172.21.15.150:0/3201557731 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f800e070 con 0x7f53001338c0 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: dump: 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.969 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 5 monclient: authenticate success, global_id 4698 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 1 -- 172.21.15.150:0/3201557731 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f800be40 con 0x7f53001338c0 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 4 set_mon_vals no callback set 2024-09-18T10:10:42.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 1 -- 172.21.15.150:0/3201557731 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8012660 con 0x7f53001338c0 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f52feffd700 10 monclient: dump: 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.971 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 10 monclient: shutdown 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 -- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 msgr2=0x7f5300135ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300135ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009e60 tx=0x7f52f8009780 comp rx=0 tx=0).stop 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 -- 172.21.15.150:0/3201557731 shutdown_connections 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3201557731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300135ca0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 -- 172.21.15.150:0/3201557731 >> 172.21.15.150:0/3201557731 conn(0x7f5300068590 msgr2=0x7f53000a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:42.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 -- 172.21.15.150:0/3201557731 shutdown_connections 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 1 -- 172.21.15.150:0/3201557731 wait complete. 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.958+0000 7f5305ed5700 10 monclient: monmap: 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:42.973 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 Processor -- start 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 -- start start 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient: init 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 0x7f530006e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-b con 0x7f53000989e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-c con 0x7f530009c000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300118970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): picked mon.noname-a con 0x7f53001338c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530006f120 con 0x7f53001338c0 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530009b130 con 0x7f53000989e0 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530011b180 con 0x7f530009c000 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f5305ed5700 10 monclient(hunting): authenticate will time out at 925.057998s 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 0x7f530006e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 0x7f530006e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47356/0 (socket says 172.21.15.150:47356) 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300118970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 1 -- 172.21.15.150:0/3369438930 learned_addr learned my addr 172.21.15.150:0/3369438930 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): get_auth_request con 0x7f53000989e0 auth_method 0 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 10 monclient(hunting): get_auth_request con 0x7f530009c000 auth_method 0 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): get_auth_request con 0x7f53001338c0 auth_method 0 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:42.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52f6ffd700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient(hunting): handle_auth_done global_id 4701 payload 824 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 msgr2=0x7f5300118430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 msgr2=0x7f530006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 0x7f530006e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 monclient: found mon.noname-a 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:42.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.962+0000 7f52fffff700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f8009a20 con 0x7f53001338c0 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52fffff700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 msgr2=0x7f5300118970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52fffff700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300118970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52f8015000 tx=0x7f52f8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8026430 con 0x7f53001338c0 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: dump: 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.983 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 5 monclient: authenticate success, global_id 4701 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _renew_subs 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530011b460 con 0x7f53001338c0 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _renew_subs 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530012d3e0 con 0x7f53001338c0 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f8021070 con 0x7f53001338c0 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8026840 con 0x7f53001338c0 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 10 monclient: dump: 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.985 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52fe7fc700 4 set_mon_vals no callback set 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f52f8023030 con 0x7f53001338c0 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e40050b0 con 0x7f53001338c0 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 0x7f52e0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4776+0+0 (secure 0 0 0) 0x7f52f804a7f0 con 0x7f53001338c0 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52ff7fe700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 0x7f52e0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.966+0000 7f52ff7fe700 10 monclient: get_auth_request con 0x7f52e0035ac0 auth_method 0 2024-09-18T10:10:42.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.970+0000 7f52ff7fe700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 0x7f52e0037f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007880 tx=0x7f52f0004a80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:42.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.982+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f52f802eb70 con 0x7f53001338c0 2024-09-18T10:10:42.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.982+0000 7f52f7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:42.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:42.982+0000 7f52f7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 10 monclient: get_monmap_and_config 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 10 monclient: build_initial_monmap 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 10 monclient: monmap: 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:42.970923+0000 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:42.970923+0000 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:42.995 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 1 Processor -- start 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 1 -- start start 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.967+0000 7fa72c823700 10 monclient: init 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:42.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa72809cf70 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-b con 0x7fa72809e3d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa728137a00 0x7fa728139de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-c con 0x7fa728137a00 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280a1df0 con 0x7fa72809cf70 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72809dfb0 con 0x7fa728137a00 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72809e7c0 con 0x7fa72809e3d0 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:42.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72c823700 10 monclient(hunting): authenticate will time out at 931.896580s 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72659c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa72809cf70 0x7fa7280a0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:44642/0 (socket says 172.21.15.142:44642) 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72659c700 1 -- 172.21.15.142:0/3176042725 learned_addr learned my addr 172.21.15.142:0/3176042725 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 10 monclient(hunting): get_auth_request con 0x7fa72809e3d0 auth_method 0 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3176042725 >> v2:172.21.15.150:3300/0 conn(0x7fa728137a00 0x7fa728139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): get_auth_request con 0x7fa728137a00 auth_method 0 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:42.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 1 -- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 msgr2=0x7fa7280a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 1 -- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 msgr2=0x7fa7280a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72659c700 1 -- 172.21.15.142:0/3176042725 >> v1:172.21.15.150:6789/0 conn(0x7fa72809cf70 legacy=0x7fa7280a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72559a700 1 -- 172.21.15.142:0/3176042725 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123027096 0 0) 0x7fa7280a1df0 con 0x7fa72809cf70 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72559a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:42.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72559a700 10 monclient(hunting): my global_id is 14386 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa72559a700 1 -- 172.21.15.142:0/3176042725 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa714003570 con 0x7fa72809cf70 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient(hunting): handle_auth_done global_id 14383 payload 274 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 -- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 msgr2=0x7fa7280a0980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 -- 172.21.15.142:0/3176042725 >> v1:172.21.15.150:6789/0 conn(0x7fa72809cf70 legacy=0x7fa7280a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 monclient: found mon.noname-c 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:43.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.971+0000 7fa726d9d700 1 -- 172.21.15.142:0/3176042725 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7280a1650 con 0x7fa728137a00 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa726d9d700 1 -- 172.21.15.142:0/3176042725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 msgr2=0x7fa728139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa728139de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa720009f10 tx=0x7fa720009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0dbd7f8df3c40e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 1 -- 172.21.15.142:0/3176042725 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72000e070 con 0x7fa728137a00 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: dump: 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:43.001 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: _finish_auth 0 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 5 monclient: authenticate success, global_id 14383 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 1 -- 172.21.15.142:0/3176042725 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72000be00 con 0x7fa728137a00 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 4 set_mon_vals no callback set 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 1 -- 172.21.15.142:0/3176042725 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720012660 con 0x7fa728137a00 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72559a700 10 monclient: dump: 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 10 monclient: shutdown 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 -- 172.21.15.142:0/3176042725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 msgr2=0x7fa728139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa728139de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa720009f10 tx=0x7fa720009780 comp rx=0 tx=0).stop 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 -- 172.21.15.142:0/3176042725 shutdown_connections 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809e3d0 0x7fa7280a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 --2- 172.21.15.142:0/3176042725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa728139de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:43.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 -- 172.21.15.142:0/3176042725 >> 172.21.15.142:0/3176042725 conn(0x7fa7280a5880 msgr2=0x7fa7280a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 -- 172.21.15.142:0/3176042725 shutdown_connections 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.975+0000 7fa72c823700 1 -- 172.21.15.142:0/3176042725 wait complete. 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient: build_initial_monmap 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient: monmap: 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 Processor -- start 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 -- start start 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient: init 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-c con 0x7fa72809cf70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72809e3d0 0x7fa728133f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-a con 0x7fa72809e3d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa7281344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): picked mon.noname-b con 0x7fa728137a00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55572/0 (socket says 172.21.15.142:55572) 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 learned_addr learned my addr 172.21.15.142:0/3186809338 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72809e3d0 0x7fa728133f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72806e5f0 con 0x7fa72809e3d0 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72812b720 con 0x7fa728137a00 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280986b0 con 0x7fa72809cf70 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72c823700 10 monclient(hunting): authenticate will time out at 931.906755s 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 10 monclient(hunting): get_auth_request con 0x7fa72809cf70 auth_method 0 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): get_auth_request con 0x7fa72809e3d0 auth_method 0 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa7281344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): get_auth_request con 0x7fa728137a00 auth_method 0 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 msgr2=0x7fa728141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:43.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 msgr2=0x7fa728141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient(hunting): handle_auth_done global_id 14389 payload 824 2024-09-18T10:10:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 msgr2=0x7fa728141150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72809cf70 0x7fa728141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72809e3d0 msgr2=0x7fa728133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72809e3d0 0x7fa728133f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 monclient: found mon.noname-b 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa720009a20 con 0x7fa728137a00 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.979+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa72809e3d0 0x7fa728133f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 msgr2=0x7fa7281344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa726d9d700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa7281344c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa72806a910 tx=0x7fa720019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720012ea0 con 0x7fa728137a00 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72001b600 con 0x7fa728137a00 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 5 monclient: authenticate success, global_id 14389 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 monclient: _renew_subs 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72812b440 con 0x7fa728137a00 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72001b9c0 con 0x7fa728137a00 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa724d99700 4 set_mon_vals no callback set 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 monclient: _renew_subs 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72806eb60 con 0x7fa728137a00 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:43.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa728064850 con 0x7fa7281416f0 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa72002f450 con 0x7fa728137a00 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa708035ac0 0x7fa708037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa7281416f0 auth_method 0 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa72659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.983+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4776+0+0 (secure 0 0 0) 0x7fa720049f70 con 0x7fa728137a00 2024-09-18T10:10:43.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa708035ac0 0x7fa708037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa725d9b700 10 monclient: get_auth_request con 0x7fa708035ac0 auth_method 0 2024-09-18T10:10:43.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:42.987+0000 7fa725d9b700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa708035ac0 0x7fa708037f70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa718006f30 tx=0x7fa718008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:43.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:43.154+0000 7f5305ed5700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:43.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:43.154+0000 7f5305ed5700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:43.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:43.154+0000 7f5305ed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:43.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:43.154+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52e40058e0 con 0x7f53001338c0 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa7281416f0 auth_method 0 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:43.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.191+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa7281416f0 auth_method 0 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:43.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:43.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:43.595+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:10:44.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.322+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' removed v43) v1 ==== 149+0+0 (secure 0 0 0) 0x7f52f802e390 con 0x7f53001338c0 2024-09-18T10:10:44.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.322+0000 7f52f7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.322+0000 7f52f7fff700 10 monclient: _finish_command 2 = system:0 pool 'foo' removed 2024-09-18T10:10:44.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa7281416f0 auth_method 0 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:44.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:44.399+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:10:44.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.426+0000 7f5305ed5700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.426+0000 7f5305ed5700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.426+0000 7f5305ed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.426+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f52e4001a50 con 0x7f53001338c0 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f52f7fff700 1 -- 172.21.15.150:0/3369438930 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) v1 ==== 156+0+0 (secure 0 0 0) 0x7f52f802e570 con 0x7f53001338c0 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f52f7fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f52f7fff700 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 msgr2=0x7f52e0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 0x7f52e0037f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007880 tx=0x7f52f0004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 10 monclient: shutdown 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 msgr2=0x7f5300118970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300118970 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f52f8015000 tx=0x7f52f8018000 comp rx=0 tx=0).stop 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 shutdown_connections 2024-09-18T10:10:44.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f52e0035ac0 0x7f52e0037f70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53001338c0 0x7f5300118970 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530009c000 0x7f5300118430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 --2- 172.21.15.150:0/3369438930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53000989e0 0x7f530006e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 >> 172.21.15.150:0/3369438930 conn(0x7f5300068590 msgr2=0x7f530009b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 shutdown_connections 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.430+0000 7f5305ed5700 1 -- 172.21.15.150:0/3369438930 wait complete. 2024-09-18T10:10:44.436 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foo' does not exist 2024-09-18T10:10:44.444 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: get_monmap_and_config 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: build_initial_monmap 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: monmap: 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:44.541333+0000 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:44.541333+0000 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:44.542 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:44.543 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 1 Processor -- start 2024-09-18T10:10:44.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 1 -- start start 2024-09-18T10:10:44.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: init 2024-09-18T10:10:44.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:44.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:10:44.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f769c09e260 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:44.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.538+0000 7f76a4352700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f769c06ed30 0x7f769c135500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-a con 0x7f769c06ed30 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-b con 0x7f769c09ed40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c0a0050 con 0x7f769c09e260 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c06d480 con 0x7f769c06ed30 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c09fb60 con 0x7f769c09ed40 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient(hunting): authenticate will time out at 926.636149s 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f769c06ed30 0x7f769c135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a20ee700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f769c09e260 0x7f769c09e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:54630/0 (socket says 172.21.15.150:54630) 2024-09-18T10:10:44.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a20ee700 1 -- 172.21.15.150:0/2706708799 learned_addr learned my addr 172.21.15.150:0/2706708799 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): get_auth_request con 0x7f769c06ed30 auth_method 0 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 10 monclient(hunting): get_auth_request con 0x7f769c09ed40 auth_method 0 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:44.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 1 -- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 msgr2=0x7f769c137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 1 -- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 msgr2=0x7f769c137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a28ef700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient(hunting): handle_auth_done global_id 4707 payload 274 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 -- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 msgr2=0x7f769c137a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 -- 172.21.15.150:0/2706708799 >> v1:172.21.15.142:6789/0 conn(0x7f769c09e260 legacy=0x7f769c09e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient: _finish_hunting 0 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 monclient: found mon.noname-a 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:44.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 -- 172.21.15.150:0/2706708799 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c09fda0 con 0x7f769c06ed30 2024-09-18T10:10:44.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 -- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c06ed30 msgr2=0x7f769c135500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:44.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a18ed700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c06ed30 0x7f769c135500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f768c0098e0 tx=0x7f768c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7819fef40524118 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 1 -- 172.21.15.150:0/2706708799 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c009e80 con 0x7f769c06ed30 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: dump: 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.552 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: _finish_auth 0 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 5 monclient: authenticate success, global_id 4707 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 1 -- 172.21.15.150:0/2706708799 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c00ba70 con 0x7f769c06ed30 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a4352700 4 set_mon_vals no callback set 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 1 -- 172.21.15.150:0/2706708799 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00bdf0 con 0x7f769c06ed30 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:44.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.542+0000 7f76a10ec700 10 monclient: dump: 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 10 monclient: shutdown 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c06ed30 msgr2=0x7f769c135500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c06ed30 0x7f769c135500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f768c0098e0 tx=0x7f768c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- 172.21.15.150:0/2706708799 shutdown_connections 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c09ed40 0x7f769c137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 --2- 172.21.15.150:0/2706708799 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c06ed30 0x7f769c135500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- 172.21.15.150:0/2706708799 >> 172.21.15.150:0/2706708799 conn(0x7f769c0a3e10 msgr2=0x7f769c0a0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- 172.21.15.150:0/2706708799 shutdown_connections 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- 172.21.15.150:0/2706708799 wait complete. 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 10 monclient: build_initial_monmap 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 10 monclient: monmap: 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.555 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 Processor -- start 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 1 -- start start 2024-09-18T10:10:44.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.546+0000 7f76a4352700 10 monclient: init 2024-09-18T10:10:44.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:44.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:44.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 0x7f769c1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-a con 0x7f769c09e260 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:44.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-b con 0x7f769c09ed40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): picked mon.noname-c con 0x7f769c1328d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1217b0 con 0x7f769c09e260 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1330c0 con 0x7f769c09ed40 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c142530 con 0x7f769c1328d0 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient(hunting): authenticate will time out at 926.644201s 2024-09-18T10:10:44.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47366/0 (socket says 172.21.15.150:47366) 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 -- 172.21.15.150:0/4058122329 learned_addr learned my addr 172.21.15.150:0/4058122329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): get_auth_request con 0x7f769c09ed40 auth_method 0 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:44.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 0x7f769c1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 10 monclient(hunting): get_auth_request con 0x7f769c1328d0 auth_method 0 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 10 monclient(hunting): get_auth_request con 0x7f769c09e260 auth_method 0 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 msgr2=0x7f769c0699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 msgr2=0x7f769c0699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:44.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient(hunting): handle_auth_done global_id 14392 payload 824 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a28ef700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 msgr2=0x7f769c0699c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 msgr2=0x7f769c1416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 0x7f769c1416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient: _finish_hunting 0 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 monclient: found mon.noname-b 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c009420 con 0x7f769c09ed40 2024-09-18T10:10:44.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a20ee700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 0x7f769c1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 msgr2=0x7f769c132390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a18ed700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f768c0030c0 tx=0x7f768c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c006af0 con 0x7f769c09ed40 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: dump: 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.565 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 5 monclient: authenticate success, global_id 14392 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _renew_subs 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c1214d0 con 0x7f769c09ed40 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c020070 con 0x7f769c09ed40 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c01b6a0 con 0x7f769c09ed40 2024-09-18T10:10:44.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76927fc700 10 monclient: dump: 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a08eb700 4 set_mon_vals no callback set 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _renew_subs 2024-09-18T10:10:44.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.550+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c141da0 con 0x7f769c09ed40 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a4352700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a4352700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a4352700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c121920 con 0x7f769c09ed40 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f768c022030 con 0x7f769c09ed40 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76927fc700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 0x7f7680037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f768c04a080 con 0x7f769c09ed40 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a20ee700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 0x7f7680037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.554+0000 7f76a20ee700 10 monclient: get_auth_request con 0x7f7680035af0 auth_method 0 2024-09-18T10:10:44.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.558+0000 7f76a20ee700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 0x7f7680037fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7698005300 tx=0x7f7698009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.562+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f768c07a670 con 0x7f769c09ed40 2024-09-18T10:10:44.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.562+0000 7f76927fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.562+0000 7f76927fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:44.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76a4352700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76a4352700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76a4352700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f769c121920 con 0x7f769c09ed40 2024-09-18T10:10:44.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) v1 ==== 156+0+0 (secure 0 0 0) 0x7f768c02c380 con 0x7f769c09ed40 2024-09-18T10:10:44.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76927fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.726+0000 7f76927fc700 10 monclient: _finish_command 2 = system:0 pool 'foo' does not exist 2024-09-18T10:10:44.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.822+0000 7f76a4352700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.822+0000 7f76a4352700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.822+0000 7f76a4352700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.822+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f769c121920 con 0x7f769c09ed40 2024-09-18T10:10:44.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76927fc700 1 -- 172.21.15.150:0/4058122329 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}]=0 pool 'foo' does not exist v43) v1 ==== 156+0+0 (secure 0 0 0) 0x7f768c02c380 con 0x7f769c09ed40 2024-09-18T10:10:44.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76927fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "foo", "pool2": "foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:44.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76927fc700 10 monclient: _finish_command 3 = system:0 pool 'foo' does not exist 2024-09-18T10:10:44.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 msgr2=0x7f7680037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 0x7f7680037fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7698005300 tx=0x7f7698009040 comp rx=0 tx=0).stop 2024-09-18T10:10:44.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 10 monclient: shutdown 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 msgr2=0x7f769c132390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f768c0030c0 tx=0x7f768c016000 comp rx=0 tx=0).stop 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 shutdown_connections 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7680035af0 0x7f7680037fa0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f769c1328d0 0x7f769c0699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f769c09ed40 0x7f769c132390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 --2- 172.21.15.150:0/4058122329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f769c09e260 0x7f769c1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 >> 172.21.15.150:0/4058122329 conn(0x7f769c0a3e10 msgr2=0x7f769c1332f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 shutdown_connections 2024-09-18T10:10:44.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.826+0000 7f76a4352700 1 -- 172.21.15.150:0/4058122329 wait complete. 2024-09-18T10:10:44.833 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'foo' does not exist 2024-09-18T10:10:44.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T10:10:44.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: get_monmap_and_config 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: build_initial_monmap 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: monmap: 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:44.941155+0000 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:44.941155+0000 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:44.942 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:44.943 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 Processor -- start 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 -- start start 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: init 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:10:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-b con 0x7fadf409d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fadf409e780 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fadf41379b0 0x7fadf4139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-a con 0x7fadf41379b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf40699a0 con 0x7fadf409e780 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf40a1640 con 0x7fadf41379b0 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf409e3d0 con 0x7fadf409d000 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadfada7700 10 monclient(hunting): authenticate will time out at 927.034968s 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf3fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fadf409e780 0x7fadf40a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38254/0 (socket says 172.21.15.150:38254) 2024-09-18T10:10:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf3fff700 1 -- 172.21.15.150:0/3429843458 learned_addr learned my addr 172.21.15.150:0/3429843458 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 1 --2- 172.21.15.150:0/3429843458 >> v2:172.21.15.142:3300/0 conn(0x7fadf41379b0 0x7fadf4139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 10 monclient(hunting): get_auth_request con 0x7fadf409d000 auth_method 0 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): get_auth_request con 0x7fadf41379b0 auth_method 0 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:44.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 1 -- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 msgr2=0x7fadf40a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 1 -- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 msgr2=0x7fadf40a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.938+0000 7fadf8b43700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 10 monclient(hunting): handle_auth_done global_id 4713 payload 274 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 -- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 msgr2=0x7fadf40a0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 -- 172.21.15.150:0/3429843458 >> v1:172.21.15.150:6789/0 conn(0x7fadf409e780 legacy=0x7fadf40a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 10 monclient: _finish_hunting 0 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 monclient: found mon.noname-a 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:44.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 -- 172.21.15.150:0/3429843458 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf409ebf0 con 0x7fadf41379b0 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 -- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf41379b0 msgr2=0x7fadf4139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf9344700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf41379b0 0x7fadf4139d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fadf409ede0 tx=0x7fadec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e327648c51da6f9b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 1 -- 172.21.15.150:0/3429843458 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec00e070 con 0x7fadf41379b0 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: dump: 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 5 monclient: authenticate success, global_id 4713 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 1 -- 172.21.15.150:0/3429843458 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec00be40 con 0x7fadf41379b0 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 1 -- 172.21.15.150:0/3429843458 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec012660 con 0x7fadf41379b0 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 4 set_mon_vals no callback set 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadf37fe700 10 monclient: dump: 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.953 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 10 monclient: shutdown 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 1 -- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf41379b0 msgr2=0x7fadf4139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.942+0000 7fadfada7700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf41379b0 0x7fadf4139d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fadf409ede0 tx=0x7fadec009780 comp rx=0 tx=0).stop 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- 172.21.15.150:0/3429843458 shutdown_connections 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf41379b0 0x7fadf4139d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 --2- 172.21.15.150:0/3429843458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409d000 0x7fadf40a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- 172.21.15.150:0/3429843458 >> 172.21.15.150:0/3429843458 conn(0x7fadf40a5830 msgr2=0x7fadf40a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- 172.21.15.150:0/3429843458 shutdown_connections 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- 172.21.15.150:0/3429843458 wait complete. 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient: build_initial_monmap 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient: monmap: 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 Processor -- start 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- start start 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient: init 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 0x7fadf41363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-a con 0x7fadf409d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-c con 0x7fadf409e780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): picked mon.noname-b con 0x7fadf41379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf412d110 con 0x7fadf409d000 2024-09-18T10:10:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf409bb50 con 0x7fadf41379b0 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf412e720 con 0x7fadf409e780 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadfada7700 10 monclient(hunting): authenticate will time out at 927.042224s 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47374/0 (socket says 172.21.15.150:47374) 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 -- 172.21.15.150:0/2347957802 learned_addr learned my addr 172.21.15.150:0/2347957802 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): get_auth_request con 0x7fadf41379b0 auth_method 0 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 10 monclient(hunting): get_auth_request con 0x7fadf409e780 auth_method 0 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 0x7fadf41363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): get_auth_request con 0x7fadf409d000 auth_method 0 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 msgr2=0x7fadf412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:44.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 msgr2=0x7fadf412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:44.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:44.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:44.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:44.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:44.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf3fff700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient(hunting): handle_auth_done global_id 14395 payload 824 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 msgr2=0x7fadf412ae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 msgr2=0x7fadf41363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 0x7fadf41363e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:44.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient: _finish_hunting 0 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 monclient: found mon.noname-b 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf9344700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec009a20 con 0x7fadf41379b0 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.946+0000 7fadf8b43700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 0x7fadf41363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf9344700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 msgr2=0x7fadf412b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf9344700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fadec009fd0 tx=0x7fadec019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec012e90 con 0x7fadf41379b0 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: dump: 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.962 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 5 monclient: authenticate success, global_id 14395 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec01b600 con 0x7fadf41379b0 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 10 monclient: _renew_subs 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf41359f0 con 0x7fadf41379b0 2024-09-18T10:10:44.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec01b9c0 con 0x7fadf41379b0 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 10 monclient: dump: 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:44.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf27fc700 4 set_mon_vals no callback set 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 10 monclient: _renew_subs 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadfada7700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf412e4f0 con 0x7fadf41379b0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadde7fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fadec02f450 con 0x7fadf41379b0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadde7fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadde7fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.950+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc0050b0 con 0x7fadf41379b0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.954+0000 7fadf0ff9700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 0x7fadd8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.954+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fadec049e20 con 0x7fadf41379b0 2024-09-18T10:10:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.954+0000 7fadf8b43700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 0x7fadd8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:44.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.954+0000 7fadf8b43700 10 monclient: get_auth_request con 0x7fadd8035ac0 auth_method 0 2024-09-18T10:10:44.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.954+0000 7fadf8b43700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 0x7fadd8037f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fade0006f30 tx=0x7fade0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:44.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.958+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fadec02dda0 con 0x7fadf41379b0 2024-09-18T10:10:44.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.958+0000 7fadf0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:44.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:44.958+0000 7fadf0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:45.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.122+0000 7fadde7fc700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.122+0000 7fadde7fc700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.122+0000 7fadde7fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.122+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fadcc0058e0 con 0x7fadf41379b0 2024-09-18T10:10:45.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.126+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) v1 ==== 159+0+0 (secure 0 0 0) 0x7fadec02d7c0 con 0x7fadf41379b0 2024-09-18T10:10:45.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.126+0000 7fadf0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.126+0000 7fadf0ff9700 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-18T10:10:45.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.218+0000 7fadde7fc700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.218+0000 7fadde7fc700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.218+0000 7fadde7fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.218+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fadcc0062b0 con 0x7fadf41379b0 2024-09-18T10:10:45.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadf0ff9700 1 -- 172.21.15.150:0/2347957802 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) v1 ==== 159+0+0 (secure 0 0 0) 0x7fadec02dda0 con 0x7fadf41379b0 2024-09-18T10:10:45.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadf0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadf0ff9700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T10:10:45.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 msgr2=0x7fadd8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 0x7fadd8037f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fade0006f30 tx=0x7fade0008040 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 10 monclient: shutdown 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 msgr2=0x7fadf412b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fadec009fd0 tx=0x7fadec019000 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 shutdown_connections 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadd8035ac0 0x7fadd8037f70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadf41379b0 0x7fadf412b3b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fadf409e780 0x7fadf412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 --2- 172.21.15.150:0/2347957802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadf409d000 0x7fadf41363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 >> 172.21.15.150:0/2347957802 conn(0x7fadf40a5830 msgr2=0x7fadf4137130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 shutdown_connections 2024-09-18T10:10:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.222+0000 7fadde7fc700 1 -- 172.21.15.150:0/2347957802 wait complete. 2024-09-18T10:10:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fooo' does not exist 2024-09-18T10:10:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete fooo fooo --yes-i-really-really-mean-it 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: get_monmap_and_config 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: build_initial_monmap 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: monmap: 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:45.337648+0000 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:45.337648+0000 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:45.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 Processor -- start 2024-09-18T10:10:45.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 -- start start 2024-09-18T10:10:45.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: init 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-b con 0x7f687009d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 0x7f68700a0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-a con 0x7f687009e780 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700a0ba0 0x7f68701389b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-c con 0x7f68700a0ba0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700a1640 con 0x7f687009e780 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700699a0 con 0x7f68700a0ba0 2024-09-18T10:10:45.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687009e3d0 con 0x7f687009d000 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f6875d83700 10 monclient(hunting): authenticate will time out at 927.431374s 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700a0ba0 0x7f68701389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f68700a0ba0 0x7f68701389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47388/0 (socket says 172.21.15.150:47388) 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 1 -- 172.21.15.150:0/694985588 learned_addr learned my addr 172.21.15.150:0/694985588 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): get_auth_request con 0x7f68700a0ba0 auth_method 0 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686effd700 1 --2- 172.21.15.150:0/694985588 >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 0x7f68700a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 10 monclient(hunting): get_auth_request con 0x7f687009d000 auth_method 0 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686effd700 10 monclient(hunting): get_auth_request con 0x7f687009e780 auth_method 0 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 1 -- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 msgr2=0x7f68700a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:10:45.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 1 -- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 msgr2=0x7f68700a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.334+0000 7f686f7fe700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686f7fe700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 10 monclient(hunting): handle_auth_done global_id 14398 payload 274 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 -- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 msgr2=0x7f68700a0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 -- 172.21.15.150:0/694985588 >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 msgr2=0x7f68700a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 --2- 172.21.15.150:0/694985588 >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 0x7f68700a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 monclient: found mon.noname-c 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 -- 172.21.15.150:0/694985588 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687009ebf0 con 0x7f68700a0ba0 2024-09-18T10:10:45.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686effd700 1 --2- 172.21.15.150:0/694985588 >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 0x7f68700a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:45.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 -- 172.21.15.150:0/694985588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700a0ba0 msgr2=0x7f68701389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686ffff700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700a0ba0 0x7f68701389b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f687009c9a0 tx=0x7f6868009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e9b639cdce6909e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 1 -- 172.21.15.150:0/694985588 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686800e070 con 0x7f68700a0ba0 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: dump: 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.350 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 5 monclient: authenticate success, global_id 14398 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 1 -- 172.21.15.150:0/694985588 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686800bc60 con 0x7f68700a0ba0 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 1 -- 172.21.15.150:0/694985588 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68680092e0 con 0x7f68700a0ba0 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 4 set_mon_vals no callback set 2024-09-18T10:10:45.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f686e7fc700 10 monclient: dump: 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 10 monclient: shutdown 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 1 -- 172.21.15.150:0/694985588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700a0ba0 msgr2=0x7f68701389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.338+0000 7f6875d83700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700a0ba0 0x7f68701389b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f687009c9a0 tx=0x7f6868009840 comp rx=0 tx=0).stop 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- 172.21.15.150:0/694985588 shutdown_connections 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68700a0ba0 0x7f68701389b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- 172.21.15.150:0/694985588 >> v2:172.21.15.142:3300/0 conn(0x7f687009e780 0x7f68700a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- 172.21.15.150:0/694985588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687009d000 0x7f68700a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- 172.21.15.150:0/694985588 >> 172.21.15.150:0/694985588 conn(0x7f68700a5830 msgr2=0x7f68700a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- 172.21.15.150:0/694985588 shutdown_connections 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- 172.21.15.150:0/694985588 wait complete. 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient: build_initial_monmap 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient: monmap: 2024-09-18T10:10:45.353 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 Processor -- start 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- start start 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient: init 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:45.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-b con 0x7f687009d000 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687009e780 0x7f687008db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-a con 0x7f687009e780 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68700a0ba0 0x7f687008e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): picked mon.noname-c con 0x7f68700a0ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700986a0 con 0x7f687009e780 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687009be50 con 0x7f687009d000 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687008fa40 con 0x7f68700a0ba0 2024-09-18T10:10:45.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f6875d83700 10 monclient(hunting): authenticate will time out at 927.437488s 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47396/0 (socket says 172.21.15.150:47396) 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 -- 172.21.15.150:0/1209398610 learned_addr learned my addr 172.21.15.150:0/1209398610 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): get_auth_request con 0x7f687009d000 auth_method 0 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686ffff700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68700a0ba0 0x7f687008e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686effd700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687009e780 0x7f687008db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686ffff700 10 monclient(hunting): get_auth_request con 0x7f68700a0ba0 auth_method 0 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686effd700 10 monclient(hunting): get_auth_request con 0x7f687009e780 auth_method 0 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient(hunting): handle_auth_done global_id 14401 payload 824 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 -- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68700a0ba0 msgr2=0x7f687008e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68700a0ba0 0x7f687008e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 -- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687009e780 msgr2=0x7f687008db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687009e780 0x7f687008db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.342+0000 7f686f7fe700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6868009ae0 con 0x7f687009d000 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f686f7fe700 1 -- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 msgr2=0x7f6870099000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f686f7fe700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f686400d860 tx=0x7f6864008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6864012800 con 0x7f687009d000 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: dump: 2024-09-18T10:10:45.358 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 5 monclient: authenticate success, global_id 14401 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6864012960 con 0x7f687009d000 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _renew_subs 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687008f5d0 con 0x7f687009d000 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6864012d20 con 0x7f687009d000 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 10 monclient: dump: 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f686dffb700 4 set_mon_vals no callback set 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _renew_subs 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6870090100 con 0x7f687009d000 2024-09-18T10:10:45.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6864014640 con 0x7f687009d000 2024-09-18T10:10:45.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 0x7f685c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f6864000e40 con 0x7f687009d000 2024-09-18T10:10:45.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68700914e0 con 0x7f687009d000 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f686effd700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 0x7f685c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.346+0000 7f686effd700 10 monclient: get_auth_request con 0x7f685c035b10 auth_method 0 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.350+0000 7f686effd700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 0x7f685c037fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6860006f00 tx=0x7f6860008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.354+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f686401d400 con 0x7f687009d000 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.354+0000 7f68577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.354+0000 7f68577fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:45.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.566+0000 7f6875d83700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.566+0000 7f6875d83700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.566+0000 7f6875d83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.566+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f687006e3e0 con 0x7f687009d000 2024-09-18T10:10:45.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.570+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) v1 ==== 159+0+0 (secure 0 0 0) 0x7f686401c360 con 0x7f687009d000 2024-09-18T10:10:45.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.570+0000 7f68577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.570+0000 7f68577fe700 10 monclient: _finish_command 2 = system:0 pool 'fooo' does not exist 2024-09-18T10:10:45.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f6875d83700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f6875d83700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f6875d83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6870090320 con 0x7f687009d000 2024-09-18T10:10:45.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f68577fe700 1 -- 172.21.15.150:0/1209398610 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}]=0 pool 'fooo' does not exist v43) v1 ==== 159+0+0 (secure 0 0 0) 0x7f686401c9b0 con 0x7f687009d000 2024-09-18T10:10:45.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f68577fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fooo", "pool2": "fooo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.666+0000 7f68577fe700 10 monclient: _finish_command 3 = system:0 pool 'fooo' does not exist 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.670+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 msgr2=0x7f685c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.670+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 0x7f685c037fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6860006f00 tx=0x7f6860008040 comp rx=0 tx=0).stop 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 10 monclient: shutdown 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 msgr2=0x7f6870099000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f686400d860 tx=0x7f6864008040 comp rx=0 tx=0).stop 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 shutdown_connections 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f685c035b10 0x7f685c037fc0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6860006f00 tx=0x7f6860008040 comp rx=0 tx=0).stop 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68700a0ba0 0x7f687008e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687009e780 0x7f687008db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 --2- 172.21.15.150:0/1209398610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687009d000 0x7f6870099000 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 >> 172.21.15.150:0/1209398610 conn(0x7f68700a5830 msgr2=0x7f687009b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:45.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 shutdown_connections 2024-09-18T10:10:45.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.674+0000 7f6875d83700 1 -- 172.21.15.150:0/1209398610 wait complete. 2024-09-18T10:10:45.681 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fooo' does not exist 2024-09-18T10:10:45.693 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool delete fuggg fuggg --yes-i-really-really-mean-it 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: get_monmap_and_config 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: build_initial_monmap 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: monmap: 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:45.801531+0000 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:45.801531+0000 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:45.802 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:45.803 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:45.803 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:45.803 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 Processor -- start 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 -- start start 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: init 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5d80a0290 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5d80a0d70 0x7fb5d809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-c con 0x7fb5d80a0d70 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5d8137a20 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d809e360 con 0x7fb5d80a0290 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d809da50 con 0x7fb5d8137a20 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d809ef80 con 0x7fb5d80a0d70 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5dd279700 10 monclient(hunting): authenticate will time out at 927.895073s 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d77fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb5d8137a20 0x7fb5d8139e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38266/0 (socket says 172.21.15.150:38266) 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d77fe700 1 -- 172.21.15.150:0/83890902 learned_addr learned my addr 172.21.15.150:0/83890902 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 1 --2- 172.21.15.150:0/83890902 >> v2:172.21.15.150:3300/0 conn(0x7fb5d80a0d70 0x7fb5d809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): get_auth_request con 0x7fb5d80a0d70 auth_method 0 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_done global_id 14404 payload 274 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 1 -- 172.21.15.150:0/83890902 >> v1:172.21.15.150:6789/0 conn(0x7fb5d8137a20 legacy=0x7fb5d8139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 1 -- 172.21.15.150:0/83890902 >> v1:172.21.15.142:6789/0 conn(0x7fb5d80a0290 legacy=0x7fb5d80a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 1 monclient: found mon.noname-c 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.798+0000 7fb5d67fc700 1 -- 172.21.15.150:0/83890902 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d809dd30 con 0x7fb5d80a0d70 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d67fc700 1 -- 172.21.15.150:0/83890902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 msgr2=0x7fb5d809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d67fc700 1 --2- 172.21.15.150:0/83890902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d809ea40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80098e0 tx=0x7fb5c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2902a440b4d12982 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 1 -- 172.21.15.150:0/83890902 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c800ba70 con 0x7fb5d80a0d70 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:45.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: dump: 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:10:45.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 5 monclient: authenticate success, global_id 14404 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 1 -- 172.21.15.150:0/83890902 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c800bbd0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 4 set_mon_vals no callback set 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 1 -- 172.21.15.150:0/83890902 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8011420 con 0x7fb5d80a0d70 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5d5ffb700 10 monclient: dump: 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.811 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 10 monclient: shutdown 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 -- 172.21.15.150:0/83890902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 msgr2=0x7fb5d809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 --2- 172.21.15.150:0/83890902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d809ea40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80098e0 tx=0x7fb5c80091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 -- 172.21.15.150:0/83890902 shutdown_connections 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 --2- 172.21.15.150:0/83890902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d809ea40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 -- 172.21.15.150:0/83890902 >> 172.21.15.150:0/83890902 conn(0x7fb5d80a58a0 msgr2=0x7fb5d80a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 -- 172.21.15.150:0/83890902 shutdown_connections 2024-09-18T10:10:45.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.802+0000 7fb5dd279700 1 -- 172.21.15.150:0/83890902 wait complete. 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient: build_initial_monmap 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient: monmap: 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 Processor -- start 2024-09-18T10:10:45.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 -- start start 2024-09-18T10:10:45.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient: init 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5d80a0290 0x7fb5d80984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-c con 0x7fb5d80a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-b con 0x7fb5d80a0d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8137a20 0x7fb5d8098f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): picked mon.noname-a con 0x7fb5d8137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8130f20 con 0x7fb5d8137a20 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d812e1d0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d812f8f0 con 0x7fb5d80a0290 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5dd279700 10 monclient(hunting): authenticate will time out at 927.902153s 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47402/0 (socket says 172.21.15.150:47402) 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 -- 172.21.15.150:0/2747354517 learned_addr learned my addr 172.21.15.150:0/2747354517 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): get_auth_request con 0x7fb5d80a0d70 auth_method 0 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8137a20 0x7fb5d8098f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): get_auth_request con 0x7fb5d8137a20 auth_method 0 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5d80a0290 0x7fb5d80984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb5d80a0290 auth_method 0 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:45.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient(hunting): handle_auth_done global_id 14407 payload 824 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 -- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5d80a0290 msgr2=0x7fb5d80984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5d80a0290 0x7fb5d80984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 -- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8137a20 msgr2=0x7fb5d8098f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8137a20 0x7fb5d8098f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:10:45.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 monclient: found mon.noname-b 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.806+0000 7fb5d67fc700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c8009420 con 0x7fb5d80a0d70 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5d67fc700 1 -- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 msgr2=0x7fb5d8098a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5d67fc700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800b3c0 tx=0x7fb5c8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c801a460 con 0x7fb5d80a0d70 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: dump: 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.822 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 5 monclient: authenticate success, global_id 14407 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c801a5c0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _renew_subs 2024-09-18T10:10:45.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c801a960 con 0x7fb5d80a0d70 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d809d3f0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5d57fa700 4 set_mon_vals no callback set 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 10 monclient: dump: 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:45.824 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _renew_subs 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d809c130 con 0x7fb5d80a0d70 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb5c801ad50 con 0x7fb5d80a0d70 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 0x7fb5c4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fb5c804a2f0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5d6ffd700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 0x7fb5c4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:45.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.810+0000 7fb5d6ffd700 10 monclient: get_auth_request con 0x7fb5c4035ac0 auth_method 0 2024-09-18T10:10:45.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.814+0000 7fb5d6ffd700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 0x7fb5c4037f70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009af0 tx=0x7fb5cc009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:45.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.822+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:45.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.822+0000 7fb5beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:45.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.822+0000 7fb5beffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:45.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:45.979+0000 7fa71d7fa700 10 monclient: tick 2024-09-18T10:10:45.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:45.979+0000 7fa71d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:45.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:45.979+0000 7fa71d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:45.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.982+0000 7fb5dd279700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.986+0000 7fb5dd279700 10 monclient: _send_command 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:45.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.986+0000 7fb5dd279700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:45.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:45.986+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa7281416f0 auth_method 0 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 10 monclient(hunting): handle_auth_done global_id 24104 payload 824 2024-09-18T10:10:46.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa72812b960 con 0x7fa7281416f0 2024-09-18T10:10:46.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7100148f0 tx=0x7fa710009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710005b10 con 0x7fa7281416f0 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.003+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.008 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa710005c70 con 0x7fa7281416f0 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7100148f0 tx=0x7fa710009040 comp rx=0 tx=0).stop 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 msgr2=0x7fa72806d8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:46.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.007+0000 7fa71e7fc700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7281416f0 0x7fa72806d8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72c823700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72c823700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 0x7fa72802a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72c823700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72809bea0 con 0x7fa728029d00 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 0x7fa72802a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient: get_auth_request con 0x7fa728029d00 auth_method 0 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 10 monclient(hunting): handle_auth_done global_id 24107 payload 824 2024-09-18T10:10:46.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.019+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7280696a0 con 0x7fa728029d00 2024-09-18T10:10:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa72659c700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 msgr2=0x7fa72802a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa72659c700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 0x7fa72802a100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa71001c000 tx=0x7fa71001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71001f940 con 0x7fa728029d00 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.027 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.028 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fa710005900 con 0x7fa728029d00 2024-09-18T10:10:46.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 msgr2=0x7fa72802a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 0x7fa72802a100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa71001c000 tx=0x7fa71001f000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 msgr2=0x7fa72802a100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.023+0000 7fa71e7fc700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa728029d00 0x7fa72802a100 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:46.029 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.030 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.031 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:46.032 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:46.033 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.034 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.037 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:46.038 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:46.039 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.040 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:46.041 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:46.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.042 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.042 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.042 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:46.042 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa708035ac0 msgr2=0x7fa708037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa708035ac0 0x7fa708037f70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa718006f30 tx=0x7fa718008040 comp rx=0 tx=0).stop 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 10 monclient: shutdown 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 msgr2=0x7fa7281344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 --2- 172.21.15.142:0/3186809338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa728137a00 0x7fa7281344c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa72806a910 tx=0x7fa720019000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 reap_dead start 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa726d9d700 1 -- 172.21.15.142:0/3186809338 reap_dead start 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 shutdown_connections 2024-09-18T10:10:46.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 >> 172.21.15.142:0/3186809338 conn(0x7fa7280a5880 msgr2=0x7fa728136ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 shutdown_connections 2024-09-18T10:10:46.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.027+0000 7fa72c823700 1 -- 172.21.15.142:0/3186809338 wait complete. 2024-09-18T10:10:46.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:46.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:10:46.045 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 10 monclient: get_monmap_and_config 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 10 monclient: monmap: 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.154 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:46.153817+0000 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:46.153817+0000 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:46.155 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 1 Processor -- start 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 1 -- start start 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.151+0000 7f1c096c0700 10 monclient: init 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c040999e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-b con 0x7f1c0409c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:46.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c04137910 0x7f1c04139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-c con 0x7f1c04137910 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c040a20c0 con 0x7f1c040999e0 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0409a000 con 0x7f1c04137910 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040a1c80 con 0x7f1c0409c410 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient(hunting): authenticate will time out at 935.080307s 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c03fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1c040999e0 0x7f1c040a0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60352/0 (socket says 172.21.15.142:60352) 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55638/0 (socket says 172.21.15.142:55638) 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c03fff700 1 -- 172.21.15.142:0/252223544 learned_addr learned my addr 172.21.15.142:0/252223544 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:46.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): get_auth_request con 0x7f1c0409c410 auth_method 0 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 1 --2- 172.21.15.142:0/252223544 >> v2:172.21.15.150:3300/0 conn(0x7f1c04137910 0x7f1c04139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): get_auth_request con 0x7f1c04137910 auth_method 0 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_done global_id 24110 payload 274 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/252223544 >> v2:172.21.15.150:3300/0 conn(0x7f1c04137910 msgr2=0x7f1c04139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 --2- 172.21.15.142:0/252223544 >> v2:172.21.15.150:3300/0 conn(0x7f1c04137910 0x7f1c04139cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/252223544 >> v1:172.21.15.142:6789/0 conn(0x7f1c040999e0 legacy=0x7f1c040a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/252223544 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0409a580 con 0x7f1c0409c410 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/252223544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 msgr2=0x7f1c040a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1bfb7fe700 1 --2- 172.21.15.142:0/252223544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009910 tx=0x7f1bec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d23446578321561 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 1 -- 172.21.15.142:0/252223544 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00b850 con 0x7f1c0409c410 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: dump: 2024-09-18T10:10:46.165 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 5 monclient: authenticate success, global_id 24110 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:46.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 1 -- 172.21.15.142:0/252223544 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec00b9b0 con 0x7f1c0409c410 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c096c0700 4 set_mon_vals no callback set 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 1 -- 172.21.15.142:0/252223544 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00bd70 con 0x7f1c0409c410 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.155+0000 7f1c037fe700 10 monclient: dump: 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.167 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: shutdown 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- 172.21.15.142:0/252223544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 msgr2=0x7f1c040a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- 172.21.15.142:0/252223544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009910 tx=0x7f1bec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- 172.21.15.142:0/252223544 shutdown_connections 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- 172.21.15.142:0/252223544 >> v2:172.21.15.150:3300/0 conn(0x7f1c04137910 0x7f1c04139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- 172.21.15.142:0/252223544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c040a0890 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- 172.21.15.142:0/252223544 >> 172.21.15.142:0/252223544 conn(0x7f1c040a5790 msgr2=0x7f1c040a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- 172.21.15.142:0/252223544 shutdown_connections 2024-09-18T10:10:46.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- 172.21.15.142:0/252223544 wait complete. 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: monmap: 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 Processor -- start 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- start start 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: init 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c040999e0 0x7f1c0412be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-a con 0x7f1c040999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c0412c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-c con 0x7f1c0409c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c04137910 0x7f1c0412c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): picked mon.noname-b con 0x7f1c04137910 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0412e230 con 0x7f1c040999e0 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0412fa80 con 0x7f1c04137910 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0409f0d0 con 0x7f1c0409c410 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c096c0700 10 monclient(hunting): authenticate will time out at 935.086960s 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c040999e0 0x7f1c0412be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c040999e0 0x7f1c0412be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36240/0 (socket says 172.21.15.142:36240) 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c0412c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 1 -- 172.21.15.142:0/1542070417 learned_addr learned my addr 172.21.15.142:0/1542070417 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): get_auth_request con 0x7f1c040999e0 auth_method 0 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): get_auth_request con 0x7f1c0409c410 auth_method 0 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c04137910 0x7f1c0412c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): get_auth_request con 0x7f1c04137910 auth_method 0 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1c08ebf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient(hunting): handle_auth_done global_id 24113 payload 824 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c04137910 msgr2=0x7f1c0412c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c04137910 0x7f1c0412c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c040999e0 msgr2=0x7f1c0412be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c040999e0 0x7f1c0412be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.159+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec009420 con 0x7f1c0409c410 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1bfb7fe700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 msgr2=0x7f1c0412c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1bfb7fe700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c0412c360 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1bec015000 tx=0x7f1bec016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec006a90 con 0x7f1c0409c410 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: dump: 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 5 monclient: authenticate success, global_id 24113 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec020070 con 0x7f1c0409c410 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec011d50 con 0x7f1c0409c410 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 10 monclient: _renew_subs 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c027fc700 4 set_mon_vals no callback set 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0412f7a0 con 0x7f1c0409c410 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c00ff9700 10 monclient: dump: 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 10 monclient: _renew_subs 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.163+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0412f4c0 con 0x7f1c0409c410 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c096c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c096c0700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 0x7f1c0412cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040647f0 con 0x7f1c0402ae80 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 0x7f1c0412cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient: get_auth_request con 0x7f1c0402ae80 auth_method 0 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1bec022030 con 0x7f1c0409c410 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c00ff9700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1be4035b10 0x7f1be4037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 10 monclient(hunting): handle_auth_done global_id 14416 payload 824 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c0412fbf0 con 0x7f1c0402ae80 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f1bec049ef0 con 0x7f1c0409c410 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c08ebf700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1be4035b10 0x7f1be4037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c08ebf700 10 monclient: get_auth_request con 0x7f1be4035b10 auth_method 0 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 msgr2=0x7f1c0412cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c03fff700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 0x7f1c0412cde0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400eac0 tx=0x7f1bf4004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf400cc40 con 0x7f1c0402ae80 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.167+0000 7f1c00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 10 monclient: dump: 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.182 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c08ebf700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1be4035b10 0x7f1be4037fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009fd0 tx=0x7f1bfc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1bf400cda0 con 0x7f1c0402ae80 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 msgr2=0x7f1c0412cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 0x7f1c0412cde0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400eac0 tx=0x7f1bf4004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 msgr2=0x7f1c0412cde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:46.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.171+0000 7f1c00ff9700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0402ae80 0x7f1c0412cde0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400eac0 tx=0x7f1bf4004a80 comp rx=0 tx=0).stop 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c096c0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c096c0700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 0x7f1c0405ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c096c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0406e220 con 0x7f1c0405e890 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 0x7f1c0405ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient: get_auth_request con 0x7f1c0405e890 auth_method 0 2024-09-18T10:10:46.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.179+0000 7f1c03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c03fff700 10 monclient(hunting): handle_auth_done global_id 14419 payload 824 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c03fff700 1 -- 172.21.15.142:0/1542070417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1c04072a80 con 0x7f1c0405e890 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c03fff700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 msgr2=0x7f1c0405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c03fff700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 0x7f1c0405ec90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40070c0 tx=0x7f1bf4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4013c20 con 0x7f1c0405e890 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 10 monclient: dump: 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.187 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.188 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f1bf4025070 con 0x7f1c0405e890 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 msgr2=0x7f1c0405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 0x7f1c0405ec90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40070c0 tx=0x7f1bf4016000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 msgr2=0x7f1c0405ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.183+0000 7f1c00ff9700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c0405e890 0x7f1c0405ec90 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40070c0 tx=0x7f1bf4016000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.189 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:46.190 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.191 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:46.192 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:46.193 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:46.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.195 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:46.196 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:46.198 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:46.199 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.200 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:10:46.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:46.202 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:46.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:46.202 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:46.202 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:46.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1be4035b10 msgr2=0x7f1be4037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1be4035b10 0x7f1be4037fc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009fd0 tx=0x7f1bfc00b040 comp rx=0 tx=0).stop 2024-09-18T10:10:46.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 10 monclient: shutdown 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 msgr2=0x7f1c0412c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 --2- 172.21.15.142:0/1542070417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c0409c410 0x7f1c0412c360 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1bec015000 tx=0x7f1bec016000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c08ebf700 1 -- 172.21.15.142:0/1542070417 reap_dead start 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c08ebf700 1 -- 172.21.15.142:0/1542070417 reap_dead start 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 shutdown_connections 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 >> 172.21.15.142:0/1542070417 conn(0x7f1c040a5790 msgr2=0x7f1c0409e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 shutdown_connections 2024-09-18T10:10:46.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.187+0000 7f1c096c0700 1 -- 172.21.15.142:0/1542070417 wait complete. 2024-09-18T10:10:46.205 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:46.205 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:10:46.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:10:46.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.310+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v43) v1 ==== 162+0+0 (secure 0 0 0) 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:46.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.310+0000 7fb5beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:46.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.310+0000 7fb5beffd700 10 monclient: _finish_command 2 = system:0 pool 'fuggg' does not exist 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.315+0000 7fb36cb25700 10 monclient: get_monmap_and_config 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.315+0000 7fb36cb25700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.315+0000 7fb36cb25700 10 monclient: monmap: 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:46.318580+0000 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:46.318580+0000 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:46.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:46.320 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:46.320 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:46.320 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:46.320 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 Processor -- start 2024-09-18T10:10:46.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 -- start start 2024-09-18T10:10:46.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient: init 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-b con 0x7fb36809b360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb36809e710 0x7fb36809d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-c con 0x7fb36809e710 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:46.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb36809b730 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368099ab0 con 0x7fb36809b730 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36809abf0 con 0x7fb36809e710 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36809e4d0 con 0x7fb36809b360 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36cb25700 10 monclient(hunting): authenticate will time out at 935.245832s 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55662/0 (socket says 172.21.15.142:55662) 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 -- 172.21.15.142:0/1087913262 learned_addr learned my addr 172.21.15.142:0/1087913262 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:46.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): get_auth_request con 0x7fb36809b360 auth_method 0 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 1 --2- 172.21.15.142:0/1087913262 >> v2:172.21.15.150:3300/0 conn(0x7fb36809e710 0x7fb36809d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): get_auth_request con 0x7fb36809e710 auth_method 0 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient(hunting): handle_auth_done global_id 24116 payload 274 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 -- 172.21.15.142:0/1087913262 >> v2:172.21.15.150:3300/0 conn(0x7fb36809e710 msgr2=0x7fb36809d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 --2- 172.21.15.142:0/1087913262 >> v2:172.21.15.150:3300/0 conn(0x7fb36809e710 0x7fb36809d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 -- 172.21.15.142:0/1087913262 >> v1:172.21.15.150:6789/0 conn(0x7fb36809b730 legacy=0x7fb368135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 monclient: found mon.noname-b 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.319+0000 7fb36659c700 1 -- 172.21.15.142:0/1087913262 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36809eb90 con 0x7fb36809b360 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36659c700 1 -- 172.21.15.142:0/1087913262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 msgr2=0x7fb36809cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36659c700 1 --2- 172.21.15.142:0/1087913262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb3500099b0 tx=0x7fb350009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8e337d71744ea2b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 1 -- 172.21.15.142:0/1087913262 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb350009ea0 con 0x7fb36809b360 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: dump: 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 1 -- 172.21.15.142:0/1087913262 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35000ba70 con 0x7fb36809b360 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 5 monclient: authenticate success, global_id 24116 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:46.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 1 -- 172.21.15.142:0/1087913262 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35000be30 con 0x7fb36809b360 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 4 set_mon_vals no callback set 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb365d9b700 10 monclient: dump: 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 10 monclient: shutdown 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- 172.21.15.142:0/1087913262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 msgr2=0x7fb36809cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 --2- 172.21.15.142:0/1087913262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb3500099b0 tx=0x7fb350009250 comp rx=0 tx=0).stop 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- 172.21.15.142:0/1087913262 shutdown_connections 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 --2- 172.21.15.142:0/1087913262 >> v2:172.21.15.150:3300/0 conn(0x7fb36809e710 0x7fb36809d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 --2- 172.21.15.142:0/1087913262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809b360 0x7fb36809cfc0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- 172.21.15.142:0/1087913262 >> 172.21.15.142:0/1087913262 conn(0x7fb36806a2e0 msgr2=0x7fb3680a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- 172.21.15.142:0/1087913262 shutdown_connections 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- 172.21.15.142:0/1087913262 wait complete. 2024-09-18T10:10:46.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 10 monclient: monmap: 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 Processor -- start 2024-09-18T10:10:46.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 1 -- start start 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.323+0000 7fb36cb25700 10 monclient: init 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-a con 0x7fb36809b360 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36809b730 0x7fb368141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-b con 0x7fb36809b730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 0x7fb368096550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): picked mon.noname-c con 0x7fb36809e710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368091aa0 con 0x7fb36809b360 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368099f10 con 0x7fb36809b730 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb368098e20 con 0x7fb36809e710 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient(hunting): authenticate will time out at 935.252320s 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36244/0 (socket says 172.21.15.142:36244) 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 0x7fb368096550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 1 -- 172.21.15.142:0/2834952580 learned_addr learned my addr 172.21.15.142:0/2834952580 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): get_auth_request con 0x7fb36809b360 auth_method 0 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): get_auth_request con 0x7fb36809e710 auth_method 0 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36809b730 0x7fb368141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): get_auth_request con 0x7fb36809b730 auth_method 0 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient(hunting): handle_auth_done global_id 24119 payload 824 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 -- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36809b730 msgr2=0x7fb368141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36809b730 0x7fb368141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 -- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 msgr2=0x7fb368141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 monclient: found mon.noname-c 2024-09-18T10:10:46.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 -- 172.21.15.142:0/2834952580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500094f0 con 0x7fb36809e710 2024-09-18T10:10:46.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36659c700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 -- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 msgr2=0x7fb368096550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb366d9d700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 0x7fb368096550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb36000c870 tx=0x7fb360002620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360012840 con 0x7fb36809e710 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: dump: 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.341 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 5 monclient: authenticate success, global_id 24119 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3600129c0 con 0x7fb36809e710 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _renew_subs 2024-09-18T10:10:46.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680980a0 con 0x7fb36809e710 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb360012d80 con 0x7fb36809e710 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb35e7fc700 10 monclient: dump: 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.343 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36559a700 4 set_mon_vals no callback set 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _renew_subs 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.327+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb368090560 con 0x7fb36809e710 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb360011590 con 0x7fb36809e710 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36cb25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36cb25700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36cb25700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb368072a80 con 0x7fb36809e710 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb35e7fc700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 0x7fb348037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fb3600484c0 con 0x7fb36809e710 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36659c700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 0x7fb348037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.331+0000 7fb36659c700 10 monclient: get_auth_request con 0x7fb348035b10 auth_method 0 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.335+0000 7fb36659c700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 0x7fb348037fc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb350009340 tx=0x7fb350016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.339+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb360010ac0 con 0x7fb36809e710 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.339+0000 7fb35e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.339+0000 7fb35e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:46.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.418+0000 7fb5dd279700 10 start_mon_command cmd=[{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:46.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.418+0000 7fb5dd279700 10 monclient: _send_command 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:46.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.418+0000 7fb5dd279700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.418+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:46.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.563+0000 7fb36cb25700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:10:46.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.563+0000 7fb36cb25700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:10:46.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.563+0000 7fb36cb25700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:46.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.563+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb368067bf0 con 0x7fb36809e710 2024-09-18T10:10:46.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.566+0000 7fb5beffd700 1 -- 172.21.15.150:0/2747354517 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}]=0 pool 'fuggg' does not exist v43) v1 ==== 162+0+0 (secure 0 0 0) 0x7fb5d81303e0 con 0x7fb5d80a0d70 2024-09-18T10:10:46.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.566+0000 7fb5beffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool delete", "pool": "fuggg", "pool2": "fuggg", "yes_i_really_really_mean_it": true}] 2024-09-18T10:10:46.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.566+0000 7fb5beffd700 10 monclient: _finish_command 3 = system:0 pool 'fuggg' does not exist 2024-09-18T10:10:46.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.567+0000 7fb35e7fc700 1 -- 172.21.15.142:0/2834952580 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb360016070 con 0x7fb36809e710 2024-09-18T10:10:46.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.567+0000 7fb35e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:10:46.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.567+0000 7fb35e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 msgr2=0x7fb348037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 0x7fb348037fc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb350009340 tx=0x7fb350016000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 10 monclient: shutdown 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 msgr2=0x7fb368096550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 0x7fb368096550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb36000c870 tx=0x7fb360002620 comp rx=0 tx=0).stop 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 shutdown_connections 2024-09-18T10:10:46.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb348035b10 0x7fb348037fc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb36809e710 0x7fb368096550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36809b730 0x7fb368141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 --2- 172.21.15.142:0/2834952580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb36809b360 0x7fb368141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 >> 172.21.15.142:0/2834952580 conn(0x7fb36806a2e0 msgr2=0x7fb368070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 shutdown_connections 2024-09-18T10:10:46.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:46.571+0000 7fb36cb25700 1 -- 172.21.15.142:0/2834952580 wait complete. 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 msgr2=0x7fb5c4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 0x7fb5c4037f70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009af0 tx=0x7fb5cc009060 comp rx=0 tx=0).stop 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 10 monclient: shutdown 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 msgr2=0x7fb5d8098a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800b3c0 tx=0x7fb5c8016000 comp rx=0 tx=0).stop 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 shutdown_connections 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb5c4035ac0 0x7fb5c4037f70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5d8137a20 0x7fb5d8098f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5d80a0d70 0x7fb5d8098a10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 --2- 172.21.15.150:0/2747354517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5d80a0290 0x7fb5d80984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 >> 172.21.15.150:0/2747354517 conn(0x7fb5d80a58a0 msgr2=0x7fb5d809acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 shutdown_connections 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.570+0000 7fb5dd279700 1 -- 172.21.15.150:0/2747354517 wait complete. 2024-09-18T10:10:46.576 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fuggg' does not exist 2024-09-18T10:10:46.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:10:46.587 INFO:tasks.workunit.client.0.smithi150.stdout:OK 2024-09-18T10:10:46.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo OK 2024-09-18T10:10:46.588 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:10:46.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:10:46.601 INFO:tasks.workunit:Running workunits matching mon/crush_ops.sh on client.0... 2024-09-18T10:10:46.602 INFO:tasks.workunit:Running workunit mon/crush_ops.sh... 2024-09-18T10:10:46.602 DEBUG:teuthology.orchestra.run.smithi150:workunit test mon/crush_ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh 2024-09-18T10:10:46.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:46.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=11,kv=6,logm=73} crc {auth=2179186541,config=3140514009,health=588522927,kv=1479720900,logm=1721862951}) 2024-09-18T10:10:46.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=11,kv=8,logm=71} crc {auth=3911066050,config=3140514009,health=2681726216,kv=4261215878,logm=1606035626}) 2024-09-18T10:10:46.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:46.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=11,kv=6,logm=73} crc {auth=2179186541,config=3140514009,health=588522927,kv=1479720900,logm=1721862951}) 2024-09-18T10:10:46.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.599+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=1510965087,config=3140514009,health=3897446751,kv=4261215878,logm=2425103087}) 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101537255}) 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615280131}) 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101537255}) 2024-09-18T10:10:46.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:46.603+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=211345447}) 2024-09-18T10:10:46.659 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush dump 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.754+0000 7f3567a1a700 10 monclient: get_monmap_and_config 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.754+0000 7f3567a1a700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient: monmap: 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:46.760050+0000 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:46.760050+0000 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.761 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 Processor -- start 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 -- start start 2024-09-18T10:10:46.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient: init 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f356009ac60 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f356006f9e0 0x7f356009c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-c con 0x7f356006f9e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3560135710 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356009b9e0 con 0x7f3560135710 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356009b780 con 0x7f356009ac60 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356009b030 con 0x7f356006f9e0 2024-09-18T10:10:46.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3567a1a700 10 monclient(hunting): authenticate will time out at 928.853388s 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f356006f9e0 0x7f356009c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f356006f9e0 0x7f356009c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47404/0 (socket says 172.21.15.150:47404) 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 1 -- 172.21.15.150:0/1699521081 learned_addr learned my addr 172.21.15.150:0/1699521081 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): get_auth_request con 0x7f356006f9e0 auth_method 0 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.758+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f35657b6700 1 -- 172.21.15.150:0/1699521081 >> v1:172.21.15.150:6789/0 conn(0x7f356009ac60 legacy=0x7f356009c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3565fb7700 1 -- 172.21.15.150:0/1699521081 >> v1:172.21.15.142:6789/0 conn(0x7f3560135710 legacy=0x7f3560137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 10 monclient(hunting): handle_auth_done global_id 14428 payload 274 2024-09-18T10:10:46.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 -- 172.21.15.150:0/1699521081 >> v1:172.21.15.150:6789/0 conn(0x7f356009ac60 legacy=0x7f356009c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 -- 172.21.15.150:0/1699521081 >> v1:172.21.15.142:6789/0 conn(0x7f3560135710 legacy=0x7f3560137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 monclient: found mon.noname-c 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 -- 172.21.15.150:0/1699521081 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3560098270 con 0x7f356006f9e0 2024-09-18T10:10:46.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 -- 172.21.15.150:0/1699521081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356006f9e0 msgr2=0x7f356009c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3564fb5700 1 --2- 172.21.15.150:0/1699521081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356006f9e0 0x7f356009c9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f355800b0d0 tx=0x7f3558009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b347a6397cd9f00 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 1 -- 172.21.15.150:0/1699521081 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355800e070 con 0x7f356006f9e0 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: dump: 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.769 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 5 monclient: authenticate success, global_id 14428 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 1 -- 172.21.15.150:0/1699521081 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355800bd90 con 0x7f356006f9e0 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 4 set_mon_vals no callback set 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 1 -- 172.21.15.150:0/1699521081 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558007650 con 0x7f356006f9e0 2024-09-18T10:10:46.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3557fff700 10 monclient: dump: 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.771 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 10 monclient: shutdown 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 -- 172.21.15.150:0/1699521081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356006f9e0 msgr2=0x7f356009c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 --2- 172.21.15.150:0/1699521081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356006f9e0 0x7f356009c9f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f355800b0d0 tx=0x7f3558009840 comp rx=0 tx=0).stop 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 -- 172.21.15.150:0/1699521081 shutdown_connections 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 --2- 172.21.15.150:0/1699521081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356006f9e0 0x7f356009c9f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 -- 172.21.15.150:0/1699521081 >> 172.21.15.150:0/1699521081 conn(0x7f35600a1710 msgr2=0x7f35600a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:46.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 -- 172.21.15.150:0/1699521081 shutdown_connections 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.762+0000 7f3567a1a700 1 -- 172.21.15.150:0/1699521081 wait complete. 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient: build_initial_monmap 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient: monmap: 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:46.773 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 Processor -- start 2024-09-18T10:10:46.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 -- start start 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient: init 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 0x7f3560134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-a con 0x7f356006f9e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-b con 0x7f356009ac60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3560135710 0x7f356008d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): picked mon.noname-c con 0x7f3560135710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3560098480 con 0x7f356006f9e0 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356006ef20 con 0x7f356009ac60 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3560090af0 con 0x7f3560135710 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3567a1a700 10 monclient(hunting): authenticate will time out at 928.861712s 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47416/0 (socket says 172.21.15.150:47416) 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 -- 172.21.15.150:0/4101660178 learned_addr learned my addr 172.21.15.150:0/4101660178 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): get_auth_request con 0x7f356009ac60 auth_method 0 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 0x7f3560134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): get_auth_request con 0x7f356006f9e0 auth_method 0 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3560135710 0x7f356008d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): get_auth_request con 0x7f3560135710 auth_method 0 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3565fb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient(hunting): handle_auth_done global_id 14431 payload 824 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 -- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3560135710 msgr2=0x7f356008d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3560135710 0x7f356008d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 -- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 msgr2=0x7f3560134930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 0x7f3560134930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient: _finish_hunting 0 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 monclient: found mon.noname-b 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:46.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f3564fb5700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558009ae0 con 0x7f356009ac60 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.766+0000 7f35657b6700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 0x7f3560134930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3564fb5700 1 -- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 msgr2=0x7f3560093310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3564fb5700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f355800b820 tx=0x7f3558018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3558007e30 con 0x7f356009ac60 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: dump: 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.782 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: _finish_auth 0 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35580070d0 con 0x7f356009ac60 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580266b0 con 0x7f356009ac60 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:46.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 10 monclient: dump: 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 5 monclient: authenticate success, global_id 14431 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _renew_subs 2024-09-18T10:10:46.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3560135500 con 0x7f356009ac60 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35577fe700 4 set_mon_vals no callback set 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _renew_subs 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356008f640 con 0x7f356009ac60 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f355801d420 con 0x7f356009ac60 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3560063ed0 con 0x7f356009ac60 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 0x7f354c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f355804a2c0 con 0x7f356009ac60 2024-09-18T10:10:46.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35657b6700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 0x7f354c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:46.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.770+0000 7f35657b6700 10 monclient: get_auth_request con 0x7f354c035ac0 auth_method 0 2024-09-18T10:10:46.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.774+0000 7f35657b6700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 0x7f354c037f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f355c00f380 tx=0x7f355c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:46.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.778+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3558011990 con 0x7f356009ac60 2024-09-18T10:10:46.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.778+0000 7f35557fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:46.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.778+0000 7f35557fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.946+0000 7f3567a1a700 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-18T10:10:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.946+0000 7f3567a1a700 10 monclient: _send_command 2 [{"prefix": "osd crush dump"}] 2024-09-18T10:10:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.946+0000 7f3567a1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:46.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.946+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f3560095c50 con 0x7f356009ac60 2024-09-18T10:10:46.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.950+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v43) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f355802b020 con 0x7f356009ac60 2024-09-18T10:10:46.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.950+0000 7f35557fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush dump"}] 2024-09-18T10:10:46.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:46.950+0000 7f35557fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:47.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f3567a1a700 10 start_mon_command cmd=[{"prefix": "osd crush dump"}] 2024-09-18T10:10:47.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f3567a1a700 10 monclient: _send_command 3 [{"prefix": "osd crush dump"}] 2024-09-18T10:10:47.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f3567a1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush dump"} v 0) v1 -- 0x7f3560090730 con 0x7f356009ac60 2024-09-18T10:10:47.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f35557fa700 1 -- 172.21.15.150:0/4101660178 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush dump"}]=0 v43) v1 ==== 62+0+7238 (secure 0 0 0) 0x7f3558021070 con 0x7f356009ac60 2024-09-18T10:10:47.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f35557fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush dump"}] 2024-09-18T10:10:47.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.050+0000 7f35557fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:47.056 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "devices": [ 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 0, 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.0", 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 1, 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.1", 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.057 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.2", 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.3", 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 4, 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.4", 2024-09-18T10:10:47.058 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd.5", 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "class": "ssd" 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "types": [ 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 0, 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "osd" 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.059 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 1, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "host" 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 2, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "chassis" 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 3, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "rack" 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.060 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 4, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "row" 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 5, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "pdu" 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 6, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "pod" 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 7, 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "room" 2024-09-18T10:10:47.061 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 8, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "datacenter" 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 9, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "zone" 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 10, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "region" 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.062 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 11, 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "root" 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "buckets": [ 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -1, 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "default", 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 11, 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "root", 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 34560, 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.063 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -3, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -5, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.064 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -2, 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "default~ssd", 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 11, 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "root", 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 34560, 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -4, 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.065 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -6, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -3, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "smithi142", 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 1, 2024-09-18T10:10:47.066 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "host", 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 0, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 1, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.067 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 2 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -4, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "smithi142~ssd", 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 1, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "host", 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.068 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 0, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 1, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.069 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 2, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 2 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -5, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "smithi150", 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 1, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "host", 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.070 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 4, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.071 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 2 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "id": -6, 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "name": "smithi150~ssd", 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "type_id": 1, 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "type_name": "host", 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 17280, 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "alg": "straw2", 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "hash": "rjenkins1", 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "items": [ 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.072 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 5, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 0 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 3, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 1 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "id": 4, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "weight": 5760, 2024-09-18T10:10:47.073 INFO:tasks.workunit.client.0.smithi150.stdout: "pos": 2 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "rules": [ 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_id": 0, 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_name": "replicated_rule", 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "type": 1, 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "steps": [ 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "take", 2024-09-18T10:10:47.074 INFO:tasks.workunit.client.0.smithi150.stdout: "item": -1, 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: "item_name": "default" 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "choose_firstn", 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 0, 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: "type": "osd" 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "emit" 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.075 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_id": 1, 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_name": "erasure-code", 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "type": 3, 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "steps": [ 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "set_chooseleaf_tries", 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 5 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "set_choose_tries", 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 100 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.076 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "take", 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "item": -1, 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "item_name": "default" 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "choose_indep", 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 0, 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "type": "osd" 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "emit" 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.077 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "tunables": { 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "choose_local_tries": 0, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "choose_local_fallback_tries": 0, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "choose_total_tries": 50, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "chooseleaf_descend_once": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "chooseleaf_vary_r": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "chooseleaf_stable": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "straw_calc_version": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "allowed_bucket_algs": 54, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "profile": "jewel", 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "optimal_tunables": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "legacy_tunables": 0, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "minimum_required_version": "jewel", 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "require_feature_tunables": 1, 2024-09-18T10:10:47.078 INFO:tasks.workunit.client.0.smithi150.stdout: "require_feature_tunables2": 1, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "has_v2_rules": 1, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "require_feature_tunables3": 1, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "has_v3_rules": 0, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "has_v4_buckets": 1, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "require_feature_tunables5": 1, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "has_v5_rules": 0 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: "choose_args": {} 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout:} 2024-09-18T10:10:47.079 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-18T10:10:47.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 msgr2=0x7f354c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 0x7f354c037f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f355c00f380 tx=0x7f355c009040 comp rx=0 tx=0).stop 2024-09-18T10:10:47.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 10 monclient: shutdown 2024-09-18T10:10:47.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 msgr2=0x7f3560093310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f355800b820 tx=0x7f3558018000 comp rx=0 tx=0).stop 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 shutdown_connections 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f354c035ac0 0x7f354c037f70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3560135710 0x7f356008d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f356009ac60 0x7f3560093310 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 --2- 172.21.15.150:0/4101660178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356006f9e0 0x7f3560134930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 >> 172.21.15.150:0/4101660178 conn(0x7f35600a1710 msgr2=0x7f356009df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 shutdown_connections 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.054+0000 7f3567a1a700 1 -- 172.21.15.150:0/4101660178 wait complete. 2024-09-18T10:10:47.081 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule dump 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: get_monmap_and_config 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: monmap: 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:47.165596+0000 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:47.165596+0000 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.166 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:47.167 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:47.167 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:47.167 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:47.167 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:47.167 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 Processor -- start 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 -- start start 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: init 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7d80a6090 0x7ff7d809d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-c con 0x7ff7d80a6090 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7d809e460 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7d80a0a80 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d80a6460 con 0x7ff7d809e460 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d809fd00 con 0x7ff7d80a0a80 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80a6850 con 0x7ff7d80a6090 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7df1bc700 10 monclient(hunting): authenticate will time out at 929.259342s 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dcf58700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7d80a6090 0x7ff7d809d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dd759700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff7d80a0a80 0x7ff7d8139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38276/0 (socket says 172.21.15.150:38276) 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dd759700 1 -- 172.21.15.150:0/550389086 learned_addr learned my addr 172.21.15.150:0/550389086 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dcf58700 10 monclient(hunting): get_auth_request con 0x7ff7d80a6090 auth_method 0 2024-09-18T10:10:47.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dcf58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dcf58700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.162+0000 7ff7dcf58700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dd759700 1 -- 172.21.15.150:0/550389086 >> v1:172.21.15.150:6789/0 conn(0x7ff7d80a0a80 legacy=0x7ff7d8139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7d7fff700 1 -- 172.21.15.150:0/550389086 >> v1:172.21.15.142:6789/0 conn(0x7ff7d809e460 legacy=0x7ff7d80a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_done global_id 14434 payload 274 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 1 -- 172.21.15.150:0/550389086 >> v1:172.21.15.150:6789/0 conn(0x7ff7d80a0a80 legacy=0x7ff7d8139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 1 -- 172.21.15.150:0/550389086 >> v1:172.21.15.142:6789/0 conn(0x7ff7d809e460 legacy=0x7ff7d80a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 1 monclient: found mon.noname-c 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.166+0000 7ff7dcf58700 1 -- 172.21.15.150:0/550389086 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d809d7a0 con 0x7ff7d80a6090 2024-09-18T10:10:47.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7dcf58700 1 -- 172.21.15.150:0/550389086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 msgr2=0x7ff7d809d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7dcf58700 1 --2- 172.21.15.150:0/550389086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d809d0f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099e0 tx=0x7ff7c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b73c99a0fc648c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 1 -- 172.21.15.150:0/550389086 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800bba0 con 0x7ff7d80a6090 2024-09-18T10:10:47.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: dump: 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 5 monclient: authenticate success, global_id 14434 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 1 -- 172.21.15.150:0/550389086 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c800bd20 con 0x7ff7d80a6090 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 1 -- 172.21.15.150:0/550389086 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8011670 con 0x7ff7d80a6090 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 4 set_mon_vals no callback set 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7d77fe700 10 monclient: dump: 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.176 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 10 monclient: shutdown 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- 172.21.15.150:0/550389086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 msgr2=0x7ff7d809d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/550389086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d809d0f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099e0 tx=0x7ff7c80091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- 172.21.15.150:0/550389086 shutdown_connections 2024-09-18T10:10:47.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/550389086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d809d0f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- 172.21.15.150:0/550389086 >> 172.21.15.150:0/550389086 conn(0x7ff7d8071830 msgr2=0x7ff7d80a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- 172.21.15.150:0/550389086 shutdown_connections 2024-09-18T10:10:47.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- 172.21.15.150:0/550389086 wait complete. 2024-09-18T10:10:47.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 10 monclient: monmap: 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 Processor -- start 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 1 -- start start 2024-09-18T10:10:47.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.170+0000 7ff7df1bc700 10 monclient: init 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7d809e460 0x7ff7d8141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-a con 0x7ff7d809e460 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7d80a0a80 0x7ff7d8142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-c con 0x7ff7d80a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): picked mon.noname-b con 0x7ff7d80a6090 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d808f760 con 0x7ff7d809e460 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80984d0 con 0x7ff7d80a6090 2024-09-18T10:10:47.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80a4500 con 0x7ff7d80a0a80 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 10 monclient(hunting): authenticate will time out at 929.268486s 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47430/0 (socket says 172.21.15.150:47430) 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 -- 172.21.15.150:0/3944477920 learned_addr learned my addr 172.21.15.150:0/3944477920 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): get_auth_request con 0x7ff7d80a6090 auth_method 0 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7d809e460 0x7ff7d8141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7d80a0a80 0x7ff7d8142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): get_auth_request con 0x7ff7d809e460 auth_method 0 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): get_auth_request con 0x7ff7d80a0a80 auth_method 0 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dcf58700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient(hunting): handle_auth_done global_id 14440 payload 824 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 -- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7d80a0a80 msgr2=0x7ff7d8142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7d80a0a80 0x7ff7d8142100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 -- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7d809e460 msgr2=0x7ff7d8141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7d809e460 0x7ff7d8141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 monclient: found mon.noname-b 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8009420 con 0x7ff7d80a6090 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 -- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 msgr2=0x7ff7d8142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7dd759700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000cf60 tx=0x7ff7d0001c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d000aa90 con 0x7ff7d80a6090 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 10 monclient: dump: 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.189 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d000abf0 con 0x7ff7d80a6090 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.174+0000 7ff7df1bc700 5 monclient: authenticate success, global_id 14440 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _renew_subs 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d808f060 con 0x7ff7d80a6090 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d00070f0 con 0x7ff7d80a6090 2024-09-18T10:10:47.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 10 monclient: dump: 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d6ffd700 4 set_mon_vals no callback set 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _renew_subs 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d808eaf0 con 0x7ff7d80a6090 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff7d00074a0 con 0x7ff7d80a6090 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 0x7ff7b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7ff7d0048400 con 0x7ff7d80a6090 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d808f000 con 0x7ff7d80a6090 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7dcf58700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 0x7ff7b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.178+0000 7ff7dcf58700 10 monclient: get_auth_request con 0x7ff7b8035ac0 auth_method 0 2024-09-18T10:10:47.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.182+0000 7ff7dcf58700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 0x7ff7b8037f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099e0 tx=0x7ff7c8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.186+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff7d001b400 con 0x7ff7d80a6090 2024-09-18T10:10:47.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.186+0000 7ff7d4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.186+0000 7ff7d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633694826}) 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832984651}) 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3633694826}) 2024-09-18T10:10:47.243 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.239+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646459067}) 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826539698}) 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1714387928}) 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826539698}) 2024-09-18T10:10:47.245 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4091941222}) 2024-09-18T10:10:47.247 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.247 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930944039}) 2024-09-18T10:10:47.248 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968016124}) 2024-09-18T10:10:47.248 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.248 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930944039}) 2024-09-18T10:10:47.248 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.243+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711012178}) 2024-09-18T10:10:47.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=1970200845,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1216880831,mon_config_key=1147460298}) 2024-09-18T10:10:47.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=73,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=424796314,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3207683715,mon_config_key=620546692}) 2024-09-18T10:10:47.251 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.251 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=1970200845,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1216880831,mon_config_key=1147460298}) 2024-09-18T10:10:47.251 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.247+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=1215724916,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1216880831,mon_config_key=1147460298}) 2024-09-18T10:10:47.255 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.255 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=1141929180,osd_snap=454132730,osdmap=2770014293}) 2024-09-18T10:10:47.255 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=82} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=1141929180,osd_snap=2673976383,osdmap=852203899}) 2024-09-18T10:10:47.255 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:10:47.256 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=1141929180,osd_snap=454132730,osdmap=2770014293}) 2024-09-18T10:10:47.256 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:10:47.251+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=82} crc {mon_config_key=2951930826,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=1141929180,osd_snap=454132730,osdmap=2100631374}) 2024-09-18T10:10:47.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7df1bc700 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7df1bc700 10 monclient: _send_command 2 [{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7df1bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7ff7d809da80 con 0x7ff7d80a6090 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v43) v1 ==== 67+0+1058 (secure 0 0 0) 0x7ff7d0029080 con 0x7ff7d80a6090 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7d4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.358+0000 7ff7d4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:47.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.450+0000 7ff7df1bc700 10 start_mon_command cmd=[{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.450+0000 7ff7df1bc700 10 monclient: _send_command 3 [{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.450+0000 7ff7df1bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.450+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule dump"} v 0) v1 -- 0x7ff7d8063ed0 con 0x7ff7d80a6090 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7d4ff9700 1 -- 172.21.15.150:0/3944477920 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule dump"}]=0 v43) v1 ==== 67+0+1058 (secure 0 0 0) 0x7ff7d0010af0 con 0x7ff7d80a6090 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7d4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule dump"}] 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7d4ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout:[ 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_id": 0, 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_name": "replicated_rule", 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout: "type": 1, 2024-09-18T10:10:47.457 INFO:tasks.workunit.client.0.smithi150.stdout: "steps": [ 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "take", 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "item": -1, 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "item_name": "default" 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "choose_firstn", 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 0, 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "type": "osd" 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "emit" 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.458 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_id": 1, 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "rule_name": "erasure-code", 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "type": 3, 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "steps": [ 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "set_chooseleaf_tries", 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 5 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "set_choose_tries", 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 100 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.459 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "take", 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "item": -1, 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "item_name": "default" 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "choose_indep", 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "num": 0, 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "type": "osd" 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: "op": "emit" 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:10:47.460 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:10:47.461 INFO:tasks.workunit.client.0.smithi150.stdout:] 2024-09-18T10:10:47.461 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 msgr2=0x7ff7b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 0x7ff7b8037f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80099e0 tx=0x7ff7c8017000 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 10 monclient: shutdown 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 msgr2=0x7ff7d8142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000cf60 tx=0x7ff7d0001c70 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 shutdown_connections 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff7b8035ac0 0x7ff7b8037f70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7d80a6090 0x7ff7d8142640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7d80a0a80 0x7ff7d8142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 --2- 172.21.15.150:0/3944477920 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7d809e460 0x7ff7d8141bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 >> 172.21.15.150:0/3944477920 conn(0x7ff7d8071830 msgr2=0x7ff7d809e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 shutdown_connections 2024-09-18T10:10:47.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.454+0000 7ff7df1bc700 1 -- 172.21.15.150:0/3944477920 wait complete. 2024-09-18T10:10:47.468 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: get_monmap_and_config 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: monmap: 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:47.564722+0000 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:47.564722+0000 2024-09-18T10:10:47.565 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:47.566 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 Processor -- start 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 -- start start 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: init 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f43000a0120 0x7f430009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): picked mon.noname-a con 0x7f43000a0120 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43000a04f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 0x7f4300139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): picked mon.noname-c con 0x7f43001379b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430009dae0 con 0x7f43000a04f0 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43000a21d0 con 0x7f43000a0120 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430009ddc0 con 0x7f43001379b0 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f4305169700 10 monclient(hunting): authenticate will time out at 929.658115s 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 0x7f4300139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 0x7f4300139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47440/0 (socket says 172.21.15.150:47440) 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fe59c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f43000a04f0 0x7f43000a0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38290/0 (socket says 172.21.15.150:38290) 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 1 -- 172.21.15.150:0/960002815 learned_addr learned my addr 172.21.15.150:0/960002815 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): get_auth_request con 0x7f43001379b0 auth_method 0 2024-09-18T10:10:47.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 --2- 172.21.15.150:0/960002815 >> v2:172.21.15.142:3300/0 conn(0x7f43000a0120 0x7f430009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42ff59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): get_auth_request con 0x7f43000a0120 auth_method 0 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient(hunting): handle_auth_done global_id 4746 payload 274 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 -- 172.21.15.150:0/960002815 >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 msgr2=0x7f4300139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 --2- 172.21.15.150:0/960002815 >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 0x7f4300139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 -- 172.21.15.150:0/960002815 >> v1:172.21.15.150:6789/0 conn(0x7f43000a04f0 legacy=0x7f43000a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 monclient: found mon.noname-a 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:47.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.562+0000 7f42fed9d700 1 -- 172.21.15.150:0/960002815 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43000a1590 con 0x7f43000a0120 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fed9d700 1 -- 172.21.15.150:0/960002815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a0120 msgr2=0x7f430009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fed9d700 1 --2- 172.21.15.150:0/960002815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a0120 0x7f430009e7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43000a36e0 tx=0x7f42f40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c88e6aa4d9edc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 1 -- 172.21.15.150:0/960002815 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4011840 con 0x7f43000a0120 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: dump: 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.578 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 5 monclient: authenticate success, global_id 4746 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 1 -- 172.21.15.150:0/960002815 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f40119a0 con 0x7f43000a0120 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 4 set_mon_vals no callback set 2024-09-18T10:10:47.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 1 -- 172.21.15.150:0/960002815 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4011d40 con 0x7f43000a0120 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f42fdd9b700 10 monclient: dump: 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 10 monclient: shutdown 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 -- 172.21.15.150:0/960002815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a0120 msgr2=0x7f430009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 --2- 172.21.15.150:0/960002815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a0120 0x7f430009e7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43000a36e0 tx=0x7f42f40091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 -- 172.21.15.150:0/960002815 shutdown_connections 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 --2- 172.21.15.150:0/960002815 >> v2:172.21.15.150:3300/0 conn(0x7f43001379b0 0x7f4300139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 --2- 172.21.15.150:0/960002815 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a0120 0x7f430009e7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 -- 172.21.15.150:0/960002815 >> 172.21.15.150:0/960002815 conn(0x7f43000a5830 msgr2=0x7f43000a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 -- 172.21.15.150:0/960002815 shutdown_connections 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 1 -- 172.21.15.150:0/960002815 wait complete. 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.566+0000 7f4305169700 10 monclient: monmap: 2024-09-18T10:10:47.581 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 Processor -- start 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 -- start start 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient: init 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): picked mon.noname-b con 0x7f43000a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a04f0 0x7f430012c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): picked mon.noname-a con 0x7f43000a04f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 0x7f430012ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): picked mon.noname-c con 0x7f43001379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430012d9a0 con 0x7f43000a04f0 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43001301b0 con 0x7f43000a0120 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430012f9d0 con 0x7f43001379b0 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f4305169700 10 monclient(hunting): authenticate will time out at 929.665036s 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47452/0 (socket says 172.21.15.150:47452) 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 -- 172.21.15.150:0/1723531614 learned_addr learned my addr 172.21.15.150:0/1723531614 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): get_auth_request con 0x7f43000a0120 auth_method 0 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fe59c700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a04f0 0x7f430012c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42ff59e700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 0x7f430012ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fe59c700 10 monclient(hunting): get_auth_request con 0x7f43000a04f0 auth_method 0 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fe59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient(hunting): handle_auth_done global_id 14446 payload 824 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 -- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 msgr2=0x7f430012ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 0x7f430012ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 -- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a04f0 msgr2=0x7f430012c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a04f0 0x7f430012c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 monclient: found mon.noname-b 2024-09-18T10:10:47.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4009420 con 0x7f43000a0120 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42ff59e700 10 monclient: get_auth_request con 0x7f43001379b0 auth_method 0 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42ff59e700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 0x7f430012ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.570+0000 7f42fed9d700 1 -- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 msgr2=0x7f43001305d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42fed9d700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009f50 tx=0x7f42f4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4006af0 con 0x7f43000a0120 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: dump: 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.586 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 5 monclient: authenticate success, global_id 14446 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f4020070 con 0x7f43000a0120 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _renew_subs 2024-09-18T10:10:47.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430012fed0 con 0x7f43000a0120 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f400be60 con 0x7f43000a0120 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 10 monclient: dump: 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.588 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _renew_subs 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42fd59a700 4 set_mon_vals no callback set 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430012f470 con 0x7f43000a0120 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4300063ed0 con 0x7f43000a0120 2024-09-18T10:10:47.590 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:10:47.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:10:47.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f42f4022030 con 0x7f43000a0120 2024-09-18T10:10:47.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 0x7f42ec037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f42f4049f20 con 0x7f43000a0120 2024-09-18T10:10:47.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42fe59c700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 0x7f42ec037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.574+0000 7f42fe59c700 10 monclient: get_auth_request con 0x7f42ec035ac0 auth_method 0 2024-09-18T10:10:47.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.578+0000 7f42fe59c700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 0x7f42ec037f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4300099dd0 tx=0x7f42f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.586+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f42f401b660 con 0x7f43000a0120 2024-09-18T10:10:47.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.586+0000 7f42e6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.586+0000 7f42e6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:47.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: get_monmap_and_config 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: monmap: 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:47.702206+0000 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:47.702206+0000 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:47.703 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:47.704 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:47.704 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 Processor -- start 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 -- start start 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: init 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d6409cdd0 0x7f7d6409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-c con 0x7f7d6409cdd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-b con 0x7f7d640a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d640a0bf0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:47.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6409d1a0 con 0x7f7d640a0bf0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6409e980 con 0x7f7d6409cdd0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6409d780 con 0x7f7d640a02e0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.699+0000 7f7d6a332700 10 monclient(hunting): authenticate will time out at 936.627985s 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7d6409cdd0 0x7f7d6409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55678/0 (socket says 172.21.15.142:55678) 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 -- 172.21.15.142:0/831877612 learned_addr learned my addr 172.21.15.142:0/831877612 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): get_auth_request con 0x7f7d640a02e0 auth_method 0 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): get_auth_request con 0x7f7d6409cdd0 auth_method 0 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient(hunting): handle_auth_done global_id 24128 payload 274 2024-09-18T10:10:47.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 -- 172.21.15.142:0/831877612 >> v2:172.21.15.150:3300/0 conn(0x7f7d6409cdd0 msgr2=0x7f7d6409e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 --2- 172.21.15.142:0/831877612 >> v2:172.21.15.150:3300/0 conn(0x7f7d6409cdd0 0x7f7d6409e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 -- 172.21.15.142:0/831877612 >> v1:172.21.15.150:6789/0 conn(0x7f7d640a0bf0 legacy=0x7f7d64138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 -- 172.21.15.142:0/831877612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6409dd40 con 0x7f7d640a02e0 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 -- 172.21.15.142:0/831877612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 msgr2=0x7f7d640a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d637fe700 1 --2- 172.21.15.142:0/831877612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7d540099b0 tx=0x7f7d54009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f18f9be2504bc0e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 1 -- 172.21.15.142:0/831877612 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54009ea0 con 0x7f7d640a02e0 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: dump: 2024-09-18T10:10:47.711 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d6a332700 5 monclient: authenticate success, global_id 24128 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d6a332700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:47.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 1 -- 172.21.15.142:0/831877612 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5400ba70 con 0x7f7d640a02e0 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d6a332700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d6a332700 4 set_mon_vals no callback set 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 1 -- 172.21.15.142:0/831877612 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400be30 con 0x7f7d640a02e0 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.703+0000 7f7d62ffd700 10 monclient: dump: 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.713 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 10 monclient: shutdown 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- 172.21.15.142:0/831877612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 msgr2=0x7f7d640a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 --2- 172.21.15.142:0/831877612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7d540099b0 tx=0x7f7d54009250 comp rx=0 tx=0).stop 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- 172.21.15.142:0/831877612 shutdown_connections 2024-09-18T10:10:47.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 --2- 172.21.15.142:0/831877612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d640a06b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 --2- 172.21.15.142:0/831877612 >> v2:172.21.15.150:3300/0 conn(0x7f7d6409cdd0 0x7f7d6409e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- 172.21.15.142:0/831877612 >> 172.21.15.142:0/831877612 conn(0x7f7d64071680 msgr2=0x7f7d640a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- 172.21.15.142:0/831877612 shutdown_connections 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- 172.21.15.142:0/831877612 wait complete. 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 10 monclient: monmap: 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:47.715 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:47.716 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:47.716 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 Processor -- start 2024-09-18T10:10:47.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 1 -- start start 2024-09-18T10:10:47.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.707+0000 7f7d6a332700 10 monclient: init 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6409cdd0 0x7f7d64141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-b con 0x7f7d6409cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-c con 0x7f7d640a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 0x7f7d6408cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): picked mon.noname-a con 0x7f7d640a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d64090b10 con 0x7f7d640a0bf0 2024-09-18T10:10:47.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d640907a0 con 0x7f7d6409cdd0 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6408f560 con 0x7f7d640a02e0 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient(hunting): authenticate will time out at 936.636294s 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 0x7f7d6408cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55690/0 (socket says 172.21.15.142:55690) 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 -- 172.21.15.142:0/310822322 learned_addr learned my addr 172.21.15.142:0/310822322 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): get_auth_request con 0x7f7d640a02e0 auth_method 0 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6409cdd0 0x7f7d64141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): get_auth_request con 0x7f7d640a0bf0 auth_method 0 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): get_auth_request con 0x7f7d6409cdd0 auth_method 0 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient(hunting): handle_auth_done global_id 24131 payload 824 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 -- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6409cdd0 msgr2=0x7f7d64141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6409cdd0 0x7f7d64141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 -- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 msgr2=0x7f7d6408cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 0x7f7d6408cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 monclient: found mon.noname-c 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 -- 172.21.15.142:0/310822322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d540094f0 con 0x7f7d640a02e0 2024-09-18T10:10:47.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d688cf700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 0x7f7d6408cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 -- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 msgr2=0x7f7d64142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d637fe700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7d6409d350 tx=0x7f7d54016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54006b60 con 0x7f7d640a02e0 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: dump: 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.724 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54020070 con 0x7f7d640a02e0 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 5 monclient: authenticate success, global_id 24131 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _renew_subs 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d640a3fa0 con 0x7f7d640a02e0 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54011cd0 con 0x7f7d640a02e0 2024-09-18T10:10:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d5bfff700 10 monclient: dump: 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d627fc700 4 set_mon_vals no callback set 2024-09-18T10:10:47.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _renew_subs 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.711+0000 7f7d6a332700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d64090570 con 0x7f7d640a02e0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d6a332700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d6a332700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7d54022030 con 0x7f7d640a02e0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d6a332700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6408e950 con 0x7f7d640a02e0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d5bfff700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 0x7f7d48037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f7d54049da0 con 0x7f7d640a02e0 2024-09-18T10:10:47.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d63fff700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 0x7f7d48037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.715+0000 7f7d63fff700 10 monclient: get_auth_request con 0x7f7d48035b10 auth_method 0 2024-09-18T10:10:47.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.719+0000 7f7d63fff700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 0x7f7d48037fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009d20 tx=0x7f7d4c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.723+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7d5401c450 con 0x7f7d640a02e0 2024-09-18T10:10:47.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.723+0000 7f7d5bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.723+0000 7f7d5bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:47.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f4305169700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f4305169700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f4305169700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f4300063ed0 con 0x7f43000a0120 2024-09-18T10:10:47.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v43) v1 ==== 65+0+29 (secure 0 0 0) 0x7f42f4023e60 con 0x7f43000a0120 2024-09-18T10:10:47.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f42e6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.754+0000 7f42e6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:47.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f4305169700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f4305169700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f4305169700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f4300063ed0 con 0x7f43000a0120 2024-09-18T10:10:47.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f42e6ffd700 1 -- 172.21.15.150:0/1723531614 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v43) v1 ==== 65+0+29 (secure 0 0 0) 0x7f42f401a3d0 con 0x7f43000a0120 2024-09-18T10:10:47.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f42e6ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:10:47.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.854+0000 7f42e6ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:47.860 INFO:tasks.workunit.client.0.smithi150.stdout:replicated_rule 2024-09-18T10:10:47.861 INFO:tasks.workunit.client.0.smithi150.stdout:erasure-code 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 msgr2=0x7f42ec037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 0x7f42ec037f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4300099dd0 tx=0x7f42f0009040 comp rx=0 tx=0).stop 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 10 monclient: shutdown 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 msgr2=0x7f43001305d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009f50 tx=0x7f42f4016000 comp rx=0 tx=0).stop 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 shutdown_connections 2024-09-18T10:10:47.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f42ec035ac0 0x7f42ec037f70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43001379b0 0x7f430012ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43000a04f0 0x7f430012c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 --2- 172.21.15.150:0/1723531614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43000a0120 0x7f43001305d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 >> 172.21.15.150:0/1723531614 conn(0x7f43000a5830 msgr2=0x7f430009abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 shutdown_connections 2024-09-18T10:10:47.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.858+0000 7f4305169700 1 -- 172.21.15.150:0/1723531614 wait complete. 2024-09-18T10:10:47.870 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule list 2024-09-18T10:10:47.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d6408e950 con 0x7f7d640a02e0 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d5bfff700 1 -- 172.21.15.142:0/310822322 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7d540243d0 con 0x7f7d640a02e0 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d5bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:10:47.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d5bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:47.958 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:10:47.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 msgr2=0x7f7d48037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 0x7f7d48037fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009d20 tx=0x7f7d4c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 10 monclient: shutdown 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 msgr2=0x7f7d64142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7d6409d350 tx=0x7f7d54016000 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 shutdown_connections 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d48035b10 0x7f7d48037fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d640a0bf0 0x7f7d6408cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d640a02e0 0x7f7d64142210 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 --2- 172.21.15.142:0/310822322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d6409cdd0 0x7f7d64141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 >> 172.21.15.142:0/310822322 conn(0x7f7d64071680 msgr2=0x7f7d6409eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 shutdown_connections 2024-09-18T10:10:47.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:47.955+0000 7f7d6a332700 1 -- 172.21.15.142:0/310822322 wait complete. 2024-09-18T10:10:47.971 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:10:47.971 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:10:47.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:10:47.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 10 monclient: get_monmap_and_config 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 10 monclient: monmap: 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:47.970439+0000 2024-09-18T10:10:47.975 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:47.970439+0000 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 1 Processor -- start 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 1 -- start start 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.966+0000 7fdcac2f4700 10 monclient: init 2024-09-18T10:10:47.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdca40a0120 0x7fdca409e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-c con 0x7fdca40a0120 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdca40a04f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 0x7fdca4139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-b con 0x7fdca41379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca409dae0 con 0x7fdca40a04f0 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca40a21d0 con 0x7fdca40a0120 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca409ddc0 con 0x7fdca41379b0 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcac2f4700 10 monclient(hunting): authenticate will time out at 930.064321s 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa090700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdca40a0120 0x7fdca409e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdca988f700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdca40a04f0 0x7fdca40a0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38294/0 (socket says 172.21.15.150:38294) 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdca988f700 1 -- 172.21.15.150:0/1738886456 learned_addr learned my addr 172.21.15.150:0/1738886456 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa090700 10 monclient(hunting): get_auth_request con 0x7fdca40a0120 auth_method 0 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa090700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa090700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa090700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 --2- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 0x7fdca4139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): get_auth_request con 0x7fdca41379b0 auth_method 0 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient(hunting): handle_auth_done global_id 24134 payload 274 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 -- 172.21.15.150:0/1738886456 >> v2:172.21.15.150:3300/0 conn(0x7fdca40a0120 msgr2=0x7fdca409e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 --2- 172.21.15.150:0/1738886456 >> v2:172.21.15.150:3300/0 conn(0x7fdca40a0120 0x7fdca409e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 -- 172.21.15.150:0/1738886456 >> v1:172.21.15.150:6789/0 conn(0x7fdca40a04f0 legacy=0x7fdca40a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 monclient: found mon.noname-b 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:47.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 -- 172.21.15.150:0/1738886456 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40a1590 con 0x7fdca41379b0 2024-09-18T10:10:47.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 -- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 msgr2=0x7fdca4139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.970+0000 7fdcaa891700 1 --2- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 0x7fdca4139d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009e60 tx=0x7fdc9c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d09942b6b98cf2a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 1 -- 172.21.15.150:0/1738886456 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c00e070 con 0x7fdca41379b0 2024-09-18T10:10:47.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: dump: 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 5 monclient: authenticate success, global_id 24134 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 1 -- 172.21.15.150:0/1738886456 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c00be40 con 0x7fdca41379b0 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 4 set_mon_vals no callback set 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 1 -- 172.21.15.150:0/1738886456 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c012660 con 0x7fdca41379b0 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca908e700 10 monclient: dump: 2024-09-18T10:10:47.988 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: shutdown 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 msgr2=0x7fdca4139d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 0x7fdca4139d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009e60 tx=0x7fdc9c009780 comp rx=0 tx=0).stop 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- 172.21.15.150:0/1738886456 shutdown_connections 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/1738886456 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca41379b0 0x7fdca4139d90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/1738886456 >> v2:172.21.15.150:3300/0 conn(0x7fdca40a0120 0x7fdca409e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- 172.21.15.150:0/1738886456 >> 172.21.15.150:0/1738886456 conn(0x7fdca40a5830 msgr2=0x7fdca40a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- 172.21.15.150:0/1738886456 shutdown_connections 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- 172.21.15.150:0/1738886456 wait complete. 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: build_initial_monmap 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: monmap: 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:47.990 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 Processor -- start 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- start start 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: init 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdca40a0120 0x7fdca412c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-a con 0x7fdca40a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca40a04f0 0x7fdca412c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-c con 0x7fdca40a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): picked mon.noname-b con 0x7fdca41379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca409d720 con 0x7fdca40a0120 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca409bef0 con 0x7fdca41379b0 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca412e3a0 con 0x7fdca40a04f0 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcac2f4700 10 monclient(hunting): authenticate will time out at 930.071347s 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47466/0 (socket says 172.21.15.150:47466) 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 1 -- 172.21.15.150:0/671792032 learned_addr learned my addr 172.21.15.150:0/671792032 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): get_auth_request con 0x7fdca41379b0 auth_method 0 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdca40a0120 0x7fdca412c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): get_auth_request con 0x7fdca40a0120 auth_method 0 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca988f700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca40a04f0 0x7fdca412c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca988f700 10 monclient(hunting): get_auth_request con 0x7fdca40a04f0 auth_method 0 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca988f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca988f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdca988f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa090700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 10 monclient(hunting): handle_auth_done global_id 14458 payload 824 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 1 -- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca40a04f0 msgr2=0x7fdca412c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.974+0000 7fdcaa891700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca40a04f0 0x7fdca412c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 -- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdca40a0120 msgr2=0x7fdca412c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:47.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdca40a0120 0x7fdca412c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 10 monclient: _finish_hunting 0 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 monclient: found mon.noname-b 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c009a20 con 0x7fdca41379b0 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 -- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 msgr2=0x7fdca412ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcaa891700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c003000 tx=0x7fdc9c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c012e20 con 0x7fdca41379b0 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: dump: 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c021070 con 0x7fdca41379b0 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 5 monclient: authenticate success, global_id 14458 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c026660 con 0x7fdca41379b0 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _renew_subs 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdca888d700 4 set_mon_vals no callback set 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca412f740 con 0x7fdca41379b0 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdc967fc700 10 monclient: dump: 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:47.997 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _renew_subs 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4137520 con 0x7fdca41379b0 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.978+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdc9c023030 con 0x7fdca41379b0 2024-09-18T10:10:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdc967fc700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 0x7fdc88037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fdc9c04a470 con 0x7fdca41379b0 2024-09-18T10:10:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdcaa090700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 0x7fdc88037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdcaa090700 10 monclient: get_auth_request con 0x7fdc88035ac0 auth_method 0 2024-09-18T10:10:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.982+0000 7fdcaa090700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 0x7fdc88037f70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdca0006fd0 tx=0x7fdca0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.994+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.994+0000 7fdc967fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:47.994+0000 7fdc967fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:48.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: get_monmap_and_config 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: monmap: 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:48.081205+0000 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:48.081205+0000 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:48.082 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:48.083 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:48.083 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:48.083 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 Processor -- start 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 -- start start 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: init 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9000999e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-b con 0x7fd90009c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd900137910 0x7fd900139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-a con 0x7fd900137910 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9000a20c0 con 0x7fd9000999e0 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90009a000 con 0x7fd900137910 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000a1c80 con 0x7fd90009c410 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd9057d2700 10 monclient(hunting): authenticate will time out at 937.007499s 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55694/0 (socket says 172.21.15.142:55694) 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/4236226711 learned_addr learned my addr 172.21.15.142:0/4236226711 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): get_auth_request con 0x7fd90009c410 auth_method 0 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd900137910 0x7fd900139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): get_auth_request con 0x7fd900137910 auth_method 0 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.079+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_done global_id 24140 payload 274 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/4236226711 >> v2:172.21.15.142:3300/0 conn(0x7fd900137910 msgr2=0x7fd900139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 --2- 172.21.15.142:0/4236226711 >> v2:172.21.15.142:3300/0 conn(0x7fd900137910 0x7fd900139cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/4236226711 >> v1:172.21.15.142:6789/0 conn(0x7fd9000999e0 legacy=0x7fd9000a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 monclient: found mon.noname-b 2024-09-18T10:10:48.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:48.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/4236226711 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90009a580 con 0x7fd90009c410 2024-09-18T10:10:48.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/4236226711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 msgr2=0x7fd9000a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8ff7fe700 1 --2- 172.21.15.142:0/4236226711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00099e0 tx=0x7fd8f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52dd1fcd1b3273da server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 1 -- 172.21.15.142:0/4236226711 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f000ba30 con 0x7fd90009c410 2024-09-18T10:10:48.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: dump: 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 1 -- 172.21.15.142:0/4236226711 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f000bb90 con 0x7fd90009c410 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 5 monclient: authenticate success, global_id 24140 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 1 -- 172.21.15.142:0/4236226711 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0011420 con 0x7fd90009c410 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 4 set_mon_vals no callback set 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd8feffd700 10 monclient: dump: 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.095 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 10 monclient: shutdown 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 1 -- 172.21.15.142:0/4236226711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 msgr2=0x7fd9000a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 1 --2- 172.21.15.142:0/4236226711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00099e0 tx=0x7fd8f0009250 comp rx=0 tx=0).stop 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 1 -- 172.21.15.142:0/4236226711 shutdown_connections 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 1 --2- 172.21.15.142:0/4236226711 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd90009c410 0x7fd9000a0890 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.083+0000 7fd9057d2700 1 --2- 172.21.15.142:0/4236226711 >> v2:172.21.15.142:3300/0 conn(0x7fd900137910 0x7fd900139cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- 172.21.15.142:0/4236226711 >> 172.21.15.142:0/4236226711 conn(0x7fd9000a5790 msgr2=0x7fd9000a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- 172.21.15.142:0/4236226711 shutdown_connections 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- 172.21.15.142:0/4236226711 wait complete. 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient: monmap: 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:48.097 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 Processor -- start 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- start start 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient: init 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9000999e0 0x7fd90006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-c con 0x7fd9000999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-a con 0x7fd90009c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd900137910 0x7fd90012bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): picked mon.noname-b con 0x7fd900137910 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90009f0d0 con 0x7fd90009c410 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90012ea30 con 0x7fd900137910 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90012e750 con 0x7fd9000999e0 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd9057d2700 10 monclient(hunting): authenticate will time out at 937.015477s 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9000999e0 0x7fd90006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36274/0 (socket says 172.21.15.142:36274) 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9000999e0 0x7fd90006f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55704/0 (socket says 172.21.15.142:55704) 2024-09-18T10:10:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/3269073722 learned_addr learned my addr 172.21.15.142:0/3269073722 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): get_auth_request con 0x7fd90009c410 auth_method 0 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 10 monclient(hunting): get_auth_request con 0x7fd9000999e0 auth_method 0 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd904fd1700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd900137910 0x7fd90012bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd904fd1700 10 monclient(hunting): get_auth_request con 0x7fd900137910 auth_method 0 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd904fd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd904fd1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd904fd1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.087+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd904fd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 10 monclient(hunting): handle_auth_done global_id 4761 payload 824 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9000999e0 msgr2=0x7fd90006f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9000999e0 0x7fd90006f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd900137910 msgr2=0x7fd90012bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd900137910 0x7fd90012bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 monclient: found mon.noname-a 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f00094f0 con 0x7fd90009c410 2024-09-18T10:10:48.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd904fd1700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd900137910 0x7fd90012bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:48.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 msgr2=0x7fd90012b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8ff7fe700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd9000a1e10 tx=0x7fd8f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0011e70 con 0x7fd90009c410 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: dump: 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.106 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 5 monclient: authenticate success, global_id 4761 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0020070 con 0x7fd90009c410 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 10 monclient: _renew_subs 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90012e470 con 0x7fd90009c410 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0011420 con 0x7fd90009c410 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8fe7fc700 4 set_mon_vals no callback set 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd8f7fff700 10 monclient: dump: 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.108 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 10 monclient: _renew_subs 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.091+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90012d760 con 0x7fd90009c410 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd9057d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd9057d2700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 0x7fd90012c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000647f0 con 0x7fd90012f190 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 0x7fd90012c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient: get_auth_request con 0x7fd90012f190 auth_method 0 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd8f0022030 con 0x7fd90009c410 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8f7fff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8e0035ac0 0x7fd8e0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd90012ebc0 con 0x7fd90012f190 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fd8f0049df0 con 0x7fd90009c410 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd904fd1700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8e0035ac0 0x7fd8e0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd904fd1700 10 monclient: get_auth_request con 0x7fd8e0035ac0 auth_method 0 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 msgr2=0x7fd90012c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.095+0000 7fd8fffff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 0x7fd90012c2b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800f7b0 tx=0x7fd8e8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8014890 con 0x7fd90012f190 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 10 monclient: dump: 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.111 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd8e80149f0 con 0x7fd90012f190 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 msgr2=0x7fd90012c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 0x7fd90012c2b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800f7b0 tx=0x7fd8e8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 msgr2=0x7fd90012c2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd8f7fff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90012f190 0x7fd90012c2b0 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800f7b0 tx=0x7fd8e8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.099+0000 7fd904fd1700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8e0035ac0 0x7fd8e0037f70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd90009fdc0 tx=0x7fd8f800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.107+0000 7fd9057d2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:48.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.107+0000 7fd9057d2700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 0x7fd90006cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.107+0000 7fd9057d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.107+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90012d760 con 0x7fd90006cac0 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 0x7fd90006cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient: get_auth_request con 0x7fd90006cac0 auth_method 0 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2024-09-18T10:10:48.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 1 -- 172.21.15.142:0/3269073722 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd900072a80 con 0x7fd90006cac0 2024-09-18T10:10:48.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 msgr2=0x7fd90006cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8fffff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 0x7fd90006cec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8003480 tx=0x7fd8e8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8021620 con 0x7fd90006cac0 2024-09-18T10:10:48.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.111+0000 7fd8f7fff700 10 monclient: dump: 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.116 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7fd8e8021780 con 0x7fd90006cac0 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 msgr2=0x7fd90006cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 0x7fd90006cec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8003480 tx=0x7fd8e8019000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 msgr2=0x7fd90006cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd8f7fff700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90006cac0 0x7fd90006cec0 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8003480 tx=0x7fd8e8019000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:10:48.118 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.119 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:48.120 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:48.121 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:48.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.123 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.124 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:48.125 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.126 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.127 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:48.128 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:10:48.129 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:48.130 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:48.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8e0035ac0 msgr2=0x7fd8e0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8e0035ac0 0x7fd8e0037f70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd90009fdc0 tx=0x7fd8f800b040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 10 monclient: shutdown 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 msgr2=0x7fd90012b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd9000a1e10 tx=0x7fd8f0016000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd904fd1700 1 -- 172.21.15.142:0/3269073722 reap_dead start 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 shutdown_connections 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 --2- 172.21.15.142:0/3269073722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd90009c410 0x7fd90012b830 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 >> 172.21.15.142:0/3269073722 conn(0x7fd9000a5790 msgr2=0x7fd900136f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 shutdown_connections 2024-09-18T10:10:48.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.115+0000 7fd9057d2700 1 -- 172.21.15.142:0/3269073722 wait complete. 2024-09-18T10:10:48.133 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:48.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:10:48.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:10:48.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdcac2f4700 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdcac2f4700 10 monclient: _send_command 2 [{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdcac2f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:48.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v44) v1 ==== 67+0+29 (secure 0 0 0) 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:48.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdc967fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.162+0000 7fdc967fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:48.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:10:48.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: monmap: 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:48.237137+0000 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:48.237137+0000 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:48.238 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 Processor -- start 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 -- start start 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: init 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f542c09a6a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f542c06f7f0 0x7f542c09b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f542c06f7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:48.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c13d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f542c09b6c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542c06fbf0 con 0x7f542c09a6a0 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542c09dae0 con 0x7f542c06f7f0 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c099bc0 con 0x7f542c09b6c0 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542b59e700 10 monclient(hunting): authenticate will time out at 937.162604s 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542a59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f542c09a6a0 0x7f542c09aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60376/0 (socket says 172.21.15.142:60376) 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f542c06f7f0 0x7f542c09b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542a59c700 1 -- 172.21.15.142:0/3750169262 learned_addr learned my addr 172.21.15.142:0/3750169262 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c13d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): get_auth_request con 0x7f542c06f7f0 auth_method 0 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): get_auth_request con 0x7f542c09b6c0 auth_method 0 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient(hunting): handle_auth_done global_id 14464 payload 274 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 -- 172.21.15.142:0/3750169262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 msgr2=0x7f542c13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c13d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 -- 172.21.15.142:0/3750169262 >> v1:172.21.15.142:6789/0 conn(0x7f542c09a6a0 legacy=0x7f542c09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 monclient: found mon.noname-c 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f5429d9b700 1 -- 172.21.15.142:0/3750169262 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542c0998e0 con 0x7f542c06f7f0 2024-09-18T10:10:48.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.235+0000 7f542ad9d700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c13d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:48.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f5429d9b700 1 -- 172.21.15.142:0/3750169262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 msgr2=0x7f542c09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f5429d9b700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c09b180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5420009910 tx=0x7f54200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=686e01787a8ec9b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 1 -- 172.21.15.142:0/3750169262 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542000b850 con 0x7f542c06f7f0 2024-09-18T10:10:48.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: dump: 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 5 monclient: authenticate success, global_id 14464 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 1 -- 172.21.15.142:0/3750169262 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f542000b9b0 con 0x7f542c06f7f0 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 1 -- 172.21.15.142:0/3750169262 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542000bd70 con 0x7f542c06f7f0 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 4 set_mon_vals no callback set 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542959a700 10 monclient: dump: 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.252 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 10 monclient: shutdown 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- 172.21.15.142:0/3750169262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 msgr2=0x7f542c09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c09b180 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5420009910 tx=0x7f54200091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- 172.21.15.142:0/3750169262 shutdown_connections 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c13d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 --2- 172.21.15.142:0/3750169262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c09b180 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- 172.21.15.142:0/3750169262 >> 172.21.15.142:0/3750169262 conn(0x7f542c070ef0 msgr2=0x7f542c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- 172.21.15.142:0/3750169262 shutdown_connections 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- 172.21.15.142:0/3750169262 wait complete. 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 10 monclient: monmap: 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:48.254 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 Processor -- start 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 1 -- start start 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.239+0000 7f542b59e700 10 monclient: init 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f542c06f7f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 0x7f542c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f542c09a6a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f542c09b6c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c1265b0 con 0x7f542c09a6a0 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542c094830 con 0x7f542c06f7f0 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542c0a0570 con 0x7f542c09b6c0 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient(hunting): authenticate will time out at 937.168632s 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 0x7f542c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 0x7f542c06c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36318/0 (socket says 172.21.15.142:36318) 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 -- 172.21.15.142:0/2980460284 learned_addr learned my addr 172.21.15.142:0/2980460284 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): get_auth_request con 0x7f542c09a6a0 auth_method 0 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): get_auth_request con 0x7f542c09b6c0 auth_method 0 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient(hunting): handle_auth_done global_id 4770 payload 824 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 msgr2=0x7f542c06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f542c09b6c0 0x7f542c06cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 msgr2=0x7f542c06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c06c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 monclient: found mon.noname-a 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424009a20 con 0x7f542c09a6a0 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542a59c700 10 monclient: get_auth_request con 0x7f542c06f7f0 auth_method 0 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c06f7f0 0x7f542c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 msgr2=0x7f542c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f5429d9b700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 0x7f542c06c9d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f542c071df0 tx=0x7f5420016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5420011b70 con 0x7f542c09a6a0 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: dump: 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.259 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 5 monclient: authenticate success, global_id 4770 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _renew_subs 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5420009420 con 0x7f542c09a6a0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _renew_subs 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c126920 con 0x7f542c09a6a0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5420020070 con 0x7f542c09a6a0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542001b450 con 0x7f542c09a6a0 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.243+0000 7f541a7fc700 10 monclient: dump: 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f5428d99700 4 set_mon_vals no callback set 2024-09-18T10:10:48.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5420022030 con 0x7f542c09a6a0 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542b59e700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 0x7f5410004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5410007cd0 con 0x7f5410002860 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 0x7f5410004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient: get_auth_request con 0x7f5410002860 auth_method 0 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f541a7fc700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f540c035ac0 0x7f540c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f5420049db0 con 0x7f542c09a6a0 2024-09-18T10:10:48.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 10 monclient(hunting): handle_auth_done global_id 24152 payload 824 2024-09-18T10:10:48.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542a59c700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5410007e90 con 0x7f5410002860 2024-09-18T10:10:48.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542ad9d700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f540c035ac0 0x7f540c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.247+0000 7f542ad9d700 10 monclient: get_auth_request con 0x7f540c035ac0 auth_method 0 2024-09-18T10:10:48.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f542a59c700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 msgr2=0x7f5410004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 0x7f5410004cf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d730 tx=0x7f541c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c00b070 con 0x7f5410002860 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 10 monclient: dump: 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.265 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f541c005d40 con 0x7f5410002860 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 msgr2=0x7f5410004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 0x7f5410004cf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d730 tx=0x7f541c009040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 msgr2=0x7f5410004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f541a7fc700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410002860 0x7f5410004cf0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d730 tx=0x7f541c009040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.251+0000 7f542ad9d700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f540c035ac0 0x7f540c037f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5424009fd0 tx=0x7f542400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542b59e700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 0x7f5410007190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100077d0 con 0x7f5410006d10 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 0x7f5410007190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient: get_auth_request con 0x7f5410006d10 auth_method 0 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 10 monclient(hunting): handle_auth_done global_id 24155 payload 824 2024-09-18T10:10:48.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.259+0000 7f542a59c700 1 -- 172.21.15.142:0/2980460284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5410007990 con 0x7f5410006d10 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 10 start_mon_command cmd=[{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 10 monclient: _send_command 3 [{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule list"} v 0) v1 -- 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdc967fc700 1 -- 172.21.15.150:0/671792032 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule list"}]=0 v44) v1 ==== 67+0+29 (secure 0 0 0) 0x7fdca412fa80 con 0x7fdca41379b0 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdc967fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule list"}] 2024-09-18T10:10:48.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdc967fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:10:48.270 INFO:tasks.workunit.client.0.smithi150.stdout:replicated_rule 2024-09-18T10:10:48.270 INFO:tasks.workunit.client.0.smithi150.stdout:erasure-code 2024-09-18T10:10:48.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542a59c700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 msgr2=0x7f5410007190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542a59c700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 0x7f5410007190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d730 tx=0x7f541c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c024070 con 0x7f5410006d10 2024-09-18T10:10:48.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 10 monclient: dump: 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f541c020430 con 0x7f5410006d10 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 msgr2=0x7f5410007190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 0x7f5410007190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f541c00d730 tx=0x7f541c018000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 msgr2=0x7f5410007190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f541a7fc700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5410006d10 0x7f5410007190 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:48.272 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:48.273 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.274 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:48.275 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.276 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:48.277 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.280 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.281 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.282 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.283 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.284 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:48.284 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 msgr2=0x7fdc88037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 0x7fdc88037f70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdca0006fd0 tx=0x7fdca0008040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 10 monclient: shutdown 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 msgr2=0x7fdca412ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c003000 tx=0x7fdc9c018000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 shutdown_connections 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc88035ac0 0x7fdc88037f70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca41379b0 0x7fdca412ccc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdca40a04f0 0x7fdca412c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 --2- 172.21.15.150:0/671792032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdca40a0120 0x7fdca412c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 >> 172.21.15.150:0/671792032 conn(0x7fdca40a5830 msgr2=0x7fdca409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 shutdown_connections 2024-09-18T10:10:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.262+0000 7fdcac2f4700 1 -- 172.21.15.150:0/671792032 wait complete. 2024-09-18T10:10:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f540c035ac0 msgr2=0x7f540c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f540c035ac0 0x7f540c037f70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5424009fd0 tx=0x7f542400b040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 10 monclient: shutdown 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 msgr2=0x7f542c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 --2- 172.21.15.142:0/2980460284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f542c09a6a0 0x7f542c06c9d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f542c071df0 tx=0x7f5420016000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542ad9d700 1 -- 172.21.15.142:0/2980460284 reap_dead start 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542ad9d700 1 -- 172.21.15.142:0/2980460284 reap_dead start 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 shutdown_connections 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 >> 172.21.15.142:0/2980460284 conn(0x7f542c070ef0 msgr2=0x7f542c09c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 shutdown_connections 2024-09-18T10:10:48.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.263+0000 7f542b59e700 1 -- 172.21.15.142:0/2980460284 wait complete. 2024-09-18T10:10:48.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:48.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:10:48.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.370+0000 7fc5c5cf4700 10 monclient: get_monmap_and_config 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.370+0000 7fc5c5cf4700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.370+0000 7fc5c5cf4700 10 monclient: monmap: 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:10:48.375492+0000 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:10:48.375492+0000 2024-09-18T10:10:48.376 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:48.377 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 Processor -- start 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 -- start start 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient: init 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5c00a2810 0x7fc5c00a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-c con 0x7fc5c00a2810 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5c00a1550 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc5c00a1950 0x7fc5c0132e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-a con 0x7fc5c00a1950 addr v2:172.21.15.142:3300/0 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c00697a0 con 0x7fc5c00a1550 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c006a240 con 0x7fc5c00a1950 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c009d170 con 0x7fc5c00a2810 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c5cf4700 10 monclient(hunting): authenticate will time out at 930.469353s 2024-09-18T10:10:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5c00a2810 0x7fc5c00a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5c00a2810 0x7fc5c00a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47480/0 (socket says 172.21.15.150:47480) 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/2336498378 learned_addr learned my addr 172.21.15.150:0/2336498378 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request con 0x7fc5c00a2810 auth_method 0 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/2336498378 >> v2:172.21.15.142:3300/0 conn(0x7fc5c00a1950 0x7fc5c0132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request con 0x7fc5c00a1950 auth_method 0 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_done global_id 14467 payload 274 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/2336498378 >> v2:172.21.15.142:3300/0 conn(0x7fc5c00a1950 msgr2=0x7fc5c0132e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/2336498378 >> v2:172.21.15.142:3300/0 conn(0x7fc5c00a1950 0x7fc5c0132e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/2336498378 >> v1:172.21.15.142:6789/0 conn(0x7fc5c00a1550 legacy=0x7fc5c0132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 monclient: found mon.noname-c 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.374+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/2336498378 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c009cec0 con 0x7fc5c00a2810 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/2336498378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a2810 msgr2=0x7fc5c00a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/2336498378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a2810 0x7fc5c00a1010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0069e90 tx=0x7fc5b4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24bc1c6be0ebe28d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 1 -- 172.21.15.150:0/2336498378 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4009e70 con 0x7fc5c00a2810 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: dump: 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 5 monclient: authenticate success, global_id 14467 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 1 -- 172.21.15.150:0/2336498378 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400ba70 con 0x7fc5c00a2810 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:48.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 1 -- 172.21.15.150:0/2336498378 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400be30 con 0x7fc5c00a2810 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 4 set_mon_vals no callback set 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bf7fe700 10 monclient: dump: 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.388 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: shutdown 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/2336498378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a2810 msgr2=0x7fc5c00a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/2336498378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a2810 0x7fc5c00a1010 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0069e90 tx=0x7fc5b4009250 comp rx=0 tx=0).stop 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/2336498378 shutdown_connections 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/2336498378 >> v2:172.21.15.142:3300/0 conn(0x7fc5c00a1950 0x7fc5c0132e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/2336498378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a2810 0x7fc5c00a1010 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/2336498378 >> 172.21.15.150:0/2336498378 conn(0x7fc5c00a60b0 msgr2=0x7fc5c00a2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/2336498378 shutdown_connections 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/2336498378 wait complete. 2024-09-18T10:10:48.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: monmap: 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 Processor -- start 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- start start 2024-09-18T10:10:48.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: init 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-b con 0x7fc5c00a1550 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5c0141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-a con 0x7fc5c00a1950 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5c012f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): picked mon.noname-c con 0x7fc5c00a2810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c008c900 con 0x7fc5c00a1950 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c00a0d60 con 0x7fc5c00a1550 2024-09-18T10:10:48.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c0132d10 con 0x7fc5c00a2810 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c5cf4700 10 monclient(hunting): authenticate will time out at 930.475642s 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47484/0 (socket says 172.21.15.150:47484) 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 learned_addr learned my addr 172.21.15.150:0/3113951296 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request con 0x7fc5c00a1550 auth_method 0 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.378+0000 7fc5bffff700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5c0141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5c012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request con 0x7fc5c00a2810 auth_method 0 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c54f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c54f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_done global_id 14470 payload 824 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bffff700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 msgr2=0x7fc5c0141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bffff700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 msgr2=0x7fc5c0141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 msgr2=0x7fc5c012f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5c012f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bffff700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5c0141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:10:48.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 msgr2=0x7fc5c0141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bffff700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5c0141fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5c0141fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 monclient: found mon.noname-b 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b40094f0 con 0x7fc5c00a1550 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 msgr2=0x7fc5c0141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00a55b0 tx=0x7fc5b4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40119d0 con 0x7fc5c00a1550 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: dump: 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.396 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 5 monclient: authenticate success, global_id 14470 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4020070 con 0x7fc5c00a1550 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 10 monclient: _renew_subs 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c0131860 con 0x7fc5c00a1550 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4011da0 con 0x7fc5c00a1550 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5bcff9700 10 monclient: dump: 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.398 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5beffd700 4 set_mon_vals no callback set 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 10 monclient: _renew_subs 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.382+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c0131590 con 0x7fc5c00a1550 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc5b4022030 con 0x7fc5c00a1550 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5c5cf4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5c5cf4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5c5cf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c0132f50 con 0x7fc5c00a1550 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5bcff9700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 0x7fc5ac037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fc5b4049fc0 con 0x7fc5c00a1550 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5bffff700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 0x7fc5ac037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.386+0000 7fc5bffff700 10 monclient: get_auth_request con 0x7fc5ac035ac0 auth_method 0 2024-09-18T10:10:48.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.390+0000 7fc5bffff700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 0x7fc5ac037f70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006fd0 tx=0x7fc5b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.394+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc5b407a670 con 0x7fc5c00a1550 2024-09-18T10:10:48.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.394+0000 7fc5bcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.394+0000 7fc5bcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.403+0000 7f5a31c39700 10 monclient: get_monmap_and_config 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.403+0000 7f5a31c39700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.403+0000 7f5a31c39700 10 monclient: monmap: 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:10:48.407485+0000 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:10:48.407485+0000 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:10:48.408 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:10:48.409 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:10:48.409 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:10:48.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 Processor -- start 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 -- start start 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient: init 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a2c09abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5a2c09b740 0x7f5a2c134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-c con 0x7f5a2c09b740 addr v2:172.21.15.150:3300/0 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-b con 0x7f5a2c134bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2c09dc90 con 0x7f5a2c09abd0 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2c09b500 con 0x7f5a2c09b740 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c0967f0 con 0x7f5a2c134bb0 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31c39700 10 monclient(hunting): authenticate will time out at 937.332959s 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a30c37700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5a2c09abd0 0x7f5a2c0960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:60380/0 (socket says 172.21.15.142:60380) 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a30c37700 1 -- 172.21.15.142:0/3731020465 learned_addr learned my addr 172.21.15.142:0/3731020465 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 --2- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): get_auth_request con 0x7f5a2c134bb0 auth_method 0 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3731020465 >> v2:172.21.15.150:3300/0 conn(0x7f5a2c09b740 0x7f5a2c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request con 0x7f5a2c09b740 auth_method 0 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient(hunting): handle_auth_done global_id 24161 payload 274 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 -- 172.21.15.142:0/3731020465 >> v2:172.21.15.150:3300/0 conn(0x7f5a2c09b740 msgr2=0x7f5a2c134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 --2- 172.21.15.142:0/3731020465 >> v2:172.21.15.150:3300/0 conn(0x7f5a2c09b740 0x7f5a2c134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 -- 172.21.15.142:0/3731020465 >> v1:172.21.15.142:6789/0 conn(0x7f5a2c09abd0 legacy=0x7f5a2c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 monclient: found mon.noname-b 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.407+0000 7f5a31438700 1 -- 172.21.15.142:0/3731020465 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c09afa0 con 0x7f5a2c134bb0 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31438700 1 -- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 msgr2=0x7f5a2c136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31438700 1 --2- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c136f90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009e60 tx=0x7f5a24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e119f1bb584b92f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 1 -- 172.21.15.142:0/3731020465 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2400e070 con 0x7f5a2c134bb0 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: dump: 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.419 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 1 -- 172.21.15.142:0/3731020465 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2400bd70 con 0x7f5a2c134bb0 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 5 monclient: authenticate success, global_id 24161 2024-09-18T10:10:48.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 1 -- 172.21.15.142:0/3731020465 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24012680 con 0x7f5a2c134bb0 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 10 monclient: get_monmap_and_config success 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 4 set_mon_vals no callback set 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a2b7fe700 10 monclient: dump: 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.421 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 10 monclient: shutdown 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 msgr2=0x7f5a2c136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c136f90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009e60 tx=0x7f5a24009780 comp rx=0 tx=0).stop 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- 172.21.15.142:0/3731020465 shutdown_connections 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3731020465 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c136f90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3731020465 >> v2:172.21.15.150:3300/0 conn(0x7f5a2c09b740 0x7f5a2c134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- 172.21.15.142:0/3731020465 >> 172.21.15.142:0/3731020465 conn(0x7f5a2c070ef0 msgr2=0x7f5a2c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- 172.21.15.142:0/3731020465 shutdown_connections 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- 172.21.15.142:0/3731020465 wait complete. 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 10 monclient: build_initial_monmap 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 10 monclient: monmap: 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 Processor -- start 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.411+0000 7f5a31c39700 1 -- start start 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient: init 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient: _reopen_session rank -1 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c09abd0 0x7f5a2c08b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-b con 0x7f5a2c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 0x7f5a2c088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-a con 0x7f5a2c09b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c0894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): picked mon.noname-c con 0x7f5a2c134bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c09f510 con 0x7f5a2c09b740 2024-09-18T10:10:48.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c08c380 con 0x7f5a2c09abd0 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c09d1c0 con 0x7f5a2c134bb0 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): _renew_subs 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31c39700 10 monclient(hunting): authenticate will time out at 937.341093s 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 0x7f5a2c088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c0894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 0x7f5a2c088fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36320/0 (socket says 172.21.15.142:36320) 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 1 -- 172.21.15.142:0/3184508963 learned_addr learned my addr 172.21.15.142:0/3184508963 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c0894f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55774/0 (socket says 172.21.15.142:55774) 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request con 0x7f5a2c09b740 auth_method 0 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): get_auth_request con 0x7f5a2c134bb0 auth_method 0 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c09abd0 0x7f5a2c08b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 10 monclient(hunting): get_auth_request con 0x7f5a2c09abd0 auth_method 0 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient(hunting): handle_auth_done global_id 24164 payload 824 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c09abd0 msgr2=0x7f5a2c08b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c09abd0 0x7f5a2c08b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 msgr2=0x7f5a2c088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 0x7f5a2c088fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient: _finish_hunting 0 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 monclient: found mon.noname-c 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24009a20 con 0x7f5a2c134bb0 2024-09-18T10:10:48.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a30c37700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c09abd0 0x7f5a2c08b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:10:48.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.415+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2c09b740 0x7f5a2c088fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:10:48.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 msgr2=0x7f5a2c0894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31438700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c0894f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c076de0 tx=0x7f5a24018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24012e30 con 0x7f5a2c134bb0 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: dump: 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: _finish_auth 0 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 5 monclient: authenticate success, global_id 24164 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2401e640 con 0x7f5a2c134bb0 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 monclient: _renew_subs 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2401ea00 con 0x7f5a2c134bb0 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c08af30 con 0x7f5a2c134bb0 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2affd700 4 set_mon_vals no callback set 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 10 monclient: dump: 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 monclient: _renew_subs 2024-09-18T10:10:48.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c08c150 con 0x7f5a2c134bb0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5a24024070 con 0x7f5a2c134bb0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a10035ac0 0x7f5a10037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f5a2404aa40 con 0x7f5a2c134bb0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 0x7f5a2c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c0648a0 con 0x7f5a2c089a30 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a30c37700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a10035ac0 0x7f5a10037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a30c37700 10 monclient: get_auth_request con 0x7f5a10035ac0 auth_method 0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 0x7f5a2c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient: get_auth_request con 0x7f5a2c089a30 auth_method 0 2024-09-18T10:10:48.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_done global_id 14479 payload 824 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.419+0000 7f5a2bfff700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a2c072a80 con 0x7f5a2c089a30 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a30c37700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a10035ac0 0x7f5a10037f70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a18005300 tx=0x7f5a18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a2bfff700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 msgr2=0x7f5a2c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 0x7f5a2c06c370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a20011ef0 tx=0x7f5a200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20014860 con 0x7f5a2c089a30 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:48.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 10 monclient: dump: 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5a200149c0 con 0x7f5a2c089a30 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:10:48.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:10:48.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 msgr2=0x7f5a2c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 0x7f5a2c06c370 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a20011ef0 tx=0x7f5a200091a0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 msgr2=0x7f5a2c06c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.423+0000 7f5a28ff9700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c089a30 0x7f5a2c06c370 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a31c39700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 0x7f5a2c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a31c39700 10 monclient(hunting): start opening mon connection 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c08c5c0 con 0x7f5a2c06c8b0 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 0x7f5a2c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:10:48.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient: get_auth_request con 0x7f5a2c06c8b0 auth_method 0 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 10 monclient(hunting): handle_auth_done global_id 14482 payload 824 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.435+0000 7f5a2bfff700 1 -- 172.21.15.142:0/3184508963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a2c089e00 con 0x7f5a2c06c8b0 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a2bfff700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 msgr2=0x7f5a2c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a2bfff700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 0x7f5a2c06ccb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5a20011ef0 tx=0x7f5a2001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20024040 con 0x7f5a2c06c8b0 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:10:48.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 10 monclient: dump: 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f5a20014650 con 0x7f5a2c06c8b0 2024-09-18T10:10:48.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 msgr2=0x7f5a2c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 0x7f5a2c06ccb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5a20011ef0 tx=0x7f5a2001c000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 msgr2=0x7f5a2c06ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a28ff9700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2c06c8b0 0x7f5a2c06ccb0 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5a20011ef0 tx=0x7f5a2001c000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:10:48.445 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 32, 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.446 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:10:48.448 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:10:48.449 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:10:48.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:10:48.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:10:48.452 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.455 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:10:48.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:10:48.457 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:10:48.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:10:48.458 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:10:48.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:10:48.458 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:10:48.458 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a10035ac0 msgr2=0x7f5a10037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a10035ac0 0x7f5a10037f70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5a18005300 tx=0x7f5a18009040 comp rx=0 tx=0).stop 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 10 monclient: shutdown 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 msgr2=0x7f5a2c0894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 --2- 172.21.15.142:0/3184508963 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2c134bb0 0x7f5a2c0894f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c076de0 tx=0x7f5a24018000 comp rx=0 tx=0).stop 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 reap_dead start 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31438700 1 -- 172.21.15.142:0/3184508963 reap_dead start 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 shutdown_connections 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 >> 172.21.15.142:0/3184508963 conn(0x7f5a2c070ef0 msgr2=0x7f5a2c09de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 shutdown_connections 2024-09-18T10:10:48.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:10:48.439+0000 7f5a31c39700 1 -- 172.21.15.142:0/3184508963 wait complete. 2024-09-18T10:10:48.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:10:48.461 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:10:48.461 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-18T10:10:48.461 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:10:48.461 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:10:48.461 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:10:48.461 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:10:48.474 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:10:48.474 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T10:10:48.475 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T10:10:48.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:10:48.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.562+0000 7fc5c5cf4700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:10:48.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.562+0000 7fc5c5cf4700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:10:48.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.562+0000 7fc5c5cf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:10:48.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:48.562+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fc5c004bbf0 con 0x7fc5c00a1550 2024-09-18T10:10:51.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:51.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:51.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:51.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:51.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:51.382+0000 7fc5a77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:10:52.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:52.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:52.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:52.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:53.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:53.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:53.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:53.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:54.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:54.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:54.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:54.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:55.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:55.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:55.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:55.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:56.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:56.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:57.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:57.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:58.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:58.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:10:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:59.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:10:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:10:59.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:00.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:00.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:00.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:00.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:01.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:01.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:01.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:01.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:01.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:01.382+0000 7fc5a77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:02.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:02.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:02.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:02.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:03.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:03.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:03.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:03.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:04.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:04.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:04.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:04.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:05.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:05.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:05.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:05.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:06.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:06.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:06.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:06.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:07.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:07.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:07.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:07.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:08.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:08.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:08.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:08.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:08.476 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T10:11:08.476 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T10:11:08.476 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:11:08.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:11:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: build_initial_monmap 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: monmap: 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:08.582158+0000 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:08.582158+0000 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:08.583 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 Processor -- start 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 -- start start 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: init 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f403c09aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f403c0967a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f403c09c320 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c095880 con 0x7f403c09c320 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403c09aeb0 con 0x7f403c0967a0 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c06f6d0 con 0x7f403c09aae0 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403b59e700 10 monclient(hunting): authenticate will time out at 957.507646s 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403a59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 msgr2=0x7f403c096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f403a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49384/0 (socket says 172.21.15.142:49384) 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 learned_addr learned my addr 172.21.15.142:0/2250483685 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f403c0967a0 auth_method 0 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.578+0000 7f4039d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.582+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.582+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.582+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.582+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.582+0000 7f403ad9d700 1 -- 172.21.15.142:0/2250483685 >> v1:172.21.15.142:6789/0 conn(0x7f403c09c320 legacy=0x7f403c136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:08.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.782+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 msgr2=0x7f403c096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:08.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.782+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:11:08.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f403c0967a0 auth_method 0 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:08.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:08.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:11:09.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.186+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 msgr2=0x7f403c096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:09.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.186+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f403c0967a0 auth_method 0 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:09.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:11:09.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:09.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:09.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:09.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:09.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.990+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 msgr2=0x7f403c096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:09.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.990+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:11:09.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:09.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f403c0967a0 auth_method 0 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:09.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:09.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:11:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:10.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 msgr2=0x7fc5c0141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 msgr2=0x7fc5c0141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00a55b0 tx=0x7fc5b4016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5c00a55b0 tx=0x7fc5b4016000 comp rx=0 tx=0).stop 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 msgr2=0x7fc5c0141a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c00a1550 0x7fc5c0141a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac03b840 0x7fc5ac03dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): picked mon.b con 0x7fc5ac03b840 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5ac040520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:10.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): picked mon.a con 0x7fc5c00a1950 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5ac042f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): picked mon.c con 0x7fc5c00a2810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac043890 con 0x7fc5c00a1950 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac0439d0 con 0x7fc5ac03b840 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ac043b60 con 0x7fc5c00a2810 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5bcff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 msgr2=0x7fc5ac042f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:10.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5ac042f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5ac040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac03b840 0x7fc5ac03dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request con 0x7fc5c00a1950 auth_method 0 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:10.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request con 0x7fc5ac03b840 auth_method 0 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.878+0000 7fc5c4cf2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient(hunting): handle_auth_done global_id 14470 payload 293 2024-09-18T10:11:10.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 msgr2=0x7fc5ac042f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5ac042f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac03b840 msgr2=0x7fc5ac03dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac03b840 0x7fc5ac03dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient: _finish_hunting 0 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 monclient: found mon.a 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fc5ac043cf0 con 0x7fc5c00a1950 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fc5b8007540 con 0x7fc5c00a1950 2024-09-18T10:11:10.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 10 monclient: _finish_auth 0 2024-09-18T10:11:10.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.882+0000 7fc5c54f3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:10.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.886+0000 7fc5c54f3700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 msgr2=0x7fc5ac040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:10.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.886+0000 7fc5c54f3700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5ac040520 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80079d0 tx=0x7fc5b8007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:10.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800e030 con 0x7fc5c00a1950 2024-09-18T10:11:10.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:10.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 10 monclient: dump: 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:10.894 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b8011790 con 0x7fc5c00a1950 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc5b8011900 con 0x7fc5c00a1950 2024-09-18T10:11:10.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:10.890+0000 7fc5beffd700 4 set_mon_vals no callback set 2024-09-18T10:11:11.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:11.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:11.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:11.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:11.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:11.382+0000 7fc5a77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): tick 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 monclient(hunting): continuing hunt 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 msgr2=0x7f403c096260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f403c09aae0 0x7f403c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 msgr2=0x7f403c09bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.142:3300/0 conn(0x7f403c0967a0 0x7f403c09bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 -- 172.21.15.142:0/2250483685 >> v1:172.21.15.142:6789/0 conn(0x7f403c09c320 legacy=0x7f403c136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient: _add_conns ranks=[1,4] 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f401c0037b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f401c0074d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 -- 172.21.15.142:0/2250483685 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403c0955a0 con 0x7f401c0037b0 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 1 -- 172.21.15.142:0/2250483685 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f403c09aeb0 con 0x7f401c0074d0 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.578+0000 7f402bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f401c0074d0 auth_method 0 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:11.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.582+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> v1:172.21.15.150:6789/0 conn(0x7f401c0037b0 legacy=0x7f401c005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f401c0074d0 auth_method 0 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:11.786+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:11:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f401c0074d0 auth_method 0 2024-09-18T10:11:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:11:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.190+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:11:12.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:12.386+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:12.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:12.386+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f401c0074d0 auth_method 0 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:12.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:12.994+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:11:13.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:13.386+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:13.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:13.386+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168172901 0 0) 0x7f403c0955a0 con 0x7f401c0037b0 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): my global_id is 14485 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024003570 con 0x7f401c0037b0 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360225085 0 0) 0x7f4024003570 con 0x7f401c0037b0 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 msgr2=0x7f401c009980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 --2- 172.21.15.142:0/2250483685 >> v2:172.21.15.150:3300/0 conn(0x7f401c0074d0 0x7f401c009980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: _finish_hunting 0 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c00b050 con 0x7f401c0037b0 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: _finish_auth 0 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403b59e700 5 monclient: authenticate success, global_id 14485 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f4030004450 con 0x7f401c0037b0 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:11:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: dump: 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 >> v1:172.21.15.150:6789/0 conn(0x7f401c0037b0 legacy=0x7f401c005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403ad9d700 1 -- 172.21.15.142:0/2250483685 reap_dead start 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f4024003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): picked mon.a con 0x7f4024003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 0x7f4024004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): picked mon.b con 0x7f4024004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 0x7f402400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): picked mon.c con 0x7f40240052e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c09aeb0 con 0x7f4024003a60 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402400c250 con 0x7f4024004810 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402400c410 con 0x7f40240052e0 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403959a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403b59e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403ad9d700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 msgr2=0x7f402400ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f4024003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403ad9d700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 0x7f402400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): get_auth_request con 0x7f4024003a60 auth_method 0 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 0x7f4024004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): get_auth_request con 0x7f4024004810 auth_method 0 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient(hunting): handle_auth_done global_id 14485 payload 293 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 msgr2=0x7f402400ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 0x7f402400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 msgr2=0x7f4024004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 0x7f4024004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 monclient: found mon.a 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4024003570 con 0x7f4024003a60 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 10 monclient: _finish_auth 0 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f403a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.530+0000 7f4039d9b700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 0x7f4024004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.534+0000 7f403a59c700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 msgr2=0x7f4024003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.534+0000 7f403a59c700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f4024003e50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4030004630 tx=0x7f4030007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8399bf89bad42d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030004970 con 0x7f4024003a60 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: dump: 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4030004ad0 con 0x7f4024003a60 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 4 set_mon_vals no callback set 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 1 -- 172.21.15.142:0/2250483685 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030004e90 con 0x7f4024003a60 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403959a700 10 monclient: dump: 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.553 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 10 monclient: shutdown 2024-09-18T10:11:13.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:13.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 -- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 msgr2=0x7f4024003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f4024003e50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4030004630 tx=0x7f4030007d20 comp rx=0 tx=0).stop 2024-09-18T10:11:13.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 -- 172.21.15.142:0/2250483685 shutdown_connections 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40240052e0 0x7f402400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4024004810 0x7f4024004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 --2- 172.21.15.142:0/2250483685 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f4024003e50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 -- 172.21.15.142:0/2250483685 >> 172.21.15.142:0/2250483685 conn(0x7f403c0a1fe0 msgr2=0x7f403c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 -- 172.21.15.142:0/2250483685 shutdown_connections 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.546+0000 7f403b59e700 1 -- 172.21.15.142:0/2250483685 wait complete. 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient: build_initial_monmap 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient: monmap: 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:13.557 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 Processor -- start 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 -- start start 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient: init 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f4024003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 0x7f403c08bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f4024004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f40240052e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c08d360 con 0x7f4024003a60 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403c06f8d0 con 0x7f40240052e0 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403c06e830 con 0x7f4024004810 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403b59e700 10 monclient(hunting): authenticate will time out at 962.478541s 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f4039d9b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 msgr2=0x7f403c08bd20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:13.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f4039d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 0x7f403c08bd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403a59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 msgr2=0x7f403c08b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403a59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 msgr2=0x7f403c08b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50030/0 (socket says 172.21.15.142:50030) 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 -- 172.21.15.142:0/1357799583 learned_addr learned my addr 172.21.15.142:0/1357799583 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): get_auth_request con 0x7f40240052e0 auth_method 0 2024-09-18T10:11:13.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient(hunting): handle_auth_done global_id 14488 payload 824 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 -- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 msgr2=0x7f403c08bd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 0x7f403c08bd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 -- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 msgr2=0x7f403c08b7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 monclient: found mon.noname-b 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:13.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:13.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.550+0000 7f403ad9d700 1 -- 172.21.15.142:0/1357799583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403c08ee00 con 0x7f40240052e0 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403ad9d700 1 -- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 msgr2=0x7f403c08c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403ad9d700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f401c009fe0 tx=0x7f4034007b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340072e0 con 0x7f40240052e0 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:13.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: dump: 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 5 monclient: authenticate success, global_id 14488 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034007460 con 0x7f40240052e0 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 10 monclient: _renew_subs 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403c08dea0 con 0x7f40240052e0 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034005810 con 0x7f40240052e0 2024-09-18T10:11:13.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f402a7fc700 10 monclient: dump: 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f4038d99700 4 set_mon_vals no callback set 2024-09-18T10:11:13.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 10 monclient: _renew_subs 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.554+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403c094f20 con 0x7f40240052e0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4034005c40 con 0x7f40240052e0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403c08f200 con 0x7f40240052e0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f402a7fc700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 0x7f401c0389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f40340473f0 con 0x7f40240052e0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403a59c700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 0x7f401c0389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.558+0000 7f403a59c700 10 monclient: get_auth_request con 0x7f401c036500 auth_method 0 2024-09-18T10:11:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.562+0000 7f403a59c700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 0x7f401c0389b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f403000d010 tx=0x7f4030017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.570+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f40340196e0 con 0x7f40240052e0 2024-09-18T10:11:13.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.570+0000 7f402a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:13.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.570+0000 7f402a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:13.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f403b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:11:13.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f403b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:11:13.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f403b59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:13.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f403c072a80 con 0x7f40240052e0 2024-09-18T10:11:13.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f402a7fc700 1 -- 172.21.15.142:0/1357799583 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f4034017080 con 0x7f40240052e0 2024-09-18T10:11:13.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f402a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:11:13.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.794+0000 7f402a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:13.800 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 msgr2=0x7f401c0389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 0x7f401c0389b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f403000d010 tx=0x7f4030017000 comp rx=0 tx=0).stop 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 10 monclient: shutdown 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 msgr2=0x7f403c08c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f401c009fe0 tx=0x7f4034007b60 comp rx=0 tx=0).stop 2024-09-18T10:11:13.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 shutdown_connections 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f401c036500 0x7f401c0389b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40240052e0 0x7f403c08c260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4024004810 0x7f403c08bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 --2- 172.21.15.142:0/1357799583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4024003a60 0x7f403c08b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 >> 172.21.15.142:0/1357799583 conn(0x7f403c0a1fe0 msgr2=0x7f403c06de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 shutdown_connections 2024-09-18T10:11:13.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.798+0000 7f403b59e700 1 -- 172.21.15.142:0/1357799583 wait complete. 2024-09-18T10:11:13.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:11:13.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:11:13.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: get_monmap_and_config 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: build_initial_monmap 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: monmap: 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:13.917467+0000 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:13.917467+0000 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:13.918 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:13.919 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:13.919 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:13.919 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 Processor -- start 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 -- start start 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: init 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f834409a6a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 0x7f834409b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-c con 0x7f834406f7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f834409b6c0 0x7f834413d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-a con 0x7f834409b6c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834406fbf0 con 0x7f834409a6a0 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834409dae0 con 0x7f834409b6c0 2024-09-18T10:11:13.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8344099bc0 con 0x7f834406f7f0 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834b0b9700 10 monclient(hunting): authenticate will time out at 962.843016s 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f834409b6c0 0x7f834413d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f834409b6c0 0x7f834413d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56330/0 (socket says 172.21.15.142:56330) 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 -- 172.21.15.142:0/2506593087 learned_addr learned my addr 172.21.15.142:0/2506593087 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:13.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): get_auth_request con 0x7f834409b6c0 auth_method 0 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 1 --2- 172.21.15.142:0/2506593087 >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 0x7f834409b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 10 monclient(hunting): get_auth_request con 0x7f834406f7f0 auth_method 0 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient(hunting): handle_auth_done global_id 4779 payload 274 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 -- 172.21.15.142:0/2506593087 >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 msgr2=0x7f834409b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 --2- 172.21.15.142:0/2506593087 >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 0x7f834409b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 -- 172.21.15.142:0/2506593087 >> v1:172.21.15.150:6789/0 conn(0x7f834409a6a0 legacy=0x7f834409aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient: _finish_hunting 0 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 monclient: found mon.noname-a 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:13.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f834a8b8700 1 -- 172.21.15.142:0/2506593087 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83440998e0 con 0x7f834409b6c0 2024-09-18T10:11:13.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.914+0000 7f83498b6700 1 --2- 172.21.15.142:0/2506593087 >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 0x7f834409b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:11:13.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834a8b8700 1 -- 172.21.15.142:0/2506593087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 msgr2=0x7f834413d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834a8b8700 1 --2- 172.21.15.142:0/2506593087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834413d6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f833c0099b0 tx=0x7f833c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9eb43b1c1c01b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 1 -- 172.21.15.142:0/2506593087 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c00ba30 con 0x7f834409b6c0 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: dump: 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.935 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: _finish_auth 0 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 5 monclient: authenticate success, global_id 4779 2024-09-18T10:11:13.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 1 -- 172.21.15.142:0/2506593087 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c00bb90 con 0x7f834409b6c0 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 4 set_mon_vals no callback set 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 1 -- 172.21.15.142:0/2506593087 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c011420 con 0x7f834409b6c0 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f83490b5700 10 monclient: dump: 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.937 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 10 monclient: shutdown 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 -- 172.21.15.142:0/2506593087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 msgr2=0x7f834413d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 --2- 172.21.15.142:0/2506593087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834413d6f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f833c0099b0 tx=0x7f833c009250 comp rx=0 tx=0).stop 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 -- 172.21.15.142:0/2506593087 shutdown_connections 2024-09-18T10:11:13.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 --2- 172.21.15.142:0/2506593087 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834413d6f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 --2- 172.21.15.142:0/2506593087 >> v2:172.21.15.150:3300/0 conn(0x7f834406f7f0 0x7f834409b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 -- 172.21.15.142:0/2506593087 >> 172.21.15.142:0/2506593087 conn(0x7f8344070ef0 msgr2=0x7f83440712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 -- 172.21.15.142:0/2506593087 shutdown_connections 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 1 -- 172.21.15.142:0/2506593087 wait complete. 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 10 monclient: build_initial_monmap 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.918+0000 7f834b0b9700 10 monclient: monmap: 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:13.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 Processor -- start 2024-09-18T10:11:13.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 -- start start 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient: init 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f834406f7f0 0x7f834413d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-c con 0x7f834406f7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f834409a6a0 0x7f834406bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-b con 0x7f834409a6a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): picked mon.noname-a con 0x7f834409b6c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834406e160 con 0x7f834409b6c0 2024-09-18T10:11:13.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834409ff60 con 0x7f834409a6a0 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8344126a30 con 0x7f834406f7f0 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834b0b9700 10 monclient(hunting): authenticate will time out at 962.848919s 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a0b7700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f834406f7f0 msgr2=0x7f834413d500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56332/0 (socket says 172.21.15.142:56332) 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 learned_addr learned my addr 172.21.15.142:0/3394840983 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): get_auth_request con 0x7f834409b6c0 auth_method 0 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a0b7700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f834406f7f0 0x7f834413d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f83498b6700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f834409a6a0 0x7f834406bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f83498b6700 10 monclient(hunting): get_auth_request con 0x7f834409a6a0 auth_method 0 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f83498b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f83498b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f83498b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient(hunting): handle_auth_done global_id 4782 payload 824 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f834406f7f0 msgr2=0x7f834413d500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f834406f7f0 0x7f834413d500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f834409a6a0 msgr2=0x7f834406bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f834409a6a0 0x7f834406bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient: _finish_hunting 0 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 monclient: found mon.noname-a 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0094f0 con 0x7f834409b6c0 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 msgr2=0x7f834406c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f834a8b8700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8344098ba0 tx=0x7f833c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01b650 con 0x7f834409b6c0 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f833a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f833a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.922+0000 7f833a7fc700 10 monclient: dump: 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.945 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833c01f070 con 0x7f834409b6c0 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 5 monclient: authenticate success, global_id 4782 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 monclient: _renew_subs 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8344126750 con 0x7f834409b6c0 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833c01ba20 con 0x7f834409b6c0 2024-09-18T10:11:13.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 10 monclient: dump: 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 monclient: _renew_subs 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f834406d930 con 0x7f834409b6c0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833bfff700 4 set_mon_vals no callback set 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 0x7f8344092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83440648a0 con 0x7f834406c9a0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f833c01a440 con 0x7f834409b6c0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 0x7f8344092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient: get_auth_request con 0x7f834406c9a0 auth_method 0 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8328035a70 0x7f8328037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7f833c021070 con 0x7f834409b6c0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 10 monclient(hunting): handle_auth_done global_id 4785 payload 824 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f834a0b7700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f834406cd70 con 0x7f834406c9a0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f83498b6700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8328035a70 0x7f8328037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.926+0000 7f83498b6700 10 monclient: get_auth_request con 0x7f8328035a70 auth_method 0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f834a0b7700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 msgr2=0x7f8344092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f834a0b7700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 0x7f8344092800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334007620 con 0x7f834406c9a0 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 10 monclient: dump: 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.950 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f83498b6700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8328035a70 0x7f8328037f20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f834406dcb0 tx=0x7f8340009860 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8334005810 con 0x7f834406c9a0 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 msgr2=0x7f8344092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 0x7f8344092800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334007e80 comp rx=0 tx=0).stop 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 msgr2=0x7f8344092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:13.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.930+0000 7f833a7fc700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834406c9a0 0x7f8344092800 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334007e80 comp rx=0 tx=0).stop 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834b0b9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834b0b9700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 0x7f8344093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834b0b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834406fbf0 con 0x7f8344092d40 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 0x7f8344093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient: get_auth_request con 0x7f8344092d40 auth_method 0 2024-09-18T10:11:13.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.938+0000 7f834a0b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f834a0b7700 10 monclient(hunting): handle_auth_done global_id 4788 payload 824 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f834a0b7700 1 -- 172.21.15.142:0/3394840983 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8344072a80 con 0x7f8344092d40 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f834a0b7700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 msgr2=0x7f8344093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:13.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f834a0b7700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 0x7f8344093140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334020620 con 0x7f8344092d40 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 10 monclient: dump: 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:13.955 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f833401f370 con 0x7f8344092d40 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 msgr2=0x7f8344093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 0x7f8344093140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334015000 comp rx=0 tx=0).stop 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 msgr2=0x7f8344093140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:13.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.942+0000 7f833a7fc700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8344092d40 0x7f8344093140 secure :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f833400c440 tx=0x7f8334015000 comp rx=0 tx=0).stop 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 36, 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.957 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:11:13.958 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:11:13.959 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:11:13.960 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:11:13.961 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:13.962 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:13.963 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:13.964 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:13.965 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:13.966 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:11:13.967 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.968 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:13.969 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:11:13.970 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:11:13.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8328035a70 msgr2=0x7f8328037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8328035a70 0x7f8328037f20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f834406dcb0 tx=0x7f8340009860 comp rx=0 tx=0).stop 2024-09-18T10:11:13.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 10 monclient: shutdown 2024-09-18T10:11:13.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 msgr2=0x7f834406c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8344098ba0 tx=0x7f833c016000 comp rx=0 tx=0).stop 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834a8b8700 1 -- 172.21.15.142:0/3394840983 reap_dead start 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 shutdown_connections 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 --2- 172.21.15.142:0/3394840983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f834409b6c0 0x7f834406c460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 >> 172.21.15.142:0/3394840983 conn(0x7f8344070ef0 msgr2=0x7f834409c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 shutdown_connections 2024-09-18T10:11:13.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:13.946+0000 7f834b0b9700 1 -- 172.21.15.142:0/3394840983 wait complete. 2024-09-18T10:11:13.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:11:13.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:11:13.974 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: get_monmap_and_config 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: build_initial_monmap 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: monmap: 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:14.088000+0000 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:14.088000+0000 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.089 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 Processor -- start 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 -- start start 2024-09-18T10:11:14.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: init 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc48c09c2c0 0x7fc48c0999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-a con 0x7fc48c09c2c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc48c0a0110 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 0x7fc48c1398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-c con 0x7fc48c0a0c20 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:14.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48c0a6620 con 0x7fc48c0a0110 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48c09ab70 con 0x7fc48c09c2c0 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48c0a1c70 con 0x7fc48c0a0c20 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc490e7f700 10 monclient(hunting): authenticate will time out at 963.013640s 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48affd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc48c0a0110 0x7fc48c0a0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:58302/0 (socket says 172.21.15.142:58302) 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48affd700 1 -- 172.21.15.142:0/1643266434 learned_addr learned my addr 172.21.15.142:0/1643266434 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/1643266434 >> v2:172.21.15.142:3300/0 conn(0x7fc48c09c2c0 0x7fc48c0999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 1 --2- 172.21.15.142:0/1643266434 >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 0x7fc48c1398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): get_auth_request con 0x7fc48c09c2c0 auth_method 0 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): get_auth_request con 0x7fc48c0a0c20 auth_method 0 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_done global_id 4791 payload 274 2024-09-18T10:11:14.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 -- 172.21.15.142:0/1643266434 >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 msgr2=0x7fc48c1398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/1643266434 >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 0x7fc48c1398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 -- 172.21.15.142:0/1643266434 >> v1:172.21.15.142:6789/0 conn(0x7fc48c0a0110 legacy=0x7fc48c0a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48b7fe700 1 -- 172.21.15.142:0/1643266434 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c09c690 con 0x7fc48c09c2c0 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.086+0000 7fc48bfff700 1 --2- 172.21.15.142:0/1643266434 >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 0x7fc48c1398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48b7fe700 1 -- 172.21.15.142:0/1643266434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c09c2c0 msgr2=0x7fc48c0999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/1643266434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c09c2c0 0x7fc48c0999e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc48c067de0 tx=0x7fc4800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de551e55d326cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 1 -- 172.21.15.142:0/1643266434 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480011840 con 0x7fc48c09c2c0 2024-09-18T10:11:14.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:14.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: dump: 2024-09-18T10:11:14.102 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 5 monclient: authenticate success, global_id 4791 2024-09-18T10:11:14.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 1 -- 172.21.15.142:0/1643266434 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4800119a0 con 0x7fc48c09c2c0 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 4 set_mon_vals no callback set 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 1 -- 172.21.15.142:0/1643266434 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480011d40 con 0x7fc48c09c2c0 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc48a7fc700 10 monclient: dump: 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.104 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: shutdown 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- 172.21.15.142:0/1643266434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c09c2c0 msgr2=0x7fc48c0999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 --2- 172.21.15.142:0/1643266434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c09c2c0 0x7fc48c0999e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc48c067de0 tx=0x7fc4800091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- 172.21.15.142:0/1643266434 shutdown_connections 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 --2- 172.21.15.142:0/1643266434 >> v2:172.21.15.150:3300/0 conn(0x7fc48c0a0c20 0x7fc48c1398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 --2- 172.21.15.142:0/1643266434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c09c2c0 0x7fc48c0999e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- 172.21.15.142:0/1643266434 >> 172.21.15.142:0/1643266434 conn(0x7fc48c0a5720 msgr2=0x7fc48c0a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:14.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- 172.21.15.142:0/1643266434 shutdown_connections 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- 172.21.15.142:0/1643266434 wait complete. 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: build_initial_monmap 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: monmap: 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:14.106 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 Processor -- start 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 -- start start 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: init 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.090+0000 7fc490e7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc48c0a0110 0x7fc48c141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-c con 0x7fc48c0a0110 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 0x7fc48c1420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-a con 0x7fc48c0a0c20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08e030 0x7fc48c08e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): picked mon.noname-b con 0x7fc48c08e030 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c092870 con 0x7fc48c0a0c20 2024-09-18T10:11:14.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c0918a0 con 0x7fc48c08e030 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c08ffa0 con 0x7fc48c0a0110 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient(hunting): authenticate will time out at 963.020052s 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc48c0a0110 msgr2=0x7fc48c141b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 0x7fc48c1420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc48c0a0110 0x7fc48c141b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 0x7fc48c1420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56358/0 (socket says 172.21.15.142:56358) 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 -- 172.21.15.142:0/2154344347 learned_addr learned my addr 172.21.15.142:0/2154344347 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): get_auth_request con 0x7fc48c0a0c20 auth_method 0 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48bfff700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08e030 0x7fc48c08e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48bfff700 10 monclient(hunting): get_auth_request con 0x7fc48c08e030 auth_method 0 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient(hunting): handle_auth_done global_id 4794 payload 824 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc48c0a0110 msgr2=0x7fc48c141b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc48c0a0110 0x7fc48c141b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08e030 msgr2=0x7fc48c08e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08e030 0x7fc48c08e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 monclient: found mon.noname-a 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484009420 con 0x7fc48c0a0c20 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 msgr2=0x7fc48c1420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc48affd700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 0x7fc48c1420d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00cca0 tx=0x7fc47c005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c00ad80 con 0x7fc48c0a0c20 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: dump: 2024-09-18T10:11:14.110 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c0070f0 con 0x7fc48c0a0c20 2024-09-18T10:11:14.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 5 monclient: authenticate success, global_id 4794 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient: _renew_subs 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc480009420 con 0x7fc48c0a0c20 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c0074b0 con 0x7fc48c0a0c20 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc4737fe700 10 monclient: dump: 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.112 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc489ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient: _renew_subs 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.094+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c090660 con 0x7fc48c0a0c20 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc490e7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc47c0078a0 con 0x7fc48c0a0c20 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc490e7f700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 0x7fc48c029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c064750 con 0x7fc48c08f150 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc4737fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc478035a70 0x7fc478037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4434+0+0 (secure 0 0 0) 0x7fc47c048880 con 0x7fc48c0a0c20 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 0x7fc48c029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient: get_auth_request con 0x7fc48c08f150 auth_method 0 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48bfff700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc478035a70 0x7fc478037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48bfff700 10 monclient: get_auth_request con 0x7fc478035a70 auth_method 0 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_done global_id 14503 payload 824 2024-09-18T10:11:14.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.098+0000 7fc48b7fe700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc48c06e190 con 0x7fc48c08f150 2024-09-18T10:11:14.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc48bfff700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc478035a70 0x7fc478037f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4840093f0 tx=0x7fc484008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc48b7fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 msgr2=0x7fc48c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 0x7fc48c029d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc48c067de0 tx=0x7fc480016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48000bb20 con 0x7fc48c08f150 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 10 monclient: dump: 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.117 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc48000bc80 con 0x7fc48c08f150 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 msgr2=0x7fc48c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 0x7fc48c029d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc48c067de0 tx=0x7fc480016000 comp rx=0 tx=0).stop 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 msgr2=0x7fc48c029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.102+0000 7fc4737fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c08f150 0x7fc48c029d00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc490e7f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc490e7f700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 0x7fc48c02a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc490e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c072a80 con 0x7fc48c02a240 2024-09-18T10:11:14.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 0x7fc48c02a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc48b7fe700 10 monclient: get_auth_request con 0x7fc48c02a240 auth_method 0 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc48b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc48b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.110+0000 7fc48b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 10 monclient(hunting): handle_auth_done global_id 14506 payload 824 2024-09-18T10:11:14.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 1 -- 172.21.15.142:0/2154344347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc48c091a10 con 0x7fc48c02a240 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 msgr2=0x7fc48c02a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc48b7fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 0x7fc48c02a640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc48c072190 tx=0x7fc48001f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48001a720 con 0x7fc48c02a240 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 10 monclient: dump: 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:14.120 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fc48000b910 con 0x7fc48c02a240 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 msgr2=0x7fc48c02a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 0x7fc48c02a640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc48c072190 tx=0x7fc48001f040 comp rx=0 tx=0).stop 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 msgr2=0x7fc48c02a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.114+0000 7fc4737fe700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc48c02a240 0x7fc48c02a640 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc48c072190 tx=0x7fc48001f040 comp rx=0 tx=0).stop 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 36, 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:11:14.122 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:14.123 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:14.124 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:14.124 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:14.124 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:14.125 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:11:14.126 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:14.127 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:14.128 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:14.128 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:14.128 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:14.128 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:14.128 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:11:14.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:14.130 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:11:14.131 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.131 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.131 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:14.131 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:11:14.132 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.133 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:11:14.134 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.135 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.136 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.136 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.136 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:11:14.136 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:11:14.136 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:11:14.137 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:11:14.138 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:14.139 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.140 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.140 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:11:14.140 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:11:14.140 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.141 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:11:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:11:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:11:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:11:14.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:14.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:14.146 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:11:14.146 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:11:14.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc478035a70 msgr2=0x7fc478037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc478035a70 0x7fc478037f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc4840093f0 tx=0x7fc484008040 comp rx=0 tx=0).stop 2024-09-18T10:11:14.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 10 monclient: shutdown 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 msgr2=0x7fc48c1420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 --2- 172.21.15.142:0/2154344347 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc48c0a0c20 0x7fc48c1420d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00cca0 tx=0x7fc47c005480 comp rx=0 tx=0).stop 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc48bfff700 1 -- 172.21.15.142:0/2154344347 reap_dead start 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc48bfff700 1 -- 172.21.15.142:0/2154344347 reap_dead start 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 shutdown_connections 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 >> 172.21.15.142:0/2154344347 conn(0x7fc48c0a5720 msgr2=0x7fc48c09e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 shutdown_connections 2024-09-18T10:11:14.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:14.118+0000 7fc490e7f700 1 -- 172.21.15.142:0/2154344347 wait complete. 2024-09-18T10:11:14.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:11:14.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:11:14.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.382+0000 7fc5a77fe700 10 monclient: tick 2024-09-18T10:11:14.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.382+0000 7fc5a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:14.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.510+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 v45) v1 ==== 126+0+0 (secure 0 0 0) 0x7fc5b801fe10 con 0x7fc5c00a1950 2024-09-18T10:11:14.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.510+0000 7fc5bcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.510+0000 7fc5bcff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:14.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5c5cf4700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5c5cf4700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5c5cf4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fc5c004bbf0 con 0x7fc5c00a1950 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5bcff9700 1 -- 172.21.15.150:0/3113951296 <== mon.0 v2:172.21.15.142:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v45) v1 ==== 149+0+0 (secure 0 0 0) 0x7fc5b801fe10 con 0x7fc5c00a1950 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5bcff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.614+0000 7fc5bcff9700 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-18T10:11:14.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 msgr2=0x7fc5ac037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 0x7fc5ac037f70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006fd0 tx=0x7fc5b0008040 comp rx=0 tx=0).stop 2024-09-18T10:11:14.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 10 monclient: shutdown 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 msgr2=0x7fc5ac040520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5ac040520 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80079d0 tx=0x7fc5b8007a70 comp rx=0 tx=0).stop 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 shutdown_connections 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc5c00a2810 0x7fc5ac042f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc5c00a1950 0x7fc5ac040520 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5ac03b840 0x7fc5ac03dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 --2- 172.21.15.150:0/3113951296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc5ac035ac0 0x7fc5ac037f70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 >> 172.21.15.150:0/3113951296 conn(0x7fc5c00a60b0 msgr2=0x7fc5c006db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 shutdown_connections 2024-09-18T10:11:14.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.618+0000 7fc5c5cf4700 1 -- 172.21.15.150:0/3113951296 wait complete. 2024-09-18T10:11:14.624 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo already exists 2024-09-18T10:11:14.636 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-simple foo default host 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 10 monclient: get_monmap_and_config 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 10 monclient: build_initial_monmap 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 10 monclient: monmap: 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:14.742275+0000 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:14.742275+0000 2024-09-18T10:11:14.742 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 1 Processor -- start 2024-09-18T10:11:14.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 1 -- start start 2024-09-18T10:11:14.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.738+0000 7fae12093700 10 monclient: init 2024-09-18T10:11:14.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:14.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae0c071a50 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae0c130b80 0x7fae0c1352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): picked mon.noname-c con 0x7fae0c130b80 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fae0c06e850 0x7fae0c137820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): picked mon.noname-a con 0x7fae0c06e850 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c097820 con 0x7fae0c071a50 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0c096f80 con 0x7fae0c06e850 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0c071e20 con 0x7fae0c130b80 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:14.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae12093700 10 monclient(hunting): authenticate will time out at 956.837126s 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae0c130b80 0x7fae0c1352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0b7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fae0c071a50 0x7fae0c130470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41106/0 (socket says 172.21.15.150:41106) 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0b7fe700 1 -- 172.21.15.150:0/328010800 learned_addr learned my addr 172.21.15.150:0/328010800 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): get_auth_request con 0x7fae0c130b80 auth_method 0 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 1 --2- 172.21.15.150:0/328010800 >> v2:172.21.15.142:3300/0 conn(0x7fae0c06e850 0x7fae0c137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): get_auth_request con 0x7fae0c06e850 auth_method 0 2024-09-18T10:11:14.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient(hunting): handle_auth_done global_id 14509 payload 274 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 -- 172.21.15.150:0/328010800 >> v2:172.21.15.142:3300/0 conn(0x7fae0c06e850 msgr2=0x7fae0c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 --2- 172.21.15.150:0/328010800 >> v2:172.21.15.142:3300/0 conn(0x7fae0c06e850 0x7fae0c137820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 -- 172.21.15.150:0/328010800 >> v1:172.21.15.150:6789/0 conn(0x7fae0c071a50 legacy=0x7fae0c130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 monclient: found mon.noname-c 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.742+0000 7fae0affd700 1 -- 172.21.15.150:0/328010800 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c097540 con 0x7fae0c130b80 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0affd700 1 -- 172.21.15.150:0/328010800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c130b80 msgr2=0x7fae0c1352d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0affd700 1 --2- 172.21.15.150:0/328010800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c130b80 0x7fae0c1352d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae0c06dea0 tx=0x7fae000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dea6ceefb842733 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 1 -- 172.21.15.150:0/328010800 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00011840 con 0x7fae0c130b80 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: dump: 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.756 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 5 monclient: authenticate success, global_id 14509 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 1 -- 172.21.15.150:0/328010800 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae000119a0 con 0x7fae0c130b80 2024-09-18T10:11:14.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 4 set_mon_vals no callback set 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 1 -- 172.21.15.150:0/328010800 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae00011d40 con 0x7fae0c130b80 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae0a7fc700 10 monclient: dump: 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.758 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 10 monclient: shutdown 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 -- 172.21.15.150:0/328010800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c130b80 msgr2=0x7fae0c1352d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 --2- 172.21.15.150:0/328010800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c130b80 0x7fae0c1352d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae0c06dea0 tx=0x7fae000091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 -- 172.21.15.150:0/328010800 shutdown_connections 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 --2- 172.21.15.150:0/328010800 >> v2:172.21.15.142:3300/0 conn(0x7fae0c06e850 0x7fae0c137820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 --2- 172.21.15.150:0/328010800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c130b80 0x7fae0c1352d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 -- 172.21.15.150:0/328010800 >> 172.21.15.150:0/328010800 conn(0x7fae0c0a41f0 msgr2=0x7fae0c0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 -- 172.21.15.150:0/328010800 shutdown_connections 2024-09-18T10:11:14.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 1 -- 172.21.15.150:0/328010800 wait complete. 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 10 monclient: build_initial_monmap 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.746+0000 7fae12093700 10 monclient: monmap: 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 Processor -- start 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 -- start start 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient: init 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): picked mon.noname-b con 0x7fae0c06e850 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae0c071a50 0x7fae0c12f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): picked mon.noname-a con 0x7fae0c071a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 0x7fae0c119dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): picked mon.noname-c con 0x7fae0c130b80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c11b240 con 0x7fae0c071a50 2024-09-18T10:11:14.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c11be40 con 0x7fae0c06e850 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c1403c0 con 0x7fae0c130b80 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae12093700 10 monclient(hunting): authenticate will time out at 956.845837s 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0bfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 msgr2=0x7fae0c119dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 0x7fae0c119dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36668/0 (socket says 172.21.15.150:36668) 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 -- 172.21.15.150:0/3091139494 learned_addr learned my addr 172.21.15.150:0/3091139494 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): get_auth_request con 0x7fae0c06e850 auth_method 0 2024-09-18T10:11:14.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae0c071a50 0x7fae0c12f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): get_auth_request con 0x7fae0c071a50 auth_method 0 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:14.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient(hunting): handle_auth_done global_id 14512 payload 824 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 -- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 msgr2=0x7fae0c119dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 0x7fae0c119dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 -- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae0c071a50 msgr2=0x7fae0c12f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae0c071a50 0x7fae0c12f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.750+0000 7fae0b7fe700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00009420 con 0x7fae0c06e850 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae0b7fe700 1 -- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 msgr2=0x7fae0c13f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae0b7fe700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00c5e0 tx=0x7fadfc004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc005d40 con 0x7fae0c06e850 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: dump: 2024-09-18T10:11:14.766 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 5 monclient: authenticate success, global_id 14512 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _renew_subs 2024-09-18T10:11:14.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc0070d0 con 0x7fae0c06e850 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c06edb0 con 0x7fae0c06e850 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc007490 con 0x7fae0c06e850 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae09ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 10 monclient: dump: 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:14.768 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _renew_subs 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c11c490 con 0x7fae0c06e850 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fadfc01b3b0 con 0x7fae0c06e850 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c097260 con 0x7fae0c06e850 2024-09-18T10:11:14.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 0x7fadf8037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4493+0+0 (secure 0 0 0) 0x7fadfc047790 con 0x7fae0c06e850 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae0affd700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 0x7fadf8037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.754+0000 7fae0affd700 10 monclient: get_auth_request con 0x7fadf8035aa0 auth_method 0 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.758+0000 7fae0affd700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 0x7fadf8037f50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae000030c0 tx=0x7fae00016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.758+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fadfc00f680 con 0x7fae0c06e850 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.762+0000 7fadf37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:14.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.762+0000 7fadf37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:14.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.926+0000 7fae12093700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.926+0000 7fae12093700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.926+0000 7fae12093700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:14.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.926+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fae0c063ed0 con 0x7fae0c06e850 2024-09-18T10:11:14.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.930+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v45) v1 ==== 149+0+0 (secure 0 0 0) 0x7fadfc017070 con 0x7fae0c06e850 2024-09-18T10:11:14.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.930+0000 7fadf37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:14.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:14.930+0000 7fadf37fe700 10 monclient: _finish_command 2 = system:0 rule foo already exists 2024-09-18T10:11:15.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.026+0000 7fae12093700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:15.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.026+0000 7fae12093700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:15.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.026+0000 7fae12093700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.026+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"} v 0) v1 -- 0x7fae0c067b10 con 0x7fae0c06e850 2024-09-18T10:11:15.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fadf37fe700 1 -- 172.21.15.150:0/3091139494 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}]=0 rule foo already exists v45) v1 ==== 149+0+0 (secure 0 0 0) 0x7fadfc00e290 con 0x7fae0c06e850 2024-09-18T10:11:15.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fadf37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo", "root": "default", "type": "host"}] 2024-09-18T10:11:15.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fadf37fe700 10 monclient: _finish_command 3 = system:0 rule foo already exists 2024-09-18T10:11:15.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 msgr2=0x7fadf8037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 0x7fadf8037f50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fae000030c0 tx=0x7fae00016000 comp rx=0 tx=0).stop 2024-09-18T10:11:15.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 10 monclient: shutdown 2024-09-18T10:11:15.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:15.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 msgr2=0x7fae0c13f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00c5e0 tx=0x7fadfc004a80 comp rx=0 tx=0).stop 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 shutdown_connections 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadf8035aa0 0x7fadf8037f50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae0c130b80 0x7fae0c119dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae0c071a50 0x7fae0c12f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.030+0000 7fae12093700 1 --2- 172.21.15.150:0/3091139494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae0c06e850 0x7fae0c13f4f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.034+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 >> 172.21.15.150:0/3091139494 conn(0x7fae0c0a41f0 msgr2=0x7fae0c0a1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.034+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 shutdown_connections 2024-09-18T10:11:15.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.034+0000 7fae12093700 1 -- 172.21.15.150:0/3091139494 wait complete. 2024-09-18T10:11:15.038 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo already exists 2024-09-18T10:11:15.046 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-simple bar default host 2024-09-18T10:11:15.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: get_monmap_and_config 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: build_initial_monmap 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: monmap: 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:15.154331+0000 2024-09-18T10:11:15.155 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:15.154331+0000 2024-09-18T10:11:15.156 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:15.156 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.156 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:15.156 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:15.156 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:15.157 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:15.157 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:15.157 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 Processor -- start 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 -- start start 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: init 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d1c0a40d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1d1c0a1be0 0x7f1d1c1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-a con 0x7f1d1c0a1be0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d1c137a10 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:15.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c06ac90 con 0x7f1d1c0a40d0 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c0a44a0 con 0x7f1d1c137a10 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1c0a20b0 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.150+0000 7f1d244df700 10 monclient(hunting): authenticate will time out at 957.247989s 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d22a7c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1d1c137a10 0x7f1d1c139df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41116/0 (socket says 172.21.15.150:41116) 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d22a7c700 1 -- 172.21.15.150:0/2185594081 learned_addr learned my addr 172.21.15.150:0/2185594081 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 1 --2- 172.21.15.150:0/2185594081 >> v2:172.21.15.142:3300/0 conn(0x7f1d1c0a1be0 0x7f1d1c1374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): get_auth_request con 0x7f1d1c0a1be0 auth_method 0 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d22a7c700 1 -- 172.21.15.150:0/2185594081 >> v1:172.21.15.150:6789/0 conn(0x7f1d1c137a10 legacy=0x7f1d1c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21279700 1 -- 172.21.15.150:0/2185594081 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878576566 0 0) 0x7f1d1c0a44a0 con 0x7f1d1c137a10 2024-09-18T10:11:15.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21279700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21279700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21279700 10 monclient(hunting): my global_id is 14515 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21279700 1 -- 172.21.15.150:0/2185594081 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d08003570 con 0x7f1d1c137a10 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d2227b700 1 -- 172.21.15.150:0/2185594081 >> v1:172.21.15.142:6789/0 conn(0x7f1d1c0a40d0 legacy=0x7f1d1c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient(hunting): handle_auth_done global_id 4803 payload 274 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 1 -- 172.21.15.150:0/2185594081 >> v1:172.21.15.150:6789/0 conn(0x7f1d1c137a10 legacy=0x7f1d1c139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 1 -- 172.21.15.150:0/2185594081 >> v1:172.21.15.142:6789/0 conn(0x7f1d1c0a40d0 legacy=0x7f1d1c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient: _finish_hunting 0 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 1 monclient: found mon.noname-a 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.154+0000 7f1d21a7a700 1 -- 172.21.15.150:0/2185594081 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c06b920 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21a7a700 1 -- 172.21.15.150:0/2185594081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 msgr2=0x7f1d1c1374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21a7a700 1 --2- 172.21.15.150:0/2185594081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c1374d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009f60 tx=0x7f1d18009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641d35d9e3018c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 1 -- 172.21.15.150:0/2185594081 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800e070 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: dump: 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.168 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.169 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.169 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.169 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.169 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: _finish_auth 0 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 5 monclient: authenticate success, global_id 4803 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 1 -- 172.21.15.150:0/2185594081 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d1800be40 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 4 set_mon_vals no callback set 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 1 -- 172.21.15.150:0/2185594081 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d18012660 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d21279700 10 monclient: dump: 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.172 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 10 monclient: shutdown 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 -- 172.21.15.150:0/2185594081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 msgr2=0x7f1d1c1374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 --2- 172.21.15.150:0/2185594081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c1374d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009f60 tx=0x7f1d18009780 comp rx=0 tx=0).stop 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 -- 172.21.15.150:0/2185594081 shutdown_connections 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 --2- 172.21.15.150:0/2185594081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c1374d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 -- 172.21.15.150:0/2185594081 >> 172.21.15.150:0/2185594081 conn(0x7f1d1c0716e0 msgr2=0x7f1d1c0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 -- 172.21.15.150:0/2185594081 shutdown_connections 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.158+0000 7f1d244df700 1 -- 172.21.15.150:0/2185594081 wait complete. 2024-09-18T10:11:15.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient: build_initial_monmap 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient: monmap: 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:15.174 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 Processor -- start 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 -- start start 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient: init 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c11ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-a con 0x7f1d1c0a1be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:15.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 0x7f1d1c11b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-c con 0x7f1d1c0a40d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): picked mon.noname-b con 0x7f1d1c137a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c06f0a0 con 0x7f1d1c0a1be0 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c06c8c0 con 0x7f1d1c137a10 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c11f160 con 0x7f1d1c0a40d0 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d244df700 10 monclient(hunting): authenticate will time out at 957.258175s 2024-09-18T10:11:15.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d21a7a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 msgr2=0x7f1d1c11b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d21a7a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 0x7f1d1c11b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36682/0 (socket says 172.21.15.150:36682) 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 -- 172.21.15.150:0/489477284 learned_addr learned my addr 172.21.15.150:0/489477284 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): get_auth_request con 0x7f1d1c137a10 auth_method 0 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c11ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): get_auth_request con 0x7f1d1c0a1be0 auth_method 0 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d2227b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient(hunting): handle_auth_done global_id 14518 payload 824 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 -- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 msgr2=0x7f1d1c11b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 0x7f1d1c11b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 -- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 msgr2=0x7f1d1c11ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c11ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient: _finish_hunting 0 2024-09-18T10:11:15.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 monclient: found mon.noname-b 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.162+0000 7f1d22a7c700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18009a20 con 0x7f1d1c137a10 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d22a7c700 1 -- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 msgr2=0x7f1d1c11b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d22a7c700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400c7c0 tx=0x7f1d14007b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14005d40 con 0x7f1d1c137a10 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: dump: 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.183 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 5 monclient: authenticate success, global_id 14518 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d140070d0 con 0x7f1d1c137a10 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _renew_subs 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c11c430 con 0x7f1d1c137a10 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14005ea0 con 0x7f1d1c137a10 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 10 monclient: dump: 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.185 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d20a78700 4 set_mon_vals no callback set 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _renew_subs 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c11d000 con 0x7f1d1c137a10 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c11e3c0 con 0x7f1d1c137a10 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1d1401b3b0 con 0x7f1d1c137a10 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 0x7f1d00037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.166+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4493+0+0 (secure 0 0 0) 0x7f1d14047630 con 0x7f1d1c137a10 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.170+0000 7f1d2227b700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 0x7f1d00037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.170+0000 7f1d2227b700 10 monclient: get_auth_request con 0x7f1d00035a70 auth_method 0 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.170+0000 7f1d2227b700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 0x7f1d00037f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d10001c10 tx=0x7f1d10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.174+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1d140196e0 con 0x7f1d1c137a10 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.174+0000 7f1d0e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.174+0000 7f1d0e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:15.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.346+0000 7f1d244df700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.346+0000 7f1d244df700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.346+0000 7f1d244df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.346+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f1d1c06a9b0 con 0x7f1d1c137a10 2024-09-18T10:11:15.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.530+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 v46) v1 ==== 126+0+0 (secure 0 0 0) 0x7f1d140170c0 con 0x7f1d1c137a10 2024-09-18T10:11:15.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.530+0000 7f1d0e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.530+0000 7f1d0e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:15.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d244df700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d244df700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d244df700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"} v 0) v1 -- 0x7f1d1c134960 con 0x7f1d1c137a10 2024-09-18T10:11:15.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d0e7fc700 1 -- 172.21.15.150:0/489477284 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}]=0 rule bar already exists v46) v1 ==== 149+0+0 (secure 0 0 0) 0x7f1d1400f260 con 0x7f1d1c137a10 2024-09-18T10:11:15.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d0e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "bar", "root": "default", "type": "host"}] 2024-09-18T10:11:15.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.638+0000 7f1d0e7fc700 10 monclient: _finish_command 3 = system:0 rule bar already exists 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 msgr2=0x7f1d00037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 0x7f1d00037f20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1d10001c10 tx=0x7f1d10009040 comp rx=0 tx=0).stop 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 10 monclient: shutdown 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 msgr2=0x7f1d1c11b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400c7c0 tx=0x7f1d14007b90 comp rx=0 tx=0).stop 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 shutdown_connections 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d00035a70 0x7f1d00037f20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d1c137a10 0x7f1d1c11b5a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d1c0a40d0 0x7f1d1c11b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 --2- 172.21.15.150:0/489477284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d1c0a1be0 0x7f1d1c11ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 >> 172.21.15.150:0/489477284 conn(0x7f1d1c0716e0 msgr2=0x7f1d1c09d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 shutdown_connections 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.642+0000 7f1d244df700 1 -- 172.21.15.150:0/489477284 wait complete. 2024-09-18T10:11:15.647 INFO:tasks.workunit.client.0.smithi150.stderr:rule bar already exists 2024-09-18T10:11:15.656 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm-device-class all 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.750+0000 7f1d6c6af700 10 monclient: get_monmap_and_config 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.750+0000 7f1d6c6af700 10 monclient: build_initial_monmap 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.750+0000 7f1d6c6af700 10 monclient: monmap: 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:15.755077+0000 2024-09-18T10:11:15.755 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:15.755077+0000 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:15.756 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.750+0000 7f1d6c6af700 1 Processor -- start 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 -- start start 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient: init 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:11:15.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d6409e6d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1d640a09a0 0x7f1d64137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-a con 0x7f1d640a09a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d640a0da0 0x7f1d64139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-c con 0x7f1d640a0da0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6409e220 con 0x7f1d6409e6d0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6409ee30 con 0x7f1d640a09a0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6409eaa0 con 0x7f1d640a0da0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6c6af700 10 monclient(hunting): authenticate will time out at 957.848789s 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d640a0da0 0x7f1d64139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d640a0da0 0x7f1d64139f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36696/0 (socket says 172.21.15.150:36696) 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 -- 172.21.15.150:0/3245580111 learned_addr learned my addr 172.21.15.150:0/3245580111 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): get_auth_request con 0x7f1d640a0da0 auth_method 0 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 1 --2- 172.21.15.150:0/3245580111 >> v2:172.21.15.142:3300/0 conn(0x7f1d640a09a0 0x7f1d64137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): get_auth_request con 0x7f1d640a09a0 auth_method 0 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d69c4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_done global_id 14521 payload 274 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 -- 172.21.15.150:0/3245580111 >> v2:172.21.15.142:3300/0 conn(0x7f1d640a09a0 msgr2=0x7f1d64137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 --2- 172.21.15.150:0/3245580111 >> v2:172.21.15.142:3300/0 conn(0x7f1d640a09a0 0x7f1d64137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 -- 172.21.15.150:0/3245580111 >> v1:172.21.15.142:6789/0 conn(0x7f1d6409e6d0 legacy=0x7f1d640a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient: _finish_hunting 0 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 monclient: found mon.noname-c 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 -- 172.21.15.150:0/3245580111 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6409d930 con 0x7f1d640a0da0 2024-09-18T10:11:15.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 -- 172.21.15.150:0/3245580111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d640a0da0 msgr2=0x7f1d64139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.754+0000 7f1d6ac4c700 1 --2- 172.21.15.150:0/3245580111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d640a0da0 0x7f1d64139f70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d640a6970 tx=0x7f1d5c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96997c0312d5c7e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 1 -- 172.21.15.150:0/3245580111 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00e070 con 0x7f1d640a0da0 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: dump: 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.764 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: _finish_auth 0 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 5 monclient: authenticate success, global_id 14521 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 1 -- 172.21.15.150:0/3245580111 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c00bc30 con 0x7f1d640a0da0 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 4 set_mon_vals no callback set 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 1 -- 172.21.15.150:0/3245580111 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c0092e0 con 0x7f1d640a0da0 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:15.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69449700 10 monclient: dump: 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.766 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: shutdown 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- 172.21.15.150:0/3245580111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d640a0da0 msgr2=0x7f1d64139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/3245580111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d640a0da0 0x7f1d64139f70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1d640a6970 tx=0x7f1d5c009840 comp rx=0 tx=0).stop 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- 172.21.15.150:0/3245580111 shutdown_connections 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/3245580111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d640a0da0 0x7f1d64139f70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/3245580111 >> v2:172.21.15.142:3300/0 conn(0x7f1d640a09a0 0x7f1d64137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- 172.21.15.150:0/3245580111 >> 172.21.15.150:0/3245580111 conn(0x7f1d640a58a0 msgr2=0x7f1d640a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:15.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- 172.21.15.150:0/3245580111 shutdown_connections 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- 172.21.15.150:0/3245580111 wait complete. 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: build_initial_monmap 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: monmap: 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:15.772 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 Processor -- start 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- start start 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: init 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-b con 0x7f1d6409e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 0x7f1d64131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-c con 0x7f1d640a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d640a0da0 0x7f1d6406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): picked mon.noname-a con 0x7f1d640a0da0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6406f880 con 0x7f1d640a0da0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6412bea0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64142280 con 0x7f1d640a09a0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6c6af700 10 monclient(hunting): authenticate will time out at 957.854395s 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69c4a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 msgr2=0x7f1d64131cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d69c4a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 0x7f1d64131cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36704/0 (socket says 172.21.15.150:36704) 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 -- 172.21.15.150:0/890524620 learned_addr learned my addr 172.21.15.150:0/890524620 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): get_auth_request con 0x7f1d6409e6d0 auth_method 0 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d640a0da0 0x7f1d6406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): get_auth_request con 0x7f1d640a0da0 auth_method 0 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6ac4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient(hunting): handle_auth_done global_id 14524 payload 824 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 -- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 msgr2=0x7f1d64131cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 0x7f1d64131cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 -- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d640a0da0 msgr2=0x7f1d6406deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d640a0da0 0x7f1d6406deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient: _finish_hunting 0 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 monclient: found mon.noname-b 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.758+0000 7f1d6a44b700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c009ae0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6a44b700 1 -- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 msgr2=0x7f1d641413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6a44b700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000c5d0 tx=0x7f1d60007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600117a0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: dump: 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 5 monclient: authenticate success, global_id 14524 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d60011900 con 0x7f1d6409e6d0 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _renew_subs 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6412c1e0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60011cc0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:15.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d68c48700 4 set_mon_vals no callback set 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 10 monclient: dump: 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _renew_subs 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6412ab10 con 0x7f1d6409e6d0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1d60012630 con 0x7f1d6409e6d0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6409dbf0 con 0x7f1d6409e6d0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 0x7f1d48037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1d60048050 con 0x7f1d6409e6d0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.762+0000 7f1d69c4a700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 0x7f1d48037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.766+0000 7f1d69c4a700 10 monclient: get_auth_request con 0x7f1d48035ac0 auth_method 0 2024-09-18T10:11:15.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.766+0000 7f1d69c4a700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 0x7f1d48037f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1d640a2180 tx=0x7f1d540091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:15.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.770+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1d60078670 con 0x7f1d6409e6d0 2024-09-18T10:11:15.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.770+0000 7f1d5a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:15.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.770+0000 7f1d5a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:15.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.938+0000 7f1d6c6af700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:15.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.938+0000 7f1d6c6af700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:15.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.938+0000 7f1d6c6af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:15.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:15.938+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f1d64063ed0 con 0x7f1d6409e6d0 2024-09-18T10:11:16.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.534+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 done removing class of osd(s): 0,1,2,3,4,5 v47) v1 ==== 131+0+0 (secure 0 0 0) 0x7f1d600108c0 con 0x7f1d6409e6d0 2024-09-18T10:11:16.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.534+0000 7f1d5a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:16.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.534+0000 7f1d5a7fc700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0,1,2,3,4,5 2024-09-18T10:11:16.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.642+0000 7f1d6c6af700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:16.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.642+0000 7f1d6c6af700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:16.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.642+0000 7f1d6c6af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:16.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:16.642+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["all"]} v 0) v1 -- 0x7f1d640008d0 con 0x7f1d6409e6d0 2024-09-18T10:11:17.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.546+0000 7f1d5a7fc700 1 -- 172.21.15.150:0/890524620 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["all"]}]=0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): v48) v1 ==== 282+0+0 (secure 0 0 0) 0x7f1d600102e0 con 0x7f1d6409e6d0 2024-09-18T10:11:17.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.546+0000 7f1d5a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["all"]}] 2024-09-18T10:11:17.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.546+0000 7f1d5a7fc700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-18T10:11:17.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 msgr2=0x7f1d48037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 0x7f1d48037f70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1d640a2180 tx=0x7f1d540091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 10 monclient: shutdown 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 msgr2=0x7f1d641413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000c5d0 tx=0x7f1d60007f60 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 shutdown_connections 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d48035ac0 0x7f1d48037f70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d640a0da0 0x7f1d6406deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d640a09a0 0x7f1d64131cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 --2- 172.21.15.150:0/890524620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6409e6d0 0x7f1d641413f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 >> 172.21.15.150:0/890524620 conn(0x7f1d640a58a0 msgr2=0x7f1d64134e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:17.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.550+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 shutdown_connections 2024-09-18T10:11:17.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.554+0000 7f1d6c6af700 1 -- 172.21.15.150:0/890524620 wait complete. 2024-09-18T10:11:17.557 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 belongs to no class, osd.1 belongs to no class, osd.2 belongs to no class, osd.3 belongs to no class, osd.4 belongs to no class, osd.5 belongs to no class, done removing class of osd(s): 2024-09-18T10:11:17.571 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush set-device-class ssd osd.0 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: get_monmap_and_config 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: build_initial_monmap 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: monmap: 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:17.678 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:17.678306+0000 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:17.678306+0000 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:17.679 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 1 Processor -- start 2024-09-18T10:11:17.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 1 -- start start 2024-09-18T10:11:17.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: init 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f721009cf90 0x7f72100a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient(hunting): picked mon.noname-c con 0x7f721009cf90 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.674+0000 7f721795e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f721009e480 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:17.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72100990a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721009e060 con 0x7f721009e480 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721009d750 con 0x7f72100990a0 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100a66c0 con 0x7f721009cf90 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 10 monclient(hunting): authenticate will time out at 959.772070s 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f721009cf90 0x7f72100a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:17.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7215efb700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f72100990a0 0x7f7210139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41118/0 (socket says 172.21.15.150:41118) 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7215efb700 1 -- 172.21.15.150:0/3831617095 learned_addr learned my addr 172.21.15.150:0/3831617095 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): get_auth_request con 0x7f721009cf90 auth_method 0 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient(hunting): handle_auth_done global_id 14527 payload 274 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 -- 172.21.15.150:0/3831617095 >> v1:172.21.15.150:6789/0 conn(0x7f72100990a0 legacy=0x7f7210139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 -- 172.21.15.150:0/3831617095 >> v1:172.21.15.142:6789/0 conn(0x7f721009e480 legacy=0x7f72100a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient: _finish_hunting 0 2024-09-18T10:11:17.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 monclient: found mon.noname-c 2024-09-18T10:11:17.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:17.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 -- 172.21.15.150:0/3831617095 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721009d360 con 0x7f721009cf90 2024-09-18T10:11:17.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 -- 172.21.15.150:0/3831617095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009cf90 msgr2=0x7f72100a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:17.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f72156fa700 1 --2- 172.21.15.150:0/3831617095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009cf90 0x7f72100a0290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72000098e0 tx=0x7f72000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6866cd5248ca0309 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:17.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 1 -- 172.21.15.150:0/3831617095 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720000ba70 con 0x7f721009cf90 2024-09-18T10:11:17.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:17.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: dump: 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.691 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: _finish_auth 0 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 1 -- 172.21.15.150:0/3831617095 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720000bbd0 con 0x7f721009cf90 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 5 monclient: authenticate success, global_id 14527 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f721795e700 4 set_mon_vals no callback set 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 1 -- 172.21.15.150:0/3831617095 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200011420 con 0x7f721009cf90 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.678+0000 7f7207fff700 10 monclient: dump: 2024-09-18T10:11:17.693 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:17.694 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:17.696 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.696 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.696 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: shutdown 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:17.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- 172.21.15.150:0/3831617095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009cf90 msgr2=0x7f72100a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 --2- 172.21.15.150:0/3831617095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009cf90 0x7f72100a0290 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72000098e0 tx=0x7f72000091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- 172.21.15.150:0/3831617095 shutdown_connections 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 --2- 172.21.15.150:0/3831617095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009cf90 0x7f72100a0290 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- 172.21.15.150:0/3831617095 >> 172.21.15.150:0/3831617095 conn(0x7f72100a58a0 msgr2=0x7f72100a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- 172.21.15.150:0/3831617095 shutdown_connections 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- 172.21.15.150:0/3831617095 wait complete. 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: build_initial_monmap 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: monmap: 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:17.698 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 Processor -- start 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- start start 2024-09-18T10:11:17.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: init 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 0x7f721008fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): picked mon.noname-a con 0x7f72100990a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 0x7f7210097620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): picked mon.noname-c con 0x7f721009cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): picked mon.noname-b con 0x7f721009e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210097b60 con 0x7f72100990a0 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100a4110 con 0x7f721009e480 2024-09-18T10:11:17.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7210091110 con 0x7f721009cf90 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f721795e700 10 monclient(hunting): authenticate will time out at 959.779813s 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f7215efb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.682+0000 7f7214ef9700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 msgr2=0x7f7210097620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36716/0 (socket says 172.21.15.150:36716) 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 -- 172.21.15.150:0/1134318846 learned_addr learned my addr 172.21.15.150:0/1134318846 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): get_auth_request con 0x7f721009e480 auth_method 0 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7214ef9700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 0x7f7210097620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:17.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 0x7f721008fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): get_auth_request con 0x7f72100990a0 auth_method 0 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient(hunting): handle_auth_done global_id 14530 payload 824 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 -- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 msgr2=0x7f7210097620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 0x7f7210097620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 -- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 msgr2=0x7f721008fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 0x7f721008fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient: _finish_hunting 0 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 monclient: found mon.noname-b 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7200009420 con 0x7f721009e480 2024-09-18T10:11:17.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72156fa700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 0x7f721008fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:17.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 -- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 msgr2=0x7f721008cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:17.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f7215efb700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f720800cc20 tx=0x7f72080053d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:17.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720800aae0 con 0x7f721009e480 2024-09-18T10:11:17.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:17.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: dump: 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.708 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: _finish_auth 0 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720800ac40 con 0x7f721009e480 2024-09-18T10:11:17.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 5 monclient: authenticate success, global_id 14530 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 10 monclient: _renew_subs 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7210090450 con 0x7f721009e480 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72080070f0 con 0x7f721009e480 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72077fe700 4 set_mon_vals no callback set 2024-09-18T10:11:17.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f72057fa700 10 monclient: dump: 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:17.711 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:17.712 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:17.712 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:17.712 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:17.712 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:17.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 10 monclient: _renew_subs 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.686+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721008f4f0 con 0x7f721009e480 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f72080074e0 con 0x7f721009e480 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f72057fa700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 0x7f71f4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4248+0+0 (secure 0 0 0) 0x7f7208048600 con 0x7f721009e480 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f721795e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:17.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f721795e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f721795e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72100a0ee0 con 0x7f721009e480 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f72156fa700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 0x7f71f4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.690+0000 7f72156fa700 10 monclient: get_auth_request con 0x7f71f4035a70 auth_method 0 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.694+0000 7f72156fa700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 0x7f71f4037f20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f72000030c0 tx=0x7f7200016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:17.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.698+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7208007790 con 0x7f721009e480 2024-09-18T10:11:17.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.698+0000 7f72057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:17.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.698+0000 7f72057fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:17.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.866+0000 7f721795e700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:17.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.870+0000 7f721795e700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:17.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.870+0000 7f721795e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:17.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:17.870+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f721008e980 con 0x7f721009e480 2024-09-18T10:11:18.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.486+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'ssd' v49) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7208010af0 con 0x7f721009e480 2024-09-18T10:11:18.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.486+0000 7f72057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:18.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.486+0000 7f72057fa700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'ssd' 2024-09-18T10:11:18.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.594+0000 7f721795e700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:18.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.594+0000 7f721795e700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:18.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.594+0000 7f721795e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:18.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:18.594+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]} v 0) v1 -- 0x7f7210063ed0 con 0x7f721009e480 2024-09-18T10:11:19.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.562+0000 7f72057fa700 1 -- 172.21.15.150:0/1134318846 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}]=0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' v50) v1 ==== 237+0+0 (secure 0 0 0) 0x7f720801a570 con 0x7f721009e480 2024-09-18T10:11:19.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.562+0000 7f72057fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["osd.0"]}] 2024-09-18T10:11:19.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.562+0000 7f72057fa700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-18T10:11:19.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 msgr2=0x7f71f4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 0x7f71f4037f20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f72000030c0 tx=0x7f7200016000 comp rx=0 tx=0).stop 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 10 monclient: shutdown 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 msgr2=0x7f721008cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f720800cc20 tx=0x7f72080053d0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 shutdown_connections 2024-09-18T10:11:19.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71f4035a70 0x7f71f4037f20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721009e480 0x7f721008cde0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721009cf90 0x7f7210097620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 --2- 172.21.15.150:0/1134318846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f72100990a0 0x7f721008fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 >> 172.21.15.150:0/1134318846 conn(0x7f72100a58a0 msgr2=0x7f72100a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 shutdown_connections 2024-09-18T10:11:19.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.566+0000 7f721795e700 1 -- 172.21.15.150:0/1134318846 wait complete. 2024-09-18T10:11:19.574 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. set osd(s) to class 'ssd' 2024-09-18T10:11:19.585 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush set-device-class hdd osd.1 2024-09-18T10:11:19.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: get_monmap_and_config 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: build_initial_monmap 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: monmap: 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:19.692669+0000 2024-09-18T10:11:19.693 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:19.692669+0000 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:19.694 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:19.695 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:19.695 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 Processor -- start 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 -- start start 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: init 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdd40a1810 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:19.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcdd40a22f0 0x7fcdd406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-a con 0x7fcdd40a22f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdd4135410 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd406c5f0 con 0x7fcdd4135410 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd40a33e0 con 0x7fcdd40a1810 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd406d750 con 0x7fcdd40a22f0 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdda52f700 10 monclient(hunting): authenticate will time out at 961.786226s 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd3fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcdd40a1810 0x7fcdd40a1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38234/0 (socket says 172.21.15.150:38234) 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd3fff700 1 -- 172.21.15.150:0/2422535790 learned_addr learned my addr 172.21.15.150:0/2422535790 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/2422535790 >> v2:172.21.15.142:3300/0 conn(0x7fcdd40a22f0 0x7fcdd406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request con 0x7fcdd40a22f0 auth_method 0 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.690+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd3fff700 1 -- 172.21.15.150:0/2422535790 >> v1:172.21.15.150:6789/0 conn(0x7fcdd40a1810 legacy=0x7fcdd40a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998547615 0 0) 0x7fcdd40a33e0 con 0x7fcdd40a1810 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): my global_id is 14533 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdbc003570 con 0x7fcdd40a1810 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 >> v1:172.21.15.142:6789/0 conn(0x7fcdd4135410 legacy=0x7fcdd41377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2112030383 0 0) 0x7fcdbc003570 con 0x7fcdd40a1810 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 >> v2:172.21.15.142:3300/0 conn(0x7fcdd40a22f0 msgr2=0x7fcdd406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 --2- 172.21.15.150:0/2422535790 >> v2:172.21.15.142:3300/0 conn(0x7fcdd40a22f0 0x7fcdd406e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 >> v1:172.21.15.142:6789/0 conn(0x7fcdd4135410 legacy=0x7fcdd41377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:19.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd406da30 con 0x7fcdd40a1810 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fcdc8001860 con 0x7fcdd40a1810 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdda52f700 5 monclient: authenticate success, global_id 14533 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdda52f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: dump: 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:19.721 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 >> v1:172.21.15.150:6789/0 conn(0x7fcdd40a1810 legacy=0x7fcdd40a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdbc003b80 0x7fcdbc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): picked mon.c con 0x7fcdbc003b80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): picked mon.a con 0x7fcdbc004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 0x7fcdbc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): picked mon.b con 0x7fcdbc0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd406d750 con 0x7fcdbc004810 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc00c320 con 0x7fcdbc0053c0 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc00c4e0 con 0x7fcdbc003b80 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:19.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdda52f700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd3fff700 1 -- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdbc003b80 msgr2=0x7fcdbc003f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd3fff700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdbc003b80 0x7fcdbc003f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 0x7fcdbc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): get_auth_request con 0x7fcdbc0053c0 auth_method 0 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request con 0x7fcdbc004810 auth_method 0 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient(hunting): handle_auth_done global_id 14533 payload 293 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdbc003b80 msgr2=0x7fcdbc003f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:19.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdbc003b80 0x7fcdbc003f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc004810 msgr2=0x7fcdbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 monclient: found mon.b 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc003570 con 0x7fcdbc0053c0 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 10 monclient: _finish_auth 0 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.694+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 reap_dead start 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd8acc700 1 -- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 msgr2=0x7fcdbc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:19.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd8acc700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 0x7fcdbc00ba60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c190 tx=0x7fcdcc001920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a86f893bc3a00d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc001c50 con 0x7fcdbc0053c0 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: dump: 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:19.727 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc005b90 con 0x7fcdbc0053c0 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 1 -- 172.21.15.150:0/2422535790 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00cac0 con 0x7fcdbc0053c0 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:19.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:19.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdd2ffd700 10 monclient: dump: 2024-09-18T10:11:19.729 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:19.729 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.729 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.729 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.730 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:19.730 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.730 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:19.730 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:19.730 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 4 set_mon_vals no callback set 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 10 monclient: shutdown 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:19.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 msgr2=0x7fcdbc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 0x7fcdbc00ba60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00c190 tx=0x7fcdcc001920 comp rx=0 tx=0).stop 2024-09-18T10:11:19.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- 172.21.15.150:0/2422535790 shutdown_connections 2024-09-18T10:11:19.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 --2- 172.21.15.150:0/2422535790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdbc0053c0 0x7fcdbc00ba60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- 172.21.15.150:0/2422535790 >> 172.21.15.150:0/2422535790 conn(0x7fcdd40716e0 msgr2=0x7fcdd40a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:19.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- 172.21.15.150:0/2422535790 shutdown_connections 2024-09-18T10:11:19.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- 172.21.15.150:0/2422535790 wait complete. 2024-09-18T10:11:19.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 10 monclient: build_initial_monmap 2024-09-18T10:11:19.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 10 monclient: monmap: 2024-09-18T10:11:19.733 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:19.733 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:19.734 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 Processor -- start 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 1 -- start start 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.698+0000 7fcdda52f700 10 monclient: init 2024-09-18T10:11:19.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc0053c0 0x7fcdd4134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-a con 0x7fcdbc0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-b con 0x7fcdd4134940 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 0x7fcdd406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): picked mon.noname-c con 0x7fcdd4069f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4122b80 con 0x7fcdbc0053c0 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd406ef80 con 0x7fcdd4134940 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd409db30 con 0x7fcdd4069f90 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdda52f700 10 monclient(hunting): authenticate will time out at 961.796732s 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd8acc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 msgr2=0x7fcdd406a3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:19.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd8acc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 0x7fcdd406a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:19.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57692/0 (socket says 172.21.15.150:57692) 2024-09-18T10:11:19.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 -- 172.21.15.150:0/3336061832 learned_addr learned my addr 172.21.15.150:0/3336061832 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:19.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request con 0x7fcdd4134940 auth_method 0 2024-09-18T10:11:19.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc0053c0 0x7fcdd4134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): get_auth_request con 0x7fcdbc0053c0 auth_method 0 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:19.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:19.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient(hunting): handle_auth_done global_id 14536 payload 824 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 -- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 msgr2=0x7fcdd406a3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 0x7fcdd406a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 -- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc0053c0 msgr2=0x7fcdd4134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc0053c0 0x7fcdd4134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:19.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 monclient: found mon.noname-b 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd406b970 con 0x7fcdd4134940 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.702+0000 7fcdd37fe700 1 -- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 msgr2=0x7fcdd4069a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdd37fe700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4011da0 tx=0x7fcdc40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400fcd0 con 0x7fcdd4134940 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:19.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: dump: 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:19.744 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: _finish_auth 0 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 5 monclient: authenticate success, global_id 14536 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400fe30 con 0x7fcdd4134940 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _renew_subs 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd406b460 con 0x7fcdd4134940 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4014690 con 0x7fcdd4134940 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 10 monclient: dump: 2024-09-18T10:11:19.745 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _renew_subs 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdd27fc700 4 set_mon_vals no callback set 2024-09-18T10:11:19.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd4135cb0 con 0x7fcdd4134940 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcdc4014a00 con 0x7fcdd4134940 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 0x7fcdc0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4472+0+0 (secure 0 0 0) 0x7fcdc4018070 con 0x7fcdd4134940 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd409c750 con 0x7fcdd4134940 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdd3fff700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 0x7fcdc0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.706+0000 7fcdd3fff700 10 monclient: get_auth_request con 0x7fcdc0035a70 auth_method 0 2024-09-18T10:11:19.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.710+0000 7fcdd3fff700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 0x7fcdc0037f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80014a0 tx=0x7fcdc80018e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:19.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.718+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcdc4014cb0 con 0x7fcdd4134940 2024-09-18T10:11:19.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.718+0000 7fcdbbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:19.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.718+0000 7fcdbbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:19.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.882+0000 7fcdda52f700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:19.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.882+0000 7fcdda52f700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:19.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.882+0000 7fcdda52f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:19.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:19.882+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7fcdd40a44a0 con 0x7fcdd4134940 2024-09-18T10:11:20.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.570+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 set osd(s) 1 to class 'hdd' v51) v1 ==== 135+0+0 (secure 0 0 0) 0x7fcdc4025e00 con 0x7fcdd4134940 2024-09-18T10:11:20.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.570+0000 7fcdbbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:20.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.570+0000 7fcdbbfff700 10 monclient: _finish_command 2 = system:0 set osd(s) 1 to class 'hdd' 2024-09-18T10:11:20.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.678+0000 7fcdda52f700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.678+0000 7fcdda52f700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.678+0000 7fcdda52f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:20.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:20.678+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]} v 0) v1 -- 0x7fcdd4063ed0 con 0x7fcdd4134940 2024-09-18T10:11:21.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.578+0000 7fcdbbfff700 1 -- 172.21.15.150:0/3336061832 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}]=0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' v52) v1 ==== 237+0+0 (secure 0 0 0) 0x7fcdc401ab20 con 0x7fcdd4134940 2024-09-18T10:11:21.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.578+0000 7fcdbbfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["osd.1"]}] 2024-09-18T10:11:21.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.578+0000 7fcdbbfff700 10 monclient: _finish_command 3 = system:0 osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-18T10:11:21.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 msgr2=0x7fcdc0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:21.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 0x7fcdc0037f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80014a0 tx=0x7fcdc80018e0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 10 monclient: shutdown 2024-09-18T10:11:21.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:21.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 msgr2=0x7fcdd4069a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4011da0 tx=0x7fcdc40091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 shutdown_connections 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdc0035a70 0x7fcdc0037f20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdd4069f90 0x7fcdd406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdd4134940 0x7fcdd4069a50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 --2- 172.21.15.150:0/3336061832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdbc0053c0 0x7fcdd4134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 >> 172.21.15.150:0/3336061832 conn(0x7fcdd40716e0 msgr2=0x7fcdd409ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 shutdown_connections 2024-09-18T10:11:21.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.582+0000 7fcdda52f700 1 -- 172.21.15.150:0/3336061832 wait complete. 2024-09-18T10:11:21.588 INFO:tasks.workunit.client.0.smithi150.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. set osd(s) to class 'hdd' 2024-09-18T10:11:21.599 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-replicated foo-ssd default host ssd 2024-09-18T10:11:21.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: get_monmap_and_config 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: build_initial_monmap 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: monmap: 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:21.708658+0000 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:21.708658+0000 2024-09-18T10:11:21.709 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:21.710 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:21.711 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 Processor -- start 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 -- start start 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: init 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9224071240 0x7f9224096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): picked mon.noname-a con 0x7f9224071240 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f922409e1e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240994f0 0x7f92241364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): picked mon.noname-c con 0x7f92240994f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9224098cf0 con 0x7f922409e1e0 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922409e640 con 0x7f9224071240 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922409df80 con 0x7f92240994f0 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9229197700 10 monclient(hunting): authenticate will time out at 963.803538s 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240994f0 0x7f92241364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240994f0 0x7f92241364d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57698/0 (socket says 172.21.15.150:57698) 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f921bfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f922409e1e0 0x7f9224096fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38250/0 (socket says 172.21.15.150:38250) 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 1 -- 172.21.15.150:0/2626070046 learned_addr learned my addr 172.21.15.150:0/2626070046 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 10 monclient(hunting): get_auth_request con 0x7f92240994f0 auth_method 0 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f922359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9222d9d700 1 --2- 172.21.15.150:0/2626070046 >> v2:172.21.15.142:3300/0 conn(0x7f9224071240 0x7f9224096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9222d9d700 10 monclient(hunting): get_auth_request con 0x7f9224071240 auth_method 0 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9222d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9222d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.706+0000 7f9222d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient(hunting): handle_auth_done global_id 14539 payload 274 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 -- 172.21.15.150:0/2626070046 >> v2:172.21.15.142:3300/0 conn(0x7f9224071240 msgr2=0x7f9224096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 --2- 172.21.15.150:0/2626070046 >> v2:172.21.15.142:3300/0 conn(0x7f9224071240 0x7f9224096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 -- 172.21.15.150:0/2626070046 >> v1:172.21.15.150:6789/0 conn(0x7f922409e1e0 legacy=0x7f9224096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient: _finish_hunting 0 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 monclient: found mon.noname-c 2024-09-18T10:11:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 -- 172.21.15.150:0/2626070046 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9224098a10 con 0x7f92240994f0 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 -- 172.21.15.150:0/2626070046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240994f0 msgr2=0x7f92241364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922359e700 1 --2- 172.21.15.150:0/2626070046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240994f0 0x7f92241364d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f922406a8d0 tx=0x7f921c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b36cba0380a054bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 1 -- 172.21.15.150:0/2626070046 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c00e070 con 0x7f92240994f0 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: dump: 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.720 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: _finish_auth 0 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9229197700 5 monclient: authenticate success, global_id 14539 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9229197700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 1 -- 172.21.15.150:0/2626070046 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c00bc30 con 0x7f92240994f0 2024-09-18T10:11:21.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9229197700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f9229197700 4 set_mon_vals no callback set 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 1 -- 172.21.15.150:0/2626070046 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0092e0 con 0x7f92240994f0 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.710+0000 7f922259c700 10 monclient: dump: 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:21.722 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: shutdown 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- 172.21.15.150:0/2626070046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240994f0 msgr2=0x7f92241364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- 172.21.15.150:0/2626070046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240994f0 0x7f92241364d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f922406a8d0 tx=0x7f921c009840 comp rx=0 tx=0).stop 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- 172.21.15.150:0/2626070046 shutdown_connections 2024-09-18T10:11:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- 172.21.15.150:0/2626070046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240994f0 0x7f92241364d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- 172.21.15.150:0/2626070046 >> v2:172.21.15.142:3300/0 conn(0x7f9224071240 0x7f9224096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- 172.21.15.150:0/2626070046 >> 172.21.15.150:0/2626070046 conn(0x7f92240a1cc0 msgr2=0x7f92240a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- 172.21.15.150:0/2626070046 shutdown_connections 2024-09-18T10:11:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- 172.21.15.150:0/2626070046 wait complete. 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: build_initial_monmap 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: monmap: 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 Processor -- start 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- start start 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: init 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): picked mon.noname-b con 0x7f9224071240 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:21.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 0x7f922413d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): picked mon.noname-c con 0x7f92240994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f922409e1e0 0x7f9224141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): picked mon.noname-a con 0x7f922409e1e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9224122cd0 con 0x7f922409e1e0 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92241228f0 con 0x7f9224071240 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9224120be0 con 0x7f92240994f0 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9229197700 10 monclient(hunting): authenticate will time out at 963.810650s 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f921bfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 msgr2=0x7f922413d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57704/0 (socket says 172.21.15.150:57704) 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f921bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 0x7f922413d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 -- 172.21.15.150:0/3513457665 learned_addr learned my addr 172.21.15.150:0/3513457665 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): get_auth_request con 0x7f9224071240 auth_method 0 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f922409e1e0 0x7f9224141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): get_auth_request con 0x7f922409e1e0 auth_method 0 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f922359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient(hunting): handle_auth_done global_id 14542 payload 824 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 -- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 msgr2=0x7f922413d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 0x7f922413d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 -- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f922409e1e0 msgr2=0x7f9224141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f922409e1e0 0x7f9224141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 monclient: found mon.noname-b 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.714+0000 7f9222d9d700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009ae0 con 0x7f9224071240 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9222d9d700 1 -- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 msgr2=0x7f922413dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9222d9d700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f921400ecb0 tx=0x7f92140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921400cb30 con 0x7f9224071240 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: dump: 2024-09-18T10:11:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: _finish_auth 0 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921400cc90 con 0x7f9224071240 2024-09-18T10:11:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 5 monclient: authenticate success, global_id 14542 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _renew_subs 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9224120460 con 0x7f9224071240 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214013470 con 0x7f9224071240 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f921affd700 10 monclient: dump: 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f922159a700 4 set_mon_vals no callback set 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _renew_subs 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922413e8d0 con 0x7f9224071240 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:21.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.718+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9224063ed0 con 0x7f9224071240 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9214013820 con 0x7f9224071240 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921affd700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 0x7f9204037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4704+0+0 (secure 0 0 0) 0x7f92140486d0 con 0x7f9224071240 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921bfff700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 0x7f9204037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921bfff700 10 monclient: get_auth_request con 0x7f9204035a70 auth_method 0 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.722+0000 7f921bfff700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 0x7f9204037f20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f92240654c0 tx=0x7f920c007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.726+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f92140234d0 con 0x7f9224071240 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.726+0000 7f921affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:21.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.726+0000 7f921affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:21.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.898+0000 7f9229197700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.898+0000 7f9229197700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.898+0000 7f9229197700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:21.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:21.898+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f9224071930 con 0x7f9224071240 2024-09-18T10:11:22.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.586+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 v53) v1 ==== 150+0+0 (secure 0 0 0) 0x7f9214021070 con 0x7f9224071240 2024-09-18T10:11:22.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.586+0000 7f921affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:22.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.586+0000 7f921affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:22.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.690+0000 7f9229197700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:22.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.690+0000 7f9229197700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:22.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.690+0000 7f9229197700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.690+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"} v 0) v1 -- 0x7f922413dab0 con 0x7f9224071240 2024-09-18T10:11:22.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f921affd700 1 -- 172.21.15.150:0/3513457665 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}]=0 rule foo-ssd already exists v53) v1 ==== 177+0+0 (secure 0 0 0) 0x7f92140232f0 con 0x7f9224071240 2024-09-18T10:11:22.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f921affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-ssd", "root": "default", "type": "host", "class": "ssd"}] 2024-09-18T10:11:22.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f921affd700 10 monclient: _finish_command 3 = system:0 rule foo-ssd already exists 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 msgr2=0x7f9204037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 0x7f9204037f20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f92240654c0 tx=0x7f920c007b50 comp rx=0 tx=0).stop 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 10 monclient: shutdown 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 msgr2=0x7f922413dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:22.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f921400ecb0 tx=0x7f92140091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 shutdown_connections 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.694+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9204035a70 0x7f9204037f20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f922409e1e0 0x7f9224141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92240994f0 0x7f922413d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 --2- 172.21.15.150:0/3513457665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9224071240 0x7f922413dec0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 >> 172.21.15.150:0/3513457665 conn(0x7f92240a1cc0 msgr2=0x7f9224099990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 shutdown_connections 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.698+0000 7f9229197700 1 -- 172.21.15.150:0/3513457665 wait complete. 2024-09-18T10:11:22.701 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo-ssd already exists 2024-09-18T10:11:22.714 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-replicated foo-hdd default host hdd 2024-09-18T10:11:22.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.814+0000 7f84d0de7700 10 monclient: get_monmap_and_config 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.814+0000 7f84d0de7700 10 monclient: build_initial_monmap 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.814+0000 7f84d0de7700 10 monclient: monmap: 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:22.818987+0000 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:22.818987+0000 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:22.819 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:22.820 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:22.820 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 Processor -- start 2024-09-18T10:11:22.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 -- start start 2024-09-18T10:11:22.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient: init 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84cc09ee70 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc09f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-b con 0x7f84cc09f240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f84cc137420 0x7f84cc139800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-c con 0x7f84cc137420 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:22.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84cc0a02a0 con 0x7f84cc09ee70 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84cc0a1a90 con 0x7f84cc137420 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0a16b0 con 0x7f84cc09f240 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84d0de7700 10 monclient(hunting): authenticate will time out at 964.913609s 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84caffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 msgr2=0x7f84cc09f610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f84cc137420 0x7f84cc139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cb7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f84cc09ee70 0x7f84cc132710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38252/0 (socket says 172.21.15.150:38252) 2024-09-18T10:11:22.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cb7fe700 1 -- 172.21.15.150:0/391560186 learned_addr learned my addr 172.21.15.150:0/391560186 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84caffd700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc09f610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84ca7fc700 1 -- 172.21.15.150:0/391560186 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): get_auth_request con 0x7f84cc137420 auth_method 0 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient(hunting): handle_auth_done global_id 14545 payload 274 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 -- 172.21.15.150:0/391560186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 msgr2=0x7f84cc09f610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc09f610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 -- 172.21.15.150:0/391560186 >> v1:172.21.15.150:6789/0 conn(0x7f84cc09ee70 legacy=0x7f84cc132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 monclient: found mon.noname-c 2024-09-18T10:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.818+0000 7f84cbfff700 1 -- 172.21.15.150:0/391560186 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc132e20 con 0x7f84cc137420 2024-09-18T10:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84cbfff700 1 -- 172.21.15.150:0/391560186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 msgr2=0x7f84cc139800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:22.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84cbfff700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc139800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009e60 tx=0x7f84c4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa10eadc5115333a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 1 -- 172.21.15.150:0/391560186 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c400e070 con 0x7f84cc137420 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: dump: 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:22.829 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 5 monclient: authenticate success, global_id 14545 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 1 -- 172.21.15.150:0/391560186 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4012430 con 0x7f84cc137420 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 1 -- 172.21.15.150:0/391560186 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40127d0 con 0x7f84cc137420 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 4 set_mon_vals no callback set 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:22.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84ca7fc700 10 monclient: dump: 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:22.831 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 10 monclient: shutdown 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 -- 172.21.15.150:0/391560186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 msgr2=0x7f84cc139800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc139800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009e60 tx=0x7f84c4009780 comp rx=0 tx=0).stop 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 -- 172.21.15.150:0/391560186 shutdown_connections 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc09f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 --2- 172.21.15.150:0/391560186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc139800 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 -- 172.21.15.150:0/391560186 >> 172.21.15.150:0/391560186 conn(0x7f84cc0a56b0 msgr2=0x7f84cc0a5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 -- 172.21.15.150:0/391560186 shutdown_connections 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 1 -- 172.21.15.150:0/391560186 wait complete. 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 10 monclient: build_initial_monmap 2024-09-18T10:11:22.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.822+0000 7f84d0de7700 10 monclient: monmap: 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 Processor -- start 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 -- start start 2024-09-18T10:11:22.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient: init 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 0x7f84cc141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-a con 0x7f84cc09ee70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-c con 0x7f84cc09f240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:22.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): picked mon.noname-b con 0x7f84cc137420 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc06d890 con 0x7f84cc09ee70 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc09ec40 con 0x7f84cc137420 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:22.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc09da10 con 0x7f84cc09f240 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84d0de7700 10 monclient(hunting): authenticate will time out at 964.921802s 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84caffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 msgr2=0x7f84cc142000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84caffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc142000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57724/0 (socket says 172.21.15.150:57724) 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 -- 172.21.15.150:0/4273635261 learned_addr learned my addr 172.21.15.150:0/4273635261 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): get_auth_request con 0x7f84cc137420 auth_method 0 2024-09-18T10:11:22.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 0x7f84cc141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): get_auth_request con 0x7f84cc09ee70 auth_method 0 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:22.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient(hunting): handle_auth_done global_id 14548 payload 824 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 -- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 msgr2=0x7f84cc142000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc142000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 -- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 msgr2=0x7f84cc141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 0x7f84cc141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 monclient: found mon.noname-b 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cbfff700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4009a20 con 0x7f84cc137420 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.826+0000 7f84cb7fe700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 0x7f84cc141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84cbfff700 1 -- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 msgr2=0x7f84cc06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84cbfff700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f84c4003000 tx=0x7f84c4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:22.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4026430 con 0x7f84cc137420 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: dump: 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:22.844 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c4021070 con 0x7f84cc137420 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 5 monclient: authenticate success, global_id 14548 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _renew_subs 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc06dee0 con 0x7f84cc137420 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40267e0 con 0x7f84cc137420 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84c9ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:22.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 10 monclient: dump: 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _renew_subs 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc06d310 con 0x7f84cc137420 2024-09-18T10:11:22.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc0a13d0 con 0x7f84cc137420 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f84c4023030 con 0x7f84cc137420 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 0x7f84b8037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4767+0+0 (secure 0 0 0) 0x7f84c404a620 con 0x7f84cc137420 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84cb7fe700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 0x7f84b8037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.830+0000 7f84cb7fe700 10 monclient: get_auth_request con 0x7f84b8035af0 auth_method 0 2024-09-18T10:11:22.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.834+0000 7f84cb7fe700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 0x7f84b8037fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005220 tx=0x7f84bc007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:22.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.838+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f84c402e3e0 con 0x7f84cc137420 2024-09-18T10:11:22.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.838+0000 7f84b37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:22.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:22.838+0000 7f84b37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:23.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:23.006+0000 7f84d0de7700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:23.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:23.006+0000 7f84d0de7700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:23.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:23.006+0000 7f84d0de7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:23.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:23.006+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f84cc04bbf0 con 0x7f84cc137420 2024-09-18T10:11:24.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.098+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 v54) v1 ==== 150+0+0 (secure 0 0 0) 0x7f84c4024380 con 0x7f84cc137420 2024-09-18T10:11:24.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.098+0000 7f84b37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:24.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.098+0000 7f84b37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:24.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.202+0000 7f84d0de7700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:24.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.202+0000 7f84d0de7700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:24.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.202+0000 7f84d0de7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.202+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"} v 0) v1 -- 0x7f84cc04bbf0 con 0x7f84cc137420 2024-09-18T10:11:24.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84b37fe700 1 -- 172.21.15.150:0/4273635261 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}]=0 rule foo-hdd already exists v54) v1 ==== 177+0+0 (secure 0 0 0) 0x7f84c4024560 con 0x7f84cc137420 2024-09-18T10:11:24.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84b37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-replicated", "name": "foo-hdd", "root": "default", "type": "host", "class": "hdd"}] 2024-09-18T10:11:24.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84b37fe700 10 monclient: _finish_command 3 = system:0 rule foo-hdd already exists 2024-09-18T10:11:24.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 msgr2=0x7f84b8037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 0x7f84b8037fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005220 tx=0x7f84bc007f90 comp rx=0 tx=0).stop 2024-09-18T10:11:24.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 10 monclient: shutdown 2024-09-18T10:11:24.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:24.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 msgr2=0x7f84cc06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.206+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f84c4003000 tx=0x7f84c4018000 comp rx=0 tx=0).stop 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 shutdown_connections 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84b8035af0 0x7f84b8037fa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84cc137420 0x7f84cc06b8c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84cc09f240 0x7f84cc142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 --2- 172.21.15.150:0/4273635261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84cc09ee70 0x7f84cc141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 >> 172.21.15.150:0/4273635261 conn(0x7f84cc0a56b0 msgr2=0x7f84cc131080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 shutdown_connections 2024-09-18T10:11:24.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.210+0000 7f84d0de7700 1 -- 172.21.15.150:0/4273635261 wait complete. 2024-09-18T10:11:24.214 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo-hdd already exists 2024-09-18T10:11:24.223 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-18T10:11:24.223 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-ssd 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 10 monclient: get_monmap_and_config 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 10 monclient: build_initial_monmap 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 10 monclient: monmap: 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.322 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:24.318758+0000 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:24.318758+0000 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 1 Processor -- start 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 1 -- start start 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.314+0000 7fd033043700 10 monclient: init 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:24.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd02c09e6d0 0x7fd02c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): picked mon.noname-a con 0x7fd02c09e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 0x7fd02c0a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): picked mon.noname-b con 0x7fd02c0a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd02c136a10 0x7fd02c138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): picked mon.noname-c con 0x7fd02c136a10 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c09ee30 con 0x7fd02c09e6d0 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02c0a1110 con 0x7fd02c136a10 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c09e220 con 0x7fd02c0a07d0 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient(hunting): authenticate will time out at 966.412137s 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd02c136a10 0x7fd02c138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02bfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 msgr2=0x7fd02c0a0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 0x7fd02c0a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd02c136a10 0x7fd02c138df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57732/0 (socket says 172.21.15.150:57732) 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 -- 172.21.15.150:0/2635994090 learned_addr learned my addr 172.21.15.150:0/2635994090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): get_auth_request con 0x7fd02c136a10 auth_method 0 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd030ddf700 1 --2- 172.21.15.150:0/2635994090 >> v2:172.21.15.142:3300/0 conn(0x7fd02c09e6d0 0x7fd02c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd030ddf700 10 monclient(hunting): get_auth_request con 0x7fd02c09e6d0 auth_method 0 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd030ddf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd030ddf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd030ddf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient(hunting): handle_auth_done global_id 14551 payload 274 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 -- 172.21.15.150:0/2635994090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 msgr2=0x7fd02c0a0bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2635994090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 0x7fd02c0a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 -- 172.21.15.150:0/2635994090 >> v2:172.21.15.142:3300/0 conn(0x7fd02c09e6d0 msgr2=0x7fd02c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2635994090 >> v2:172.21.15.142:3300/0 conn(0x7fd02c09e6d0 0x7fd02c0a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient: _finish_hunting 0 2024-09-18T10:11:24.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 monclient: found mon.noname-c 2024-09-18T10:11:24.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:24.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 -- 172.21.15.150:0/2635994090 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c09eaa0 con 0x7fd02c136a10 2024-09-18T10:11:24.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 -- 172.21.15.150:0/2635994090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 msgr2=0x7fd02c138df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2635994090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c138df0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd02c06a580 tx=0x7fd01c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c27b10cd31322e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 1 -- 172.21.15.150:0/2635994090 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00e070 con 0x7fd02c136a10 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: dump: 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.331 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 1 -- 172.21.15.150:0/2635994090 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c00bc60 con 0x7fd02c136a10 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 5 monclient: authenticate success, global_id 14551 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:24.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 1 -- 172.21.15.150:0/2635994090 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c0092e0 con 0x7fd02c136a10 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd033043700 4 set_mon_vals no callback set 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.318+0000 7fd02b7fe700 10 monclient: dump: 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.333 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: shutdown 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- 172.21.15.150:0/2635994090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 msgr2=0x7fd02c138df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- 172.21.15.150:0/2635994090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c138df0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd02c06a580 tx=0x7fd01c009840 comp rx=0 tx=0).stop 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- 172.21.15.150:0/2635994090 shutdown_connections 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- 172.21.15.150:0/2635994090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c138df0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- 172.21.15.150:0/2635994090 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c0a07d0 0x7fd02c0a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- 172.21.15.150:0/2635994090 >> v2:172.21.15.142:3300/0 conn(0x7fd02c09e6d0 0x7fd02c0a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- 172.21.15.150:0/2635994090 >> 172.21.15.150:0/2635994090 conn(0x7fd02c0a58a0 msgr2=0x7fd02c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- 172.21.15.150:0/2635994090 shutdown_connections 2024-09-18T10:11:24.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- 172.21.15.150:0/2635994090 wait complete. 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: build_initial_monmap 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: monmap: 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 Processor -- start 2024-09-18T10:11:24.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- start start 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: init 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 0x7fd02c094d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): picked mon.noname-c con 0x7fd02c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd02c0a07d0 0x7fd02c06c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): picked mon.noname-a con 0x7fd02c0a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): picked mon.noname-b con 0x7fd02c136a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c06e450 con 0x7fd02c0a07d0 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c099510 con 0x7fd02c136a10 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c06f750 con 0x7fd02c09e6d0 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd033043700 10 monclient(hunting): authenticate will time out at 966.418868s 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd030ddf700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 msgr2=0x7fd02c094d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd030ddf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 0x7fd02c094d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57734/0 (socket says 172.21.15.150:57734) 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 -- 172.21.15.150:0/2935599135 learned_addr learned my addr 172.21.15.150:0/2935599135 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): get_auth_request con 0x7fd02c136a10 auth_method 0 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd02c0a07d0 0x7fd02c06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): get_auth_request con 0x7fd02c0a07d0 auth_method 0 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd02bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient(hunting): handle_auth_done global_id 14554 payload 824 2024-09-18T10:11:24.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 -- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 msgr2=0x7fd02c094d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 0x7fd02c094d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 -- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd02c0a07d0 msgr2=0x7fd02c06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd02c0a07d0 0x7fd02c06c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient: _finish_hunting 0 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 monclient: found mon.noname-b 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.322+0000 7fd0315e0700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c009ae0 con 0x7fd02c136a10 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd0315e0700 1 -- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 msgr2=0x7fd02c06c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd0315e0700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd02c06a940 tx=0x7fd01c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c008c70 con 0x7fd02c136a10 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: dump: 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:24.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 5 monclient: authenticate success, global_id 14554 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c008dd0 con 0x7fd02c136a10 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c026690 con 0x7fd02c136a10 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd028ff9700 10 monclient: dump: 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.341 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd02affd700 4 set_mon_vals no callback set 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 10 monclient: _renew_subs 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c08c410 con 0x7fd02c136a10 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 10 monclient: _renew_subs 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.326+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c095290 con 0x7fd02c136a10 2024-09-18T10:11:24.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd033043700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd033043700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd033043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c08c8f0 con 0x7fd02c136a10 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd01c003040 con 0x7fd02c136a10 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd028ff9700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 0x7fd00c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 28..54) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fd01c04a280 con 0x7fd02c136a10 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd030ddf700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 0x7fd00c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd030ddf700 10 monclient: get_auth_request con 0x7fd00c035ac0 auth_method 0 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.330+0000 7fd030ddf700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 0x7fd00c037f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd024006fd0 tx=0x7fd024009530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.334+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd01c02e9a0 con 0x7fd02c136a10 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.334+0000 7fd028ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.334+0000 7fd028ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:24.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd033043700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd033043700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd033043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fd02c063ed0 con 0x7fd02c136a10 2024-09-18T10:11:24.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fd01c0210a0 con 0x7fd02c136a10 2024-09-18T10:11:24.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd028ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.502+0000 7fd028ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:24.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd033043700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd033043700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd033043700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7fd02c0008d0 con 0x7fd02c136a10 2024-09-18T10:11:24.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd028ff9700 1 -- 172.21.15.150:0/2935599135 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fd01c01c3b0 con 0x7fd02c136a10 2024-09-18T10:11:24.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd028ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.598+0000 7fd028ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:24.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 msgr2=0x7fd00c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 0x7fd00c037f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd024006fd0 tx=0x7fd024009530 comp rx=0 tx=0).stop 2024-09-18T10:11:24.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 10 monclient: shutdown 2024-09-18T10:11:24.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:24.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 msgr2=0x7fd02c06c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd02c06a940 tx=0x7fd01c018000 comp rx=0 tx=0).stop 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 shutdown_connections 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd00c035ac0 0x7fd00c037f70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd02c136a10 0x7fd02c06c6c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd02c0a07d0 0x7fd02c06c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 --2- 172.21.15.150:0/2935599135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd02c09e6d0 0x7fd02c094d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 >> 172.21.15.150:0/2935599135 conn(0x7fd02c0a58a0 msgr2=0x7fd02c098a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:24.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 shutdown_connections 2024-09-18T10:11:24.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.602+0000 7fd033043700 1 -- 172.21.15.150:0/2935599135 wait complete. 2024-09-18T10:11:24.614 INFO:tasks.workunit.client.0.smithi150.stdout:foo-ssd 2024-09-18T10:11:24.615 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls-by-class ssd 2024-09-18T10:11:24.615 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false grep foo-hdd 2024-09-18T10:11:24.615 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:24.615 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-hdd 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: get_monmap_and_config 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: build_initial_monmap 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: monmap: 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:24.716953+0000 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:24.716953+0000 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:24.717 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:24.718 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:24.718 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:24.718 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 Processor -- start 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 -- start start 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: init 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f354009e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f35400a09a0 0x7f3540137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-a con 0x7f35400a09a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f35400a0da0 0x7f3540139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-c con 0x7f35400a0da0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354009e220 con 0x7f354009e6d0 2024-09-18T10:11:24.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354009ee30 con 0x7f35400a09a0 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354009eaa0 con 0x7f35400a0da0 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3546c79700 10 monclient(hunting): authenticate will time out at 966.810941s 2024-09-18T10:11:24.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f35400a0da0 0x7f3540139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3544a15700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f354009e6d0 0x7f35400a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38254/0 (socket says 172.21.15.150:38254) 2024-09-18T10:11:24.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3544a15700 1 -- 172.21.15.150:0/919695772 learned_addr learned my addr 172.21.15.150:0/919695772 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): get_auth_request con 0x7f35400a0da0 auth_method 0 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 1 --2- 172.21.15.150:0/919695772 >> v2:172.21.15.142:3300/0 conn(0x7f35400a09a0 0x7f3540137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): get_auth_request con 0x7f35400a09a0 auth_method 0 2024-09-18T10:11:24.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient(hunting): handle_auth_done global_id 14557 payload 274 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 -- 172.21.15.150:0/919695772 >> v2:172.21.15.142:3300/0 conn(0x7f35400a09a0 msgr2=0x7f3540137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 --2- 172.21.15.150:0/919695772 >> v2:172.21.15.142:3300/0 conn(0x7f35400a09a0 0x7f3540137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 -- 172.21.15.150:0/919695772 >> v1:172.21.15.150:6789/0 conn(0x7f354009e6d0 legacy=0x7f35400a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient: _finish_hunting 0 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 monclient: found mon.noname-c 2024-09-18T10:11:24.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:24.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.714+0000 7f3545216700 1 -- 172.21.15.150:0/919695772 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354009d930 con 0x7f35400a0da0 2024-09-18T10:11:24.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3545216700 1 -- 172.21.15.150:0/919695772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a0da0 msgr2=0x7f3540139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:24.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3545216700 1 --2- 172.21.15.150:0/919695772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a0da0 0x7f3540139f70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f35400a6970 tx=0x7f3538009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22b2aef6772f5da5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 1 -- 172.21.15.150:0/919695772 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353800e070 con 0x7f35400a0da0 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: dump: 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.733 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 5 monclient: authenticate success, global_id 14557 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 1 -- 172.21.15.150:0/919695772 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353800bd70 con 0x7f35400a0da0 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 4 set_mon_vals no callback set 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 1 -- 172.21.15.150:0/919695772 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538012660 con 0x7f35400a0da0 2024-09-18T10:11:24.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f353f7fe700 10 monclient: dump: 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.735 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 10 monclient: shutdown 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 -- 172.21.15.150:0/919695772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a0da0 msgr2=0x7f3540139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 --2- 172.21.15.150:0/919695772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a0da0 0x7f3540139f70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f35400a6970 tx=0x7f3538009780 comp rx=0 tx=0).stop 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 -- 172.21.15.150:0/919695772 shutdown_connections 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 --2- 172.21.15.150:0/919695772 >> v2:172.21.15.142:3300/0 conn(0x7f35400a09a0 0x7f3540137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 --2- 172.21.15.150:0/919695772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a0da0 0x7f3540139f70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 -- 172.21.15.150:0/919695772 >> 172.21.15.150:0/919695772 conn(0x7f35400a58a0 msgr2=0x7f35400a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 -- 172.21.15.150:0/919695772 shutdown_connections 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 -- 172.21.15.150:0/919695772 wait complete. 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 10 monclient: build_initial_monmap 2024-09-18T10:11:24.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 10 monclient: monmap: 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.718+0000 7f3546c79700 1 Processor -- start 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 -- start start 2024-09-18T10:11:24.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient: init 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 0x7f35401413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-c con 0x7f354009e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-b con 0x7f35400a09a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35400a0da0 0x7f354006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): picked mon.noname-a con 0x7f35400a0da0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354006f880 con 0x7f35400a0da0 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354012bea0 con 0x7f35400a09a0 2024-09-18T10:11:24.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3540142280 con 0x7f354009e6d0 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3546c79700 10 monclient(hunting): authenticate will time out at 966.816577s 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3544a15700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 msgr2=0x7f35401413f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3544a15700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 0x7f35401413f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57758/0 (socket says 172.21.15.150:57758) 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 -- 172.21.15.150:0/741833645 learned_addr learned my addr 172.21.15.150:0/741833645 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): get_auth_request con 0x7f35400a09a0 auth_method 0 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35400a0da0 0x7f354006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): get_auth_request con 0x7f35400a0da0 auth_method 0 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f3545216700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient(hunting): handle_auth_done global_id 14560 payload 824 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 -- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 msgr2=0x7f35401413f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:24.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 0x7f35401413f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 -- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35400a0da0 msgr2=0x7f354006deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35400a0da0 0x7f354006deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 monclient: found mon.noname-b 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3538009a20 con 0x7f35400a09a0 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 -- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 msgr2=0x7f3540131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.722+0000 7f353ffff700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f352800ec80 tx=0x7f35280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352800ca60 con 0x7f35400a09a0 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: dump: 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 5 monclient: authenticate success, global_id 14560 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _renew_subs 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352800cbc0 con 0x7f35400a09a0 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354012c1e0 con 0x7f35400a09a0 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353e7fc700 4 set_mon_vals no callback set 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _renew_subs 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354012ab10 con 0x7f35400a09a0 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528013470 con 0x7f35400a09a0 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 10 monclient: dump: 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:24.744 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354012c3c0 con 0x7f35400a09a0 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f35280137e0 con 0x7f35400a09a0 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 0x7f3520037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 28..54) v4 ==== 4830+0+0 (secure 0 0 0) 0x7f3528048780 con 0x7f35400a09a0 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3544a15700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 0x7f3520037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:24.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.726+0000 7f3544a15700 10 monclient: get_auth_request con 0x7f3520035a70 auth_method 0 2024-09-18T10:11:24.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.730+0000 7f3544a15700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 0x7f3520037f20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3530004860 tx=0x7f3530007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:24.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.734+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f35280236b0 con 0x7f35400a09a0 2024-09-18T10:11:24.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.734+0000 7f353cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:24.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.734+0000 7f353cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:24.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f3546c79700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f3546c79700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f3546c79700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:24.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f354009dbf0 con 0x7f35400a09a0 2024-09-18T10:11:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7f35280232f0 con 0x7f35400a09a0 2024-09-18T10:11:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f353cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.902+0000 7f353cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:25.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.998+0000 7f3546c79700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:25.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.998+0000 7f3546c79700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:25.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.998+0000 7f3546c79700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:24.998+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "ssd"} v 0) v1 -- 0x7f3540063ed0 con 0x7f35400a09a0 2024-09-18T10:11:25.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f353cff9700 1 -- 172.21.15.150:0/741833645 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "ssd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7f35280234d0 con 0x7f35400a09a0 2024-09-18T10:11:25.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f353cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "ssd"}] 2024-09-18T10:11:25.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f353cff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:25.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 msgr2=0x7f3520037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 0x7f3520037f20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3530004860 tx=0x7f3530007c10 comp rx=0 tx=0).stop 2024-09-18T10:11:25.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 10 monclient: shutdown 2024-09-18T10:11:25.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 msgr2=0x7f3540131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f352800ec80 tx=0x7f35280091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 shutdown_connections 2024-09-18T10:11:25.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3520035a70 0x7f3520037f20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35400a0da0 0x7f354006deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35400a09a0 0x7f3540131cb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 --2- 172.21.15.150:0/741833645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f354009e6d0 0x7f35401413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 >> 172.21.15.150:0/741833645 conn(0x7f35400a58a0 msgr2=0x7f3540134e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 shutdown_connections 2024-09-18T10:11:25.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.002+0000 7f3546c79700 1 -- 172.21.15.150:0/741833645 wait complete. 2024-09-18T10:11:25.015 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:25.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-18T10:11:25.016 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-hdd 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.110+0000 7fd59ea3a700 10 monclient: get_monmap_and_config 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.110+0000 7fd59ea3a700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.110+0000 7fd59ea3a700 10 monclient: monmap: 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.116 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:25.115732+0000 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:25.115732+0000 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:25.117 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 Processor -- start 2024-09-18T10:11:25.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 -- start start 2024-09-18T10:11:25.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient: init 2024-09-18T10:11:25.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59809d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-b con 0x7fd5980a6090 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd59809e460 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5980a0a80 0x7fd598139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-c con 0x7fd5980a0a80 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980a6460 con 0x7fd59809e460 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59809fd00 con 0x7fd5980a0a80 2024-09-18T10:11:25.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980a6850 con 0x7fd5980a6090 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59ea3a700 10 monclient(hunting): authenticate will time out at 967.209289s 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd597fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 msgr2=0x7fd59809d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd597fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59809d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5980a0a80 0x7fd598139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5980a0a80 0x7fd598139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57762/0 (socket says 172.21.15.150:57762) 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 -- 172.21.15.150:0/3537263388 learned_addr learned my addr 172.21.15.150:0/3537263388 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): get_auth_request con 0x7fd5980a0a80 auth_method 0 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient(hunting): handle_auth_done global_id 14563 payload 274 2024-09-18T10:11:25.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 -- 172.21.15.150:0/3537263388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 msgr2=0x7fd59809d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:25.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 --2- 172.21.15.150:0/3537263388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59809d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 -- 172.21.15.150:0/3537263388 >> v1:172.21.15.142:6789/0 conn(0x7fd59809e460 legacy=0x7fd5980a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 monclient: found mon.noname-c 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.114+0000 7fd59cfd7700 1 -- 172.21.15.150:0/3537263388 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59809d7a0 con 0x7fd5980a0a80 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59cfd7700 1 -- 172.21.15.150:0/3537263388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5980a0a80 msgr2=0x7fd598139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59cfd7700 1 --2- 172.21.15.150:0/3537263388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5980a0a80 0x7fd598139b10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd590009f20 tx=0x7fd590009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ea2f9ed610280f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 1 -- 172.21.15.150:0/3537263388 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59000e070 con 0x7fd5980a0a80 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:25.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: dump: 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 1 -- 172.21.15.150:0/3537263388 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59000bc90 con 0x7fd5980a0a80 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 5 monclient: authenticate success, global_id 14563 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 4 set_mon_vals no callback set 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 1 -- 172.21.15.150:0/3537263388 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590012660 con 0x7fd5980a0a80 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd596ffd700 10 monclient: dump: 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.129 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: shutdown 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3537263388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5980a0a80 msgr2=0x7fd598139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3537263388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5980a0a80 0x7fd598139b10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd590009f20 tx=0x7fd590009840 comp rx=0 tx=0).stop 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3537263388 shutdown_connections 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3537263388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5980a0a80 0x7fd598139b10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3537263388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59809d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3537263388 >> 172.21.15.150:0/3537263388 conn(0x7fd598071830 msgr2=0x7fd5980a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3537263388 shutdown_connections 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3537263388 wait complete. 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: monmap: 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.131 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 Processor -- start 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- start start 2024-09-18T10:11:25.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: init 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-b con 0x7fd59809e460 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5980a0a80 0x7fd59808d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-a con 0x7fd5980a0a80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:25.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59808d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient(hunting): picked mon.noname-c con 0x7fd5980a6090 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.118+0000 7fd59ea3a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59808edb0 con 0x7fd5980a0a80 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980902f0 con 0x7fd59809e460 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59809c190 con 0x7fd5980a6090 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient(hunting): authenticate will time out at 967.216051s 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59cfd7700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 msgr2=0x7fd59808d950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59cfd7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59808d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57764/0 (socket says 172.21.15.150:57764) 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 -- 172.21.15.150:0/3675673526 learned_addr learned my addr 172.21.15.150:0/3675673526 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): get_auth_request con 0x7fd59809e460 auth_method 0 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5980a0a80 0x7fd59808d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): get_auth_request con 0x7fd5980a0a80 auth_method 0 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd5977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient(hunting): handle_auth_done global_id 14566 payload 824 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 -- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 msgr2=0x7fd59808d950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59808d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 -- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5980a0a80 msgr2=0x7fd59808d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5980a0a80 0x7fd59808d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 monclient: found mon.noname-b 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590009ae0 con 0x7fd59809e460 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 -- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 msgr2=0x7fd59808ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd597fff700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b810 tx=0x7fd580009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580003e20 con 0x7fd59809e460 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: dump: 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.141 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 5 monclient: authenticate success, global_id 14566 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd580006150 con 0x7fd59809e460 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient: _renew_subs 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580006510 con 0x7fd59809e460 2024-09-18T10:11:25.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5980905d0 con 0x7fd59809e460 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd595ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd58ffff700 10 monclient: dump: 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.143 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient: _renew_subs 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.122+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5980984a0 con 0x7fd59809e460 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd59ea3a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd59ea3a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd59ea3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5980909b0 con 0x7fd59809e460 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd58001b3b0 con 0x7fd59809e460 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd58ffff700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 0x7fd578037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 28..54) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fd5800479b0 con 0x7fd59809e460 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd5977fe700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 0x7fd578037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.126+0000 7fd5977fe700 10 monclient: get_auth_request con 0x7fd578035ac0 auth_method 0 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.130+0000 7fd5977fe700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 0x7fd578037f70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd59808e430 tx=0x7fd588007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.134+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd58000e820 con 0x7fd59809e460 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.134+0000 7fd58ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.134+0000 7fd58ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:25.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd59ea3a700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd59ea3a700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd59ea3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fd59806d3c0 con 0x7fd59809e460 2024-09-18T10:11:25.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fd580017030 con 0x7fd59809e460 2024-09-18T10:11:25.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd58ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.290+0000 7fd58ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:25.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd59ea3a700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd59ea3a700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd59ea3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fd59809da80 con 0x7fd59809e460 2024-09-18T10:11:25.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd58ffff700 1 -- 172.21.15.150:0/3675673526 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fd580017020 con 0x7fd59809e460 2024-09-18T10:11:25.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd58ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.378+0000 7fd58ffff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:25.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 msgr2=0x7fd578037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 0x7fd578037f70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd59808e430 tx=0x7fd588007ea0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 10 monclient: shutdown 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 msgr2=0x7fd59808ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd58000b810 tx=0x7fd580009220 comp rx=0 tx=0).stop 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 shutdown_connections 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035ac0 0x7fd578037f70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5980a6090 0x7fd59808d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5980a0a80 0x7fd59808d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 --2- 172.21.15.150:0/3675673526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd59809e460 0x7fd59808ced0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 >> 172.21.15.150:0/3675673526 conn(0x7fd598071830 msgr2=0x7fd59809c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 shutdown_connections 2024-09-18T10:11:25.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.382+0000 7fd59ea3a700 1 -- 172.21.15.150:0/3675673526 wait complete. 2024-09-18T10:11:25.394 INFO:tasks.workunit.client.0.smithi150.stdout:foo-hdd 2024-09-18T10:11:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls-by-class hdd 2024-09-18T10:11:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false grep foo-ssd 2024-09-18T10:11:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-ssd 2024-09-18T10:11:25.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.490+0000 7fbe29723700 10 monclient: get_monmap_and_config 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.490+0000 7fbe29723700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.490+0000 7fbe29723700 10 monclient: monmap: 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:25.495238+0000 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:25.495238+0000 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:25.496 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.490+0000 7fbe29723700 1 Processor -- start 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 -- start start 2024-09-18T10:11:25.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient: init 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe240a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-b con 0x7fbe240a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe2406e610 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe24137590 0x7fbe24139970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-a con 0x7fbe24137590 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2406afe0 con 0x7fbe2406e610 2024-09-18T10:11:25.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe240a2220 con 0x7fbe24137590 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240a2830 con 0x7fbe240a4270 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe29723700 10 monclient(hunting): authenticate will time out at 967.588692s 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe227fc700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbe2406e610 0x7fbe2406e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38266/0 (socket says 172.21.15.150:38266) 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe227fc700 1 -- 172.21.15.150:0/39466775 learned_addr learned my addr 172.21.15.150:0/39466775 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe22ffd700 1 -- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 msgr2=0x7fbe240a1ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe22ffd700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe240a1ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 --2- 172.21.15.150:0/39466775 >> v2:172.21.15.142:3300/0 conn(0x7fbe24137590 0x7fbe24139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): get_auth_request con 0x7fbe24137590 auth_method 0 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient(hunting): handle_auth_done global_id 4851 payload 274 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 -- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 msgr2=0x7fbe240a1ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe240a1ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 -- 172.21.15.150:0/39466775 >> v1:172.21.15.150:6789/0 conn(0x7fbe2406e610 legacy=0x7fbe2406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 monclient: found mon.noname-a 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:25.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 -- 172.21.15.150:0/39466775 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240a4640 con 0x7fbe24137590 2024-09-18T10:11:25.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 -- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe24137590 msgr2=0x7fbe24139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.494+0000 7fbe237fe700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe24137590 0x7fbe24139970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009e60 tx=0x7fbe1c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229908dfcfc795e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 1 -- 172.21.15.150:0/39466775 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c00e070 con 0x7fbe24137590 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: dump: 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.507 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 5 monclient: authenticate success, global_id 4851 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 1 -- 172.21.15.150:0/39466775 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c012430 con 0x7fbe24137590 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 4 set_mon_vals no callback set 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 1 -- 172.21.15.150:0/39466775 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c0127f0 con 0x7fbe24137590 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe21ffb700 10 monclient: dump: 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: shutdown 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe24137590 msgr2=0x7fbe24139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe24137590 0x7fbe24139970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009e60 tx=0x7fbe1c009780 comp rx=0 tx=0).stop 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- 172.21.15.150:0/39466775 shutdown_connections 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe24137590 0x7fbe24139970 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- 172.21.15.150:0/39466775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe240a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- 172.21.15.150:0/39466775 >> 172.21.15.150:0/39466775 conn(0x7fbe240716e0 msgr2=0x7fbe240a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- 172.21.15.150:0/39466775 shutdown_connections 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- 172.21.15.150:0/39466775 wait complete. 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: monmap: 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 Processor -- start 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- start start 2024-09-18T10:11:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: init 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 0x7fbe2411eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-a con 0x7fbe2406e610 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe2411f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-c con 0x7fbe240a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): picked mon.noname-b con 0x7fbe24137590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406f160 con 0x7fbe2406e610 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406e000 con 0x7fbe24137590 2024-09-18T10:11:25.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe24122510 con 0x7fbe240a4270 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe29723700 10 monclient(hunting): authenticate will time out at 967.594762s 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe227fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 msgr2=0x7fbe2411f3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe227fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe2411f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57768/0 (socket says 172.21.15.150:57768) 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 -- 172.21.15.150:0/3187442469 learned_addr learned my addr 172.21.15.150:0/3187442469 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): get_auth_request con 0x7fbe24137590 auth_method 0 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 0x7fbe2411eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 10 monclient(hunting): get_auth_request con 0x7fbe2406e610 auth_method 0 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient(hunting): handle_auth_done global_id 14569 payload 824 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 -- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 msgr2=0x7fbe2411f3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe2411f3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 -- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 msgr2=0x7fbe2411eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 0x7fbe2411eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 monclient: found mon.noname-b 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe237fe700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c009a20 con 0x7fbe24137590 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.498+0000 7fbe22ffd700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 0x7fbe2411eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe237fe700 1 -- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 msgr2=0x7fbe2411f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe237fe700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe240a2b50 tx=0x7fbe1c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c012e80 con 0x7fbe24137590 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: dump: 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 5 monclient: authenticate success, global_id 14569 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _renew_subs 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c01e850 con 0x7fbe24137590 2024-09-18T10:11:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe24122230 con 0x7fbe24137590 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c01ebf0 con 0x7fbe24137590 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe217fa700 4 set_mon_vals no callback set 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 10 monclient: dump: 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _renew_subs 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe24120770 con 0x7fbe24137590 2024-09-18T10:11:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbe1c024070 con 0x7fbe24137590 2024-09-18T10:11:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 0x7fbe10037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 28..54) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fbe1c04b010 con 0x7fbe24137590 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.502+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe24122650 con 0x7fbe24137590 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.506+0000 7fbe22ffd700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 0x7fbe10037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.506+0000 7fbe22ffd700 10 monclient: get_auth_request con 0x7fbe10035a70 auth_method 0 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.506+0000 7fbe22ffd700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 0x7fbe10037f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe18004fd0 tx=0x7fbe18004aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.510+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbe1c011b70 con 0x7fbe24137590 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.510+0000 7fbe0affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.510+0000 7fbe0affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe29723700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe29723700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe29723700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fbe2406ad00 con 0x7fbe24137590 2024-09-18T10:11:25.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fbe1c011390 con 0x7fbe24137590 2024-09-18T10:11:25.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe0affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.686+0000 7fbe0affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:25.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.778+0000 7fbe29723700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.778+0000 7fbe29723700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.778+0000 7fbe29723700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.778+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls-by-class", "class": "hdd"} v 0) v1 -- 0x7fbe2406ad00 con 0x7fbe24137590 2024-09-18T10:11:25.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe0affd700 1 -- 172.21.15.150:0/3187442469 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls-by-class", "class": "hdd"}]=0 v54) v1 ==== 90+0+8 (secure 0 0 0) 0x7fbe1c011570 con 0x7fbe24137590 2024-09-18T10:11:25.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe0affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls-by-class", "class": "hdd"}] 2024-09-18T10:11:25.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe0affd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:25.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 msgr2=0x7fbe10037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 0x7fbe10037f20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbe18004fd0 tx=0x7fbe18004aa0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 10 monclient: shutdown 2024-09-18T10:11:25.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 msgr2=0x7fbe2411f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe240a2b50 tx=0x7fbe1c018000 comp rx=0 tx=0).stop 2024-09-18T10:11:25.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 shutdown_connections 2024-09-18T10:11:25.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe10035a70 0x7fbe10037f20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe24137590 0x7fbe2411f920 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe240a4270 0x7fbe2411f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 --2- 172.21.15.150:0/3187442469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe2406e610 0x7fbe2411eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 >> 172.21.15.150:0/3187442469 conn(0x7fbe240716e0 msgr2=0x7fbe2406c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 shutdown_connections 2024-09-18T10:11:25.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.782+0000 7fbe29723700 1 -- 172.21.15.150:0/3187442469 wait complete. 2024-09-18T10:11:25.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:25.796 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd erasure-code-profile set ec-foo-ssd crush-device-class=ssd m=2 k=2 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: get_monmap_and_config 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: monmap: 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:25.898155+0000 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:25.898155+0000 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.898 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:25.899 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 Processor -- start 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 -- start start 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: init 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed5406b5a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed5406e710 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:25.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fed5406eb10 0x7fed5413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-a con 0x7fed5406eb10 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5406b970 con 0x7fed5406b5a0 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5406ef90 con 0x7fed5406e710 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed5409fb10 con 0x7fed5406eb10 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed5b1b9700 10 monclient(hunting): authenticate will time out at 967.991650s 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed53fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fed5406e710 0x7fed541374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38272/0 (socket says 172.21.15.150:38272) 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.894+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 learned_addr learned my addr 172.21.15.150:0/3928340878 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 1 --2- 172.21.15.150:0/3928340878 >> v2:172.21.15.142:3300/0 conn(0x7fed5406eb10 0x7fed5413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): get_auth_request con 0x7fed5406eb10 auth_method 0 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 >> v1:172.21.15.150:6789/0 conn(0x7fed5406e710 legacy=0x7fed541374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726713232 0 0) 0x7fed5406ef90 con 0x7fed5406e710 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): my global_id is 14572 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed3c003570 con 0x7fed5406e710 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4004887199 0 0) 0x7fed3c003570 con 0x7fed5406e710 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed58f55700 1 -- 172.21.15.150:0/3928340878 >> v1:172.21.15.142:6789/0 conn(0x7fed5406b5a0 legacy=0x7fed5409e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 >> v2:172.21.15.142:3300/0 conn(0x7fed5406eb10 msgr2=0x7fed5413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 --2- 172.21.15.150:0/3928340878 >> v2:172.21.15.142:3300/0 conn(0x7fed5406eb10 0x7fed5413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 >> v1:172.21.15.142:6789/0 conn(0x7fed5406b5a0 legacy=0x7fed5409e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5409ee40 con 0x7fed5406e710 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed5b1b9700 5 monclient: authenticate success, global_id 14572 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fed440017a0 con 0x7fed5406e710 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed5b1b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: dump: 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 >> v1:172.21.15.150:6789/0 conn(0x7fed5406e710 legacy=0x7fed541374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed3c003c50 0x7fed3c004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): picked mon.c con 0x7fed3c003c50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 0x7fed3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): picked mon.b con 0x7fed3c004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c009a50 0x7fed3c00be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): picked mon.a con 0x7fed3c009a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5409fb10 con 0x7fed3c009a50 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c00c3f0 con 0x7fed3c004810 2024-09-18T10:11:25.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c00c5b0 con 0x7fed3c003c50 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed537fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed5b1b9700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed58f55700 1 -- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed3c003c50 msgr2=0x7fed3c004060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 0x7fed3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed58f55700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed3c003c50 0x7fed3c004060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): get_auth_request con 0x7fed3c004810 auth_method 0 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c009a50 0x7fed3c00be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): get_auth_request con 0x7fed3c009a50 auth_method 0 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient(hunting): handle_auth_done global_id 14572 payload 293 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed3c003c50 msgr2=0x7fed3c004060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed3c003c50 0x7fed3c004060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c009a50 msgr2=0x7fed3c00be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c009a50 0x7fed3c00be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 monclient: found mon.b 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed59756700 1 -- 172.21.15.150:0/3928340878 reap_dead start 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c003570 con 0x7fed3c004810 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.898+0000 7fed53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed53fff700 1 -- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 msgr2=0x7fed3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed53fff700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 0x7fed3c004c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fed440014d0 tx=0x7fed44007b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70a2fe87f788b3d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4400ce20 con 0x7fed3c004810 2024-09-18T10:11:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: dump: 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44005920 con 0x7fed3c004810 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 4 set_mon_vals no callback set 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 1 -- 172.21.15.150:0/3928340878 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44005ce0 con 0x7fed3c004810 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed537fe700 10 monclient: dump: 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 10 monclient: shutdown 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 -- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 msgr2=0x7fed3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 0x7fed3c004c80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fed440014d0 tx=0x7fed44007b20 comp rx=0 tx=0).stop 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 -- 172.21.15.150:0/3928340878 shutdown_connections 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/3928340878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed3c004810 0x7fed3c004c80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 -- 172.21.15.150:0/3928340878 >> 172.21.15.150:0/3928340878 conn(0x7fed54071b60 msgr2=0x7fed540a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 -- 172.21.15.150:0/3928340878 shutdown_connections 2024-09-18T10:11:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 1 -- 172.21.15.150:0/3928340878 wait complete. 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 10 monclient: build_initial_monmap 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.902+0000 7fed5b1b9700 10 monclient: monmap: 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 Processor -- start 2024-09-18T10:11:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 -- start start 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient: init 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c004810 0x7fed541415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-a con 0x7fed3c004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-b con 0x7fed541341f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 0x7fed5411e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): picked mon.noname-c con 0x7fed5411dcc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54134b00 con 0x7fed3c004810 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541424d0 con 0x7fed541341f0 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed54069b30 con 0x7fed5411dcc0 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed59756700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 msgr2=0x7fed5411e130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed5b1b9700 10 monclient(hunting): authenticate will time out at 968.001768s 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed59756700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 0x7fed5411e130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57776/0 (socket says 172.21.15.150:57776) 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 -- 172.21.15.150:0/2699508269 learned_addr learned my addr 172.21.15.150:0/2699508269 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): get_auth_request con 0x7fed541341f0 auth_method 0 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c004810 0x7fed541415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): get_auth_request con 0x7fed3c004810 auth_method 0 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed58f55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient(hunting): handle_auth_done global_id 14575 payload 824 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 -- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 msgr2=0x7fed5411e130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 0x7fed5411e130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 -- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c004810 msgr2=0x7fed541415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c004810 0x7fed541415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 monclient: found mon.noname-b 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.906+0000 7fed53fff700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed54121680 con 0x7fed541341f0 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed53fff700 1 -- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 msgr2=0x7fed541345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed53fff700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fed44007bd0 tx=0x7fed44015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44022070 con 0x7fed541341f0 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: dump: 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 5 monclient: authenticate success, global_id 14575 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _renew_subs 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44001190 con 0x7fed541341f0 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed54135060 con 0x7fed541341f0 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed44007720 con 0x7fed541341f0 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 10 monclient: dump: 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _renew_subs 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5411f850 con 0x7fed541341f0 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed52ffd700 4 set_mon_vals no callback set 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fed44004a40 con 0x7fed541341f0 2024-09-18T10:11:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5406a7e0 con 0x7fed541341f0 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 0x7fed40037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 28..54) v4 ==== 4830+0+0 (secure 0 0 0) 0x7fed44049570 con 0x7fed541341f0 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed58f55700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 0x7fed40037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.910+0000 7fed58f55700 10 monclient: get_auth_request con 0x7fed40035a70 auth_method 0 2024-09-18T10:11:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.914+0000 7fed58f55700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 0x7fed40037f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed48000c00 tx=0x7fed48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:25.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.922+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fed4401e430 con 0x7fed541341f0 2024-09-18T10:11:25.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.922+0000 7fed50ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:25.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:25.922+0000 7fed50ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:26.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.094+0000 7fed5b1b9700 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.094+0000 7fed5b1b9700 10 monclient: _send_command 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.094+0000 7fed5b1b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.094+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7fed54063ed0 con 0x7fed541341f0 2024-09-18T10:11:26.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.110+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v55) v1 ==== 151+0+0 (secure 0 0 0) 0x7fed44017070 con 0x7fed541341f0 2024-09-18T10:11:26.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.110+0000 7fed50ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.110+0000 7fed50ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:26.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.210+0000 7fed5b1b9700 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.210+0000 7fed5b1b9700 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.214+0000 7fed5b1b9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.214+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]} v 0) v1 -- 0x7fed54063ed0 con 0x7fed541341f0 2024-09-18T10:11:26.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.214+0000 7fed50ff9700 1 -- 172.21.15.150:0/2699508269 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}]=0 v55) v1 ==== 151+0+0 (secure 0 0 0) 0x7fed4401d390 con 0x7fed541341f0 2024-09-18T10:11:26.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.214+0000 7fed50ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "ec-foo-ssd", "profile": ["crush-device-class=ssd", "m=2", "k=2"]}] 2024-09-18T10:11:26.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.214+0000 7fed50ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 msgr2=0x7fed40037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 0x7fed40037f20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed48000c00 tx=0x7fed48009040 comp rx=0 tx=0).stop 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 10 monclient: shutdown 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 msgr2=0x7fed541345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fed44007bd0 tx=0x7fed44015000 comp rx=0 tx=0).stop 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 shutdown_connections 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed40035a70 0x7fed40037f20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed5411dcc0 0x7fed5411e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed541341f0 0x7fed541345c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 --2- 172.21.15.150:0/2699508269 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed3c004810 0x7fed541415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 >> 172.21.15.150:0/2699508269 conn(0x7fed54071b60 msgr2=0x7fed5406d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 shutdown_connections 2024-09-18T10:11:26.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.218+0000 7fed5b1b9700 1 -- 172.21.15.150:0/2699508269 wait complete. 2024-09-18T10:11:26.229 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create ec-foo 2 erasure ec-foo-ssd 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.326+0000 7f9601304700 10 monclient: get_monmap_and_config 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.326+0000 7f9601304700 10 monclient: build_initial_monmap 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.326+0000 7f9601304700 10 monclient: monmap: 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:26.331428+0000 2024-09-18T10:11:26.331 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:26.331428+0000 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:26.332 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 Processor -- start 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 -- start start 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient: init 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95fc0a05e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95fc09e800 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f95fc09d230 0x7f95fc13b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:26.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): picked mon.noname-a con 0x7f95fc09d230 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc09e060 con 0x7f95fc0a05e0 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95fc09daa0 con 0x7f95fc09e800 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95fc068b30 con 0x7f95fc09d230 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f9601304700 10 monclient(hunting): authenticate will time out at 968.424941s 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fa7fc700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95fc09e800 0x7f95fc137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38276/0 (socket says 172.21.15.150:38276) 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fa7fc700 1 -- 172.21.15.150:0/1678486284 learned_addr learned my addr 172.21.15.150:0/1678486284 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 1 --2- 172.21.15.150:0/1678486284 >> v2:172.21.15.142:3300/0 conn(0x7f95fc09d230 0x7f95fc13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): get_auth_request con 0x7f95fc09d230 auth_method 0 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:26.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:26.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:26.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:26.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:26.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fa7fc700 1 -- 172.21.15.150:0/1678486284 >> v1:172.21.15.150:6789/0 conn(0x7f95fc09e800 legacy=0x7f95fc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:26.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95f9ffb700 1 -- 172.21.15.150:0/1678486284 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564802318 0 0) 0x7f95fc09daa0 con 0x7f95fc09e800 2024-09-18T10:11:26.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95f9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:26.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95f9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:26.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95f9ffb700 10 monclient(hunting): my global_id is 14578 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95f9ffb700 1 -- 172.21.15.150:0/1678486284 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4003570 con 0x7f95fc09e800 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95faffd700 1 -- 172.21.15.150:0/1678486284 >> v1:172.21.15.142:6789/0 conn(0x7f95fc0a05e0 legacy=0x7f95fc0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 10 monclient(hunting): handle_auth_done global_id 4863 payload 274 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.330+0000 7f95fb7fe700 1 -- 172.21.15.150:0/1678486284 >> v1:172.21.15.150:6789/0 conn(0x7f95fc09e800 legacy=0x7f95fc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 1 -- 172.21.15.150:0/1678486284 >> v1:172.21.15.142:6789/0 conn(0x7f95fc0a05e0 legacy=0x7f95fc0a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 1 monclient: found mon.noname-a 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:26.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:26.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 1 -- 172.21.15.150:0/1678486284 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc09fca0 con 0x7f95fc09d230 2024-09-18T10:11:26.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 1 -- 172.21.15.150:0/1678486284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 msgr2=0x7f95fc13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:26.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95fb7fe700 1 --2- 172.21.15.150:0/1678486284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc13b150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009f60 tx=0x7f95f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e1f331dec545b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:26.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 1 -- 172.21.15.150:0/1678486284 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f400e070 con 0x7f95fc09d230 2024-09-18T10:11:26.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:26.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:26.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: dump: 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:26.342 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:26.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 1 -- 172.21.15.150:0/1678486284 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f400be40 con 0x7f95fc09d230 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 5 monclient: authenticate success, global_id 4863 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:26.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 1 -- 172.21.15.150:0/1678486284 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4012640 con 0x7f95fc09d230 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 4 set_mon_vals no callback set 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f95f9ffb700 10 monclient: dump: 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:26.345 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:26.346 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 10 monclient: shutdown 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 1 -- 172.21.15.150:0/1678486284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 msgr2=0x7f95fc13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.334+0000 7f9601304700 1 --2- 172.21.15.150:0/1678486284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc13b150 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009f60 tx=0x7f95f4009780 comp rx=0 tx=0).stop 2024-09-18T10:11:26.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- 172.21.15.150:0/1678486284 shutdown_connections 2024-09-18T10:11:26.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 --2- 172.21.15.150:0/1678486284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc13b150 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- 172.21.15.150:0/1678486284 >> 172.21.15.150:0/1678486284 conn(0x7f95fc0a5a40 msgr2=0x7f95fc0a5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:26.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- 172.21.15.150:0/1678486284 shutdown_connections 2024-09-18T10:11:26.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- 172.21.15.150:0/1678486284 wait complete. 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient: build_initial_monmap 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient: monmap: 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:26.358 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:26.359 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 Processor -- start 2024-09-18T10:11:26.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- start start 2024-09-18T10:11:26.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient: init 2024-09-18T10:11:26.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:26.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc08dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): picked mon.noname-a con 0x7f95fc09d230 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): picked mon.noname-b con 0x7f95fc09e800 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 0x7f95fc08e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:26.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): picked mon.noname-c con 0x7f95fc0a05e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc09bbb0 con 0x7f95fc09d230 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc099500 con 0x7f95fc09e800 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:26.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc08fda0 con 0x7f95fc0a05e0 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f9601304700 10 monclient(hunting): authenticate will time out at 968.435326s 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fb7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 msgr2=0x7f95fc08e960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 0x7f95fc08e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:26.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57792/0 (socket says 172.21.15.150:57792) 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 -- 172.21.15.150:0/500847556 learned_addr learned my addr 172.21.15.150:0/500847556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): get_auth_request con 0x7f95fc09e800 auth_method 0 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:26.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc08dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:26.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:26.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:26.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:26.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): get_auth_request con 0x7f95fc09d230 auth_method 0 2024-09-18T10:11:26.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:26.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:26.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:26.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:26.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 10 monclient(hunting): handle_auth_done global_id 14581 payload 824 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 msgr2=0x7f95fc08e960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 0x7f95fc08e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 msgr2=0x7f95fc08dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:26.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.338+0000 7f95fa7fc700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc08dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:26.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:26.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 1 monclient: found mon.noname-b 2024-09-18T10:11:26.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:26.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4009a20 con 0x7f95fc09e800 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 msgr2=0x7f95fc08e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95fa7fc700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f95f000c7c0 tx=0x7f95f0007c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0005d40 con 0x7f95fc09e800 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:26.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: dump: 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:26.372 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:26.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f00070d0 con 0x7f95fc09e800 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 5 monclient: authenticate success, global_id 14581 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _renew_subs 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc091350 con 0x7f95fc09e800 2024-09-18T10:11:26.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:26.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0005ea0 con 0x7f95fc09e800 2024-09-18T10:11:26.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:26.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:26.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 10 monclient: dump: 2024-09-18T10:11:26.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:26.376 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:26.376 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.376 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:26.376 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:26.376 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:26.377 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:26.377 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:26.377 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:26.377 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:26.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _renew_subs 2024-09-18T10:11:26.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95f97fa700 4 set_mon_vals no callback set 2024-09-18T10:11:26.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc08f890 con 0x7f95fc09e800 2024-09-18T10:11:26.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:26.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:26.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc06e3e0 con 0x7f95fc09e800 2024-09-18T10:11:26.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.342+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f95f0007740 con 0x7f95fc09e800 2024-09-18T10:11:26.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95e2ffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:26.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 28..55) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f95f0047960 con 0x7f95fc09e800 2024-09-18T10:11:26.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:26.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 10 monclient: get_auth_request con 0x7f95e8035a70 auth_method 0 2024-09-18T10:11:26.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 msgr2=0x7f95e8037f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:11:26.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 msgr2=0x7f95e8037f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:26.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:26.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.346+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:26.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.354+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f95f001a360 con 0x7f95fc09e800 2024-09-18T10:11:26.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.354+0000 7f95e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:26.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.354+0000 7f95e2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.502+0000 7f9601304700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.502+0000 7f9601304700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.502+0000 7f9601304700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.502+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f95fc063ed0 con 0x7f95fc09e800 2024-09-18T10:11:26.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.550+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:26.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.550+0000 7f95faffd700 10 monclient: get_auth_request con 0x7f95e8035a70 auth_method 0 2024-09-18T10:11:26.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:26.550+0000 7f95faffd700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f95ec001570 tx=0x7f95ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:28.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.462+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' created v57) v1 ==== 177+0+0 (secure 0 0 0) 0x7f95f0017070 con 0x7f95fc09e800 2024-09-18T10:11:28.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.462+0000 7f95e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:28.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.462+0000 7f95e2ffd700 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' created 2024-09-18T10:11:28.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.570+0000 7f9601304700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:28.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.570+0000 7f9601304700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:28.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.570+0000 7f9601304700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.570+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"} v 0) v1 -- 0x7f95fc067b10 con 0x7f95fc09e800 2024-09-18T10:11:28.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f95e2ffd700 1 -- 172.21.15.150:0/500847556 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}]=0 pool 'ec-foo' already exists v57) v1 ==== 184+0+0 (secure 0 0 0) 0x7f95f000e290 con 0x7f95fc09e800 2024-09-18T10:11:28.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f95e2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "ec-foo", "pg_num": 2, "pool_type": "erasure", "erasure_code_profile": "ec-foo-ssd"}] 2024-09-18T10:11:28.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f95e2ffd700 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' already exists 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 msgr2=0x7f95e8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f95ec001570 tx=0x7f95ec009040 comp rx=0 tx=0).stop 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 10 monclient: shutdown 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 msgr2=0x7f95fc08e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f95f000c7c0 tx=0x7f95f0007c30 comp rx=0 tx=0).stop 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 shutdown_connections 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95fc0a05e0 0x7f95fc08e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95fc09e800 0x7f95fc08e420 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 --2- 172.21.15.150:0/500847556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95fc09d230 0x7f95fc08dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 >> 172.21.15.150:0/500847556 conn(0x7f95fc0a5a40 msgr2=0x7f95fc09b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 shutdown_connections 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.574+0000 7f9601304700 1 -- 172.21.15.150:0/500847556 wait complete. 2024-09-18T10:11:28.580 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'ec-foo' already exists 2024-09-18T10:11:28.589 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool rm ec-foo ec-foo --yes-i-really-really-mean-it 2024-09-18T10:11:28.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: get_monmap_and_config 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: build_initial_monmap 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: monmap: 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:28.685806+0000 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:28.685806+0000 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:28.686 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:28.687 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:28.687 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 Processor -- start 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 -- start start 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: init 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f985c09b2b0 0x7f985c09b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): picked mon.noname-a con 0x7f985c09b2b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f985c0955b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f985c098180 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c0985a0 con 0x7f985c0955b0 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c09aab0 con 0x7f985c098180 2024-09-18T10:11:28.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c09a7d0 con 0x7f985c09b2b0 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f986308b700 10 monclient(hunting): authenticate will time out at 970.779313s 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9861628700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f985c098180 0x7f985c135980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38286/0 (socket says 172.21.15.150:38286) 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 learned_addr learned my addr 172.21.15.150:0/1271308354 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9860e27700 1 --2- 172.21.15.150:0/1271308354 >> v2:172.21.15.142:3300/0 conn(0x7f985c09b2b0 0x7f985c09b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9860e27700 10 monclient(hunting): get_auth_request con 0x7f985c09b2b0 auth_method 0 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9860e27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9860e27700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.682+0000 7f9860e27700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:28.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 >> v1:172.21.15.150:6789/0 conn(0x7f985c098180 legacy=0x7f985c135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:28.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226544505 0 0) 0x7f985c09aab0 con 0x7f985c098180 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): my global_id is 14584 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984c003570 con 0x7f985c098180 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 447193611 0 0) 0x7f984c003570 con 0x7f985c098180 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 >> v2:172.21.15.142:3300/0 conn(0x7f985c09b2b0 msgr2=0x7f985c09b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 --2- 172.21.15.150:0/1271308354 >> v2:172.21.15.142:3300/0 conn(0x7f985c09b2b0 0x7f985c09b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 >> v1:172.21.15.142:6789/0 conn(0x7f985c0955b0 legacy=0x7f985c097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:28.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c0992f0 con 0x7f985c098180 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f986308b700 5 monclient: authenticate success, global_id 14584 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f986308b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f9854001be0 con 0x7f985c098180 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: dump: 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.697 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 >> v1:172.21.15.150:6789/0 conn(0x7f985c098180 legacy=0x7f985c135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f984c003b30 0x7f984c003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): picked mon.a con 0x7f984c003b30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c004810 0x7f984c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): picked mon.c con 0x7f984c004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): picked mon.b con 0x7f984c0053a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c09a7d0 con 0x7f984c003b30 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984c00e570 con 0x7f984c0053a0 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c00e730 con 0x7f984c004810 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:28.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f986308b700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985bfff700 1 -- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c004810 msgr2=0x7f984c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): get_auth_request con 0x7f984c0053a0 auth_method 0 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f985bfff700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c004810 0x7f984c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f984c003b30 0x7f984c003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:28.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 10 monclient(hunting): get_auth_request con 0x7f984c003b30 auth_method 0 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient(hunting): handle_auth_done global_id 14584 payload 293 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c004810 msgr2=0x7f984c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c004810 0x7f984c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f984c003b30 msgr2=0x7f984c003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f984c003b30 0x7f984c003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient: _finish_hunting 0 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 monclient: found mon.b 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c003570 con 0x7f984c0053a0 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 10 monclient: _finish_auth 0 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9860e27700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f984c003b30 0x7f984c003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.686+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 reap_dead start 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f9861628700 1 -- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 msgr2=0x7f984c00dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f9861628700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9854001850 tx=0x7f985400a3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5cc970bde71e734 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f9861628700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9854001850 tx=0x7f985400a3b0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f985400c670 mon_map magic: 0 v1 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985400c670 con 0x7f984c0053a0 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 10 monclient: dump: 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:28.702 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985400c7f0 con 0x7f984c0053a0 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 4 set_mon_vals no callback set 2024-09-18T10:11:28.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 10 monclient: shutdown 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 -- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 msgr2=0x7f984c00dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9854001850 tx=0x7f985400a3b0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 1 -- 172.21.15.150:0/1271308354 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f985400cbb0 con 0x7f984c0053a0 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f985b7fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 -- 172.21.15.150:0/1271308354 shutdown_connections 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 --2- 172.21.15.150:0/1271308354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f984c0053a0 0x7f984c00dc90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 -- 172.21.15.150:0/1271308354 >> 172.21.15.150:0/1271308354 conn(0x7f985c0a15e0 msgr2=0x7f985c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 -- 172.21.15.150:0/1271308354 shutdown_connections 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 -- 172.21.15.150:0/1271308354 wait complete. 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 10 monclient: build_initial_monmap 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 10 monclient: monmap: 2024-09-18T10:11:28.704 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.690+0000 7f986308b700 1 Processor -- start 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 -- start start 2024-09-18T10:11:28.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient: init 2024-09-18T10:11:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:11:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 0x7f985c13d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): picked mon.noname-c con 0x7f984c0053a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): picked mon.noname-b con 0x7f985c13dd60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c089be0 0x7f985c08a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): picked mon.noname-a con 0x7f985c089be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c08d850 con 0x7f985c089be0 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c06fba0 con 0x7f985c13dd60 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c0977c0 con 0x7f984c0053a0 2024-09-18T10:11:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f986308b700 10 monclient(hunting): authenticate will time out at 970.789104s 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9860e27700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 msgr2=0x7f985c13d820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9860e27700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 0x7f985c13d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57814/0 (socket says 172.21.15.150:57814) 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 -- 172.21.15.150:0/261453591 learned_addr learned my addr 172.21.15.150:0/261453591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): get_auth_request con 0x7f985c13dd60 auth_method 0 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c089be0 0x7f985c08a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:28.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): get_auth_request con 0x7f985c089be0 auth_method 0 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f9861628700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient(hunting): handle_auth_done global_id 14587 payload 824 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 -- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 msgr2=0x7f985c13d820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 0x7f985c13d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 -- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c089be0 msgr2=0x7f985c08a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:28.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c089be0 0x7f985c08a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 monclient: found mon.noname-b 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.694+0000 7f985bfff700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c08bbe0 con 0x7f985c13dd60 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f985bfff700 1 -- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 msgr2=0x7f985c13e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f985bfff700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f985000c840 tx=0x7f9850007f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98500074f0 con 0x7f985c13dd60 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: dump: 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 5 monclient: authenticate success, global_id 14587 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9850007670 con 0x7f985c13dd60 2024-09-18T10:11:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _renew_subs 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c08b6d0 con 0x7f985c13dd60 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850005a60 con 0x7f985c13dd60 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 10 monclient: dump: 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f985affd700 4 set_mon_vals no callback set 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _renew_subs 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c08c2a0 con 0x7f985c13dd60 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f985001b3b0 con 0x7f985c13dd60 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 0x7f9840037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c09a4f0 con 0x7f985c13dd60 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 28..57) v4 ==== 5496+0+0 (secure 0 0 0) 0x7f9850047bd0 con 0x7f985c13dd60 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9860e27700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 0x7f9840037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.698+0000 7f9860e27700 10 monclient: get_auth_request con 0x7f9840035aa0 auth_method 0 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.702+0000 7f9860e27700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 0x7f9840037f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.710+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9850017070 con 0x7f985c13dd60 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.710+0000 7f9858ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:28.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.710+0000 7f9858ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:28.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.878+0000 7f986308b700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:28.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.878+0000 7f986308b700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:28.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.878+0000 7f986308b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:28.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:28.878+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f985c0990d0 con 0x7f985c13dd60 2024-09-18T10:11:29.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.466+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' removed v58) v1 ==== 154+0+0 (secure 0 0 0) 0x7f985001a360 con 0x7f985c13dd60 2024-09-18T10:11:29.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.466+0000 7f9858ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:29.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.466+0000 7f9858ff9700 10 monclient: _finish_command 2 = system:0 pool 'ec-foo' removed 2024-09-18T10:11:29.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f986308b700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:29.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f986308b700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:29.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f986308b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f985c063ed0 con 0x7f985c13dd60 2024-09-18T10:11:29.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f9858ff9700 1 -- 172.21.15.150:0/261453591 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}]=0 pool 'ec-foo' does not exist v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f985000e290 con 0x7f985c13dd60 2024-09-18T10:11:29.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f9858ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "ec-foo", "pool2": "ec-foo", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:29.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.570+0000 7f9858ff9700 10 monclient: _finish_command 3 = system:0 pool 'ec-foo' does not exist 2024-09-18T10:11:29.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 msgr2=0x7f9840037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 0x7f9840037f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 10 monclient: shutdown 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 msgr2=0x7f985c13e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f985000c840 tx=0x7f9850007f10 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 shutdown_connections 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9840035aa0 0x7f9840037f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c089be0 0x7f985c08a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c13dd60 0x7f985c13e130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 --2- 172.21.15.150:0/261453591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f984c0053a0 0x7f985c13d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 >> 172.21.15.150:0/261453591 conn(0x7f985c0a15e0 msgr2=0x7f985c096ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:29.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 shutdown_connections 2024-09-18T10:11:29.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.574+0000 7f986308b700 1 -- 172.21.15.150:0/261453591 wait complete. 2024-09-18T10:11:29.579 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'ec-foo' does not exist 2024-09-18T10:11:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:11:29.588 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: get_monmap_and_config 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: build_initial_monmap 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: monmap: 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:29.682129+0000 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:29.682129+0000 2024-09-18T10:11:29.682 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:29.683 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 Processor -- start 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 -- start start 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: init 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f467009e370 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f467006cbc0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f46700699a0 0x7f467013ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-c con 0x7f46700699a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467006af30 con 0x7f467009e370 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467006b630 con 0x7f467006cbc0 2024-09-18T10:11:29.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467006ac50 con 0x7f46700699a0 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f4675cc2700 10 monclient(hunting): authenticate will time out at 971.775537s 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f46700699a0 0x7f467013ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466effd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f467006cbc0 0x7f4670137740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:56452/0 (socket says 172.21.15.150:56452) 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466effd700 1 -- 172.21.15.150:0/619503039 learned_addr learned my addr 172.21.15.150:0/619503039 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466ffff700 10 monclient(hunting): get_auth_request con 0x7f46700699a0 auth_method 0 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.678+0000 7f466ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:29.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient(hunting): handle_auth_done global_id 14590 payload 274 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 -- 172.21.15.150:0/619503039 >> v1:172.21.15.150:6789/0 conn(0x7f467006cbc0 legacy=0x7f4670137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 -- 172.21.15.150:0/619503039 >> v1:172.21.15.142:6789/0 conn(0x7f467009e370 legacy=0x7f467009e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 monclient: found mon.noname-c 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 -- 172.21.15.150:0/619503039 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46700a4540 con 0x7f46700699a0 2024-09-18T10:11:29.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 -- 172.21.15.150:0/619503039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 msgr2=0x7f467013ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:29.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466ffff700 1 --2- 172.21.15.150:0/619503039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f467013ae70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f467006f0e0 tx=0x7f4668009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=911b4933e85b256b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 1 -- 172.21.15.150:0/619503039 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466800e070 con 0x7f46700699a0 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: dump: 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:29.694 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 1 -- 172.21.15.150:0/619503039 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4668012430 con 0x7f46700699a0 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 5 monclient: authenticate success, global_id 14590 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 1 -- 172.21.15.150:0/619503039 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680127d0 con 0x7f46700699a0 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 4 set_mon_vals no callback set 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f466e7fc700 10 monclient: dump: 2024-09-18T10:11:29.695 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 10 monclient: shutdown 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 1 -- 172.21.15.150:0/619503039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 msgr2=0x7f467013ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 1 --2- 172.21.15.150:0/619503039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f467013ae70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f467006f0e0 tx=0x7f4668009780 comp rx=0 tx=0).stop 2024-09-18T10:11:29.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 1 -- 172.21.15.150:0/619503039 shutdown_connections 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 1 --2- 172.21.15.150:0/619503039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f467013ae70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.682+0000 7f4675cc2700 1 -- 172.21.15.150:0/619503039 >> 172.21.15.150:0/619503039 conn(0x7f4670071850 msgr2=0x7f46700a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- 172.21.15.150:0/619503039 shutdown_connections 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- 172.21.15.150:0/619503039 wait complete. 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient: build_initial_monmap 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient: monmap: 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.697 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 Processor -- start 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- start start 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient: init 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-b con 0x7f46700699a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 0x7f467011ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-c con 0x7f467006cbc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f467009e370 0x7f467011f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:29.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): picked mon.noname-a con 0x7f467009e370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:29.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:29.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467006f110 con 0x7f467009e370 2024-09-18T10:11:29.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:29.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467006d2b0 con 0x7f46700699a0 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46701226e0 con 0x7f467006cbc0 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f4675cc2700 10 monclient(hunting): authenticate will time out at 971.782357s 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466effd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 msgr2=0x7f467011ede0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 0x7f467011ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47160/0 (socket says 172.21.15.150:47160) 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 -- 172.21.15.150:0/3974488235 learned_addr learned my addr 172.21.15.150:0/3974488235 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): get_auth_request con 0x7f46700699a0 auth_method 0 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:29.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f467009e370 0x7f467011f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): get_auth_request con 0x7f467009e370 auth_method 0 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient(hunting): handle_auth_done global_id 14593 payload 824 2024-09-18T10:11:29.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 -- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 msgr2=0x7f467011ede0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 0x7f467011ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 -- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f467009e370 msgr2=0x7f467011f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f467009e370 0x7f467011f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.686+0000 7f466f7fe700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668009a20 con 0x7f46700699a0 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f466f7fe700 1 -- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 msgr2=0x7f46701322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:11:29.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f466f7fe700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f466400c5d0 tx=0x7f4664007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:29.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46640117a0 con 0x7f46700699a0 2024-09-18T10:11:29.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:29.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:29.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: dump: 2024-09-18T10:11:29.704 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4664011900 con 0x7f46700699a0 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 5 monclient: authenticate success, global_id 14593 2024-09-18T10:11:29.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 10 monclient: _renew_subs 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46701208d0 con 0x7f46700699a0 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664011cc0 con 0x7f46700699a0 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f46577fe700 10 monclient: dump: 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:29.706 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 10 monclient: _renew_subs 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46701214a0 con 0x7f46700699a0 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.690+0000 7f466dffb700 4 set_mon_vals no callback set 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4664012630 con 0x7f46700699a0 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f4675cc2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f4675cc2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f4675cc2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4670063ed0 con 0x7f46700699a0 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f46577fe700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 0x7f465c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:29.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 28..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f4664048280 con 0x7f46700699a0 2024-09-18T10:11:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f466effd700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 0x7f465c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.694+0000 7f466effd700 10 monclient: get_auth_request con 0x7f465c035ac0 auth_method 0 2024-09-18T10:11:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.698+0000 7f466effd700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 0x7f465c037f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4660004860 tx=0x7f4660007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:29.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.706+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4664010d50 con 0x7f46700699a0 2024-09-18T10:11:29.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.706+0000 7f46577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:29.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.706+0000 7f46577fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:29.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f4675cc2700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f4675cc2700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f4675cc2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f4670067b10 con 0x7f46700699a0 2024-09-18T10:11:29.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7f4664010750 con 0x7f46700699a0 2024-09-18T10:11:29.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f46577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.870+0000 7f46577fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:29.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:29.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f4670122820 con 0x7f46700699a0 2024-09-18T10:11:29.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f46577fe700 1 -- 172.21.15.150:0/3974488235 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v58) v1 ==== 65+0+60 (secure 0 0 0) 0x7f4664010930 con 0x7f46700699a0 2024-09-18T10:11:29.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f46577fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:29.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f46577fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:29.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 msgr2=0x7f465c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 0x7f465c037f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4660004860 tx=0x7f4660007c10 comp rx=0 tx=0).stop 2024-09-18T10:11:29.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 10 monclient: shutdown 2024-09-18T10:11:29.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:29.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 msgr2=0x7f46701322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f466400c5d0 tx=0x7f4664007f60 comp rx=0 tx=0).stop 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 shutdown_connections 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f465c035ac0 0x7f465c037f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f467009e370 0x7f467011f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f467006cbc0 0x7f467011ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 --2- 172.21.15.150:0/3974488235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46700699a0 0x7f46701322b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 >> 172.21.15.150:0/3974488235 conn(0x7f4670071850 msgr2=0x7f467009ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 shutdown_connections 2024-09-18T10:11:29.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:29.978+0000 7f4675cc2700 1 -- 172.21.15.150:0/3974488235 wait complete. 2024-09-18T10:11:29.992 INFO:tasks.workunit.client.0.smithi150.stdout:foo 2024-09-18T10:11:29.992 INFO:tasks.workunit.client.0.smithi150.stdout:foo-ssd 2024-09-18T10:11:29.993 INFO:tasks.workunit.client.0.smithi150.stdout:foo-hdd 2024-09-18T10:11:29.993 INFO:tasks.workunit.client.0.smithi150.stdout:ec-foo 2024-09-18T10:11:29.993 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-18T10:11:30.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.090+0000 7f7838e7f700 10 monclient: get_monmap_and_config 2024-09-18T10:11:30.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.090+0000 7f7838e7f700 10 monclient: build_initial_monmap 2024-09-18T10:11:30.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.090+0000 7f7838e7f700 10 monclient: monmap: 2024-09-18T10:11:30.095 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:30.095 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:30.095444+0000 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:30.095444+0000 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:30.096 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 Processor -- start 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 -- start start 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient: init 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f783409d000 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f783409e780 0x7f78340a0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-c con 0x7f783409e780 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f78341379b0 0x7f7834139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-a con 0x7f78341379b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78340699a0 con 0x7f783409d000 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78340a1640 con 0x7f78341379b0 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783409e3d0 con 0x7f783409e780 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7838e7f700 10 monclient(hunting): authenticate will time out at 972.189383s 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f783409e780 0x7f78340a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f783259c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f783409d000 0x7f78340a0120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:56454/0 (socket says 172.21.15.150:56454) 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f783409e780 0x7f78340a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47168/0 (socket says 172.21.15.150:47168) 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f783259c700 1 -- 172.21.15.150:0/3813065566 learned_addr learned my addr 172.21.15.150:0/3813065566 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:30.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): get_auth_request con 0x7f783409e780 auth_method 0 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7832d9d700 1 --2- 172.21.15.150:0/3813065566 >> v2:172.21.15.142:3300/0 conn(0x7f78341379b0 0x7f7834139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7832d9d700 10 monclient(hunting): get_auth_request con 0x7f78341379b0 auth_method 0 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7832d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7832d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f7832d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient(hunting): handle_auth_done global_id 14596 payload 274 2024-09-18T10:11:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 -- 172.21.15.150:0/3813065566 >> v2:172.21.15.142:3300/0 conn(0x7f78341379b0 msgr2=0x7f7834139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 --2- 172.21.15.150:0/3813065566 >> v2:172.21.15.142:3300/0 conn(0x7f78341379b0 0x7f7834139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 -- 172.21.15.150:0/3813065566 >> v1:172.21.15.150:6789/0 conn(0x7f783409d000 legacy=0x7f78340a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 monclient: found mon.noname-c 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.094+0000 7f782bfff700 1 -- 172.21.15.150:0/3813065566 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783409ebf0 con 0x7f783409e780 2024-09-18T10:11:30.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f782bfff700 1 -- 172.21.15.150:0/3813065566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409e780 msgr2=0x7f78340a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f782bfff700 1 --2- 172.21.15.150:0/3813065566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409e780 0x7f78340a0830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f78340698a0 tx=0x7f781c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12e217bf278230f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 1 -- 172.21.15.150:0/3813065566 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c00b8c0 con 0x7f783409e780 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: dump: 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.106 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: _finish_auth 0 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 5 monclient: authenticate success, global_id 14596 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 1 -- 172.21.15.150:0/3813065566 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c00ba20 con 0x7f783409e780 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 4 set_mon_vals no callback set 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 1 -- 172.21.15.150:0/3813065566 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c00bdc0 con 0x7f783409e780 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7831d9b700 10 monclient: dump: 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: shutdown 2024-09-18T10:11:30.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- 172.21.15.150:0/3813065566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409e780 msgr2=0x7f78340a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- 172.21.15.150:0/3813065566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409e780 0x7f78340a0830 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f78340698a0 tx=0x7f781c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- 172.21.15.150:0/3813065566 shutdown_connections 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- 172.21.15.150:0/3813065566 >> v2:172.21.15.142:3300/0 conn(0x7f78341379b0 0x7f7834139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- 172.21.15.150:0/3813065566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409e780 0x7f78340a0830 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- 172.21.15.150:0/3813065566 >> 172.21.15.150:0/3813065566 conn(0x7f78340a5830 msgr2=0x7f78340a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- 172.21.15.150:0/3813065566 shutdown_connections 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- 172.21.15.150:0/3813065566 wait complete. 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: build_initial_monmap 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: monmap: 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:30.109 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 Processor -- start 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- start start 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: init 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-b con 0x7f783409d000 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 0x7f783412c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-c con 0x7f783409e780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): picked mon.noname-a con 0x7f78341379b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783409beb0 con 0x7f78341379b0 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783412fce0 con 0x7f783409d000 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783412fa00 con 0x7f783409e780 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f7838e7f700 10 monclient(hunting): authenticate will time out at 972.195727s 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f782bfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 msgr2=0x7f783412c670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:30.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f782bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 0x7f783412c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.098+0000 7f783259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:51938/0 (socket says 172.21.15.150:51938) 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47174/0 (socket says 172.21.15.150:47174) 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 1 -- 172.21.15.150:0/960941114 learned_addr learned my addr 172.21.15.150:0/960941114 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): get_auth_request con 0x7f78341379b0 auth_method 0 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): get_auth_request con 0x7f783409d000 auth_method 0 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient(hunting): handle_auth_done global_id 14599 payload 824 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 -- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 msgr2=0x7f783412c670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 0x7f783412c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 -- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 msgr2=0x7f783412cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient: _finish_hunting 0 2024-09-18T10:11:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 monclient: found mon.noname-b 2024-09-18T10:11:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c009420 con 0x7f783409d000 2024-09-18T10:11:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7832d9d700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 -- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 msgr2=0x7f783412c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f783259c700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f782400b4e0 tx=0x7f7824009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7824011840 con 0x7f783409d000 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: dump: 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 5 monclient: authenticate success, global_id 14599 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78240119a0 con 0x7f783409d000 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 10 monclient: _renew_subs 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783412f720 con 0x7f783409d000 2024-09-18T10:11:30.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7824011d60 con 0x7f783409d000 2024-09-18T10:11:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f782a7fc700 10 monclient: dump: 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7830d99700 4 set_mon_vals no callback set 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 10 monclient: _renew_subs 2024-09-18T10:11:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.102+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78341376c0 con 0x7f783409d000 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f7838e7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f7838e7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f7838e7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783412fe70 con 0x7f783409d000 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7824010590 con 0x7f783409d000 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f782a7fc700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 0x7f7814037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 28..58) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f7824048560 con 0x7f783409d000 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f782bfff700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 0x7f7814037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.106+0000 7f782bfff700 10 monclient: get_auth_request con 0x7f7814035a70 auth_method 0 2024-09-18T10:11:30.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.110+0000 7f782bfff700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 0x7f7814037f20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7834136e80 tx=0x7f781c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.114+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f782400f8e0 con 0x7f783409d000 2024-09-18T10:11:30.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.114+0000 7f782a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.114+0000 7f782a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:30.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.282+0000 7f7838e7f700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.282+0000 7f7838e7f700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.282+0000 7f7838e7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.282+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f783406e3e0 con 0x7f783409d000 2024-09-18T10:11:30.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.494+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 v59) v1 ==== 110+0+0 (secure 0 0 0) 0x7f7824015070 con 0x7f783409d000 2024-09-18T10:11:30.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.494+0000 7f782a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.494+0000 7f782a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f7838e7f700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f7838e7f700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f7838e7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f7834063ed0 con 0x7f783409d000 2024-09-18T10:11:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f782a7fc700 1 -- 172.21.15.150:0/960941114 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v59) v1 ==== 139+0+0 (secure 0 0 0) 0x7f782400f2e0 con 0x7f783409d000 2024-09-18T10:11:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f782a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.602+0000 7f782a7fc700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 msgr2=0x7f7814037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 0x7f7814037f20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7834136e80 tx=0x7f781c016000 comp rx=0 tx=0).stop 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 10 monclient: shutdown 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 msgr2=0x7f783412c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f782400b4e0 tx=0x7f7824009060 comp rx=0 tx=0).stop 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 shutdown_connections 2024-09-18T10:11:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7814035a70 0x7f7814037f20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f78341379b0 0x7f783412cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f783409e780 0x7f783412c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 --2- 172.21.15.150:0/960941114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f783409d000 0x7f783412c130 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 >> 172.21.15.150:0/960941114 conn(0x7f78340a5830 msgr2=0x7f783409b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 shutdown_connections 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.606+0000 7f7838e7f700 1 -- 172.21.15.150:0/960941114 wait complete. 2024-09-18T10:11:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'foo-asdf' 2024-09-18T10:11:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename foo foo-asdf 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: get_monmap_and_config 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: build_initial_monmap 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: monmap: 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:30.725380+0000 2024-09-18T10:11:30.725 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:30.725380+0000 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:30.726 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 Processor -- start 2024-09-18T10:11:30.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 -- start start 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: init 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 0x7f322c0a0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): picked mon.noname-b con 0x7f322c09d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f322c0a0740 0x7f322c0a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): picked mon.noname-a con 0x7f322c0a0740 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f322c136a80 0x7f322c138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): picked mon.noname-c con 0x7f322c136a80 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0a1070 con 0x7f322c0a0740 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0a1680 con 0x7f322c136a80 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c09d3d0 con 0x7f322c09d000 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3233064700 10 monclient(hunting): authenticate will time out at 972.818802s 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f322c136a80 0x7f322c138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f322c136a80 0x7f322c138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47186/0 (socket says 172.21.15.150:47186) 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 -- 172.21.15.150:0/1097013455 learned_addr learned my addr 172.21.15.150:0/1097013455 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): get_auth_request con 0x7f322c136a80 auth_method 0 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3230e00700 1 -- 172.21.15.150:0/1097013455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 msgr2=0x7f322c0a0200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3230e00700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 0x7f322c0a0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 1 --2- 172.21.15.150:0/1097013455 >> v2:172.21.15.142:3300/0 conn(0x7f322c0a0740 0x7f322c0a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): get_auth_request con 0x7f322c0a0740 auth_method 0 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient(hunting): handle_auth_done global_id 14602 payload 274 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 -- 172.21.15.150:0/1097013455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 msgr2=0x7f322c0a0200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 0x7f322c0a0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 -- 172.21.15.150:0/1097013455 >> v2:172.21.15.142:3300/0 conn(0x7f322c0a0740 msgr2=0x7f322c0a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 --2- 172.21.15.150:0/1097013455 >> v2:172.21.15.142:3300/0 conn(0x7f322c0a0740 0x7f322c0a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient: _finish_hunting 0 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 monclient: found mon.noname-c 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:30.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.722+0000 7f3231601700 1 -- 172.21.15.150:0/1097013455 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0a1e60 con 0x7f322c136a80 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3231601700 1 -- 172.21.15.150:0/1097013455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c136a80 msgr2=0x7f322c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3231601700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c136a80 0x7f322c138e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f322c0a37c0 tx=0x7f3224009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29c390b82596ab53 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 1 -- 172.21.15.150:0/1097013455 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400e070 con 0x7f322c136a80 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: dump: 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.737 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 5 monclient: authenticate success, global_id 14602 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 1 -- 172.21.15.150:0/1097013455 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322400bc30 con 0x7f322c136a80 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 4 set_mon_vals no callback set 2024-09-18T10:11:30.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 1 -- 172.21.15.150:0/1097013455 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32240092e0 con 0x7f322c136a80 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f322b7fe700 10 monclient: dump: 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.739 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 10 monclient: shutdown 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 -- 172.21.15.150:0/1097013455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c136a80 msgr2=0x7f322c138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c136a80 0x7f322c138e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f322c0a37c0 tx=0x7f3224009840 comp rx=0 tx=0).stop 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 -- 172.21.15.150:0/1097013455 shutdown_connections 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c136a80 0x7f322c138e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 --2- 172.21.15.150:0/1097013455 >> v2:172.21.15.142:3300/0 conn(0x7f322c0a0740 0x7f322c0a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 --2- 172.21.15.150:0/1097013455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c09d000 0x7f322c0a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 -- 172.21.15.150:0/1097013455 >> 172.21.15.150:0/1097013455 conn(0x7f322c0a5910 msgr2=0x7f322c0a5d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 -- 172.21.15.150:0/1097013455 shutdown_connections 2024-09-18T10:11:30.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.726+0000 7f3233064700 1 -- 172.21.15.150:0/1097013455 wait complete. 2024-09-18T10:11:30.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient: build_initial_monmap 2024-09-18T10:11:30.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient: monmap: 2024-09-18T10:11:30.745 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 Processor -- start 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 -- start start 2024-09-18T10:11:30.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient: init 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): picked mon.noname-b con 0x7f322c09d000 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): picked mon.noname-a con 0x7f322c0a0740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 0x7f322c12af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): picked mon.noname-c con 0x7f322c136a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c12e210 con 0x7f322c0a0740 2024-09-18T10:11:30.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c12d830 con 0x7f322c09d000 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c12d550 con 0x7f322c136a80 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3233064700 10 monclient(hunting): authenticate will time out at 972.826137s 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3231601700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 msgr2=0x7f322c12af20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3231601700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 0x7f322c12af20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47198/0 (socket says 172.21.15.150:47198) 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:51946/0 (socket says 172.21.15.150:51946) 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 -- 172.21.15.150:0/3305164326 learned_addr learned my addr 172.21.15.150:0/3305164326 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): get_auth_request con 0x7f322c09d000 auth_method 0 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): get_auth_request con 0x7f322c0a0740 auth_method 0 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient(hunting): handle_auth_done global_id 14605 payload 824 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 -- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 msgr2=0x7f322c12af20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 0x7f322c12af20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 -- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 msgr2=0x7f322c12a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient: _finish_hunting 0 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 monclient: found mon.noname-b 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:30.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f3230e00700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009ae0 con 0x7f322c09d000 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.730+0000 7f322bfff700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3230e00700 1 -- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 msgr2=0x7f322c06f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3230e00700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f322000b7e0 tx=0x7f3220009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220003e20 con 0x7f322c09d000 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: dump: 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.751 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 5 monclient: authenticate success, global_id 14605 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _renew_subs 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3220006150 con 0x7f322c09d000 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c12db10 con 0x7f322c09d000 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:30.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220006510 con 0x7f322c09d000 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3228ff9700 10 monclient: dump: 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:30.753 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _renew_subs 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c134b10 con 0x7f322c09d000 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f322affd700 4 set_mon_vals no callback set 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.734+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c06c3d0 con 0x7f322c09d000 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f32200103c0 con 0x7f322c09d000 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f3228ff9700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 0x7f3218037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 28..59) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f3220047c90 con 0x7f322c09d000 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f322bfff700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 0x7f3218037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:30.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f322bfff700 10 monclient: get_auth_request con 0x7f3218035a70 auth_method 0 2024-09-18T10:11:30.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.738+0000 7f322bfff700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 0x7f3218037f20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f322c06fae0 tx=0x7f321c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:30.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.742+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f322000e880 con 0x7f322c09d000 2024-09-18T10:11:30.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.742+0000 7f3228ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:30.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.742+0000 7f3228ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:30.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.914+0000 7f3233064700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.914+0000 7f3233064700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.914+0000 7f3233064700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:30.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.914+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f322c063ed0 con 0x7f322c09d000 2024-09-18T10:11:30.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.918+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v59) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3220017030 con 0x7f322c09d000 2024-09-18T10:11:30.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.918+0000 7f3228ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:30.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:30.918+0000 7f3228ff9700 10 monclient: _finish_command 2 = system:0 already renamed to 'foo-asdf' 2024-09-18T10:11:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.026+0000 7f3233064700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.026+0000 7f3233064700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.026+0000 7f3233064700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.026+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"} v 0) v1 -- 0x7f322c09f480 con 0x7f322c09d000 2024-09-18T10:11:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.030+0000 7f3228ff9700 1 -- 172.21.15.150:0/3305164326 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}]=0 already renamed to 'foo-asdf' v59) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3220017020 con 0x7f322c09d000 2024-09-18T10:11:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.030+0000 7f3228ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo", "dstname": "foo-asdf"}] 2024-09-18T10:11:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.030+0000 7f3228ff9700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo-asdf' 2024-09-18T10:11:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 msgr2=0x7f3218037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 0x7f3218037f20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f322c06fae0 tx=0x7f321c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 10 monclient: shutdown 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 msgr2=0x7f322c06f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f322000b7e0 tx=0x7f3220009220 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 shutdown_connections 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3218035a70 0x7f3218037f20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c136a80 0x7f322c12af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c0a0740 0x7f322c12a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 --2- 172.21.15.150:0/3305164326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c09d000 0x7f322c06f360 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 >> 172.21.15.150:0/3305164326 conn(0x7f322c0a5910 msgr2=0x7f322c098810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 shutdown_connections 2024-09-18T10:11:31.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.034+0000 7f3233064700 1 -- 172.21.15.150:0/3305164326 wait complete. 2024-09-18T10:11:31.038 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'foo-asdf' 2024-09-18T10:11:31.046 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename bar bar-asdf 2024-09-18T10:11:31.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: get_monmap_and_config 2024-09-18T10:11:31.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: build_initial_monmap 2024-09-18T10:11:31.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: monmap: 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:31.149762+0000 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:31.149762+0000 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:31.150 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:31.151 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:31.151 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 Processor -- start 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 -- start start 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: init 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f72300a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): picked mon.noname-b con 0x7f723009d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f723009e780 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:31.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72301379b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72300699a0 con 0x7f72301379b0 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723009e3d0 con 0x7f723009e780 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72300a1640 con 0x7f723009d000 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7237643700 10 monclient(hunting): authenticate will time out at 973.243449s 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7234bde700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f723009e780 0x7f72300a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:56462/0 (socket says 172.21.15.150:56462) 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7234bde700 1 -- 172.21.15.150:0/3874012914 learned_addr learned my addr 172.21.15.150:0/3874012914 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f72353df700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 msgr2=0x7f72300a0120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f7234bde700 1 -- 172.21.15.150:0/3874012914 >> v1:172.21.15.150:6789/0 conn(0x7f723009e780 legacy=0x7f72300a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-18T10:11:31.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.146+0000 7f72353df700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f72300a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 >> v1:172.21.15.142:6789/0 conn(0x7f72301379b0 legacy=0x7f7230139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268694251 0 0) 0x7f72300699a0 con 0x7f72301379b0 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): my global_id is 4893 2024-09-18T10:11:31.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f721c003570 con 0x7f72301379b0 2024-09-18T10:11:31.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409986676 0 0) 0x7f721c003570 con 0x7f72301379b0 2024-09-18T10:11:31.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 msgr2=0x7f72300a0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:31.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f72300a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 >> v1:172.21.15.150:6789/0 conn(0x7f723009e780 legacy=0x7f72300a0660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723009ebf0 con 0x7f72301379b0 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: _finish_auth 0 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f7228001b70 con 0x7f72301379b0 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7237643700 5 monclient: authenticate success, global_id 4893 2024-09-18T10:11:31.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7237643700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: dump: 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.159 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 >> v1:172.21.15.142:6789/0 conn(0x7f72301379b0 legacy=0x7f7230139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f721c003b10 0x7f721c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): picked mon.a con 0x7f721c003b10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f721c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): picked mon.c con 0x7f723009d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 0x7f721c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): picked mon.b con 0x7f721c004f30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c0056b0 con 0x7f721c003b10 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c00bfa0 con 0x7f721c004f30 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721c00c160 con 0x7f723009d000 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7227fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7237643700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7234bde700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 msgr2=0x7f721c004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7234bde700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f721c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 0x7f721c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 10 monclient(hunting): get_auth_request con 0x7f721c004f30 auth_method 0 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f7235be0700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f72353df700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f721c003b10 0x7f721c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f72353df700 10 monclient(hunting): get_auth_request con 0x7f721c003b10 auth_method 0 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f72353df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f72353df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.150+0000 7f72353df700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f72353df700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f72353df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f72353df700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient(hunting): handle_auth_done global_id 4893 payload 293 2024-09-18T10:11:31.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 msgr2=0x7f721c004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f723009d000 0x7f721c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f721c003b10 msgr2=0x7f721c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f721c003b10 0x7f721c003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient: _finish_hunting 0 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 monclient: found mon.b 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c003570 con 0x7f721c004f30 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 10 monclient: _finish_auth 0 2024-09-18T10:11:31.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:31.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 msgr2=0x7f721c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:31.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7235be0700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 0x7f721c00ba60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f723006ab50 tx=0x7f7228007e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=121266eca3fd1a0d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228005b90 con 0x7f721c004f30 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: dump: 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.169 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228005d10 con 0x7f721c004f30 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 1 -- 172.21.15.150:0/3874012914 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228007320 con 0x7f721c004f30 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 4 set_mon_vals no callback set 2024-09-18T10:11:31.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7227fff700 10 monclient: dump: 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.171 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 10 monclient: shutdown 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 1 -- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 msgr2=0x7f721c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.154+0000 7f7237643700 1 --2- 172.21.15.150:0/3874012914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f721c004f30 0x7f721c00ba60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f723006ab50 tx=0x7f7228007e70 comp rx=0 tx=0).stop 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 -- 172.21.15.150:0/3874012914 reap_dead start 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- 172.21.15.150:0/3874012914 shutdown_connections 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- 172.21.15.150:0/3874012914 >> 172.21.15.150:0/3874012914 conn(0x7f72300a5830 msgr2=0x7f72300a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- 172.21.15.150:0/3874012914 shutdown_connections 2024-09-18T10:11:31.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- 172.21.15.150:0/3874012914 wait complete. 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient: build_initial_monmap 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient: monmap: 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 Processor -- start 2024-09-18T10:11:31.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- start start 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient: init 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 0x7f723012ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): picked mon.noname-c con 0x7f721c004f30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 0x7f723012b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): picked mon.noname-a con 0x7f723012f1b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): picked mon.noname-b con 0x7f723012b900 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723009bbe0 con 0x7f723012f1b0 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723012d1a0 con 0x7f723012b900 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723012c290 con 0x7f721c004f30 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7237643700 10 monclient(hunting): authenticate will time out at 973.254838s 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f72353df700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 msgr2=0x7f723012ae80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f72353df700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 0x7f723012ae80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47216/0 (socket says 172.21.15.150:47216) 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 0x7f723012b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 -- 172.21.15.150:0/3729340969 learned_addr learned my addr 172.21.15.150:0/3729340969 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:31.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): get_auth_request con 0x7f723012b900 auth_method 0 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): get_auth_request con 0x7f723012f1b0 auth_method 0 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:31.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient(hunting): handle_auth_done global_id 14608 payload 824 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 -- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 msgr2=0x7f723012ae80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 0x7f723012ae80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 -- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 msgr2=0x7f723012b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 0x7f723012b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient: _finish_hunting 0 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 monclient: found mon.noname-b 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7235be0700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723012e7d0 con 0x7f723012b900 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.158+0000 7f7234bde700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 0x7f723012b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7235be0700 1 -- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 msgr2=0x7f723012bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7235be0700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7228001d50 tx=0x7f7228016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228005b90 con 0x7f723012b900 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: dump: 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: _finish_auth 0 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 5 monclient: authenticate success, global_id 14608 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72280072e0 con 0x7f723012b900 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 10 monclient: _renew_subs 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7230135aa0 con 0x7f723012b900 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228020620 con 0x7f723012b900 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7226ffd700 4 set_mon_vals no callback set 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f72257fa700 10 monclient: dump: 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.179 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 10 monclient: _renew_subs 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723012d590 con 0x7f723012b900 2024-09-18T10:11:31.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.162+0000 7f7237643700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f7237643700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f7237643700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7230063ed0 con 0x7f723012b900 2024-09-18T10:11:31.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7228020990 con 0x7f723012b900 2024-09-18T10:11:31.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f72257fa700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 0x7f7214037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 28..59) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f72280497e0 con 0x7f723012b900 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f72353df700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 0x7f7214037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.166+0000 7f72353df700 10 monclient: get_auth_request con 0x7f7214035a70 auth_method 0 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.170+0000 7f72353df700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 0x7f7214037f20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7220003980 tx=0x7f72200090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.174+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7228012400 con 0x7f723012b900 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.174+0000 7f72257fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.174+0000 7f72257fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:31.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.342+0000 7f7237643700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.342+0000 7f7237643700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.342+0000 7f7237643700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.342+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f723012d780 con 0x7f723012b900 2024-09-18T10:11:31.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.682+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 v60) v1 ==== 110+0+0 (secure 0 0 0) 0x7f7228018030 con 0x7f723012b900 2024-09-18T10:11:31.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.682+0000 7f72257fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.682+0000 7f72257fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.786+0000 7f7237643700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.786+0000 7f7237643700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.786+0000 7f7237643700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.786+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"} v 0) v1 -- 0x7f72300008d0 con 0x7f723012b900 2024-09-18T10:11:31.792 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f72257fa700 1 -- 172.21.15.150:0/3729340969 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}]=0 already renamed to 'bar-asdf' v60) v1 ==== 139+0+0 (secure 0 0 0) 0x7f7228018020 con 0x7f723012b900 2024-09-18T10:11:31.792 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f72257fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar", "dstname": "bar-asdf"}] 2024-09-18T10:11:31.792 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f72257fa700 10 monclient: _finish_command 3 = system:0 already renamed to 'bar-asdf' 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 msgr2=0x7f7214037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 0x7f7214037f20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7220003980 tx=0x7f72200090f0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 10 monclient: shutdown 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 msgr2=0x7f723012bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7228001d50 tx=0x7f7228016000 comp rx=0 tx=0).stop 2024-09-18T10:11:31.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 shutdown_connections 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7214035a70 0x7f7214037f20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f723012b900 0x7f723012bd50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f723012f1b0 0x7f723012b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 --2- 172.21.15.150:0/3729340969 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f721c004f30 0x7f723012ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 >> 172.21.15.150:0/3729340969 conn(0x7f72300a5830 msgr2=0x7f72301370a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 shutdown_connections 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.790+0000 7f7237643700 1 -- 172.21.15.150:0/3729340969 wait complete. 2024-09-18T10:11:31.796 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'bar-asdf' 2024-09-18T10:11:31.805 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:11:31.805 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-asdf 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: get_monmap_and_config 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: build_initial_monmap 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: monmap: 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:31.904976+0000 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:31.904976+0000 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.905 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:31.906 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:31.906 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:31.906 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:31.906 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:31.906 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 Processor -- start 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 -- start start 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: init 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f486c09cf90 0x7f486c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): picked mon.noname-a con 0x7f486c09cf90 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f486c09e480 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:31.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 0x7f486c139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): picked mon.noname-b con 0x7f486c0990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486c09e060 con 0x7f486c09e480 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f486c0a66c0 con 0x7f486c09cf90 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c09d750 con 0x7f486c0990a0 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f487446d700 10 monclient(hunting): authenticate will time out at 973.998727s 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872a0a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 msgr2=0x7f486c139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4871a08700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f486c09e480 0x7f486c0a07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:56464/0 (socket says 172.21.15.150:56464) 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4871a08700 1 -- 172.21.15.150:0/3709226535 learned_addr learned my addr 172.21.15.150:0/3709226535 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872a0a700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 0x7f486c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:31.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 --2- 172.21.15.150:0/3709226535 >> v2:172.21.15.142:3300/0 conn(0x7f486c09cf90 0x7f486c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): get_auth_request con 0x7f486c09cf90 auth_method 0 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient(hunting): handle_auth_done global_id 4899 payload 274 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 -- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 msgr2=0x7f486c139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 0x7f486c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 -- 172.21.15.150:0/3709226535 >> v1:172.21.15.150:6789/0 conn(0x7f486c09e480 legacy=0x7f486c0a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient: _finish_hunting 0 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 monclient: found mon.noname-a 2024-09-18T10:11:31.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:31.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:31.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.902+0000 7f4872209700 1 -- 172.21.15.150:0/3709226535 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c09d360 con 0x7f486c09cf90 2024-09-18T10:11:31.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4872209700 1 -- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c09cf90 msgr2=0x7f486c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:31.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4872209700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c09cf90 0x7f486c0a0290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f48680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81770939564bebf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 1 -- 172.21.15.150:0/3709226535 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486800ba70 con 0x7f486c09cf90 2024-09-18T10:11:31.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:31.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: dump: 2024-09-18T10:11:31.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.924 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: _finish_auth 0 2024-09-18T10:11:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 5 monclient: authenticate success, global_id 4899 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 1 -- 172.21.15.150:0/3709226535 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486800bbf0 con 0x7f486c09cf90 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 4 set_mon_vals no callback set 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 1 -- 172.21.15.150:0/3709226535 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868011420 con 0x7f486c09cf90 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f4871207700 10 monclient: dump: 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.928 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.928 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.928 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.928 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.928 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 10 monclient: shutdown 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:31.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 -- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c09cf90 msgr2=0x7f486c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c09cf90 0x7f486c0a0290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f48680091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 -- 172.21.15.150:0/3709226535 shutdown_connections 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c0990a0 0x7f486c139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 --2- 172.21.15.150:0/3709226535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c09cf90 0x7f486c0a0290 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 -- 172.21.15.150:0/3709226535 >> 172.21.15.150:0/3709226535 conn(0x7f486c0a58a0 msgr2=0x7f486c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 -- 172.21.15.150:0/3709226535 shutdown_connections 2024-09-18T10:11:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.906+0000 7f487446d700 1 -- 172.21.15.150:0/3709226535 wait complete. 2024-09-18T10:11:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient: build_initial_monmap 2024-09-18T10:11:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient: monmap: 2024-09-18T10:11:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:31.933 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:31.933 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 Processor -- start 2024-09-18T10:11:31.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 -- start start 2024-09-18T10:11:31.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient: init 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 0x7f486c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): picked mon.noname-a con 0x7f486c0990a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 0x7f486c142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): picked mon.noname-c con 0x7f486c09cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:31.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): picked mon.noname-b con 0x7f486c09e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:31.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0a41b0 con 0x7f486c0990a0 2024-09-18T10:11:31.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486c0919f0 con 0x7f486c09e480 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c08f080 con 0x7f486c09cf90 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f487446d700 10 monclient(hunting): authenticate will time out at 974.005338s 2024-09-18T10:11:31.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4871a08700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 msgr2=0x7f486c142020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:31.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4871a08700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 0x7f486c142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:31.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 0x7f486c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47218/0 (socket says 172.21.15.150:47218) 2024-09-18T10:11:31.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 0x7f486c141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:51988/0 (socket says 172.21.15.150:51988) 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 -- 172.21.15.150:0/12407795 learned_addr learned my addr 172.21.15.150:0/12407795 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): get_auth_request con 0x7f486c09e480 auth_method 0 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): get_auth_request con 0x7f486c0990a0 auth_method 0 2024-09-18T10:11:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872209700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient(hunting): handle_auth_done global_id 14611 payload 824 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 -- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 msgr2=0x7f486c142020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 0x7f486c142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 -- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 msgr2=0x7f486c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 0x7f486c141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient: _finish_hunting 0 2024-09-18T10:11:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 monclient: found mon.noname-b 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.910+0000 7f4872a0a700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868009420 con 0x7f486c09e480 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f4872a0a700 1 -- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 msgr2=0x7f486c142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f4872a0a700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f486400bb10 tx=0x7f48640097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486400ca50 con 0x7f486c09e480 2024-09-18T10:11:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: dump: 2024-09-18T10:11:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.945 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 5 monclient: authenticate success, global_id 14611 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _renew_subs 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486400cbd0 con 0x7f486c09e480 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c08eb70 con 0x7f486c09e480 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f4870a06700 4 set_mon_vals no callback set 2024-09-18T10:11:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _renew_subs 2024-09-18T10:11:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486c090980 con 0x7f486c09e480 2024-09-18T10:11:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486400cd80 con 0x7f486c09e480 2024-09-18T10:11:31.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 10 monclient: dump: 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486c08f800 con 0x7f486c09e480 2024-09-18T10:11:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4864012020 con 0x7f486c09e480 2024-09-18T10:11:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 0x7f4850037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 28..60) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f48640482e0 con 0x7f486c09e480 2024-09-18T10:11:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f4872209700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 0x7f4850037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.914+0000 7f4872209700 10 monclient: get_auth_request con 0x7f4850035a70 auth_method 0 2024-09-18T10:11:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.918+0000 7f4872209700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 0x7f4850037f20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f4868016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.922+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f486401d900 con 0x7f486c09e480 2024-09-18T10:11:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.922+0000 7f48627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:31.922+0000 7f48627fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:32.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f487446d700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f487446d700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f487446d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f486c0a0ee0 con 0x7f486c09e480 2024-09-18T10:11:32.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+70 (secure 0 0 0) 0x7f4864014340 con 0x7f486c09e480 2024-09-18T10:11:32.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f48627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.090+0000 7f48627fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f487446d700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f487446d700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f487446d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f486c063ed0 con 0x7f486c09e480 2024-09-18T10:11:32.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f48627fc700 1 -- 172.21.15.150:0/12407795 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+70 (secure 0 0 0) 0x7f486401d300 con 0x7f486c09e480 2024-09-18T10:11:32.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f48627fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:32.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.182+0000 7f48627fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:32.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 msgr2=0x7f4850037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 0x7f4850037f20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48680098e0 tx=0x7f4868016000 comp rx=0 tx=0).stop 2024-09-18T10:11:32.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 10 monclient: shutdown 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 msgr2=0x7f486c142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f486400bb10 tx=0x7f48640097d0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 shutdown_connections 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4850035a70 0x7f4850037f20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f486c09e480 0x7f486c142560 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f486c09cf90 0x7f486c142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 --2- 172.21.15.150:0/12407795 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f486c0990a0 0x7f486c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 >> 172.21.15.150:0/12407795 conn(0x7f486c0a58a0 msgr2=0x7f486c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 shutdown_connections 2024-09-18T10:11:32.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.186+0000 7f487446d700 1 -- 172.21.15.150:0/12407795 wait complete. 2024-09-18T10:11:32.196 INFO:tasks.workunit.client.0.smithi150.stdout:foo-asdf 2024-09-18T10:11:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:11:32.197 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep bar-asdf 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.294+0000 7f7950625700 10 monclient: get_monmap_and_config 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.294+0000 7f7950625700 10 monclient: build_initial_monmap 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.294+0000 7f7950625700 10 monclient: monmap: 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:32.299455+0000 2024-09-18T10:11:32.302 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:32.299455+0000 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 Processor -- start 2024-09-18T10:11:32.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 -- start start 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient: init 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79480a0120 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f79480a0c00 0x7f794809e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): picked mon.noname-a con 0x7f79480a0c00 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79481379b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794809dae0 con 0x7f79480a0120 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79480a1590 con 0x7f79481379b0 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79480a21d0 con 0x7f79480a0c00 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:32.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f7950625700 10 monclient(hunting): authenticate will time out at 974.392972s 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794ebc2700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f79481379b0 0x7f7948139d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:56478/0 (socket says 172.21.15.150:56478) 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794ebc2700 1 -- 172.21.15.150:0/2652820849 learned_addr learned my addr 172.21.15.150:0/2652820849 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 1 --2- 172.21.15.150:0/2652820849 >> v2:172.21.15.142:3300/0 conn(0x7f79480a0c00 0x7f794809e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): get_auth_request con 0x7f79480a0c00 auth_method 0 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:32.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:32.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794ebc2700 1 -- 172.21.15.150:0/2652820849 >> v1:172.21.15.150:6789/0 conn(0x7f79481379b0 legacy=0x7f7948139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:32.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794d3bf700 1 -- 172.21.15.150:0/2652820849 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998425642 0 0) 0x7f79480a1590 con 0x7f79481379b0 2024-09-18T10:11:32.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794d3bf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:32.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794d3bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:32.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794d3bf700 10 monclient(hunting): my global_id is 14614 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794d3bf700 1 -- 172.21.15.150:0/2652820849 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7934003570 con 0x7f79481379b0 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794e3c1700 1 -- 172.21.15.150:0/2652820849 >> v1:172.21.15.142:6789/0 conn(0x7f79480a0120 legacy=0x7f79480a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.298+0000 7f794dbc0700 10 monclient(hunting): handle_auth_done global_id 4905 payload 274 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 -- 172.21.15.150:0/2652820849 >> v1:172.21.15.150:6789/0 conn(0x7f79481379b0 legacy=0x7f7948139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 -- 172.21.15.150:0/2652820849 >> v1:172.21.15.142:6789/0 conn(0x7f79480a0120 legacy=0x7f79480a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 10 monclient: _finish_hunting 0 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 monclient: found mon.noname-a 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 -- 172.21.15.150:0/2652820849 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79480a1da0 con 0x7f79480a0c00 2024-09-18T10:11:32.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 -- 172.21.15.150:0/2652820849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0c00 msgr2=0x7f794809e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794dbc0700 1 --2- 172.21.15.150:0/2652820849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0c00 0x7f794809e7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f79380099e0 tx=0x7f79380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17210ab818099a90 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 1 -- 172.21.15.150:0/2652820849 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793800ba70 con 0x7f79480a0c00 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: dump: 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:32.314 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: _finish_auth 0 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 5 monclient: authenticate success, global_id 4905 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 1 -- 172.21.15.150:0/2652820849 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793800bbd0 con 0x7f79480a0c00 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:32.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 4 set_mon_vals no callback set 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 1 -- 172.21.15.150:0/2652820849 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938011420 con 0x7f79480a0c00 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f794d3bf700 10 monclient: dump: 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.316 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 10 monclient: shutdown 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 -- 172.21.15.150:0/2652820849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0c00 msgr2=0x7f794809e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 --2- 172.21.15.150:0/2652820849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0c00 0x7f794809e7a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f79380099e0 tx=0x7f79380091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 -- 172.21.15.150:0/2652820849 shutdown_connections 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 --2- 172.21.15.150:0/2652820849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0c00 0x7f794809e7a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 -- 172.21.15.150:0/2652820849 >> 172.21.15.150:0/2652820849 conn(0x7f79480a5830 msgr2=0x7f79480a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 -- 172.21.15.150:0/2652820849 shutdown_connections 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.302+0000 7f7950625700 1 -- 172.21.15.150:0/2652820849 wait complete. 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient: build_initial_monmap 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient: monmap: 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:32.318 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 Processor -- start 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 -- start start 2024-09-18T10:11:32.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient: init 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0120 0x7f79481305e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): picked mon.noname-a con 0x7f79480a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 0x7f794812c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): picked mon.noname-c con 0x7f79480a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): picked mon.noname-b con 0x7f79481379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794812df70 con 0x7f79480a0120 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948130b20 con 0x7f79481379b0 2024-09-18T10:11:32.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:32.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79481301c0 con 0x7f79480a0c00 2024-09-18T10:11:32.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:32.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f7950625700 10 monclient(hunting): authenticate will time out at 974.401816s 2024-09-18T10:11:32.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794dbc0700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 msgr2=0x7f794812c5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:11:32.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794dbc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 0x7f794812c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47224/0 (socket says 172.21.15.150:47224) 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 -- 172.21.15.150:0/96871560 learned_addr learned my addr 172.21.15.150:0/96871560 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): get_auth_request con 0x7f79481379b0 auth_method 0 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:32.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0120 0x7f79481305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:32.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): get_auth_request con 0x7f79480a0120 auth_method 0 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794e3c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient(hunting): handle_auth_done global_id 14617 payload 824 2024-09-18T10:11:32.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 -- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 msgr2=0x7f794812c5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 0x7f794812c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 -- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0120 msgr2=0x7f79481305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0120 0x7f79481305e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient: _finish_hunting 0 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 monclient: found mon.noname-b 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:32.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.306+0000 7f794ebc2700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938009420 con 0x7f79481379b0 2024-09-18T10:11:32.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f794ebc2700 1 -- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 msgr2=0x7f794812cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:32.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f794ebc2700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f794000cf60 tx=0x7f7940001c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794000ab70 con 0x7f79481379b0 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: dump: 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:32.329 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 5 monclient: authenticate success, global_id 14617 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794000acd0 con 0x7f79481379b0 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940007320 con 0x7f79481379b0 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:32.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f794cbbe700 4 set_mon_vals no callback set 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f793e7fc700 10 monclient: dump: 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:32.331 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 10 monclient: _renew_subs 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794812fee0 con 0x7f79481379b0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 10 monclient: _renew_subs 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.310+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794812f6b0 con 0x7f79481379b0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f7950625700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f794ebc2700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f794000cf60 tx=0x7f7940001c70 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f7940007750 mgrmap(e 5) v1 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f7950625700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f7950625700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794806e3e0 con 0x7f79481379b0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7940007750 con 0x7f79481379b0 2024-09-18T10:11:32.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f793e7fc700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 0x7f792c037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:32.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f794e3c1700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 0x7f792c037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:32.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f794e3c1700 10 monclient: get_auth_request con 0x7f792c035aa0 auth_method 0 2024-09-18T10:11:32.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.314+0000 7f794e3c1700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 0x7f792c037f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7944001bf0 tx=0x7f7944009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:32.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:32.322+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 28..60) v4 ==== 5151+0+0 (secure 0 0 0) 0x7f79400488c0 con 0x7f79481379b0 2024-09-18T10:11:33.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.142+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f794000ae40 con 0x7f79481379b0 2024-09-18T10:11:33.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.142+0000 7f793e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:33.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.142+0000 7f793e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:33.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.314+0000 7f7950625700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.314+0000 7f7950625700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.314+0000 7f7950625700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:33.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.314+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f7948063ed0 con 0x7f79481379b0 2024-09-18T10:11:33.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.318+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+70 (secure 0 0 0) 0x7f7940010af0 con 0x7f79481379b0 2024-09-18T10:11:33.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.318+0000 7f793e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.318+0000 7f793e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:33.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:33.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f79480008d0 con 0x7f79481379b0 2024-09-18T10:11:33.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f793e7fc700 1 -- 172.21.15.150:0/96871560 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v60) v1 ==== 65+0+70 (secure 0 0 0) 0x7f794001a360 con 0x7f79481379b0 2024-09-18T10:11:33.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f793e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:33.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f793e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:33.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 msgr2=0x7f792c037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:33.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 0x7f792c037f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7944001bf0 tx=0x7f7944009040 comp rx=0 tx=0).stop 2024-09-18T10:11:33.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 10 monclient: shutdown 2024-09-18T10:11:33.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:33.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 msgr2=0x7f794812cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f794000cf60 tx=0x7f7940001c70 comp rx=0 tx=0).stop 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 shutdown_connections 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f792c035aa0 0x7f792c037f50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79481379b0 0x7f794812cae0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79480a0c00 0x7f794812c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 --2- 172.21.15.150:0/96871560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79480a0120 0x7f79481305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 >> 172.21.15.150:0/96871560 conn(0x7f79480a5830 msgr2=0x7f79480a40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 shutdown_connections 2024-09-18T10:11:33.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:33.410+0000 7f7950625700 1 -- 172.21.15.150:0/96871560 wait complete. 2024-09-18T10:11:33.424 INFO:tasks.workunit.client.0.smithi150.stdout:bar-asdf 2024-09-18T10:11:33.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm foo 2024-09-18T10:11:33.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'does not exist' 2024-09-18T10:11:33.828 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:33.698+0000 7f7077fff700 1 -- 172.21.15.150:0/1968628532 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f7088046b80 con 0x7f707c03ed20 2024-09-18T10:11:33.829 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:33.698+0000 7f7077fff700 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-18T10:11:33.829 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:33.806+0000 7f7077fff700 1 -- 172.21.15.150:0/1968628532 <== mon.1 v2:172.21.15.150:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f7088046d60 con 0x7f707c03ed20 2024-09-18T10:11:33.829 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:33.806+0000 7f7077fff700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T10:11:33.829 INFO:tasks.workunit.client.0.smithi150.stdout:rule foo does not exist 2024-09-18T10:11:33.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm bar 2024-09-18T10:11:33.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'does not exist' 2024-09-18T10:11:34.151 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:11:34.151 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:11:34.151 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:11:34.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:11:34.154 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:11:34.154 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T10:11:34.155 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T10:11:34.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:11:50.747 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:34.102+0000 7f4eab7fe700 1 -- 172.21.15.150:0/2190327651 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f4eac02b070 con 0x7f4eb41307a0 2024-09-18T10:11:50.747 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:34.102+0000 7f4eab7fe700 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-18T10:11:50.747 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:50.722+0000 7f4eab7fe700 1 -- 172.21.15.150:0/2190327651 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v60) v1 ==== 103+0+0 (secure 0 0 0) 0x7f4eac034070 con 0x7f4eb41307a0 2024-09-18T10:11:50.748 INFO:tasks.workunit.client.0.smithi150.stdout:2024-09-18T10:11:50.722+0000 7f4eab7fe700 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-18T10:11:50.748 INFO:tasks.workunit.client.0.smithi150.stdout:rule bar does not exist 2024-09-18T10:11:50.748 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-18T10:11:50.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: get_monmap_and_config 2024-09-18T10:11:50.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: build_initial_monmap 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: monmap: 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:50.857845+0000 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:50.857845+0000 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:50.858 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:50.859 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 Processor -- start 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 -- start start 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: init 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:11:50.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd3809d970 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd38131750 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdd38135250 0x7fdd38137630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-c con 0x7fdd38135250 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3809b0a0 con 0x7fdd38131750 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3809dd40 con 0x7fdd3809d970 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd380982f0 con 0x7fdd38135250 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3fd91700 10 monclient(hunting): authenticate will time out at 992.951281s 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdd38135250 0x7fdd38137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdd38135250 0x7fdd38137630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49458/0 (socket says 172.21.15.150:49458) 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 1 -- 172.21.15.150:0/1229385642 learned_addr learned my addr 172.21.15.150:0/1229385642 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:50.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): get_auth_request con 0x7fdd38135250 auth_method 0 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3db2d700 1 --1- 172.21.15.150:0/1229385642 >> v1:172.21.15.150:6789/0 conn(0x7fdd3809d970 0x7fdd380955b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39592/0 (socket says 172.21.15.150:39592) 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:50.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.854+0000 7fdd3e32e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3db2d700 1 -- 172.21.15.150:0/1229385642 >> v1:172.21.15.150:6789/0 conn(0x7fdd3809d970 legacy=0x7fdd380955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 10 monclient(hunting): handle_auth_done global_id 14632 payload 274 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 1 -- 172.21.15.150:0/1229385642 >> v1:172.21.15.150:6789/0 conn(0x7fdd3809d970 legacy=0x7fdd380955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 1 -- 172.21.15.150:0/1229385642 >> v1:172.21.15.142:6789/0 conn(0x7fdd38131750 legacy=0x7fdd38133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 10 monclient: _finish_hunting 0 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 1 monclient: found mon.noname-c 2024-09-18T10:11:50.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:50.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:50.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.858+0000 7fdd3e32e700 1 -- 172.21.15.150:0/1229385642 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3809c560 con 0x7fdd38135250 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3e32e700 1 -- 172.21.15.150:0/1229385642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38135250 msgr2=0x7fdd38137630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3e32e700 1 --2- 172.21.15.150:0/1229385642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38135250 0x7fdd38137630 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009fa0 tx=0x7fdd28009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51d2a8787350cc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 1 -- 172.21.15.150:0/1229385642 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800e070 con 0x7fdd38135250 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: dump: 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:50.865 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: _finish_auth 0 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 1 -- 172.21.15.150:0/1229385642 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2800bd10 con 0x7fdd38135250 2024-09-18T10:11:50.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 1 -- 172.21.15.150:0/1229385642 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd28012660 con 0x7fdd38135250 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3cb2b700 10 monclient: dump: 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:50.867 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:50.868 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 5 monclient: authenticate success, global_id 14632 2024-09-18T10:11:50.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:50.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 4 set_mon_vals no callback set 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 10 monclient: shutdown 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 -- 172.21.15.150:0/1229385642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38135250 msgr2=0x7fdd38137630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/1229385642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38135250 0x7fdd38137630 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009fa0 tx=0x7fdd28009840 comp rx=0 tx=0).stop 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 -- 172.21.15.150:0/1229385642 shutdown_connections 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/1229385642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd38135250 0x7fdd38137630 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 -- 172.21.15.150:0/1229385642 >> 172.21.15.150:0/1229385642 conn(0x7fdd380a15e0 msgr2=0x7fdd380a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 -- 172.21.15.150:0/1229385642 shutdown_connections 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 1 -- 172.21.15.150:0/1229385642 wait complete. 2024-09-18T10:11:50.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 10 monclient: build_initial_monmap 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.862+0000 7fdd3fd91700 10 monclient: monmap: 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 Processor -- start 2024-09-18T10:11:50.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 -- start start 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient: init 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-b con 0x7fdd3809d970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-c con 0x7fdd38131750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd38135250 0x7fdd3806c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): picked mon.noname-a con 0x7fdd38135250 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3806e180 con 0x7fdd38135250 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:50.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd380984e0 con 0x7fdd3809d970 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd380a10b0 con 0x7fdd38131750 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3fd91700 10 monclient(hunting): authenticate will time out at 992.962180s 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49470/0 (socket says 172.21.15.150:49470) 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 -- 172.21.15.150:0/3304311385 learned_addr learned my addr 172.21.15.150:0/3304311385 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): get_auth_request con 0x7fdd3809d970 auth_method 0 2024-09-18T10:11:50.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 10 monclient(hunting): get_auth_request con 0x7fdd38131750 auth_method 0 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:50.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 msgr2=0x7fdd38094830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 msgr2=0x7fdd38094830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3d32c700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient(hunting): handle_auth_done global_id 14638 payload 824 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 msgr2=0x7fdd38094830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd38135250 msgr2=0x7fdd3806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd38135250 0x7fdd3806c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient: _finish_hunting 0 2024-09-18T10:11:50.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 monclient: found mon.noname-b 2024-09-18T10:11:50.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:50.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:50.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.866+0000 7fdd3db2d700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28009ae0 con 0x7fdd3809d970 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3db2d700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 msgr2=0x7fdd380942f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3db2d700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000c7f0 tx=0x7fdd30007eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30007510 con 0x7fdd3809d970 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: dump: 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.880 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd300076b0 con 0x7fdd3809d970 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 5 monclient: authenticate success, global_id 14638 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004a80 con 0x7fdd3809d970 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _renew_subs 2024-09-18T10:11:50.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd38126da0 con 0x7fdd3809d970 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd377fe700 4 set_mon_vals no callback set 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd35ffb700 10 monclient: dump: 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:50.882 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _renew_subs 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3806dc40 con 0x7fdd3809d970 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.870+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd380a0b60 con 0x7fdd3809d970 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdd30013400 con 0x7fdd3809d970 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd35ffb700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 0x7fdd24037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 35..60) v4 ==== 5151+0+0 (secure 0 0 0) 0x7fdd30047890 con 0x7fdd3809d970 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd3d32c700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 0x7fdd24037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:50.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd3d32c700 10 monclient: get_auth_request con 0x7fdd24035ac0 auth_method 0 2024-09-18T10:11:50.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.874+0000 7fdd3d32c700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 0x7fdd24037f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806e030 tx=0x7fdd2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:50.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.878+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdd30004e30 con 0x7fdd3809d970 2024-09-18T10:11:50.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.878+0000 7fdd35ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:50.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:50.878+0000 7fdd35ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.050+0000 7fdd3fd91700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.050+0000 7fdd3fd91700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.050+0000 7fdd3fd91700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.050+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7fdd38063ed0 con 0x7fdd3809d970 2024-09-18T10:11:51.792 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.790+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 v61) v1 ==== 110+0+0 (secure 0 0 0) 0x7fdd3000e290 con 0x7fdd3809d970 2024-09-18T10:11:51.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.790+0000 7fdd35ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.790+0000 7fdd35ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:51.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd3fd91700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd3fd91700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd3fd91700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:51.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7fdd3806e520 con 0x7fdd3809d970 2024-09-18T10:11:51.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd35ffb700 1 -- 172.21.15.150:0/3304311385 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v61) v1 ==== 134+0+0 (secure 0 0 0) 0x7fdd30019670 con 0x7fdd3809d970 2024-09-18T10:11:51.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd35ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:51.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.898+0000 7fdd35ffb700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 msgr2=0x7fdd24037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 0x7fdd24037f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd3806e030 tx=0x7fdd2c008040 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 10 monclient: shutdown 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 msgr2=0x7fdd380942f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000c7f0 tx=0x7fdd30007eb0 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 shutdown_connections 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdd24035ac0 0x7fdd24037f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd38135250 0x7fdd3806c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd38131750 0x7fdd38094830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 --2- 172.21.15.150:0/3304311385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd3809d970 0x7fdd380942f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:51.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 >> 172.21.15.150:0/3304311385 conn(0x7fdd380a15e0 msgr2=0x7fdd38130e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:51.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 shutdown_connections 2024-09-18T10:11:51.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:51.902+0000 7fdd3fd91700 1 -- 172.21.15.150:0/3304311385 wait complete. 2024-09-18T10:11:51.906 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'foo' 2024-09-18T10:11:51.914 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename foo-asdf foo 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: get_monmap_and_config 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: build_initial_monmap 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: monmap: 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:52.014153+0000 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:52.014153+0000 2024-09-18T10:11:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:52.015 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 Processor -- start 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 -- start start 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: init 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28b409bca0 0x7f28b409a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-c con 0x7f28b409bca0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f28b409e140 0x7f28b409c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-a con 0x7f28b409e140 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-b con 0x7f28b409c5b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b4099b90 con 0x7f28b409e140 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b409edd0 con 0x7f28b409bca0 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b4064bf0 con 0x7f28b409c5b0 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28bb245700 10 monclient(hunting): authenticate will time out at 994.107568s 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28b409bca0 0x7f28b409a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f28b409bca0 0x7f28b409a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49482/0 (socket says 172.21.15.150:49482) 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 1 -- 172.21.15.150:0/1429710843 learned_addr learned my addr 172.21.15.150:0/1429710843 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 10 monclient(hunting): get_auth_request con 0x7f28b409bca0 auth_method 0 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.010+0000 7f28b8fe1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 10 monclient(hunting): get_auth_request con 0x7f28b409c5b0 auth_method 0 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:52.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 1 -- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b409ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 1 -- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b409ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient(hunting): handle_auth_done global_id 14641 payload 274 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 -- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b409ce90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 -- 172.21.15.150:0/1429710843 >> v2:172.21.15.142:3300/0 conn(0x7f28b409e140 msgr2=0x7f28b409c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 --2- 172.21.15.150:0/1429710843 >> v2:172.21.15.142:3300/0 conn(0x7f28b409e140 0x7f28b409c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient: _finish_hunting 0 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 monclient: found mon.noname-c 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 -- 172.21.15.150:0/1429710843 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b409e810 con 0x7f28b409bca0 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 -- 172.21.15.150:0/1429710843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409bca0 msgr2=0x7f28b409a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b8fe1700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409bca0 0x7f28b409a410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f28b4072460 tx=0x7f28a8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba051bad62a694a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 1 -- 172.21.15.150:0/1429710843 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a800ba70 con 0x7f28b409bca0 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: dump: 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.024 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28bb245700 5 monclient: authenticate success, global_id 14641 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28bb245700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 1 -- 172.21.15.150:0/1429710843 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a800bbd0 con 0x7f28b409bca0 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28bb245700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:52.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28bb245700 4 set_mon_vals no callback set 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 1 -- 172.21.15.150:0/1429710843 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8011420 con 0x7f28b409bca0 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.014+0000 7f28b37fe700 10 monclient: dump: 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.026 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: shutdown 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- 172.21.15.150:0/1429710843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409bca0 msgr2=0x7f28b409a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409bca0 0x7f28b409a410 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f28b4072460 tx=0x7f28a8009250 comp rx=0 tx=0).stop 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- 172.21.15.150:0/1429710843 shutdown_connections 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b409ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- 172.21.15.150:0/1429710843 >> v2:172.21.15.142:3300/0 conn(0x7f28b409e140 0x7f28b409c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- 172.21.15.150:0/1429710843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409bca0 0x7f28b409a410 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- 172.21.15.150:0/1429710843 >> 172.21.15.150:0/1429710843 conn(0x7f28b40a24a0 msgr2=0x7f28b40a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- 172.21.15.150:0/1429710843 shutdown_connections 2024-09-18T10:11:52.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- 172.21.15.150:0/1429710843 wait complete. 2024-09-18T10:11:52.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: build_initial_monmap 2024-09-18T10:11:52.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: monmap: 2024-09-18T10:11:52.031 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:52.031 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:52.031 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 Processor -- start 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- start start 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: init 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:52.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28b409bca0 0x7f28b41417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-a con 0x7f28b409bca0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-c con 0x7f28b409c5b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): picked mon.noname-b con 0x7f28b409e140 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b408efa0 con 0x7f28b409bca0 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40a06d0 con 0x7f28b409e140 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b41426f0 con 0x7f28b409c5b0 2024-09-18T10:11:52.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28bb245700 10 monclient(hunting): authenticate will time out at 994.113850s 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49492/0 (socket says 172.21.15.150:49492) 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 -- 172.21.15.150:0/1893028986 learned_addr learned my addr 172.21.15.150:0/1893028986 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): get_auth_request con 0x7f28b409e140 auth_method 0 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 10 monclient(hunting): get_auth_request con 0x7f28b409c5b0 auth_method 0 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b4094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b4094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b3fff700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient(hunting): handle_auth_done global_id 14644 payload 824 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 msgr2=0x7f28b4094770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28b409bca0 msgr2=0x7f28b41417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28b409bca0 0x7f28b41417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient: _finish_hunting 0 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 monclient: found mon.noname-b 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.018+0000 7f28b97e2700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a80094f0 con 0x7f28b409e140 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b97e2700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 msgr2=0x7f28b408d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b97e2700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f28ac007f50 tx=0x7f28ac009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac00ee20 con 0x7f28b409e140 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: dump: 2024-09-18T10:11:52.036 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac014420 con 0x7f28b409e140 2024-09-18T10:11:52.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 5 monclient: authenticate success, global_id 14644 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _renew_subs 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b4090980 con 0x7f28b409e140 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac0147e0 con 0x7f28b409e140 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 10 monclient: dump: 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.038 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b27fc700 4 set_mon_vals no callback set 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _renew_subs 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b4141ed0 con 0x7f28b409e140 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b409fe50 con 0x7f28b409e140 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.022+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f28ac014bd0 con 0x7f28b409e140 2024-09-18T10:11:52.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.026+0000 7f28b0ff9700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 0x7f28a0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.026+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 35..61) v4 ==== 5146+0+0 (secure 0 0 0) 0x7f28ac048c30 con 0x7f28b409e140 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.026+0000 7f28b8fe1700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 0x7f28a0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.026+0000 7f28b8fe1700 10 monclient: get_auth_request con 0x7f28a0035ac0 auth_method 0 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.026+0000 7f28b8fe1700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 0x7f28a0037f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f28b4072460 tx=0x7f28a8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.030+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f28ac0098c0 con 0x7f28b409e140 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.030+0000 7f28b0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.030+0000 7f28b0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:52.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28bb245700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28bb245700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28bb245700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f28b4063ed0 con 0x7f28b409e140 2024-09-18T10:11:52.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v61) v1 ==== 134+0+0 (secure 0 0 0) 0x7f28ac0122e0 con 0x7f28b409e140 2024-09-18T10:11:52.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28b0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.198+0000 7f28b0ff9700 10 monclient: _finish_command 2 = system:0 already renamed to 'foo' 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28bb245700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28bb245700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28bb245700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"} v 0) v1 -- 0x7f28b4090cc0 con 0x7f28b409e140 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28b0ff9700 1 -- 172.21.15.150:0/1893028986 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}]=0 already renamed to 'foo' v61) v1 ==== 134+0+0 (secure 0 0 0) 0x7f28ac0124c0 con 0x7f28b409e140 2024-09-18T10:11:52.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28b0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "foo-asdf", "dstname": "foo"}] 2024-09-18T10:11:52.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.302+0000 7f28b0ff9700 10 monclient: _finish_command 3 = system:0 already renamed to 'foo' 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 msgr2=0x7f28a0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 0x7f28a0037f70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f28b4072460 tx=0x7f28a8017000 comp rx=0 tx=0).stop 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 10 monclient: shutdown 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 msgr2=0x7f28b408d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f28ac007f50 tx=0x7f28ac009830 comp rx=0 tx=0).stop 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 shutdown_connections 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f28a0035ac0 0x7f28a0037f70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28b409e140 0x7f28b408d400 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f28b409c5b0 0x7f28b4094770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 --2- 172.21.15.150:0/1893028986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28b409bca0 0x7f28b41417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 >> 172.21.15.150:0/1893028986 conn(0x7f28b40a24a0 msgr2=0x7f28b4095900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:52.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 shutdown_connections 2024-09-18T10:11:52.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.306+0000 7f28bb245700 1 -- 172.21.15.150:0/1893028986 wait complete. 2024-09-18T10:11:52.311 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'foo' 2024-09-18T10:11:52.320 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rename bar-asdf bar 2024-09-18T10:11:52.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 10 monclient: get_monmap_and_config 2024-09-18T10:11:52.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 10 monclient: build_initial_monmap 2024-09-18T10:11:52.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 10 monclient: monmap: 2024-09-18T10:11:52.422 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:52.422722+0000 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:52.422722+0000 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:52.423 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 1 Processor -- start 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 1 -- start start 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.418+0000 7ffb5f384700 10 monclient: init 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:11:52.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb58130120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-b con 0x7ffb580a3ca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb5806ee00 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffb58135370 0x7ffb58137750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-a con 0x7ffb58135370 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb58130f00 con 0x7ffb5806ee00 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb580a4460 con 0x7ffb58135370 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5806d940 con 0x7ffb580a3ca0 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5f384700 10 monclient(hunting): authenticate will time out at 994.516994s 2024-09-18T10:11:52.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5db81700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ffb5806ee00 0x7ffb5806f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39594/0 (socket says 172.21.15.150:39594) 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5db81700 1 -- 172.21.15.150:0/580676630 learned_addr learned my addr 172.21.15.150:0/580676630 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb58130120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 10 monclient(hunting): get_auth_request con 0x7ffb580a3ca0 auth_method 0 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 msgr2=0x7ffb58130120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 msgr2=0x7ffb58130120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb58130120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:52.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5e382700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb58130120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:52.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5db81700 1 -- 172.21.15.150:0/580676630 >> v1:172.21.15.150:6789/0 conn(0x7ffb5806ee00 legacy=0x7ffb5806f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065526074 0 0) 0x7ffb58130f00 con 0x7ffb5806ee00 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5d380700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5d380700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5d380700 10 monclient(hunting): my global_id is 14647 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.422+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb44003570 con 0x7ffb5806ee00 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880214165 0 0) 0x7ffb44003570 con 0x7ffb5806ee00 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 msgr2=0x7ffb58130120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb58130120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 >> v2:172.21.15.142:3300/0 conn(0x7ffb58135370 msgr2=0x7ffb58137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 --2- 172.21.15.150:0/580676630 >> v2:172.21.15.142:3300/0 conn(0x7ffb58135370 0x7ffb58137750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: _finish_hunting 0 2024-09-18T10:11:52.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:11:52.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:52.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:11:52.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5806d5f0 con 0x7ffb5806ee00 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: _finish_auth 0 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 5 monclient: authenticate success, global_id 14647 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ffb4c0017a0 con 0x7ffb5806ee00 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: dump: 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.437 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 >> v1:172.21.15.150:6789/0 conn(0x7ffb5806ee00 legacy=0x7ffb5806f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb44003c00 0x7ffb44004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): picked mon.a con 0x7ffb44003c00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb44004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): picked mon.c con 0x7ffb580a3ca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 0x7ffb4400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): picked mon.b con 0x7ffb44004f50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5806d940 con 0x7ffb44003c00 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4400bfa0 con 0x7ffb44004f50 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4400c0e0 con 0x7ffb580a3ca0 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 0x7ffb4400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): get_auth_request con 0x7ffb44004f50 auth_method 0 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:11:52.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5db81700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb44004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5db81700 10 monclient(hunting): get_auth_request con 0x7ffb580a3ca0 auth_method 0 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5db81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5db81700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5db81700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_done global_id 14647 payload 293 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 msgr2=0x7ffb44004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb580a3ca0 0x7ffb44004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb44003c00 msgr2=0x7ffb44004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb44003c00 0x7ffb44004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient: _finish_hunting 0 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 monclient: found mon.b 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 -- 172.21.15.150:0/580676630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb44003570 con 0x7ffb44004f50 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 10 monclient: _finish_auth 0 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 msgr2=0x7ffb4400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 0x7ffb4400ba60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009df0 tx=0x7ffb500095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12ed96820cb25ca5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000bda0 con 0x7ffb44004f50 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: dump: 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb50014440 con 0x7ffb44004f50 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5f384700 4 set_mon_vals no callback set 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 1 -- 172.21.15.150:0/580676630 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50014800 con 0x7ffb44004f50 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.426+0000 7ffb5d380700 10 monclient: dump: 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: shutdown 2024-09-18T10:11:52.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 msgr2=0x7ffb4400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 --2- 172.21.15.150:0/580676630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb44004f50 0x7ffb4400ba60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffb50009df0 tx=0x7ffb500095f0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5eb83700 1 -- 172.21.15.150:0/580676630 reap_dead start 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- 172.21.15.150:0/580676630 shutdown_connections 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- 172.21.15.150:0/580676630 >> 172.21.15.150:0/580676630 conn(0x7ffb58071830 msgr2=0x7ffb580a3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- 172.21.15.150:0/580676630 shutdown_connections 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- 172.21.15.150:0/580676630 wait complete. 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: build_initial_monmap 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: monmap: 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:52.445 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 Processor -- start 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- start start 2024-09-18T10:11:52.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: init 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb581312f0 0x7ffb581316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-a con 0x7ffb581312f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-c con 0x7ffb58131c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): picked mon.noname-b con 0x7ffb58143800 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:52.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb581342d0 con 0x7ffb581312f0 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58132510 con 0x7ffb58143800 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb58134800 con 0x7ffb58131c00 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.430+0000 7ffb5f384700 10 monclient(hunting): authenticate will time out at 994.527800s 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49500/0 (socket says 172.21.15.150:49500) 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 -- 172.21.15.150:0/1173004698 learned_addr learned my addr 172.21.15.150:0/1173004698 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): get_auth_request con 0x7ffb58143800 auth_method 0 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 10 monclient(hunting): get_auth_request con 0x7ffb58131c00 auth_method 0 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:52.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 msgr2=0x7ffb58131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 msgr2=0x7ffb58131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5db81700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient(hunting): handle_auth_done global_id 14650 payload 824 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 msgr2=0x7ffb58131fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb581312f0 msgr2=0x7ffb581316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb581312f0 0x7ffb581316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient: _finish_hunting 0 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 monclient: found mon.noname-b 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:52.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58130aa0 con 0x7ffb58143800 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 msgr2=0x7ffb58145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5eb83700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb44005750 tx=0x7ffb50017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000c640 con 0x7ffb58143800 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: dump: 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.455 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 5 monclient: authenticate success, global_id 14650 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5000c7a0 con 0x7ffb58143800 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 10 monclient: _renew_subs 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5812e0e0 con 0x7ffb58143800 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000cb60 con 0x7ffb58143800 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb4a7fc700 10 monclient: dump: 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:52.457 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 10 monclient: _renew_subs 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5cb7f700 4 set_mon_vals no callback set 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.434+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb58132710 con 0x7ffb58143800 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5f384700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5f384700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5f384700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb58133e60 con 0x7ffb58143800 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffb5001b070 con 0x7ffb58143800 2024-09-18T10:11:52.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb4a7fc700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 0x7ffb3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 35..61) v4 ==== 5146+0+0 (secure 0 0 0) 0x7ffb5004aaa0 con 0x7ffb58143800 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5e382700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 0x7ffb3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.438+0000 7ffb5e382700 10 monclient: get_auth_request con 0x7ffb3c035ac0 auth_method 0 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.442+0000 7ffb5e382700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 0x7ffb3c037f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006f00 tx=0x7ffb54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.446+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ffb5001d970 con 0x7ffb58143800 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.446+0000 7ffb4a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:52.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.446+0000 7ffb4a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:52.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.622+0000 7ffb5f384700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.622+0000 7ffb5f384700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.622+0000 7ffb5f384700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.622+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7ffb5804bbf0 con 0x7ffb58143800 2024-09-18T10:11:52.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.798+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 v62) v1 ==== 110+0+0 (secure 0 0 0) 0x7ffb50019070 con 0x7ffb58143800 2024-09-18T10:11:52.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.798+0000 7ffb4a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.798+0000 7ffb4a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:52.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb5f384700 10 start_mon_command cmd=[{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb5f384700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb5f384700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:52.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"} v 0) v1 -- 0x7ffb5804bbf0 con 0x7ffb58143800 2024-09-18T10:11:52.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb4a7fc700 1 -- 172.21.15.150:0/1173004698 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}]=0 already renamed to 'bar' v62) v1 ==== 134+0+0 (secure 0 0 0) 0x7ffb50030380 con 0x7ffb58143800 2024-09-18T10:11:52.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb4a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rename", "srcname": "bar-asdf", "dstname": "bar"}] 2024-09-18T10:11:52.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.906+0000 7ffb4a7fc700 10 monclient: _finish_command 3 = system:0 already renamed to 'bar' 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 msgr2=0x7ffb3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 0x7ffb3c037f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006f00 tx=0x7ffb54008040 comp rx=0 tx=0).stop 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 10 monclient: shutdown 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 msgr2=0x7ffb58145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffb44005750 tx=0x7ffb50017000 comp rx=0 tx=0).stop 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 shutdown_connections 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb3c035ac0 0x7ffb3c037f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58143800 0x7ffb58145be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb58131c00 0x7ffb58131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 --2- 172.21.15.150:0/1173004698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb581312f0 0x7ffb581316c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 >> 172.21.15.150:0/1173004698 conn(0x7ffb58071830 msgr2=0x7ffb5812e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 shutdown_connections 2024-09-18T10:11:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:52.910+0000 7ffb5f384700 1 -- 172.21.15.150:0/1173004698 wait complete. 2024-09-18T10:11:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:already renamed to 'bar' 2024-09-18T10:11:52.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:11:52.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false grep foo-asdf 2024-09-18T10:11:52.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:52.924 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo-asdf 2024-09-18T10:11:53.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: get_monmap_and_config 2024-09-18T10:11:53.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: monmap: 2024-09-18T10:11:53.024 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.024 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.025 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:53.024308+0000 2024-09-18T10:11:53.025 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:53.024308+0000 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:53.026 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 Processor -- start 2024-09-18T10:11:53.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 -- start start 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: init 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12b409e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12b40a09a0 0x7f12b4137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-c con 0x7f12b40a09a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12b40a0da0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b409e220 con 0x7f12b40a0da0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b409eaa0 con 0x7f12b409e6d0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b409ee30 con 0x7f12b40a09a0 2024-09-18T10:11:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12ba8bc700 10 monclient(hunting): authenticate will time out at 995.117780s 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b37fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12b40a09a0 0x7f12b4137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b3fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f12b409e6d0 0x7f12b40a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39606/0 (socket says 172.21.15.150:39606) 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b3fff700 1 -- 172.21.15.150:0/4031828031 learned_addr learned my addr 172.21.15.150:0/4031828031 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b37fe700 10 monclient(hunting): get_auth_request con 0x7f12b40a09a0 auth_method 0 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.022+0000 7f12b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b3fff700 1 -- 172.21.15.150:0/4031828031 >> v1:172.21.15.150:6789/0 conn(0x7f12b409e6d0 legacy=0x7f12b40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient(hunting): handle_auth_done global_id 14653 payload 274 2024-09-18T10:11:53.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 -- 172.21.15.150:0/4031828031 >> v1:172.21.15.150:6789/0 conn(0x7f12b409e6d0 legacy=0x7f12b40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 -- 172.21.15.150:0/4031828031 >> v1:172.21.15.142:6789/0 conn(0x7f12b40a0da0 legacy=0x7f12b4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 1 -- 172.21.15.150:0/4031828031 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869904329 0 0) 0x7f12b409eaa0 con 0x7f12b409e6d0 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 monclient: found mon.noname-c 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 -- 172.21.15.150:0/4031828031 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b409d930 con 0x7f12b40a09a0 2024-09-18T10:11:53.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:11:53.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 -- 172.21.15.150:0/4031828031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40a09a0 msgr2=0x7f12b4137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b37fe700 1 --2- 172.21.15.150:0/4031828031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40a09a0 0x7f12b4137a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f129c009960 tx=0x7f129c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e229c36cbf7830 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 1 -- 172.21.15.150:0/4031828031 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c00ba70 con 0x7f12b40a09a0 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: dump: 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.033 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12ba8bc700 5 monclient: authenticate success, global_id 14653 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12ba8bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 1 -- 172.21.15.150:0/4031828031 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c00bbf0 con 0x7f12b40a09a0 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12ba8bc700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12ba8bc700 4 set_mon_vals no callback set 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 1 -- 172.21.15.150:0/4031828031 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c011420 con 0x7f12b40a09a0 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.026+0000 7f12b2ffd700 10 monclient: dump: 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: shutdown 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:53.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- 172.21.15.150:0/4031828031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40a09a0 msgr2=0x7f12b4137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/4031828031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40a09a0 0x7f12b4137a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f129c009960 tx=0x7f129c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- 172.21.15.150:0/4031828031 shutdown_connections 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/4031828031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b40a09a0 0x7f12b4137a20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- 172.21.15.150:0/4031828031 >> 172.21.15.150:0/4031828031 conn(0x7f12b40a58a0 msgr2=0x7f12b40a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- 172.21.15.150:0/4031828031 shutdown_connections 2024-09-18T10:11:53.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- 172.21.15.150:0/4031828031 wait complete. 2024-09-18T10:11:53.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: monmap: 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 Processor -- start 2024-09-18T10:11:53.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- start start 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: init 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-b con 0x7f12b409e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b40a09a0 0x7f12b4141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-a con 0x7f12b40a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): picked mon.noname-c con 0x7f12b40a0da0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40994e0 con 0x7f12b40a09a0 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b409bfe0 con 0x7f12b409e6d0 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b408fe90 con 0x7f12b40a0da0 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12ba8bc700 10 monclient(hunting): authenticate will time out at 995.127602s 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49514/0 (socket says 172.21.15.150:49514) 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 1 -- 172.21.15.150:0/283264806 learned_addr learned my addr 172.21.15.150:0/283264806 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 10 monclient(hunting): get_auth_request con 0x7f12b409e6d0 auth_method 0 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.030+0000 7f12b8e59700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 10 monclient(hunting): get_auth_request con 0x7f12b40a0da0 auth_method 0 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 10 monclient(hunting): handle_auth_done global_id 14659 payload 824 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 msgr2=0x7f12b4142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 msgr2=0x7f12b4142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b8e59700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:53.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 msgr2=0x7f12b4142230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b40a09a0 msgr2=0x7f12b4141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b40a09a0 0x7f12b4141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 monclient: found mon.noname-b 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c009420 con 0x7f12b409e6d0 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 msgr2=0x7f12b4092750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b3fff700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12a400cf60 tx=0x7f12a4001cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a400ad80 con 0x7f12b409e6d0 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:53.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: dump: 2024-09-18T10:11:53.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 5 monclient: authenticate success, global_id 14659 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a40070f0 con 0x7f12b409e6d0 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 10 monclient: _renew_subs 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b408f980 con 0x7f12b409e6d0 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40074b0 con 0x7f12b409e6d0 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12b1ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12abfff700 10 monclient: dump: 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.048 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 10 monclient: _renew_subs 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.034+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b4142910 con 0x7f12b409e6d0 2024-09-18T10:11:53.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12ba8bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12ba8bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12ba8bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b409dbf0 con 0x7f12b409e6d0 2024-09-18T10:11:53.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f12a40078e0 con 0x7f12b409e6d0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12abfff700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 0x7f1294037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f12a4048ad0 con 0x7f12b409e6d0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12b37fe700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 0x7f1294037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.038+0000 7f12b37fe700 10 monclient: get_auth_request con 0x7f1294035ac0 auth_method 0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.042+0000 7f12b37fe700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 0x7f1294037f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f129c009960 tx=0x7f129c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.046+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f12a40125c0 con 0x7f12b409e6d0 2024-09-18T10:11:53.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.046+0000 7f12abfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.046+0000 7f12abfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:53.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12ba8bc700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12ba8bc700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12ba8bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f12b40992c0 con 0x7f12b409e6d0 2024-09-18T10:11:53.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+60 (secure 0 0 0) 0x7f12a401a360 con 0x7f12b409e6d0 2024-09-18T10:11:53.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12abfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.214+0000 7f12abfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:53.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12ba8bc700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12ba8bc700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12ba8bc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7f12b4063ed0 con 0x7f12b409e6d0 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12abfff700 1 -- 172.21.15.150:0/283264806 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+60 (secure 0 0 0) 0x7f12a4029080 con 0x7f12b409e6d0 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12abfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.310+0000 7f12abfff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 msgr2=0x7f1294037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 0x7f1294037f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f129c009960 tx=0x7f129c016000 comp rx=0 tx=0).stop 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 10 monclient: shutdown 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 msgr2=0x7f12b4092750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f12a400cf60 tx=0x7f12a4001cb0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 shutdown_connections 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1294035ac0 0x7f1294037f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12b40a0da0 0x7f12b4142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12b40a09a0 0x7f12b4141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 --2- 172.21.15.150:0/283264806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12b409e6d0 0x7f12b4092750 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 >> 172.21.15.150:0/283264806 conn(0x7f12b40a58a0 msgr2=0x7f12b409a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:53.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 shutdown_connections 2024-09-18T10:11:53.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.314+0000 7f12ba8bc700 1 -- 172.21.15.150:0/283264806 wait complete. 2024-09-18T10:11:53.326 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:53.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule ls 2024-09-18T10:11:53.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false grep bar-asdf 2024-09-18T10:11:53.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:53.327 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep bar-asdf 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.422+0000 7fc0b6360700 10 monclient: get_monmap_and_config 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.422+0000 7fc0b6360700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.422+0000 7fc0b6360700 10 monclient: monmap: 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:53.427306+0000 2024-09-18T10:11:53.427 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:53.427306+0000 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:53.428 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 Processor -- start 2024-09-18T10:11:53.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 -- start start 2024-09-18T10:11:53.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient: init 2024-09-18T10:11:53.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0b00a0120 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc0b00a0c00 0x7fc0b009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-c con 0x7fc0b00a0c00 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc0b01379b0 0x7fc0b0139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-a con 0x7fc0b01379b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b009dae0 con 0x7fc0b00a0120 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b00a21d0 con 0x7fc0b01379b0 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b009ddc0 con 0x7fc0b00a0c00 2024-09-18T10:11:53.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0b6360700 10 monclient(hunting): authenticate will time out at 995.521546s 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc0b00a0c00 0x7fc0b009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc0b00a0c00 0x7fc0b009e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49526/0 (socket says 172.21.15.150:49526) 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1062328657 learned_addr learned my addr 172.21.15.150:0/1062328657 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): get_auth_request con 0x7fc0b00a0c00 auth_method 0 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_done global_id 14662 payload 274 2024-09-18T10:11:53.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1062328657 >> v2:172.21.15.142:3300/0 conn(0x7fc0b01379b0 msgr2=0x7fc0b0139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 --2- 172.21.15.150:0/1062328657 >> v2:172.21.15.142:3300/0 conn(0x7fc0b01379b0 0x7fc0b0139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1062328657 >> v1:172.21.15.142:6789/0 conn(0x7fc0b00a0120 legacy=0x7fc0b00a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 monclient: found mon.noname-c 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.426+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1062328657 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00a1590 con 0x7fc0b00a0c00 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1062328657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 msgr2=0x7fc0b009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0af7fe700 1 --2- 172.21.15.150:0/1062328657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b009e7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00a6850 tx=0x7fc0a4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f9d5a928910b45 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 1 -- 172.21.15.150:0/1062328657 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a400ba70 con 0x7fc0b00a0c00 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: dump: 2024-09-18T10:11:53.435 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 5 monclient: authenticate success, global_id 14662 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 1 -- 172.21.15.150:0/1062328657 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a400bbd0 con 0x7fc0b00a0c00 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 4 set_mon_vals no callback set 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 1 -- 172.21.15.150:0/1062328657 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4011420 con 0x7fc0b00a0c00 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0aeffd700 10 monclient: dump: 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.437 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 10 monclient: shutdown 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- 172.21.15.150:0/1062328657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 msgr2=0x7fc0b009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1062328657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b009e7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00a6850 tx=0x7fc0a4009250 comp rx=0 tx=0).stop 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- 172.21.15.150:0/1062328657 shutdown_connections 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1062328657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b009e7a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1062328657 >> v2:172.21.15.142:3300/0 conn(0x7fc0b01379b0 0x7fc0b0139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- 172.21.15.150:0/1062328657 >> 172.21.15.150:0/1062328657 conn(0x7fc0b00a5830 msgr2=0x7fc0b00a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- 172.21.15.150:0/1062328657 shutdown_connections 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- 172.21.15.150:0/1062328657 wait complete. 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 10 monclient: monmap: 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:53.439 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:53.440 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 Processor -- start 2024-09-18T10:11:53.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 1 -- start start 2024-09-18T10:11:53.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.430+0000 7fc0b6360700 10 monclient: init 2024-09-18T10:11:53.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0b00a0120 0x7fc0b0098700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-a con 0x7fc0b00a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-b con 0x7fc0b00a0c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): picked mon.noname-c con 0x7fc0b01379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b009d720 con 0x7fc0b00a0120 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b009bef0 con 0x7fc0b00a0c00 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b0130ed0 con 0x7fc0b01379b0 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b6360700 10 monclient(hunting): authenticate will time out at 995.528729s 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49538/0 (socket says 172.21.15.150:49538) 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1895553700 learned_addr learned my addr 172.21.15.150:0/1895553700 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): get_auth_request con 0x7fc0b00a0c00 auth_method 0 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 10 monclient(hunting): get_auth_request con 0x7fc0b01379b0 auth_method 0 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 msgr2=0x7fc0b012c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 msgr2=0x7fc0b012c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:53.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0b48fd700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient(hunting): handle_auth_done global_id 14665 payload 824 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 msgr2=0x7fc0b012c7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0b00a0120 msgr2=0x7fc0b0098700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0b00a0120 0x7fc0b0098700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 monclient: found mon.noname-b 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40094f0 con 0x7fc0b00a0c00 2024-09-18T10:11:53.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 msgr2=0x7fc0b0098c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.434+0000 7fc0af7fe700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4006fd0 tx=0x7fc0a4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4006b30 con 0x7fc0b00a0c00 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: dump: 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.448 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 5 monclient: authenticate success, global_id 14665 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _renew_subs 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a401f070 con 0x7fc0b00a0c00 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b012f9c0 con 0x7fc0b00a0c00 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a401b880 con 0x7fc0b00a0c00 2024-09-18T10:11:53.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 10 monclient: dump: 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _renew_subs 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0ae7fc700 4 set_mon_vals no callback set 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b012fc50 con 0x7fc0b00a0c00 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc0a401bc30 con 0x7fc0b00a0c00 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 0x7fc09c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fc0a4021070 con 0x7fc0b00a0c00 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0affff700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 0x7fc09c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0affff700 10 monclient: get_auth_request con 0x7fc09c035ac0 auth_method 0 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.438+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.442+0000 7fc0affff700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 0x7fc09c037f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097c0 tx=0x7fc0a0006ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.446+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.446+0000 7fc097fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.446+0000 7fc097fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:53.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc0b6360700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc0b6360700 10 monclient: _send_command 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc0b6360700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+60 (secure 0 0 0) 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc097fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.614+0000 7fc097fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:53.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc0b6360700 10 start_mon_command cmd=[{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc0b6360700 10 monclient: _send_command 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc0b6360700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule ls"} v 0) v1 -- 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc097fff700 1 -- 172.21.15.150:0/1895553700 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule ls"}]=0 v62) v1 ==== 65+0+60 (secure 0 0 0) 0x7fc0b0130150 con 0x7fc0b00a0c00 2024-09-18T10:11:53.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc097fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule ls"}] 2024-09-18T10:11:53.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.714+0000 7fc097fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 msgr2=0x7fc09c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 0x7fc09c037f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097c0 tx=0x7fc0a0006ce0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 10 monclient: shutdown 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 msgr2=0x7fc0b0098c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4006fd0 tx=0x7fc0a4016000 comp rx=0 tx=0).stop 2024-09-18T10:11:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 shutdown_connections 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc09c035ac0 0x7fc09c037f70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc0b01379b0 0x7fc0b012c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0b00a0c00 0x7fc0b0098c40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 --2- 172.21.15.150:0/1895553700 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc0b00a0120 0x7fc0b0098700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 >> 172.21.15.150:0/1895553700 conn(0x7fc0b00a5830 msgr2=0x7fc0b009ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 shutdown_connections 2024-09-18T10:11:53.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.718+0000 7fc0b6360700 1 -- 172.21.15.150:0/1895553700 wait complete. 2024-09-18T10:11:53.729 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:53.729 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm foo 2024-09-18T10:11:53.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 10 monclient: get_monmap_and_config 2024-09-18T10:11:53.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 10 monclient: monmap: 2024-09-18T10:11:53.813 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:53.813602+0000 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:53.813602+0000 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:53.814 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 1 Processor -- start 2024-09-18T10:11:53.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 1 -- start start 2024-09-18T10:11:53.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.810+0000 7fe2116a2700 10 monclient: init 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe20c09e6d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe20c0a09a0 0x7fe20c137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-c con 0x7fe20c0a09a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe20c0a0da0 0x7fe20c139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-a con 0x7fe20c0a0da0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c09e220 con 0x7fe20c09e6d0 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c09ee30 con 0x7fe20c0a0da0 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20c09eaa0 con 0x7fe20c0a09a0 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient(hunting): authenticate will time out at 995.908418s 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe20c0a09a0 0x7fe20c137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe20c0a09a0 0x7fe20c137a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49554/0 (socket says 172.21.15.150:49554) 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2243489953 learned_addr learned my addr 172.21.15.150:0/2243489953 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): get_auth_request con 0x7fe20c0a09a0 auth_method 0 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_done global_id 14668 payload 274 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2243489953 >> v2:172.21.15.142:3300/0 conn(0x7fe20c0a0da0 msgr2=0x7fe20c139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 --2- 172.21.15.150:0/2243489953 >> v2:172.21.15.142:3300/0 conn(0x7fe20c0a0da0 0x7fe20c139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2243489953 >> v1:172.21.15.142:6789/0 conn(0x7fe20c09e6d0 legacy=0x7fe20c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 monclient: found mon.noname-c 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2243489953 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c09d930 con 0x7fe20c0a09a0 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2243489953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a09a0 msgr2=0x7fe20c137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe20a7fc700 1 --2- 172.21.15.150:0/2243489953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a09a0 0x7fe20c137a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe20c069dd0 tx=0x7fe200009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=738e144d9b7745ad server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 1 -- 172.21.15.150:0/2243489953 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20000ba70 con 0x7fe20c0a09a0 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: dump: 2024-09-18T10:11:53.824 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 5 monclient: authenticate success, global_id 14668 2024-09-18T10:11:53.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 1 -- 172.21.15.150:0/2243489953 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20000bbd0 con 0x7fe20c0a09a0 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe2116a2700 4 set_mon_vals no callback set 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 1 -- 172.21.15.150:0/2243489953 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200011420 con 0x7fe20c0a09a0 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.814+0000 7fe209ffb700 10 monclient: dump: 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: shutdown 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- 172.21.15.150:0/2243489953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a09a0 msgr2=0x7fe20c137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2243489953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a09a0 0x7fe20c137a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe20c069dd0 tx=0x7fe200009250 comp rx=0 tx=0).stop 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- 172.21.15.150:0/2243489953 shutdown_connections 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2243489953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a09a0 0x7fe20c137a20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2243489953 >> v2:172.21.15.142:3300/0 conn(0x7fe20c0a0da0 0x7fe20c139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- 172.21.15.150:0/2243489953 >> 172.21.15.150:0/2243489953 conn(0x7fe20c0a58a0 msgr2=0x7fe20c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- 172.21.15.150:0/2243489953 shutdown_connections 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- 172.21.15.150:0/2243489953 wait complete. 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: build_initial_monmap 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: monmap: 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:53.828 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 Processor -- start 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- start start 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: init 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-c con 0x7fe20c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-b con 0x7fe20c0a0da0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe20c142790 0x7fe20c08e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): picked mon.noname-a con 0x7fe20c142790 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c091af0 con 0x7fe20c142790 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c08fce0 con 0x7fe20c0a0da0 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20c091810 con 0x7fe20c09e6d0 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe2116a2700 10 monclient(hunting): authenticate will time out at 995.915056s 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49558/0 (socket says 172.21.15.150:49558) 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2720162368 learned_addr learned my addr 172.21.15.150:0/2720162368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): get_auth_request con 0x7fe20c0a0da0 auth_method 0 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 10 monclient(hunting): get_auth_request con 0x7fe20c09e6d0 auth_method 0 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 msgr2=0x7fe20c141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 msgr2=0x7fe20c141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.818+0000 7fe20affd700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 10 monclient(hunting): handle_auth_done global_id 14671 payload 824 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 msgr2=0x7fe20c141d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe20c142790 msgr2=0x7fe20c08e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe20c142790 0x7fe20c08e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 monclient: found mon.noname-b 2024-09-18T10:11:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2000094f0 con 0x7fe20c0a0da0 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 msgr2=0x7fe20c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe20a7fc700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe20c09ebe0 tx=0x7fe200016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200026480 con 0x7fe20c0a0da0 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: dump: 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.837 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20001f070 con 0x7fe20c0a0da0 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 5 monclient: authenticate success, global_id 14671 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 10 monclient: _renew_subs 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c091dd0 con 0x7fe20c0a0da0 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe200026870 con 0x7fe20c0a0da0 2024-09-18T10:11:53.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe1f2ffd700 10 monclient: dump: 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 10 monclient: _renew_subs 2024-09-18T10:11:53.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c090a50 con 0x7fe20c0a0da0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.822+0000 7fe2097fa700 4 set_mon_vals no callback set 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe2116a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe2116a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe2116a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c09dbf0 con 0x7fe20c0a0da0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe200026c60 con 0x7fe20c0a0da0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe1f2ffd700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 0x7fe1f8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fe200021070 con 0x7fe20c0a0da0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe20affd700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 0x7fe1f8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.826+0000 7fe20affd700 10 monclient: get_auth_request con 0x7fe1f8035ac0 auth_method 0 2024-09-18T10:11:53.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.830+0000 7fe20affd700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 0x7fe1f8037f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc006fd0 tx=0x7fe1fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:53.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.834+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe200010980 con 0x7fe20c0a0da0 2024-09-18T10:11:53.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.834+0000 7fe1f2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:53.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:53.834+0000 7fe1f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:54.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.002+0000 7fe2116a2700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:54.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.002+0000 7fe2116a2700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:54.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.002+0000 7fe2116a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:54.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.002+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fe20c0a5490 con 0x7fe20c0a0da0 2024-09-18T10:11:54.156 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T10:11:54.156 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T10:11:54.156 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:11:54.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: get_monmap_and_config 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: monmap: 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:54.270244+0000 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:54.270244+0000 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:54.272 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 Processor -- start 2024-09-18T10:11:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 -- start start 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: init 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99ec068bd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99ec1325f0 0x7f99ec1349d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-a con 0x7f99ec1325f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-b con 0x7f99ec134f10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec095970 con 0x7f99ec068bd0 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ec068fa0 con 0x7f99ec1325f0 2024-09-18T10:11:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec095690 con 0x7f99ec134f10 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99ebfff700 10 monclient(hunting): authenticate will time out at 1003.195765s 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99eaffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f99ec068bd0 0x7f99ec096070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48474/0 (socket says 172.21.15.142:48474) 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eaffd700 1 -- 172.21.15.142:0/3531510213 learned_addr learned my addr 172.21.15.142:0/3531510213 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.265+0000 7f99eb7fe700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/3531510213 >> v2:172.21.15.142:3300/0 conn(0x7f99ec1325f0 0x7f99ec1349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 10 monclient(hunting): get_auth_request con 0x7f99ec134f10 auth_method 0 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): get_auth_request con 0x7f99ec1325f0 auth_method 0 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 1 -- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 msgr2=0x7f99ec1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 1 -- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 msgr2=0x7f99ec1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eb7fe700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99eaffd700 1 -- 172.21.15.142:0/3531510213 >> v1:172.21.15.142:6789/0 conn(0x7f99ec068bd0 legacy=0x7f99ec096070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient(hunting): handle_auth_done global_id 4920 payload 274 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 -- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 msgr2=0x7f99ec1372f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 -- 172.21.15.142:0/3531510213 >> v1:172.21.15.142:6789/0 conn(0x7f99ec068bd0 legacy=0x7f99ec096070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.269+0000 7f99ea7fc700 1 -- 172.21.15.142:0/3531510213 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec06f6f0 con 0x7f99ec1325f0 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ea7fc700 1 -- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec1325f0 msgr2=0x7f99ec1349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec1325f0 0x7f99ec1349d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009960 tx=0x7f99e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8bffa83a4c951f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 1 -- 172.21.15.142:0/3531510213 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e000ba70 con 0x7f99ec1325f0 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: dump: 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.284 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 1 -- 172.21.15.142:0/3531510213 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e000bbf0 con 0x7f99ec1325f0 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 5 monclient: authenticate success, global_id 4920 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:54.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 1 -- 172.21.15.142:0/3531510213 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0011420 con 0x7f99ec1325f0 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 4 set_mon_vals no callback set 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99e9ffb700 10 monclient: dump: 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.286 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 10 monclient: shutdown 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:54.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 -- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec1325f0 msgr2=0x7f99ec1349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec1325f0 0x7f99ec1349d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009960 tx=0x7f99e00091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 -- 172.21.15.142:0/3531510213 shutdown_connections 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec134f10 0x7f99ec1372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 --2- 172.21.15.142:0/3531510213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec1325f0 0x7f99ec1349d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 -- 172.21.15.142:0/3531510213 >> 172.21.15.142:0/3531510213 conn(0x7f99ec0a1470 msgr2=0x7f99ec0a1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 -- 172.21.15.142:0/3531510213 shutdown_connections 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 1 -- 172.21.15.142:0/3531510213 wait complete. 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.273+0000 7f99ebfff700 10 monclient: monmap: 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:54.288 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 Processor -- start 2024-09-18T10:11:54.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 -- start start 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient: init 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-a con 0x7f99ec068bd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-c con 0x7f99ec1325f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ec134f10 0x7f99ec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): picked mon.noname-b con 0x7f99ec134f10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec131ac0 con 0x7f99ec068bd0 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec1286c0 con 0x7f99ec134f10 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec06f9b0 con 0x7f99ec1325f0 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ebfff700 10 monclient(hunting): authenticate will time out at 1003.205181s 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53194/0 (socket says 172.21.15.142:53194) 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 -- 172.21.15.142:0/2022505138 learned_addr learned my addr 172.21.15.142:0/2022505138 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): get_auth_request con 0x7f99ec068bd0 auth_method 0 2024-09-18T10:11:54.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36208/0 (socket says 172.21.15.142:36208) 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 10 monclient(hunting): get_auth_request con 0x7f99ec1325f0 auth_method 0 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ec134f10 0x7f99ec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): get_auth_request con 0x7f99ec134f10 auth_method 0 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 msgr2=0x7f99ec06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:11:54.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 msgr2=0x7f99ec06e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient(hunting): handle_auth_done global_id 4923 payload 824 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 msgr2=0x7f99ec06e560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ec134f10 msgr2=0x7f99ec06eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ec134f10 0x7f99ec06eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.277+0000 7f99eaffd700 1 -- 172.21.15.142:0/2022505138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0009420 con 0x7f99ec068bd0 2024-09-18T10:11:54.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99eaffd700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 msgr2=0x7f99ec06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99eaffd700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00c790 tx=0x7f99dc007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc005d40 con 0x7f99ec068bd0 2024-09-18T10:11:54.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: dump: 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.297 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc0070d0 con 0x7f99ec068bd0 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 5 monclient: authenticate success, global_id 4923 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _renew_subs 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec1268d0 con 0x7f99ec068bd0 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc005ea0 con 0x7f99ec068bd0 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 10 monclient: dump: 2024-09-18T10:11:54.298 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99e97fa700 4 set_mon_vals no callback set 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _renew_subs 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec1283e0 con 0x7f99ec068bd0 2024-09-18T10:11:54.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f99dc007680 con 0x7f99ec068bd0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec1288e0 con 0x7f99ec068bd0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 0x7f99d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f99dc0479d0 con 0x7f99ec068bd0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 0x7f99d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.281+0000 7f99ea7fc700 10 monclient: get_auth_request con 0x7f99d8035ac0 auth_method 0 2024-09-18T10:11:54.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.285+0000 7f99ea7fc700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 0x7f99d8037f70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009960 tx=0x7f99e0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.293+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f99ec1288e0 con 0x7f99ec068bd0 2024-09-18T10:11:54.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.293+0000 7f99d2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.293+0000 7f99d2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:11:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:11:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f99ec1288e0 con 0x7f99ec068bd0 2024-09-18T10:11:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99d2ffd700 1 -- 172.21.15.142:0/2022505138 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f99ec1288e0 con 0x7f99ec068bd0 2024-09-18T10:11:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99d2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:11:54.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99d2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:54.522 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:11:54.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 msgr2=0x7f99d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 0x7f99d8037f70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009960 tx=0x7f99e0017000 comp rx=0 tx=0).stop 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 10 monclient: shutdown 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 msgr2=0x7f99ec06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00c790 tx=0x7f99dc007e50 comp rx=0 tx=0).stop 2024-09-18T10:11:54.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 shutdown_connections 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99d8035ac0 0x7f99d8037f70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99ec134f10 0x7f99ec06eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99ec1325f0 0x7f99ec06e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 --2- 172.21.15.142:0/2022505138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99ec068bd0 0x7f99ec06e020 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 >> 172.21.15.142:0/2022505138 conn(0x7f99ec0a1470 msgr2=0x7f99ec096a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 shutdown_connections 2024-09-18T10:11:54.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.517+0000 7f99ebfff700 1 -- 172.21.15.142:0/2022505138 wait complete. 2024-09-18T10:11:54.535 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:11:54.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:11:54.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:11:54.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: get_monmap_and_config 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: monmap: 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:54.641021+0000 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:54.641021+0000 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:54.643 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 Processor -- start 2024-09-18T10:11:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 -- start start 2024-09-18T10:11:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: init 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe7d80999e0 0x7fe7d80a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80999e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 0x7fe7d80a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-c con 0x7fe7d809c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-b con 0x7fe7d80a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d809a000 con 0x7fe7d80999e0 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d80a20c0 con 0x7fe7d809c410 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80a1c80 con 0x7fe7d80a0c00 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7df0f5700 10 monclient(hunting): authenticate will time out at 1003.567425s 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de0f3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe7d80999e0 0x7fe7d80a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36224/0 (socket says 172.21.15.142:36224) 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 1 -- 172.21.15.142:0/1503387953 learned_addr learned my addr 172.21.15.142:0/1503387953 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 10 monclient(hunting): get_auth_request con 0x7fe7d80a0c00 auth_method 0 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request con 0x7fe7d80999e0 auth_method 0 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de0f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de0f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/1503387953 >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 0x7fe7d80a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7dd8f2700 10 monclient(hunting): get_auth_request con 0x7fe7d809c410 auth_method 0 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7dd8f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7dd8f2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7dd8f2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.637+0000 7fe7de8f4700 1 -- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d8138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de8f4700 1 -- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d8138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de8f4700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de8f4700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd8f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd8f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd8f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_done global_id 4926 payload 274 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 -- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d8138910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 -- 172.21.15.142:0/1503387953 >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 msgr2=0x7fe7d80a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/1503387953 >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 0x7fe7d80a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 -- 172.21.15.142:0/1503387953 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d809a580 con 0x7fe7d80999e0 2024-09-18T10:11:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/1503387953 >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 0x7fe7d80a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 -- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80999e0 msgr2=0x7fe7d80a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80999e0 0x7fe7d80a0180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806fbb0 tx=0x7fe7c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905557f964b27a37 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 1 -- 172.21.15.142:0/1503387953 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8011840 con 0x7fe7d80999e0 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: dump: 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 5 monclient: authenticate success, global_id 4926 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 1 -- 172.21.15.142:0/1503387953 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c80119a0 con 0x7fe7d80999e0 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 4 set_mon_vals no callback set 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 1 -- 172.21.15.142:0/1503387953 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8011d40 con 0x7fe7d80999e0 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7dd0f1700 10 monclient: dump: 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 10 monclient: shutdown 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 -- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80999e0 msgr2=0x7fe7d80a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80999e0 0x7fe7d80a0180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806fbb0 tx=0x7fe7c80091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 -- 172.21.15.142:0/1503387953 shutdown_connections 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d8138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/1503387953 >> v2:172.21.15.150:3300/0 conn(0x7fe7d809c410 0x7fe7d80a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/1503387953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80999e0 0x7fe7d80a0180 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.641+0000 7fe7df0f5700 1 -- 172.21.15.142:0/1503387953 >> 172.21.15.142:0/1503387953 conn(0x7fe7d80a5790 msgr2=0x7fe7d80a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- 172.21.15.142:0/1503387953 shutdown_connections 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- 172.21.15.142:0/1503387953 wait complete. 2024-09-18T10:11:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient: monmap: 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 Processor -- start 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- start start 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient: init 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d809c410 0x7fe7d808dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-b con 0x7fe7d809c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d808e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-c con 0x7fe7d80a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 0x7fe7d808e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): picked mon.noname-a con 0x7fe7d80902d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80914a0 con 0x7fe7d80902d0 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80991f0 con 0x7fe7d809c410 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d808fc10 con 0x7fe7d80a0c00 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7df0f5700 10 monclient(hunting): authenticate will time out at 1003.573499s 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 0x7fe7d808e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 0x7fe7d808e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53216/0 (socket says 172.21.15.142:53216) 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 learned_addr learned my addr 172.21.15.142:0/165494842 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): get_auth_request con 0x7fe7d80902d0 auth_method 0 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d808e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d809c410 0x7fe7d808dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request con 0x7fe7d809c410 auth_method 0 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 10 monclient(hunting): get_auth_request con 0x7fe7d80a0c00 auth_method 0 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d808e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d808e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d808e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d808e2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient(hunting): handle_auth_done global_id 4929 payload 824 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 msgr2=0x7fe7d808e2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe7d80a0c00 0x7fe7d808e2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d809c410 msgr2=0x7fe7d808dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d809c410 0x7fe7d808dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d4009420 con 0x7fe7d80902d0 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.645+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7d809c410 0x7fe7d808dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 msgr2=0x7fe7d808e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de8f4700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 0x7fe7d808e810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0007c70 tx=0x7fe7d00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0017070 con 0x7fe7d80902d0 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: dump: 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.672 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 5 monclient: authenticate success, global_id 4929 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d000f040 con 0x7fe7d80902d0 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d000dd70 con 0x7fe7d80902d0 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 monclient: _renew_subs 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7dc8f0700 4 set_mon_vals no callback set 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c8009420 con 0x7fe7d80902d0 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 10 monclient: dump: 2024-09-18T10:11:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 monclient: _renew_subs 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d808ed50 con 0x7fe7d80902d0 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 0x7fe7d806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d8091360 con 0x7fe7d808ef70 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 0x7fe7d806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient: get_auth_request con 0x7fe7d808ef70 auth_method 0 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe7d0003020 con 0x7fe7d80902d0 2024-09-18T10:11:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_done global_id 4932 payload 824 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7de0f3700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7d8091660 con 0x7fe7d808ef70 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7bc035ac0 0x7fe7bc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7fe7d0048380 con 0x7fe7d80902d0 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7bc035ac0 0x7fe7bc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.649+0000 7fe7dd8f2700 10 monclient: get_auth_request con 0x7fe7bc035ac0 auth_method 0 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7de0f3700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 msgr2=0x7fe7d806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 0x7fe7d806dad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806fbb0 tx=0x7fe7c8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8021070 con 0x7fe7d808ef70 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 10 monclient: dump: 2024-09-18T10:11:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.678 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe7c8006c50 con 0x7fe7d808ef70 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 msgr2=0x7fe7d806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 0x7fe7d806dad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806fbb0 tx=0x7fe7c8017000 comp rx=0 tx=0).stop 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7dd8f2700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7bc035ac0 0x7fe7bc037f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40093f0 tx=0x7fe7d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 msgr2=0x7fe7d806dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.653+0000 7fe7ce7fc700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d808ef70 0x7fe7d806dad0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe7d806fbb0 tx=0x7fe7c8017000 comp rx=0 tx=0).stop 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7df0f5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 0x7fe7d806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7df0f5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80647f0 con 0x7fe7d806e010 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 0x7fe7d806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient: get_auth_request con 0x7fe7d806e010 auth_method 0 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 10 monclient(hunting): handle_auth_done global_id 4935 payload 824 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.665+0000 7fe7de0f3700 1 -- 172.21.15.142:0/165494842 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe7d808f7a0 con 0x7fe7d806e010 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7de0f3700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 msgr2=0x7fe7d806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7de0f3700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 0x7fe7d806e3e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80093f0 tx=0x7fe7c8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8025b50 con 0x7fe7d806e010 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 10 monclient: dump: 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fe7c80193b0 con 0x7fe7d806e010 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 msgr2=0x7fe7d806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 0x7fe7d806e3e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80093f0 tx=0x7fe7c8021000 comp rx=0 tx=0).stop 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 msgr2=0x7fe7d806e3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7ce7fc700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d806e010 0x7fe7d806e3e0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80093f0 tx=0x7fe7c8021000 comp rx=0 tx=0).stop 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:11:54.684 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:54.685 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:11:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:11:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:11:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:11:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:11:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:11:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:11:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:11:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:11:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:11:54.698 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:11:54.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7bc035ac0 msgr2=0x7fe7bc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7bc035ac0 0x7fe7bc037f70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40093f0 tx=0x7fe7d4008040 comp rx=0 tx=0).stop 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 10 monclient: shutdown 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 msgr2=0x7fe7d808e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 --2- 172.21.15.142:0/165494842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7d80902d0 0x7fe7d808e810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0007c70 tx=0x7fe7d00092d0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 reap_dead start 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7de8f4700 1 -- 172.21.15.142:0/165494842 reap_dead start 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 shutdown_connections 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 >> 172.21.15.142:0/165494842 conn(0x7fe7d80a5790 msgr2=0x7fe7d809d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:54.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 shutdown_connections 2024-09-18T10:11:54.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.669+0000 7fe7df0f5700 1 -- 172.21.15.142:0/165494842 wait complete. 2024-09-18T10:11:54.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:11:54.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:11:54.701 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:11:54.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.809+0000 7f02f2cb4700 10 monclient: get_monmap_and_config 2024-09-18T10:11:54.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.809+0000 7f02f2cb4700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.809+0000 7f02f2cb4700 10 monclient: monmap: 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:11:54.815681+0000 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:11:54.815681+0000 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:54.817 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:54.818 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:54.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 Processor -- start 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 -- start start 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient: init 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:11:54.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02ec068e50 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f02ec131750 0x7f02ec133b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-a con 0x7f02ec131750 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02ec134070 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec099c90 con 0x7f02ec068e50 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ec09aff0 con 0x7f02ec134070 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ec09b2a0 con 0x7f02ec131750 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f2cb4700 10 monclient(hunting): authenticate will time out at 1003.741226s 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f02ec131750 0x7f02ec133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f02ec131750 0x7f02ec133b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53244/0 (socket says 172.21.15.142:53244) 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 -- 172.21.15.142:0/1462715867 learned_addr learned my addr 172.21.15.142:0/1462715867 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): get_auth_request con 0x7f02ec131750 auth_method 0 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f1cb2700 1 -- 172.21.15.142:0/1462715867 >> v1:172.21.15.142:6789/0 conn(0x7f02ec068e50 legacy=0x7f02ec09a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient(hunting): handle_auth_done global_id 4938 payload 274 2024-09-18T10:11:54.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 -- 172.21.15.142:0/1462715867 >> v1:172.21.15.150:6789/0 conn(0x7f02ec134070 legacy=0x7f02ec136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f24b3700 1 -- 172.21.15.142:0/1462715867 >> v1:172.21.15.150:6789/0 conn(0x7f02ec134070 legacy=0x7f02ec136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 -- 172.21.15.142:0/1462715867 >> v1:172.21.15.142:6789/0 conn(0x7f02ec068e50 legacy=0x7f02ec09a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.813+0000 7f02f14b1700 1 -- 172.21.15.142:0/1462715867 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ec0989c0 con 0x7f02ec131750 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f14b1700 1 -- 172.21.15.142:0/1462715867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec131750 msgr2=0x7f02ec133b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f14b1700 1 --2- 172.21.15.142:0/1462715867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec131750 0x7f02ec133b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f02e400b0d0 tx=0x7f02e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75905d7edbcf7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 1 -- 172.21.15.142:0/1462715867 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e400e070 con 0x7f02ec131750 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: dump: 2024-09-18T10:11:54.824 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 1 -- 172.21.15.142:0/1462715867 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e400bd90 con 0x7f02ec131750 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 5 monclient: authenticate success, global_id 4938 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 1 -- 172.21.15.142:0/1462715867 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e4007630 con 0x7f02ec131750 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 4 set_mon_vals no callback set 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f0cb0700 10 monclient: dump: 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.827 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.827 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.827 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.827 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 10 monclient: shutdown 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 -- 172.21.15.142:0/1462715867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec131750 msgr2=0x7f02ec133b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/1462715867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec131750 0x7f02ec133b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f02e400b0d0 tx=0x7f02e4009840 comp rx=0 tx=0).stop 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 -- 172.21.15.142:0/1462715867 shutdown_connections 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/1462715867 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec131750 0x7f02ec133b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 -- 172.21.15.142:0/1462715867 >> 172.21.15.142:0/1462715867 conn(0x7f02ec070ef0 msgr2=0x7f02ec0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.817+0000 7f02f2cb4700 1 -- 172.21.15.142:0/1462715867 shutdown_connections 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 -- 172.21.15.142:0/1462715867 wait complete. 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient: build_initial_monmap 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient: monmap: 2024-09-18T10:11:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 Processor -- start 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 -- start start 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient: init 2024-09-18T10:11:54.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ec068e50 0x7f02ec134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-b con 0x7f02ec068e50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-c con 0x7f02ec131750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 0x7f02ec134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): picked mon.noname-a con 0x7f02ec134070 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec06f3b0 con 0x7f02ec134070 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec1397c0 con 0x7f02ec068e50 2024-09-18T10:11:54.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec137550 con 0x7f02ec131750 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f2cb4700 10 monclient(hunting): authenticate will time out at 1003.749532s 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36244/0 (socket says 172.21.15.142:36244) 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 -- 172.21.15.142:0/924900672 learned_addr learned my addr 172.21.15.142:0/924900672 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 10 monclient(hunting): get_auth_request con 0x7f02ec131750 auth_method 0 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 0x7f02ec134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): get_auth_request con 0x7f02ec134070 auth_method 0 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 msgr2=0x7f02ec134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:54.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 msgr2=0x7f02ec134980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ec068e50 0x7f02ec134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request con 0x7f02ec068e50 auth_method 0 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f14b1700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient(hunting): handle_auth_done global_id 4941 payload 824 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 msgr2=0x7f02ec134980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec131750 0x7f02ec134980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ec068e50 msgr2=0x7f02ec134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ec068e50 0x7f02ec134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient: _finish_hunting 0 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 monclient: found mon.noname-a 2024-09-18T10:11:54.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.821+0000 7f02f24b3700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4009ae0 con 0x7f02ec134070 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f24b3700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 msgr2=0x7f02ec134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f24b3700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 0x7f02ec134ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c560 tx=0x7f02e0007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e00074f0 con 0x7f02ec134070 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: dump: 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.840 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 5 monclient: authenticate success, global_id 4941 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e0007670 con 0x7f02ec134070 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 monclient: _renew_subs 2024-09-18T10:11:54.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ec135a90 con 0x7f02ec134070 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0005a60 con 0x7f02ec134070 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02dffff700 4 set_mon_vals no callback set 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 10 monclient: dump: 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:11:54.842 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 monclient: _renew_subs 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ec137240 con 0x7f02ec134070 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec0648a0 con 0x7f02ec138540 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec138540 auth_method 0 2024-09-18T10:11:54.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f02e001b3b0 con 0x7f02ec134070 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 0x7f02d0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(62..62 src has 35..62) v4 ==== 5141+0+0 (secure 0 0 0) 0x7f02e00478a0 con 0x7f02ec134070 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f14b1700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 0x7f02d0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:54.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.825+0000 7f02f14b1700 10 monclient: get_auth_request con 0x7f02d0035ac0 auth_method 0 2024-09-18T10:11:54.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:54.829+0000 7f02f14b1700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 0x7f02d0037f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009a60 tx=0x7f02e4018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:54.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.914+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 v63) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe200010380 con 0x7fe20c0a0da0 2024-09-18T10:11:54.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.914+0000 7fe1f2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:54.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:54.914+0000 7fe1f2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:11:55.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.018+0000 7fe2116a2700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.018+0000 7fe2116a2700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.018+0000 7fe2116a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.022+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7fe20c0a5490 con 0x7fe20c0a0da0 2024-09-18T10:11:55.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.022+0000 7fe1f2ffd700 1 -- 172.21.15.150:0/2720162368 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v63) v1 ==== 103+0+0 (secure 0 0 0) 0x7fe20002b020 con 0x7fe20c0a0da0 2024-09-18T10:11:55.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.022+0000 7fe1f2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.022+0000 7fe1f2ffd700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 msgr2=0x7fe1f8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 0x7fe1f8037f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc006fd0 tx=0x7fe1fc008040 comp rx=0 tx=0).stop 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 10 monclient: shutdown 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 msgr2=0x7fe20c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe20c09ebe0 tx=0x7fe200016000 comp rx=0 tx=0).stop 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 shutdown_connections 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1f8035ac0 0x7fe1f8037f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe20c142790 0x7fe20c08e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20c0a0da0 0x7fe20c142250 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 --2- 172.21.15.150:0/2720162368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe20c09e6d0 0x7fe20c141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 >> 172.21.15.150:0/2720162368 conn(0x7fe20c0a58a0 msgr2=0x7fe20c09a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 shutdown_connections 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.026+0000 7fe2116a2700 1 -- 172.21.15.150:0/2720162368 wait complete. 2024-09-18T10:11:55.030 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo does not exist 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec138540 auth_method 0 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:55.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.029+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:11:55.038 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm foo 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: get_monmap_and_config 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: monmap: 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:55.134071+0000 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:55.134071+0000 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:55.135 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 Processor -- start 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 -- start start 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: init 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f617c0a0120 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:55.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f617c0a0c00 0x7f617c09e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-c con 0x7f617c0a0c00 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f617c1379b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c09dae0 con 0x7f617c1379b0 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0a1590 con 0x7f617c0a0120 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c0a21d0 con 0x7f617c0a0c00 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61814e8700 10 monclient(hunting): authenticate will time out at 997.227365s 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f617affd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f617c0a0120 0x7f617c0a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39608/0 (socket says 172.21.15.150:39608) 2024-09-18T10:11:55.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f617affd700 1 -- 172.21.15.150:0/2121653504 learned_addr learned my addr 172.21.15.150:0/2121653504 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:55.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61727fc700 1 --2- 172.21.15.150:0/2121653504 >> v2:172.21.15.150:3300/0 conn(0x7f617c0a0c00 0x7f617c09e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61727fc700 10 monclient(hunting): get_auth_request con 0x7f617c0a0c00 auth_method 0 2024-09-18T10:11:55.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.129+0000 7f61727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f617affd700 1 -- 172.21.15.150:0/2121653504 >> v1:172.21.15.150:6789/0 conn(0x7f617c0a0120 legacy=0x7f617c0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f617b7fe700 1 -- 172.21.15.150:0/2121653504 >> v1:172.21.15.142:6789/0 conn(0x7f617c1379b0 legacy=0x7f617c139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient(hunting): handle_auth_done global_id 14686 payload 274 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 1 -- 172.21.15.150:0/2121653504 >> v1:172.21.15.150:6789/0 conn(0x7f617c0a0120 legacy=0x7f617c0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 1 -- 172.21.15.150:0/2121653504 >> v1:172.21.15.142:6789/0 conn(0x7f617c1379b0 legacy=0x7f617c139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 1 monclient: found mon.noname-c 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.133+0000 7f61727fc700 1 -- 172.21.15.150:0/2121653504 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0a1da0 con 0x7f617c0a0c00 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61727fc700 1 -- 172.21.15.150:0/2121653504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0c00 msgr2=0x7f617c09e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61727fc700 1 --2- 172.21.15.150:0/2121653504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0c00 0x7f617c09e7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f616c0099e0 tx=0x7f616c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f580948ec333ec4c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 1 -- 172.21.15.150:0/2121653504 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00bba0 con 0x7f617c0a0c00 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: dump: 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.142 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 5 monclient: authenticate success, global_id 14686 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 1 -- 172.21.15.150:0/2121653504 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c00bd20 con 0x7f617c0a0c00 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 4 set_mon_vals no callback set 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 1 -- 172.21.15.150:0/2121653504 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c011670 con 0x7f617c0a0c00 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f617a7fc700 10 monclient: dump: 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.144 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 10 monclient: shutdown 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- 172.21.15.150:0/2121653504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0c00 msgr2=0x7f617c09e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 --2- 172.21.15.150:0/2121653504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0c00 0x7f617c09e7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f616c0099e0 tx=0x7f616c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- 172.21.15.150:0/2121653504 shutdown_connections 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 --2- 172.21.15.150:0/2121653504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0c00 0x7f617c09e7a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- 172.21.15.150:0/2121653504 >> 172.21.15.150:0/2121653504 conn(0x7f617c0a5830 msgr2=0x7f617c0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- 172.21.15.150:0/2121653504 shutdown_connections 2024-09-18T10:11:55.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- 172.21.15.150:0/2121653504 wait complete. 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 10 monclient: monmap: 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:55.149 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 Processor -- start 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 1 -- start start 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.137+0000 7f61814e8700 10 monclient: init 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-b con 0x7f617c0a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:55.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f617c0a0c00 0x7f617c098c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-c con 0x7f617c0a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 0x7f617c12c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): picked mon.noname-a con 0x7f617c1379b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c09bff0 con 0x7f617c1379b0 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c130290 con 0x7f617c0a0120 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c12e250 con 0x7f617c0a0c00 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient(hunting): authenticate will time out at 997.236248s 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49576/0 (socket says 172.21.15.150:49576) 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 -- 172.21.15.150:0/454169402 learned_addr learned my addr 172.21.15.150:0/454169402 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): get_auth_request con 0x7f617c0a0120 auth_method 0 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61727fc700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f617c0a0c00 0x7f617c098c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617b7fe700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 0x7f617c12c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61727fc700 10 monclient(hunting): get_auth_request con 0x7f617c0a0c00 auth_method 0 2024-09-18T10:11:55.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient(hunting): handle_auth_done global_id 14692 payload 824 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 -- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f617c0a0c00 msgr2=0x7f617c098c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f617c0a0c00 0x7f617c098c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 -- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 msgr2=0x7f617c12c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 0x7f617c12c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 monclient: found mon.noname-b 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c009420 con 0x7f617c0a0120 2024-09-18T10:11:55.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617b7fe700 10 monclient: get_auth_request con 0x7f617c1379b0 auth_method 0 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617b7fe700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 0x7f617c12c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 -- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 msgr2=0x7f617c098700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f617affd700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f616400c790 tx=0x7f6164007e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164005d40 con 0x7f617c0a0120 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: dump: 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.155 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 5 monclient: authenticate success, global_id 14692 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640070d0 con 0x7f617c0a0120 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _renew_subs 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c12ede0 con 0x7f617c0a0120 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164005ea0 con 0x7f617c0a0120 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f6179ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61737fe700 10 monclient: dump: 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.158 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.159 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _renew_subs 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.141+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c130f60 con 0x7f617c0a0120 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f616401b3b0 con 0x7f617c0a0120 2024-09-18T10:11:55.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61814e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61814e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61814e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c063ed0 con 0x7f617c0a0120 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61737fe700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 0x7f615c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 35..63) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f61640479d0 con 0x7f617c0a0120 2024-09-18T10:11:55.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61727fc700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 0x7f615c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.145+0000 7f61727fc700 10 monclient: get_auth_request con 0x7f615c035ac0 auth_method 0 2024-09-18T10:11:55.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.149+0000 7f61727fc700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 0x7f615c037f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f616c0099e0 tx=0x7f616c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.153+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f616401a360 con 0x7f617c0a0120 2024-09-18T10:11:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.153+0000 7f61737fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.153+0000 7f61737fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:55.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.325+0000 7f61814e8700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.325+0000 7f61814e8700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.325+0000 7f61814e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.325+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f617c12fed0 con 0x7f617c0a0120 2024-09-18T10:11:55.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.329+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v63) v1 ==== 103+0+0 (secure 0 0 0) 0x7f6164017020 con 0x7f617c0a0120 2024-09-18T10:11:55.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.329+0000 7f61737fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.329+0000 7f61737fe700 10 monclient: _finish_command 2 = system:0 rule foo does not exist 2024-09-18T10:11:55.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.425+0000 7f61814e8700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.425+0000 7f61814e8700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.425+0000 7f61814e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.425+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo"} v 0) v1 -- 0x7f617c0008d0 con 0x7f617c0a0120 2024-09-18T10:11:55.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61737fe700 1 -- 172.21.15.150:0/454169402 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo"}]=0 rule foo does not exist v63) v1 ==== 103+0+0 (secure 0 0 0) 0x7f616400e290 con 0x7f617c0a0120 2024-09-18T10:11:55.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61737fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo"}] 2024-09-18T10:11:55.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61737fe700 10 monclient: _finish_command 3 = system:0 rule foo does not exist 2024-09-18T10:11:55.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 msgr2=0x7f615c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 0x7f615c037f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f616c0099e0 tx=0x7f616c017000 comp rx=0 tx=0).stop 2024-09-18T10:11:55.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 10 monclient: shutdown 2024-09-18T10:11:55.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 msgr2=0x7f617c098700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f616400c790 tx=0x7f6164007e50 comp rx=0 tx=0).stop 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 shutdown_connections 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f615c035ac0 0x7f615c037f70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f617c1379b0 0x7f617c12c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f617c0a0c00 0x7f617c098c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 --2- 172.21.15.150:0/454169402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f617c0a0120 0x7f617c098700 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 >> 172.21.15.150:0/454169402 conn(0x7f617c0a5830 msgr2=0x7f617c09ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 shutdown_connections 2024-09-18T10:11:55.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.429+0000 7f61814e8700 1 -- 172.21.15.150:0/454169402 wait complete. 2024-09-18T10:11:55.437 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo does not exist 2024-09-18T10:11:55.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec138540 auth_method 0 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:55.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:55.433+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:11:55.445 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm bar 2024-09-18T10:11:55.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: get_monmap_and_config 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: monmap: 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:55.548927+0000 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:55.548927+0000 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:55.549 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:55.550 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:55.550 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 Processor -- start 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 -- start start 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: init 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f86e409c930 0x7f86e40977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-a con 0x7f86e409c930 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:55.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86e409cd00 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 0x7f86e4139b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-c con 0x7f86e4137770 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e4098f00 con 0x7f86e409cd00 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e4097d00 con 0x7f86e409c930 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e4097030 con 0x7f86e4137770 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86ea0fd700 10 monclient(hunting): authenticate will time out at 997.642466s 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 0x7f86e4139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 0x7f86e4139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49588/0 (socket says 172.21.15.150:49588) 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 1 -- 172.21.15.150:0/134830575 learned_addr learned my addr 172.21.15.150:0/134830575 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): get_auth_request con 0x7f86e4137770 auth_method 0 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 --2- 172.21.15.150:0/134830575 >> v2:172.21.15.142:3300/0 conn(0x7f86e409c930 0x7f86e40977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): get_auth_request con 0x7f86e409c930 auth_method 0 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient(hunting): handle_auth_done global_id 4947 payload 274 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 -- 172.21.15.150:0/134830575 >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 msgr2=0x7f86e4139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 --2- 172.21.15.150:0/134830575 >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 0x7f86e4139b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 -- 172.21.15.150:0/134830575 >> v1:172.21.15.150:6789/0 conn(0x7f86e409cd00 legacy=0x7f86e409d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 monclient: found mon.noname-a 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:55.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.545+0000 7f86e37fe700 1 -- 172.21.15.150:0/134830575 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e406f730 con 0x7f86e409c930 2024-09-18T10:11:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e37fe700 1 -- 172.21.15.150:0/134830575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409c930 msgr2=0x7f86e40977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e37fe700 1 --2- 172.21.15.150:0/134830575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409c930 0x7f86e40977c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f86e409e8c0 tx=0x7f86d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc2c2eb3a0b9c33 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 1 -- 172.21.15.150:0/134830575 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4011840 con 0x7f86e409c930 2024-09-18T10:11:55.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: dump: 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.562 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 5 monclient: authenticate success, global_id 4947 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 1 -- 172.21.15.150:0/134830575 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d40119a0 con 0x7f86e409c930 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 4 set_mon_vals no callback set 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 1 -- 172.21.15.150:0/134830575 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d4011d40 con 0x7f86e409c930 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86e27fc700 10 monclient: dump: 2024-09-18T10:11:55.563 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 10 monclient: shutdown 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 -- 172.21.15.150:0/134830575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409c930 msgr2=0x7f86e40977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/134830575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409c930 0x7f86e40977c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f86e409e8c0 tx=0x7f86d40091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 -- 172.21.15.150:0/134830575 shutdown_connections 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/134830575 >> v2:172.21.15.150:3300/0 conn(0x7f86e4137770 0x7f86e4139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/134830575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409c930 0x7f86e40977c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 -- 172.21.15.150:0/134830575 >> 172.21.15.150:0/134830575 conn(0x7f86e4068360 msgr2=0x7f86e40a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 -- 172.21.15.150:0/134830575 shutdown_connections 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.549+0000 7f86ea0fd700 1 -- 172.21.15.150:0/134830575 wait complete. 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient: monmap: 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:55.565 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 Processor -- start 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 -- start start 2024-09-18T10:11:55.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient: init 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-c con 0x7f86e409c930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409cd00 0x7f86e411f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-a con 0x7f86e409cd00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): picked mon.noname-b con 0x7f86e4137770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:55.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e409e420 con 0x7f86e409cd00 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e409bb60 con 0x7f86e4137770 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e4122840 con 0x7f86e409c930 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86ea0fd700 10 monclient(hunting): authenticate will time out at 997.649336s 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49594/0 (socket says 172.21.15.150:49594) 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 -- 172.21.15.150:0/567297953 learned_addr learned my addr 172.21.15.150:0/567297953 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:55.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): get_auth_request con 0x7f86e4137770 auth_method 0 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409cd00 0x7f86e411f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): get_auth_request con 0x7f86e409cd00 auth_method 0 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 10 monclient(hunting): get_auth_request con 0x7f86e409c930 auth_method 0 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient(hunting): handle_auth_done global_id 14698 payload 824 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 msgr2=0x7f86e411f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 msgr2=0x7f86e411f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 msgr2=0x7f86e411f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:55.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e37fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409cd00 msgr2=0x7f86e411f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409cd00 0x7f86e411f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 monclient: found mon.noname-b 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.553+0000 7f86e3fff700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d4009420 con 0x7f86e4137770 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86e3fff700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 msgr2=0x7f86e411fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86e3fff700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009e50 tx=0x7f86dc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc009c30 con 0x7f86e4137770 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:55.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: dump: 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.572 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 5 monclient: authenticate success, global_id 14698 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _renew_subs 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e41224e0 con 0x7f86e4137770 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _renew_subs 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e4123660 con 0x7f86e4137770 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc00f040 con 0x7f86e4137770 2024-09-18T10:11:55.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc00cd60 con 0x7f86e4137770 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 10 monclient: dump: 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.574 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.575 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86e1ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:55.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f86dc014440 con 0x7f86e4137770 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c40050b0 con 0x7f86e4137770 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 0x7f86d0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 35..63) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f86dc048980 con 0x7f86e4137770 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86e37fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 0x7f86d0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.557+0000 7f86e37fe700 10 monclient: get_auth_request con 0x7f86d0035ac0 auth_method 0 2024-09-18T10:11:55.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.561+0000 7f86e37fe700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 0x7f86d0037f70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f86d40030c0 tx=0x7f86d4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.565+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f86dc01cdb0 con 0x7f86e4137770 2024-09-18T10:11:55.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.565+0000 7f86cb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.565+0000 7f86cb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:55.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.725+0000 7f86ea0fd700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.725+0000 7f86ea0fd700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.725+0000 7f86ea0fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.725+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f86c4005360 con 0x7f86e4137770 2024-09-18T10:11:55.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.729+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v63) v1 ==== 103+0+0 (secure 0 0 0) 0x7f86dc01c7d0 con 0x7f86e4137770 2024-09-18T10:11:55.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.729+0000 7f86cb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.729+0000 7f86cb7fe700 10 monclient: _finish_command 2 = system:0 rule bar does not exist 2024-09-18T10:11:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.817+0000 7f86ea0fd700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.817+0000 7f86ea0fd700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.817+0000 7f86ea0fd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.817+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "bar"} v 0) v1 -- 0x7f86c4005b40 con 0x7f86e4137770 2024-09-18T10:11:55.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86cb7fe700 1 -- 172.21.15.150:0/567297953 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "bar"}]=0 rule bar does not exist v63) v1 ==== 103+0+0 (secure 0 0 0) 0x7f86dc01cdb0 con 0x7f86e4137770 2024-09-18T10:11:55.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86cb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "bar"}] 2024-09-18T10:11:55.833 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86cb7fe700 10 monclient: _finish_command 3 = system:0 rule bar does not exist 2024-09-18T10:11:55.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 msgr2=0x7f86d0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 0x7f86d0037f70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f86d40030c0 tx=0x7f86d4016000 comp rx=0 tx=0).stop 2024-09-18T10:11:55.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 10 monclient: shutdown 2024-09-18T10:11:55.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 msgr2=0x7f86e411fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f86dc009e50 tx=0x7f86dc00b040 comp rx=0 tx=0).stop 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 shutdown_connections 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f86d0035ac0 0x7f86d0037f70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e4137770 0x7f86e411fcd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86e409cd00 0x7f86e411f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 --2- 172.21.15.150:0/567297953 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f86e409c930 0x7f86e411f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 >> 172.21.15.150:0/567297953 conn(0x7f86e4068360 msgr2=0x7f86e4099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 shutdown_connections 2024-09-18T10:11:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.829+0000 7f86ea0fd700 1 -- 172.21.15.150:0/567297953 wait complete. 2024-09-18T10:11:55.837 INFO:tasks.workunit.client.0.smithi150.stderr:rule bar does not exist 2024-09-18T10:11:55.846 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create pinning_pool 1 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: get_monmap_and_config 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: monmap: 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.949 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:55.949542+0000 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:55.949542+0000 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:55.950 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 Processor -- start 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 -- start start 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: init 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe409cf90 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0fe409e480 0x7f0fe40a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-c con 0x7f0fe409e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fe40990a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe409e060 con 0x7f0fe409cf90 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe409d750 con 0x7f0fe40990a0 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe40a66c0 con 0x7f0fe409e480 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0feada5700 10 monclient(hunting): authenticate will time out at 998.043074s 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0fe409e480 0x7f0fe40a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0fe409e480 0x7f0fe40a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49600/0 (socket says 172.21.15.150:49600) 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 1 -- 172.21.15.150:0/2752415094 learned_addr learned my addr 172.21.15.150:0/2752415094 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): get_auth_request con 0x7f0fe409e480 auth_method 0 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.945+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe9342700 1 -- 172.21.15.150:0/2752415094 >> v1:172.21.15.150:6789/0 conn(0x7f0fe40990a0 legacy=0x7f0fe4139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe8b41700 1 -- 172.21.15.150:0/2752415094 >> v1:172.21.15.142:6789/0 conn(0x7f0fe409cf90 legacy=0x7f0fe40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_done global_id 14701 payload 274 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 1 -- 172.21.15.150:0/2752415094 >> v1:172.21.15.150:6789/0 conn(0x7f0fe40990a0 legacy=0x7f0fe4139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 1 -- 172.21.15.150:0/2752415094 >> v1:172.21.15.142:6789/0 conn(0x7f0fe409cf90 legacy=0x7f0fe40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 1 monclient: found mon.noname-c 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.949+0000 7f0fe3fff700 1 -- 172.21.15.150:0/2752415094 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe409d360 con 0x7f0fe409e480 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe3fff700 1 -- 172.21.15.150:0/2752415094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe409e480 msgr2=0x7f0fe40a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe3fff700 1 --2- 172.21.15.150:0/2752415094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe409e480 0x7f0fe40a09a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009ab0 tx=0x7f0fd4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24456c81029808cb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 1 -- 172.21.15.150:0/2752415094 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd400bc70 con 0x7f0fe409e480 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: dump: 2024-09-18T10:11:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 1 -- 172.21.15.150:0/2752415094 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd400bdd0 con 0x7f0fe409e480 2024-09-18T10:11:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 5 monclient: authenticate success, global_id 14701 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 1 -- 172.21.15.150:0/2752415094 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4011670 con 0x7f0fe409e480 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 4 set_mon_vals no callback set 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0fe37fe700 10 monclient: dump: 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.960 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 10 monclient: shutdown 2024-09-18T10:11:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 -- 172.21.15.150:0/2752415094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe409e480 msgr2=0x7f0fe40a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 --2- 172.21.15.150:0/2752415094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe409e480 0x7f0fe40a09a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009ab0 tx=0x7f0fd4009250 comp rx=0 tx=0).stop 2024-09-18T10:11:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 -- 172.21.15.150:0/2752415094 shutdown_connections 2024-09-18T10:11:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 --2- 172.21.15.150:0/2752415094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe409e480 0x7f0fe40a09a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 -- 172.21.15.150:0/2752415094 >> 172.21.15.150:0/2752415094 conn(0x7f0fe40a58a0 msgr2=0x7f0fe40a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 -- 172.21.15.150:0/2752415094 shutdown_connections 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 1 -- 172.21.15.150:0/2752415094 wait complete. 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 10 monclient: build_initial_monmap 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.953+0000 7f0feada5700 10 monclient: monmap: 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 Processor -- start 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 -- start start 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient: init 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-b con 0x7f0fe40990a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 0x7f0fe408cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-a con 0x7f0fe409cf90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): picked mon.noname-c con 0x7f0fe409e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe40a4110 con 0x7f0fe409cf90 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe4090a40 con 0x7f0fe40990a0 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe408e930 con 0x7f0fe409e480 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0feada5700 10 monclient(hunting): authenticate will time out at 998.053601s 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49610/0 (socket says 172.21.15.150:49610) 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 -- 172.21.15.150:0/821865052 learned_addr learned my addr 172.21.15.150:0/821865052 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): get_auth_request con 0x7f0fe40990a0 auth_method 0 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 0x7f0fe408cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): get_auth_request con 0x7f0fe409cf90 auth_method 0 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 10 monclient(hunting): get_auth_request con 0x7f0fe409e480 auth_method 0 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:55.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 msgr2=0x7f0fe408d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 msgr2=0x7f0fe408d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe9342700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient(hunting): handle_auth_done global_id 14704 payload 824 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 msgr2=0x7f0fe408d320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 msgr2=0x7f0fe408cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:55.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 0x7f0fe408cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient: _finish_hunting 0 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 monclient: found mon.noname-b 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe8b41700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40094f0 con 0x7f0fe40990a0 2024-09-18T10:11:55.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.957+0000 7f0fe3fff700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 0x7f0fe408cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:55.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe8b41700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 msgr2=0x7f0fe40975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe8b41700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c7c0 tx=0x7f0fd8007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80074f0 con 0x7f0fe40990a0 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: dump: 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.973 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd8007670 con 0x7f0fe40990a0 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8005a60 con 0x7f0fe40990a0 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 10 monclient: dump: 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:55.974 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 5 monclient: authenticate success, global_id 14704 2024-09-18T10:11:55.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _renew_subs 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe40903f0 con 0x7f0fe40990a0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe2ffd700 4 set_mon_vals no callback set 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _renew_subs 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe40910b0 con 0x7f0fe40990a0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe40907f0 con 0x7f0fe40990a0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0fd801b3b0 con 0x7f0fe40990a0 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 0x7f0fd0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:55.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 35..63) v4 ==== 5031+0+0 (secure 0 0 0) 0x7f0fd80478c0 con 0x7f0fe40990a0 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe3fff700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 0x7f0fd0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.961+0000 7f0fe3fff700 10 monclient: get_auth_request con 0x7f0fd0035ac0 auth_method 0 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.965+0000 7f0fe3fff700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 0x7f0fd0037f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009ab0 tx=0x7f0fd4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.973+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0fd800f680 con 0x7f0fe40990a0 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.973+0000 7f0fe0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:55.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:55.973+0000 7f0fe0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:56.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.137+0000 7f0feada5700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:56.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.137+0000 7f0feada5700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:56.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.137+0000 7f0feada5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:56.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.137+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f0fe40a0ee0 con 0x7f0fe40990a0 2024-09-18T10:11:56.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:56.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec138540 auth_method 0 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:56.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:56.237+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:11:56.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.929+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' created v64) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0fd8017020 con 0x7f0fe40990a0 2024-09-18T10:11:56.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.929+0000 7f0fe0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:56.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:56.929+0000 7f0fe0ff9700 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' created 2024-09-18T10:11:57.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.033+0000 7f0feada5700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:57.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.033+0000 7f0feada5700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:57.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.033+0000 7f0feada5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.033+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1} v 0) v1 -- 0x7f0fe4090580 con 0x7f0fe40990a0 2024-09-18T10:11:57.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.037+0000 7f0fe0ff9700 1 -- 172.21.15.150:0/821865052 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}]=0 pool 'pinning_pool' already exists v64) v1 ==== 134+0+0 (secure 0 0 0) 0x7f0fd800e290 con 0x7f0fe40990a0 2024-09-18T10:11:57.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.037+0000 7f0fe0ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "pinning_pool", "pg_num": 1}] 2024-09-18T10:11:57.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.037+0000 7f0fe0ff9700 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' already exists 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 msgr2=0x7f0fd0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 0x7f0fd0037f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009ab0 tx=0x7f0fd4017000 comp rx=0 tx=0).stop 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 10 monclient: shutdown 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 msgr2=0x7f0fe40975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c7c0 tx=0x7f0fd8007e00 comp rx=0 tx=0).stop 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 shutdown_connections 2024-09-18T10:11:57.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0fd0035ac0 0x7f0fd0037f70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fe409e480 0x7f0fe408d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fe409cf90 0x7f0fe408cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 --2- 172.21.15.150:0/821865052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fe40990a0 0x7f0fe40975b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 >> 172.21.15.150:0/821865052 conn(0x7f0fe40a58a0 msgr2=0x7f0fe40a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 shutdown_connections 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.041+0000 7f0feada5700 1 -- 172.21.15.150:0/821865052 wait complete. 2024-09-18T10:11:57.045 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'pinning_pool' already exists 2024-09-18T10:11:57.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rule rm replicated_rule 2024-09-18T10:11:57.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:57.054 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm replicated_rule 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: get_monmap_and_config 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: build_initial_monmap 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: monmap: 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:57.156296+0000 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:57.156296+0000 2024-09-18T10:11:57.156 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:57.157 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 Processor -- start 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 -- start start 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: init 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): picked mon.noname-b con 0x7ff38c0a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff38c0a04f0 0x7ff38c0a0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): picked mon.noname-c con 0x7ff38c0a04f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff38c1369a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c09dae0 con 0x7ff38c1369a0 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0a21d0 con 0x7ff38c0a04f0 2024-09-18T10:11:57.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c09ddc0 con 0x7ff38c0a0120 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff392265700 10 monclient(hunting): authenticate will time out at 999.249718s 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff38c0a04f0 0x7ff38c0a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff38c0a04f0 0x7ff38c0a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49622/0 (socket says 172.21.15.150:49622) 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 1 -- 172.21.15.150:0/1247417904 learned_addr learned my addr 172.21.15.150:0/1247417904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): get_auth_request con 0x7ff38c0a04f0 auth_method 0 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:57.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 10 monclient(hunting): get_auth_request con 0x7ff38c0a0120 auth_method 0 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 1 -- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 msgr2=0x7ff38c09e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 1 -- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 msgr2=0x7ff38c09e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.153+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff390802700 1 -- 172.21.15.150:0/1247417904 >> v1:172.21.15.150:6789/0 conn(0x7ff38c1369a0 legacy=0x7ff38c138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 10 monclient(hunting): handle_auth_done global_id 14707 payload 274 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 -- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 msgr2=0x7ff38c09e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:57.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 -- 172.21.15.150:0/1247417904 >> v1:172.21.15.150:6789/0 conn(0x7ff38c1369a0 legacy=0x7ff38c138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 -- 172.21.15.150:0/1247417904 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0a1590 con 0x7ff38c0a04f0 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 -- 172.21.15.150:0/1247417904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a04f0 msgr2=0x7ff38c0a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38b7fe700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a04f0 0x7ff38c0a0920 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009fa0 tx=0x7ff37c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e39dea399c01ef server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 1 -- 172.21.15.150:0/1247417904 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00e070 con 0x7ff38c0a04f0 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: dump: 2024-09-18T10:11:57.167 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: _finish_auth 0 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff392265700 5 monclient: authenticate success, global_id 14707 2024-09-18T10:11:57.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff392265700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 1 -- 172.21.15.150:0/1247417904 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c00bd10 con 0x7ff38c0a04f0 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 1 -- 172.21.15.150:0/1247417904 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0092e0 con 0x7ff38c0a04f0 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff392265700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff392265700 4 set_mon_vals no callback set 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.157+0000 7ff38affd700 10 monclient: dump: 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: shutdown 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- 172.21.15.150:0/1247417904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a04f0 msgr2=0x7ff38c0a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a04f0 0x7ff38c0a0920 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009fa0 tx=0x7ff37c009840 comp rx=0 tx=0).stop 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- 172.21.15.150:0/1247417904 shutdown_connections 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a04f0 0x7ff38c0a0920 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- 172.21.15.150:0/1247417904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a0120 0x7ff38c09e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- 172.21.15.150:0/1247417904 >> 172.21.15.150:0/1247417904 conn(0x7ff38c0a5830 msgr2=0x7ff38c0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:57.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- 172.21.15.150:0/1247417904 shutdown_connections 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- 172.21.15.150:0/1247417904 wait complete. 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: build_initial_monmap 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: monmap: 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:57.171 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 Processor -- start 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- start start 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: init 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a0120 0x7ff38c12c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): picked mon.noname-b con 0x7ff38c0a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a04f0 0x7ff38c12cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): picked mon.noname-c con 0x7ff38c0a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff38c1369a0 0x7ff38c12d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): picked mon.noname-a con 0x7ff38c1369a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c09d720 con 0x7ff38c1369a0 2024-09-18T10:11:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c09bf80 con 0x7ff38c0a0120 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c12fc30 con 0x7ff38c0a04f0 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff392265700 10 monclient(hunting): authenticate will time out at 999.257901s 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a0120 0x7ff38c12c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a0120 0x7ff38c12c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49626/0 (socket says 172.21.15.150:49626) 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 -- 172.21.15.150:0/1091281838 learned_addr learned my addr 172.21.15.150:0/1091281838 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): get_auth_request con 0x7ff38c0a0120 auth_method 0 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38b7fe700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a04f0 0x7ff38c12cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff38c1369a0 0x7ff38c12d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 10 monclient(hunting): get_auth_request con 0x7ff38c1369a0 auth_method 0 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38b7fe700 10 monclient(hunting): get_auth_request con 0x7ff38c0a04f0 auth_method 0 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient(hunting): handle_auth_done global_id 14713 payload 824 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 -- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a04f0 msgr2=0x7ff38c12cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff38c0a04f0 0x7ff38c12cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 -- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff38c1369a0 msgr2=0x7ff38c12d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff38c1369a0 0x7ff38c12d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 monclient: found mon.noname-b 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff38bfff700 1 -- 172.21.15.150:0/1091281838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c009ae0 con 0x7ff38c0a0120 2024-09-18T10:11:57.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.161+0000 7ff390802700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff38c1369a0 0x7ff38c12d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff38bfff700 1 -- 172.21.15.150:0/1091281838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a0120 msgr2=0x7ff38c12c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff38bfff700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff38c0a0120 0x7ff38c12c6c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff38000d890 tx=0x7ff380008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380012800 con 0x7ff38c0a0120 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: dump: 2024-09-18T10:11:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: _finish_auth 0 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff380012980 con 0x7ff38c0a0120 2024-09-18T10:11:57.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 5 monclient: authenticate success, global_id 14713 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _renew_subs 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 1 -- 172.21.15.150:0/1091281838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c12deb0 con 0x7ff38c0a0120 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff380012d40 con 0x7ff38c0a0120 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 10 monclient: dump: 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.179 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _renew_subs 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff38a7fc700 4 set_mon_vals no callback set 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 1 -- 172.21.15.150:0/1091281838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c12ea60 con 0x7ff38c0a0120 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff392265700 1 -- 172.21.15.150:0/1091281838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c09d860 con 0x7ff38c0a0120 2024-09-18T10:11:57.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff380014640 con 0x7ff38c0a0120 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.165+0000 7ff373fff700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff378035ac0 0x7ff378037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.169+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 35..64) v4 ==== 5382+0+0 (secure 0 0 0) 0x7ff380048da0 con 0x7ff38c0a0120 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.169+0000 7ff38b7fe700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff378035ac0 0x7ff378037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.169+0000 7ff38b7fe700 10 monclient: get_auth_request con 0x7ff378035ac0 auth_method 0 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.169+0000 7ff38b7fe700 1 --2- 172.21.15.150:0/1091281838 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff378035ac0 0x7ff378037f70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009a60 tx=0x7ff37c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.177+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff3800148f0 con 0x7ff38c0a0120 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.177+0000 7ff373fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.177+0000 7ff373fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.337+0000 7ff392265700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T10:11:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.337+0000 7ff392265700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T10:11:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.337+0000 7ff392265700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.337+0000 7ff392265700 1 -- 172.21.15.150:0/1091281838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "replicated_rule"} v 0) v1 -- 0x7ff38c12fdf0 con 0x7ff38c0a0120 2024-09-18T10:11:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.341+0000 7ff373fff700 1 -- 172.21.15.150:0/1091281838 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "replicated_rule"}]=-16 crush rule replicated_rule (0) is in use v64) v1 ==== 132+0+0 (secure 0 0 0) 0x7ff38001dbb0 con 0x7ff38c0a0120 2024-09-18T10:11:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.341+0000 7ff373fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "replicated_rule"}] 2024-09-18T10:11:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.341+0000 7ff373fff700 10 monclient: _finish_command 2 = mon:16 crush rule replicated_rule (0) is in use 2024-09-18T10:11:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:Error EBUSY: crush rule replicated_rule (0) is in use 2024-09-18T10:11:57.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:57.349 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool rm pinning_pool pinning_pool --yes-i-really-really-mean-it 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: get_monmap_and_config 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: build_initial_monmap 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: monmap: 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:57.452997+0000 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:57.452997+0000 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:57.453 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:57.454 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:57.454 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:57.454 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:57.454 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 Processor -- start 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 -- start start 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: init 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:11:57.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efe7c09d000 0x7efe7c0a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-a con 0x7efe7c09d000 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe7c09e780 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-b con 0x7efe7c1379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c0699a0 con 0x7efe7c09e780 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c0a1640 con 0x7efe7c09d000 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c09e3d0 con 0x7efe7c1379b0 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe83d80700 10 monclient(hunting): authenticate will time out at 999.546598s 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efe7c09d000 0x7efe7c0a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8131b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7efe7c09e780 0x7efe7c0a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:59758/0 (socket says 172.21.15.150:59758) 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8131b700 1 -- 172.21.15.150:0/3593907982 learned_addr learned my addr 172.21.15.150:0/3593907982 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:39316/0 (socket says 172.21.15.150:39316) 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 10 monclient(hunting): get_auth_request con 0x7efe7c1379b0 auth_method 0 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): get_auth_request con 0x7efe7c09d000 auth_method 0 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 -- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 msgr2=0x7efe7c139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:11:57.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 -- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 msgr2=0x7efe7c139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe8231d700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient(hunting): handle_auth_done global_id 4962 payload 274 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 -- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 msgr2=0x7efe7c139d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 -- 172.21.15.150:0/3593907982 >> v1:172.21.15.142:6789/0 conn(0x7efe7c09e780 legacy=0x7efe7c0a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:57.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient: _finish_hunting 0 2024-09-18T10:11:57.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 monclient: found mon.noname-a 2024-09-18T10:11:57.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:57.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:57.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.449+0000 7efe81b1c700 1 -- 172.21.15.150:0/3593907982 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c09ebf0 con 0x7efe7c09d000 2024-09-18T10:11:57.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe81b1c700 1 -- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 msgr2=0x7efe7c0a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:57.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c0a0120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009910 tx=0x7efe6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc433536cfcfbf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 1 -- 172.21.15.150:0/3593907982 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c00ba30 con 0x7efe7c09d000 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: dump: 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.464 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: _finish_auth 0 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 5 monclient: authenticate success, global_id 4962 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 1 -- 172.21.15.150:0/3593907982 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c00bb90 con 0x7efe7c09d000 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 4 set_mon_vals no callback set 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 1 -- 172.21.15.150:0/3593907982 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c011420 con 0x7efe7c09d000 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:57.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe80b1a700 10 monclient: dump: 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 10 monclient: shutdown 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 -- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 msgr2=0x7efe7c0a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c0a0120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009910 tx=0x7efe6c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 -- 172.21.15.150:0/3593907982 shutdown_connections 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c1379b0 0x7efe7c139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 --2- 172.21.15.150:0/3593907982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c0a0120 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 -- 172.21.15.150:0/3593907982 >> 172.21.15.150:0/3593907982 conn(0x7efe7c0a5830 msgr2=0x7efe7c0a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 -- 172.21.15.150:0/3593907982 shutdown_connections 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.453+0000 7efe83d80700 1 -- 172.21.15.150:0/3593907982 wait complete. 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient: build_initial_monmap 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient: monmap: 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:57.467 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 Processor -- start 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 -- start start 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient: init 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c12c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-a con 0x7efe7c09d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-c con 0x7efe7c09e780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): picked mon.noname-b con 0x7efe7c1379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c09bf00 con 0x7efe7c09d000 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c12fdb0 con 0x7efe7c1379b0 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c12fad0 con 0x7efe7c09e780 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe83d80700 10 monclient(hunting): authenticate will time out at 999.554363s 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49642/0 (socket says 172.21.15.150:49642) 2024-09-18T10:11:57.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 -- 172.21.15.150:0/3556212284 learned_addr learned my addr 172.21.15.150:0/3556212284 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): get_auth_request con 0x7efe7c1379b0 auth_method 0 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c12c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): get_auth_request con 0x7efe7c09d000 auth_method 0 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 10 monclient(hunting): get_auth_request con 0x7efe7c09e780 auth_method 0 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 msgr2=0x7efe7c12c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 msgr2=0x7efe7c12c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8131b700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient(hunting): handle_auth_done global_id 14716 payload 824 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 msgr2=0x7efe7c12c710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:57.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 msgr2=0x7efe7c12c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c12c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient: _finish_hunting 0 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 monclient: found mon.noname-b 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe8231d700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c009420 con 0x7efe7c1379b0 2024-09-18T10:11:57.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.457+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c12c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe8231d700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 msgr2=0x7efe7c12cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe8231d700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe74009fd0 tx=0x7efe74009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7400ee50 con 0x7efe7c1379b0 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: dump: 2024-09-18T10:11:57.474 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: _finish_auth 0 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe74014420 con 0x7efe7c1379b0 2024-09-18T10:11:57.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 5 monclient: authenticate success, global_id 14716 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _renew_subs 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c130ba0 con 0x7efe7c1379b0 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740147e0 con 0x7efe7c1379b0 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe71ffb700 10 monclient: dump: 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:57.476 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe73fff700 4 set_mon_vals no callback set 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _renew_subs 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c12f7f0 con 0x7efe7c1379b0 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.461+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c12fef0 con 0x7efe7c1379b0 2024-09-18T10:11:57.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efe74014c50 con 0x7efe7c1379b0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe71ffb700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 0x7efe60037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 35..64) v4 ==== 5382+0+0 (secure 0 0 0) 0x7efe74048a60 con 0x7efe7c1379b0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 0x7efe60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe81b1c700 10 monclient: get_auth_request con 0x7efe60035ac0 auth_method 0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.465+0000 7efe81b1c700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 0x7efe60037f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0093f0 tx=0x7efe6c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.473+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7efe7c12fef0 con 0x7efe7c1379b0 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.473+0000 7efe71ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:57.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.473+0000 7efe71ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:57.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.637+0000 7efe83d80700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:57.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.637+0000 7efe83d80700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:57.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.637+0000 7efe83d80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:57.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.637+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe7c130ba0 con 0x7efe7c1379b0 2024-09-18T10:11:57.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.821+0000 7f02dcff9700 10 monclient: tick 2024-09-18T10:11:57.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.821+0000 7f02dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:57.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.821+0000 7f02dcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec138540 auth_method 0 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:57.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:57.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:57.841+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T10:11:57.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.937+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' removed v65) v1 ==== 172+0+0 (secure 0 0 0) 0x7efe7c130ba0 con 0x7efe7c1379b0 2024-09-18T10:11:57.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.937+0000 7efe71ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:57.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:57.937+0000 7efe71ffb700 10 monclient: _finish_command 2 = system:0 pool 'pinning_pool' removed 2024-09-18T10:11:58.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.041+0000 7efe83d80700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:58.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.041+0000 7efe83d80700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:58.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.041+0000 7efe83d80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.041+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7efe7c130ba0 con 0x7efe7c1379b0 2024-09-18T10:11:58.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.157+0000 7efe71ffb700 1 -- 172.21.15.150:0/3556212284 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}]=0 pool 'pinning_pool' does not exist v65) v1 ==== 179+0+0 (secure 0 0 0) 0x7efe7c130ba0 con 0x7efe7c1379b0 2024-09-18T10:11:58.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.157+0000 7efe71ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "pinning_pool", "pool2": "pinning_pool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:11:58.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.157+0000 7efe71ffb700 10 monclient: _finish_command 3 = system:0 pool 'pinning_pool' does not exist 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 msgr2=0x7efe60037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 0x7efe60037f70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efe6c0093f0 tx=0x7efe6c016000 comp rx=0 tx=0).stop 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 10 monclient: shutdown 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 msgr2=0x7efe7c12cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe74009fd0 tx=0x7efe74009870 comp rx=0 tx=0).stop 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 shutdown_connections 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efe60035ac0 0x7efe60037f70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe7c1379b0 0x7efe7c12cc50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efe7c09e780 0x7efe7c12c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 --2- 172.21.15.150:0/3556212284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efe7c09d000 0x7efe7c12c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 >> 172.21.15.150:0/3556212284 conn(0x7efe7c0a5830 msgr2=0x7efe7c0a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:58.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 shutdown_connections 2024-09-18T10:11:58.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.161+0000 7efe83d80700 1 -- 172.21.15.150:0/3556212284 wait complete. 2024-09-18T10:11:58.167 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'pinning_pool' does not exist 2024-09-18T10:11:58.180 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush add-bucket foo osd 2024-09-18T10:11:58.180 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:11:58.180 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket foo osd 2024-09-18T10:11:58.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: get_monmap_and_config 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: build_initial_monmap 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: monmap: 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:58.284834+0000 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:58.284834+0000 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:58.285 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:58.286 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:58.286 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:58.286 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 Processor -- start 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 -- start start 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: init 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb5ac09ae80 0x7fb5ac06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac09ae80 addr v2:172.21.15.142:3300/0 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5ac06b2d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac09b960 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0a13a0 con 0x7fb5ac06b2d0 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac06a990 con 0x7fb5ac09ae80 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a1680 con 0x7fb5ac09b960 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5b1fcc700 10 monclient(hunting): authenticate will time out at 1000.378300s 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5aaffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb5ac06b2d0 0x7fb5ac09b250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39644/0 (socket says 172.21.15.150:39644) 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5aaffd700 1 -- 172.21.15.150:0/886054612 learned_addr learned my addr 172.21.15.150:0/886054612 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 --2- 172.21.15.150:0/886054612 >> v2:172.21.15.142:3300/0 conn(0x7fb5ac09ae80 0x7fb5ac06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb5ac09ae80 auth_method 0 2024-09-18T10:11:58.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 10 monclient(hunting): get_auth_request con 0x7fb5ac09b960 auth_method 0 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 1 -- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 msgr2=0x7fb5ac1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 1 -- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 msgr2=0x7fb5ac1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5abfff700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:58.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_done global_id 4968 payload 274 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 -- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 msgr2=0x7fb5ac1397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 -- 172.21.15.150:0/886054612 >> v1:172.21.15.150:6789/0 conn(0x7fb5ac06b2d0 legacy=0x7fb5ac09b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 monclient: found mon.noname-a 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:11:58.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.281+0000 7fb5ab7fe700 1 -- 172.21.15.150:0/886054612 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0a10c0 con 0x7fb5ac09ae80 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5ab7fe700 1 -- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac09ae80 msgr2=0x7fb5ac06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5ab7fe700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac09ae80 0x7fb5ac06ad90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac09bff0 tx=0x7fb5a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3ff83804857b33 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 1 -- 172.21.15.150:0/886054612 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a000b8c0 con 0x7fb5ac09ae80 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: dump: 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:58.294 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 5 monclient: authenticate success, global_id 4968 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 1 -- 172.21.15.150:0/886054612 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a000ba20 con 0x7fb5ac09ae80 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:58.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 4 set_mon_vals no callback set 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 1 -- 172.21.15.150:0/886054612 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a000bdc0 con 0x7fb5ac09ae80 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5aa7fc700 10 monclient: dump: 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:58.296 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 10 monclient: shutdown 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac09ae80 msgr2=0x7fb5ac06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac09ae80 0x7fb5ac06ad90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac09bff0 tx=0x7fb5a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/886054612 shutdown_connections 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09b960 0x7fb5ac1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 --2- 172.21.15.150:0/886054612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac09ae80 0x7fb5ac06ad90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/886054612 >> 172.21.15.150:0/886054612 conn(0x7fb5ac0a2440 msgr2=0x7fb5ac09e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/886054612 shutdown_connections 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/886054612 wait complete. 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 10 monclient: build_initial_monmap 2024-09-18T10:11:58.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 10 monclient: monmap: 2024-09-18T10:11:58.298 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:58.298 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:58.298 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:58.298 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:58.298 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 Processor -- start 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 1 -- start start 2024-09-18T10:11:58.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.285+0000 7fb5b1fcc700 10 monclient: init 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac06b2d0 0x7fb5ac145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac06b2d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 0x7fb5ac1327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-c con 0x7fb5ac09ae80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5ac09b960 0x7fb5ac136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac09b960 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac146ea0 con 0x7fb5ac06b2d0 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac120be0 con 0x7fb5ac09b960 2024-09-18T10:11:58.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac120900 con 0x7fb5ac09ae80 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient(hunting): authenticate will time out at 1000.384305s 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5ac09b960 0x7fb5ac136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac06b2d0 0x7fb5ac145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5ac09b960 0x7fb5ac136170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49656/0 (socket says 172.21.15.150:49656) 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 -- 172.21.15.150:0/57127221 learned_addr learned my addr 172.21.15.150:0/57127221 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): get_auth_request con 0x7fb5ac09b960 auth_method 0 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): get_auth_request con 0x7fb5ac06b2d0 auth_method 0 2024-09-18T10:11:58.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 0x7fb5ac1327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 10 monclient(hunting): get_auth_request con 0x7fb5ac09ae80 auth_method 0 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 1 -- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 msgr2=0x7fb5ac1327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 1 -- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 msgr2=0x7fb5ac1327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 0x7fb5ac1327d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient(hunting): handle_auth_done global_id 14719 payload 824 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5aaffd700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 0x7fb5ac1327d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 -- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 msgr2=0x7fb5ac1327d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5ac09ae80 0x7fb5ac1327d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 -- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac06b2d0 msgr2=0x7fb5ac145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5ac06b2d0 0x7fb5ac145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:58.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 monclient: found mon.noname-b 2024-09-18T10:11:58.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:58.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 -- 172.21.15.150:0/57127221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0009420 con 0x7fb5ac09b960 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 -- 172.21.15.150:0/57127221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5ac09b960 msgr2=0x7fb5ac136170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5abfff700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5ac09b960 0x7fb5ac136170 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009fd0 tx=0x7fb5a4009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400ee50 con 0x7fb5ac09b960 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: dump: 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.306 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a4014420 con 0x7fb5ac09b960 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 5 monclient: authenticate success, global_id 14719 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40147e0 con 0x7fb5ac09b960 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _renew_subs 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/57127221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac120ec0 con 0x7fb5ac09b960 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5937fe700 10 monclient: dump: 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:58.308 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5a9ffb700 4 set_mon_vals no callback set 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _renew_subs 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.289+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/57127221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1466a0 con 0x7fb5ac09b960 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb5a4014c50 con 0x7fb5ac09b960 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5937fe700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb598035ac0 0x7fb598037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5b1fcc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5b1fcc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5b1fcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 35..65) v4 ==== 5031+0+0 (secure 0 0 0) 0x7fb5a4048c10 con 0x7fb5ac09b960 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/57127221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac069940 con 0x7fb5ac09b960 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5ab7fe700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb598035ac0 0x7fb598037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.293+0000 7fb5ab7fe700 10 monclient: get_auth_request con 0x7fb598035ac0 auth_method 0 2024-09-18T10:11:58.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.297+0000 7fb5ab7fe700 1 --2- 172.21.15.150:0/57127221 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb598035ac0 0x7fb598037f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00093f0 tx=0x7fb5a0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:58.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.301+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb5a4013380 con 0x7fb5ac09b960 2024-09-18T10:11:58.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.301+0000 7fb5937fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:58.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.301+0000 7fb5937fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:58.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.473+0000 7fb5b1fcc700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T10:11:58.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.473+0000 7fb5b1fcc700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T10:11:58.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.473+0000 7fb5b1fcc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:58.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.473+0000 7fb5b1fcc700 1 -- 172.21.15.150:0/57127221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"} v 0) v1 -- 0x7fb5ac063ed0 con 0x7fb5ac09b960 2024-09-18T10:11:58.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 10 monclient: tick 2024-09-18T10:11:58.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:58.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 msgr2=0x7f02ec095390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec138540 0x7f02ec095390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02dcff9700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40060c0 con 0x7f02c4003590 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02c4003590 auth_method 0 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:58.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:58.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:58.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:58.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:58.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:58.821+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:11:58.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.933+0000 7fb5937fe700 1 -- 172.21.15.150:0/57127221 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}]=-22 type 'osd' is for devices, not buckets v65) v1 ==== 136+0+0 (secure 0 0 0) 0x7fb5a401b310 con 0x7fb5ac09b960 2024-09-18T10:11:58.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.933+0000 7fb5937fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "osd"}] 2024-09-18T10:11:58.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:58.933+0000 7fb5937fe700 10 monclient: _finish_command 2 = mon:22 type 'osd' is for devices, not buckets 2024-09-18T10:11:58.939 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: type 'osd' is for devices, not buckets 2024-09-18T10:11:58.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:11:58.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket foo root 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02c4003590 auth_method 0 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:59.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.025+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:11:59.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.049+0000 7fe565b48700 10 monclient: get_monmap_and_config 2024-09-18T10:11:59.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.049+0000 7fe565b48700 10 monclient: build_initial_monmap 2024-09-18T10:11:59.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.049+0000 7fe565b48700 10 monclient: monmap: 2024-09-18T10:11:59.055 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:59.055 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:11:59.055767+0000 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:11:59.055767+0000 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:11:59.056 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:11:59.057 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 Processor -- start 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 -- start start 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient: init 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5600a0120 addr v1:172.21.15.150:6789/0 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5600a0c00 addr v1:172.21.15.142:6789/0 2024-09-18T10:11:59.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe56009e7a0 0x7fe56013b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-c con 0x7fe56009e7a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56009dae0 con 0x7fe5600a0c00 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56009eba0 con 0x7fe5600a0120 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600a21d0 con 0x7fe56009e7a0 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe565b48700 10 monclient(hunting): authenticate will time out at 1001.149423s 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55f7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe5600a0120 0x7fe5600a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58896/0 (socket says 172.21.15.150:58896) 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55f7fe700 1 -- 172.21.15.150:0/653676737 learned_addr learned my addr 172.21.15.150:0/653676737 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 1 --2- 172.21.15.150:0/653676737 >> v2:172.21.15.150:3300/0 conn(0x7fe56009e7a0 0x7fe56013b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): get_auth_request con 0x7fe56009e7a0 auth_method 0 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:59.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55f7fe700 1 -- 172.21.15.150:0/653676737 >> v1:172.21.15.150:6789/0 conn(0x7fe5600a0120 legacy=0x7fe5600a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:59.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:59.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:59.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.053+0000 7fe55ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:59.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55effd700 1 -- 172.21.15.150:0/653676737 >> v1:172.21.15.142:6789/0 conn(0x7fe5600a0c00 legacy=0x7fe5601379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 10 monclient(hunting): handle_auth_done global_id 14722 payload 274 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 -- 172.21.15.150:0/653676737 >> v1:172.21.15.150:6789/0 conn(0x7fe5600a0120 legacy=0x7fe5600a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 -- 172.21.15.150:0/653676737 >> v1:172.21.15.142:6789/0 conn(0x7fe5600a0c00 legacy=0x7fe5601379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 1 -- 172.21.15.150:0/653676737 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805419151 0 0) 0x7fe56009eba0 con 0x7fe5600a0120 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 monclient: found mon.noname-c 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 -- 172.21.15.150:0/653676737 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5600a1590 con 0x7fe56009e7a0 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 -- 172.21.15.150:0/653676737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56009e7a0 msgr2=0x7fe56013b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:59.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55ffff700 1 --2- 172.21.15.150:0/653676737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56009e7a0 0x7fe56013b0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe558009f60 tx=0x7fe558009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e62af38a830e03c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 1 -- 172.21.15.150:0/653676737 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800e070 con 0x7fe56009e7a0 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: dump: 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:59.066 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe565b48700 5 monclient: authenticate success, global_id 14722 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 1 -- 172.21.15.150:0/653676737 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55800be10 con 0x7fe56009e7a0 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe565b48700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:11:59.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe565b48700 10 monclient: get_monmap_and_config success 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 1 -- 172.21.15.150:0/653676737 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5580117d0 con 0x7fe56009e7a0 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe565b48700 4 set_mon_vals no callback set 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.057+0000 7fe55e7fc700 10 monclient: dump: 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:59.068 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: shutdown 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- 172.21.15.150:0/653676737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56009e7a0 msgr2=0x7fe56013b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 --2- 172.21.15.150:0/653676737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56009e7a0 0x7fe56013b0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe558009f60 tx=0x7fe558009780 comp rx=0 tx=0).stop 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- 172.21.15.150:0/653676737 shutdown_connections 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 --2- 172.21.15.150:0/653676737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56009e7a0 0x7fe56013b0e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- 172.21.15.150:0/653676737 >> 172.21.15.150:0/653676737 conn(0x7fe5600a5830 msgr2=0x7fe5600a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:11:59.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- 172.21.15.150:0/653676737 shutdown_connections 2024-09-18T10:11:59.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- 172.21.15.150:0/653676737 wait complete. 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: build_initial_monmap 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: monmap: 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:11:59.075 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 Processor -- start 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- start start 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: init 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: _reopen_session rank -1 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe56009e7a0 0x7fe560141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-a con 0x7fe56009e7a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-b con 0x7fe5600a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5600a0c00 0x7fe56008dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): picked mon.noname-c con 0x7fe5600a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:11:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56009d720 con 0x7fe56009e7a0 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56009c0a0 con 0x7fe5600a0120 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): start opening mon connection 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe560090070 con 0x7fe5600a0c00 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): _renew_subs 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe565b48700 10 monclient(hunting): authenticate will time out at 1001.159375s 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60786/0 (socket says 172.21.15.150:60786) 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 1 -- 172.21.15.150:0/1919439115 learned_addr learned my addr 172.21.15.150:0/1919439115 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): get_auth_request con 0x7fe5600a0120 auth_method 0 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe56009e7a0 0x7fe560141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): get_auth_request con 0x7fe56009e7a0 auth_method 0 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55ffff700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5600a0c00 0x7fe56008dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:11:59.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55ffff700 10 monclient(hunting): get_auth_request con 0x7fe5600a0c00 auth_method 0 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.061+0000 7fe55ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 10 monclient(hunting): handle_auth_done global_id 14728 payload 824 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 -- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5600a0c00 msgr2=0x7fe56008dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5600a0c00 0x7fe56008dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 -- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe56009e7a0 msgr2=0x7fe560141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe56009e7a0 0x7fe560141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 10 monclient: _finish_hunting 0 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 monclient: found mon.noname-b 2024-09-18T10:11:59.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558009a20 con 0x7fe5600a0120 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 -- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 msgr2=0x7fe5601421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55effd700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c740 tx=0x7fe550007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5500074f0 con 0x7fe5600a0120 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: dump: 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:59.080 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: _finish_auth 0 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe550007670 con 0x7fe5600a0120 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 5 monclient: authenticate success, global_id 14728 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 10 monclient: _renew_subs 2024-09-18T10:11:59.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56008fb60 con 0x7fe5600a0120 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550005a60 con 0x7fe5600a0120 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe5477fe700 10 monclient: dump: 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:11:59.082 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 10 monclient: _renew_subs 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe55dffb700 4 set_mon_vals no callback set 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.065+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560092840 con 0x7fe5600a0120 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe55001b3b0 con 0x7fe5600a0120 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe565b48700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe565b48700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe565b48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe560063ed0 con 0x7fe5600a0120 2024-09-18T10:11:59.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe5477fe700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 0x7fe54c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 35..65) v4 ==== 5031+0+0 (secure 0 0 0) 0x7fe550047850 con 0x7fe5600a0120 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe55f7fe700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 0x7fe54c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe55f7fe700 10 monclient: get_auth_request con 0x7fe54c035ac0 auth_method 0 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.069+0000 7fe55f7fe700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 0x7fe54c037f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe5540058d0 tx=0x7fe554007bb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.073+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe5500196e0 con 0x7fe5600a0120 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.073+0000 7fe5477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:11:59.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.073+0000 7fe5477fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:11:59.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.245+0000 7fe565b48700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:11:59.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.245+0000 7fe565b48700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:11:59.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.245+0000 7fe565b48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:11:59.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:11:59.245+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7fe560067b10 con 0x7fe5600a0120 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02c4003590 auth_method 0 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:11:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:11:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:11:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.429+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:11:59.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.821+0000 7f02dcff9700 10 monclient: tick 2024-09-18T10:11:59.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:11:59.821+0000 7f02dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.197+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 added bucket foo type root to crush map v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe5500170a0 con 0x7fe5600a0120 2024-09-18T10:12:00.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.197+0000 7fe5477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:12:00.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.197+0000 7fe5477fe700 10 monclient: _finish_command 2 = system:0 added bucket foo type root to crush map 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02c4003590 auth_method 0 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-09-18T10:12:00.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02c4006230 con 0x7f02c4003590 2024-09-18T10:12:00.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8018990 con 0x7f02c4003590 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.233+0000 7f02ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 10 monclient: dump: 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.240 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f02e8018af0 con 0x7f02c4003590 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8009040 comp rx=0 tx=0).stop 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 msgr2=0x7f02c4005a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:00.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.237+0000 7f02ddffb700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02c4003590 0x7f02c4005a10 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8009040 comp rx=0 tx=0).stop 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f2cb4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 0x7f02ec095cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f2cb4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ec1379f0 con 0x7f02ec0958d0 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 0x7f02ec095cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient: get_auth_request con 0x7f02ec0958d0 auth_method 0 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-18T10:12:00.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.249+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f02ec072a80 con 0x7f02ec0958d0 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f1cb2700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 msgr2=0x7f02ec095cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f1cb2700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 0x7f02ec095cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8034070 con 0x7f02ec0958d0 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 10 monclient: dump: 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.258 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.258 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f02e80238f0 con 0x7f02ec0958d0 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 msgr2=0x7f02ec095cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 0x7f02ec095cd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8022000 comp rx=0 tx=0).stop 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 msgr2=0x7f02ec095cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02ddffb700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ec0958d0 0x7f02ec095cd0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f02e8001660 tx=0x7f02e8022000 comp rx=0 tx=0).stop 2024-09-18T10:12:00.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f24b3700 1 -- 172.21.15.142:0/924900672 reap_dead start 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:00.260 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.261 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.262 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:00.263 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.264 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:00.266 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:00.269 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.271 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.272 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:00.273 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 msgr2=0x7f02d0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 0x7f02d0037f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009a60 tx=0x7f02e4018000 comp rx=0 tx=0).stop 2024-09-18T10:12:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 10 monclient: shutdown 2024-09-18T10:12:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 msgr2=0x7f02ec134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 0x7f02ec134ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c560 tx=0x7f02e0007cc0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 shutdown_connections 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d0035ac0 0x7f02d0037f70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 --2- 172.21.15.142:0/924900672 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ec134070 0x7f02ec134ec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 >> 172.21.15.142:0/924900672 conn(0x7f02ec070ef0 msgr2=0x7f02ec0a0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 shutdown_connections 2024-09-18T10:12:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.253+0000 7f02f2cb4700 1 -- 172.21.15.142:0/924900672 wait complete. 2024-09-18T10:12:00.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:00.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:12:00.277 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:12:00.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.301+0000 7fe565b48700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:12:00.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.301+0000 7fe565b48700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:12:00.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.301+0000 7fe565b48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:00.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.301+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "root"} v 0) v1 -- 0x7fe560142720 con 0x7fe5600a0120 2024-09-18T10:12:00.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe5477fe700 1 -- 172.21.15.150:0/1919439115 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}]=0 bucket 'foo' already exists v66) v1 ==== 126+0+0 (secure 0 0 0) 0x7fe55000f260 con 0x7fe5600a0120 2024-09-18T10:12:00.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe5477fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "root"}] 2024-09-18T10:12:00.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe5477fe700 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 msgr2=0x7fe54c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 0x7fe54c037f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe5540058d0 tx=0x7fe554007bb0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 10 monclient: shutdown 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 msgr2=0x7fe5601421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe55000c740 tx=0x7fe550007e00 comp rx=0 tx=0).stop 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.305+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 shutdown_connections 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035ac0 0x7fe54c037f70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe5600a0c00 0x7fe56008dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5600a0120 0x7fe5601421e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 --2- 172.21.15.150:0/1919439115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe56009e7a0 0x7fe560141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 >> 172.21.15.150:0/1919439115 conn(0x7fe5600a5830 msgr2=0x7fe56009a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 shutdown_connections 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.309+0000 7fe565b48700 1 -- 172.21.15.150:0/1919439115 wait complete. 2024-09-18T10:12:00.313 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'foo' already exists 2024-09-18T10:12:00.323 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph osd create 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.385+0000 7f79ec82f700 10 monclient: get_monmap_and_config 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.385+0000 7f79ec82f700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.385+0000 7f79ec82f700 10 monclient: monmap: 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:00.391385+0000 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:00.391385+0000 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.392 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:00.393 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 Processor -- start 2024-09-18T10:12:00.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 -- start start 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient: init 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79e809ae60 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79e809cdd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f79e809b940 0x7f79e806b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-a con 0x7f79e809b940 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e8064a30 con 0x7f79e809ae60 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e809cb90 con 0x7f79e809cdd0 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e806b660 con 0x7f79e809b940 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79ec82f700 10 monclient(hunting): authenticate will time out at 1009.317315s 2024-09-18T10:12:00.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e6d9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f79e809ae60 0x7f79e809b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48484/0 (socket says 172.21.15.142:48484) 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e6d9d700 1 -- 172.21.15.142:0/2052008393 learned_addr learned my addr 172.21.15.142:0/2052008393 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 1 --2- 172.21.15.142:0/2052008393 >> v2:172.21.15.142:3300/0 conn(0x7f79e809b940 0x7f79e806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): get_auth_request con 0x7f79e809b940 auth_method 0 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e6d9d700 1 -- 172.21.15.142:0/2052008393 >> v1:172.21.15.142:6789/0 conn(0x7f79e809ae60 legacy=0x7f79e809b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.389+0000 7f79e759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79de59c700 1 -- 172.21.15.142:0/2052008393 >> v1:172.21.15.150:6789/0 conn(0x7f79e809cdd0 legacy=0x7f79e81378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 10 monclient(hunting): handle_auth_done global_id 4980 payload 274 2024-09-18T10:12:00.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 -- 172.21.15.142:0/2052008393 >> v1:172.21.15.150:6789/0 conn(0x7f79e809cdd0 legacy=0x7f79e81378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 -- 172.21.15.142:0/2052008393 >> v1:172.21.15.142:6789/0 conn(0x7f79e809ae60 legacy=0x7f79e809b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 monclient: found mon.noname-a 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 -- 172.21.15.142:0/2052008393 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e809e2d0 con 0x7f79e809b940 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 -- 172.21.15.142:0/2052008393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809b940 msgr2=0x7f79e806b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e759e700 1 --2- 172.21.15.142:0/2052008393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809b940 0x7f79e806b120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009f60 tx=0x7f79e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c28e1176f1c9ac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 1 -- 172.21.15.142:0/2052008393 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000e070 con 0x7f79e809b940 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: dump: 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.400 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 5 monclient: authenticate success, global_id 4980 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 1 -- 172.21.15.142:0/2052008393 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000be10 con 0x7f79e809b940 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:00.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 1 -- 172.21.15.142:0/2052008393 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e00117d0 con 0x7f79e809b940 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 4 set_mon_vals no callback set 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79e659c700 10 monclient: dump: 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.402 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 10 monclient: shutdown 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 1 -- 172.21.15.142:0/2052008393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809b940 msgr2=0x7f79e806b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 1 --2- 172.21.15.142:0/2052008393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809b940 0x7f79e806b120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f79e0009f60 tx=0x7f79e0009780 comp rx=0 tx=0).stop 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 1 -- 172.21.15.142:0/2052008393 shutdown_connections 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 1 --2- 172.21.15.142:0/2052008393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809b940 0x7f79e806b120 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.393+0000 7f79ec82f700 1 -- 172.21.15.142:0/2052008393 >> 172.21.15.142:0/2052008393 conn(0x7f79e8071ae0 msgr2=0x7f79e80a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- 172.21.15.142:0/2052008393 shutdown_connections 2024-09-18T10:12:00.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- 172.21.15.142:0/2052008393 wait complete. 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient: monmap: 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:00.406 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 Processor -- start 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- start start 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient: init 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809ae60 0x7f79e81416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-a con 0x7f79e809ae60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 0x7f79e813a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-c con 0x7f79e809b940 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e809cdd0 0x7f79e811f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): picked mon.noname-b con 0x7f79e809cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e806e5b0 con 0x7f79e809ae60 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8120b30 con 0x7f79e809cdd0 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8122660 con 0x7f79e809b940 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79ec82f700 10 monclient(hunting): authenticate will time out at 1009.325589s 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 0x7f79e813a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 0x7f79e813a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36362/0 (socket says 172.21.15.142:36362) 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 -- 172.21.15.142:0/3520159297 learned_addr learned my addr 172.21.15.142:0/3520159297 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): get_auth_request con 0x7f79e809b940 auth_method 0 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809ae60 0x7f79e81416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e809cdd0 0x7f79e811f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): get_auth_request con 0x7f79e809ae60 auth_method 0 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 10 monclient(hunting): get_auth_request con 0x7f79e809cdd0 auth_method 0 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient(hunting): handle_auth_done global_id 34106 payload 824 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e809cdd0 msgr2=0x7f79e811f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e809cdd0 0x7f79e811f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809ae60 msgr2=0x7f79e81416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79e809ae60 0x7f79e81416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 monclient: found mon.noname-c 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79de59c700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009a20 con 0x7f79e809b940 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.397+0000 7f79e759e700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e809cdd0 0x7f79e811f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79de59c700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 msgr2=0x7f79e813a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79de59c700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 0x7f79e813a590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c7c0 tx=0x7f79d0007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d00074f0 con 0x7f79e809b940 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: dump: 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d0007670 con 0x7f79e809b940 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 5 monclient: authenticate success, global_id 34106 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 monclient: _renew_subs 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e8122940 con 0x7f79e809b940 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d0005a60 con 0x7f79e809b940 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 10 monclient: dump: 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.413 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e5d9b700 4 set_mon_vals no callback set 2024-09-18T10:12:00.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 monclient: _renew_subs 2024-09-18T10:12:00.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8141be0 con 0x7f79e809b940 2024-09-18T10:12:00.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 0x7f79e8029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8064680 con 0x7f79e8121700 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f79d001b3b0 con 0x7f79e809b940 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79c8035ac0 0x7f79c8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(66..66 src has 40..66) v4 ==== 5058+0+0 (secure 0 0 0) 0x7f79d00479a0 con 0x7f79e809b940 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 0x7f79e8029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient: get_auth_request con 0x7f79e8121700 auth_method 0 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e759e700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79c8035ac0 0x7f79c8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e759e700 10 monclient: get_auth_request con 0x7f79c8035ac0 auth_method 0 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_done global_id 14737 payload 824 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.401+0000 7f79e6d9d700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79e811b9e0 con 0x7f79e8121700 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79e6d9d700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 msgr2=0x7f79e8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79e6d9d700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 0x7f79e8029d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001670 tx=0x7f79d8007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79e759e700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79c8035ac0 0x7f79c8037f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79e00099f0 tx=0x7f79e0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d8012770 con 0x7f79e8121700 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 10 monclient: dump: 2024-09-18T10:12:00.417 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f79d80128d0 con 0x7f79e8121700 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 msgr2=0x7f79e8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 0x7f79e8029d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001670 tx=0x7f79d8007df0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 msgr2=0x7f79e8029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:00.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.405+0000 7f79df7fe700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e8121700 0x7f79e8029d00 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001670 tx=0x7f79d8007df0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 10 monclient: get_monmap_and_config 2024-09-18T10:12:00.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 10 monclient: monmap: 2024-09-18T10:12:00.428 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:00.428 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:00.415174+0000 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:00.415174+0000 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 1 Processor -- start 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 1 -- start start 2024-09-18T10:12:00.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.409+0000 7fa4017a0700 10 monclient: init 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3fc09c930 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc09d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc09cd00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3fc137770 0x7fa3fc139b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc137770 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc098f00 con 0x7fa3fc09c930 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc097030 con 0x7fa3fc137770 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc06f730 con 0x7fa3fc09cd00 2024-09-18T10:12:00.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient(hunting): authenticate will time out at 1002.508513s 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3fc137770 0x7fa3fc139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3fc137770 0x7fa3fc139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60792/0 (socket says 172.21.15.150:60792) 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/373308520 learned_addr learned my addr 172.21.15.150:0/373308520 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa3fc137770 auth_method 0 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc09d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request con 0x7fa3fc09cd00 auth_method 0 2024-09-18T10:12:00.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_done global_id 14740 payload 274 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/373308520 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 msgr2=0x7fa3fc09d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc09d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/373308520 >> v1:172.21.15.142:6789/0 conn(0x7fa3fc09c930 legacy=0x7fa3fc0977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/373308520 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc09e1f0 con 0x7fa3fc137770 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/373308520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 msgr2=0x7fa3fc139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3fc139b50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009f20 tx=0x7fa3f4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=868d8e7554d36261 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 1 -- 172.21.15.150:0/373308520 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400e070 con 0x7fa3fc137770 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: dump: 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.433 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 5 monclient: authenticate success, global_id 14740 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 1 -- 172.21.15.150:0/373308520 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f400bc90 con 0x7fa3fc137770 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 1 -- 172.21.15.150:0/373308520 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40092e0 con 0x7fa3fc137770 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa4017a0700 4 set_mon_vals no callback set 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.413+0000 7fa3fa7fc700 10 monclient: dump: 2024-09-18T10:12:00.434 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: shutdown 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- 172.21.15.150:0/373308520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 msgr2=0x7fa3fc139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3fc139b50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009f20 tx=0x7fa3f4009840 comp rx=0 tx=0).stop 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- 172.21.15.150:0/373308520 shutdown_connections 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3fc139b50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- 172.21.15.150:0/373308520 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc09d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- 172.21.15.150:0/373308520 >> 172.21.15.150:0/373308520 conn(0x7fa3fc068360 msgr2=0x7fa3fc0a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- 172.21.15.150:0/373308520 shutdown_connections 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- 172.21.15.150:0/373308520 wait complete. 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: monmap: 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:00.436 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 Processor -- start 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- start start 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: init 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:00.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3fc11ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-a con 0x7fa3fc09c930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc09cd00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3fc11f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc137770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc120dd0 con 0x7fa3fc09c930 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc09e3f0 con 0x7fa3fc137770 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc09b880 con 0x7fa3fc09cd00 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa4017a0700 10 monclient(hunting): authenticate will time out at 1002.514060s 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:56072/0 (socket says 172.21.15.150:56072) 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 learned_addr learned my addr 172.21.15.150:0/2097442524 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request con 0x7fa3fc09cd00 auth_method 0 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3fc11ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): get_auth_request con 0x7fa3fc09c930 auth_method 0 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_done global_id 34112 payload 824 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 msgr2=0x7fa3fc11f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3fc11f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 msgr2=0x7fa3fc11ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3fc11ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 monclient: found mon.noname-c 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.417+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f4009ae0 con 0x7fa3fc09cd00 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 msgr2=0x7fa3fc11f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00d8c0 tx=0x7fa3ec008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec012800 con 0x7fa3fc09cd00 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:00.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: dump: 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 5 monclient: authenticate success, global_id 34112 2024-09-18T10:12:00.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _renew_subs 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec012960 con 0x7fa3fc09cd00 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc120850 con 0x7fa3fc09cd00 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec012d00 con 0x7fa3fc09cd00 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 10 monclient: dump: 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.443 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f9ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _renew_subs 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc121420 con 0x7fa3fc09cd00 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa3ec014640 con 0x7fa3fc09cd00 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 0x7fa3dc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(66..66 src has 40..66) v4 ==== 5058+0+0 (secure 0 0 0) 0x7fa3ec048be0 con 0x7fa3fc09cd00 2024-09-18T10:12:00.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa3faffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 0x7fa3dc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.421+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc063ed0 con 0x7fa3fc09cd00 2024-09-18T10:12:00.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.425+0000 7fa3faffd700 10 monclient: get_auth_request con 0x7fa3dc035ac0 auth_method 0 2024-09-18T10:12:00.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.425+0000 7fa3faffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 0x7fa3dc037f70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4003a30 tx=0x7fa3e4007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.433+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa3ec011b20 con 0x7fa3fc09cd00 2024-09-18T10:12:00.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.433+0000 7fa3f2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.433+0000 7fa3f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79ec82f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79ec82f700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 0x7f79e802a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79ec82f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8072a80 con 0x7f79e813aad0 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 0x7f79e802a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient: get_auth_request con 0x7f79e813aad0 auth_method 0 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 10 monclient(hunting): handle_auth_done global_id 14743 payload 824 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 1 -- 172.21.15.142:0/3520159297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f79e80a2a70 con 0x7f79e813aad0 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 msgr2=0x7f79e802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.417+0000 7f79e6d9d700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 0x7f79e802a240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001670 tx=0x7f79d8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d801f940 con 0x7f79e813aad0 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 10 monclient: dump: 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.448 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f79d8026070 con 0x7f79e813aad0 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 msgr2=0x7f79e802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 0x7f79e802a240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f79d8001670 tx=0x7f79d8015000 comp rx=0 tx=0).stop 2024-09-18T10:12:00.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 msgr2=0x7f79e802a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79df7fe700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79e813aad0 0x7f79e802a240 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:00.450 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.451 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.452 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:00.453 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:00.454 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.455 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:00.456 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:00.457 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.458 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.459 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.460 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.461 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:00.462 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79c8035ac0 msgr2=0x7f79c8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79c8035ac0 0x7f79c8037f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79e00099f0 tx=0x7f79e0017000 comp rx=0 tx=0).stop 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 10 monclient: shutdown 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 msgr2=0x7f79e813a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 --2- 172.21.15.142:0/3520159297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79e809b940 0x7f79e813a590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c7c0 tx=0x7f79d0007e00 comp rx=0 tx=0).stop 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79e759e700 1 -- 172.21.15.142:0/3520159297 reap_dead start 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79e759e700 1 -- 172.21.15.142:0/3520159297 reap_dead start 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 shutdown_connections 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 >> 172.21.15.142:0/3520159297 conn(0x7f79e8071ae0 msgr2=0x7f79e806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 shutdown_connections 2024-09-18T10:12:00.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.421+0000 7f79ec82f700 1 -- 172.21.15.142:0/3520159297 wait complete. 2024-09-18T10:12:00.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:00.465 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:12:00.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: get_monmap_and_config 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: monmap: 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:00.572565+0000 2024-09-18T10:12:00.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:00.572565+0000 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:00.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 Processor -- start 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 -- start start 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: init 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 0x7fd53409a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-a con 0x7fd53409bd80 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd53409a740 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-b con 0x7fd5341376a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534068ff0 con 0x7fd53409a740 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd534099b00 con 0x7fd53409bd80 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53409e5f0 con 0x7fd5341376a0 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd5389c8700 10 monclient(hunting): authenticate will time out at 1009.498557s 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 0x7fd53409a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 0x7fd53409a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53278/0 (socket says 172.21.15.142:53278) 2024-09-18T10:12:00.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 1 -- 172.21.15.142:0/2459778222 learned_addr learned my addr 172.21.15.142:0/2459778222 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 10 monclient(hunting): get_auth_request con 0x7fd53409bd80 auth_method 0 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd53259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 --2- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): get_auth_request con 0x7fd5341376a0 auth_method 0 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient(hunting): handle_auth_done global_id 34115 payload 274 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 -- 172.21.15.142:0/2459778222 >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 msgr2=0x7fd53409a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 --2- 172.21.15.142:0/2459778222 >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 0x7fd53409a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 -- 172.21.15.142:0/2459778222 >> v1:172.21.15.142:6789/0 conn(0x7fd53409a740 legacy=0x7fd53409c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 monclient: found mon.noname-b 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.569+0000 7fd532d9d700 1 -- 172.21.15.142:0/2459778222 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53409ea80 con 0x7fd5341376a0 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd532d9d700 1 -- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 msgr2=0x7fd534139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd532d9d700 1 --2- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534139a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009e60 tx=0x7fd52c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a3689661c4557f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 1 -- 172.21.15.142:0/2459778222 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00e070 con 0x7fd5341376a0 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: dump: 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 1 -- 172.21.15.142:0/2459778222 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c00be40 con 0x7fd5341376a0 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 5 monclient: authenticate success, global_id 34115 2024-09-18T10:12:00.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 1 -- 172.21.15.142:0/2459778222 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c012680 con 0x7fd5341376a0 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 4 set_mon_vals no callback set 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:00.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd53159a700 10 monclient: dump: 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.587 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 10 monclient: shutdown 2024-09-18T10:12:00.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 -- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 msgr2=0x7fd534139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 --2- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534139a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009e60 tx=0x7fd52c009780 comp rx=0 tx=0).stop 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 -- 172.21.15.142:0/2459778222 shutdown_connections 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 --2- 172.21.15.142:0/2459778222 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534139a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 --2- 172.21.15.142:0/2459778222 >> v2:172.21.15.142:3300/0 conn(0x7fd53409bd80 0x7fd53409a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 -- 172.21.15.142:0/2459778222 >> 172.21.15.142:0/2459778222 conn(0x7fd5340a24f0 msgr2=0x7fd5340a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 -- 172.21.15.142:0/2459778222 shutdown_connections 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.573+0000 7fd5389c8700 1 -- 172.21.15.142:0/2459778222 wait complete. 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient: build_initial_monmap 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient: monmap: 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:00.590 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 Processor -- start 2024-09-18T10:12:00.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 -- start start 2024-09-18T10:12:00.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient: init 2024-09-18T10:12:00.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:00.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:00.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 0x7fd53412f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-a con 0x7fd53409a740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 0x7fd5340963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-b con 0x7fd53409bd80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): picked mon.noname-c con 0x7fd5341376a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd534098c40 con 0x7fd53409a740 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340976f0 con 0x7fd53409bd80 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5340988d0 con 0x7fd5341376a0 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:00.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd5389c8700 10 monclient(hunting): authenticate will time out at 1009.505520s 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 0x7fd53412f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36378/0 (socket says 172.21.15.142:36378) 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 1 -- 172.21.15.142:0/3593596376 learned_addr learned my addr 172.21.15.142:0/3593596376 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): get_auth_request con 0x7fd5341376a0 auth_method 0 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): get_auth_request con 0x7fd53409a740 auth_method 0 2024-09-18T10:12:00.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 0x7fd5340963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): get_auth_request con 0x7fd53409bd80 auth_method 0 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient(hunting): handle_auth_done global_id 4995 payload 824 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 -- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 msgr2=0x7fd534096900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 -- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 msgr2=0x7fd5340963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 0x7fd5340963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 monclient: found mon.noname-a 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd53259c700 1 -- 172.21.15.142:0/3593596376 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c009a20 con 0x7fd53409a740 2024-09-18T10:12:00.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd532d9d700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:00.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.577+0000 7fd531d9b700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 0x7fd5340963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd53259c700 1 -- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 msgr2=0x7fd53412f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd53259c700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 0x7fd53412f870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd52400e870 tx=0x7fd524008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52400c070 con 0x7fd53409a740 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: dump: 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.602 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.603 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 5 monclient: authenticate success, global_id 4995 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd524013620 con 0x7fd53409a740 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5240139e0 con 0x7fd53409a740 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _renew_subs 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd534130240 con 0x7fd53409a740 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 10 monclient: dump: 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:00.605 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd530d99700 4 set_mon_vals no callback set 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _renew_subs 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd534097970 con 0x7fd53409a740 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd534098a90 con 0x7fd53409a740 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd52401b3b0 con 0x7fd53409a740 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 0x7fd514037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(66..66 src has 40..66) v4 ==== 5058+0+0 (secure 0 0 0) 0x7fd524048040 con 0x7fd53409a740 2024-09-18T10:12:00.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd531d9b700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 0x7fd514037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:00.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.581+0000 7fd531d9b700 10 monclient: get_auth_request con 0x7fd514035ac0 auth_method 0 2024-09-18T10:12:00.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.585+0000 7fd531d9b700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 0x7fd514037f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd528009ed0 tx=0x7fd528009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:00.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.589+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd5240222a0 con 0x7fd53409a740 2024-09-18T10:12:00.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.589+0000 7fd5227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:00.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.589+0000 7fd5227fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:00.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.613+0000 7fa4017a0700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:12:00.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.613+0000 7fa4017a0700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T10:12:00.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.613+0000 7fa4017a0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:00.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:00.613+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa3fc071930 con 0x7fa3fc09cd00 2024-09-18T10:12:00.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.813+0000 7fd5389c8700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:12:00.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.813+0000 7fd5389c8700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:12:00.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.813+0000 7fd5389c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:00.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.813+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd5340696a0 con 0x7fd53409a740 2024-09-18T10:12:00.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5227fc700 1 -- 172.21.15.142:0/3593596376 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fd524011a90 con 0x7fd53409a740 2024-09-18T10:12:00.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:12:00.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5227fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2142097008,config=3140514009,health=906682742,kv=4261215878,logm=2835998130}) 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=2711614101,config=3140514009,health=906682742,kv=4261215878,logm=3469494986}) 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2142097008,config=3140514009,health=906682742,kv=4261215878,logm=2835998130}) 2024-09-18T10:12:00.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=2142097008,config=3140514009,health=906682742,kv=4261215878,logm=2029926918}) 2024-09-18T10:12:00.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537823017}) 2024-09-18T10:12:00.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 msgr2=0x7fd514037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178017187}) 2024-09-18T10:12:00.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537823017}) 2024-09-18T10:12:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 0x7fd514037f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd528009ed0 tx=0x7fd528009060 comp rx=0 tx=0).stop 2024-09-18T10:12:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 10 monclient: shutdown 2024-09-18T10:12:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 msgr2=0x7fd53412f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:00.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 0x7fd53412f870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd52400e870 tx=0x7fd524008040 comp rx=0 tx=0).stop 2024-09-18T10:12:00.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4109500726}) 2024-09-18T10:12:00.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 shutdown_connections 2024-09-18T10:12:00.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd514035ac0 0x7fd514037f70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5341376a0 0x7fd534096900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd53409bd80 0x7fd5340963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 --2- 172.21.15.142:0/3593596376 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd53409a740 0x7fd53412f870 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:00.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 >> 172.21.15.142:0/3593596376 conn(0x7fd5340a24f0 msgr2=0x7fd5340a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:00.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 shutdown_connections 2024-09-18T10:12:00.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:00.817+0000 7fd5389c8700 1 -- 172.21.15.142:0/3593596376 wait complete. 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175416495}) 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1531612556}) 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175416495}) 2024-09-18T10:12:00.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191672950}) 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068375628}) 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231995093}) 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068375628}) 2024-09-18T10:12:00.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.821+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360371763}) 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153509543}) 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3041991771}) 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153509543}) 2024-09-18T10:12:00.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462902084}) 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270490134}) 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3292942889}) 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270490134}) 2024-09-18T10:12:00.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.825+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291253630}) 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719805567}) 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2440908179}) 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719805567}) 2024-09-18T10:12:00.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:00.829+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647409481}) 2024-09-18T10:12:00.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038499354}) 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1525904143}) 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038499354}) 2024-09-18T10:12:01.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.161+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292759832}) 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1607257286,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922385107}) 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1607257286,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T10:12:01.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.181+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3298136372,mdsmap=3585761405}) 2024-09-18T10:12:01.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=53} crc {mgr=3564340610,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2656916866,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=405563648,osd_snap=3823785925,osdmap=1970658138}) 2024-09-18T10:12:01.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=48} crc {mgr=3564340610,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2656916866,mon_config_key=4160305146,monmap=3505902565,osd_metadata=500417744,osd_pg_creating=405563648,osd_snap=454132730,osdmap=3729536232}) 2024-09-18T10:12:01.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:01.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=53} crc {mgr=3564340610,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2656916866,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=405563648,osd_snap=3823785925,osdmap=1970658138}) 2024-09-18T10:12:01.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:01.197+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mgr=1176933844,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3346566414,mon_config_key=620546692,monmap=4028308432,osd_metadata=1440571423,osd_pg_creating=405563648,osd_snap=454132730,osdmap=1900850267}) 2024-09-18T10:12:01.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.465+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v67) v1 ==== 58+0+2 (secure 0 0 0) 0x7fa3ec011340 con 0x7fa3fc09cd00 2024-09-18T10:12:01.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.465+0000 7fa3f2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T10:12:01.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.465+0000 7fa3f2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:01.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.581+0000 7fa4017a0700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:12:01.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.581+0000 7fa4017a0700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:12:01.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.581+0000 7fa4017a0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:01.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:01.581+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa3fc137450 con 0x7fa3fc09cd00 2024-09-18T10:12:01.836 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:12:01.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: get_monmap_and_config 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: build_initial_monmap 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: monmap: 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:01.947967+0000 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:01.947967+0000 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:01.949 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 Processor -- start 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 -- start start 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: init 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39ec132030 0x7f39ec132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-c con 0x7f39ec132030 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f39ec09ece0 0x7f39ec09f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-a con 0x7f39ec09ece0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39ec09ae60 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ec132940 con 0x7f39ec09ae60 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec09f960 con 0x7f39ec09ece0 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39ec0a1a90 con 0x7f39ec132030 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f13e3700 10 monclient(hunting): authenticate will time out at 1010.873594s 2024-09-18T10:12:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f0be2700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f39ec09ae60 0x7f39ec09b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48496/0 (socket says 172.21.15.142:48496) 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39f0be2700 1 -- 172.21.15.142:0/2102680487 learned_addr learned my addr 172.21.15.142:0/2102680487 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39eb7fe700 1 --2- 172.21.15.142:0/2102680487 >> v2:172.21.15.142:3300/0 conn(0x7f39ec09ece0 0x7f39ec09f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39eb7fe700 10 monclient(hunting): get_auth_request con 0x7f39ec09ece0 auth_method 0 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 --2- 172.21.15.142:0/2102680487 >> v2:172.21.15.150:3300/0 conn(0x7f39ec132030 0x7f39ec132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): get_auth_request con 0x7f39ec132030 auth_method 0 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient(hunting): handle_auth_done global_id 14749 payload 274 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 -- 172.21.15.142:0/2102680487 >> v2:172.21.15.142:3300/0 conn(0x7f39ec09ece0 msgr2=0x7f39ec09f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 --2- 172.21.15.142:0/2102680487 >> v2:172.21.15.142:3300/0 conn(0x7f39ec09ece0 0x7f39ec09f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 -- 172.21.15.142:0/2102680487 >> v1:172.21.15.142:6789/0 conn(0x7f39ec09ae60 legacy=0x7f39ec09b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 monclient: found mon.noname-c 2024-09-18T10:12:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.945+0000 7f39ebfff700 1 -- 172.21.15.142:0/2102680487 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec0a17b0 con 0x7f39ec132030 2024-09-18T10:12:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39ebfff700 1 -- 172.21.15.142:0/2102680487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec132030 msgr2=0x7f39ec132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39ebfff700 1 --2- 172.21.15.142:0/2102680487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec132030 0x7f39ec132400 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f39ec06a4d0 tx=0x7f39e00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf7d9b4b40c9aa00 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 1 -- 172.21.15.142:0/2102680487 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000b8c0 con 0x7f39ec132030 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: dump: 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:01.965 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 1 -- 172.21.15.142:0/2102680487 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e000ba20 con 0x7f39ec132030 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 5 monclient: authenticate success, global_id 14749 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 1 -- 172.21.15.142:0/2102680487 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000bdc0 con 0x7f39ec132030 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 4 set_mon_vals no callback set 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39eaffd700 10 monclient: dump: 2024-09-18T10:12:01.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 10 monclient: shutdown 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 -- 172.21.15.142:0/2102680487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec132030 msgr2=0x7f39ec132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 --2- 172.21.15.142:0/2102680487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec132030 0x7f39ec132400 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f39ec06a4d0 tx=0x7f39e00091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 -- 172.21.15.142:0/2102680487 shutdown_connections 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 --2- 172.21.15.142:0/2102680487 >> v2:172.21.15.142:3300/0 conn(0x7f39ec09ece0 0x7f39ec09f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 --2- 172.21.15.142:0/2102680487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec132030 0x7f39ec132400 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 -- 172.21.15.142:0/2102680487 >> 172.21.15.142:0/2102680487 conn(0x7f39ec0a5700 msgr2=0x7f39ec0a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 -- 172.21.15.142:0/2102680487 shutdown_connections 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.949+0000 7f39f13e3700 1 -- 172.21.15.142:0/2102680487 wait complete. 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient: build_initial_monmap 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient: monmap: 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 Processor -- start 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 -- start start 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient: init 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec09ae60 0x7f39ec06e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-b con 0x7f39ec09ae60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39ec09ece0 0x7f39ec130f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-a con 0x7f39ec09ece0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): picked mon.noname-c con 0x7f39ec132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0a4820 con 0x7f39ec09ece0 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec142670 con 0x7f39ec09ae60 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec11e5f0 con 0x7f39ec132030 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f13e3700 10 monclient(hunting): authenticate will time out at 1010.882443s 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39ec09ece0 0x7f39ec130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36392/0 (socket says 172.21.15.142:36392) 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 -- 172.21.15.142:0/589204810 learned_addr learned my addr 172.21.15.142:0/589204810 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): get_auth_request con 0x7f39ec132030 auth_method 0 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec09ae60 0x7f39ec06e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): get_auth_request con 0x7f39ec09ece0 auth_method 0 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): get_auth_request con 0x7f39ec09ae60 auth_method 0 2024-09-18T10:12:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:01.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient(hunting): handle_auth_done global_id 34121 payload 824 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 -- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec09ae60 msgr2=0x7f39ec06e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec09ae60 0x7f39ec06e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 -- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39ec09ece0 msgr2=0x7f39ec130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39ec09ece0 0x7f39ec130f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient: _finish_hunting 0 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 monclient: found mon.noname-c 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:01.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.953+0000 7f39f0be2700 1 -- 172.21.15.142:0/589204810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009420 con 0x7f39ec132030 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f0be2700 1 -- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 msgr2=0x7f39ec11bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f0be2700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f39e400b530 tx=0x7f39e4006f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4005900 con 0x7f39ec132030 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:01.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: dump: 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 5 monclient: authenticate success, global_id 34121 2024-09-18T10:12:01.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _renew_subs 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec11e310 con 0x7f39ec132030 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e4005a60 con 0x7f39ec132030 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4005e20 con 0x7f39ec132030 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 10 monclient: dump: 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:01.978 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _renew_subs 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec11e030 con 0x7f39ec132030 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39ea7fc700 4 set_mon_vals no callback set 2024-09-18T10:12:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec11e730 con 0x7f39ec132030 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.957+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f39e4011a50 con 0x7f39ec132030 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.961+0000 7f39d3fff700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 0x7f39d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.961+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(67..67 src has 40..67) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f39e40481b0 con 0x7f39ec132030 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.961+0000 7f39ebfff700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 0x7f39d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.961+0000 7f39ebfff700 10 monclient: get_auth_request con 0x7f39d8035ac0 auth_method 0 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.961+0000 7f39ebfff700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 0x7f39d8037f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39e00030c0 tx=0x7f39e0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.969+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f39ec11e730 con 0x7f39ec132030 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.969+0000 7f39d3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:01.969+0000 7f39d3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:02.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.189+0000 7f39f13e3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:02.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.189+0000 7f39f13e3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.189+0000 7f39f13e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.189+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39ec11e730 con 0x7f39ec132030 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39d3fff700 1 -- 172.21.15.142:0/589204810 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f39ec11e730 con 0x7f39ec132030 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39d3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39d3fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:02.196 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 msgr2=0x7f39d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 0x7f39d8037f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39e00030c0 tx=0x7f39e0016000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 10 monclient: shutdown 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 msgr2=0x7f39ec11bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f39e400b530 tx=0x7f39e4006f90 comp rx=0 tx=0).stop 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 shutdown_connections 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f39d8035ac0 0x7f39d8037f70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39ec132030 0x7f39ec11bd60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39ec09ece0 0x7f39ec130f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 --2- 172.21.15.142:0/589204810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39ec09ae60 0x7f39ec06e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 >> 172.21.15.142:0/589204810 conn(0x7f39ec0a5700 msgr2=0x7f39ec135450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 shutdown_connections 2024-09-18T10:12:02.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.193+0000 7f39f13e3700 1 -- 172.21.15.142:0/589204810 wait complete. 2024-09-18T10:12:02.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:12:02.210 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:12:02.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:12:02.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:12:02.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: get_monmap_and_config 2024-09-18T10:12:02.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: monmap: 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:02.315949+0000 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:02.315949+0000 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:02.317 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:02.318 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 Processor -- start 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 -- start start 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: init 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faddc09c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:02.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faddc134880 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faddc138380 0x7faddc13a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-c con 0x7faddc138380 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0a6620 con 0x7faddc134880 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0a1c70 con 0x7faddc09c2c0 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faddc09ab90 con 0x7faddc138380 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7fade0b16700 10 monclient(hunting): authenticate will time out at 1011.241373s 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddaffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7faddc134880 0x7faddc136c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41140/0 (socket says 172.21.15.142:41140) 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddaffd700 1 -- 172.21.15.142:0/4038842450 learned_addr learned my addr 172.21.15.142:0/4038842450 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 1 --2- 172.21.15.142:0/4038842450 >> v2:172.21.15.150:3300/0 conn(0x7faddc138380 0x7faddc13a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): get_auth_request con 0x7faddc138380 auth_method 0 2024-09-18T10:12:02.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.313+0000 7faddaffd700 1 -- 172.21.15.142:0/4038842450 >> v1:172.21.15.142:6789/0 conn(0x7faddc134880 legacy=0x7faddc136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541258385 0 0) 0x7fadd0001450 con 0x7faddc134880 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddb7fe700 1 -- 172.21.15.142:0/4038842450 >> v1:172.21.15.150:6789/0 conn(0x7faddc09c2c0 legacy=0x7faddc0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient(hunting): my global_id is 5004 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4003570 con 0x7faddc134880 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 10 monclient(hunting): handle_auth_done global_id 14755 payload 274 2024-09-18T10:12:02.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 -- 172.21.15.142:0/4038842450 >> v1:172.21.15.150:6789/0 conn(0x7faddc09c2c0 legacy=0x7faddc0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 -- 172.21.15.142:0/4038842450 >> v1:172.21.15.142:6789/0 conn(0x7faddc134880 legacy=0x7faddc136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 monclient: found mon.noname-c 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235312697 0 0) 0x7fadc4003570 con 0x7faddc134880 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 -- 172.21.15.142:0/4038842450 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc09c690 con 0x7faddc138380 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fadd0001930 con 0x7faddc134880 2024-09-18T10:12:02.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T10:12:02.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 -- 172.21.15.142:0/4038842450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc138380 msgr2=0x7faddc13a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7faddbfff700 1 --2- 172.21.15.142:0/4038842450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc138380 0x7faddc13a760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009f60 tx=0x7fadd4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58639538825039d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd400e070 con 0x7faddc138380 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: dump: 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fade0b16700 5 monclient: authenticate success, global_id 14755 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fade0b16700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd400be40 con 0x7faddc138380 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fade0b16700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fade0b16700 4 set_mon_vals no callback set 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 1 -- 172.21.15.142:0/4038842450 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4012640 con 0x7faddc138380 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.317+0000 7fadda7fc700 10 monclient: dump: 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: shutdown 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- 172.21.15.142:0/4038842450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc138380 msgr2=0x7faddc13a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 --2- 172.21.15.142:0/4038842450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc138380 0x7faddc13a760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009f60 tx=0x7fadd4009780 comp rx=0 tx=0).stop 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- 172.21.15.142:0/4038842450 shutdown_connections 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 --2- 172.21.15.142:0/4038842450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc138380 0x7faddc13a760 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- 172.21.15.142:0/4038842450 >> 172.21.15.142:0/4038842450 conn(0x7faddc0a5720 msgr2=0x7faddc0a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- 172.21.15.142:0/4038842450 shutdown_connections 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- 172.21.15.142:0/4038842450 wait complete. 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: monmap: 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:02.333 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 Processor -- start 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- start start 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: init 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc09c2c0 0x7faddc06b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-b con 0x7faddc09c2c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faddc134880 0x7faddc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-c con 0x7faddc134880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): picked mon.noname-a con 0x7faddc138380 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc06d9c0 con 0x7faddc138380 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc06dbf0 con 0x7faddc09c2c0 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc13a950 con 0x7faddc134880 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7fade0b16700 10 monclient(hunting): authenticate will time out at 1011.250875s 2024-09-18T10:12:02.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faddc134880 0x7faddc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faddc134880 0x7faddc06b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58790/0 (socket says 172.21.15.142:58790) 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc09c2c0 0x7faddc06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36896/0 (socket says 172.21.15.142:36896) 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 1 -- 172.21.15.142:0/1187722854 learned_addr learned my addr 172.21.15.142:0/1187722854 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): get_auth_request con 0x7faddc134880 auth_method 0 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): get_auth_request con 0x7faddc09c2c0 auth_method 0 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): get_auth_request con 0x7faddc138380 auth_method 0 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient(hunting): handle_auth_done global_id 5007 payload 824 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faddc134880 msgr2=0x7faddc06b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faddc134880 0x7faddc06b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc09c2c0 msgr2=0x7faddc06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faddc09c2c0 0x7faddc06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 monclient: found mon.noname-a 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.321+0000 7faddbfff700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4009a20 con 0x7faddc138380 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7faddbfff700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 msgr2=0x7faddc06bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7faddbfff700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faddc06a3b0 tx=0x7fadd4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4012e10 con 0x7faddc138380 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:02.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: dump: 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd401b630 con 0x7faddc138380 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 5 monclient: authenticate success, global_id 5007 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 10 monclient: _renew_subs 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc06d5b0 con 0x7faddc138380 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd401b9f0 con 0x7faddc138380 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadc37fe700 10 monclient: dump: 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.344 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fadd9ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 10 monclient: _renew_subs 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.325+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc06eb90 con 0x7faddc138380 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fade0b16700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fade0b16700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 0x7faddc09cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc064750 con 0x7faddc06c1d0 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fadd401c400 con 0x7faddc138380 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 0x7faddc09cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient: get_auth_request con 0x7faddc06c1d0 auth_method 0 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadc8035ac0 0x7fadc8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 10 monclient(hunting): handle_auth_done global_id 5010 payload 824 2024-09-18T10:12:02.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faddc072a80 con 0x7faddc06c1d0 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(67..67 src has 40..67) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fadd404a240 con 0x7faddc138380 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddaffd700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadc8035ac0 0x7fadc8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddaffd700 10 monclient: get_auth_request con 0x7fadc8035ac0 auth_method 0 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 msgr2=0x7faddc09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7faddb7fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 0x7faddc09cb00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fadcc007840 tx=0x7fadcc007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc017070 con 0x7faddc06c1d0 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 10 monclient: dump: 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.347 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fadcc005810 con 0x7faddc06c1d0 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 msgr2=0x7faddc09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 0x7faddc09cb00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fadcc007840 tx=0x7fadcc007da0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 msgr2=0x7faddc09cb00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.329+0000 7fadc37fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc06c1d0 0x7faddc09cb00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.333+0000 7faddaffd700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadc8035ac0 0x7fadc8037f70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faddc06c9f0 tx=0x7fadd0001a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7fade0b16700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7fade0b16700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 0x7faddc09d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7fade0b16700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc13a2b0 con 0x7faddc09d040 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 0x7faddc09d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient: get_auth_request con 0x7faddc09d040 auth_method 0 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 10 monclient(hunting): handle_auth_done global_id 5013 payload 824 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.341+0000 7faddb7fe700 1 -- 172.21.15.142:0/1187722854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faddc06c5a0 con 0x7faddc09d040 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7faddb7fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 msgr2=0x7faddc09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7faddb7fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 0x7faddc09d440 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fadcc007840 tx=0x7fadcc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc022640 con 0x7faddc09d040 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 10 monclient: dump: 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.352 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fadcc011590 con 0x7faddc09d040 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 msgr2=0x7faddc09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 0x7faddc09d440 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fadcc007840 tx=0x7fadcc017000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 msgr2=0x7faddc09d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fadc37fe700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc09d040 0x7faddc09d440 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fadcc007840 tx=0x7fadcc017000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 8, 2024-09-18T10:12:02.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.355 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.356 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:02.357 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.358 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.359 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.360 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:02.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.364 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:02.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:02.366 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadc8035ac0 msgr2=0x7fadc8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fadc8035ac0 0x7fadc8037f70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faddc06c9f0 tx=0x7fadd0001a20 comp rx=0 tx=0).stop 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 10 monclient: shutdown 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7faddbfff700 1 -- 172.21.15.142:0/1187722854 reap_dead start 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 msgr2=0x7faddc06bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faddc06a3b0 tx=0x7fadd4017000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 shutdown_connections 2024-09-18T10:12:02.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 --2- 172.21.15.142:0/1187722854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faddc138380 0x7faddc06bc90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 >> 172.21.15.142:0/1187722854 conn(0x7faddc0a5720 msgr2=0x7faddc0a4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 shutdown_connections 2024-09-18T10:12:02.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.345+0000 7fade0b16700 1 -- 172.21.15.142:0/1187722854 wait complete. 2024-09-18T10:12:02.369 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:02.369 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:12:02.369 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:12:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.473+0000 7f8aa4389700 10 monclient: get_monmap_and_config 2024-09-18T10:12:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.473+0000 7f8aa4389700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.473+0000 7f8aa4389700 10 monclient: monmap: 2024-09-18T10:12:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:02.478624+0000 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:02.478624+0000 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:02.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 Processor -- start 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 -- start start 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient: init 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a9c099cc0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c09c7d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a9c134750 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c09a8b0 con 0x7f8a9c134750 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c09aee0 con 0x7f8a9c099cc0 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a9c09a090 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa4389700 10 monclient(hunting): authenticate will time out at 1011.405113s 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa2926700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8a9c134750 0x7f8a9c136b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41152/0 (socket says 172.21.15.142:41152) 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36914/0 (socket says 172.21.15.142:36914) 2024-09-18T10:12:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa2926700 1 -- 172.21.15.142:0/3748013868 learned_addr learned my addr 172.21.15.142:0/3748013868 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:02.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): get_auth_request con 0x7f8a9c09c7d0 auth_method 0 2024-09-18T10:12:02.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa2926700 1 -- 172.21.15.142:0/3748013868 >> v1:172.21.15.142:6789/0 conn(0x7f8a9c134750 legacy=0x7f8a9c136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa2125700 1 -- 172.21.15.142:0/3748013868 >> v1:172.21.15.150:6789/0 conn(0x7f8a9c099cc0 legacy=0x7f8a9c09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient(hunting): handle_auth_done global_id 5016 payload 274 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 -- 172.21.15.142:0/3748013868 >> v1:172.21.15.150:6789/0 conn(0x7f8a9c099cc0 legacy=0x7f8a9c09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 -- 172.21.15.142:0/3748013868 >> v1:172.21.15.142:6789/0 conn(0x7f8a9c134750 legacy=0x7f8a9c136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 monclient: found mon.noname-a 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1924700 1 -- 172.21.15.142:0/3748013868 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c09b580 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.477+0000 7f8aa1123700 1 -- 172.21.15.142:0/3748013868 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494311232 0 0) 0x7f8a9c09a8b0 con 0x7f8a9c134750 2024-09-18T10:12:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:12:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1924700 1 -- 172.21.15.142:0/3748013868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c09c7d0 msgr2=0x7f8a9c09d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1924700 1 --2- 172.21.15.142:0/3748013868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0099e0 tx=0x7f8a8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657a6952f9499b54 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 1 -- 172.21.15.142:0/3748013868 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00bba0 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: dump: 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 1 -- 172.21.15.142:0/3748013868 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c00bd20 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 5 monclient: authenticate success, global_id 5016 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 1 -- 172.21.15.142:0/3748013868 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c011670 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 4 set_mon_vals no callback set 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa1123700 10 monclient: dump: 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 10 monclient: shutdown 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 -- 172.21.15.142:0/3748013868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c09c7d0 msgr2=0x7f8a9c09d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 --2- 172.21.15.142:0/3748013868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0099e0 tx=0x7f8a8c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 -- 172.21.15.142:0/3748013868 shutdown_connections 2024-09-18T10:12:02.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 --2- 172.21.15.142:0/3748013868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c09d6d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 -- 172.21.15.142:0/3748013868 >> 172.21.15.142:0/3748013868 conn(0x7f8a9c0a25d0 msgr2=0x7f8a9c0a29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 -- 172.21.15.142:0/3748013868 shutdown_connections 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.481+0000 7f8aa4389700 1 -- 172.21.15.142:0/3748013868 wait complete. 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient: monmap: 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:02.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:02.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 Processor -- start 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 -- start start 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient: init 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 0x7f8a9c13f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c099cc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c1339e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-b con 0x7f8a9c09c7d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:02.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 0x7f8a9c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): picked mon.noname-c con 0x7f8a9c134750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c128e60 con 0x7f8a9c099cc0 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c09bc70 con 0x7f8a9c09c7d0 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0a0cd0 con 0x7f8a9c134750 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa4389700 10 monclient(hunting): authenticate will time out at 1011.413712s 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 0x7f8a9c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 0x7f8a9c13f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 0x7f8a9c13f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36928/0 (socket says 172.21.15.142:36928) 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 0x7f8a9c06dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58802/0 (socket says 172.21.15.142:58802) 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 -- 172.21.15.142:0/227181710 learned_addr learned my addr 172.21.15.142:0/227181710 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:02.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): get_auth_request con 0x7f8a9c099cc0 auth_method 0 2024-09-18T10:12:02.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): get_auth_request con 0x7f8a9c134750 auth_method 0 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c1339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): get_auth_request con 0x7f8a9c09c7d0 auth_method 0 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient(hunting): handle_auth_done global_id 5022 payload 824 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 msgr2=0x7f8a9c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 0x7f8a9c06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a9c09c7d0 msgr2=0x7f8a9c1339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a9c09c7d0 0x7f8a9c1339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 monclient: found mon.noname-a 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2125700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c009420 con 0x7f8a9c099cc0 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.485+0000 7f8aa2926700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c134750 0x7f8a9c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa2125700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 msgr2=0x7f8a9c13f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa2125700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 0x7f8a9c13f620 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c570 tx=0x7f8a98007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800cea0 con 0x7f8a9c099cc0 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: dump: 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.508 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 5 monclient: authenticate success, global_id 5022 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 monclient: _renew_subs 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a980074f0 con 0x7f8a9c099cc0 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c140470 con 0x7f8a9c099cc0 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a98005a40 con 0x7f8a9c099cc0 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa0922700 4 set_mon_vals no callback set 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 10 monclient: dump: 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 monclient: _renew_subs 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c06f680 con 0x7f8a9c099cc0 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 0x7f8a9c06e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c128d20 con 0x7f8a9c133f20 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 0x7f8a9c06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient: get_auth_request con 0x7f8a9c133f20 auth_method 0 2024-09-18T10:12:02.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 10 monclient(hunting): handle_auth_done global_id 34130 payload 824 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8aa1924700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a9c129060 con 0x7f8a9c133f20 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8a9801b3b0 con 0x7f8a9c099cc0 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.489+0000 7f8a927fc700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a80035b10 0x7f8a80037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(67..67 src has 40..67) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f8a98047840 con 0x7f8a9c099cc0 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8aa2926700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a80035b10 0x7f8a80037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8aa2926700 10 monclient: get_auth_request con 0x7f8a80035b10 auth_method 0 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8aa1924700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 msgr2=0x7f8a9c06e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8aa1924700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 0x7f8a9c06e120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c09a4d0 tx=0x7f8a8c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c01f050 con 0x7f8a9c133f20 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:02.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 10 monclient: dump: 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8a8c020040 con 0x7f8a9c133f20 2024-09-18T10:12:02.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 msgr2=0x7f8a9c06e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 0x7f8a9c06e120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c09a4d0 tx=0x7f8a8c017000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 msgr2=0x7f8a9c06e120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8a927fc700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c133f20 0x7f8a9c06e120 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c09a4d0 tx=0x7f8a8c017000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.493+0000 7f8aa2926700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a80035b10 0x7f8a80037fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a94001820 tx=0x7f8a94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa4389700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa4389700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 0x7f8a9c135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa4389700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c064a50 con 0x7f8a9c06e660 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 0x7f8a9c135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient: get_auth_request con 0x7f8a9c06e660 auth_method 0 2024-09-18T10:12:02.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 10 monclient(hunting): handle_auth_done global_id 34133 payload 824 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 1 -- 172.21.15.142:0/227181710 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8a9c06ea30 con 0x7f8a9c06e660 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 msgr2=0x7f8a9c135b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.505+0000 7f8aa1924700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 0x7f8a9c135b30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0992e0 tx=0x7f8a8c022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c026bf0 con 0x7f8a9c06e660 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 10 monclient: dump: 2024-09-18T10:12:02.518 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f8a8c026d50 con 0x7f8a9c06e660 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:02.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 msgr2=0x7f8a9c135b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 0x7f8a9c135b30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0992e0 tx=0x7f8a8c022000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 msgr2=0x7f8a9c135b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8a927fc700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9c06e660 0x7f8a9c135b30 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0992e0 tx=0x7f8a8c022000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:12:02.520 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 8, 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.521 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.522 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:02.523 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.525 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.526 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.528 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.530 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:02.531 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.533 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:02.533 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:02.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a80035b10 msgr2=0x7f8a80037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8a80035b10 0x7f8a80037fc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a94001820 tx=0x7f8a94009040 comp rx=0 tx=0).stop 2024-09-18T10:12:02.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 10 monclient: shutdown 2024-09-18T10:12:02.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 msgr2=0x7f8a9c13f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 --2- 172.21.15.142:0/227181710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9c099cc0 0x7f8a9c13f620 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800c570 tx=0x7f8a98007e60 comp rx=0 tx=0).stop 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa2926700 1 -- 172.21.15.142:0/227181710 reap_dead start 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa2926700 1 -- 172.21.15.142:0/227181710 reap_dead start 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 shutdown_connections 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 >> 172.21.15.142:0/227181710 conn(0x7f8a9c0a25d0 msgr2=0x7f8a9c09ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 shutdown_connections 2024-09-18T10:12:02.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.509+0000 7f8aa4389700 1 -- 172.21.15.142:0/227181710 wait complete. 2024-09-18T10:12:02.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:02.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:12:02.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: get_monmap_and_config 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: monmap: 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:02.650169+0000 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:02.650169+0000 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.657 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 Processor -- start 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 -- start start 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: init 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd49c12e540 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd49c06ec30 0x7fd49c06f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-c con 0x7fd49c06ec30 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 0x7fd49c135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-b con 0x7fd49c133870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c064a30 con 0x7fd49c12e540 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c06c130 con 0x7fd49c06ec30 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c06be50 con 0x7fd49c133870 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.645+0000 7fd4a3365700 10 monclient(hunting): authenticate will time out at 1011.575872s 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2363700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd49c12e540 0x7fd49c12e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41158/0 (socket says 172.21.15.142:41158) 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2363700 1 -- 172.21.15.142:0/2031095073 learned_addr learned my addr 172.21.15.142:0/2031095073 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 0x7fd49c135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): get_auth_request con 0x7fd49c133870 auth_method 0 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/2031095073 >> v2:172.21.15.150:3300/0 conn(0x7fd49c06ec30 0x7fd49c06f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request con 0x7fd49c06ec30 auth_method 0 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_done global_id 14770 payload 274 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 -- 172.21.15.142:0/2031095073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 msgr2=0x7fd49c135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 0x7fd49c135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 -- 172.21.15.142:0/2031095073 >> v1:172.21.15.142:6789/0 conn(0x7fd49c12e540 legacy=0x7fd49c12e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 monclient: found mon.noname-c 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 -- 172.21.15.142:0/2031095073 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c06f570 con 0x7fd49c06ec30 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a2b64700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 0x7fd49c135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 -- 172.21.15.142:0/2031095073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c06ec30 msgr2=0x7fd49c06f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c06ec30 0x7fd49c06f030 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009e60 tx=0x7fd48c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b98b9fa81850c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 1 -- 172.21.15.142:0/2031095073 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c00e070 con 0x7fd49c06ec30 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 10 monclient: dump: 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.662 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a3365700 5 monclient: authenticate success, global_id 14770 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a3365700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 1 -- 172.21.15.142:0/2031095073 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c00bd70 con 0x7fd49c06ec30 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a1361700 1 -- 172.21.15.142:0/2031095073 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c012660 con 0x7fd49c06ec30 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.649+0000 7fd4a3365700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 4 set_mon_vals no callback set 2024-09-18T10:12:02.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1361700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1361700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1361700 10 monclient: dump: 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: shutdown 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- 172.21.15.142:0/2031095073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c06ec30 msgr2=0x7fd49c06f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c06ec30 0x7fd49c06f030 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009e60 tx=0x7fd48c009780 comp rx=0 tx=0).stop 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- 172.21.15.142:0/2031095073 shutdown_connections 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c133870 0x7fd49c135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- 172.21.15.142:0/2031095073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c06ec30 0x7fd49c06f030 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- 172.21.15.142:0/2031095073 >> 172.21.15.142:0/2031095073 conn(0x7fd49c071ae0 msgr2=0x7fd49c0a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- 172.21.15.142:0/2031095073 shutdown_connections 2024-09-18T10:12:02.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- 172.21.15.142:0/2031095073 wait complete. 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: monmap: 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:02.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 Processor -- start 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- start start 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: init 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 0x7fd49c09af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-c con 0x7fd49c06ec30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12e540 0x7fd49c12d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-b con 0x7fd49c12e540 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c133870 0x7fd49c1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): picked mon.noname-a con 0x7fd49c133870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c09bfc0 con 0x7fd49c133870 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0a1330 con 0x7fd49c12e540 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c118e20 con 0x7fd49c06ec30 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a3365700 10 monclient(hunting): authenticate will time out at 1011.582464s 2024-09-18T10:12:02.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 0x7fd49c09af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c133870 0x7fd49c1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 0x7fd49c09af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58838/0 (socket says 172.21.15.142:58838) 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 -- 172.21.15.142:0/1045515967 learned_addr learned my addr 172.21.15.142:0/1045515967 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): get_auth_request con 0x7fd49c06ec30 auth_method 0 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): get_auth_request con 0x7fd49c133870 auth_method 0 2024-09-18T10:12:02.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12e540 0x7fd49c12d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request con 0x7fd49c12e540 auth_method 0 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient(hunting): handle_auth_done global_id 34139 payload 824 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12e540 msgr2=0x7fd49c12d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12e540 0x7fd49c12d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c133870 msgr2=0x7fd49c1416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c133870 0x7fd49c1416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 monclient: found mon.noname-c 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2363700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009420 con 0x7fd49c06ec30 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a2b64700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd49c133870 0x7fd49c1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.653+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12e540 0x7fd49c12d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a2363700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 msgr2=0x7fd49c09af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a2363700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 0x7fd49c09af10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c5e0 tx=0x7fd490004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490005d40 con 0x7fd49c06ec30 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: dump: 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 5 monclient: authenticate success, global_id 34139 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4900070d0 con 0x7fd49c06ec30 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 monclient: _renew_subs 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c009a20 con 0x7fd49c06ec30 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490007490 con 0x7fd49c06ec30 2024-09-18T10:12:02.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a0b60700 4 set_mon_vals no callback set 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 10 monclient: dump: 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.679 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 monclient: _renew_subs 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c1188e0 con 0x7fd49c06ec30 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 0x7fd49c135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c064680 con 0x7fd49c12df00 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 0x7fd49c135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a1b62700 10 monclient: get_auth_request con 0x7fd49c12df00 auth_method 0 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a1b62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd4a1b62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.657+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd49001b3b0 con 0x7fd49c06ec30 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd488035ac0 0x7fd488037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_done global_id 14776 payload 824 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd49c072a80 con 0x7fd49c12df00 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(67..67 src has 40..67) v4 ==== 5173+0+0 (secure 0 0 0) 0x7fd490047b90 con 0x7fd49c06ec30 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a2b64700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd488035ac0 0x7fd488037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a2b64700 10 monclient: get_auth_request con 0x7fd488035ac0 auth_method 0 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 msgr2=0x7fd49c135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 0x7fd49c135f10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009e60 tx=0x7fd48c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c012e30 con 0x7fd49c12df00 2024-09-18T10:12:02.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 10 monclient: dump: 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd4a2b64700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd488035ac0 0x7fd488037f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd494009fd0 tx=0x7fd4940091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd48c01c440 con 0x7fd49c12df00 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 msgr2=0x7fd49c135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 0x7fd49c135f10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009e60 tx=0x7fd48c018000 comp rx=0 tx=0).stop 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 msgr2=0x7fd49c135f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.661+0000 7fd49a7fc700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c12df00 0x7fd49c135f10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a3365700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:02.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a3365700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 0x7fd49c142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a3365700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c142540 con 0x7fd49c141c00 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 0x7fd49c142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient: get_auth_request con 0x7fd49c141c00 auth_method 0 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 10 monclient(hunting): handle_auth_done global_id 14779 payload 824 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.673+0000 7fd4a1b62700 1 -- 172.21.15.142:0/1045515967 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd49c068360 con 0x7fd49c141c00 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a1b62700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 msgr2=0x7fd49c142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a1b62700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 0x7fd49c142000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0099a0 tx=0x7fd48c020040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c01dea0 con 0x7fd49c141c00 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 10 monclient: dump: 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fd48c012c20 con 0x7fd49c141c00 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 msgr2=0x7fd49c142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 0x7fd49c142000 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0099a0 tx=0x7fd48c020040 comp rx=0 tx=0).stop 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 msgr2=0x7fd49c142000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd49a7fc700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd49c141c00 0x7fd49c142000 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0099a0 tx=0x7fd48c020040 comp rx=0 tx=0).stop 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:02.687 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 40, 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 8, 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.688 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:02.690 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:02.691 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:02.692 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.695 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.698 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:02.699 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:02.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd488035ac0 msgr2=0x7fd488037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd488035ac0 0x7fd488037f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd494009fd0 tx=0x7fd4940091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 10 monclient: shutdown 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 msgr2=0x7fd49c09af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 --2- 172.21.15.142:0/1045515967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd49c06ec30 0x7fd49c09af10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c5e0 tx=0x7fd490004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a2b64700 1 -- 172.21.15.142:0/1045515967 reap_dead start 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a2b64700 1 -- 172.21.15.142:0/1045515967 reap_dead start 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 shutdown_connections 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 >> 172.21.15.142:0/1045515967 conn(0x7fd49c071ae0 msgr2=0x7fd49c09ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 shutdown_connections 2024-09-18T10:12:02.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.677+0000 7fd4a3365700 1 -- 172.21.15.142:0/1045515967 wait complete. 2024-09-18T10:12:02.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:02.702 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:12:02.702 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:12:02.702 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:12:02.702 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:12:02.702 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:12:02.702 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 msgr2=0x7fa3fc11f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f3fff700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 msgr2=0x7fa3fc11f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00d8c0 tx=0x7fa3ec008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00d8c0 tx=0x7fa3ec008040 comp rx=0 tx=0).stop 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f2ffd700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f2ffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 msgr2=0x7fa3fc11f240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.701+0000 7fa3f2ffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc09cd00 0x7fa3fc11f240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3dc03e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): picked mon.a con 0x7fa3fc09c930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3dc040cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): picked mon.b con 0x7fa3fc137770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 0x7fa3dc043ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): picked mon.c con 0x7fa3dc041650 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc044580 con 0x7fa3fc09c930 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0446c0 con 0x7fa3fc137770 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc044830 con 0x7fa3dc041650 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f2ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3faffd700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 msgr2=0x7fa3dc043ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3faffd700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 0x7fa3dc043ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3dc040cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3dc03e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa3fc137770 auth_method 0 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request con 0x7fa3fc09c930 auth_method 0 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.712 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_done global_id 34112 payload 293 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 msgr2=0x7fa3dc043ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 0x7fa3dc043ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 msgr2=0x7fa3dc03e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3dc03e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 monclient: found mon.b 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:02.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fa3dc0449f0 con 0x7fa3fc137770 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/2097442524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fa3fc09e1f0 con 0x7fa3fc137770 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.705+0000 7fa3f3fff700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3dc03e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:02.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3fb7fe700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 msgr2=0x7fa3dc040cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3fb7fe700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3dc040cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc071460 tx=0x7fa3f4012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4022070 con 0x7fa3fc137770 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 10 monclient: dump: 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.717 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.718 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:02.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f40098a0 con 0x7fa3fc137770 2024-09-18T10:12:02.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f97fa700 4 set_mon_vals no callback set 2024-09-18T10:12:02.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:02.709+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa3f4012430 con 0x7fa3fc137770 2024-09-18T10:12:02.719 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:12:02.719 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:12:02.719 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: monmap: 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:02.872480+0000 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:02.872480+0000 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.873 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:02.874 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 Processor -- start 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 -- start start 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: init 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 0x7f7134098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-b con 0x7f713409a830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f713409ac00 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f713413b6e0 0x7f713413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-a con 0x7f713413b6e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7134099820 con 0x7f713409ac00 2024-09-18T10:12:02.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713409afd0 con 0x7f713413b6e0 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7134099d50 con 0x7f713409a830 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713c33a700 10 monclient(hunting): authenticate will time out at 1011.797965s 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a0d6700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 msgr2=0x7f7134098bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:02.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a0d6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 0x7f7134098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f713413b6e0 0x7f713413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f71398d5700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f713409ac00 0x7f7134099110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:57402/0 (socket says 172.21.15.142:57402) 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f71398d5700 1 -- 172.21.15.142:0/252690306 learned_addr learned my addr 172.21.15.142:0/252690306 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f713413b6e0 0x7f713413dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36960/0 (socket says 172.21.15.142:36960) 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): get_auth_request con 0x7f713413b6e0 auth_method 0 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient(hunting): handle_auth_done global_id 5028 payload 274 2024-09-18T10:12:02.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 -- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 msgr2=0x7f7134098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 --2- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 0x7f7134098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 -- 172.21.15.142:0/252690306 >> v1:172.21.15.150:6789/0 conn(0x7f713409ac00 legacy=0x7f7134099110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 monclient: found mon.noname-a 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.869+0000 7f713a8d7700 1 -- 172.21.15.142:0/252690306 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713409dca0 con 0x7f713413b6e0 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713a8d7700 1 -- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 msgr2=0x7f713413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713a8d7700 1 --2- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f713413dac0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f712c0098e0 tx=0x7f712c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212d42a9c032a6d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 1 -- 172.21.15.142:0/252690306 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c00ba70 con 0x7f713413b6e0 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: dump: 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.884 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 5 monclient: authenticate success, global_id 5028 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 1 -- 172.21.15.142:0/252690306 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c00bbf0 con 0x7f713413b6e0 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 4 set_mon_vals no callback set 2024-09-18T10:12:02.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 1 -- 172.21.15.142:0/252690306 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c011420 con 0x7f713413b6e0 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f71390d4700 10 monclient: dump: 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.886 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 10 monclient: shutdown 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 -- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 msgr2=0x7f713413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 --2- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f713413dac0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f712c0098e0 tx=0x7f712c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 -- 172.21.15.142:0/252690306 shutdown_connections 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 --2- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f713413dac0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 --2- 172.21.15.142:0/252690306 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409a830 0x7f7134098bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 -- 172.21.15.142:0/252690306 >> 172.21.15.142:0/252690306 conn(0x7f71340a14e0 msgr2=0x7f71340a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 -- 172.21.15.142:0/252690306 shutdown_connections 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 1 -- 172.21.15.142:0/252690306 wait complete. 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 10 monclient: build_initial_monmap 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.873+0000 7f713c33a700 10 monclient: monmap: 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:02.888 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 Processor -- start 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 -- start start 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient: init 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 0x7f7134093380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-b con 0x7f713409a830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 0x7f71340943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-c con 0x7f713409ac00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:02.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): picked mon.noname-a con 0x7f713413b6e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713413d220 con 0x7f713413b6e0 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713413b430 con 0x7f713409a830 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7134139d70 con 0x7f713409ac00 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f71398d5700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 msgr2=0x7f71340943a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713c33a700 10 monclient(hunting): authenticate will time out at 1011.805083s 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f71398d5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 0x7f71340943a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:02.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36976/0 (socket says 172.21.15.142:36976) 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 -- 172.21.15.142:0/3274135862 learned_addr learned my addr 172.21.15.142:0/3274135862 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): get_auth_request con 0x7f713413b6e0 auth_method 0 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 0x7f7134093380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): get_auth_request con 0x7f713409a830 auth_method 0 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient(hunting): handle_auth_done global_id 5031 payload 824 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 -- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 msgr2=0x7f71340943a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 0x7f71340943a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 -- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 msgr2=0x7f7134093380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 0x7f7134093380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient: _finish_hunting 0 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 monclient: found mon.noname-a 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a8d7700 1 -- 172.21.15.142:0/3274135862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c009420 con 0x7f713413b6e0 2024-09-18T10:12:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.877+0000 7f713a0d6700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 0x7f7134093380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713a8d7700 1 -- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 msgr2=0x7f7134137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713a8d7700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f712c0091e0 tx=0x7f712c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c00be60 con 0x7f713413b6e0 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: dump: 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c01c8a0 con 0x7f713413b6e0 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 5 monclient: authenticate success, global_id 5031 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _renew_subs 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713413b150 con 0x7f713413b6e0 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c01cc60 con 0x7f713413b6e0 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 10 monclient: dump: 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f71388d3700 4 set_mon_vals no callback set 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _renew_subs 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713413a390 con 0x7f713413b6e0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71340696a0 con 0x7f713413b6e0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f712c01a450 con 0x7f713413b6e0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 0x7f7118037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(67..67 src has 40..67) v4 ==== 5173+0+0 (secure 0 0 0) 0x7f712c04a890 con 0x7f713413b6e0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713a0d6700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 0x7f7118037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:02.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.881+0000 7f713a0d6700 10 monclient: get_auth_request con 0x7f7118035ac0 auth_method 0 2024-09-18T10:12:02.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.885+0000 7f713a0d6700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 0x7f7118037f70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7124003fd0 tx=0x7f71240091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:02.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.889+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f712c0187c0 con 0x7f713413b6e0 2024-09-18T10:12:02.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.889+0000 7f712a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:02.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:02.889+0000 7f712a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:03.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f713c33a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:03.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f713c33a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:03.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f713c33a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:03.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7134067950 con 0x7f713413b6e0 2024-09-18T10:12:03.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f712a7fc700 1 -- 172.21.15.142:0/3274135862 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f712c0187c0 con 0x7f713413b6e0 2024-09-18T10:12:03.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f712a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:03.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.109+0000 7f712a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:03.115 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 msgr2=0x7f7118037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 0x7f7118037f70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7124003fd0 tx=0x7f71240091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 10 monclient: shutdown 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 msgr2=0x7f7134137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:03.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f712c0091e0 tx=0x7f712c016000 comp rx=0 tx=0).stop 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 shutdown_connections 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7118035ac0 0x7f7118037f70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f713413b6e0 0x7f7134137cc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f713409ac00 0x7f71340943a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 --2- 172.21.15.142:0/3274135862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f713409a830 0x7f7134093380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 >> 172.21.15.142:0/3274135862 conn(0x7f71340a14e0 msgr2=0x7f713409dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 shutdown_connections 2024-09-18T10:12:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:03.113+0000 7f713c33a700 1 -- 172.21.15.142:0/3274135862 wait complete. 2024-09-18T10:12:03.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:03.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:03.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:03.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:03.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:03.417+0000 7fa3f1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:04.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:04.421+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:04.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:04.421+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:05.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:05.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:05.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:05.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:06.130 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 10 monclient: get_monmap_and_config 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 10 monclient: build_initial_monmap 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 10 monclient: monmap: 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:06.241495+0000 2024-09-18T10:12:06.242 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:06.241495+0000 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:06.243 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 1 Processor -- start 2024-09-18T10:12:06.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 1 -- start start 2024-09-18T10:12:06.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.237+0000 7fa40158e700 10 monclient: init 2024-09-18T10:12:06.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:06.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3fc132b80 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3fc0a2670 0x7fa3fc132070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc0a2670 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3fc137230 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc132730 con 0x7fa3fc137230 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc096ca0 con 0x7fa3fc132b80 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc096f80 con 0x7fa3fc0a2670 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:06.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa40158e700 10 monclient(hunting): authenticate will time out at 1015.168789s 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa400d8d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa3fc137230 0x7fa3fc139610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41168/0 (socket says 172.21.15.142:41168) 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa400d8d700 1 -- 172.21.15.142:0/2253440760 learned_addr learned my addr 172.21.15.142:0/2253440760 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 1 --2- 172.21.15.142:0/2253440760 >> v2:172.21.15.150:3300/0 conn(0x7fa3fc0a2670 0x7fa3fc132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa3fc0a2670 auth_method 0 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:06.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_done global_id 14785 payload 274 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 1 -- 172.21.15.142:0/2253440760 >> v1:172.21.15.150:6789/0 conn(0x7fa3fc132b80 legacy=0x7fa3fc098790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 1 -- 172.21.15.142:0/2253440760 >> v1:172.21.15.142:6789/0 conn(0x7fa3fc137230 legacy=0x7fa3fc139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:06.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.241+0000 7fa3fb7fe700 1 -- 172.21.15.142:0/2253440760 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc098ea0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3fb7fe700 1 -- 172.21.15.142:0/2253440760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc0a2670 msgr2=0x7fa3fc132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:06.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3fb7fe700 1 --2- 172.21.15.142:0/2253440760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc132070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00098e0 tx=0x7fa3f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3db279b229de47af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 1 -- 172.21.15.142:0/2253440760 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f000ba70 con 0x7fa3fc0a2670 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: dump: 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.252 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 5 monclient: authenticate success, global_id 14785 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 1 -- 172.21.15.142:0/2253440760 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f000bbd0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:06.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 4 set_mon_vals no callback set 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 1 -- 172.21.15.142:0/2253440760 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0011420 con 0x7fa3fc0a2670 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa3faffd700 10 monclient: dump: 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:06.254 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 10 monclient: shutdown 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 -- 172.21.15.142:0/2253440760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc0a2670 msgr2=0x7fa3fc132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 --2- 172.21.15.142:0/2253440760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc132070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00098e0 tx=0x7fa3f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 -- 172.21.15.142:0/2253440760 shutdown_connections 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 --2- 172.21.15.142:0/2253440760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc132070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 -- 172.21.15.142:0/2253440760 >> 172.21.15.142:0/2253440760 conn(0x7fa3fc071700 msgr2=0x7fa3fc0a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 -- 172.21.15.142:0/2253440760 shutdown_connections 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 1 -- 172.21.15.142:0/2253440760 wait complete. 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 10 monclient: build_initial_monmap 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.245+0000 7fa40158e700 10 monclient: monmap: 2024-09-18T10:12:06.255 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:06.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 Processor -- start 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 -- start start 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient: init 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:06.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-a con 0x7fa3fc0a2670 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137230 0x7fa3fc0990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc137230 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 0x7fa3fc099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc099630 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc136310 con 0x7fa3fc0a2670 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc133080 con 0x7fa3fc137230 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc099fa0 con 0x7fa3fc099630 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa40158e700 10 monclient(hunting): authenticate will time out at 1015.177399s 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa400d8d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 msgr2=0x7fa3fc099a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:06.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa400d8d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 0x7fa3fc099a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36980/0 (socket says 172.21.15.142:36980) 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 -- 172.21.15.142:0/1391157913 learned_addr learned my addr 172.21.15.142:0/1391157913 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): get_auth_request con 0x7fa3fc0a2670 auth_method 0 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137230 0x7fa3fc0990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa3fc137230 auth_method 0 2024-09-18T10:12:06.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient(hunting): handle_auth_done global_id 5034 payload 824 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 -- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 msgr2=0x7fa3fc099a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 0x7fa3fc099a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 -- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137230 msgr2=0x7fa3fc0990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137230 0x7fa3fc0990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 monclient: found mon.noname-a 2024-09-18T10:12:06.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.249+0000 7fa3fbfff700 1 -- 172.21.15.142:0/1391157913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0009420 con 0x7fa3fc0a2670 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3fbfff700 1 -- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 msgr2=0x7fa3fc12f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3fbfff700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c5d0 tx=0x7fa3ec007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0117a0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: dump: 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:06.265 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec011900 con 0x7fa3fc0a2670 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec011cc0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:06.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3e3fff700 10 monclient: dump: 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa3f9ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 5 monclient: authenticate success, global_id 5034 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 10 monclient: _renew_subs 2024-09-18T10:12:06.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc09a550 con 0x7fa3fc0a2670 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 10 monclient: _renew_subs 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.253+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc13ab00 con 0x7fa3fc0a2670 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa40158e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa40158e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa40158e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc072a80 con 0x7fa3fc0a2670 2024-09-18T10:12:06.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa3ec012630 con 0x7fa3fc0a2670 2024-09-18T10:12:06.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa3e3fff700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 0x7fa3e8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:06.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(68..68 src has 40..68) v4 ==== 5288+0+0 (secure 0 0 0) 0x7fa3ec0482e0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa3fb7fe700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 0x7fa3e8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:06.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.257+0000 7fa3fb7fe700 10 monclient: get_auth_request con 0x7fa3e8035ac0 auth_method 0 2024-09-18T10:12:06.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.261+0000 7fa3fb7fe700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 0x7fa3e8037f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00030c0 tx=0x7fa3f0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:06.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.269+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa3ec0108c0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.269+0000 7fa3e3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:06.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.269+0000 7fa3e3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:06.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:06.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:06.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:06.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa40158e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa40158e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa40158e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa3fc09c070 con 0x7fa3fc0a2670 2024-09-18T10:12:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa3e3fff700 1 -- 172.21.15.142:0/1391157913 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa3ec0102e0 con 0x7fa3fc0a2670 2024-09-18T10:12:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa3e3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.497+0000 7fa3e3fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:06.504 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 msgr2=0x7fa3e8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 0x7fa3e8037f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00030c0 tx=0x7fa3f0016000 comp rx=0 tx=0).stop 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 10 monclient: shutdown 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 msgr2=0x7fa3fc12f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00c5d0 tx=0x7fa3ec007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 shutdown_connections 2024-09-18T10:12:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3e8035ac0 0x7fa3e8037f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3fc099630 0x7fa3fc099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137230 0x7fa3fc0990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 --2- 172.21.15.142:0/1391157913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc0a2670 0x7fa3fc12f210 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 >> 172.21.15.142:0/1391157913 conn(0x7fa3fc071700 msgr2=0x7fa3fc1315a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 shutdown_connections 2024-09-18T10:12:06.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:06.501+0000 7fa40158e700 1 -- 172.21.15.142:0/1391157913 wait complete. 2024-09-18T10:12:07.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:07.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:07.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:07.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:08.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:08.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:08.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:08.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:09.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:09.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:09.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:09.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:09.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:09.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: get_monmap_and_config 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: build_initial_monmap 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: monmap: 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:09.625859+0000 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:09.625859+0000 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:09.627 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:09.628 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:09.628 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 Processor -- start 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 -- start start 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: init 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f900c0a1cf0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:09.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient(hunting): picked mon.noname-b con 0x7f900c12f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f900c0a02d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c09fb90 con 0x7f900c0a02d0 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f900c09d990 con 0x7f900c0a1cf0 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900c09dc70 con 0x7f900c12f810 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.621+0000 7f901297d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901297d700 10 monclient(hunting): authenticate will time out at 1018.552009s 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901117a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 msgr2=0x7f900c12f030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901217c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f900c0a02d0 0x7f900c139420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41178/0 (socket says 172.21.15.142:41178) 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901117a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:09.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901217c700 1 -- 172.21.15.142:0/3376554769 learned_addr learned my addr 172.21.15.142:0/3376554769 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:09.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901217c700 1 -- 172.21.15.142:0/3376554769 >> v1:172.21.15.142:6789/0 conn(0x7f900c0a02d0 legacy=0x7f900c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:09.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.625+0000 7f901197b700 1 -- 172.21.15.142:0/3376554769 >> v1:172.21.15.150:6789/0 conn(0x7f900c0a1cf0 legacy=0x7f900c09f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:09.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.829+0000 7f901117a700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 msgr2=0x7f900c12f030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:09.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:09.829+0000 7f901117a700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:10.233+0000 7f901117a700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 msgr2=0x7f900c12f030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:10.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:10.233+0000 7f901117a700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:10.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:10.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:10.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:10.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:11.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:11.037+0000 7f901117a700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 msgr2=0x7f900c12f030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:11.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:11.037+0000 7f901117a700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:11.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:11.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:11.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:11.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:12.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:12.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:12.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:12.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:12.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): tick 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 msgr2=0x7f900c12f030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f900c12f810 0x7f900c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 -- 172.21.15.142:0/3376554769 >> v1:172.21.15.150:6789/0 conn(0x7f900c0a1cf0 legacy=0x7f900c09f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 -- 172.21.15.142:0/3376554769 >> v1:172.21.15.142:6789/0 conn(0x7f900c0a02d0 legacy=0x7f900c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient: _add_conns ranks=[3,4] 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.142:3300/0 conn(0x7f8ff0003860 0x7f8ff0005cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): picked mon.noname-a con 0x7f8ff0003860 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.150:3300/0 conn(0x7f8ff0006320 0x7f8ff0008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f8ff0006320 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:12.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 -- 172.21.15.142:0/3376554769 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900c09dc70 con 0x7f8ff0003860 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 1 -- 172.21.15.142:0/3376554769 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff0008e50 con 0x7f8ff0006320 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f8fff7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.142:3300/0 conn(0x7f8ff0003860 0x7f8ff0005cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.150:3300/0 conn(0x7f8ff0006320 0x7f8ff0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): get_auth_request con 0x7f8ff0003860 auth_method 0 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): get_auth_request con 0x7f8ff0006320 auth_method 0 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:12.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:12.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient(hunting): handle_auth_done global_id 5037 payload 274 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 -- 172.21.15.142:0/3376554769 >> v2:172.21.15.150:3300/0 conn(0x7f8ff0006320 msgr2=0x7f8ff0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.150:3300/0 conn(0x7f8ff0006320 0x7f8ff0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 monclient: found mon.noname-a 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 -- 172.21.15.142:0/3376554769 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c0a20c0 con 0x7f8ff0003860 2024-09-18T10:12:12.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901117a700 1 --2- 172.21.15.142:0/3376554769 >> v2:172.21.15.150:3300/0 conn(0x7f8ff0006320 0x7f8ff0008790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:12.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ff0003860 msgr2=0x7f8ff0005cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:12.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.625+0000 7f901197b700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ff0003860 0x7f8ff0005cd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00062f0 tx=0x7f9000007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1ca9b3b72293a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 1 -- 172.21.15.142:0/3376554769 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90000073b0 con 0x7f8ff0003860 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: dump: 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:12.637 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: _finish_auth 0 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 5 monclient: authenticate success, global_id 5037 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 1 -- 172.21.15.142:0/3376554769 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9000007510 con 0x7f8ff0003860 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 1 -- 172.21.15.142:0/3376554769 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000005a60 con 0x7f8ff0003860 2024-09-18T10:12:12.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 4 set_mon_vals no callback set 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f9010979700 10 monclient: dump: 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:12.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 10 monclient: shutdown 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 -- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ff0003860 msgr2=0x7f8ff0005cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ff0003860 0x7f8ff0005cd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8ff00062f0 tx=0x7f9000007dc0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901217c700 1 -- 172.21.15.142:0/3376554769 reap_dead start 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 -- 172.21.15.142:0/3376554769 shutdown_connections 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 --2- 172.21.15.142:0/3376554769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ff0003860 0x7f8ff0005cd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 -- 172.21.15.142:0/3376554769 >> 172.21.15.142:0/3376554769 conn(0x7f900c071820 msgr2=0x7f900c0a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 -- 172.21.15.142:0/3376554769 shutdown_connections 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 1 -- 172.21.15.142:0/3376554769 wait complete. 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 10 monclient: build_initial_monmap 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.629+0000 7f901297d700 10 monclient: monmap: 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:12.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 Processor -- start 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 -- start start 2024-09-18T10:12:12.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient: init 2024-09-18T10:12:12.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:12.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:12.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 0x7f900c09c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): picked mon.noname-c con 0x7f8ff0003860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): picked mon.noname-b con 0x7f900c06bb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): picked mon.noname-a con 0x7f900c06c480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c06ec40 con 0x7f900c06c480 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c0a5530 con 0x7f900c06bb30 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900c09cdc0 con 0x7f8ff0003860 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901297d700 10 monclient(hunting): authenticate will time out at 1021.561837s 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901197b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 msgr2=0x7f900c09c230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:12.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901197b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 0x7f900c09c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38530/0 (socket says 172.21.15.142:38530) 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:55940/0 (socket says 172.21.15.142:55940) 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 1 -- 172.21.15.142:0/1300922105 learned_addr learned my addr 172.21.15.142:0/1300922105 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): get_auth_request con 0x7f900c06c480 auth_method 0 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): get_auth_request con 0x7f900c06bb30 auth_method 0 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:12.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient(hunting): handle_auth_done global_id 14794 payload 824 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 -- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 msgr2=0x7f900c09c230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 0x7f900c09c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 -- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 msgr2=0x7f900c148200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient: _finish_hunting 0 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 monclient: found mon.noname-b 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901117a700 1 -- 172.21.15.142:0/1300922105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900c09d990 con 0x7f900c06bb30 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.633+0000 7f901217c700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901117a700 1 -- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 msgr2=0x7f900c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901117a700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f90080039d0 tx=0x7f9008009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008007e80 con 0x7f900c06bb30 2024-09-18T10:12:12.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: dump: 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:12.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900800f040 con 0x7f900c06bb30 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 5 monclient: authenticate success, global_id 14794 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 10 monclient: _renew_subs 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c06f200 con 0x7f900c06bb30 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008013400 con 0x7f900c06bb30 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:12.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffdffb700 10 monclient: dump: 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:12.652 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f8ffffff700 4 set_mon_vals no callback set 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 10 monclient: _renew_subs 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.637+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c06c8b0 con 0x7f900c06bb30 2024-09-18T10:12:12.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9008013770 con 0x7f900c06bb30 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901297d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901297d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901297d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f8ffdffb700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 0x7f8ff0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 40..68) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f90080489d0 con 0x7f900c06bb30 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c06ee40 con 0x7f900c06bb30 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901197b700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 0x7f8ff0038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.641+0000 7f901197b700 10 monclient: get_auth_request con 0x7f8ff00368c0 auth_method 0 2024-09-18T10:12:12.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.645+0000 7f901197b700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 0x7f8ff0038d70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9000007e40 tx=0x7f9000015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:12.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.653+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f900801bba0 con 0x7f900c06bb30 2024-09-18T10:12:12.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.653+0000 7f8ffdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:12.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.653+0000 7f8ffdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f901297d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f901297d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f901297d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f900c072a80 con 0x7f900c06bb30 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f8ffdffb700 1 -- 172.21.15.142:0/1300922105 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f900802a620 con 0x7f900c06bb30 2024-09-18T10:12:12.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f8ffdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:12.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.881+0000 7f8ffdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:12.886 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:12.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 msgr2=0x7f8ff0038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 0x7f8ff0038d70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9000007e40 tx=0x7f9000015000 comp rx=0 tx=0).stop 2024-09-18T10:12:12.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 10 monclient: shutdown 2024-09-18T10:12:12.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 msgr2=0x7f900c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f90080039d0 tx=0x7f9008009780 comp rx=0 tx=0).stop 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 shutdown_connections 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ff00368c0 0x7f8ff0038d70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f900c06c480 0x7f900c148200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f900c06bb30 0x7f900c06bf40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 --2- 172.21.15.142:0/1300922105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8ff0003860 0x7f900c09c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 >> 172.21.15.142:0/1300922105 conn(0x7f900c071820 msgr2=0x7f900c13a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 shutdown_connections 2024-09-18T10:12:12.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:12.885+0000 7f901297d700 1 -- 172.21.15.142:0/1300922105 wait complete. 2024-09-18T10:12:13.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:13.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:13.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:13.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:13.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:13.417+0000 7fa3f1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:14.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:14.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:14.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:14.417+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:15.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:15.417+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:15.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:15.421+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:15.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.013+0000 7f5d07242700 10 monclient: get_monmap_and_config 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.013+0000 7f5d07242700 10 monclient: build_initial_monmap 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.013+0000 7f5d07242700 10 monclient: monmap: 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:16.020 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:16.019573+0000 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:16.019573+0000 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:16.021 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 Processor -- start 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 -- start start 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient: init 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:12:16.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-a con 0x7f5d000a0500 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d0006ee10 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-c con 0x7f5d00135160 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0006d720 con 0x7f5d0006ee10 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0009e3c0 con 0x7f5d000a0500 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0009edc0 con 0x7f5d00135160 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d07242700 10 monclient(hunting): authenticate will time out at 1024.945145s 2024-09-18T10:12:16.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38542/0 (socket says 172.21.15.142:38542) 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 learned_addr learned my addr 172.21.15.142:0/2422271790 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5d000a0500 auth_method 0 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 10 monclient(hunting): get_auth_request con 0x7f5d00135160 auth_method 0 2024-09-18T10:12:16.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:16.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.017+0000 7f5d05a3f700 1 -- 172.21.15.142:0/2422271790 >> v1:172.21.15.150:6789/0 conn(0x7f5d0006ee10 legacy=0x7f5d0006f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:16.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5d000a0500 auth_method 0 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 10 monclient(hunting): get_auth_request con 0x7f5d00135160 auth_method 0 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:16.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:16.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.221+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:16.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:16.421+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:16.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:16.421+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:16.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5d000a0500 auth_method 0 2024-09-18T10:12:16.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:16.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 10 monclient(hunting): get_auth_request con 0x7f5d00135160 auth_method 0 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:16.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:16.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:16.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:16.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:16.625+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:17.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:17.421+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:17.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:17.421+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:17.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:17.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5d000a0500 auth_method 0 2024-09-18T10:12:17.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 10 monclient(hunting): get_auth_request con 0x7f5d00135160 auth_method 0 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:17.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:17.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:17.429+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:18.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220559399 0 0) 0x7f5d0006d720 con 0x7f5d0006ee10 2024-09-18T10:12:18.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): my global_id is 14797 2024-09-18T10:12:18.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec003570 con 0x7f5d0006ee10 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735548732 0 0) 0x7f5cec003570 con 0x7f5d0006ee10 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 msgr2=0x7f5d00137540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.150:3300/0 conn(0x7f5d00135160 0x7f5d00137540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 msgr2=0x7f5d0009e700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 --2- 172.21.15.142:0/2422271790 >> v2:172.21.15.142:3300/0 conn(0x7f5d000a0500 0x7f5d0009e700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0006ce10 con 0x7f5d0006ee10 2024-09-18T10:12:18.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d07242700 5 monclient: authenticate success, global_id 14797 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f5cf40017e0 con 0x7f5d0006ee10 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: dump: 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.238 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 >> v1:172.21.15.150:6789/0 conn(0x7f5d0006ee10 legacy=0x7f5d0006f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 0x7f5cec004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): picked mon.a con 0x7f5cec003c50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec004810 0x7f5cec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): picked mon.b con 0x7f5cec004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cec005440 0x7f5cec00bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): picked mon.c con 0x7f5cec005440 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0009e3c0 con 0x7f5cec003c50 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0009edc0 con 0x7f5cec004810 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cec00c460 con 0x7f5cec005440 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d0523e700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d07242700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:18.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 0x7f5cec004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cec005440 msgr2=0x7f5cec00bb00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5cec003c50 auth_method 0 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06a41700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cec005440 0x7f5cec00bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec004810 0x7f5cec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): get_auth_request con 0x7f5cec004810 auth_method 0 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient(hunting): handle_auth_done global_id 14797 payload 293 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cec005440 msgr2=0x7f5cec00bb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cec005440 0x7f5cec00bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec004810 msgr2=0x7f5cec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec004810 0x7f5cec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 monclient: found mon.a 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06a41700 1 -- 172.21.15.142:0/2422271790 reap_dead start 2024-09-18T10:12:18.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec003570 con 0x7f5cec003c50 2024-09-18T10:12:18.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d06240700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.229+0000 7f5d05a3f700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec004810 0x7f5cec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.233+0000 7f5d06240700 1 -- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 msgr2=0x7f5cec004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.233+0000 7f5d06240700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 0x7f5cec004060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009cd0 tx=0x7f5cf8009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d1406e42773e9a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8007bb0 con 0x7f5cec003c50 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: dump: 2024-09-18T10:12:18.244 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf800f040 con 0x7f5cec003c50 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:18.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 1 -- 172.21.15.142:0/2422271790 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8017470 con 0x7f5cec003c50 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 4 set_mon_vals no callback set 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d0523e700 10 monclient: dump: 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.246 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 10 monclient: shutdown 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 1 -- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 msgr2=0x7f5cec004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 0x7f5cec004060 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009cd0 tx=0x7f5cf8009040 comp rx=0 tx=0).stop 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.237+0000 7f5d07242700 1 -- 172.21.15.142:0/2422271790 shutdown_connections 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 --2- 172.21.15.142:0/2422271790 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cec003c50 0x7f5cec004060 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- 172.21.15.142:0/2422271790 >> 172.21.15.142:0/2422271790 conn(0x7f5d00096a50 msgr2=0x7f5d00071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- 172.21.15.142:0/2422271790 shutdown_connections 2024-09-18T10:12:18.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- 172.21.15.142:0/2422271790 wait complete. 2024-09-18T10:12:18.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient: build_initial_monmap 2024-09-18T10:12:18.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient: monmap: 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 Processor -- start 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- start start 2024-09-18T10:12:18.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient: init 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 0x7f5d001341b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-b con 0x7f5cec003c50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-a con 0x7f5d001346f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 0x7f5d0011f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): picked mon.noname-c con 0x7f5d0011f1b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00134ac0 con 0x7f5d001346f0 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00120d00 con 0x7f5cec003c50 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d001222e0 con 0x7f5d0011f1b0 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d07242700 10 monclient(hunting): authenticate will time out at 1027.170172s 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06a41700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 msgr2=0x7f5d0011f600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06a41700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 0x7f5d0011f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38586/0 (socket says 172.21.15.142:38586) 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 -- 172.21.15.142:0/129007719 learned_addr learned my addr 172.21.15.142:0/129007719 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): get_auth_request con 0x7f5d001346f0 auth_method 0 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 0x7f5d001341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): get_auth_request con 0x7f5cec003c50 auth_method 0 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient(hunting): handle_auth_done global_id 5043 payload 824 2024-09-18T10:12:18.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 -- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 msgr2=0x7f5d0011f600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 0x7f5d0011f600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 -- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 msgr2=0x7f5d001341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 0x7f5d001341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 monclient: found mon.noname-a 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d05a3f700 1 -- 172.21.15.142:0/129007719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00142410 con 0x7f5d001346f0 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.241+0000 7f5d06240700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 0x7f5d001341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d05a3f700 1 -- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 msgr2=0x7f5d0011ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d05a3f700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4007070 tx=0x7f5cf4007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4013810 con 0x7f5d001346f0 2024-09-18T10:12:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: dump: 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 5 monclient: authenticate success, global_id 5043 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _renew_subs 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0006fa40 con 0x7f5d001346f0 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf4013970 con 0x7f5d001346f0 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4013d30 con 0x7f5d001346f0 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5cf27fc700 10 monclient: dump: 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _renew_subs 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d04a3d700 4 set_mon_vals no callback set 2024-09-18T10:12:18.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d00121380 con 0x7f5d001346f0 2024-09-18T10:12:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.245+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d00120870 con 0x7f5d001346f0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.249+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5cf400c5d0 con 0x7f5d001346f0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.249+0000 7f5cf27fc700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 0x7f5ce4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.249+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(68..68 src has 40..68) v4 ==== 5288+0+0 (secure 0 0 0) 0x7f5cf4048af0 con 0x7f5d001346f0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.249+0000 7f5d06240700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 0x7f5ce4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.249+0000 7f5d06240700 10 monclient: get_auth_request con 0x7f5ce4035a70 auth_method 0 2024-09-18T10:12:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.253+0000 7f5d06240700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 0x7f5ce4037f20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80097f0 tx=0x7f5cf801d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.257+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5cf4023aa0 con 0x7f5d001346f0 2024-09-18T10:12:18.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.257+0000 7f5cf27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:18.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.257+0000 7f5cf27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:18.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:18.421+0000 7fa3f1ffb700 10 monclient: tick 2024-09-18T10:12:18.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:18.421+0000 7fa3f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.485+0000 7f5d07242700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:18.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.485+0000 7f5d07242700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.485+0000 7f5d07242700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.485+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5d00072a80 con 0x7f5d001346f0 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5cf27fc700 1 -- 172.21.15.142:0/129007719 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f5cf4023680 con 0x7f5d001346f0 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5cf27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5cf27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:18.492 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 msgr2=0x7f5ce4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 0x7f5ce4037f20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5cf80097f0 tx=0x7f5cf801d000 comp rx=0 tx=0).stop 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 10 monclient: shutdown 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 msgr2=0x7f5d0011ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4007070 tx=0x7f5cf4007a90 comp rx=0 tx=0).stop 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 shutdown_connections 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce4035a70 0x7f5ce4037f20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5d0011f1b0 0x7f5d0011f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5d001346f0 0x7f5d0011ec70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 --2- 172.21.15.142:0/129007719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cec003c50 0x7f5d001341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 >> 172.21.15.142:0/129007719 conn(0x7f5d00096a50 msgr2=0x7f5d000a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:18.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 shutdown_connections 2024-09-18T10:12:18.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.489+0000 7f5d07242700 1 -- 172.21.15.142:0/129007719 wait complete. 2024-09-18T10:12:18.506 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:12:18.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:12:18.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.609+0000 7fa237291700 10 monclient: get_monmap_and_config 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.609+0000 7fa237291700 10 monclient: build_initial_monmap 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.609+0000 7fa237291700 10 monclient: monmap: 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:18.614407+0000 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:18.614407+0000 2024-09-18T10:12:18.615 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:18.616 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 Processor -- start 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 -- start start 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient: init 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 0x7fa23009e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): picked mon.noname-b con 0x7fa23009cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2300a02e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2300a0dc0 0x7fa230139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): picked mon.noname-c con 0x7fa2300a0dc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23009da60 con 0x7fa2300a02e0 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa23009d1a0 con 0x7fa2300a0dc0 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23009e980 con 0x7fa23009cdd0 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa237291700 10 monclient(hunting): authenticate will time out at 1027.541051s 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23502d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 msgr2=0x7fa23009e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23482c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa2300a02e0 0x7fa2300a06b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33708/0 (socket says 172.21.15.142:33708) 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23482c700 1 -- 172.21.15.142:0/2115724401 learned_addr learned my addr 172.21.15.142:0/2115724401 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:18.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23502d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 0x7fa23009e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 --2- 172.21.15.142:0/2115724401 >> v2:172.21.15.150:3300/0 conn(0x7fa2300a0dc0 0x7fa230139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): get_auth_request con 0x7fa2300a0dc0 auth_method 0 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient(hunting): handle_auth_done global_id 14803 payload 274 2024-09-18T10:12:18.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 -- 172.21.15.142:0/2115724401 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 msgr2=0x7fa23009e440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 --2- 172.21.15.142:0/2115724401 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 0x7fa23009e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 -- 172.21.15.142:0/2115724401 >> v1:172.21.15.142:6789/0 conn(0x7fa2300a02e0 legacy=0x7fa2300a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 monclient: found mon.noname-c 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:18.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.613+0000 7fa23582e700 1 -- 172.21.15.142:0/2115724401 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23009d780 con 0x7fa2300a0dc0 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa23582e700 1 -- 172.21.15.142:0/2115724401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 msgr2=0x7fa230139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa23582e700 1 --2- 172.21.15.142:0/2115724401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa230139a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe469ad7a1a92166 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 1 -- 172.21.15.142:0/2115724401 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22800e070 con 0x7fa2300a0dc0 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: dump: 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.626 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 1 -- 172.21.15.142:0/2115724401 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa228012430 con 0x7fa2300a0dc0 2024-09-18T10:12:18.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 5 monclient: authenticate success, global_id 14803 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 4 set_mon_vals no callback set 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 1 -- 172.21.15.142:0/2115724401 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2280127d0 con 0x7fa2300a0dc0 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa227fff700 10 monclient: dump: 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.628 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: shutdown 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- 172.21.15.142:0/2115724401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 msgr2=0x7fa230139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 --2- 172.21.15.142:0/2115724401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa230139a80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228009780 comp rx=0 tx=0).stop 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- 172.21.15.142:0/2115724401 shutdown_connections 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 --2- 172.21.15.142:0/2115724401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa230139a80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 --2- 172.21.15.142:0/2115724401 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa23009cdd0 0x7fa23009e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- 172.21.15.142:0/2115724401 >> 172.21.15.142:0/2115724401 conn(0x7fa230071680 msgr2=0x7fa2300a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- 172.21.15.142:0/2115724401 shutdown_connections 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- 172.21.15.142:0/2115724401 wait complete. 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: build_initial_monmap 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: monmap: 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:18.630 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 Processor -- start 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 1 -- start start 2024-09-18T10:12:18.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: init 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.617+0000 7fa237291700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): picked mon.noname-a con 0x7fa23009cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2300a02e0 0x7fa23008cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): picked mon.noname-c con 0x7fa2300a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa23008d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): picked mon.noname-b con 0x7fa2300a0dc0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23009be90 con 0x7fa23009cdd0 2024-09-18T10:12:18.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa230090750 con 0x7fa2300a0dc0 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa230090470 con 0x7fa2300a02e0 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23482c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2300a02e0 msgr2=0x7fa23008cf10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient(hunting): authenticate will time out at 1027.548199s 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23482c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2300a02e0 0x7fa23008cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38602/0 (socket says 172.21.15.142:38602) 2024-09-18T10:12:18.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 -- 172.21.15.142:0/771846102 learned_addr learned my addr 172.21.15.142:0/771846102 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): get_auth_request con 0x7fa23009cdd0 auth_method 0 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23582e700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa23008d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23582e700 10 monclient(hunting): get_auth_request con 0x7fa2300a0dc0 auth_method 0 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23582e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23582e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23582e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient(hunting): handle_auth_done global_id 5046 payload 824 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2300a02e0 msgr2=0x7fa23008cf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2300a02e0 0x7fa23008cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 msgr2=0x7fa23008d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2300a0dc0 0x7fa23008d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 monclient: found mon.noname-a 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009a20 con 0x7fa23009cdd0 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 msgr2=0x7fa230090e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa23502d700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00b7e0 tx=0x7fa22c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c015070 con 0x7fa23009cdd0 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:18.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: dump: 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.640 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 5 monclient: authenticate success, global_id 5046 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient: _renew_subs 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22c003e20 con 0x7fa23009cdd0 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa230090190 con 0x7fa23009cdd0 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c006380 con 0x7fa23009cdd0 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2277fe700 4 set_mon_vals no callback set 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa2257fa700 10 monclient: dump: 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient: _renew_subs 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.621+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23008f480 con 0x7fa23009cdd0 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa237291700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa237291700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 0x7fa23008dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300648c0 con 0x7fa23002ae80 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 0x7fa23008dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient: get_auth_request con 0x7fa23002ae80 auth_method 0 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 10 monclient(hunting): handle_auth_done global_id 5049 payload 824 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa23006d320 con 0x7fa23002ae80 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa22c006730 con 0x7fa23009cdd0 2024-09-18T10:12:18.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa2257fa700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa214035a70 0x7fa214037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(68..68 src has 40..68) v4 ==== 5288+0+0 (secure 0 0 0) 0x7fa22c047ff0 con 0x7fa23009cdd0 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23582e700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa214035a70 0x7fa214037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23582e700 10 monclient: get_auth_request con 0x7fa214035a70 auth_method 0 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 msgr2=0x7fa23008dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.625+0000 7fa23482c700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 0x7fa23008dae0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa2200048c0 tx=0x7fa220007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220005e00 con 0x7fa23002ae80 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 10 monclient: dump: 2024-09-18T10:12:18.645 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa2200070d0 con 0x7fa23002ae80 2024-09-18T10:12:18.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 msgr2=0x7fa23008dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 0x7fa23008dae0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa2200048c0 tx=0x7fa220007f90 comp rx=0 tx=0).stop 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 msgr2=0x7fa23008dae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa2257fa700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23002ae80 0x7fa23008dae0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.629+0000 7fa23582e700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa214035a70 0x7fa214037f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa237291700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa237291700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 0x7fa23005ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa237291700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa230090970 con 0x7fa23005e890 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 0x7fa23005ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient: get_auth_request con 0x7fa23005e890 auth_method 0 2024-09-18T10:12:18.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 10 monclient(hunting): handle_auth_done global_id 5052 payload 824 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.637+0000 7fa23482c700 1 -- 172.21.15.142:0/771846102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa23008f480 con 0x7fa23005e890 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa23482c700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 msgr2=0x7fa23005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa23482c700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 0x7fa23005ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa220014000 tx=0x7fa220015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220023070 con 0x7fa23005e890 2024-09-18T10:12:18.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 10 monclient: dump: 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7fa220005bf0 con 0x7fa23005e890 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 msgr2=0x7fa23005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 0x7fa23005ec90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa220014000 tx=0x7fa220015000 comp rx=0 tx=0).stop 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 msgr2=0x7fa23005ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa2257fa700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23005e890 0x7fa23005ec90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.652 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 44, 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:18.653 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:18.654 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:18.655 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:18.656 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:18.657 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:18.658 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.659 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:18.661 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:18.663 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.665 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:18.666 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:12:18.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:18.668 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa214035a70 msgr2=0x7fa214037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa214035a70 0x7fa214037f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa228009e60 tx=0x7fa228018000 comp rx=0 tx=0).stop 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 10 monclient: shutdown 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 msgr2=0x7fa230090e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa23582e700 1 -- 172.21.15.142:0/771846102 reap_dead start 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00b7e0 tx=0x7fa22c009220 comp rx=0 tx=0).stop 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 shutdown_connections 2024-09-18T10:12:18.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 --2- 172.21.15.142:0/771846102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa23009cdd0 0x7fa230090e50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 >> 172.21.15.142:0/771846102 conn(0x7fa230071680 msgr2=0x7fa23009c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:18.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 shutdown_connections 2024-09-18T10:12:18.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.641+0000 7fa237291700 1 -- 172.21.15.142:0/771846102 wait complete. 2024-09-18T10:12:18.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:12:18.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:12:18.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:12:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: get_monmap_and_config 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: build_initial_monmap 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: monmap: 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:18.780994+0000 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:18.780994+0000 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:18.783 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 Processor -- start 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 -- start start 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: init 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3cc09a6a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc06f7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc09b6c0 0x7fd3cc13d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-b con 0x7fd3cc09b6c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:18.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3cc06fbf0 con 0x7fd3cc09a6a0 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3cc09dae0 con 0x7fd3cc06f7f0 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc099bc0 con 0x7fd3cc09b6c0 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d2213700 10 monclient(hunting): authenticate will time out at 1027.706898s 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d1a12700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc09b6c0 msgr2=0x7fd3cc13d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d1a12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc09b6c0 0x7fd3cc13d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:56014/0 (socket says 172.21.15.142:56014) 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 learned_addr learned my addr 172.21.15.142:0/3717756408 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d1211700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd3cc09a6a0 0x7fd3cc09aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37452/0 (socket says 172.21.15.142:37452) 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): get_auth_request con 0x7fd3cc06f7f0 auth_method 0 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_done global_id 14809 payload 274 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc09b6c0 msgr2=0x7fd3cc13d6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc09b6c0 0x7fd3cc13d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 >> v1:172.21.15.150:6789/0 conn(0x7fd3cc09a6a0 legacy=0x7fd3cc09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 monclient: found mon.noname-c 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.777+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc0998e0 con 0x7fd3cc06f7f0 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.781+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 msgr2=0x7fd3cc09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:18.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.781+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 msgr2=0x7fd3cc09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:18.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.781+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fd3bc0099e0 tx=0x7fd3bc009250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:18.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.781+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:18.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d0a10700 10 monclient: get_auth_request con 0x7fd3cc06f7f0 auth_method 0 2024-09-18T10:12:18.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d0a10700 0 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:12:18.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient: ms_handle_reset current mon v2:172.21.15.150:3300/0 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 msgr2=0x7fd3cc09b180 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.150:3300/0 conn(0x7fd3cc06f7f0 0x7fd3cc09b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient: _add_conns ranks=[3,0] 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.142:3300/0 conn(0x7fd3b8003550 0x7fd3b80059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd3b8003550 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3b80062f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b800a250 con 0x7fd3b80062f0 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b800b410 con 0x7fd3b8003550 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3c3fff700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.150:3300/0 2024-09-18T10:12:18.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3717756408 >> v2:172.21.15.142:3300/0 conn(0x7fd3b8003550 0x7fd3b80059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): get_auth_request con 0x7fd3b8003550 auth_method 0 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient(hunting): handle_auth_done global_id 14809 payload 293 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 1 -- 172.21.15.142:0/3717756408 >> v1:172.21.15.142:6789/0 conn(0x7fd3b80062f0 legacy=0x7fd3b80087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:18.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient: _finish_hunting 0 2024-09-18T10:12:18.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 1 monclient: found mon.noname-a 2024-09-18T10:12:18.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:18.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:18.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.985+0000 7fd3d1211700 1 -- 172.21.15.142:0/3717756408 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b800b640 con 0x7fd3b8003550 2024-09-18T10:12:18.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d1211700 1 -- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3b8003550 msgr2=0x7fd3b80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:18.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3b8003550 0x7fd3b80059c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4005b90 tx=0x7fd3c4005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1842b5e92693bce server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:18.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4005dd0 con 0x7fd3b8003550 2024-09-18T10:12:18.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: dump: 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.996 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c400c9e0 con 0x7fd3b8003550 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 5 monclient: authenticate success, global_id 14809 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 1 -- 172.21.15.142:0/3717756408 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400cda0 con 0x7fd3b8003550 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 4 set_mon_vals no callback set 2024-09-18T10:12:18.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3c3fff700 10 monclient: dump: 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 10 monclient: shutdown 2024-09-18T10:12:18.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 -- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3b8003550 msgr2=0x7fd3b80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3b8003550 0x7fd3b80059c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4005b90 tx=0x7fd3c4005250 comp rx=0 tx=0).stop 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d1a12700 1 -- 172.21.15.142:0/3717756408 reap_dead start 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 -- 172.21.15.142:0/3717756408 shutdown_connections 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3717756408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3b8003550 0x7fd3b80059c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 -- 172.21.15.142:0/3717756408 >> 172.21.15.142:0/3717756408 conn(0x7fd3cc070ef0 msgr2=0x7fd3cc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 -- 172.21.15.142:0/3717756408 shutdown_connections 2024-09-18T10:12:18.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 1 -- 172.21.15.142:0/3717756408 wait complete. 2024-09-18T10:12:19.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 10 monclient: build_initial_monmap 2024-09-18T10:12:19.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.989+0000 7fd3d2213700 10 monclient: monmap: 2024-09-18T10:12:19.004 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:19.004 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:19.004 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:19.005 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:19.006 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 Processor -- start 2024-09-18T10:12:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 -- start start 2024-09-18T10:12:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient: init 2024-09-18T10:12:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3b8003550 0x7fd3cc1462d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-b con 0x7fd3b8003550 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-a con 0x7fd3cc146810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc146be0 0x7fd3cc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc146be0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc06e940 con 0x7fd3cc146810 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc094830 con 0x7fd3b8003550 2024-09-18T10:12:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0a0960 con 0x7fd3cc146be0 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d2213700 10 monclient(hunting): authenticate will time out at 1027.921037s 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1a12700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc146be0 msgr2=0x7fd3cc06c9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1a12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc146be0 0x7fd3cc06c9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38656/0 (socket says 172.21.15.142:38656) 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3716178607 learned_addr learned my addr 172.21.15.142:0/3716178607 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): get_auth_request con 0x7fd3cc146810 auth_method 0 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3b8003550 0x7fd3cc1462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): get_auth_request con 0x7fd3b8003550 auth_method 0 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient(hunting): handle_auth_done global_id 5055 payload 824 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc146be0 msgr2=0x7fd3cc06c9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3cc146be0 0x7fd3cc06c9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3b8003550 msgr2=0x7fd3cc1462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3b8003550 0x7fd3cc1462d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient: _finish_hunting 0 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 monclient: found mon.noname-a 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:19.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.993+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3cc126ae0 con 0x7fd3cc146810 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d0a10700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 msgr2=0x7fd3cc06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d0a10700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc0099b0 tx=0x7fd3bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc017070 con 0x7fd3cc146810 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: dump: 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.011 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 5 monclient: authenticate success, global_id 5055 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 monclient: _renew_subs 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3bc0107a0 con 0x7fd3cc146810 2024-09-18T10:12:19.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3cc1270a0 con 0x7fd3cc146810 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc00bee0 con 0x7fd3cc146810 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c37fe700 4 set_mon_vals no callback set 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 10 monclient: dump: 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 monclient: _renew_subs 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3cc127c70 con 0x7fd3cc146810 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 0x7fd3cc06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc0648a0 con 0x7fd3cc06cf10 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 0x7fd3cc06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd3bc01a390 con 0x7fd3cc146810 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient: get_auth_request con 0x7fd3cc06cf10 auth_method 0 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 10 monclient(hunting): handle_auth_done global_id 14815 payload 824 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3d1211700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd3cc126c70 con 0x7fd3cc06cf10 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd3b0035aa0 0x7fd3b0037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:18.997+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(68..68 src has 40..68) v4 ==== 5288+0+0 (secure 0 0 0) 0x7fd3bc048b70 con 0x7fd3cc146810 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3d1a12700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd3b0035aa0 0x7fd3b0037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3d1a12700 10 monclient: get_auth_request con 0x7fd3b0035aa0 auth_method 0 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3d1211700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 msgr2=0x7fd3cc06d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 0x7fd3cc06d2e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4005b90 tx=0x7fd3c4015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4007830 con 0x7fd3cc06cf10 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 10 monclient: dump: 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3d1a12700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd3b0035aa0 0x7fd3b0037f50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc099d50 tx=0x7fd3c8006bb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd3c4007990 con 0x7fd3cc06cf10 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 msgr2=0x7fd3cc06d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 0x7fd3cc06d2e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4005b90 tx=0x7fd3c4015000 comp rx=0 tx=0).stop 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 msgr2=0x7fd3cc06d2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.001+0000 7fd3c17fa700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc06cf10 0x7fd3cc06d2e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d2213700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 0x7fd3cc092c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d2213700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3cc06fbf0 con 0x7fd3cc092800 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 0x7fd3cc092c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient: get_auth_request con 0x7fd3cc092800 auth_method 0 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 10 monclient(hunting): handle_auth_done global_id 14818 payload 824 2024-09-18T10:12:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.013+0000 7fd3d1211700 1 -- 172.21.15.142:0/3716178607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd3cc072a80 con 0x7fd3cc092800 2024-09-18T10:12:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3d1211700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 msgr2=0x7fd3cc092c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3d1211700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 0x7fd3cc092c00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0a0b40 tx=0x7fd3c4022070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c402e070 con 0x7fd3cc092800 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 10 monclient: dump: 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.023 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.024 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:19.024 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.024 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.024 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.024 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fd3c4007620 con 0x7fd3cc092800 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 msgr2=0x7fd3cc092c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 0x7fd3cc092c00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0a0b40 tx=0x7fd3c4022070 comp rx=0 tx=0).stop 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 msgr2=0x7fd3cc092c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:19.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.017+0000 7fd3c17fa700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc092800 0x7fd3cc092c00 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0a0b40 tx=0x7fd3c4022070 comp rx=0 tx=0).stop 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 44, 2024-09-18T10:12:19.026 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:19.027 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:19.027 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:12:19.027 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:19.027 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:12:19.027 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:19.028 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:19.029 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:19.030 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:19.031 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:19.032 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.033 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:19.034 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.035 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.036 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:19.037 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:19.038 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:19.039 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.039 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:19.039 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:19.040 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:19.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.041 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:19.041 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:19.041 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:19.041 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:19.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:19.043 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:19.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd3b0035aa0 msgr2=0x7fd3b0037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd3b0035aa0 0x7fd3b0037f50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc099d50 tx=0x7fd3c8006bb0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 10 monclient: shutdown 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 msgr2=0x7fd3cc06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d1a12700 1 -- 172.21.15.142:0/3716178607 reap_dead start 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc0099b0 tx=0x7fd3bc008040 comp rx=0 tx=0).stop 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 shutdown_connections 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 --2- 172.21.15.142:0/3716178607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3cc146810 0x7fd3cc06c490 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 >> 172.21.15.142:0/3716178607 conn(0x7fd3cc070ef0 msgr2=0x7fd3cc09c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 shutdown_connections 2024-09-18T10:12:19.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:19.021+0000 7fd3d2213700 1 -- 172.21.15.142:0/3716178607 wait complete. 2024-09-18T10:12:19.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:12:19.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:12:19.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.217+0000 7fa3f2ffd700 1 -- 172.21.15.150:0/2097442524 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mon_command_ack([{"prefix": "osd create"}]=0 v69) v1 ==== 58+0+2 (secure 0 0 0) 0x7fa3f4010d30 con 0x7fa3fc137770 2024-09-18T10:12:19.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.217+0000 7fa3f2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T10:12:19.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.217+0000 7fa3f2ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 msgr2=0x7fa3dc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 0x7fa3dc037f70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4003a30 tx=0x7fa3e4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 10 monclient: shutdown 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 msgr2=0x7fa3dc040cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3dc040cd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc071460 tx=0x7fa3f4012000 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 shutdown_connections 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3dc041650 0x7fa3dc043ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3fc137770 0x7fa3dc040cd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa3fc09c930 0x7fa3dc03e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 --2- 172.21.15.150:0/2097442524 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa3dc035ac0 0x7fa3dc037f70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 >> 172.21.15.150:0/2097442524 conn(0x7fa3fc068360 msgr2=0x7fa3fc09af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:19.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 shutdown_connections 2024-09-18T10:12:19.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.221+0000 7fa4017a0700 1 -- 172.21.15.150:0/2097442524 wait complete. 2024-09-18T10:12:19.237 INFO:tasks.workunit.client.0.smithi150.stderr:+ o1=8 2024-09-18T10:12:19.238 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph osd create 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: get_monmap_and_config 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: build_initial_monmap 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: monmap: 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:19.333256+0000 2024-09-18T10:12:19.333 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:19.333256+0000 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:19.334 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 1 Processor -- start 2024-09-18T10:12:19.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 1 -- start start 2024-09-18T10:12:19.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: init 2024-09-18T10:12:19.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:19.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:12:19.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f16f409cf10 0x7f16f40a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-a con 0x7f16f409cf10 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16f40a0750 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16f40a0b50 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f409e6e0 con 0x7f16f40a0750 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.329+0000 7f16f9fd2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f409d2e0 con 0x7f16f40a0b50 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f9fd2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f9fd2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f409e2e0 con 0x7f16f409cf10 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f9fd2700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f9fd2700 10 monclient(hunting): authenticate will time out at 1021.428052s 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f3fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f16f40a0b50 0x7f16f413a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57774/0 (socket says 172.21.15.150:57774) 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f3fff700 1 -- 172.21.15.150:0/2709425012 learned_addr learned my addr 172.21.15.150:0/2709425012 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 --2- 172.21.15.150:0/2709425012 >> v2:172.21.15.142:3300/0 conn(0x7f16f409cf10 0x7f16f40a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): get_auth_request con 0x7f16f409cf10 auth_method 0 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient(hunting): handle_auth_done global_id 5058 payload 274 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 -- 172.21.15.150:0/2709425012 >> v1:172.21.15.150:6789/0 conn(0x7f16f40a0b50 legacy=0x7f16f413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 -- 172.21.15.150:0/2709425012 >> v1:172.21.15.142:6789/0 conn(0x7f16f40a0750 legacy=0x7f16f4136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 monclient: found mon.noname-a 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:19.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:19.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 -- 172.21.15.150:0/2709425012 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40a1f90 con 0x7f16f409cf10 2024-09-18T10:12:19.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 -- 172.21.15.150:0/2709425012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 msgr2=0x7f16f40a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:19.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.333+0000 7f16f37fe700 1 --2- 172.21.15.150:0/2709425012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f40a0210 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f16e40098e0 tx=0x7f16e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cd1c25e4b3f71c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 1 -- 172.21.15.150:0/2709425012 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e400ba70 con 0x7f16f409cf10 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: dump: 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.341 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 1 -- 172.21.15.150:0/2709425012 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e400bbd0 con 0x7f16f409cf10 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 5 monclient: authenticate success, global_id 5058 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 1 -- 172.21.15.150:0/2709425012 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4011420 con 0x7f16f409cf10 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 4 set_mon_vals no callback set 2024-09-18T10:12:19.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f2ffd700 10 monclient: dump: 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.343 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: shutdown 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/2709425012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 msgr2=0x7f16f40a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/2709425012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f40a0210 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f16e40098e0 tx=0x7f16e40091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/2709425012 shutdown_connections 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/2709425012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f40a0210 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/2709425012 >> 172.21.15.150:0/2709425012 conn(0x7f16f40a5820 msgr2=0x7f16f40a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/2709425012 shutdown_connections 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/2709425012 wait complete. 2024-09-18T10:12:19.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: build_initial_monmap 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: monmap: 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 Processor -- start 2024-09-18T10:12:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- start start 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: init 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f4141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-a con 0x7f16f409cf10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 0x7f16f4141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-c con 0x7f16f40a0750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): picked mon.noname-b con 0x7f16f40a0b50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f40994e0 con 0x7f16f409cf10 2024-09-18T10:12:19.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f409c0d0 con 0x7f16f40a0b50 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4091830 con 0x7f16f40a0750 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.337+0000 7f16f9fd2700 10 monclient(hunting): authenticate will time out at 1021.435974s 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16ebfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 msgr2=0x7f16f4141fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16ebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 0x7f16f4141fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44782/0 (socket says 172.21.15.150:44782) 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 -- 172.21.15.150:0/285874057 learned_addr learned my addr 172.21.15.150:0/285874057 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): get_auth_request con 0x7f16f40a0b50 auth_method 0 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f4141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): get_auth_request con 0x7f16f409cf10 auth_method 0 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:19.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient(hunting): handle_auth_done global_id 14821 payload 824 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 -- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 msgr2=0x7f16f4141fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 0x7f16f4141fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 -- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 msgr2=0x7f16f4141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f4141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 monclient: found mon.noname-b 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:19.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:19.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4009420 con 0x7f16f40a0b50 2024-09-18T10:12:19.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 -- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 msgr2=0x7f16f41424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:19.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f3fff700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00ccc0 tx=0x7f16ec005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec00ab40 con 0x7f16f40a0b50 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: dump: 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:19.355 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 5 monclient: authenticate success, global_id 14821 2024-09-18T10:12:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec00acc0 con 0x7f16f40a0b50 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 10 monclient: _renew_subs 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f4091260 con 0x7f16f40a0b50 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec007340 con 0x7f16f40a0b50 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16eb7fe700 10 monclient: dump: 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f27fc700 4 set_mon_vals no callback set 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 10 monclient: _renew_subs 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.341+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f408fa90 con 0x7f16f40a0b50 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f9fd2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f9fd2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f9fd2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f16ec0076f0 con 0x7f16f40a0b50 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16eb7fe700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 0x7f16d4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(69..69 src has 40..69) v4 ==== 5403+0+0 (secure 0 0 0) 0x7f16ec048af0 con 0x7f16f40a0b50 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f37fe700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 0x7f16d4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.345+0000 7f16f37fe700 10 monclient: get_auth_request con 0x7f16d4035a70 auth_method 0 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.349+0000 7f16f37fe700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 0x7f16d4037f20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16e40030c0 tx=0x7f16e4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.353+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.353+0000 7f16eb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.353+0000 7f16eb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:19.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.529+0000 7f16f9fd2700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:12:19.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.529+0000 7f16f9fd2700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T10:12:19.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.529+0000 7f16f9fd2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:19.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:19.529+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:20.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.225+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v70) v1 ==== 58+0+2 (secure 0 0 0) 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:20.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.225+0000 7f16eb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T10:12:20.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.225+0000 7f16eb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:20.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.337+0000 7f16f9fd2700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:12:20.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.337+0000 7f16f9fd2700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:12:20.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.337+0000 7f16f9fd2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:20.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:20.337+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:21.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.237+0000 7f16eb7fe700 1 -- 172.21.15.150:0/285874057 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v71) v1 ==== 58+0+3 (secure 0 0 0) 0x7f16f4092030 con 0x7f16f40a0b50 2024-09-18T10:12:21.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.237+0000 7f16eb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T10:12:21.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.237+0000 7f16eb7fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 msgr2=0x7f16d4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 0x7f16d4037f20 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16e40030c0 tx=0x7f16e4016000 comp rx=0 tx=0).stop 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 10 monclient: shutdown 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 msgr2=0x7f16f41424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:21.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00ccc0 tx=0x7f16ec005400 comp rx=0 tx=0).stop 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 shutdown_connections 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f16d4035a70 0x7f16d4037f20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f40a0b50 0x7f16f41424e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16f40a0750 0x7f16f4141fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 --2- 172.21.15.150:0/285874057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16f409cf10 0x7f16f4141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 >> 172.21.15.150:0/285874057 conn(0x7f16f40a5820 msgr2=0x7f16f409b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 shutdown_connections 2024-09-18T10:12:21.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.241+0000 7f16f9fd2700 1 -- 172.21.15.150:0/285874057 wait complete. 2024-09-18T10:12:21.264 INFO:tasks.workunit.client.0.smithi150.stderr:+ o2=10 2024-09-18T10:12:21.264 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 8 1 host=host1 root=foo 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.369+0000 7fafa128f700 10 monclient: get_monmap_and_config 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.369+0000 7fafa128f700 10 monclient: build_initial_monmap 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.369+0000 7fafa128f700 10 monclient: monmap: 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:21.374876+0000 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:21.374876+0000 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:21.375 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 Processor -- start 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 -- start start 2024-09-18T10:12:21.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient: init 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf9c09c7c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7faf9c0978d0 0x7faf9c09d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-a con 0x7faf9c0978d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf9c137600 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9c071630 con 0x7faf9c09c7c0 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9c096ba0 con 0x7faf9c137600 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:21.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9c097cd0 con 0x7faf9c0978d0 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9b7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7faf9c137600 0x7faf9c1399e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57786/0 (socket says 172.21.15.150:57786) 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9b7fe700 1 -- 172.21.15.150:0/1072022310 learned_addr learned my addr 172.21.15.150:0/1072022310 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7fafa128f700 10 monclient(hunting): authenticate will time out at 1023.470203s 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 1 --2- 172.21.15.150:0/1072022310 >> v2:172.21.15.142:3300/0 conn(0x7faf9c0978d0 0x7faf9c09d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): get_auth_request con 0x7faf9c0978d0 auth_method 0 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:21.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_done global_id 5064 payload 274 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 1 -- 172.21.15.150:0/1072022310 >> v1:172.21.15.150:6789/0 conn(0x7faf9c137600 legacy=0x7faf9c1399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 1 -- 172.21.15.150:0/1072022310 >> v1:172.21.15.142:6789/0 conn(0x7faf9c09c7c0 legacy=0x7faf9c09cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 1 monclient: found mon.noname-a 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.373+0000 7faf9a7fc700 1 -- 172.21.15.150:0/1072022310 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9c09e180 con 0x7faf9c0978d0 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf9a7fc700 1 -- 172.21.15.150:0/1072022310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c0978d0 msgr2=0x7faf9c09d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:21.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf9a7fc700 1 --2- 172.21.15.150:0/1072022310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c0978d0 0x7faf9c09d2a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faf900098e0 tx=0x7faf900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e4f8e5f59171d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:21.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 1 -- 172.21.15.150:0/1072022310 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000ba70 con 0x7faf9c0978d0 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: dump: 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.383 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:21.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 1 -- 172.21.15.150:0/1072022310 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000bbd0 con 0x7faf9c0978d0 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 5 monclient: authenticate success, global_id 5064 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 1 -- 172.21.15.150:0/1072022310 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90011420 con 0x7faf9c0978d0 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 4 set_mon_vals no callback set 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7faf99ffb700 10 monclient: dump: 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.385 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:21.386 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 10 monclient: shutdown 2024-09-18T10:12:21.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:21.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 1 -- 172.21.15.150:0/1072022310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c0978d0 msgr2=0x7faf9c09d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.377+0000 7fafa128f700 1 --2- 172.21.15.150:0/1072022310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c0978d0 0x7faf9c09d2a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7faf900098e0 tx=0x7faf900091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- 172.21.15.150:0/1072022310 shutdown_connections 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 --2- 172.21.15.150:0/1072022310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c0978d0 0x7faf9c09d2a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- 172.21.15.150:0/1072022310 >> 172.21.15.150:0/1072022310 conn(0x7faf9c068590 msgr2=0x7faf9c0a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- 172.21.15.150:0/1072022310 shutdown_connections 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- 172.21.15.150:0/1072022310 wait complete. 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient: build_initial_monmap 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient: monmap: 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:21.390 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 Processor -- start 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- start start 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient: init 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:21.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-b con 0x7faf9c0978d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c09c7c0 0x7faf9c134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-a con 0x7faf9c09c7c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 0x7faf9c134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): picked mon.noname-c con 0x7faf9c137600 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c136eb0 con 0x7faf9c09c7c0 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:21.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c0a17d0 con 0x7faf9c0978d0 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c1369c0 con 0x7faf9c137600 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7fafa128f700 10 monclient(hunting): authenticate will time out at 1023.479800s 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7faf9b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 msgr2=0x7faf9c134ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7faf9affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.381+0000 7faf9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 0x7faf9c134ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44790/0 (socket says 172.21.15.150:44790) 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 -- 172.21.15.150:0/1025160590 learned_addr learned my addr 172.21.15.150:0/1025160590 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:21.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): get_auth_request con 0x7faf9c0978d0 auth_method 0 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c09c7c0 0x7faf9c134960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): get_auth_request con 0x7faf9c09c7c0 auth_method 0 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:21.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient(hunting): handle_auth_done global_id 14824 payload 824 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 -- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 msgr2=0x7faf9c134ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 0x7faf9c134ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 -- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c09c7c0 msgr2=0x7faf9c134960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c09c7c0 0x7faf9c134960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 monclient: found mon.noname-b 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:21.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009420 con 0x7faf9c0978d0 2024-09-18T10:12:21.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 -- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 msgr2=0x7faf9c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf9affd700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c5d0 tx=0x7faf8c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0117a0 con 0x7faf9c0978d0 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: dump: 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.398 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 5 monclient: authenticate success, global_id 14824 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c011900 con 0x7faf9c0978d0 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 10 monclient: _renew_subs 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9c09e450 con 0x7faf9c0978d0 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c011cc0 con 0x7faf9c0978d0 2024-09-18T10:12:21.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf997fa700 4 set_mon_vals no callback set 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7faf82ffd700 10 monclient: dump: 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:21.400 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 10 monclient: _renew_subs 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.385+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9c136530 con 0x7faf9c0978d0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7fafa128f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7fafa128f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7fafa128f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9c139130 con 0x7faf9c0978d0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7faf8c012630 con 0x7faf9c0978d0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7faf82ffd700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 0x7faf88037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(71..71 src has 40..71) v4 ==== 5633+0+0 (secure 0 0 0) 0x7faf8c048450 con 0x7faf9c0978d0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7faf9a7fc700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 0x7faf88037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.389+0000 7faf9a7fc700 10 monclient: get_auth_request con 0x7faf88035a70 auth_method 0 2024-09-18T10:12:21.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.393+0000 7faf9a7fc700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 0x7faf88037f20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf900093f0 tx=0x7faf90016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:21.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.397+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7faf8c00e290 con 0x7faf9c0978d0 2024-09-18T10:12:21.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.397+0000 7faf82ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:21.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.397+0000 7faf82ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:21.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.565+0000 7fafa128f700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:21.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.565+0000 7fafa128f700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:21.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.565+0000 7fafa128f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:21.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:21.565+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7faf9c063ed0 con 0x7faf9c0978d0 2024-09-18T10:12:22.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.237+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 add item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo} to crush map v72) v1 ==== 203+0+0 (secure 0 0 0) 0x7faf8c0102e0 con 0x7faf9c0978d0 2024-09-18T10:12:22.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.241+0000 7faf82ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.241+0000 7faf82ffd700 10 monclient: _finish_command 2 = system:0 add item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo} to crush map 2024-09-18T10:12:22.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.345+0000 7fafa128f700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.345+0000 7fafa128f700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.345+0000 7fafa128f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.345+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7faf9c137140 con 0x7faf9c0978d0 2024-09-18T10:12:22.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.349+0000 7faf82ffd700 1 -- 172.21.15.150:0/1025160590 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change v72) v1 ==== 201+0+0 (secure 0 0 0) 0x7faf8c010930 con 0x7faf9c0978d0 2024-09-18T10:12:22.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.349+0000 7faf82ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.349+0000 7faf82ffd700 10 monclient: _finish_command 3 = system:0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T10:12:22.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 msgr2=0x7faf88037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 0x7faf88037f20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf900093f0 tx=0x7faf90016000 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 10 monclient: shutdown 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 msgr2=0x7faf9c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf8c00c5d0 tx=0x7faf8c007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 shutdown_connections 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035a70 0x7faf88037f20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf9c137600 0x7faf9c134ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf9c09c7c0 0x7faf9c134960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 --2- 172.21.15.150:0/1025160590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf9c0978d0 0x7faf9c12fff0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 >> 172.21.15.150:0/1025160590 conn(0x7faf9c068590 msgr2=0x7faf9c0a0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 shutdown_connections 2024-09-18T10:12:22.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.353+0000 7fafa128f700 1 -- 172.21.15.150:0/1025160590 wait complete. 2024-09-18T10:12:22.358 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T10:12:22.366 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 8 1 host=host1 root=foo 2024-09-18T10:12:22.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:22.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: build_initial_monmap 2024-09-18T10:12:22.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: monmap: 2024-09-18T10:12:22.474 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:22.474 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:22.474438+0000 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:22.474438+0000 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:22.475 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 Processor -- start 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 -- start start 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: init 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f101c06b5a0 0x7f101c1321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-a con 0x7f101c06b5a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f101c132700 0x7f101c132b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-c con 0x7f101c132700 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f101c09e770 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101c06aac0 con 0x7f101c09e770 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c06a5e0 con 0x7f101c06b5a0 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101c06b970 con 0x7f101c132700 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.469+0000 7f1021d3a700 10 monclient(hunting): authenticate will time out at 1024.567989s 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101affd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f101c132700 0x7f101c132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101bfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f101c09e770 0x7f101c09ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57800/0 (socket says 172.21.15.150:57800) 2024-09-18T10:12:22.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101bfff700 1 -- 172.21.15.150:0/1196286916 learned_addr learned my addr 172.21.15.150:0/1196286916 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101affd700 10 monclient(hunting): get_auth_request con 0x7f101c132700 auth_method 0 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 --2- 172.21.15.150:0/1196286916 >> v2:172.21.15.142:3300/0 conn(0x7f101c06b5a0 0x7f101c1321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): get_auth_request con 0x7f101c06b5a0 auth_method 0 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:22.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient(hunting): handle_auth_done global_id 5070 payload 274 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 -- 172.21.15.150:0/1196286916 >> v2:172.21.15.150:3300/0 conn(0x7f101c132700 msgr2=0x7f101c132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 --2- 172.21.15.150:0/1196286916 >> v2:172.21.15.150:3300/0 conn(0x7f101c132700 0x7f101c132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 -- 172.21.15.150:0/1196286916 >> v1:172.21.15.150:6789/0 conn(0x7f101c09e770 legacy=0x7f101c09ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:22.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 -- 172.21.15.150:0/1196286916 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c09fb10 con 0x7f101c06b5a0 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 -- 172.21.15.150:0/1196286916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c06b5a0 msgr2=0x7f101c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.473+0000 7f101b7fe700 1 --2- 172.21.15.150:0/1196286916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c06b5a0 0x7f101c1321c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f101c0723f0 tx=0x7f10100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447fd2f3b5e6e1bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 1 -- 172.21.15.150:0/1196286916 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000b8c0 con 0x7f101c06b5a0 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: dump: 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.483 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 5 monclient: authenticate success, global_id 5070 2024-09-18T10:12:22.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 1 -- 172.21.15.150:0/1196286916 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101000ba20 con 0x7f101c06b5a0 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 1 -- 172.21.15.150:0/1196286916 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000bdc0 con 0x7f101c06b5a0 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 4 set_mon_vals no callback set 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f101a7fc700 10 monclient: dump: 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.485 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 10 monclient: shutdown 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- 172.21.15.150:0/1196286916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c06b5a0 msgr2=0x7f101c1321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 --2- 172.21.15.150:0/1196286916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c06b5a0 0x7f101c1321c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f101c0723f0 tx=0x7f10100091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- 172.21.15.150:0/1196286916 shutdown_connections 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 --2- 172.21.15.150:0/1196286916 >> v2:172.21.15.150:3300/0 conn(0x7f101c132700 0x7f101c132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 --2- 172.21.15.150:0/1196286916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c06b5a0 0x7f101c1321c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- 172.21.15.150:0/1196286916 >> 172.21.15.150:0/1196286916 conn(0x7f101c071b60 msgr2=0x7f101c0a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:22.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- 172.21.15.150:0/1196286916 shutdown_connections 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- 172.21.15.150:0/1196286916 wait complete. 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 10 monclient: build_initial_monmap 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 10 monclient: monmap: 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 Processor -- start 2024-09-18T10:12:22.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 1 -- start start 2024-09-18T10:12:22.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.477+0000 7f1021d3a700 10 monclient: init 2024-09-18T10:12:22.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:22.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 0x7f101c141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-c con 0x7f101c06b5a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-b con 0x7f101c09e770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c132700 0x7f101c11ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): picked mon.noname-a con 0x7f101c132700 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c0a3010 con 0x7f101c132700 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c11e690 con 0x7f101c09e770 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c11e3b0 con 0x7f101c06b5a0 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f1021d3a700 10 monclient(hunting): authenticate will time out at 1024.576735s 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 msgr2=0x7f101c141580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 0x7f101c141580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44812/0 (socket says 172.21.15.150:44812) 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 -- 172.21.15.150:0/50268610 learned_addr learned my addr 172.21.15.150:0/50268610 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): get_auth_request con 0x7f101c09e770 auth_method 0 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c132700 0x7f101c11ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): get_auth_request con 0x7f101c132700 auth_method 0 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient(hunting): handle_auth_done global_id 14830 payload 824 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 -- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 msgr2=0x7f101c141580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 0x7f101c141580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 -- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c132700 msgr2=0x7f101c11ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c132700 0x7f101c11ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 monclient: found mon.noname-b 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1010009420 con 0x7f101c09e770 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 -- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 msgr2=0x7f101c131290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.481+0000 7f101affd700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d890 tx=0x7f100c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c012800 con 0x7f101c09e770 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: dump: 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.495 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 5 monclient: authenticate success, global_id 14830 2024-09-18T10:12:22.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c012980 con 0x7f101c09e770 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _renew_subs 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c1423a0 con 0x7f101c09e770 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c012d40 con 0x7f101c09e770 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 10 monclient: dump: 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.497 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10197fa700 4 set_mon_vals no callback set 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _renew_subs 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c141c60 con 0x7f101c09e770 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c09fd50 con 0x7f101c09e770 2024-09-18T10:12:22.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f100c014640 con 0x7f101c09e770 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 0x7f1008037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 40..72) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f100c048ec0 con 0x7f101c09e770 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f101b7fe700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 0x7f1008037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.485+0000 7f101b7fe700 10 monclient: get_auth_request con 0x7f1008035a70 auth_method 0 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.489+0000 7f101b7fe700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 0x7f1008037f20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10100093a0 tx=0x7f1010016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.493+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f100c01c360 con 0x7f101c09e770 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.493+0000 7f10037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.493+0000 7f10037fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:22.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.661+0000 7f1021d3a700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.661+0000 7f1021d3a700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.661+0000 7f1021d3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.661+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f101c063ed0 con 0x7f101c09e770 2024-09-18T10:12:22.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.665+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change v72) v1 ==== 201+0+0 (secure 0 0 0) 0x7f100c011340 con 0x7f101c09e770 2024-09-18T10:12:22.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.665+0000 7f10037fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.665+0000 7f10037fe700 10 monclient: _finish_command 2 = system:0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T10:12:22.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f1021d3a700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f1021d3a700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f1021d3a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]} v 0) v1 -- 0x7f101c067b10 con 0x7f101c09e770 2024-09-18T10:12:22.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f10037fe700 1 -- 172.21.15.150:0/50268610 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}]=0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change v72) v1 ==== 201+0+0 (secure 0 0 0) 0x7f100c02d0c0 con 0x7f101c09e770 2024-09-18T10:12:22.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f10037fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 8, "weight": 1.0, "args": ["host=host1", "root=foo"]}] 2024-09-18T10:12:22.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.773+0000 7f10037fe700 10 monclient: _finish_command 3 = system:0 set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 msgr2=0x7f1008037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 0x7f1008037f20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10100093a0 tx=0x7f1010016000 comp rx=0 tx=0).stop 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 10 monclient: shutdown 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 msgr2=0x7f101c131290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f100c00d890 tx=0x7f100c008040 comp rx=0 tx=0).stop 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 shutdown_connections 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1008035a70 0x7f1008037f20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f101c132700 0x7f101c11ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101c09e770 0x7f101c131290 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 --2- 172.21.15.150:0/50268610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f101c06b5a0 0x7f101c141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 >> 172.21.15.150:0/50268610 conn(0x7f101c071b60 msgr2=0x7f101c0a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 shutdown_connections 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.777+0000 7f1021d3a700 1 -- 172.21.15.150:0/50268610 wait complete. 2024-09-18T10:12:22.782 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 8 name 'osd.8' weight 1 at location {host=host1,root=foo}: no change 2024-09-18T10:12:22.790 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 10 1 host=host2 root=foo 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.889+0000 7fd74ae52700 10 monclient: get_monmap_and_config 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.889+0000 7fd74ae52700 10 monclient: build_initial_monmap 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient: monmap: 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:22.896045+0000 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:22.896045+0000 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:22.896 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 Processor -- start 2024-09-18T10:12:22.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 -- start start 2024-09-18T10:12:22.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient: init 2024-09-18T10:12:22.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:22.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:12:22.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7441304c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd7440a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-b con 0x7fd7440a07e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd744135390 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd744071c90 con 0x7fd744135390 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7440a0c50 con 0x7fd7441304c0 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd744130fa0 con 0x7fd7440a07e0 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd748bee700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd7441304c0 0x7fd744130890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57808/0 (socket says 172.21.15.150:57808) 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd748bee700 1 -- 172.21.15.150:0/3836550316 learned_addr learned my addr 172.21.15.150:0/3836550316 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd74ae52700 10 monclient(hunting): authenticate will time out at 1024.991249s 2024-09-18T10:12:22.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd743fff700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 msgr2=0x7fd7440a1ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:22.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.893+0000 7fd743fff700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd7440a1ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd748bee700 1 -- 172.21.15.150:0/3836550316 >> v1:172.21.15.150:6789/0 conn(0x7fd7441304c0 legacy=0x7fd744130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708160980 0 0) 0x7fd7440a0c50 con 0x7fd7441304c0 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): my global_id is 14833 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd72c003570 con 0x7fd7441304c0 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 1 -- 172.21.15.150:0/3836550316 >> v1:172.21.15.142:6789/0 conn(0x7fd744135390 legacy=0x7fd744137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269034941 0 0) 0x7fd72c003570 con 0x7fd7441304c0 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 msgr2=0x7fd7440a1ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd7440a1ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 >> v1:172.21.15.142:6789/0 conn(0x7fd744135390 legacy=0x7fd744137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7440a0f30 con 0x7fd7441304c0 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd74ae52700 5 monclient: authenticate success, global_id 14833 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fd738001860 con 0x7fd7441304c0 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd74ae52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: dump: 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:22.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 >> v1:172.21.15.150:6789/0 conn(0x7fd7441304c0 legacy=0x7fd744130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd72c003a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): picked mon.c con 0x7fd7440a07e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd72c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): picked mon.b con 0x7fd72c0041e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72c004ea0 0x7fd72c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): picked mon.a con 0x7fd72c004ea0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744130fa0 con 0x7fd72c004ea0 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0056b0 con 0x7fd72c0041e0 2024-09-18T10:12:22.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c00bff0 con 0x7fd7440a07e0 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7437fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd74ae52700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd748bee700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 msgr2=0x7fd72c003a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd72c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd748bee700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd72c003a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): get_auth_request con 0x7fd72c0041e0 auth_method 0 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72c004ea0 0x7fd72c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 10 monclient(hunting): get_auth_request con 0x7fd72c004ea0 auth_method 0 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd7493ef700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient(hunting): handle_auth_done global_id 14833 payload 293 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 msgr2=0x7fd72c003a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7440a07e0 0x7fd72c003a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72c004ea0 msgr2=0x7fd72c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72c004ea0 0x7fd72c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 monclient: found mon.b 2024-09-18T10:12:22.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:22.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 1 -- 172.21.15.150:0/3836550316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c003570 con 0x7fd72c0041e0 2024-09-18T10:12:22.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:22.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.897+0000 7fd743fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:22.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd743fff700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 msgr2=0x7fd72c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd743fff700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd72c004810 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd734003980 tx=0x7fd7340090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3265fb3cc10b8da1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734009e80 con 0x7fd72c0041e0 2024-09-18T10:12:22.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: dump: 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:22.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd734003c10 con 0x7fd72c0041e0 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 1 -- 172.21.15.150:0/3836550316 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7340050f0 con 0x7fd72c0041e0 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 4 set_mon_vals no callback set 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7437fe700 10 monclient: dump: 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.914 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 10 monclient: shutdown 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 msgr2=0x7fd72c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 --2- 172.21.15.150:0/3836550316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd72c004810 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd734003980 tx=0x7fd7340090f0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd7493ef700 1 -- 172.21.15.150:0/3836550316 reap_dead start 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- 172.21.15.150:0/3836550316 shutdown_connections 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- 172.21.15.150:0/3836550316 >> 172.21.15.150:0/3836550316 conn(0x7fd744068520 msgr2=0x7fd7440a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- 172.21.15.150:0/3836550316 shutdown_connections 2024-09-18T10:12:22.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- 172.21.15.150:0/3836550316 wait complete. 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 10 monclient: build_initial_monmap 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 10 monclient: monmap: 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 Processor -- start 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 1 -- start start 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.901+0000 7fd74ae52700 10 monclient: init 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-b con 0x7fd72c0041e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 0x7fd74412f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-c con 0x7fd74412f020 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd744118f60 0x7fd7441193b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): picked mon.noname-a con 0x7fd744118f60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74412f930 con 0x7fd744118f60 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74411b410 con 0x7fd72c0041e0 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74411ae20 con 0x7fd74412f020 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd74ae52700 10 monclient(hunting): authenticate will time out at 1025.000300s 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd743fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 msgr2=0x7fd74412f3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd743fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 0x7fd74412f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44828/0 (socket says 172.21.15.150:44828) 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 -- 172.21.15.150:0/1323312187 learned_addr learned my addr 172.21.15.150:0/1323312187 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): get_auth_request con 0x7fd72c0041e0 auth_method 0 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd744118f60 0x7fd7441193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): get_auth_request con 0x7fd744118f60 auth_method 0 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:22.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd7493ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient(hunting): handle_auth_done global_id 14836 payload 824 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 -- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 msgr2=0x7fd74412f3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 0x7fd74412f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 -- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd744118f60 msgr2=0x7fd7441193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd744118f60 0x7fd7441193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient: _finish_hunting 0 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 monclient: found mon.noname-b 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd744134cb0 con 0x7fd72c0041e0 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 -- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 msgr2=0x7fd74409b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:22.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:22.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.905+0000 7fd748bee700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c910 tx=0x7fd738001200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738007570 con 0x7fd72c0041e0 2024-09-18T10:12:22.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: dump: 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7380076f0 con 0x7fd72c0041e0 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 5 monclient: authenticate success, global_id 14836 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _renew_subs 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74411b130 con 0x7fd72c0041e0 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd738004a80 con 0x7fd72c0041e0 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 10 monclient: dump: 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:22.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd742ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _renew_subs 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74409ca70 con 0x7fd72c0041e0 2024-09-18T10:12:22.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd738014420 con 0x7fd72c0041e0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74411b5d0 con 0x7fd72c0041e0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 0x7fd730037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 40..72) v4 ==== 5915+0+0 (secure 0 0 0) 0x7fd738048680 con 0x7fd72c0041e0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.909+0000 7fd743fff700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 0x7fd730037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.913+0000 7fd743fff700 10 monclient: get_auth_request con 0x7fd730035ac0 auth_method 0 2024-09-18T10:12:22.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.913+0000 7fd743fff700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 0x7fd730037f70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd734007fd0 tx=0x7fd734015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:22.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.917+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd738004e30 con 0x7fd72c0041e0 2024-09-18T10:12:22.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.917+0000 7fd740ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:22.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:22.917+0000 7fd740ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:23.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.085+0000 7fd74ae52700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.085+0000 7fd74ae52700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.085+0000 7fd74ae52700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.085+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fd74406e900 con 0x7fd72c0041e0 2024-09-18T10:12:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.257+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 add item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo} to crush map v73) v1 ==== 206+0+0 (secure 0 0 0) 0x7fd73801b770 con 0x7fd72c0041e0 2024-09-18T10:12:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.257+0000 7fd740ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.257+0000 7fd740ff9700 10 monclient: _finish_command 2 = system:0 add item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo} to crush map 2024-09-18T10:12:23.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.361+0000 7fd74ae52700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.361+0000 7fd74ae52700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.361+0000 7fd74ae52700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.361+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7fd744063ed0 con 0x7fd72c0041e0 2024-09-18T10:12:23.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.365+0000 7fd740ff9700 1 -- 172.21.15.150:0/1323312187 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change v73) v1 ==== 204+0+0 (secure 0 0 0) 0x7fd73801bd20 con 0x7fd72c0041e0 2024-09-18T10:12:23.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.365+0000 7fd740ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.365+0000 7fd740ff9700 10 monclient: _finish_command 3 = system:0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T10:12:23.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 msgr2=0x7fd730037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 0x7fd730037f70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd734007fd0 tx=0x7fd734015000 comp rx=0 tx=0).stop 2024-09-18T10:12:23.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 10 monclient: shutdown 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 msgr2=0x7fd74409b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd73800c910 tx=0x7fd738001200 comp rx=0 tx=0).stop 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 shutdown_connections 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd730035ac0 0x7fd730037f70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd744118f60 0x7fd7441193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd74412f020 0x7fd74412f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 --2- 172.21.15.150:0/1323312187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72c0041e0 0x7fd74409b840 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 >> 172.21.15.150:0/1323312187 conn(0x7fd744068520 msgr2=0x7fd7441330f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 shutdown_connections 2024-09-18T10:12:23.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.369+0000 7fd74ae52700 1 -- 172.21.15.150:0/1323312187 wait complete. 2024-09-18T10:12:23.374 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T10:12:23.381 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 10 1 host=host2 root=foo 2024-09-18T10:12:23.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.481+0000 7ffa379f8700 10 monclient: get_monmap_and_config 2024-09-18T10:12:23.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.481+0000 7ffa379f8700 10 monclient: build_initial_monmap 2024-09-18T10:12:23.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.481+0000 7ffa379f8700 10 monclient: monmap: 2024-09-18T10:12:23.487 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:23.487345+0000 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:23.487345+0000 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:23.488 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 Processor -- start 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 -- start start 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient: init 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffa3006b5a0 0x7ffa301321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-a con 0x7ffa3006b5a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa30132700 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa30132b40 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3006aac0 con 0x7ffa30132700 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3006b970 con 0x7ffa30132b40 2024-09-18T10:12:23.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3009fb10 con 0x7ffa3006b5a0 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa379f8700 10 monclient(hunting): authenticate will time out at 1025.580941s 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35f95700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ffa30132b40 0x7ffa3009ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57816/0 (socket says 172.21.15.150:57816) 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35f95700 1 -- 172.21.15.150:0/860149915 learned_addr learned my addr 172.21.15.150:0/860149915 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 1 --2- 172.21.15.150:0/860149915 >> v2:172.21.15.142:3300/0 conn(0x7ffa3006b5a0 0x7ffa301321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): get_auth_request con 0x7ffa3006b5a0 auth_method 0 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient(hunting): handle_auth_done global_id 5082 payload 274 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 1 -- 172.21.15.150:0/860149915 >> v1:172.21.15.150:6789/0 conn(0x7ffa30132b40 legacy=0x7ffa3009ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 1 -- 172.21.15.150:0/860149915 >> v1:172.21.15.142:6789/0 conn(0x7ffa30132700 legacy=0x7ffa3009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient: _finish_hunting 0 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 1 monclient: found mon.noname-a 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:23.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.485+0000 7ffa35794700 1 -- 172.21.15.150:0/860149915 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3006a7e0 con 0x7ffa3006b5a0 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa35794700 1 -- 172.21.15.150:0/860149915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa3006b5a0 msgr2=0x7ffa301321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa35794700 1 --2- 172.21.15.150:0/860149915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa3006b5a0 0x7ffa301321c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffa200098e0 tx=0x7ffa200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a624cffcdd150c19 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 1 -- 172.21.15.150:0/860149915 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2000ba70 con 0x7ffa3006b5a0 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: dump: 2024-09-18T10:12:23.495 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: _finish_auth 0 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:23.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 1 -- 172.21.15.150:0/860149915 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2000bbd0 con 0x7ffa3006b5a0 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 5 monclient: authenticate success, global_id 5082 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 1 -- 172.21.15.150:0/860149915 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20011420 con 0x7ffa3006b5a0 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 4 set_mon_vals no callback set 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa2ffff700 10 monclient: dump: 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.497 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 10 monclient: shutdown 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- 172.21.15.150:0/860149915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa3006b5a0 msgr2=0x7ffa301321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 --2- 172.21.15.150:0/860149915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa3006b5a0 0x7ffa301321c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffa200098e0 tx=0x7ffa200091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- 172.21.15.150:0/860149915 shutdown_connections 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 --2- 172.21.15.150:0/860149915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa3006b5a0 0x7ffa301321c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- 172.21.15.150:0/860149915 >> 172.21.15.150:0/860149915 conn(0x7ffa30071b60 msgr2=0x7ffa300a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:23.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- 172.21.15.150:0/860149915 shutdown_connections 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- 172.21.15.150:0/860149915 wait complete. 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 10 monclient: build_initial_monmap 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 10 monclient: monmap: 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:23.499 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 Processor -- start 2024-09-18T10:12:23.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 1 -- start start 2024-09-18T10:12:23.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.489+0000 7ffa379f8700 10 monclient: init 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-b con 0x7ffa30132700 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 0x7ffa301314a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-c con 0x7ffa30132b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:23.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30141fc0 0x7ffa3011b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): picked mon.noname-a con 0x7ffa30141fc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa30142980 con 0x7ffa30141fc0 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300a31e0 con 0x7ffa30132700 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa30069b30 con 0x7ffa30132b40 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa379f8700 10 monclient(hunting): authenticate will time out at 1025.588340s 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa34f93700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 msgr2=0x7ffa301314a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44830/0 (socket says 172.21.15.150:44830) 2024-09-18T10:12:23.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 -- 172.21.15.150:0/270526067 learned_addr learned my addr 172.21.15.150:0/270526067 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa34f93700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 0x7ffa301314a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): get_auth_request con 0x7ffa30132700 auth_method 0 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30141fc0 0x7ffa3011b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): get_auth_request con 0x7ffa30141fc0 auth_method 0 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35f95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient(hunting): handle_auth_done global_id 14839 payload 824 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 -- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 msgr2=0x7ffa301314a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 0x7ffa301314a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 -- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30141fc0 msgr2=0x7ffa3011b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30141fc0 0x7ffa3011b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient: _finish_hunting 0 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 monclient: found mon.noname-b 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa20009420 con 0x7ffa30132700 2024-09-18T10:12:23.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 -- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 msgr2=0x7ffa30141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.493+0000 7ffa35794700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa30072050 tx=0x7ffa20016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20011420 con 0x7ffa30132700 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: dump: 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.507 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2001a670 con 0x7ffa30132700 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 5 monclient: authenticate success, global_id 14839 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _renew_subs 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3011ca90 con 0x7ffa30132700 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2001aa10 con 0x7ffa30132700 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 10 monclient: dump: 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2f7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _renew_subs 2024-09-18T10:12:23.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3011e8a0 con 0x7ffa30132700 2024-09-18T10:12:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.497+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffa2002e440 con 0x7ffa30132700 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.501+0000 7ffa2d7fa700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 0x7ffa1c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.501+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(73..73 src has 40..73) v4 ==== 6160+0+0 (secure 0 0 0) 0x7ffa2004a8d0 con 0x7ffa30132700 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.501+0000 7ffa34f93700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 0x7ffa1c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.501+0000 7ffa34f93700 10 monclient: get_auth_request con 0x7ffa1c035ac0 auth_method 0 2024-09-18T10:12:23.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.501+0000 7ffa34f93700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 0x7ffa1c037f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffa24007fd0 tx=0x7ffa24007a70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.509+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.509+0000 7ffa2d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.509+0000 7ffa2d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa379f8700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa379f8700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa379f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change v73) v1 ==== 204+0+0 (secure 0 0 0) 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa2d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.677+0000 7ffa2d7fa700 10 monclient: _finish_command 2 = system:0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T10:12:23.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.773+0000 7ffa379f8700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.773+0000 7ffa379f8700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.773+0000 7ffa379f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.773+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]} v 0) v1 -- 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.777+0000 7ffa2d7fa700 1 -- 172.21.15.150:0/270526067 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}]=0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change v73) v1 ==== 204+0+0 (secure 0 0 0) 0x7ffa3011d720 con 0x7ffa30132700 2024-09-18T10:12:23.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.777+0000 7ffa2d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 10, "weight": 1.0, "args": ["host=host2", "root=foo"]}] 2024-09-18T10:12:23.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.777+0000 7ffa2d7fa700 10 monclient: _finish_command 3 = system:0 set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 msgr2=0x7ffa1c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 0x7ffa1c037f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffa24007fd0 tx=0x7ffa24007a70 comp rx=0 tx=0).stop 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 10 monclient: shutdown 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 msgr2=0x7ffa30141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa30072050 tx=0x7ffa20016000 comp rx=0 tx=0).stop 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 shutdown_connections 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa1c035ac0 0x7ffa1c037f70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30141fc0 0x7ffa3011b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa30132b40 0x7ffa301314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 --2- 172.21.15.150:0/270526067 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa30132700 0x7ffa30141a80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 >> 172.21.15.150:0/270526067 conn(0x7ffa30071b60 msgr2=0x7ffa300a34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 shutdown_connections 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.781+0000 7ffa379f8700 1 -- 172.21.15.150:0/270526067 wait complete. 2024-09-18T10:12:23.786 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 10 name 'osd.10' weight 1 at location {host=host2,root=foo}: no change 2024-09-18T10:12:23.799 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket bar root 2024-09-18T10:12:23.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.897+0000 7fb74e5ca700 10 monclient: get_monmap_and_config 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.897+0000 7fb74e5ca700 10 monclient: build_initial_monmap 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.897+0000 7fb74e5ca700 10 monclient: monmap: 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:23.903773+0000 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:23.903773+0000 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:23.904 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:23.905 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:23.905 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:23.905 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:23.905 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 Processor -- start 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 -- start start 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient: init 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74809e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-b con 0x7fb74809e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb74806cbc0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7480699a0 0x7fb74809ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-a con 0x7fb7480699a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74806af30 con 0x7fb74806cbc0 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74809f1c0 con 0x7fb7480699a0 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74806b630 con 0x7fb74809e370 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74e5ca700 10 monclient(hunting): authenticate will time out at 1025.997372s 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb747fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 msgr2=0x7fb74809e740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb747fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74809e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb7477fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb74806cbc0 0x7fb748132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57826/0 (socket says 172.21.15.150:57826) 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb7477fe700 1 -- 172.21.15.150:0/393295297 learned_addr learned my addr 172.21.15.150:0/393295297 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 --2- 172.21.15.150:0/393295297 >> v2:172.21.15.142:3300/0 conn(0x7fb7480699a0 0x7fb74809ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): get_auth_request con 0x7fb7480699a0 auth_method 0 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient(hunting): handle_auth_done global_id 5088 payload 274 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 -- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 msgr2=0x7fb74809e740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 --2- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74809e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 -- 172.21.15.150:0/393295297 >> v1:172.21.15.150:6789/0 conn(0x7fb74806cbc0 legacy=0x7fb748132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient: _finish_hunting 0 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 monclient: found mon.noname-a 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:23.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.901+0000 7fb74cb67700 1 -- 172.21.15.150:0/393295297 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74806ac50 con 0x7fb7480699a0 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74cb67700 1 -- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7480699a0 msgr2=0x7fb74809ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74cb67700 1 --2- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7480699a0 0x7fb74809ec80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb740009e60 tx=0x7fb740009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2441d364aa34310a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 1 -- 172.21.15.150:0/393295297 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74000e070 con 0x7fb7480699a0 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: dump: 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.915 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 1 -- 172.21.15.150:0/393295297 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb740012430 con 0x7fb7480699a0 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 5 monclient: authenticate success, global_id 5088 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:23.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 1 -- 172.21.15.150:0/393295297 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7400127f0 con 0x7fb7480699a0 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 4 set_mon_vals no callback set 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb746ffd700 10 monclient: dump: 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.917 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 10 monclient: shutdown 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 -- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7480699a0 msgr2=0x7fb74809ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7480699a0 0x7fb74809ec80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb740009e60 tx=0x7fb740009780 comp rx=0 tx=0).stop 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 -- 172.21.15.150:0/393295297 shutdown_connections 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7480699a0 0x7fb74809ec80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/393295297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74809e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 -- 172.21.15.150:0/393295297 >> 172.21.15.150:0/393295297 conn(0x7fb748071850 msgr2=0x7fb7480a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:23.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 -- 172.21.15.150:0/393295297 shutdown_connections 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 1 -- 172.21.15.150:0/393295297 wait complete. 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 10 monclient: build_initial_monmap 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.905+0000 7fb74e5ca700 10 monclient: monmap: 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:23.919 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 Processor -- start 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 -- start start 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient: init 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-b con 0x7fb7480699a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 0x7fb74811e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-a con 0x7fb74806cbc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74811e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): picked mon.noname-c con 0x7fb74809e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb748121980 con 0x7fb74806cbc0 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb748132200 con 0x7fb7480699a0 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74811fbe0 con 0x7fb74809e370 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74e5ca700 10 monclient(hunting): authenticate will time out at 1026.005143s 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74cb67700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 msgr2=0x7fb74811e570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44842/0 (socket says 172.21.15.150:44842) 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 -- 172.21.15.150:0/1684930915 learned_addr learned my addr 172.21.15.150:0/1684930915 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): get_auth_request con 0x7fb7480699a0 auth_method 0 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb74cb67700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74811e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 0x7fb74811e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): get_auth_request con 0x7fb74806cbc0 auth_method 0 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:23.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient(hunting): handle_auth_done global_id 14842 payload 824 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 -- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 msgr2=0x7fb74811e570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74811e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 -- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 msgr2=0x7fb74811e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 0x7fb74811e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 monclient: found mon.noname-b 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb747fff700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740009a20 con 0x7fb7480699a0 2024-09-18T10:12:23.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.909+0000 7fb7477fe700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 0x7fb74811e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb747fff700 1 -- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 msgr2=0x7fb748131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb747fff700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb73800b810 tx=0x7fb738009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738003e20 con 0x7fb7480699a0 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: dump: 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.926 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: _finish_auth 0 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 5 monclient: authenticate success, global_id 14842 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb738006150 con 0x7fb7480699a0 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _renew_subs 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb748140020 con 0x7fb7480699a0 2024-09-18T10:12:23.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb738006510 con 0x7fb7480699a0 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 10 monclient: dump: 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:23.928 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:23.929 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _renew_subs 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb745ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74813fc30 con 0x7fb7480699a0 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7480a4540 con 0x7fb7480699a0 2024-09-18T10:12:23.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb73801a360 con 0x7fb7480699a0 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 0x7fb728037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(73..73 src has 40..73) v4 ==== 6160+0+0 (secure 0 0 0) 0x7fb738047f20 con 0x7fb7480699a0 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb7477fe700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 0x7fb728037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.913+0000 7fb7477fe700 10 monclient: get_auth_request con 0x7fb728035a70 auth_method 0 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.917+0000 7fb7477fe700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 0x7fb728037f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb748120010 tx=0x7fb730007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:23.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.921+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb738010360 con 0x7fb7480699a0 2024-09-18T10:12:23.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.921+0000 7fb73ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:23.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:23.921+0000 7fb73ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:24.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.089+0000 7fb74e5ca700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.089+0000 7fb74e5ca700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.089+0000 7fb74e5ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.089+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fb74806eef0 con 0x7fb7480699a0 2024-09-18T10:12:24.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.265+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 added bucket bar type root to crush map v74) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb738017030 con 0x7fb7480699a0 2024-09-18T10:12:24.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.265+0000 7fb73ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.265+0000 7fb73ffff700 10 monclient: _finish_command 2 = system:0 added bucket bar type root to crush map 2024-09-18T10:12:24.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb74e5ca700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb74e5ca700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb74e5ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7fb748063ed0 con 0x7fb7480699a0 2024-09-18T10:12:24.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb73ffff700 1 -- 172.21.15.150:0/1684930915 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v74) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb738017020 con 0x7fb7480699a0 2024-09-18T10:12:24.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb73ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.377+0000 7fb73ffff700 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 msgr2=0x7fb728037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 0x7fb728037f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb748120010 tx=0x7fb730007f90 comp rx=0 tx=0).stop 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 10 monclient: shutdown 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 msgr2=0x7fb748131550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb73800b810 tx=0x7fb738009220 comp rx=0 tx=0).stop 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 shutdown_connections 2024-09-18T10:12:24.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb728035a70 0x7fb728037f20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb74809e370 0x7fb74811e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb74806cbc0 0x7fb74811e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 --2- 172.21.15.150:0/1684930915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7480699a0 0x7fb748131550 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 >> 172.21.15.150:0/1684930915 conn(0x7fb748071850 msgr2=0x7fb74806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 shutdown_connections 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.381+0000 7fb74e5ca700 1 -- 172.21.15.150:0/1684930915 wait complete. 2024-09-18T10:12:24.386 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'bar' already exists 2024-09-18T10:12:24.395 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket bar root 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.489+0000 7ff8914bd700 10 monclient: get_monmap_and_config 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.489+0000 7ff8914bd700 10 monclient: build_initial_monmap 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.489+0000 7ff8914bd700 10 monclient: monmap: 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:24.495999+0000 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:24.495999+0000 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:24.496 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:24.497 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 Processor -- start 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 -- start start 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient: init 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff88c09e370 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 0x7ff88c09eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-c con 0x7ff88c09e740 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:24.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff88c135590 0x7ff88c137970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-a con 0x7ff88c135590 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0a00e0 con 0x7ff88c09e370 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c06e080 con 0x7ff88c135590 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c06d770 con 0x7ff88c09e740 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff8914bd700 10 monclient(hunting): authenticate will time out at 1026.589343s 2024-09-18T10:12:24.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 0x7ff88c09eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 0x7ff88c09eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44848/0 (socket says 172.21.15.150:44848) 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 1 -- 172.21.15.150:0/4103504015 learned_addr learned my addr 172.21.15.150:0/4103504015 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): get_auth_request con 0x7ff88c09e740 auth_method 0 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/4103504015 >> v2:172.21.15.142:3300/0 conn(0x7ff88c135590 0x7ff88c137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): get_auth_request con 0x7ff88c135590 auth_method 0 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_done global_id 5094 payload 274 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 -- 172.21.15.150:0/4103504015 >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 msgr2=0x7ff88c09eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/4103504015 >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 0x7ff88c09eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 -- 172.21.15.150:0/4103504015 >> v1:172.21.15.150:6789/0 conn(0x7ff88c09e370 legacy=0x7ff88c06e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:24.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.493+0000 7ff88b7fe700 1 -- 172.21.15.150:0/4103504015 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c09fe50 con 0x7ff88c135590 2024-09-18T10:12:24.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff88b7fe700 1 -- 172.21.15.150:0/4103504015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c135590 msgr2=0x7ff88c137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/4103504015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c135590 0x7ff88c137970 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff88c072020 tx=0x7ff884009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d9c345e9e2a0b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 1 -- 172.21.15.150:0/4103504015 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88400e070 con 0x7ff88c135590 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: dump: 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 1 -- 172.21.15.150:0/4103504015 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88400bd70 con 0x7ff88c135590 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 5 monclient: authenticate success, global_id 5094 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 4 set_mon_vals no callback set 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 1 -- 172.21.15.150:0/4103504015 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884012680 con 0x7ff88c135590 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff889ffb700 10 monclient: dump: 2024-09-18T10:12:24.510 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: shutdown 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:24.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- 172.21.15.150:0/4103504015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c135590 msgr2=0x7ff88c137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 --2- 172.21.15.150:0/4103504015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c135590 0x7ff88c137970 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff88c072020 tx=0x7ff884009780 comp rx=0 tx=0).stop 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- 172.21.15.150:0/4103504015 shutdown_connections 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 --2- 172.21.15.150:0/4103504015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c135590 0x7ff88c137970 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 --2- 172.21.15.150:0/4103504015 >> v2:172.21.15.150:3300/0 conn(0x7ff88c09e740 0x7ff88c09eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- 172.21.15.150:0/4103504015 >> 172.21.15.150:0/4103504015 conn(0x7ff88c071850 msgr2=0x7ff88c0a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- 172.21.15.150:0/4103504015 shutdown_connections 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- 172.21.15.150:0/4103504015 wait complete. 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: build_initial_monmap 2024-09-18T10:12:24.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: monmap: 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.513 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 Processor -- start 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 -- start start 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: init 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.497+0000 7ff8914bd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c09e370 0x7ff88c11ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-a con 0x7ff88c09e370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 0x7ff88c11cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-c con 0x7ff88c09e740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:24.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): picked mon.noname-b con 0x7ff88c135590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1347f0 con 0x7ff88c09e370 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c1203b0 con 0x7ff88c135590 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c135140 con 0x7ff88c09e740 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient(hunting): authenticate will time out at 1026.596118s 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88a7fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 msgr2=0x7ff88c11cfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 0x7ff88c11cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44860/0 (socket says 172.21.15.150:44860) 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 -- 172.21.15.150:0/3262364197 learned_addr learned my addr 172.21.15.150:0/3262364197 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): get_auth_request con 0x7ff88c135590 auth_method 0 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c09e370 0x7ff88c11ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): get_auth_request con 0x7ff88c09e370 auth_method 0 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient(hunting): handle_auth_done global_id 14848 payload 824 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 -- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 msgr2=0x7ff88c11cfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 0x7ff88c11cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 -- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c09e370 msgr2=0x7ff88c11ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c09e370 0x7ff88c11ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884009a20 con 0x7ff88c135590 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 -- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 msgr2=0x7ff88c11d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff88b7fe700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff884009fd0 tx=0x7ff884018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884012e70 con 0x7ff88c135590 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: dump: 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.518 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 5 monclient: authenticate success, global_id 14848 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient: _renew_subs 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c09d810 con 0x7ff88c135590 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88401d640 con 0x7ff88c135590 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient: _renew_subs 2024-09-18T10:12:24.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c11e5a0 con 0x7ff88c135590 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff8897fa700 4 set_mon_vals no callback set 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88401da20 con 0x7ff88c135590 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.501+0000 7ff882ffd700 10 monclient: dump: 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.520 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.521 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.521 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.521 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff884024070 con 0x7ff88c135590 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff882ffd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 0x7ff86c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(74..74 src has 40..74) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff88404acf0 con 0x7ff88c135590 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff88affd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 0x7ff86c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff88affd700 10 monclient: get_auth_request con 0x7ff86c035a70 auth_method 0 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff8914bd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff8914bd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff8914bd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.505+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c063eb0 con 0x7ff88c135590 2024-09-18T10:12:24.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.509+0000 7ff88affd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 0x7ff86c037f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff874003a30 tx=0x7ff874007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.513+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff88401b420 con 0x7ff88c135590 2024-09-18T10:12:24.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.513+0000 7ff882ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.513+0000 7ff882ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:24.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.681+0000 7ff8914bd700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.681+0000 7ff8914bd700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.681+0000 7ff8914bd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.681+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7ff88c067b10 con 0x7ff88c135590 2024-09-18T10:12:24.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.685+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v74) v1 ==== 126+0+0 (secure 0 0 0) 0x7ff884029080 con 0x7ff88c135590 2024-09-18T10:12:24.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.685+0000 7ff882ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.685+0000 7ff882ffd700 10 monclient: _finish_command 2 = system:0 bucket 'bar' already exists 2024-09-18T10:12:24.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff8914bd700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff8914bd700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff8914bd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "bar", "type": "root"} v 0) v1 -- 0x7ff88c067b10 con 0x7ff88c135590 2024-09-18T10:12:24.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff882ffd700 1 -- 172.21.15.150:0/3262364197 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}]=0 bucket 'bar' already exists v74) v1 ==== 126+0+0 (secure 0 0 0) 0x7ff88401c3b0 con 0x7ff88c135590 2024-09-18T10:12:24.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff882ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "bar", "type": "root"}] 2024-09-18T10:12:24.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.781+0000 7ff882ffd700 10 monclient: _finish_command 3 = system:0 bucket 'bar' already exists 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 msgr2=0x7ff86c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 0x7ff86c037f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff874003a30 tx=0x7ff874007ea0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 10 monclient: shutdown 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 msgr2=0x7ff88c11d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff884009fd0 tx=0x7ff884018000 comp rx=0 tx=0).stop 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 shutdown_connections 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff86c035a70 0x7ff86c037f20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88c135590 0x7ff88c11d4e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c09e740 0x7ff88c11cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 --2- 172.21.15.150:0/3262364197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c09e370 0x7ff88c11ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 >> 172.21.15.150:0/3262364197 conn(0x7ff88c071850 msgr2=0x7ff88c099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 shutdown_connections 2024-09-18T10:12:24.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.785+0000 7ff8914bd700 1 -- 172.21.15.150:0/3262364197 wait complete. 2024-09-18T10:12:24.792 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'bar' already exists 2024-09-18T10:12:24.799 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush link host1 root=bar 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 10 monclient: get_monmap_and_config 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 10 monclient: build_initial_monmap 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 10 monclient: monmap: 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:24.901880+0000 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:24.901880+0000 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:24.902 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 1 Processor -- start 2024-09-18T10:12:24.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 1 -- start start 2024-09-18T10:12:24.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.897+0000 7f51b43a7700 10 monclient: init 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f51ac12f8e0 0x7f51ac09c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-a con 0x7f51ac12f8e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac09cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-b con 0x7f51ac12efe0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51ac12f440 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:24.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ac09eeb0 con 0x7f51ac12f440 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ac09e820 con 0x7f51ac12f8e0 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac09a580 con 0x7f51ac12efe0 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b43a7700 10 monclient(hunting): authenticate will time out at 1026.996695s 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b1942700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 msgr2=0x7f51ac09cfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b1942700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac09cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f51ac12f8e0 0x7f51ac09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f51ac12f8e0 0x7f51ac09c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:35352/0 (socket says 172.21.15.150:35352) 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 -- 172.21.15.150:0/1347745242 learned_addr learned my addr 172.21.15.150:0/1347745242 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): get_auth_request con 0x7f51ac12f8e0 auth_method 0 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2944700 1 --1- 172.21.15.150:0/1347745242 >> v1:172.21.15.142:6789/0 conn(0x7f51ac12f440 0x7f51ac09d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:48290/0 (socket says 172.21.15.150:48290) 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient(hunting): handle_auth_done global_id 5100 payload 274 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 -- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 msgr2=0x7f51ac09cfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 --2- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac09cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 -- 172.21.15.150:0/1347745242 >> v1:172.21.15.142:6789/0 conn(0x7f51ac12f440 legacy=0x7f51ac09d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient: _finish_hunting 0 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 monclient: found mon.noname-a 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:24.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:24.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.901+0000 7f51b2143700 1 -- 172.21.15.150:0/1347745242 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ac09a860 con 0x7f51ac12f8e0 2024-09-18T10:12:24.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b2143700 1 -- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f8e0 msgr2=0x7f51ac09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:24.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b2143700 1 --2- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f8e0 0x7f51ac09c350 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f51a80099b0 tx=0x7f51a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883080c8afee2ab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 1 -- 172.21.15.150:0/1347745242 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a800bb50 con 0x7f51ac12f8e0 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: dump: 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.911 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: _finish_auth 0 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 5 monclient: authenticate success, global_id 5100 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 1 -- 172.21.15.150:0/1347745242 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a800bcb0 con 0x7f51ac12f8e0 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 4 set_mon_vals no callback set 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 1 -- 172.21.15.150:0/1347745242 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8011670 con 0x7f51ac12f8e0 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b1141700 10 monclient: dump: 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 10 monclient: shutdown 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:24.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 -- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f8e0 msgr2=0x7f51ac09c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 --2- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f8e0 0x7f51ac09c350 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f51a80099b0 tx=0x7f51a8009250 comp rx=0 tx=0).stop 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 -- 172.21.15.150:0/1347745242 shutdown_connections 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 --2- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac09cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 --2- 172.21.15.150:0/1347745242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f8e0 0x7f51ac09c350 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 -- 172.21.15.150:0/1347745242 >> 172.21.15.150:0/1347745242 conn(0x7f51ac071830 msgr2=0x7f51ac0a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 -- 172.21.15.150:0/1347745242 shutdown_connections 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 -- 172.21.15.150:0/1347745242 wait complete. 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 10 monclient: build_initial_monmap 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 10 monclient: monmap: 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:24.914 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:24.915 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.905+0000 7f51b43a7700 1 Processor -- start 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 -- start start 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient: init 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac12e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-c con 0x7f51ac12efe0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f440 0x7f51ac1276c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-a con 0x7f51ac12f440 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): picked mon.noname-b con 0x7f51ac12f8e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac12dd90 con 0x7f51ac12f440 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac09a140 con 0x7f51ac12f8e0 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac09db30 con 0x7f51ac12efe0 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b43a7700 10 monclient(hunting): authenticate will time out at 1027.005315s 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2143700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 msgr2=0x7f51ac12e730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2143700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac12e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44874/0 (socket says 172.21.15.150:44874) 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 -- 172.21.15.150:0/2137527574 learned_addr learned my addr 172.21.15.150:0/2137527574 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): get_auth_request con 0x7f51ac12f8e0 auth_method 0 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f440 0x7f51ac1276c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): get_auth_request con 0x7f51ac12f440 auth_method 0 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b1942700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient(hunting): handle_auth_done global_id 14851 payload 824 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 -- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 msgr2=0x7f51ac12e730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac12e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 -- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f440 msgr2=0x7f51ac1276c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f440 0x7f51ac1276c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient: _finish_hunting 0 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 monclient: found mon.noname-b 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.909+0000 7f51b2944700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a80094f0 con 0x7f51ac12f8e0 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b2944700 1 -- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 msgr2=0x7f51ac127c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b2944700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f51a400cde0 tx=0x7f51a4005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a400a8e0 con 0x7f51ac12f8e0 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: dump: 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a400aa60 con 0x7f51ac12f8e0 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 5 monclient: authenticate success, global_id 14851 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _renew_subs 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac12ad90 con 0x7f51ac12f8e0 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a400ae20 con 0x7f51ac12f8e0 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:24.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 10 monclient: dump: 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a3fff700 4 set_mon_vals no callback set 2024-09-18T10:12:24.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _renew_subs 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac128d10 con 0x7f51ac12f8e0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ac12fce0 con 0x7f51ac12f8e0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f51a4007300 con 0x7f51ac12f8e0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 0x7f5190037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(74..74 src has 40..74) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f51a4048ac0 con 0x7f51ac12f8e0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b2143700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 0x7f5190037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:24.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.913+0000 7f51b2143700 10 monclient: get_auth_request con 0x7f5190035a70 auth_method 0 2024-09-18T10:12:24.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.917+0000 7f51b2143700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 0x7f5190037f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51a80099b0 tx=0x7f51a8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:24.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.921+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f51a401b400 con 0x7f51ac12f8e0 2024-09-18T10:12:24.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.921+0000 7f51a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:24.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:24.921+0000 7f51a27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:25.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.077+0000 7f51b43a7700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.077+0000 7f51b43a7700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.077+0000 7f51b43a7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.077+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f51ac063eb0 con 0x7f51ac12f8e0 2024-09-18T10:12:25.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.277+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 linked item id -11 name 'host1' to location {root=bar} in crush map v75) v1 ==== 168+0+0 (secure 0 0 0) 0x7f51a4011e60 con 0x7f51ac12f8e0 2024-09-18T10:12:25.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.277+0000 7f51a27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.277+0000 7f51a27fc700 10 monclient: _finish_command 2 = system:0 linked item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.385+0000 7f51b43a7700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.385+0000 7f51b43a7700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.385+0000 7f51b43a7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.385+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7f51ac0008d0 con 0x7f51ac12f8e0 2024-09-18T10:12:25.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51a27fc700 1 -- 172.21.15.150:0/2137527574 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v75) v1 ==== 177+0+0 (secure 0 0 0) 0x7f51a4010770 con 0x7f51ac12f8e0 2024-09-18T10:12:25.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51a27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51a27fc700 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 msgr2=0x7f5190037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 0x7f5190037f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f51a80099b0 tx=0x7f51a8016000 comp rx=0 tx=0).stop 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 10 monclient: shutdown 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 msgr2=0x7f51ac127c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f51a400cde0 tx=0x7f51a4005400 comp rx=0 tx=0).stop 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 shutdown_connections 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5190035a70 0x7f5190037f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51ac12f8e0 0x7f51ac127c00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51ac12f440 0x7f51ac1276c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 --2- 172.21.15.150:0/2137527574 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51ac12efe0 0x7f51ac12e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 >> 172.21.15.150:0/2137527574 conn(0x7f51ac071830 msgr2=0x7f51ac0956b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 shutdown_connections 2024-09-18T10:12:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.389+0000 7f51b43a7700 1 -- 172.21.15.150:0/2137527574 wait complete. 2024-09-18T10:12:25.396 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.404 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush link host1 root=bar 2024-09-18T10:12:25.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: get_monmap_and_config 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: build_initial_monmap 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: monmap: 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:25.504556+0000 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:25.504556+0000 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:25.505 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:25.506 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 Processor -- start 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 -- start start 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: init 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe6640a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-b con 0x7fe66409e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6640a07d0 0x7fe6640a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-a con 0x7fe6640a07d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe664136a10 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6640a10e0 con 0x7fe664136a10 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66409ee30 con 0x7fe6640a07d0 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66409e220 con 0x7fe66409e6d0 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe66b78d700 10 monclient(hunting): authenticate will time out at 1027.598350s 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe669529700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 msgr2=0x7fe6640a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:25.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe669529700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe6640a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6640a07d0 0x7fe6640a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe669d2a700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe664136a10 0x7fe664138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:48302/0 (socket says 172.21.15.150:48302) 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6640a07d0 0x7fe6640a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:35374/0 (socket says 172.21.15.150:35374) 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe669d2a700 1 -- 172.21.15.150:0/39883266 learned_addr learned my addr 172.21.15.150:0/39883266 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): get_auth_request con 0x7fe6640a07d0 auth_method 0 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:25.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient(hunting): handle_auth_done global_id 5106 payload 274 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 -- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 msgr2=0x7fe6640a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 --2- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe6640a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 -- 172.21.15.150:0/39883266 >> v1:172.21.15.142:6789/0 conn(0x7fe664136a10 legacy=0x7fe664138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient: _finish_hunting 0 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 monclient: found mon.noname-a 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:25.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.501+0000 7fe668d28700 1 -- 172.21.15.150:0/39883266 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66409eaa0 con 0x7fe6640a07d0 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe668d28700 1 -- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 msgr2=0x7fe6640a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe668d28700 1 --2- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe6640a0ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe66406a610 tx=0x7fe6600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9475a3151c7746e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 1 -- 172.21.15.150:0/39883266 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000b9a0 con 0x7fe6640a07d0 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: dump: 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: _finish_auth 0 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 5 monclient: authenticate success, global_id 5106 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 1 -- 172.21.15.150:0/39883266 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66000bb20 con 0x7fe6640a07d0 2024-09-18T10:12:25.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 4 set_mon_vals no callback set 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 1 -- 172.21.15.150:0/39883266 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660011420 con 0x7fe6640a07d0 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe65bfff700 10 monclient: dump: 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.517 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 10 monclient: shutdown 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 -- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 msgr2=0x7fe6640a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 --2- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe6640a0ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe66406a610 tx=0x7fe6600091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 -- 172.21.15.150:0/39883266 shutdown_connections 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 --2- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe6640a0ba0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 --2- 172.21.15.150:0/39883266 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe6640a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 -- 172.21.15.150:0/39883266 >> 172.21.15.150:0/39883266 conn(0x7fe6640a58a0 msgr2=0x7fe6640a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 -- 172.21.15.150:0/39883266 shutdown_connections 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.505+0000 7fe66b78d700 1 -- 172.21.15.150:0/39883266 wait complete. 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient: build_initial_monmap 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient: monmap: 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:25.519 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 Processor -- start 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 -- start start 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient: init 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe66412c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-c con 0x7fe66409e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe66412c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-a con 0x7fe6640a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): picked mon.noname-b con 0x7fe664136a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:25.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66412f920 con 0x7fe6640a07d0 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66412dc20 con 0x7fe664136a10 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66412e2d0 con 0x7fe66409e6d0 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe66b78d700 10 monclient(hunting): authenticate will time out at 1027.605836s 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669529700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 msgr2=0x7fe66412c310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669529700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe66412c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44890/0 (socket says 172.21.15.150:44890) 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe66412c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 -- 172.21.15.150:0/2696535772 learned_addr learned my addr 172.21.15.150:0/2696535772 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): get_auth_request con 0x7fe664136a10 auth_method 0 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): get_auth_request con 0x7fe6640a07d0 auth_method 0 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:25.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe668d28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient(hunting): handle_auth_done global_id 14854 payload 824 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 -- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 msgr2=0x7fe66412c310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe66412c310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 -- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 msgr2=0x7fe66412c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe66412c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient: _finish_hunting 0 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 monclient: found mon.noname-b 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.509+0000 7fe669d2a700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe660009420 con 0x7fe664136a10 2024-09-18T10:12:25.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe669d2a700 1 -- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 msgr2=0x7fe66412cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:25.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe669d2a700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00ccc0 tx=0x7fe65c005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c00aac0 con 0x7fe664136a10 2024-09-18T10:12:25.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: dump: 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:25.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 5 monclient: authenticate success, global_id 14854 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _renew_subs 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c00ac40 con 0x7fe664136a10 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66412f640 con 0x7fe664136a10 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0070f0 con 0x7fe664136a10 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 10 monclient: dump: 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.527 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _renew_subs 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe65b7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe664136640 con 0x7fe664136a10 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe664063ed0 con 0x7fe664136a10 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe65c0074a0 con 0x7fe664136a10 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 0x7fe648037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(75..75 src has 40..75) v4 ==== 6313+0+0 (secure 0 0 0) 0x7fe65c048d90 con 0x7fe664136a10 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe669529700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 0x7fe648037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.513+0000 7fe669529700 10 monclient: get_auth_request con 0x7fe648035a70 auth_method 0 2024-09-18T10:12:25.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.517+0000 7fe669529700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 0x7fe648037f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe654006fd0 tx=0x7fe654006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.525+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe65c01a360 con 0x7fe664136a10 2024-09-18T10:12:25.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.525+0000 7fe6597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.525+0000 7fe6597fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:25.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.693+0000 7fe66b78d700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.693+0000 7fe66b78d700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.693+0000 7fe66b78d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.693+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fe664067b10 con 0x7fe664136a10 2024-09-18T10:12:25.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.697+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v75) v1 ==== 177+0+0 (secure 0 0 0) 0x7fe65c010af0 con 0x7fe664136a10 2024-09-18T10:12:25.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.697+0000 7fe6597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.697+0000 7fe6597fa700 10 monclient: _finish_command 2 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.804 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe66b78d700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.804 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe66b78d700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe66b78d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]} v 0) v1 -- 0x7fe66412fa90 con 0x7fe664136a10 2024-09-18T10:12:25.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe6597fa700 1 -- 172.21.15.150:0/2696535772 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}]=0 no need to move item id -11 name 'host1' to location {root=bar} in crush map v75) v1 ==== 177+0+0 (secure 0 0 0) 0x7fe65c02a0a0 con 0x7fe664136a10 2024-09-18T10:12:25.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe6597fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host1", "args": ["root=bar"]}] 2024-09-18T10:12:25.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.801+0000 7fe6597fa700 10 monclient: _finish_command 3 = system:0 no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 msgr2=0x7fe648037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 0x7fe648037f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe654006fd0 tx=0x7fe654006e10 comp rx=0 tx=0).stop 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 10 monclient: shutdown 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 msgr2=0x7fe66412cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00ccc0 tx=0x7fe65c005400 comp rx=0 tx=0).stop 2024-09-18T10:12:25.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 shutdown_connections 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe648035a70 0x7fe648037f20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe664136a10 0x7fe66412cd90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6640a07d0 0x7fe66412c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 --2- 172.21.15.150:0/2696535772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe66409e6d0 0x7fe66412c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 >> 172.21.15.150:0/2696535772 conn(0x7fe6640a58a0 msgr2=0x7fe66409b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 shutdown_connections 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.805+0000 7fe66b78d700 1 -- 172.21.15.150:0/2696535772 wait complete. 2024-09-18T10:12:25.810 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -11 name 'host1' to location {root=bar} in crush map 2024-09-18T10:12:25.819 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush link host2 root=bar 2024-09-18T10:12:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.917+0000 7fd5188c5700 10 monclient: get_monmap_and_config 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.917+0000 7fd5188c5700 10 monclient: build_initial_monmap 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.917+0000 7fd5188c5700 10 monclient: monmap: 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:25.923788+0000 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:25.923788+0000 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:25.924 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:25.925 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:25.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 Processor -- start 2024-09-18T10:12:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 -- start start 2024-09-18T10:12:25.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient: init 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd51409ff90 0x7fd51409e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-c con 0x7fd51409ff90 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd51409f830 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5140a0a70 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd514068ca0 con 0x7fd5140a0a70 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51409d840 con 0x7fd51409f830 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5140a1400 con 0x7fd51409ff90 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd5188c5700 10 monclient(hunting): authenticate will time out at 1028.017982s 2024-09-18T10:12:25.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd51409ff90 0x7fd51409e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd51409ff90 0x7fd51409e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44894/0 (socket says 172.21.15.150:44894) 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 1 -- 172.21.15.150:0/2302068484 learned_addr learned my addr 172.21.15.150:0/2302068484 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): get_auth_request con 0x7fd51409ff90 auth_method 0 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd51259c700 1 --1- 172.21.15.150:0/2302068484 >> v1:172.21.15.150:6789/0 conn(0x7fd51409f830 0x7fd5140a0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57838/0 (socket says 172.21.15.150:57838) 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:25.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.921+0000 7fd512d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd51259c700 1 -- 172.21.15.150:0/2302068484 >> v1:172.21.15.150:6789/0 conn(0x7fd51409f830 legacy=0x7fd5140a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 10 monclient(hunting): handle_auth_done global_id 14857 payload 274 2024-09-18T10:12:25.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 -- 172.21.15.150:0/2302068484 >> v1:172.21.15.150:6789/0 conn(0x7fd51409f830 legacy=0x7fd5140a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 -- 172.21.15.150:0/2302068484 >> v1:172.21.15.142:6789/0 conn(0x7fd5140a0a70 legacy=0x7fd514139730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 monclient: found mon.noname-c 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 -- 172.21.15.150:0/2302068484 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51409d560 con 0x7fd51409ff90 2024-09-18T10:12:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 -- 172.21.15.150:0/2302068484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 msgr2=0x7fd51409e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd512d9d700 1 --2- 172.21.15.150:0/2302068484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd51409e220 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd504009a30 tx=0x7fd504009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e72cae76b034d104 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 1 -- 172.21.15.150:0/2302068484 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50400bb50 con 0x7fd51409ff90 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: dump: 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: _finish_auth 0 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 5 monclient: authenticate success, global_id 14857 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 1 -- 172.21.15.150:0/2302068484 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50400bcb0 con 0x7fd51409ff90 2024-09-18T10:12:25.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 4 set_mon_vals no callback set 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 1 -- 172.21.15.150:0/2302068484 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504011670 con 0x7fd51409ff90 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd511d9b700 10 monclient: dump: 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.934 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.935 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.935 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 10 monclient: shutdown 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 1 -- 172.21.15.150:0/2302068484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 msgr2=0x7fd51409e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.925+0000 7fd5188c5700 1 --2- 172.21.15.150:0/2302068484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd51409e220 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd504009a30 tx=0x7fd504009250 comp rx=0 tx=0).stop 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- 172.21.15.150:0/2302068484 shutdown_connections 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 --2- 172.21.15.150:0/2302068484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd51409e220 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- 172.21.15.150:0/2302068484 >> 172.21.15.150:0/2302068484 conn(0x7fd5140a55a0 msgr2=0x7fd5140a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:25.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- 172.21.15.150:0/2302068484 shutdown_connections 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- 172.21.15.150:0/2302068484 wait complete. 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient: build_initial_monmap 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient: monmap: 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:25.941 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 Processor -- start 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- start start 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient: init 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:25.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 0x7fd5141417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-c con 0x7fd51409f830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-b con 0x7fd51409ff90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5140a0a70 0x7fd514142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): picked mon.noname-a con 0x7fd5140a0a70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd514093280 con 0x7fd5140a0a70 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd514095b80 con 0x7fd51409ff90 2024-09-18T10:12:25.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5140a11d0 con 0x7fd51409f830 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd5188c5700 10 monclient(hunting): authenticate will time out at 1028.027828s 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd51259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd512d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 msgr2=0x7fd5141417e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd512d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 0x7fd5141417e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd51259c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44896/0 (socket says 172.21.15.150:44896) 2024-09-18T10:12:25.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd51259c700 1 -- 172.21.15.150:0/502021258 learned_addr learned my addr 172.21.15.150:0/502021258 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.929+0000 7fd51259c700 10 monclient(hunting): get_auth_request con 0x7fd51409ff90 auth_method 0 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51359e700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5140a0a70 0x7fd514142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51359e700 10 monclient(hunting): get_auth_request con 0x7fd5140a0a70 auth_method 0 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient(hunting): handle_auth_done global_id 14860 payload 824 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 -- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 msgr2=0x7fd5141417e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 0x7fd5141417e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 -- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5140a0a70 msgr2=0x7fd514142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:25.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5140a0a70 0x7fd514142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 monclient: found mon.noname-b 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5040094f0 con 0x7fd51409ff90 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 -- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 msgr2=0x7fd514141d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51259c700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd50800c790 tx=0x7fd508007e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508005c80 con 0x7fd51409ff90 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: dump: 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.947 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 5 monclient: authenticate success, global_id 14860 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 10 monclient: _renew_subs 2024-09-18T10:12:25.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd508005e00 con 0x7fd51409ff90 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51406f750 con 0x7fd51409ff90 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd508007320 con 0x7fd51409ff90 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd4faffd700 10 monclient: dump: 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:25.949 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd51159a700 4 set_mon_vals no callback set 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 10 monclient: _renew_subs 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.933+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd514093970 con 0x7fd51409ff90 2024-09-18T10:12:25.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd50801b3b0 con 0x7fd51409ff90 2024-09-18T10:12:25.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd5188c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd4faffd700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 0x7fd500037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd5188c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd5188c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51404bbf0 con 0x7fd51409ff90 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(75..75 src has 40..75) v4 ==== 6313+0+0 (secure 0 0 0) 0x7fd508047eb0 con 0x7fd51409ff90 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd512d9d700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 0x7fd500037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.937+0000 7fd512d9d700 10 monclient: get_auth_request con 0x7fd500035a70 auth_method 0 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.941+0000 7fd512d9d700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 0x7fd500037f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd504009a30 tx=0x7fd504016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.945+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd508017020 con 0x7fd51409ff90 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.945+0000 7fd4faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:25.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:25.945+0000 7fd4faffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:26.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.113+0000 7fd5188c5700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.113+0000 7fd5188c5700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.113+0000 7fd5188c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.113+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fd514063ed0 con 0x7fd51409ff90 2024-09-18T10:12:26.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.277+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 linked item id -16 name 'host2' to location {root=bar} in crush map v76) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd50800f680 con 0x7fd51409ff90 2024-09-18T10:12:26.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.277+0000 7fd4faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.277+0000 7fd4faffd700 10 monclient: _finish_command 2 = system:0 linked item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd5188c5700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd5188c5700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd5188c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7fd5140008d0 con 0x7fd51409ff90 2024-09-18T10:12:26.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd4faffd700 1 -- 172.21.15.150:0/502021258 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v76) v1 ==== 177+0+0 (secure 0 0 0) 0x7fd50800e290 con 0x7fd51409ff90 2024-09-18T10:12:26.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd4faffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.385+0000 7fd4faffd700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 msgr2=0x7fd500037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 0x7fd500037f20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd504009a30 tx=0x7fd504016000 comp rx=0 tx=0).stop 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 10 monclient: shutdown 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 msgr2=0x7fd514141d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd50800c790 tx=0x7fd508007e50 comp rx=0 tx=0).stop 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 shutdown_connections 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd500035a70 0x7fd500037f20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5140a0a70 0x7fd514142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd51409ff90 0x7fd514141d20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 --2- 172.21.15.150:0/502021258 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd51409f830 0x7fd5141417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 >> 172.21.15.150:0/502021258 conn(0x7fd5140a55a0 msgr2=0x7fd51406dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 shutdown_connections 2024-09-18T10:12:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.389+0000 7fd5188c5700 1 -- 172.21.15.150:0/502021258 wait complete. 2024-09-18T10:12:26.396 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.405 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush link host2 root=bar 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.501+0000 7f25c75c4700 10 monclient: get_monmap_and_config 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.501+0000 7f25c75c4700 10 monclient: build_initial_monmap 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient: monmap: 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:26.508097+0000 2024-09-18T10:12:26.508 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:26.508097+0000 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:26.509 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 Processor -- start 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 -- start start 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient: init 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f25c009e6d0 0x7f25c00a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-a con 0x7f25c009e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25c00a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25c00a0bd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c009e220 con 0x7f25c00a07d0 2024-09-18T10:12:26.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c009eaa0 con 0x7f25c00a0bd0 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c009ee30 con 0x7f25c009e6d0 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c75c4700 10 monclient(hunting): authenticate will time out at 1028.602612s 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5b61700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f25c00a0bd0 0x7f25c013a140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57842/0 (socket says 172.21.15.150:57842) 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5b61700 1 -- 172.21.15.150:0/1838076479 learned_addr learned my addr 172.21.15.150:0/1838076479 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 1 --2- 172.21.15.150:0/1838076479 >> v2:172.21.15.142:3300/0 conn(0x7f25c009e6d0 0x7f25c00a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): get_auth_request con 0x7f25c009e6d0 auth_method 0 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient(hunting): handle_auth_done global_id 5115 payload 274 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 1 -- 172.21.15.150:0/1838076479 >> v1:172.21.15.150:6789/0 conn(0x7f25c00a0bd0 legacy=0x7f25c013a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 1 -- 172.21.15.150:0/1838076479 >> v1:172.21.15.142:6789/0 conn(0x7f25c00a07d0 legacy=0x7f25c0136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient: _finish_hunting 0 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 1 monclient: found mon.noname-a 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:26.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.505+0000 7f25c5360700 1 -- 172.21.15.150:0/1838076479 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c009d930 con 0x7f25c009e6d0 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c5360700 1 -- 172.21.15.150:0/1838076479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c009e6d0 msgr2=0x7f25c00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c5360700 1 --2- 172.21.15.150:0/1838076479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c009e6d0 0x7f25c00a0290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098e0 tx=0x7f25bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d166bec79b56a373 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 1 -- 172.21.15.150:0/1838076479 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00ba70 con 0x7f25c009e6d0 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:26.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: dump: 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:26.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 5 monclient: authenticate success, global_id 5115 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 1 -- 172.21.15.150:0/1838076479 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00bbd0 con 0x7f25c009e6d0 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 4 set_mon_vals no callback set 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 1 -- 172.21.15.150:0/1838076479 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc011420 con 0x7f25c009e6d0 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25b7fff700 10 monclient: dump: 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.518 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 10 monclient: shutdown 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 1 -- 172.21.15.150:0/1838076479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c009e6d0 msgr2=0x7f25c00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.509+0000 7f25c75c4700 1 --2- 172.21.15.150:0/1838076479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c009e6d0 0x7f25c00a0290 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098e0 tx=0x7f25bc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- 172.21.15.150:0/1838076479 shutdown_connections 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 --2- 172.21.15.150:0/1838076479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c009e6d0 0x7f25c00a0290 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- 172.21.15.150:0/1838076479 >> 172.21.15.150:0/1838076479 conn(0x7f25c00a58a0 msgr2=0x7f25c00a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:26.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- 172.21.15.150:0/1838076479 shutdown_connections 2024-09-18T10:12:26.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- 172.21.15.150:0/1838076479 wait complete. 2024-09-18T10:12:26.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient: build_initial_monmap 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient: monmap: 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 Processor -- start 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- start start 2024-09-18T10:12:26.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient: init 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 0x7f25c0141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-c con 0x7f25c009e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-b con 0x7f25c00a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c00a0bd0 0x7f25c008dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): picked mon.noname-a con 0x7f25c00a0bd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00917d0 con 0x7f25c00a0bd0 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0091ab0 con 0x7f25c00a07d0 2024-09-18T10:12:26.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c008fa30 con 0x7f25c009e6d0 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c75c4700 10 monclient(hunting): authenticate will time out at 1028.610799s 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44898/0 (socket says 172.21.15.150:44898) 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5360700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 msgr2=0x7f25c0141d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 -- 172.21.15.150:0/3591695716 learned_addr learned my addr 172.21.15.150:0/3591695716 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): get_auth_request con 0x7f25c00a07d0 auth_method 0 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5360700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 0x7f25c0141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5b61700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c00a0bd0 0x7f25c008dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5b61700 10 monclient(hunting): get_auth_request con 0x7f25c00a0bd0 auth_method 0 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5b61700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5b61700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c5b61700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient(hunting): handle_auth_done global_id 14863 payload 824 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 -- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 msgr2=0x7f25c0141d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 0x7f25c0141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 -- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c00a0bd0 msgr2=0x7f25c008dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c00a0bd0 0x7f25c008dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient: _finish_hunting 0 2024-09-18T10:12:26.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 monclient: found mon.noname-b 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.513+0000 7f25c4b5f700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc009420 con 0x7f25c00a07d0 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c4b5f700 1 -- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 msgr2=0x7f25c0142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c4b5f700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c5d0 tx=0x7f25b0007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b00117a0 con 0x7f25c00a07d0 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: dump: 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.528 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 5 monclient: authenticate success, global_id 14863 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0011900 con 0x7f25c00a07d0 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:26.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0011cc0 con 0x7f25c00a07d0 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _renew_subs 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b77fe700 4 set_mon_vals no callback set 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00914f0 con 0x7f25c00a07d0 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 10 monclient: dump: 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.530 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _renew_subs 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c00926f0 con 0x7f25c00a07d0 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f25b0012630 con 0x7f25c00a07d0 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00911e0 con 0x7f25c00a07d0 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 0x7f25a4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f25b0048840 con 0x7f25c00a07d0 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.517+0000 7f25c5360700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 0x7f25a4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.521+0000 7f25c5360700 10 monclient: get_auth_request con 0x7f25a4035a70 auth_method 0 2024-09-18T10:12:26.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.521+0000 7f25c5360700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 0x7f25a4037f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098e0 tx=0x7f25bc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.525+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f25b0010ac0 con 0x7f25c00a07d0 2024-09-18T10:12:26.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.525+0000 7f25b57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.525+0000 7f25b57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:26.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.693+0000 7f25c75c4700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.693+0000 7f25c75c4700 10 monclient: _send_command 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.693+0000 7f25c75c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.693+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f25c0091910 con 0x7f25c00a07d0 2024-09-18T10:12:26.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.697+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v76) v1 ==== 177+0+0 (secure 0 0 0) 0x7f25c0091910 con 0x7f25c00a07d0 2024-09-18T10:12:26.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.697+0000 7f25b57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.697+0000 7f25b57fa700 10 monclient: _finish_command 2 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.805+0000 7f25c75c4700 10 start_mon_command cmd=[{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.805+0000 7f25c75c4700 10 monclient: _send_command 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.805+0000 7f25c75c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.805+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]} v 0) v1 -- 0x7f25c0091910 con 0x7f25c00a07d0 2024-09-18T10:12:26.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25b57fa700 1 -- 172.21.15.150:0/3591695716 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}]=0 no need to move item id -16 name 'host2' to location {root=bar} in crush map v76) v1 ==== 177+0+0 (secure 0 0 0) 0x7f25c0091910 con 0x7f25c00a07d0 2024-09-18T10:12:26.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25b57fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush link", "name": "host2", "args": ["root=bar"]}] 2024-09-18T10:12:26.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25b57fa700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 msgr2=0x7f25a4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 0x7f25a4037f20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0098e0 tx=0x7f25bc016000 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 10 monclient: shutdown 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 msgr2=0x7f25c0142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c5d0 tx=0x7f25b0007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 shutdown_connections 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25a4035a70 0x7f25a4037f20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25c00a0bd0 0x7f25c008dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25c00a07d0 0x7f25c0142250 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 --2- 172.21.15.150:0/3591695716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25c009e6d0 0x7f25c0141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 >> 172.21.15.150:0/3591695716 conn(0x7f25c00a58a0 msgr2=0x7f25c009a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 shutdown_connections 2024-09-18T10:12:26.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.809+0000 7f25c75c4700 1 -- 172.21.15.150:0/3591695716 wait complete. 2024-09-18T10:12:26.816 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -16 name 'host2' to location {root=bar} in crush map 2024-09-18T10:12:26.825 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:26.825 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c osd.8 2024-09-18T10:12:26.825 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 2 2024-09-18T10:12:26.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: get_monmap_and_config 2024-09-18T10:12:26.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: build_initial_monmap 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: monmap: 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:26.922638+0000 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:26.922638+0000 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:26.923 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 Processor -- start 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 -- start start 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: init 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-c con 0x7f7cf009c2e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 0x7f7cf0098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-b con 0x7f7cf0098a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7cf0099360 0x7f7cf0135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-a con 0x7f7cf0099360 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf0098580 con 0x7f7cf0099360 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf009af60 con 0x7f7cf009c2e0 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf009b520 con 0x7f7cf0098a50 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf7fca700 10 monclient(hunting): authenticate will time out at 1029.015752s 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.917+0000 7f7cf5565700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 msgr2=0x7f7cf0098e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5565700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 0x7f7cf0098e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7cf0099360 0x7f7cf0135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44900/0 (socket says 172.21.15.150:44900) 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 1 -- 172.21.15.150:0/680245404 learned_addr learned my addr 172.21.15.150:0/680245404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): get_auth_request con 0x7f7cf009c2e0 auth_method 0 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): get_auth_request con 0x7f7cf0099360 auth_method 0 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient(hunting): handle_auth_done global_id 5121 payload 274 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 -- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 msgr2=0x7f7cf0098e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 --2- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 0x7f7cf0098e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 -- 172.21.15.150:0/680245404 >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 msgr2=0x7f7cf00956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 --2- 172.21.15.150:0/680245404 >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient: _finish_hunting 0 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 monclient: found mon.noname-a 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 -- 172.21.15.150:0/680245404 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf009c6b0 con 0x7f7cf0099360 2024-09-18T10:12:26.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf5d66700 1 --2- 172.21.15.150:0/680245404 >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:26.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 -- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0099360 msgr2=0x7f7cf0135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:26.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf6567700 1 --2- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0099360 0x7f7cf0135700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009e60 tx=0x7f7ce8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0676b5bec417 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 1 -- 172.21.15.150:0/680245404 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800e070 con 0x7f7cf0099360 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: dump: 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.932 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: _finish_auth 0 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 1 -- 172.21.15.150:0/680245404 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce800bd30 con 0x7f7cf0099360 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf7fca700 5 monclient: authenticate success, global_id 5121 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf7fca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf7fca700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf7fca700 4 set_mon_vals no callback set 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 1 -- 172.21.15.150:0/680245404 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800bea0 con 0x7f7cf0099360 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:26.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.921+0000 7f7cf4d64700 10 monclient: dump: 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 10 monclient: shutdown 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:26.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0099360 msgr2=0x7f7cf0135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0099360 0x7f7cf0135700 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009e60 tx=0x7f7ce8009780 comp rx=0 tx=0).stop 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- 172.21.15.150:0/680245404 shutdown_connections 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0099360 0x7f7cf0135700 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/680245404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf0098a50 0x7f7cf0098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/680245404 >> v2:172.21.15.150:3300/0 conn(0x7f7cf009c2e0 0x7f7cf00956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- 172.21.15.150:0/680245404 >> 172.21.15.150:0/680245404 conn(0x7f7cf00a1710 msgr2=0x7f7cf00a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- 172.21.15.150:0/680245404 shutdown_connections 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- 172.21.15.150:0/680245404 wait complete. 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 10 monclient: build_initial_monmap 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 10 monclient: monmap: 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:26.935 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 Processor -- start 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 1 -- start start 2024-09-18T10:12:26.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.925+0000 7f7cf7fca700 10 monclient: init 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0098a50 0x7f7cf008b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-a con 0x7f7cf0098a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-b con 0x7f7cf0099360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 0x7f7cf008c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): picked mon.noname-c con 0x7f7cf009c2e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00951f0 con 0x7f7cf0098a50 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf006ea40 con 0x7f7cf0099360 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf008eb60 con 0x7f7cf009c2e0 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf7fca700 10 monclient(hunting): authenticate will time out at 1029.024574s 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf6567700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 msgr2=0x7f7cf008c370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf6567700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 0x7f7cf008c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:26.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0098a50 0x7f7cf008b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44904/0 (socket says 172.21.15.150:44904) 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 -- 172.21.15.150:0/225608884 learned_addr learned my addr 172.21.15.150:0/225608884 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): get_auth_request con 0x7f7cf0099360 auth_method 0 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): get_auth_request con 0x7f7cf0098a50 auth_method 0 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5d66700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient(hunting): handle_auth_done global_id 14869 payload 824 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 -- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 msgr2=0x7f7cf008c370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 0x7f7cf008c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 -- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0098a50 msgr2=0x7f7cf008b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0098a50 0x7f7cf008b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient: _finish_hunting 0 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 monclient: found mon.noname-b 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce00094f0 con 0x7f7cf0099360 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 -- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 msgr2=0x7f7cf008be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.929+0000 7f7cf5565700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00b7e0 tx=0x7f7cec009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec003e20 con 0x7f7cf0099360 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: dump: 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.945 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 5 monclient: authenticate success, global_id 14869 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec006150 con 0x7f7cf0099360 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _renew_subs 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec006510 con 0x7f7cf0099360 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce8009a20 con 0x7f7cf0099360 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce7fff700 4 set_mon_vals no callback set 2024-09-18T10:12:26.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 10 monclient: dump: 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:26.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _renew_subs 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf008df70 con 0x7f7cf0099360 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf008f370 con 0x7f7cf0099360 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7cec0103c0 con 0x7f7cf0099360 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 0x7f7cd4037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f7cec048170 con 0x7f7cf0099360 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf5d66700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 0x7f7cd4037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:26.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.933+0000 7f7cf5d66700 10 monclient: get_auth_request con 0x7f7cd4035aa0 auth_method 0 2024-09-18T10:12:26.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.937+0000 7f7cf5d66700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 0x7f7cd4037f50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006d20 tx=0x7f7ce0009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:26.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.941+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7cec00e8a0 con 0x7f7cf0099360 2024-09-18T10:12:26.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.941+0000 7f7ce5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:26.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:26.941+0000 7f7ce5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:27.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.105+0000 7f7cf7fca700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.105+0000 7f7cf7fca700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.105+0000 7f7cf7fca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.105+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f7cf0063ed0 con 0x7f7cf0099360 2024-09-18T10:12:27.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.109+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7f7cec017070 con 0x7f7cf0099360 2024-09-18T10:12:27.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.109+0000 7f7ce5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.109+0000 7f7ce5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7cf7fca700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7cf7fca700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7cf7fca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f7cf00997a0 con 0x7f7cf0099360 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7ce5ffb700 1 -- 172.21.15.150:0/225608884 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7f7cec00e2c0 con 0x7f7cf0099360 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7ce5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.205+0000 7f7ce5ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:27.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 msgr2=0x7f7cd4037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 0x7f7cd4037f50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0006d20 tx=0x7f7ce0009250 comp rx=0 tx=0).stop 2024-09-18T10:12:27.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 10 monclient: shutdown 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 msgr2=0x7f7cf008be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00b7e0 tx=0x7f7cec009220 comp rx=0 tx=0).stop 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 shutdown_connections 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7cd4035aa0 0x7f7cd4037f50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cf009c2e0 0x7f7cf008c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7cf0099360 0x7f7cf008be30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 --2- 172.21.15.150:0/225608884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cf0098a50 0x7f7cf008b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 >> 172.21.15.150:0/225608884 conn(0x7f7cf00a1710 msgr2=0x7f7cf006e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 shutdown_connections 2024-09-18T10:12:27.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.209+0000 7f7cf7fca700 1 -- 172.21.15.150:0/225608884 wait complete. 2024-09-18T10:12:27.228 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:27.228 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c host1 2024-09-18T10:12:27.228 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 2 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: get_monmap_and_config 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: build_initial_monmap 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: monmap: 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:27.326155+0000 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:27.326155+0000 2024-09-18T10:12:27.326 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:27.327 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 Processor -- start 2024-09-18T10:12:27.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 -- start start 2024-09-18T10:12:27.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: init 2024-09-18T10:12:27.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:27.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffadc095540 0x7ffadc09b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-a con 0x7ffadc095540 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 0x7ffadc097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-b con 0x7ffadc097c10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffadc09c130 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc098520 con 0x7ffadc09c130 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffadc09ac00 con 0x7ffadc095540 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc06fa30 con 0x7ffadc097c10 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffae1f98700 10 monclient(hunting): authenticate will time out at 1029.419460s 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffadaffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 msgr2=0x7ffadc097fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:27.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffadbfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ffadc09c130 0x7ffadc09c590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57858/0 (socket says 172.21.15.150:57858) 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffadbfff700 1 -- 172.21.15.150:0/632572588 learned_addr learned my addr 172.21.15.150:0/632572588 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffadaffd700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 0x7ffadc097fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.321+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/632572588 >> v2:172.21.15.142:3300/0 conn(0x7ffadc095540 0x7ffadc09b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): get_auth_request con 0x7ffadc095540 auth_method 0 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_done global_id 5127 payload 274 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 -- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 msgr2=0x7ffadc097fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:27.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 0x7ffadc097fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 -- 172.21.15.150:0/632572588 >> v1:172.21.15.150:6789/0 conn(0x7ffadc09c130 legacy=0x7ffadc09c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 monclient: found mon.noname-a 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:27.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 -- 172.21.15.150:0/632572588 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc099460 con 0x7ffadc095540 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 -- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc095540 msgr2=0x7ffadc09b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc095540 0x7ffadc09b300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffad00098e0 tx=0x7ffad00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f738981f6c409f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffada7fc700 1 -- 172.21.15.150:0/632572588 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad000ba70 con 0x7ffadc095540 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffada7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffada7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffada7fc700 10 monclient: dump: 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.334 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.325+0000 7ffada7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:27.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 1 -- 172.21.15.150:0/632572588 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad000bbf0 con 0x7ffadc095540 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 5 monclient: authenticate success, global_id 5127 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 4 set_mon_vals no callback set 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 1 -- 172.21.15.150:0/632572588 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0011420 con 0x7ffadc095540 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffada7fc700 10 monclient: dump: 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.336 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: shutdown 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:27.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc095540 msgr2=0x7ffadc09b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc095540 0x7ffadc09b300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffad00098e0 tx=0x7ffad00091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- 172.21.15.150:0/632572588 shutdown_connections 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc097c10 0x7ffadc097fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- 172.21.15.150:0/632572588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc095540 0x7ffadc09b300 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- 172.21.15.150:0/632572588 >> 172.21.15.150:0/632572588 conn(0x7ffadc0a1710 msgr2=0x7ffadc0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- 172.21.15.150:0/632572588 shutdown_connections 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- 172.21.15.150:0/632572588 wait complete. 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: build_initial_monmap 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: monmap: 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:27.338 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 Processor -- start 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- start start 2024-09-18T10:12:27.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: init 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-b con 0x7ffadc095540 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc097c10 0x7ffadc141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-a con 0x7ffadc097c10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 0x7ffadc08def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): picked mon.noname-c con 0x7ffadc09c130 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc06fc50 con 0x7ffadc097c10 2024-09-18T10:12:27.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc0a0be0 con 0x7ffadc095540 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffadc08fdb0 con 0x7ffadc09c130 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffae1f98700 10 monclient(hunting): authenticate will time out at 1029.426554s 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadbfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 msgr2=0x7ffadc08def0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 0x7ffadc08def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:27.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44918/0 (socket says 172.21.15.150:44918) 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 -- 172.21.15.150:0/3677963007 learned_addr learned my addr 172.21.15.150:0/3677963007 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): get_auth_request con 0x7ffadc095540 auth_method 0 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc097c10 0x7ffadc141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): get_auth_request con 0x7ffadc097c10 auth_method 0 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient(hunting): handle_auth_done global_id 14872 payload 824 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 -- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 msgr2=0x7ffadc08def0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 0x7ffadc08def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 -- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc097c10 msgr2=0x7ffadc141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc097c10 0x7ffadc141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.329+0000 7ffadb7fe700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0009420 con 0x7ffadc095540 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffadb7fe700 1 -- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 msgr2=0x7ffadc141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffadb7fe700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ffadc069ae0 tx=0x7ffad0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0011e30 con 0x7ffadc095540 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: dump: 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.349 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:27.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad0020070 con 0x7ffadc095540 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 5 monclient: authenticate success, global_id 14872 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 10 monclient: _renew_subs 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc090470 con 0x7ffadc095540 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad001b6a0 con 0x7ffadc095540 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 10 monclient: dump: 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.351 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffad97fa700 4 set_mon_vals no callback set 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 10 monclient: _renew_subs 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc08f8a0 con 0x7ffadc095540 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.333+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffad0022030 con 0x7ffadc095540 2024-09-18T10:12:27.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffac37fe700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 0x7ffac8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7ffad004a6b0 con 0x7ffadc095540 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffae1f98700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffadaffd700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 0x7ffac8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffae1f98700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffae1f98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffadaffd700 10 monclient: get_auth_request con 0x7ffac8035a70 auth_method 0 2024-09-18T10:12:27.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.337+0000 7ffadaffd700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 0x7ffac8037f20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffacc003980 tx=0x7ffacc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.345+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.345+0000 7ffac37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.345+0000 7ffac37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:27.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffae1f98700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffae1f98700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffae1f98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffac37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.505+0000 7ffac37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffae1f98700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffae1f98700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffae1f98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffac37fe700 1 -- 172.21.15.150:0/3677963007 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7ffadc098520 con 0x7ffadc095540 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffac37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.597+0000 7ffac37fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 msgr2=0x7ffac8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 0x7ffac8037f20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffacc003980 tx=0x7ffacc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 10 monclient: shutdown 2024-09-18T10:12:27.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 msgr2=0x7ffadc141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ffadc069ae0 tx=0x7ffad0016000 comp rx=0 tx=0).stop 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 shutdown_connections 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffac8035a70 0x7ffac8037f20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffadc09c130 0x7ffadc08def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffadc097c10 0x7ffadc141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 --2- 172.21.15.150:0/3677963007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffadc095540 0x7ffadc141a20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 >> 172.21.15.150:0/3677963007 conn(0x7ffadc0a1710 msgr2=0x7ffadc0a0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 shutdown_connections 2024-09-18T10:12:27.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.601+0000 7ffae1f98700 1 -- 172.21.15.150:0/3677963007 wait complete. 2024-09-18T10:12:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c osd.10 2024-09-18T10:12:27.614 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 2 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: get_monmap_and_config 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: build_initial_monmap 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: monmap: 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:27.698772+0000 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:27.698772+0000 2024-09-18T10:12:27.699 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 Processor -- start 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 -- start start 2024-09-18T10:12:27.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: init 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f252c0a39f0 0x7f252c06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): picked mon.noname-a con 0x7f252c0a39f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 0x7f252c0a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): picked mon.noname-c con 0x7f252c0a3dc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:27.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f252c1323b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c06a490 con 0x7f252c1323b0 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c06bf90 con 0x7f252c0a39f0 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c06b660 con 0x7f252c0a3dc0 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.693+0000 7f253289d700 10 monclient(hunting): authenticate will time out at 1029.791876s 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 0x7f252c0a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 0x7f252c0a41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44928/0 (socket says 172.21.15.150:44928) 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 1 -- 172.21.15.150:0/2119174404 learned_addr learned my addr 172.21.15.150:0/2119174404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:27.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): get_auth_request con 0x7f252c0a3dc0 auth_method 0 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 --2- 172.21.15.150:0/2119174404 >> v2:172.21.15.142:3300/0 conn(0x7f252c0a39f0 0x7f252c06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): get_auth_request con 0x7f252c0a39f0 auth_method 0 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient(hunting): handle_auth_done global_id 5133 payload 274 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 -- 172.21.15.150:0/2119174404 >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 msgr2=0x7f252c0a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 --2- 172.21.15.150:0/2119174404 >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 0x7f252c0a41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 -- 172.21.15.150:0/2119174404 >> v1:172.21.15.150:6789/0 conn(0x7f252c1323b0 legacy=0x7f252c132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 monclient: found mon.noname-a 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 -- 172.21.15.150:0/2119174404 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c0a5880 con 0x7f252c0a39f0 2024-09-18T10:12:27.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 -- 172.21.15.150:0/2119174404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 msgr2=0x7f252c06c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:27.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252bfff700 1 --2- 172.21.15.150:0/2119174404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c06c7f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f252c06c350 tx=0x7f251c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c39354daed1139f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 1 -- 172.21.15.150:0/2119174404 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c00e070 con 0x7f252c0a39f0 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: dump: 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.713 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: _finish_auth 0 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f253289d700 5 monclient: authenticate success, global_id 5133 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f253289d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 1 -- 172.21.15.150:0/2119174404 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c00bd70 con 0x7f252c0a39f0 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f253289d700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:27.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f253289d700 4 set_mon_vals no callback set 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 1 -- 172.21.15.150:0/2119174404 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c012660 con 0x7f252c0a39f0 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.697+0000 7f252affd700 10 monclient: dump: 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.715 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: shutdown 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- 172.21.15.150:0/2119174404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 msgr2=0x7f252c06c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- 172.21.15.150:0/2119174404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c06c7f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f252c06c350 tx=0x7f251c009780 comp rx=0 tx=0).stop 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- 172.21.15.150:0/2119174404 shutdown_connections 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- 172.21.15.150:0/2119174404 >> v2:172.21.15.150:3300/0 conn(0x7f252c0a3dc0 0x7f252c0a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- 172.21.15.150:0/2119174404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c06c7f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- 172.21.15.150:0/2119174404 >> 172.21.15.150:0/2119174404 conn(0x7f252c0a6300 msgr2=0x7f252c0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- 172.21.15.150:0/2119174404 shutdown_connections 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- 172.21.15.150:0/2119174404 wait complete. 2024-09-18T10:12:27.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: build_initial_monmap 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: monmap: 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 Processor -- start 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- start start 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: init 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): picked mon.noname-a con 0x7f252c0a39f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 0x7f252c142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): picked mon.noname-c con 0x7f252c0a3dc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): picked mon.noname-b con 0x7f252c1323b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c121450 con 0x7f252c0a39f0 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0a0120 con 0x7f252c1323b0 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0a1820 con 0x7f252c0a3dc0 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f253289d700 10 monclient(hunting): authenticate will time out at 1029.798396s 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 msgr2=0x7f252c142050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44932/0 (socket says 172.21.15.150:44932) 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 -- 172.21.15.150:0/2217599539 learned_addr learned my addr 172.21.15.150:0/2217599539 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): get_auth_request con 0x7f252c1323b0 auth_method 0 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252b7fe700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 0x7f252c142050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:27.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): get_auth_request con 0x7f252c0a39f0 auth_method 0 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:27.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f252bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient(hunting): handle_auth_done global_id 14878 payload 824 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 -- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 msgr2=0x7f252c142050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 0x7f252c142050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 -- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 msgr2=0x7f252c141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient: _finish_hunting 0 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 monclient: found mon.noname-b 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.701+0000 7f2530e3a700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c009a20 con 0x7f252c1323b0 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2530e3a700 1 -- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 msgr2=0x7f252c11ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2530e3a700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f252400c5b0 tx=0x7f2524007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25240117a0 con 0x7f252c1323b0 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: dump: 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.722 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 5 monclient: authenticate success, global_id 14878 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2524011900 con 0x7f252c1323b0 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _renew_subs 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c122580 con 0x7f252c1323b0 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524011cc0 con 0x7f252c1323b0 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f252a7fc700 4 set_mon_vals no callback set 2024-09-18T10:12:27.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f2523fff700 10 monclient: dump: 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _renew_subs 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c121620 con 0x7f252c1323b0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.705+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c063ed0 con 0x7f252c1323b0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2524012630 con 0x7f252c1323b0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f2523fff700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 0x7f250c037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f2524048640 con 0x7f252c1323b0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f252bfff700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 0x7f250c037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:27.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f252bfff700 10 monclient: get_auth_request con 0x7f250c035aa0 auth_method 0 2024-09-18T10:12:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.709+0000 7f252bfff700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 0x7f250c037f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f251c009fd0 tx=0x7f251c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.717+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2524010ac0 con 0x7f252c1323b0 2024-09-18T10:12:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.717+0000 7f2523fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:27.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.717+0000 7f2523fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:27.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f253289d700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f253289d700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f253289d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f252c067b10 con 0x7f252c1323b0 2024-09-18T10:12:27.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7f25240102e0 con 0x7f252c1323b0 2024-09-18T10:12:27.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f2523fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.869+0000 7f2523fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:27.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.957+0000 7f253289d700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:27.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.957+0000 7f253289d700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.957+0000 7f253289d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.957+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f252c123780 con 0x7f252c1323b0 2024-09-18T10:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f2523fff700 1 -- 172.21.15.150:0/2217599539 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7f25240104c0 con 0x7f252c1323b0 2024-09-18T10:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f2523fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:27.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f2523fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 msgr2=0x7f250c037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 0x7f250c037f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f251c009fd0 tx=0x7f251c018000 comp rx=0 tx=0).stop 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 10 monclient: shutdown 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 msgr2=0x7f252c11ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f252400c5b0 tx=0x7f2524007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 shutdown_connections 2024-09-18T10:12:27.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f250c035aa0 0x7f250c037f50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f252c1323b0 0x7f252c11ef10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f252c0a3dc0 0x7f252c142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 --2- 172.21.15.150:0/2217599539 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f252c0a39f0 0x7f252c141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 >> 172.21.15.150:0/2217599539 conn(0x7f252c0a6300 msgr2=0x7f252c0a0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 shutdown_connections 2024-09-18T10:12:27.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:27.961+0000 7f253289d700 1 -- 172.21.15.150:0/2217599539 wait complete. 2024-09-18T10:12:27.976 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:27.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 2 2024-09-18T10:12:27.977 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c host2 2024-09-18T10:12:28.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: get_monmap_and_config 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: monmap: 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:28.074675+0000 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:28.074675+0000 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:28.075 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:28.076 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:28.076 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:28.076 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 1 Processor -- start 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 1 -- start start 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: init 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.069+0000 7fedc491c700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fedc009c2e0 0x7fedc00956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-a con 0x7fedc009c2e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc0098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-b con 0x7fedc0098a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedc0099360 0x7fedc0135700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-c con 0x7fedc0099360 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc0098580 con 0x7fedc009c2e0 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc009af60 con 0x7fedc0099360 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc009b520 con 0x7fedc0098a50 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient(hunting): authenticate will time out at 1030.168080s 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedc0099360 0x7fedc0135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedc0099360 0x7fedc0135700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44942/0 (socket says 172.21.15.150:44942) 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 -- 172.21.15.150:0/948371499 learned_addr learned my addr 172.21.15.150:0/948371499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:28.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): get_auth_request con 0x7fedc0099360 auth_method 0 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbdd9b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 msgr2=0x7fedc0098e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 1 -- 172.21.15.150:0/948371499 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbdd9b700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc0098e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbe59c700 1 --2- 172.21.15.150:0/948371499 >> v2:172.21.15.142:3300/0 conn(0x7fedc009c2e0 0x7fedc00956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbe59c700 10 monclient(hunting): get_auth_request con 0x7fedc009c2e0 auth_method 0 2024-09-18T10:12:28.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbe59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient(hunting): handle_auth_done global_id 14881 payload 274 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 -- 172.21.15.150:0/948371499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 msgr2=0x7fedc0098e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc0098e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 -- 172.21.15.150:0/948371499 >> v2:172.21.15.142:3300/0 conn(0x7fedc009c2e0 msgr2=0x7fedc00956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 --2- 172.21.15.150:0/948371499 >> v2:172.21.15.142:3300/0 conn(0x7fedc009c2e0 0x7fedc00956d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 monclient: found mon.noname-c 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 -- 172.21.15.150:0/948371499 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc009c6b0 con 0x7fedc0099360 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 -- 172.21.15.150:0/948371499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 msgr2=0x7fedc0135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbed9d700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc0135700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009f20 tx=0x7fedb8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20ec6c067f9f3273 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 1 -- 172.21.15.150:0/948371499 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb800e070 con 0x7fedc0099360 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: dump: 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.088 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 5 monclient: authenticate success, global_id 14881 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 1 -- 172.21.15.150:0/948371499 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb800bc50 con 0x7fedc0099360 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 4 set_mon_vals no callback set 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 1 -- 172.21.15.150:0/948371499 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb80092e0 con 0x7fedc0099360 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:28.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedbd59a700 10 monclient: dump: 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 10 monclient: shutdown 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 -- 172.21.15.150:0/948371499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 msgr2=0x7fedc0135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.073+0000 7fedc491c700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc0135700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fedb8009f20 tx=0x7fedb8009840 comp rx=0 tx=0).stop 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- 172.21.15.150:0/948371499 shutdown_connections 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc0135700 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- 172.21.15.150:0/948371499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc0098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- 172.21.15.150:0/948371499 >> v2:172.21.15.142:3300/0 conn(0x7fedc009c2e0 0x7fedc00956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- 172.21.15.150:0/948371499 >> 172.21.15.150:0/948371499 conn(0x7fedc00a1710 msgr2=0x7fedc00a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- 172.21.15.150:0/948371499 shutdown_connections 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- 172.21.15.150:0/948371499 wait complete. 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient: monmap: 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:28.091 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 Processor -- start 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- start start 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient: init 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc008dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-c con 0x7fedc0098a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:28.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-b con 0x7fedc0099360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 0x7fedc008e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): picked mon.noname-a con 0x7fedc009c2e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc00976a0 con 0x7fedc009c2e0 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0091360 con 0x7fedc0099360 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0091080 con 0x7fedc0098a50 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedc491c700 10 monclient(hunting): authenticate will time out at 1030.173870s 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbe59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 msgr2=0x7fedc008dc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbe59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc008dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:28.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44952/0 (socket says 172.21.15.150:44952) 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 -- 172.21.15.150:0/3186881036 learned_addr learned my addr 172.21.15.150:0/3186881036 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): get_auth_request con 0x7fedc0099360 auth_method 0 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 0x7fedc008e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 10 monclient(hunting): get_auth_request con 0x7fedc009c2e0 auth_method 0 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient(hunting): handle_auth_done global_id 14884 payload 824 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 -- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 msgr2=0x7fedc008dc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc008dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 -- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 msgr2=0x7fedc008e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 0x7fedc008e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 monclient: found mon.noname-b 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbdd9b700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8009ae0 con 0x7fedc0099360 2024-09-18T10:12:28.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.077+0000 7fedbed9d700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 0x7fedc008e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedbdd9b700 1 -- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 msgr2=0x7fedc008e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedbdd9b700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feda800b7e0 tx=0x7feda8009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8015070 con 0x7fedc0099360 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: dump: 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.098 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 5 monclient: authenticate success, global_id 14884 2024-09-18T10:12:28.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda8003e20 con 0x7fedc0099360 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8006380 con 0x7fedc0099360 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _renew_subs 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc0091640 con 0x7fedc0099360 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedbcd99700 4 set_mon_vals no callback set 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 10 monclient: dump: 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.100 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _renew_subs 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc0090120 con 0x7fedc0099360 2024-09-18T10:12:28.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7feda8006770 con 0x7fedc0099360 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc0095180 con 0x7fedc0099360 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 0x7feda0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7feda8048520 con 0x7fedc0099360 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedbe59c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 0x7feda0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.081+0000 7fedbe59c700 10 monclient: get_auth_request con 0x7feda0035ac0 auth_method 0 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.085+0000 7fedbe59c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 0x7feda0037f70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fedb0006fd0 tx=0x7fedb0009530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.089+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7feda8010440 con 0x7fedc0099360 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.089+0000 7fedb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.089+0000 7fedb67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:28.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.237+0000 7fedc491c700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:28.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.237+0000 7fedc491c700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:28.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.237+0000 7fedc491c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.237+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fedc0063ed0 con 0x7fedc0099360 2024-09-18T10:12:28.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.241+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7feda8018310 con 0x7fedc0099360 2024-09-18T10:12:28.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.241+0000 7fedb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:28.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.241+0000 7fedb67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedc491c700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedc491c700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedc491c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fedc00997a0 con 0x7fedc0099360 2024-09-18T10:12:28.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedb67fc700 1 -- 172.21.15.150:0/3186881036 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v76) v1 ==== 56+0+1541 (secure 0 0 0) 0x7feda80184f0 con 0x7fedc0099360 2024-09-18T10:12:28.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedb67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:28.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.329+0000 7fedb67fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:28.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 msgr2=0x7feda0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 0x7feda0037f70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fedb0006fd0 tx=0x7fedb0009530 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 10 monclient: shutdown 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 msgr2=0x7fedc008e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7feda800b7e0 tx=0x7feda8009220 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 shutdown_connections 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feda0035ac0 0x7feda0037f70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedc009c2e0 0x7fedc008e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedc0099360 0x7fedc008e1d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 --2- 172.21.15.150:0/3186881036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedc0098a50 0x7fedc008dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 >> 172.21.15.150:0/3186881036 conn(0x7fedc00a1710 msgr2=0x7fedc0096f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 shutdown_connections 2024-09-18T10:12:28.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.333+0000 7fedc491c700 1 -- 172.21.15.150:0/3186881036 wait complete. 2024-09-18T10:12:28.346 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rm host1 foo 2024-09-18T10:12:28.346 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:28.346 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm host1 foo 2024-09-18T10:12:28.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.429+0000 7fd335914700 10 monclient: get_monmap_and_config 2024-09-18T10:12:28.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.429+0000 7fd335914700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.429+0000 7fd335914700 10 monclient: monmap: 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:28.435405+0000 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:28.435405+0000 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:28.436 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:28.437 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:28.437 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:28.437 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:28.437 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 Processor -- start 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 -- start start 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient: init 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:12:28.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd33009cf90 0x7fd3300a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): picked mon.noname-a con 0x7fd33009cf90 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd3300a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): picked mon.noname-b con 0x7fd33009e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3300990a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33009e060 con 0x7fd3300990a0 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3300a66c0 con 0x7fd33009cf90 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33009d750 con 0x7fd33009e480 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd335914700 10 monclient(hunting): authenticate will time out at 1030.528946s 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32f7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd3300990a0 0x7fd330138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57878/0 (socket says 172.21.15.150:57878) 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1427472407 learned_addr learned my addr 172.21.15.150:0/1427472407 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32e7fc700 1 -- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 msgr2=0x7fd3300a07d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32e7fc700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd3300a07d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32dffb700 1 -- 172.21.15.150:0/1427472407 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 --2- 172.21.15.150:0/1427472407 >> v2:172.21.15.142:3300/0 conn(0x7fd33009cf90 0x7fd3300a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): get_auth_request con 0x7fd33009cf90 auth_method 0 2024-09-18T10:12:28.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient(hunting): handle_auth_done global_id 5145 payload 274 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 -- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 msgr2=0x7fd3300a07d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd3300a07d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 -- 172.21.15.150:0/1427472407 >> v1:172.21.15.150:6789/0 conn(0x7fd3300990a0 legacy=0x7fd330138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:28.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 monclient: found mon.noname-a 2024-09-18T10:12:28.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:28.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.433+0000 7fd32effd700 1 -- 172.21.15.150:0/1427472407 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33009d360 con 0x7fd33009cf90 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 1 -- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 msgr2=0x7fd3300a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3300a0290 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3200098e0 tx=0x7fd3200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ab5259d8ab4bed server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 1 -- 172.21.15.150:0/1427472407 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000ba70 con 0x7fd33009cf90 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: dump: 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.448 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 5 monclient: authenticate success, global_id 5145 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 1 -- 172.21.15.150:0/1427472407 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32000bbf0 con 0x7fd33009cf90 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 4 set_mon_vals no callback set 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 1 -- 172.21.15.150:0/1427472407 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320011420 con 0x7fd33009cf90 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32dffb700 10 monclient: dump: 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.450 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: shutdown 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 msgr2=0x7fd3300a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3300a0290 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd3200098e0 tx=0x7fd3200091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- 172.21.15.150:0/1427472407 shutdown_connections 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd3300a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- 172.21.15.150:0/1427472407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3300a0290 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- 172.21.15.150:0/1427472407 >> 172.21.15.150:0/1427472407 conn(0x7fd3300a58a0 msgr2=0x7fd3300a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- 172.21.15.150:0/1427472407 shutdown_connections 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- 172.21.15.150:0/1427472407 wait complete. 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: monmap: 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:28.452 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 Processor -- start 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- start start 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: init 2024-09-18T10:12:28.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3300990a0 0x7fd330141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): picked mon.noname-b con 0x7fd3300990a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3301421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): picked mon.noname-a con 0x7fd33009cf90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd33008ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): picked mon.noname-c con 0x7fd33009e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300a40f0 con 0x7fd33009cf90 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330090aa0 con 0x7fd3300990a0 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd330091670 con 0x7fd33009e480 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd335914700 10 monclient(hunting): authenticate will time out at 1030.535483s 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3300990a0 0x7fd330141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32f7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 msgr2=0x7fd33008ced0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3300990a0 0x7fd330141c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44962/0 (socket says 172.21.15.150:44962) 2024-09-18T10:12:28.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 1 -- 172.21.15.150:0/1083499909 learned_addr learned my addr 172.21.15.150:0/1083499909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): get_auth_request con 0x7fd3300990a0 auth_method 0 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32f7fe700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd33008ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32e7fc700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3301421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32e7fc700 10 monclient(hunting): get_auth_request con 0x7fd33009cf90 auth_method 0 2024-09-18T10:12:28.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.437+0000 7fd32e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 10 monclient(hunting): handle_auth_done global_id 14887 payload 824 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 -- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 msgr2=0x7fd33008ced0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd33009e480 0x7fd33008ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 -- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 msgr2=0x7fd3301421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3301421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 monclient: found mon.noname-b 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 -- 172.21.15.150:0/1083499909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320009420 con 0x7fd3300990a0 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32e7fc700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd33009cf90 0x7fd3301421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 -- 172.21.15.150:0/1083499909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3300990a0 msgr2=0x7fd330141c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32effd700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3300990a0 0x7fd330141c80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd32000b3c0 tx=0x7fd320016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32001b450 con 0x7fd3300990a0 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:28.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: dump: 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.459 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 5 monclient: authenticate success, global_id 14887 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd320020070 con 0x7fd3300990a0 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 10 monclient: _renew_subs 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 1 -- 172.21.15.150:0/1083499909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3300901b0 con 0x7fd3300990a0 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32001b7e0 con 0x7fd3300990a0 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd316ffd700 10 monclient: dump: 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.461 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd32cff9700 4 set_mon_vals no callback set 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 10 monclient: _renew_subs 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.441+0000 7fd335914700 1 -- 172.21.15.150:0/1083499909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3300904e0 con 0x7fd3300990a0 2024-09-18T10:12:28.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd335914700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd335914700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd335914700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd335914700 1 -- 172.21.15.150:0/1083499909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330090900 con 0x7fd3300990a0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd320022030 con 0x7fd3300990a0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd316ffd700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd31c035aa0 0x7fd31c037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7fd32004a930 con 0x7fd3300990a0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd32e7fc700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd31c035aa0 0x7fd31c037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd32e7fc700 10 monclient: get_auth_request con 0x7fd31c035aa0 auth_method 0 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:28.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.445+0000 7fd32e7fc700 1 --2- 172.21.15.150:0/1083499909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd31c035aa0 0x7fd31c037f50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd324004fd0 tx=0x7fd324004aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.453+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd330090900 con 0x7fd3300990a0 2024-09-18T10:12:28.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.453+0000 7fd316ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.453+0000 7fd316ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:28.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd335914700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:28.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd335914700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:28.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd335914700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd335914700 1 -- 172.21.15.150:0/1083499909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7fd330090900 con 0x7fd3300990a0 2024-09-18T10:12:28.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd316ffd700 1 -- 172.21.15.150:0/1083499909 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}]=-39 (39) Directory not empty v76) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd330090900 con 0x7fd3300990a0 2024-09-18T10:12:28.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd316ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:28.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.609+0000 7fd316ffd700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T10:12:28.616 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T10:12:28.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:28.619 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush unlink host1 foo 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: get_monmap_and_config 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: monmap: 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:28.708799+0000 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:28.708799+0000 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:28.709 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:28.710 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:28.710 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 Processor -- start 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 -- start start 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: init 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b300a6090 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b3009e460 0x7f0b3009d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-a con 0x7f0b3009e460 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b300a0a80 0x7f0b30139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-c con 0x7f0b300a0a80 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300a6460 con 0x7f0b300a6090 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3009fd00 con 0x7f0b3009e460 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b300a6850 con 0x7f0b300a0a80 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b358da700 10 monclient(hunting): authenticate will time out at 1030.802885s 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b300a0a80 0x7f0b30139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b300a0a80 0x7f0b30139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44976/0 (socket says 172.21.15.150:44976) 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 -- 172.21.15.150:0/3024295708 learned_addr learned my addr 172.21.15.150:0/3024295708 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f0b300a0a80 auth_method 0 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b267fc700 1 --2- 172.21.15.150:0/3024295708 >> v2:172.21.15.142:3300/0 conn(0x7f0b3009e460 0x7f0b3009d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b267fc700 10 monclient(hunting): get_auth_request con 0x7f0b3009e460 auth_method 0 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_done global_id 14890 payload 274 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 -- 172.21.15.150:0/3024295708 >> v2:172.21.15.142:3300/0 conn(0x7f0b3009e460 msgr2=0x7f0b3009d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 --2- 172.21.15.150:0/3024295708 >> v2:172.21.15.142:3300/0 conn(0x7f0b3009e460 0x7f0b3009d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.715 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 -- 172.21.15.150:0/3024295708 >> v1:172.21.15.142:6789/0 conn(0x7f0b300a6090 legacy=0x7f0b300a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.705+0000 7f0b2f7fe700 1 -- 172.21.15.150:0/3024295708 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3009d7a0 con 0x7f0b300a0a80 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2f7fe700 1 -- 172.21.15.150:0/3024295708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 msgr2=0x7f0b30139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2f7fe700 1 --2- 172.21.15.150:0/3024295708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b30139b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a1850 tx=0x7f0b28009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd7d376719e34a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 1 -- 172.21.15.150:0/3024295708 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800e070 con 0x7f0b300a0a80 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: dump: 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.721 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 5 monclient: authenticate success, global_id 14890 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 1 -- 172.21.15.150:0/3024295708 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2800bc60 con 0x7f0b300a0a80 2024-09-18T10:12:28.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 4 set_mon_vals no callback set 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 1 -- 172.21.15.150:0/3024295708 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b280092e0 con 0x7f0b300a0a80 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b2e7fc700 10 monclient: dump: 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.723 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 10 monclient: shutdown 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 -- 172.21.15.150:0/3024295708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 msgr2=0x7f0b30139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 --2- 172.21.15.150:0/3024295708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b30139b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a1850 tx=0x7f0b28009840 comp rx=0 tx=0).stop 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 -- 172.21.15.150:0/3024295708 shutdown_connections 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 --2- 172.21.15.150:0/3024295708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b30139b10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 --2- 172.21.15.150:0/3024295708 >> v2:172.21.15.142:3300/0 conn(0x7f0b3009e460 0x7f0b3009d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 -- 172.21.15.150:0/3024295708 >> 172.21.15.150:0/3024295708 conn(0x7f0b30071830 msgr2=0x7f0b300a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 -- 172.21.15.150:0/3024295708 shutdown_connections 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 1 -- 172.21.15.150:0/3024295708 wait complete. 2024-09-18T10:12:28.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 10 monclient: build_initial_monmap 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.709+0000 7f0b358da700 10 monclient: monmap: 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 Processor -- start 2024-09-18T10:12:28.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 -- start start 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient: init 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 0x7f0b3008ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-c con 0x7f0b3009e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-b con 0x7f0b300a0a80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a6090 0x7f0b3008d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): picked mon.noname-a con 0x7f0b300a6090 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30097870 con 0x7f0b300a6090 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30098700 con 0x7f0b300a0a80 2024-09-18T10:12:28.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300a4140 con 0x7f0b3009e460 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b358da700 10 monclient(hunting): authenticate will time out at 1030.809501s 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2effd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 msgr2=0x7f0b3008ce50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 0x7f0b3008ce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44992/0 (socket says 172.21.15.150:44992) 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 -- 172.21.15.150:0/4175491198 learned_addr learned my addr 172.21.15.150:0/4175491198 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): get_auth_request con 0x7f0b300a0a80 auth_method 0 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a6090 0x7f0b3008d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f0b300a6090 auth_method 0 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:28.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient(hunting): handle_auth_done global_id 14893 payload 824 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 -- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 msgr2=0x7f0b3008ce50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 0x7f0b3008ce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 -- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a6090 msgr2=0x7f0b3008d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a6090 0x7f0b3008d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient: _finish_hunting 0 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 monclient: found mon.noname-b 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.713+0000 7f0b267fc700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28009ae0 con 0x7f0b300a0a80 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b267fc700 1 -- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 msgr2=0x7f0b3008d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b267fc700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800d8f0 tx=0x7f0b18008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18012800 con 0x7f0b300a0a80 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: dump: 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 5 monclient: authenticate success, global_id 14893 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18012980 con 0x7f0b300a0a80 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _renew_subs 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18012d40 con 0x7f0b300a0a80 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b300902c0 con 0x7f0b300a0a80 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b2dffb700 4 set_mon_vals no callback set 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:28.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 10 monclient: dump: 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _renew_subs 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.732 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3008f3d0 con 0x7f0b300a0a80 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3009da80 con 0x7f0b300a0a80 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b18014640 con 0x7f0b300a0a80 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 0x7f0b10037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(76..76 src has 40..76) v4 ==== 6337+0+0 (secure 0 0 0) 0x7f0b180490e0 con 0x7f0b300a0a80 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b2effd700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 0x7f0b10037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.717+0000 7f0b2effd700 10 monclient: get_auth_request con 0x7f0b10035a70 auth_method 0 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.721+0000 7f0b2effd700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 0x7f0b10037f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.725+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0b18011570 con 0x7f0b300a0a80 2024-09-18T10:12:28.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.725+0000 7f0b277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:28.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.725+0000 7f0b277fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:28.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.881+0000 7f0b358da700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:28.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.881+0000 7f0b358da700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:28.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.881+0000 7f0b358da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:28.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:28.881+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f0b30063ed0 con 0x7f0b300a0a80 2024-09-18T10:12:29.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.301+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 removed item id -11 name 'host1' from crush map v77) v1 ==== 147+0+0 (secure 0 0 0) 0x7f0b1801dbb0 con 0x7f0b300a0a80 2024-09-18T10:12:29.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.301+0000 7f0b277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.301+0000 7f0b277fe700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T10:12:29.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.401+0000 7f0b358da700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.401+0000 7f0b358da700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.401+0000 7f0b358da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.401+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f0b300008d0 con 0x7f0b300a0a80 2024-09-18T10:12:29.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b277fe700 1 -- 172.21.15.150:0/4175491198 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v77) v1 ==== 141+0+0 (secure 0 0 0) 0x7f0b1801dd90 con 0x7f0b300a0a80 2024-09-18T10:12:29.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b277fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b277fe700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T10:12:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 msgr2=0x7f0b10037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 0x7f0b10037f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20007b00 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b358da700 10 monclient: shutdown 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b358da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.405+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 msgr2=0x7f0b3008d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800d8f0 tx=0x7f0b18008040 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 shutdown_connections 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035a70 0x7f0b10037f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a6090 0x7f0b3008d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b300a0a80 0x7f0b3008d390 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 --2- 172.21.15.150:0/4175491198 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3009e460 0x7f0b3008ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 >> 172.21.15.150:0/4175491198 conn(0x7f0b30071830 msgr2=0x7f0b3009ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:29.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 shutdown_connections 2024-09-18T10:12:29.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.409+0000 7f0b358da700 1 -- 172.21.15.150:0/4175491198 wait complete. 2024-09-18T10:12:29.414 INFO:tasks.workunit.client.0.smithi150.stderr:item -11 does not appear in that position 2024-09-18T10:12:29.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush unlink host1 foo 2024-09-18T10:12:29.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: get_monmap_and_config 2024-09-18T10:12:29.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: build_initial_monmap 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: monmap: 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:29.524274+0000 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:29.524274+0000 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:29.525 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:29.526 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 Processor -- start 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 -- start start 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: init 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f506809e6d0 0x7f50680a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): picked mon.noname-c con 0x7f506809e6d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f50680a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): picked mon.noname-b con 0x7f50680a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:29.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5068136a10 0x7f5068138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): picked mon.noname-a con 0x7f5068136a10 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506809ee30 con 0x7f5068136a10 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50680a1110 con 0x7f506809e6d0 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506809e220 con 0x7f50680a07d0 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f506d624700 10 monclient(hunting): authenticate will time out at 1031.617715s 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50667fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 msgr2=0x7f50680a0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f506809e6d0 0x7f50680a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50667fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f50680a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f506809e6d0 0x7f50680a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55724/0 (socket says 172.21.15.150:55724) 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 -- 172.21.15.150:0/3556954389 learned_addr learned my addr 172.21.15.150:0/3556954389 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): get_auth_request con 0x7f506809e6d0 auth_method 0 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 1 --2- 172.21.15.150:0/3556954389 >> v2:172.21.15.142:3300/0 conn(0x7f5068136a10 0x7f5068138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): get_auth_request con 0x7f5068136a10 auth_method 0 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient(hunting): handle_auth_done global_id 14896 payload 274 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 -- 172.21.15.150:0/3556954389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 msgr2=0x7f50680a0bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3556954389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f50680a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 -- 172.21.15.150:0/3556954389 >> v2:172.21.15.142:3300/0 conn(0x7f5068136a10 msgr2=0x7f5068138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3556954389 >> v2:172.21.15.142:3300/0 conn(0x7f5068136a10 0x7f5068138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 monclient: found mon.noname-c 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.521+0000 7f5066ffd700 1 -- 172.21.15.150:0/3556954389 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506809eaa0 con 0x7f506809e6d0 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 1 -- 172.21.15.150:0/3556954389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f506809e6d0 msgr2=0x7f50680a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3556954389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f506809e6d0 0x7f50680a0290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f506806a940 tx=0x7f505c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8caa61d4595260c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 1 -- 172.21.15.150:0/3556954389 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c009e70 con 0x7f506809e6d0 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: dump: 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.540 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 5 monclient: authenticate success, global_id 14896 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 1 -- 172.21.15.150:0/3556954389 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c00ba70 con 0x7f506809e6d0 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 4 set_mon_vals no callback set 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 1 -- 172.21.15.150:0/3556954389 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c00be30 con 0x7f506809e6d0 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5065ffb700 10 monclient: dump: 2024-09-18T10:12:29.541 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: shutdown 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- 172.21.15.150:0/3556954389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f506809e6d0 msgr2=0x7f50680a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- 172.21.15.150:0/3556954389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f506809e6d0 0x7f50680a0290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f506806a940 tx=0x7f505c009250 comp rx=0 tx=0).stop 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- 172.21.15.150:0/3556954389 shutdown_connections 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- 172.21.15.150:0/3556954389 >> v2:172.21.15.142:3300/0 conn(0x7f5068136a10 0x7f5068138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- 172.21.15.150:0/3556954389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f50680a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- 172.21.15.150:0/3556954389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f506809e6d0 0x7f50680a0290 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- 172.21.15.150:0/3556954389 >> 172.21.15.150:0/3556954389 conn(0x7f50680a58a0 msgr2=0x7f50680a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- 172.21.15.150:0/3556954389 shutdown_connections 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- 172.21.15.150:0/3556954389 wait complete. 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: build_initial_monmap 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: monmap: 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:29.543 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 Processor -- start 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- start start 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: init 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:29.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f506809e6d0 0x7f506812c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): picked mon.noname-a con 0x7f506809e6d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f506812cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): picked mon.noname-c con 0x7f50680a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): picked mon.noname-b con 0x7f5068136a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506812e640 con 0x7f506809e6d0 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50680a1340 con 0x7f5068136a10 2024-09-18T10:12:29.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506809be90 con 0x7f50680a07d0 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f506d624700 10 monclient(hunting): authenticate will time out at 1031.623230s 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50667fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 msgr2=0x7f506812cf30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50667fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f506812cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55734/0 (socket says 172.21.15.150:55734) 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 -- 172.21.15.150:0/3863973171 learned_addr learned my addr 172.21.15.150:0/3863973171 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): get_auth_request con 0x7f5068136a10 auth_method 0 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f506809e6d0 0x7f506812c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): get_auth_request con 0x7f506809e6d0 auth_method 0 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f5066ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient(hunting): handle_auth_done global_id 14899 payload 824 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 -- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 msgr2=0x7f506812cf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:29.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f506812cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 -- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f506809e6d0 msgr2=0x7f506812c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f506809e6d0 0x7f506812c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:29.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:29.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.525+0000 7f50677fe700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0094f0 con 0x7f5068136a10 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f50677fe700 1 -- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 msgr2=0x7f506812d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f50677fe700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5060009f20 tx=0x7f506000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506000cde0 con 0x7f5068136a10 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: dump: 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.549 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: _finish_auth 0 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 5 monclient: authenticate success, global_id 14899 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _renew_subs 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506000f040 con 0x7f5068136a10 2024-09-18T10:12:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506812ece0 con 0x7f5068136a10 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600146d0 con 0x7f5068136a10 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f504effd700 10 monclient: dump: 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.551 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _renew_subs 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506812e190 con 0x7f5068136a10 2024-09-18T10:12:29.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f50657fa700 4 set_mon_vals no callback set 2024-09-18T10:12:29.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.529+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5060014a80 con 0x7f5068136a10 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f504effd700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 0x7f5054037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(77..77 src has 40..77) v4 ==== 6329+0+0 (secure 0 0 0) 0x7f5060049790 con 0x7f5068136a10 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 0x7f5054037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f5066ffd700 10 monclient: get_auth_request con 0x7f5054035a70 auth_method 0 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.533+0000 7f5066ffd700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 0x7f5054037f20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f505c0094c0 tx=0x7f505c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.537+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.537+0000 7f504effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.537+0000 7f504effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.701+0000 7f506d624700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.701+0000 7f506d624700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.701+0000 7f506d624700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.701+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.705+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v77) v1 ==== 141+0+0 (secure 0 0 0) 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.705+0000 7f504effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.705+0000 7f504effd700 10 monclient: _finish_command 2 = system:0 item -11 does not appear in that position 2024-09-18T10:12:29.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f506d624700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f506d624700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f506d624700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"} v 0) v1 -- 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f504effd700 1 -- 172.21.15.150:0/3863973171 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}]=0 item -11 does not appear in that position v77) v1 ==== 141+0+0 (secure 0 0 0) 0x7f50680a3110 con 0x7f5068136a10 2024-09-18T10:12:29.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f504effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "foo"}] 2024-09-18T10:12:29.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.805+0000 7f504effd700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 msgr2=0x7f5054037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 0x7f5054037f20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f505c0094c0 tx=0x7f505c017000 comp rx=0 tx=0).stop 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 10 monclient: shutdown 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 msgr2=0x7f506812d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5060009f20 tx=0x7f506000b040 comp rx=0 tx=0).stop 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 shutdown_connections 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5054035a70 0x7f5054037f20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5068136a10 0x7f506812d470 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50680a07d0 0x7f506812cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 --2- 172.21.15.150:0/3863973171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f506809e6d0 0x7f506812c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 >> 172.21.15.150:0/3863973171 conn(0x7f50680a58a0 msgr2=0x7f506809b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:29.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 shutdown_connections 2024-09-18T10:12:29.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.809+0000 7f506d624700 1 -- 172.21.15.150:0/3863973171 wait complete. 2024-09-18T10:12:29.815 INFO:tasks.workunit.client.0.smithi150.stderr:item -11 does not appear in that position 2024-09-18T10:12:29.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:29.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c host1 2024-09-18T10:12:29.823 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 1 2024-09-18T10:12:29.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.905+0000 7fec0ee99700 10 monclient: get_monmap_and_config 2024-09-18T10:12:29.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.905+0000 7fec0ee99700 10 monclient: build_initial_monmap 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.905+0000 7fec0ee99700 10 monclient: monmap: 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:29.911622+0000 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:29.911622+0000 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:29.912 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:29.913 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 Processor -- start 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 -- start start 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient: init 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec0809ac60 0x7fec0809c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-c con 0x7fec0809ac60 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec0806f9e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 0x7fec08137af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-b con 0x7fec08135710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0809b9e0 con 0x7fec0806f9e0 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0809b780 con 0x7fec0809ac60 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0809b030 con 0x7fec08135710 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0ee99700 10 monclient(hunting): authenticate will time out at 1032.004925s 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0d436700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 msgr2=0x7fec08137af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0d436700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 0x7fec08137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec0809ac60 0x7fec0809c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec0809ac60 0x7fec0809c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55742/0 (socket says 172.21.15.150:55742) 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 -- 172.21.15.150:0/1762031781 learned_addr learned my addr 172.21.15.150:0/1762031781 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): get_auth_request con 0x7fec0809ac60 auth_method 0 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient(hunting): handle_auth_done global_id 14902 payload 274 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 -- 172.21.15.150:0/1762031781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 msgr2=0x7fec08137af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 --2- 172.21.15.150:0/1762031781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 0x7fec08137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 -- 172.21.15.150:0/1762031781 >> v1:172.21.15.142:6789/0 conn(0x7fec0806f9e0 legacy=0x7fec0809c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient: _finish_hunting 0 2024-09-18T10:12:29.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 monclient: found mon.noname-c 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.909+0000 7fec0cc35700 1 -- 172.21.15.150:0/1762031781 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08098270 con 0x7fec0809ac60 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0cc35700 1 -- 172.21.15.150:0/1762031781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 msgr2=0x7fec0809c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0cc35700 1 --2- 172.21.15.150:0/1762031781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec0809c2e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7febfc0099b0 tx=0x7febfc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3b960dd3029cc03 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 1 -- 172.21.15.150:0/1762031781 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00bb50 con 0x7fec0809ac60 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: dump: 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.922 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 5 monclient: authenticate success, global_id 14902 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 1 -- 172.21.15.150:0/1762031781 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00bcb0 con 0x7fec0809ac60 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 1 -- 172.21.15.150:0/1762031781 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc011670 con 0x7fec0809ac60 2024-09-18T10:12:29.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec077fe700 10 monclient: dump: 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 4 set_mon_vals no callback set 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 10 monclient: shutdown 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- 172.21.15.150:0/1762031781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 msgr2=0x7fec0809c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 --2- 172.21.15.150:0/1762031781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec0809c2e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7febfc0099b0 tx=0x7febfc009250 comp rx=0 tx=0).stop 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- 172.21.15.150:0/1762031781 shutdown_connections 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 --2- 172.21.15.150:0/1762031781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec08135710 0x7fec08137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 --2- 172.21.15.150:0/1762031781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec0809c2e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- 172.21.15.150:0/1762031781 >> 172.21.15.150:0/1762031781 conn(0x7fec080a1710 msgr2=0x7fec080a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- 172.21.15.150:0/1762031781 shutdown_connections 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- 172.21.15.150:0/1762031781 wait complete. 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 10 monclient: build_initial_monmap 2024-09-18T10:12:29.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 10 monclient: monmap: 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:29.926 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 Processor -- start 2024-09-18T10:12:29.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 1 -- start start 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.913+0000 7fec0ee99700 10 monclient: init 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 0x7fec08134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-c con 0x7fec0806f9e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-b con 0x7fec0809ac60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec08135710 0x7fec0808bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): picked mon.noname-a con 0x7fec08135710 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec08098480 con 0x7fec08135710 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0806ef40 con 0x7fec0809ac60 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0808f450 con 0x7fec0806f9e0 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0ee99700 10 monclient(hunting): authenticate will time out at 1032.013166s 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0cc35700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 msgr2=0x7fec08134950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0cc35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 0x7fec08134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55758/0 (socket says 172.21.15.150:55758) 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 -- 172.21.15.150:0/3149163870 learned_addr learned my addr 172.21.15.150:0/3149163870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): get_auth_request con 0x7fec0809ac60 auth_method 0 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec08135710 0x7fec0808bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): get_auth_request con 0x7fec08135710 auth_method 0 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec0d436700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:29.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient(hunting): handle_auth_done global_id 14905 payload 824 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 -- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 msgr2=0x7fec08134950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 0x7fec08134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 -- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec08135710 msgr2=0x7fec0808bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec08135710 0x7fec0808bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 monclient: found mon.noname-b 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.917+0000 7fec07fff700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0094f0 con 0x7fec0809ac60 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec07fff700 1 -- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 msgr2=0x7fec08134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec07fff700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7febf400c5d0 tx=0x7febf4007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40117a0 con 0x7fec0809ac60 2024-09-18T10:12:29.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: dump: 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 5 monclient: authenticate success, global_id 14905 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf4011900 con 0x7fec0809ac60 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _renew_subs 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0808e280 con 0x7fec0809ac60 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4011cc0 con 0x7fec0809ac60 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec06ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 10 monclient: dump: 2024-09-18T10:12:29.937 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _renew_subs 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec08090440 con 0x7fec0809ac60 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7febf4012630 con 0x7fec0809ac60 2024-09-18T10:12:29.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec08063ed0 con 0x7fec0809ac60 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 0x7febec037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(77..77 src has 40..77) v4 ==== 6329+0+0 (secure 0 0 0) 0x7febf4048770 con 0x7fec0809ac60 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0cc35700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 0x7febec037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.921+0000 7fec0cc35700 10 monclient: get_auth_request con 0x7febec035a70 auth_method 0 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.925+0000 7fec0cc35700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 0x7febec037f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7febfc0099b0 tx=0x7febfc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:29.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.929+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7febf4010d50 con 0x7fec0809ac60 2024-09-18T10:12:29.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.929+0000 7fec04ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:29.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:29.929+0000 7fec04ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec0ee99700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec0ee99700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec0ee99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fec08095970 con 0x7fec0809ac60 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) v1 ==== 56+0+1407 (secure 0 0 0) 0x7febf4010750 con 0x7fec0809ac60 2024-09-18T10:12:30.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec04ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:30.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.097+0000 7fec04ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:30.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:30.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:30.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fec0808f7c0 con 0x7fec0809ac60 2024-09-18T10:12:30.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec04ff9700 1 -- 172.21.15.150:0/3149163870 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v77) v1 ==== 56+0+1407 (secure 0 0 0) 0x7febf4010930 con 0x7fec0809ac60 2024-09-18T10:12:30.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec04ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:30.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec04ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 msgr2=0x7febec037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 0x7febec037f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7febfc0099b0 tx=0x7febfc016000 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 10 monclient: shutdown 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 msgr2=0x7fec08134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7febf400c5d0 tx=0x7febf4007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 shutdown_connections 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7febec035a70 0x7febec037f20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec08135710 0x7fec0808bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec0809ac60 0x7fec08134e90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 --2- 172.21.15.150:0/3149163870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec0806f9e0 0x7fec08134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.193+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 >> 172.21.15.150:0/3149163870 conn(0x7fec080a1710 msgr2=0x7fec0809df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:30.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.197+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 shutdown_connections 2024-09-18T10:12:30.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.197+0000 7fec0ee99700 1 -- 172.21.15.150:0/3149163870 wait complete. 2024-09-18T10:12:30.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rm foo 2024-09-18T10:12:30.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:30.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm foo 2024-09-18T10:12:30.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.301+0000 7fd5e9ed6700 10 monclient: get_monmap_and_config 2024-09-18T10:12:30.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.301+0000 7fd5e9ed6700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.301+0000 7fd5e9ed6700 10 monclient: monmap: 2024-09-18T10:12:30.307 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:30.307300+0000 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:30.307300+0000 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:30.308 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 Processor -- start 2024-09-18T10:12:30.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 -- start start 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient: init 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5e409e6d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5e40a09a0 0x7fd5e4137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-c con 0x7fd5e40a09a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5e40a0da0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e409e220 con 0x7fd5e409e6d0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e409eaa0 con 0x7fd5e40a0da0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e409ee30 con 0x7fd5e40a09a0 2024-09-18T10:12:30.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e9ed6700 10 monclient(hunting): authenticate will time out at 1032.400869s 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5e40a09a0 0x7fd5e4137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5e40a09a0 0x7fd5e4137a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55770/0 (socket says 172.21.15.150:55770) 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4281169224 learned_addr learned my addr 172.21.15.150:0/4281169224 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): get_auth_request con 0x7fd5e40a09a0 auth_method 0 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e3fff700 1 -- 172.21.15.150:0/4281169224 >> v1:172.21.15.150:6789/0 conn(0x7fd5e40a0da0 legacy=0x7fd5e4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4281169224 >> v1:172.21.15.142:6789/0 conn(0x7fd5e409e6d0 legacy=0x7fd5e40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient(hunting): handle_auth_done global_id 14908 payload 274 2024-09-18T10:12:30.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4281169224 >> v1:172.21.15.150:6789/0 conn(0x7fd5e40a0da0 legacy=0x7fd5e4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4281169224 >> v1:172.21.15.142:6789/0 conn(0x7fd5e409e6d0 legacy=0x7fd5e40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 monclient: found mon.noname-c 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.305+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4281169224 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e409d930 con 0x7fd5e40a09a0 2024-09-18T10:12:30.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4281169224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e40a09a0 msgr2=0x7fd5e4137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e2ffd700 1 --2- 172.21.15.150:0/4281169224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e40a09a0 0x7fd5e4137a20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009ab0 tx=0x7fd5d4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9828ab8ccbc447d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 1 -- 172.21.15.150:0/4281169224 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d400bc70 con 0x7fd5e40a09a0 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: dump: 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.316 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 5 monclient: authenticate success, global_id 14908 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 1 -- 172.21.15.150:0/4281169224 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d400bdd0 con 0x7fd5e40a09a0 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 1 -- 172.21.15.150:0/4281169224 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4011670 con 0x7fd5e40a09a0 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 4 set_mon_vals no callback set 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:30.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e27fc700 10 monclient: dump: 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 10 monclient: shutdown 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4281169224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e40a09a0 msgr2=0x7fd5e4137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 --2- 172.21.15.150:0/4281169224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e40a09a0 0x7fd5e4137a20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009ab0 tx=0x7fd5d4009250 comp rx=0 tx=0).stop 2024-09-18T10:12:30.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4281169224 shutdown_connections 2024-09-18T10:12:30.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 --2- 172.21.15.150:0/4281169224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e40a09a0 0x7fd5e4137a20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4281169224 >> 172.21.15.150:0/4281169224 conn(0x7fd5e40a58a0 msgr2=0x7fd5e40a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:30.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4281169224 shutdown_connections 2024-09-18T10:12:30.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.309+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4281169224 wait complete. 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient: monmap: 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:30.324 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 Processor -- start 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 -- start start 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient: init 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e409e6d0 0x7fd5e4141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-b con 0x7fd5e409e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:30.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5e40a09a0 0x7fd5e4142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-c con 0x7fd5e40a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5e40a0da0 0x7fd5e408e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): picked mon.noname-a con 0x7fd5e40a0da0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e4090030 con 0x7fd5e40a0da0 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e4092890 con 0x7fd5e409e6d0 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e40918c0 con 0x7fd5e40a09a0 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e9ed6700 10 monclient(hunting): authenticate will time out at 1032.410751s 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e409e6d0 0x7fd5e4141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e409e6d0 0x7fd5e4141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55780/0 (socket says 172.21.15.150:55780) 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4233277444 learned_addr learned my addr 172.21.15.150:0/4233277444 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): get_auth_request con 0x7fd5e409e6d0 auth_method 0 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e2ffd700 1 -- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5e40a09a0 msgr2=0x7fd5e4142250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e2ffd700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5e40a09a0 0x7fd5e4142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e3fff700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5e40a0da0 0x7fd5e408e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e3fff700 10 monclient(hunting): get_auth_request con 0x7fd5e40a0da0 auth_method 0 2024-09-18T10:12:30.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient(hunting): handle_auth_done global_id 14914 payload 824 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5e40a09a0 msgr2=0x7fd5e4142250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5e40a09a0 0x7fd5e4142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5e40a0da0 msgr2=0x7fd5e408e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5e40a0da0 0x7fd5e408e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.313+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4233277444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d40094f0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e37fe700 1 -- 172.21.15.150:0/4233277444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e409e6d0 msgr2=0x7fd5e4141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e37fe700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5e409e6d0 0x7fd5e4141d10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800c7c0 tx=0x7fd5d8007dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80074f0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: dump: 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.329 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d8007670 con 0x7fd5e409e6d0 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 5 monclient: authenticate success, global_id 14914 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _renew_subs 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4233277444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e408fab0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8005a60 con 0x7fd5e409e6d0 2024-09-18T10:12:30.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 10 monclient: dump: 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e1ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:30.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _renew_subs 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4233277444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e40993d0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd5d801b3b0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4233277444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e40916f0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.317+0000 7fd5cb7fe700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5d0035ac0 0x7fd5d0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.321+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(77..77 src has 40..77) v4 ==== 6329+0+0 (secure 0 0 0) 0x7fd5d8047e20 con 0x7fd5e409e6d0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.321+0000 7fd5e2ffd700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5d0035ac0 0x7fd5d0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.321+0000 7fd5e2ffd700 10 monclient: get_auth_request con 0x7fd5d0035ac0 auth_method 0 2024-09-18T10:12:30.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.321+0000 7fd5e2ffd700 1 --2- 172.21.15.150:0/4233277444 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5d0035ac0 0x7fd5d0037f70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009ab0 tx=0x7fd5d4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.325+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd5d800ea40 con 0x7fd5e409e6d0 2024-09-18T10:12:30.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.325+0000 7fd5cb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.325+0000 7fd5cb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:30.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.485+0000 7fd5e9ed6700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:30.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.485+0000 7fd5e9ed6700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:30.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.485+0000 7fd5e9ed6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.485+0000 7fd5e9ed6700 1 -- 172.21.15.150:0/4233277444 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fd5e409dbf0 con 0x7fd5e409e6d0 2024-09-18T10:12:30.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.489+0000 7fd5cb7fe700 1 -- 172.21.15.150:0/4233277444 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-39 (39) Directory not empty v77) v1 ==== 99+0+0 (secure 0 0 0) 0x7fd5d8017020 con 0x7fd5e409e6d0 2024-09-18T10:12:30.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.489+0000 7fd5cb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:30.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.489+0000 7fd5cb7fe700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T10:12:30.494 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T10:12:30.497 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:30.497 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rm bar 2024-09-18T10:12:30.497 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:30.497 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm bar 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: get_monmap_and_config 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: monmap: 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.588 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:30.588246+0000 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:30.588246+0000 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:30.589 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 Processor -- start 2024-09-18T10:12:30.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 -- start start 2024-09-18T10:12:30.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: init 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f880c0710a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f880c09ce30 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f880c09d230 0x7f880c138c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): picked mon.noname-c con 0x7f880c09d230 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880c099210 con 0x7f880c09ce30 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880c071470 con 0x7f880c0710a0 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f880c06f7c0 con 0x7f880c09d230 2024-09-18T10:12:30.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8813157700 10 monclient(hunting): authenticate will time out at 1032.683068s 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8810ef3700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f880c0710a0 0x7f880c09c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50250/0 (socket says 172.21.15.150:50250) 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f8810ef3700 1 -- 172.21.15.150:0/4000399445 learned_addr learned my addr 172.21.15.150:0/4000399445 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f88116f4700 1 --2- 172.21.15.150:0/4000399445 >> v2:172.21.15.150:3300/0 conn(0x7f880c09d230 0x7f880c138c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f88116f4700 10 monclient(hunting): get_auth_request con 0x7f880c09d230 auth_method 0 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f88116f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f88116f4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.585+0000 7f88116f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f8810ef3700 1 -- 172.21.15.150:0/4000399445 >> v1:172.21.15.150:6789/0 conn(0x7f880c0710a0 legacy=0x7f880c09c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient(hunting): handle_auth_done global_id 14917 payload 274 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 -- 172.21.15.150:0/4000399445 >> v1:172.21.15.150:6789/0 conn(0x7f880c0710a0 legacy=0x7f880c09c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 -- 172.21.15.150:0/4000399445 >> v1:172.21.15.142:6789/0 conn(0x7f880c09ce30 legacy=0x7f880c135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f880bfff700 1 -- 172.21.15.150:0/4000399445 >> v1:172.21.15.142:6789/0 conn(0x7f880c09ce30 legacy=0x7f880c135540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 monclient: found mon.noname-c 2024-09-18T10:12:30.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 -- 172.21.15.150:0/4000399445 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c099aa0 con 0x7f880c09d230 2024-09-18T10:12:30.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 -- 172.21.15.150:0/4000399445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 msgr2=0x7f880c138c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.589+0000 7f88116f4700 1 --2- 172.21.15.150:0/4000399445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c138c70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8804009f90 tx=0x7f8804009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d610d2e67ccc5632 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 1 -- 172.21.15.150:0/4000399445 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400e070 con 0x7f880c09d230 2024-09-18T10:12:30.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: dump: 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 5 monclient: authenticate success, global_id 14917 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 1 -- 172.21.15.150:0/4000399445 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804012430 con 0x7f880c09d230 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 4 set_mon_vals no callback set 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 1 -- 172.21.15.150:0/4000399445 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88040127d0 con 0x7f880c09d230 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880b7fe700 10 monclient: dump: 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.599 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: shutdown 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- 172.21.15.150:0/4000399445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 msgr2=0x7f880c138c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 --2- 172.21.15.150:0/4000399445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c138c70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8804009f90 tx=0x7f8804009780 comp rx=0 tx=0).stop 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- 172.21.15.150:0/4000399445 shutdown_connections 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 --2- 172.21.15.150:0/4000399445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c138c70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- 172.21.15.150:0/4000399445 >> 172.21.15.150:0/4000399445 conn(0x7f880c0684a0 msgr2=0x7f880c0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- 172.21.15.150:0/4000399445 shutdown_connections 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- 172.21.15.150:0/4000399445 wait complete. 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: monmap: 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:30.601 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:30.602 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 Processor -- start 2024-09-18T10:12:30.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- start start 2024-09-18T10:12:30.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: init 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f880c0710a0 0x7f880c13e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): picked mon.noname-c con 0x7f880c0710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f880c09ce30 0x7f880c13d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): picked mon.noname-a con 0x7f880c09ce30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:30.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): picked mon.noname-b con 0x7f880c09d230 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c1230b0 con 0x7f880c09ce30 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c122af0 con 0x7f880c09d230 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880c120580 con 0x7f880c0710a0 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8813157700 10 monclient(hunting): authenticate will time out at 1032.690208s 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c141630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55804/0 (socket says 172.21.15.150:55804) 2024-09-18T10:12:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 -- 172.21.15.150:0/507662055 learned_addr learned my addr 172.21.15.150:0/507662055 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): get_auth_request con 0x7f880c09d230 auth_method 0 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8810ef3700 1 -- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f880c0710a0 msgr2=0x7f880c13e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f8810ef3700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f880c0710a0 0x7f880c13e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880bfff700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f880c09ce30 0x7f880c13d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880bfff700 10 monclient(hunting): get_auth_request con 0x7f880c09ce30 auth_method 0 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f880bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient(hunting): handle_auth_done global_id 14923 payload 824 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 -- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f880c0710a0 msgr2=0x7f880c13e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f880c0710a0 0x7f880c13e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 -- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f880c09ce30 msgr2=0x7f880c13d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f880c09ce30 0x7f880c13d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 monclient: found mon.noname-b 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.593+0000 7f88116f4700 1 -- 172.21.15.150:0/507662055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8804009a20 con 0x7f880c09d230 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f88116f4700 1 -- 172.21.15.150:0/507662055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 msgr2=0x7f880c141630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f88116f4700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f880c09d230 0x7f880c141630 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f880400b770 tx=0x7f8804019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804007000 con 0x7f880c09d230 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: dump: 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880401b810 con 0x7f880c09d230 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 5 monclient: authenticate success, global_id 14923 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _renew_subs 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 1 -- 172.21.15.150:0/507662055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c122810 con 0x7f880c09d230 2024-09-18T10:12:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880401bbd0 con 0x7f880c09d230 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 10 monclient: dump: 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.611 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f880a7fc700 4 set_mon_vals no callback set 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _renew_subs 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 1 -- 172.21.15.150:0/507662055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c13dcb0 con 0x7f880c09d230 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8813157700 1 -- 172.21.15.150:0/507662055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c063ed0 con 0x7f880c09d230 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f88040263f0 con 0x7f880c09d230 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87f0035a70 0x7f87f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(77..77 src has 40..77) v4 ==== 6329+0+0 (secure 0 0 0) 0x7f880404ad40 con 0x7f880c09d230 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8810ef3700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87f0035a70 0x7f87f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.597+0000 7f8810ef3700 10 monclient: get_auth_request con 0x7f87f0035a70 auth_method 0 2024-09-18T10:12:30.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.601+0000 7f8810ef3700 1 --2- 172.21.15.150:0/507662055 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f87f0035a70 0x7f87f0037f20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8800004860 tx=0x7f8800007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.605+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f88040266a0 con 0x7f880c09d230 2024-09-18T10:12:30.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.605+0000 7f8808ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.605+0000 7f8808ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:30.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.769+0000 7f8813157700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:30.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.769+0000 7f8813157700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:30.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.769+0000 7f8813157700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.769+0000 7f8813157700 1 -- 172.21.15.150:0/507662055 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f880c071930 con 0x7f880c09d230 2024-09-18T10:12:30.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.773+0000 7f8808ff9700 1 -- 172.21.15.150:0/507662055 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v77) v1 ==== 99+0+0 (secure 0 0 0) 0x7f8804019780 con 0x7f880c09d230 2024-09-18T10:12:30.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.773+0000 7f8808ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:30.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.773+0000 7f8808ff9700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T10:12:30.778 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T10:12:30.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:30.781 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush unlink host1 bar 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: get_monmap_and_config 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: monmap: 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.880 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:30.880212+0000 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:30.880212+0000 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:30.881 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 Processor -- start 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 -- start start 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: init 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19740a0120 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19740a0c00 0x7f197409e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-a con 0x7f19740a0c00 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 0x7f1974139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-b con 0x7f19741379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197409dae0 con 0x7f19740a0120 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19740a21d0 con 0x7f19740a0c00 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197409ddc0 con 0x7f19741379b0 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197bf54700 10 monclient(hunting): authenticate will time out at 1032.973631s 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197a4f1700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 msgr2=0x7f1974139d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f197a4f1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 0x7f1974139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19740a0c00 0x7f197409e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19740a0c00 0x7f197409e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47210/0 (socket says 172.21.15.150:47210) 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f1979cf0700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f19740a0120 0x7f19740a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35102/0 (socket says 172.21.15.150:35102) 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 -- 172.21.15.150:0/1256598615 learned_addr learned my addr 172.21.15.150:0/1256598615 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): get_auth_request con 0x7f19740a0c00 auth_method 0 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient(hunting): handle_auth_done global_id 5175 payload 274 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 -- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 msgr2=0x7f1974139d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 --2- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 0x7f1974139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 -- 172.21.15.150:0/1256598615 >> v1:172.21.15.142:6789/0 conn(0x7f19740a0120 legacy=0x7f19740a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:30.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 monclient: found mon.noname-a 2024-09-18T10:12:30.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:30.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.877+0000 7f19794ef700 1 -- 172.21.15.150:0/1256598615 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740a1590 con 0x7f19740a0c00 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f19794ef700 1 -- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19740a0c00 msgr2=0x7f197409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f19794ef700 1 --2- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19740a0c00 0x7f197409e7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f19640099b0 tx=0x7f1964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c38566cfcab63f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 1 -- 172.21.15.150:0/1256598615 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196400bb50 con 0x7f19740a0c00 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: dump: 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.891 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 5 monclient: authenticate success, global_id 5175 2024-09-18T10:12:30.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 1 -- 172.21.15.150:0/1256598615 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196400bcb0 con 0x7f19740a0c00 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 4 set_mon_vals no callback set 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 1 -- 172.21.15.150:0/1256598615 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964011670 con 0x7f19740a0c00 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f1978cee700 10 monclient: dump: 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.893 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 10 monclient: shutdown 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 -- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19740a0c00 msgr2=0x7f197409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 --2- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19740a0c00 0x7f197409e7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f19640099b0 tx=0x7f1964009250 comp rx=0 tx=0).stop 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 -- 172.21.15.150:0/1256598615 shutdown_connections 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 --2- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19741379b0 0x7f1974139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 --2- 172.21.15.150:0/1256598615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19740a0c00 0x7f197409e7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 -- 172.21.15.150:0/1256598615 >> 172.21.15.150:0/1256598615 conn(0x7f19740a5830 msgr2=0x7f19740a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:30.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 -- 172.21.15.150:0/1256598615 shutdown_connections 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 1 -- 172.21.15.150:0/1256598615 wait complete. 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 10 monclient: build_initial_monmap 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.881+0000 7f197bf54700 10 monclient: monmap: 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:30.895 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 Processor -- start 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 -- start start 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient: init 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-b con 0x7f19740a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 0x7f1974098cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-c con 0x7f19740a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19741379b0 0x7f197412c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): picked mon.noname-a con 0x7f19741379b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197412eb00 con 0x7f19741379b0 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197409c010 con 0x7f19740a0120 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197409d720 con 0x7f19740a0c00 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197bf54700 10 monclient(hunting): authenticate will time out at 1032.981124s 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f19794ef700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 msgr2=0x7f1974098cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55816/0 (socket says 172.21.15.150:55816) 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 -- 172.21.15.150:0/1243137281 learned_addr learned my addr 172.21.15.150:0/1243137281 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): get_auth_request con 0x7f19740a0120 auth_method 0 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f19794ef700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 0x7f1974098cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19741379b0 0x7f197412c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): get_auth_request con 0x7f19741379b0 auth_method 0 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:30.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f197a4f1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient(hunting): handle_auth_done global_id 14926 payload 824 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 -- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 msgr2=0x7f1974098cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 0x7f1974098cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 -- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19741379b0 msgr2=0x7f197412c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19741379b0 0x7f197412c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient: _finish_hunting 0 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 monclient: found mon.noname-b 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.885+0000 7f1979cf0700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640094f0 con 0x7f19740a0120 2024-09-18T10:12:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1979cf0700 1 -- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 msgr2=0x7f1974098790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1979cf0700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f197000c5e0 tx=0x7f1970007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970005c80 con 0x7f19740a0120 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: dump: 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 5 monclient: authenticate success, global_id 14926 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _renew_subs 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970005e00 con 0x7f19740a0120 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197412eea0 con 0x7f19740a0120 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970007320 con 0x7f19740a0120 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:30.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 10 monclient: dump: 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f196b7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _renew_subs 2024-09-18T10:12:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1974131090 con 0x7f19740a0120 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f197001b3b0 con 0x7f19740a0120 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19740a53b0 con 0x7f19740a0120 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 0x7f1958037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(77..77 src has 40..77) v4 ==== 6329+0+0 (secure 0 0 0) 0x7f1970047f20 con 0x7f19740a0120 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f19794ef700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 0x7f1958037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.889+0000 7f19794ef700 10 monclient: get_auth_request con 0x7f1958035a70 auth_method 0 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.893+0000 7f19794ef700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 0x7f1958037f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f19640099b0 tx=0x7f1964016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.893+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:30.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.897+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f197000f680 con 0x7f19740a0120 2024-09-18T10:12:30.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.897+0000 7f1969ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:30.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:30.897+0000 7f1969ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:31.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.065+0000 7f197bf54700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.065+0000 7f197bf54700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.065+0000 7f197bf54700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.065+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f1974063ed0 con 0x7f19740a0120 2024-09-18T10:12:31.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.317+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 removed item id -11 name 'host1' from crush map v78) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1970017020 con 0x7f19740a0120 2024-09-18T10:12:31.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.317+0000 7f1969ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.317+0000 7f1969ffb700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T10:12:31.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f197bf54700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f197bf54700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f197bf54700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"} v 0) v1 -- 0x7f19740008d0 con 0x7f19740a0120 2024-09-18T10:12:31.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f1969ffb700 1 -- 172.21.15.150:0/1243137281 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}]=0 item -11 does not appear in that position v78) v1 ==== 141+0+0 (secure 0 0 0) 0x7f197000e290 con 0x7f19740a0120 2024-09-18T10:12:31.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f1969ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host1", "ancestor": "bar"}] 2024-09-18T10:12:31.431 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.425+0000 7f1969ffb700 10 monclient: _finish_command 3 = system:0 item -11 does not appear in that position 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 msgr2=0x7f1958037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 0x7f1958037f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f19640099b0 tx=0x7f1964016000 comp rx=0 tx=0).stop 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 10 monclient: shutdown 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 msgr2=0x7f1974098790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f197000c5e0 tx=0x7f1970007f90 comp rx=0 tx=0).stop 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 shutdown_connections 2024-09-18T10:12:31.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1958035a70 0x7f1958037f20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19741379b0 0x7f197412c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19740a0c00 0x7f1974098cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 --2- 172.21.15.150:0/1243137281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19740a0120 0x7f1974098790 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 >> 172.21.15.150:0/1243137281 conn(0x7f19740a5830 msgr2=0x7f197409ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 shutdown_connections 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.429+0000 7f197bf54700 1 -- 172.21.15.150:0/1243137281 wait complete. 2024-09-18T10:12:31.435 INFO:tasks.workunit.client.0.smithi150.stderr:item -11 does not appear in that position 2024-09-18T10:12:31.444 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:31.444 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c host1 2024-09-18T10:12:31.444 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 1 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: get_monmap_and_config 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: build_initial_monmap 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: monmap: 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:31.534583+0000 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:31.534583+0000 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:31.535 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 Processor -- start 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 -- start start 2024-09-18T10:12:31.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: init 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb04094850 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb0409ade0 0x7fdb0409b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-a con 0x7fdb0409ade0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 0x7fdb0413db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-b con 0x7fdb0413b770 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0409d640 con 0x7fdb04094850 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0409dd40 con 0x7fdb0409ade0 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb04099df0 con 0x7fdb0413b770 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb09d64700 10 monclient(hunting): authenticate will time out at 1033.627753s 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.529+0000 7fdb03fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 msgr2=0x7fdb0413db50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 0x7fdb0413db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb0409ade0 0x7fdb0409b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdb037fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdb04094850 0x7fdb04094c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35118/0 (socket says 172.21.15.150:35118) 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdb037fe700 1 -- 172.21.15.150:0/1367134350 learned_addr learned my addr 172.21.15.150:0/1367134350 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:31.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): get_auth_request con 0x7fdb0409ade0 auth_method 0 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient(hunting): handle_auth_done global_id 5181 payload 274 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 -- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 msgr2=0x7fdb0413db50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 --2- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 0x7fdb0413db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 -- 172.21.15.150:0/1367134350 >> v1:172.21.15.142:6789/0 conn(0x7fdb04094850 legacy=0x7fdb04094c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 monclient: found mon.noname-a 2024-09-18T10:12:31.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:31.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:31.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 -- 172.21.15.150:0/1367134350 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04098cc0 con 0x7fdb0409ade0 2024-09-18T10:12:31.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 -- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0409ade0 msgr2=0x7fdb0409b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.533+0000 7fdafaffd700 1 --2- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0409ade0 0x7fdb0409b1e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0098e0 tx=0x7fdaec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace21b8bad771759 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 1 -- 172.21.15.150:0/1367134350 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00ba70 con 0x7fdb0409ade0 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: dump: 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.545 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 5 monclient: authenticate success, global_id 5181 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 1 -- 172.21.15.150:0/1367134350 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec00bbf0 con 0x7fdb0409ade0 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 4 set_mon_vals no callback set 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 1 -- 172.21.15.150:0/1367134350 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec011420 con 0x7fdb0409ade0 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb02ffd700 10 monclient: dump: 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: shutdown 2024-09-18T10:12:31.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0409ade0 msgr2=0x7fdb0409b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0409ade0 0x7fdb0409b1e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0098e0 tx=0x7fdaec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- 172.21.15.150:0/1367134350 shutdown_connections 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0413b770 0x7fdb0413db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- 172.21.15.150:0/1367134350 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0409ade0 0x7fdb0409b1e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- 172.21.15.150:0/1367134350 >> 172.21.15.150:0/1367134350 conn(0x7fdb040a1710 msgr2=0x7fdb040a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- 172.21.15.150:0/1367134350 shutdown_connections 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- 172.21.15.150:0/1367134350 wait complete. 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: build_initial_monmap 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: monmap: 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:31.548 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 Processor -- start 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- start start 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: init 2024-09-18T10:12:31.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:31.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:31.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-b con 0x7fdb04094850 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 0x7fdb0413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-c con 0x7fdb0409ade0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0413b770 0x7fdb0413b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): picked mon.noname-a con 0x7fdb0413b770 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0406da70 con 0x7fdb0413b770 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0406d4b0 con 0x7fdb04094850 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0406ba10 con 0x7fdb0409ade0 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb09d64700 10 monclient(hunting): authenticate will time out at 1033.634690s 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdafaffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 msgr2=0x7fdb0413acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdafaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 0x7fdb0413acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55830/0 (socket says 172.21.15.150:55830) 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 -- 172.21.15.150:0/3134454469 learned_addr learned my addr 172.21.15.150:0/3134454469 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): get_auth_request con 0x7fdb04094850 auth_method 0 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0413b770 0x7fdb0413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): get_auth_request con 0x7fdb0413b770 auth_method 0 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient(hunting): handle_auth_done global_id 14929 payload 824 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 -- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 msgr2=0x7fdb0413acc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 0x7fdb0413acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 -- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0413b770 msgr2=0x7fdb0413b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0413b770 0x7fdb0413b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:31.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.537+0000 7fdb037fe700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec009420 con 0x7fdb04094850 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb037fe700 1 -- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 msgr2=0x7fdb0413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb037fe700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c5e0 tx=0x7fdaf4004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4005d40 con 0x7fdb04094850 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:31.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: dump: 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: _finish_auth 0 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:31.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 5 monclient: authenticate success, global_id 14929 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _renew_subs 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0406f4e0 con 0x7fdb04094850 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf40070d0 con 0x7fdb04094850 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _renew_subs 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4007490 con 0x7fdb04094850 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0406c5c0 con 0x7fdb04094850 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb027fc700 4 set_mon_vals no callback set 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 10 monclient: dump: 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb04063ed0 con 0x7fdb04094850 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.541+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdaf401b3b0 con 0x7fdb04094850 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafbfff700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 0x7fdae4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(78..78 src has 40..78) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fdaf4047f70 con 0x7fdb04094850 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafaffd700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 0x7fdae4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafaffd700 10 monclient: get_auth_request con 0x7fdae4035a70 auth_method 0 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.545+0000 7fdafaffd700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 0x7fdae4037f20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0098e0 tx=0x7fdaec016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.553+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdaf400f840 con 0x7fdb04094850 2024-09-18T10:12:31.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.553+0000 7fdafbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.553+0000 7fdafbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:31.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.713+0000 7fdb09d64700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:31.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.713+0000 7fdb09d64700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:31.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.713+0000 7fdb09d64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.713+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fdb0409a0d0 con 0x7fdb04094850 2024-09-18T10:12:31.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.717+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1407 (secure 0 0 0) 0x7fdaf400f260 con 0x7fdb04094850 2024-09-18T10:12:31.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.717+0000 7fdafbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:31.721 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.717+0000 7fdafbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:31.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdb09d64700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:31.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdb09d64700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:31.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdb09d64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fdb0409b9b0 con 0x7fdb04094850 2024-09-18T10:12:31.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdafbfff700 1 -- 172.21.15.150:0/3134454469 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v78) v1 ==== 56+0+1407 (secure 0 0 0) 0x7fdaf400f840 con 0x7fdb04094850 2024-09-18T10:12:31.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdafbfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:31.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.809+0000 7fdafbfff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 msgr2=0x7fdae4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 0x7fdae4037f20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0098e0 tx=0x7fdaec016000 comp rx=0 tx=0).stop 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 10 monclient: shutdown 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 msgr2=0x7fdb0413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c5e0 tx=0x7fdaf4004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:31.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 shutdown_connections 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdae4035a70 0x7fdae4037f20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0413b770 0x7fdb0413b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0409ade0 0x7fdb0413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 --2- 172.21.15.150:0/3134454469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb04094850 0x7fdb0413a780 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 >> 172.21.15.150:0/3134454469 conn(0x7fdb040a1710 msgr2=0x7fdb0413bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 shutdown_connections 2024-09-18T10:12:31.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.813+0000 7fdb09d64700 1 -- 172.21.15.150:0/3134454469 wait complete. 2024-09-18T10:12:31.828 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.8 host1 2024-09-18T10:12:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: get_monmap_and_config 2024-09-18T10:12:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: build_initial_monmap 2024-09-18T10:12:31.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: monmap: 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:31.925559+0000 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:31.925559+0000 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:31.926 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:31.927 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 Processor -- start 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 -- start start 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: init 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:12:31.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcc840a0110 0x7fcc840a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-a con 0x7fcc840a0110 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc8409d160 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8412c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-b con 0x7fcc840a0a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc84068c30 con 0x7fcc8409d160 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8406a6c0 con 0x7fcc840a0110 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84068e00 con 0x7fcc840a0a20 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc892a5700 10 monclient(hunting): authenticate will time out at 1034.019628s 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc88aa4700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 msgr2=0x7fcc8412c590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:31.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc88aa4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8412c590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc837fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcc8409d160 0x7fcc8412be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50258/0 (socket says 172.21.15.150:50258) 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.921+0000 7fcc837fe700 1 -- 172.21.15.150:0/1932725113 learned_addr learned my addr 172.21.15.150:0/1932725113 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 --2- 172.21.15.150:0/1932725113 >> v2:172.21.15.142:3300/0 conn(0x7fcc840a0110 0x7fcc840a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): get_auth_request con 0x7fcc840a0110 auth_method 0 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient(hunting): handle_auth_done global_id 5187 payload 274 2024-09-18T10:12:31.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 -- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 msgr2=0x7fcc8412c590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:31.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 --2- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8412c590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 -- 172.21.15.150:0/1932725113 >> v1:172.21.15.150:6789/0 conn(0x7fcc8409d160 legacy=0x7fcc8412be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 monclient: found mon.noname-a 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:31.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 -- 172.21.15.150:0/1932725113 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840a1dc0 con 0x7fcc840a0110 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 -- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc840a0110 msgr2=0x7fcc840a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc83fff700 1 --2- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc840a0110 0x7fcc840a04e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc740098e0 tx=0x7fcc740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bf20b8250103cc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 1 -- 172.21.15.150:0/1932725113 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7400ba70 con 0x7fcc840a0110 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: dump: 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.938 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc892a5700 5 monclient: authenticate success, global_id 5187 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc892a5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 1 -- 172.21.15.150:0/1932725113 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7400bbf0 con 0x7fcc840a0110 2024-09-18T10:12:31.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 1 -- 172.21.15.150:0/1932725113 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74011420 con 0x7fcc840a0110 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc892a5700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc892a5700 4 set_mon_vals no callback set 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.925+0000 7fcc82ffd700 10 monclient: dump: 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.940 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: shutdown 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc840a0110 msgr2=0x7fcc840a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc840a0110 0x7fcc840a04e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcc740098e0 tx=0x7fcc740091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- 172.21.15.150:0/1932725113 shutdown_connections 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8412c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- 172.21.15.150:0/1932725113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc840a0110 0x7fcc840a04e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- 172.21.15.150:0/1932725113 >> 172.21.15.150:0/1932725113 conn(0x7fcc840a6170 msgr2=0x7fcc840a2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- 172.21.15.150:0/1932725113 shutdown_connections 2024-09-18T10:12:31.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- 172.21.15.150:0/1932725113 wait complete. 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: build_initial_monmap 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: monmap: 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 Processor -- start 2024-09-18T10:12:31.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- start start 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: init 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc8409d160 0x7fcc84141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-a con 0x7fcc8409d160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-b con 0x7fcc840a0110 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8411e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): picked mon.noname-c con 0x7fcc840a0a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8409cf30 con 0x7fcc8409d160 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84135390 con 0x7fcc840a0110 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc84121660 con 0x7fcc840a0a20 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.929+0000 7fcc892a5700 10 monclient(hunting): authenticate will time out at 1034.027993s 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc88aa4700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 msgr2=0x7fcc8411e040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc88aa4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8411e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55842/0 (socket says 172.21.15.150:55842) 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 -- 172.21.15.150:0/834763769 learned_addr learned my addr 172.21.15.150:0/834763769 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): get_auth_request con 0x7fcc840a0110 auth_method 0 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc8409d160 0x7fcc84141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): get_auth_request con 0x7fcc8409d160 auth_method 0 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient(hunting): handle_auth_done global_id 14932 payload 824 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 -- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 msgr2=0x7fcc8411e040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:31.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8411e040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 -- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc8409d160 msgr2=0x7fcc84141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc8409d160 0x7fcc84141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc74009420 con 0x7fcc840a0110 2024-09-18T10:12:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 -- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 msgr2=0x7fcc84134540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc837fe700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c610 tx=0x7fcc78004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78005d40 con 0x7fcc840a0110 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: dump: 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.949 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: _finish_auth 0 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 5 monclient: authenticate success, global_id 14932 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 10 monclient: _renew_subs 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc84141e40 con 0x7fcc840a0110 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc780070d0 con 0x7fcc840a0110 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc827fc700 4 set_mon_vals no callback set 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 10 monclient: _renew_subs 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc84142690 con 0x7fcc840a0110 2024-09-18T10:12:31.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78007450 con 0x7fcc840a0110 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.933+0000 7fcc6bfff700 10 monclient: dump: 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:31.951 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcc78012400 con 0x7fcc840a0110 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc6bfff700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 0x7fcc70037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(78..78 src has 40..78) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fcc78047e60 con 0x7fcc840a0110 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc892a5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc892a5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc892a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8404bbf0 con 0x7fcc840a0110 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc83fff700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 0x7fcc70037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.937+0000 7fcc83fff700 10 monclient: get_auth_request con 0x7fcc70035a70 auth_method 0 2024-09-18T10:12:31.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.941+0000 7fcc83fff700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 0x7fcc70037f20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcc740098e0 tx=0x7fcc74016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:31.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.945+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcc78072670 con 0x7fcc840a0110 2024-09-18T10:12:31.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.945+0000 7fcc6bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:31.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:31.945+0000 7fcc6bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:32.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.117+0000 7fcc892a5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.117+0000 7fcc892a5700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.117+0000 7fcc892a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.117+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"} v 0) v1 -- 0x7fcc84063ed0 con 0x7fcc840a0110 2024-09-18T10:12:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.325+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}]=0 removed item id 8 name 'osd.8' from crush map v79) v1 ==== 143+0+0 (secure 0 0 0) 0x7fcc78017070 con 0x7fcc840a0110 2024-09-18T10:12:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.325+0000 7fcc6bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.325+0000 7fcc6bfff700 10 monclient: _finish_command 2 = system:0 removed item id 8 name 'osd.8' from crush map 2024-09-18T10:12:32.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.433+0000 7fcc892a5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.433+0000 7fcc892a5700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.433+0000 7fcc892a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.433+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"} v 0) v1 -- 0x7fcc840008d0 con 0x7fcc840a0110 2024-09-18T10:12:32.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc6bfff700 1 -- 172.21.15.150:0/834763769 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}]=0 device 'osd.8' does not appear in the crush map v79) v1 ==== 145+0+0 (secure 0 0 0) 0x7fcc7801b3b0 con 0x7fcc840a0110 2024-09-18T10:12:32.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc6bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.8", "ancestor": "host1"}] 2024-09-18T10:12:32.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc6bfff700 10 monclient: _finish_command 3 = system:0 device 'osd.8' does not appear in the crush map 2024-09-18T10:12:32.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 msgr2=0x7fcc70037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:32.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 0x7fcc70037f20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcc740098e0 tx=0x7fcc74016000 comp rx=0 tx=0).stop 2024-09-18T10:12:32.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 10 monclient: shutdown 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 msgr2=0x7fcc84134540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcc7800c610 tx=0x7fcc78004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 shutdown_connections 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc70035a70 0x7fcc70037f20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc840a0a20 0x7fcc8411e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc840a0110 0x7fcc84134540 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 --2- 172.21.15.150:0/834763769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc8409d160 0x7fcc84141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 >> 172.21.15.150:0/834763769 conn(0x7fcc840a6170 msgr2=0x7fcc840a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:32.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 shutdown_connections 2024-09-18T10:12:32.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.437+0000 7fcc892a5700 1 -- 172.21.15.150:0/834763769 wait complete. 2024-09-18T10:12:32.445 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.8' does not appear in the crush map 2024-09-18T10:12:32.457 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm host1 2024-09-18T10:12:32.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: get_monmap_and_config 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: build_initial_monmap 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: monmap: 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:32.553614+0000 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:32.553614+0000 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:32.554 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:32.555 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 Processor -- start 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 -- start start 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: init 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f806c09e6d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f806c0a09a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f806c0a0da0 0x7f806c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-a con 0x7f806c0a0da0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806c09e220 con 0x7f806c09e6d0 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806c09eaa0 con 0x7f806c0a09a0 2024-09-18T10:12:32.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c09ee30 con 0x7f806c0a0da0 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.549+0000 7f8073cc0700 10 monclient(hunting): authenticate will time out at 1034.647757s 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807125b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f806c0a09a0 0x7f806c137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50274/0 (socket says 172.21.15.150:50274) 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807125b700 1 -- 172.21.15.150:0/1949034167 learned_addr learned my addr 172.21.15.150:0/1949034167 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 --2- 172.21.15.150:0/1949034167 >> v2:172.21.15.142:3300/0 conn(0x7f806c0a0da0 0x7f806c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): get_auth_request con 0x7f806c0a0da0 auth_method 0 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:32.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient(hunting): handle_auth_done global_id 5193 payload 274 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 -- 172.21.15.150:0/1949034167 >> v1:172.21.15.150:6789/0 conn(0x7f806c0a09a0 legacy=0x7f806c137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 -- 172.21.15.150:0/1949034167 >> v1:172.21.15.142:6789/0 conn(0x7f806c09e6d0 legacy=0x7f806c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 monclient: found mon.noname-a 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:32.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 -- 172.21.15.150:0/1949034167 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c09d930 con 0x7f806c0a0da0 2024-09-18T10:12:32.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 -- 172.21.15.150:0/1949034167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 msgr2=0x7f806c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:32.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f807225d700 1 --2- 172.21.15.150:0/1949034167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c13b150 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e60 tx=0x7f8064009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2a55e82434433b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 1 -- 172.21.15.150:0/1949034167 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806400e070 con 0x7f806c0a0da0 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: dump: 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:32.561 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: _finish_auth 0 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8073cc0700 5 monclient: authenticate success, global_id 5193 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 1 -- 172.21.15.150:0/1949034167 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8064012430 con 0x7f806c0a0da0 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8073cc0700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8073cc0700 4 set_mon_vals no callback set 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 1 -- 172.21.15.150:0/1949034167 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640127d0 con 0x7f806c0a0da0 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.553+0000 7f8070a5a700 10 monclient: dump: 2024-09-18T10:12:32.562 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: shutdown 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- 172.21.15.150:0/1949034167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 msgr2=0x7f806c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:32.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 --2- 172.21.15.150:0/1949034167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c13b150 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e60 tx=0x7f8064009780 comp rx=0 tx=0).stop 2024-09-18T10:12:32.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- 172.21.15.150:0/1949034167 shutdown_connections 2024-09-18T10:12:32.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 --2- 172.21.15.150:0/1949034167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c13b150 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- 172.21.15.150:0/1949034167 >> 172.21.15.150:0/1949034167 conn(0x7f806c0a58a0 msgr2=0x7f806c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:32.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- 172.21.15.150:0/1949034167 shutdown_connections 2024-09-18T10:12:32.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- 172.21.15.150:0/1949034167 wait complete. 2024-09-18T10:12:32.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: build_initial_monmap 2024-09-18T10:12:32.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: monmap: 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 Processor -- start 2024-09-18T10:12:32.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- start start 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: init 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-b con 0x7f806c09e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 0x7f806c08e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-c con 0x7f806c0a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c08e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): picked mon.noname-a con 0x7f806c0a0da0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c08fd20 con 0x7f806c0a0da0 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:32.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c091260 con 0x7f806c09e6d0 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c09bf00 con 0x7f806c0a09a0 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8073cc0700 10 monclient(hunting): authenticate will time out at 1034.654707s 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807125b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 msgr2=0x7f806c08e3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807125b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 0x7f806c08e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55844/0 (socket says 172.21.15.150:55844) 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 -- 172.21.15.150:0/2825609916 learned_addr learned my addr 172.21.15.150:0/2825609916 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): get_auth_request con 0x7f806c09e6d0 auth_method 0 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:32.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c08e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): get_auth_request con 0x7f806c0a0da0 auth_method 0 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f807225d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:32.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient(hunting): handle_auth_done global_id 14935 payload 824 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 -- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 msgr2=0x7f806c08e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 0x7f806c08e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 -- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 msgr2=0x7f806c08e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c08e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 monclient: found mon.noname-b 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.557+0000 7f8071a5c700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064009a20 con 0x7f806c09e6d0 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8071a5c700 1 -- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 msgr2=0x7f806c08de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8071a5c700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f806800c600 tx=0x7f8068007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80680117a0 con 0x7f806c09e6d0 2024-09-18T10:12:32.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: dump: 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 5 monclient: authenticate success, global_id 14935 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _renew_subs 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c091540 con 0x7f806c09e6d0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _renew_subs 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c099410 con 0x7f806c09e6d0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8068011900 con 0x7f806c09e6d0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8068011cc0 con 0x7f806c09e6d0 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:32.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 10 monclient: dump: 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f80637fe700 4 set_mon_vals no callback set 2024-09-18T10:12:32.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:32.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:32.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c091920 con 0x7f806c09e6d0 2024-09-18T10:12:32.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.561+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8068012630 con 0x7f806c09e6d0 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.565+0000 7f8061ffb700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 0x7f8050037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.565+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(79..79 src has 40..79) v4 ==== 6268+0+0 (secure 0 0 0) 0x7f80680486b0 con 0x7f806c09e6d0 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.565+0000 7f807125b700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 0x7f8050037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.565+0000 7f807125b700 10 monclient: get_auth_request con 0x7f8050035a70 auth_method 0 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.565+0000 7f807125b700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 0x7f8050037f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f806c08f430 tx=0x7f805c007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:32.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.573+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8068010ac0 con 0x7f806c09e6d0 2024-09-18T10:12:32.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.573+0000 7f8061ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:32.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.573+0000 7f8061ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:32.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.745+0000 7f8073cc0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:32.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.745+0000 7f8073cc0700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:32.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.745+0000 7f8073cc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:32.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:32.745+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f806c09dbf0 con 0x7f806c09e6d0 2024-09-18T10:12:33.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.333+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 removed item id -11 name 'host1' from crush map v80) v1 ==== 124+0+0 (secure 0 0 0) 0x7f80680102e0 con 0x7f806c09e6d0 2024-09-18T10:12:33.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.333+0000 7f8061ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:33.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.333+0000 7f8061ffb700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host1' from crush map 2024-09-18T10:12:33.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8073cc0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:33.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8073cc0700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:33.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8073cc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host1"} v 0) v1 -- 0x7f806c063ed0 con 0x7f806c09e6d0 2024-09-18T10:12:33.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8061ffb700 1 -- 172.21.15.150:0/2825609916 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host1"}]=0 device 'host1' does not appear in the crush map v80) v1 ==== 124+0+0 (secure 0 0 0) 0x7f80680104c0 con 0x7f806c09e6d0 2024-09-18T10:12:33.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8061ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host1"}] 2024-09-18T10:12:33.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.445+0000 7f8061ffb700 10 monclient: _finish_command 3 = system:0 device 'host1' does not appear in the crush map 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 msgr2=0x7f8050037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 0x7f8050037f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f806c08f430 tx=0x7f805c007c10 comp rx=0 tx=0).stop 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 10 monclient: shutdown 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 msgr2=0x7f806c08de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f806800c600 tx=0x7f8068007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 shutdown_connections 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8050035a70 0x7f8050037f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f806c0a0da0 0x7f806c08e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f806c0a09a0 0x7f806c08e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 --2- 172.21.15.150:0/2825609916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f806c09e6d0 0x7f806c08de80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 >> 172.21.15.150:0/2825609916 conn(0x7f806c0a58a0 msgr2=0x7f806c09a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:33.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 shutdown_connections 2024-09-18T10:12:33.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.449+0000 7f8073cc0700 1 -- 172.21.15.150:0/2825609916 wait complete. 2024-09-18T10:12:33.454 INFO:tasks.workunit.client.0.smithi150.stderr:device 'host1' does not appear in the crush map 2024-09-18T10:12:33.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:33.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c host1 2024-09-18T10:12:33.464 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 0 2024-09-18T10:12:33.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: get_monmap_and_config 2024-09-18T10:12:33.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: build_initial_monmap 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: monmap: 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:33.558574+0000 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:33.558574+0000 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:33.560 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:33.560 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:33.560 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 Processor -- start 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 -- start start 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: init 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac0956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-b con 0x7f02ac09c2e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:33.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02ac098a50 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:33.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f02ac099530 0x7f02ac136710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-a con 0x7f02ac099530 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:33.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0984f0 con 0x7f02ac098a50 2024-09-18T10:12:33.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac09af60 con 0x7f02ac099530 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac09b520 con 0x7f02ac09c2e0 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.553+0000 7f02b3e70700 10 monclient(hunting): authenticate will time out at 1035.651884s 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b1c0c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 msgr2=0x7f02ac0956d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b1c0c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac0956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b140b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f02ac098a50 0x7f02ac098e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50280/0 (socket says 172.21.15.150:50280) 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b140b700 1 -- 172.21.15.150:0/1291968525 learned_addr learned my addr 172.21.15.150:0/1291968525 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 --2- 172.21.15.150:0/1291968525 >> v2:172.21.15.142:3300/0 conn(0x7f02ac099530 0x7f02ac136710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): get_auth_request con 0x7f02ac099530 auth_method 0 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient(hunting): handle_auth_done global_id 5199 payload 274 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 -- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 msgr2=0x7f02ac0956d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 --2- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac0956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 -- 172.21.15.150:0/1291968525 >> v1:172.21.15.150:6789/0 conn(0x7f02ac098a50 legacy=0x7f02ac098e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 monclient: found mon.noname-a 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:33.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 -- 172.21.15.150:0/1291968525 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac09c6b0 con 0x7f02ac099530 2024-09-18T10:12:33.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 -- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac099530 msgr2=0x7f02ac136710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:33.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b240d700 1 --2- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac099530 0x7f02ac136710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009e60 tx=0x7f02a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b5e76774019de5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 1 -- 172.21.15.150:0/1291968525 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a400e070 con 0x7f02ac099530 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: dump: 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.572 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: _finish_auth 0 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 1 -- 172.21.15.150:0/1291968525 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4012430 con 0x7f02ac099530 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 1 -- 172.21.15.150:0/1291968525 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a40127d0 con 0x7f02ac099530 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b3e70700 5 monclient: authenticate success, global_id 5199 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b3e70700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b3e70700 4 set_mon_vals no callback set 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.557+0000 7f02b0c0a700 10 monclient: dump: 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: shutdown 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:33.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac099530 msgr2=0x7f02ac136710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac099530 0x7f02ac136710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009e60 tx=0x7f02a4009780 comp rx=0 tx=0).stop 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- 172.21.15.150:0/1291968525 shutdown_connections 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac099530 0x7f02ac136710 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- 172.21.15.150:0/1291968525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac0956d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- 172.21.15.150:0/1291968525 >> 172.21.15.150:0/1291968525 conn(0x7f02ac0a1710 msgr2=0x7f02ac0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- 172.21.15.150:0/1291968525 shutdown_connections 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- 172.21.15.150:0/1291968525 wait complete. 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: build_initial_monmap 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: monmap: 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:33.575 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 Processor -- start 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- start start 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: init 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 0x7f02ac141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-a con 0x7f02ac098a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-b con 0x7f02ac099530 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac08da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): picked mon.noname-c con 0x7f02ac09c2e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac091400 con 0x7f02ac098a50 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0951f0 con 0x7f02ac099530 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac097430 con 0x7f02ac09c2e0 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b3e70700 10 monclient(hunting): authenticate will time out at 1035.658519s 2024-09-18T10:12:33.577 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b240d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 msgr2=0x7f02ac08da70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b240d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac08da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55858/0 (socket says 172.21.15.150:55858) 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 -- 172.21.15.150:0/42933216 learned_addr learned my addr 172.21.15.150:0/42933216 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): get_auth_request con 0x7f02ac099530 auth_method 0 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 0x7f02ac141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): get_auth_request con 0x7f02ac098a50 auth_method 0 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient(hunting): handle_auth_done global_id 14938 payload 824 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 -- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 msgr2=0x7f02ac08da70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac08da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 -- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 msgr2=0x7f02ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 0x7f02ac141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:33.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 monclient: found mon.noname-b 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b140b700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4009a20 con 0x7f02ac099530 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.561+0000 7f02b1c0c700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 0x7f02ac141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b140b700 1 -- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 msgr2=0x7f02ac141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b140b700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c5e0 tx=0x7f02a8004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8005d40 con 0x7f02ac099530 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: dump: 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.580 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 5 monclient: authenticate success, global_id 14938 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80070d0 con 0x7f02ac099530 2024-09-18T10:12:33.581 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _renew_subs 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8007490 con 0x7f02ac099530 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac090220 con 0x7f02ac099530 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a37fe700 4 set_mon_vals no callback set 2024-09-18T10:12:33.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02a1ffb700 10 monclient: dump: 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.583 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _renew_subs 2024-09-18T10:12:33.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac1424a0 con 0x7f02ac099530 2024-09-18T10:12:33.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.565+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac063ed0 con 0x7f02ac099530 2024-09-18T10:12:33.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.569+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f02a801b3b0 con 0x7f02ac099530 2024-09-18T10:12:33.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.569+0000 7f02a1ffb700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 0x7f0290037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.569+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(80..80 src has 40..80) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f02a80482b0 con 0x7f02ac099530 2024-09-18T10:12:33.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.569+0000 7f02b1c0c700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 0x7f0290037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.569+0000 7f02b1c0c700 10 monclient: get_auth_request con 0x7f0290035a70 auth_method 0 2024-09-18T10:12:33.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.573+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:12:33.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.573+0000 7f02b1c0c700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 0x7f0290037f20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f029c003fd0 tx=0x7f029c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.573+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f02a800f810 con 0x7f02ac099530 2024-09-18T10:12:33.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.577+0000 7f02a1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.577+0000 7f02a1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:33.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02b3e70700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:33.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02b3e70700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:33.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02b3e70700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f02ac099cc0 con 0x7f02ac099530 2024-09-18T10:12:33.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f02a8017070 con 0x7f02ac099530 2024-09-18T10:12:33.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02a1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:33.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.737+0000 7f02a1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:33.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02b3e70700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:33.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02b3e70700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:33.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02b3e70700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f02ac0008d0 con 0x7f02ac099530 2024-09-18T10:12:33.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02a1ffb700 1 -- 172.21.15.150:0/42933216 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v80) v1 ==== 56+0+1340 (secure 0 0 0) 0x7f02a800f260 con 0x7f02ac099530 2024-09-18T10:12:33.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02a1ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:33.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.833+0000 7f02a1ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 msgr2=0x7f0290037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 0x7f0290037f20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f029c003fd0 tx=0x7f029c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 10 monclient: shutdown 2024-09-18T10:12:33.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 msgr2=0x7f02ac141f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f02a800c5e0 tx=0x7f02a8004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 shutdown_connections 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0290035a70 0x7f0290037f20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02ac09c2e0 0x7f02ac08da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02ac099530 0x7f02ac141f60 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 --2- 172.21.15.150:0/42933216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02ac098a50 0x7f02ac141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 >> 172.21.15.150:0/42933216 conn(0x7f02ac0a1710 msgr2=0x7f02ac096a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 shutdown_connections 2024-09-18T10:12:33.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.837+0000 7f02b3e70700 1 -- 172.21.15.150:0/42933216 wait complete. 2024-09-18T10:12:33.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd tree-from host1 2024-09-18T10:12:33.852 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:33.853 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree-from host1 2024-09-18T10:12:33.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 10 monclient: get_monmap_and_config 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 10 monclient: build_initial_monmap 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 10 monclient: monmap: 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:33.950682+0000 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:33.950682+0000 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:33.951 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:33.952 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:33.952 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 1 Processor -- start 2024-09-18T10:12:33.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 1 -- start start 2024-09-18T10:12:33.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.945+0000 7f3d766d3700 10 monclient: init 2024-09-18T10:12:33.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d7009ac60 0x7f3d7009c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-a con 0x7f3d7009ac60 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 0x7f3d7009c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-b con 0x7f3d7006fa70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d7009ccb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d70095ee0 con 0x7f3d7009ccb0 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7009d0d0 con 0x7f3d7009ac60 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d70095c00 con 0x7f3d7006fa70 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:33.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d766d3700 10 monclient(hunting): authenticate will time out at 1036.044714s 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d67fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 msgr2=0x7f3d7009c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d67fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 0x7f3d7009c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d7009ac60 0x7f3d7009c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d74c70700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3d7009ccb0 0x7f3d70136710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35136/0 (socket says 172.21.15.150:35136) 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d74c70700 1 -- 172.21.15.150:0/903247468 learned_addr learned my addr 172.21.15.150:0/903247468 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d7009ac60 0x7f3d7009c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47306/0 (socket says 172.21.15.150:47306) 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): get_auth_request con 0x7f3d7009ac60 auth_method 0 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_done global_id 5205 payload 274 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 -- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 msgr2=0x7f3d7009c770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 --2- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 0x7f3d7009c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 -- 172.21.15.150:0/903247468 >> v1:172.21.15.142:6789/0 conn(0x7f3d7009ccb0 legacy=0x7f3d70136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 monclient: found mon.noname-a 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:33.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.949+0000 7f3d6ffff700 1 -- 172.21.15.150:0/903247468 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7009b030 con 0x7f3d7009ac60 2024-09-18T10:12:33.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6ffff700 1 -- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 msgr2=0x7f3d7009c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6ffff700 1 --2- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7009c230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3d600098e0 tx=0x7f3d600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e92b29c87ce69cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 1 -- 172.21.15.150:0/903247468 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000ba70 con 0x7f3d7009ac60 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: dump: 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.960 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 5 monclient: authenticate success, global_id 5205 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 1 -- 172.21.15.150:0/903247468 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000bbf0 con 0x7f3d7009ac60 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:33.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 4 set_mon_vals no callback set 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 1 -- 172.21.15.150:0/903247468 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60011420 con 0x7f3d7009ac60 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d6f7fe700 10 monclient: dump: 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.962 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 10 monclient: shutdown 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 msgr2=0x7f3d7009c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 --2- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7009c230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3d600098e0 tx=0x7f3d600091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- 172.21.15.150:0/903247468 shutdown_connections 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 --2- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7006fa70 0x7f3d7009c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 --2- 172.21.15.150:0/903247468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7009c230 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- 172.21.15.150:0/903247468 >> 172.21.15.150:0/903247468 conn(0x7f3d700a1710 msgr2=0x7f3d700a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- 172.21.15.150:0/903247468 shutdown_connections 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- 172.21.15.150:0/903247468 wait complete. 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 10 monclient: build_initial_monmap 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 10 monclient: monmap: 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:33.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 Processor -- start 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 1 -- start start 2024-09-18T10:12:33.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.953+0000 7f3d766d3700 10 monclient: init 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d7006fa70 0x7f3d7013fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-b con 0x7f3d7006fa70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7013ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-a con 0x7f3d7009ac60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7009ccb0 0x7f3d7008bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): picked mon.noname-c con 0x7f3d7009ccb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:33.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d70090420 con 0x7f3d7009ac60 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7008dbe0 con 0x7f3d7006fa70 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7008e2a0 con 0x7f3d7009ccb0 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d766d3700 10 monclient(hunting): authenticate will time out at 1036.052615s 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d74c70700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7009ccb0 msgr2=0x7f3d7008bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d7006fa70 0x7f3d7013fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d74c70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7009ccb0 0x7f3d7008bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d7006fa70 0x7f3d7013fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55860/0 (socket says 172.21.15.150:55860) 2024-09-18T10:12:33.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 -- 172.21.15.150:0/2844090071 learned_addr learned my addr 172.21.15.150:0/2844090071 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): get_auth_request con 0x7f3d7006fa70 auth_method 0 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7013ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): get_auth_request con 0x7f3d7009ac60 auth_method 0 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient(hunting): handle_auth_done global_id 14941 payload 824 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 -- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7009ccb0 msgr2=0x7f3d7008bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d7009ccb0 0x7f3d7008bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 -- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 msgr2=0x7f3d7013ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7013ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 monclient: found mon.noname-b 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d6ffff700 1 -- 172.21.15.150:0/2844090071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d60009420 con 0x7f3d7006fa70 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.957+0000 7f3d67fff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d7009ac60 0x7f3d7013ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6ffff700 1 -- 172.21.15.150:0/2844090071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d7006fa70 msgr2=0x7f3d7013fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6ffff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d7006fa70 0x7f3d7013fa00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3d70069a80 tx=0x7f3d60016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60011e30 con 0x7f3d7006fa70 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: dump: 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.974 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.975 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.975 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.975 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.975 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 5 monclient: authenticate success, global_id 14941 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d60020070 con 0x7f3d7006fa70 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _renew_subs 2024-09-18T10:12:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 1 -- 172.21.15.150:0/2844090071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7008f190 con 0x7f3d7006fa70 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6001b680 con 0x7f3d7006fa70 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 10 monclient: dump: 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _renew_subs 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6effd700 4 set_mon_vals no callback set 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 1 -- 172.21.15.150:0/2844090071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d70095080 con 0x7f3d7006fa70 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:33.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d766d3700 1 -- 172.21.15.150:0/2844090071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d70063ed0 con 0x7f3d7006fa70 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3d60022030 con 0x7f3d7006fa70 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d54035a70 0x7f3d54037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(80..80 src has 40..80) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f3d6004a7b0 con 0x7f3d7006fa70 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d67fff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d54035a70 0x7f3d54037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.961+0000 7f3d67fff700 10 monclient: get_auth_request con 0x7f3d54035a70 auth_method 0 2024-09-18T10:12:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.965+0000 7f3d67fff700 1 --2- 172.21.15.150:0/2844090071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d54035a70 0x7f3d54037f20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3d58003fd0 tx=0x7f3d580091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.973+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3d6002cb30 con 0x7f3d7006fa70 2024-09-18T10:12:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.973+0000 7f3d6cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:33.973+0000 7f3d6cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:34.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.137+0000 7f3d766d3700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T10:12:34.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.137+0000 7f3d766d3700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T10:12:34.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.137+0000 7f3d766d3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.137+0000 7f3d766d3700 1 -- 172.21.15.150:0/2844090071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host1"} v 0) v1 -- 0x7f3d70063ed0 con 0x7f3d7006fa70 2024-09-18T10:12:34.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.141+0000 7f3d6cff9700 1 -- 172.21.15.150:0/2844090071 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host1"}]=-2 bucket 'host1' does not exist v80) v1 ==== 109+0+0 (secure 0 0 0) 0x7f3d6002cb30 con 0x7f3d7006fa70 2024-09-18T10:12:34.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.141+0000 7f3d6cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host1"}] 2024-09-18T10:12:34.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.141+0000 7f3d6cff9700 10 monclient: _finish_command 2 = mon:2 bucket 'host1' does not exist 2024-09-18T10:12:34.147 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOENT: bucket 'host1' does not exist 2024-09-18T10:12:34.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:34.150 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree-from host2 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.237+0000 7f1541980700 10 monclient: get_monmap_and_config 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.237+0000 7f1541980700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.237+0000 7f1541980700 10 monclient: monmap: 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.243 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:34.243297+0000 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:34.243297+0000 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:34.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 Processor -- start 2024-09-18T10:12:34.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 -- start start 2024-09-18T10:12:34.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient: init 2024-09-18T10:12:34.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:12:34.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f153c0a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): picked mon.noname-b con 0x7f153c09cf10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f153c0a0750 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:34.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f153c0a0b20 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c09e6e0 con 0x7f153c0a0750 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c09d2e0 con 0x7f153c0a0b20 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c09e2e0 con 0x7f153c09cf10 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f153affd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 msgr2=0x7f153c0a0210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f1541980700 10 monclient(hunting): authenticate will time out at 1036.338163s 2024-09-18T10:12:34.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f153affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f153c0a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f153b7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f153c0a0b20 0x7f153c13a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50284/0 (socket says 172.21.15.150:50284) 2024-09-18T10:12:34.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.241+0000 7f153b7fe700 1 -- 172.21.15.150:0/1000027886 learned_addr learned my addr 172.21.15.150:0/1000027886 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153b7fe700 1 -- 172.21.15.150:0/1000027886 >> v1:172.21.15.150:6789/0 conn(0x7f153c0a0b20 legacy=0x7f153c13a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 1 -- 172.21.15.150:0/1000027886 >> v1:172.21.15.142:6789/0 conn(0x7f153c0a0750 legacy=0x7f153c136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475786991 0 0) 0x7f153c09d2e0 con 0x7f153c0a0b20 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): my global_id is 14944 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1524003570 con 0x7f153c0a0b20 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120839075 0 0) 0x7f153c09e6e0 con 0x7f153c0a0750 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): my global_id is 5211 2024-09-18T10:12:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c09d2e0 con 0x7f153c0a0750 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919414780 0 0) 0x7f1524003570 con 0x7f153c0a0b20 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 msgr2=0x7f153c0a0210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f153c0a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 >> v1:172.21.15.142:6789/0 conn(0x7f153c0a0750 legacy=0x7f153c136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0a1f90 con 0x7f153c0a0b20 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1534001be0 con 0x7f153c0a0b20 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: dump: 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.261 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 >> v1:172.21.15.150:6789/0 conn(0x7f153c0a0b20 legacy=0x7f153c13a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 0x7f1524004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): picked mon.b con 0x7f1524003c40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1524004810 0x7f1524004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): picked mon.a con 0x7f1524004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f152400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): picked mon.c con 0x7f153c09cf10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c09e2e0 con 0x7f1524004810 2024-09-18T10:12:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524005620 con 0x7f1524003c40 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152400bfa0 con 0x7f153c09cf10 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1539ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153b7fe700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 msgr2=0x7f152400ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153b7fe700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f152400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 0x7f1524004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1524004810 0x7f1524004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): get_auth_request con 0x7f1524003c40 auth_method 0 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): get_auth_request con 0x7f1524004810 auth_method 0 2024-09-18T10:12:34.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient(hunting): handle_auth_done global_id 14944 payload 293 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 msgr2=0x7f152400ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c09cf10 0x7f152400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1524004810 msgr2=0x7f1524004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1524004810 0x7f1524004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 monclient: found mon.b 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 1 -- 172.21.15.150:0/1000027886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524003570 con 0x7f1524003c40 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1541980700 5 monclient: authenticate success, global_id 14944 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f1541980700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.245+0000 7f153a7fc700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1524004810 0x7f1524004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:34.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f153affd700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 msgr2=0x7f1524004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f153affd700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 0x7f1524004030 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f152c009a80 tx=0x7f152c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a469dc63b2898ac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c009340 con 0x7f1524003c40 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: dump: 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.267 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c003c10 con 0x7f1524003c40 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 1 -- 172.21.15.150:0/1000027886 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c0050f0 con 0x7f1524003c40 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1539ffb700 10 monclient: dump: 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.268 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 4 set_mon_vals no callback set 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 10 monclient: shutdown 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 msgr2=0x7f1524004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 --2- 172.21.15.150:0/1000027886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1524003c40 0x7f1524004030 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f152c009a80 tx=0x7f152c0090f0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f153b7fe700 1 -- 172.21.15.150:0/1000027886 reap_dead start 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- 172.21.15.150:0/1000027886 shutdown_connections 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- 172.21.15.150:0/1000027886 >> 172.21.15.150:0/1000027886 conn(0x7f153c0a5820 msgr2=0x7f153c0a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- 172.21.15.150:0/1000027886 shutdown_connections 2024-09-18T10:12:34.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- 172.21.15.150:0/1000027886 wait complete. 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 10 monclient: monmap: 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:34.271 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 Processor -- start 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 1 -- start start 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.249+0000 7f1541980700 10 monclient: init 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): picked mon.noname-b con 0x7f153c141c90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f153c092970 0x7f153c08e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): picked mon.noname-a con 0x7f153c092970 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 0x7f153c08e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): picked mon.noname-c con 0x7f153c1425a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0a52a0 con 0x7f153c092970 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0994c0 con 0x7f153c141c90 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0917c0 con 0x7f153c1425a0 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f1541980700 10 monclient(hunting): authenticate will time out at 1036.348907s 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 msgr2=0x7f153c08e6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55884/0 (socket says 172.21.15.150:55884) 2024-09-18T10:12:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153b7fe700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 0x7f153c08e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 -- 172.21.15.150:0/487524645 learned_addr learned my addr 172.21.15.150:0/487524645 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): get_auth_request con 0x7f153c141c90 auth_method 0 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f153c092970 0x7f153c08e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): get_auth_request con 0x7f153c092970 auth_method 0 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient(hunting): handle_auth_done global_id 14947 payload 824 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 -- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 msgr2=0x7f153c08e6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 0x7f153c08e6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 -- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f153c092970 msgr2=0x7f153c08e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f153c092970 0x7f153c08e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 monclient: found mon.noname-b 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.253+0000 7f153affd700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c091aa0 con 0x7f153c141c90 2024-09-18T10:12:34.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f153affd700 1 -- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 msgr2=0x7f153c142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f153affd700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f152c00b3c0 tx=0x7f152c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c005070 con 0x7f153c141c90 2024-09-18T10:12:34.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: dump: 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 5 monclient: authenticate success, global_id 14947 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c00b7a0 con 0x7f153c141c90 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _renew_subs 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c08fd00 con 0x7f153c141c90 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c00bb60 con 0x7f153c141c90 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:34.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 10 monclient: dump: 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _renew_subs 2024-09-18T10:12:34.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c08ebf0 con 0x7f153c141c90 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f15397fa700 4 set_mon_vals no callback set 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c063ed0 con 0x7f153c141c90 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f152c023070 con 0x7f153c141c90 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 0x7f1528037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(81..81 src has 40..81) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f152c04a770 con 0x7f153c141c90 2024-09-18T10:12:34.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f153a7fc700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 0x7f1528037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.257+0000 7f153a7fc700 10 monclient: get_auth_request con 0x7f1528035a70 auth_method 0 2024-09-18T10:12:34.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.261+0000 7f153a7fc700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 0x7f1528037f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1530001800 tx=0x7f1530007cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.269+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f152c02fbd0 con 0x7f153c141c90 2024-09-18T10:12:34.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.269+0000 7f1522ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.269+0000 7f1522ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:34.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.433+0000 7f1541980700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.433+0000 7f1541980700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.433+0000 7f1541980700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.433+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f153c091c30 con 0x7f153c141c90 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.437+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v81) v1 ==== 80+0+177 (secure 0 0 0) 0x7f152c02f3f0 con 0x7f153c141c90 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.437+0000 7f1522ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.437+0000 7f1522ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:34.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1541980700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1541980700 10 monclient: _send_command 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1541980700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "host2"} v 0) v1 -- 0x7f153c0008d0 con 0x7f153c141c90 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1522ffd700 1 -- 172.21.15.150:0/487524645 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "host2"}]=0 v81) v1 ==== 80+0+177 (secure 0 0 0) 0x7f152c02f5d0 con 0x7f153c141c90 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1522ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree-from", "bucket": "host2"}] 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.529+0000 7f1522ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:34.534 INFO:tasks.workunit.client.0.smithi150.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-18T10:12:34.535 INFO:tasks.workunit.client.0.smithi150.stdout:-16 1.00000 host host2 2024-09-18T10:12:34.535 INFO:tasks.workunit.client.0.smithi150.stdout: 10 1.00000 osd.10 down 1.00000 1.00000 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 msgr2=0x7f1528037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 0x7f1528037f20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1530001800 tx=0x7f1530007cf0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 10 monclient: shutdown 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 msgr2=0x7f153c142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f152c00b3c0 tx=0x7f152c015000 comp rx=0 tx=0).stop 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 shutdown_connections 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1528035a70 0x7f1528037f20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f153c1425a0 0x7f153c08e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f153c092970 0x7f153c08e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 --2- 172.21.15.150:0/487524645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f153c141c90 0x7f153c142060 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 >> 172.21.15.150:0/487524645 conn(0x7f153c0a5820 msgr2=0x7f153c0a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:34.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 shutdown_connections 2024-09-18T10:12:34.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.533+0000 7f1541980700 1 -- 172.21.15.150:0/487524645 wait complete. 2024-09-18T10:12:34.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd tree-from osd.10 2024-09-18T10:12:34.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:34.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree-from osd.10 2024-09-18T10:12:34.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: get_monmap_and_config 2024-09-18T10:12:34.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: monmap: 2024-09-18T10:12:34.642 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.642 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:34.642306+0000 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:34.642306+0000 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:34.643 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 Processor -- start 2024-09-18T10:12:34.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 -- start start 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: init 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2db009d1f0 0x7f2db009e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-a con 0x7f2db009d1f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2db00a0110 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2db00a0c80 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db009ddd0 con 0x7f2db00a0110 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db009e400 con 0x7f2db00a0c80 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db00a66e0 con 0x7f2db009d1f0 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db8408700 10 monclient(hunting): authenticate will time out at 1036.735765s 2024-09-18T10:12:34.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db69a5700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2db00a0c80 0x7f2db01399b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50290/0 (socket says 172.21.15.150:50290) 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.637+0000 7f2db69a5700 1 -- 172.21.15.150:0/1506450088 learned_addr learned my addr 172.21.15.150:0/1506450088 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 --2- 172.21.15.150:0/1506450088 >> v2:172.21.15.142:3300/0 conn(0x7f2db009d1f0 0x7f2db009e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): get_auth_request con 0x7f2db009d1f0 auth_method 0 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient(hunting): handle_auth_done global_id 5217 payload 274 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 -- 172.21.15.150:0/1506450088 >> v1:172.21.15.150:6789/0 conn(0x7f2db00a0c80 legacy=0x7f2db01399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 -- 172.21.15.150:0/1506450088 >> v1:172.21.15.142:6789/0 conn(0x7f2db00a0110 legacy=0x7f2db00a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:34.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 monclient: found mon.noname-a 2024-09-18T10:12:34.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:34.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 -- 172.21.15.150:0/1506450088 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db009daf0 con 0x7f2db009d1f0 2024-09-18T10:12:34.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 -- 172.21.15.150:0/1506450088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 msgr2=0x7f2db009e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db61a4700 1 --2- 172.21.15.150:0/1506450088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db009e840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2da00098e0 tx=0x7f2da00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d36b6bdd17b986 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 1 -- 172.21.15.150:0/1506450088 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000ba70 con 0x7f2db009d1f0 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: dump: 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db8408700 5 monclient: authenticate success, global_id 5217 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db8408700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 1 -- 172.21.15.150:0/1506450088 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da000bbd0 con 0x7f2db009d1f0 2024-09-18T10:12:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db8408700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db8408700 4 set_mon_vals no callback set 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 1 -- 172.21.15.150:0/1506450088 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0011420 con 0x7f2db009d1f0 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.641+0000 7f2db51a2700 10 monclient: dump: 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.654 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: shutdown 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- 172.21.15.150:0/1506450088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 msgr2=0x7f2db009e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 --2- 172.21.15.150:0/1506450088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db009e840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2da00098e0 tx=0x7f2da00091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- 172.21.15.150:0/1506450088 shutdown_connections 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 --2- 172.21.15.150:0/1506450088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db009e840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- 172.21.15.150:0/1506450088 >> 172.21.15.150:0/1506450088 conn(0x7f2db00683a0 msgr2=0x7f2db00a5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- 172.21.15.150:0/1506450088 shutdown_connections 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- 172.21.15.150:0/1506450088 wait complete. 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: monmap: 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 Processor -- start 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- start start 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: init 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db0141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-a con 0x7f2db009d1f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2db00a0110 0x7f2db0141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-c con 0x7f2db00a0110 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00a0c80 0x7f2db01424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): picked mon.noname-b con 0x7f2db00a0c80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00994e0 con 0x7f2db009d1f0 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00a5010 con 0x7f2db00a0c80 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0091350 con 0x7f2db00a0110 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db8408700 10 monclient(hunting): authenticate will time out at 1036.742626s 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00a0c80 0x7f2db01424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00a0c80 0x7f2db01424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55898/0 (socket says 172.21.15.150:55898) 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 -- 172.21.15.150:0/1294429093 learned_addr learned my addr 172.21.15.150:0/1294429093 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): get_auth_request con 0x7f2db00a0c80 auth_method 0 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db59a3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2db00a0110 msgr2=0x7f2db0141fa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db59a3700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2db00a0110 0x7f2db0141fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db0141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): get_auth_request con 0x7f2db009d1f0 auth_method 0 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db61a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient(hunting): handle_auth_done global_id 14950 payload 824 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 -- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2db00a0110 msgr2=0x7f2db0141fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2db00a0110 0x7f2db0141fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 -- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 msgr2=0x7f2db0141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2db009d1f0 0x7f2db0141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 monclient: found mon.noname-b 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.645+0000 7f2db69a5700 1 -- 172.21.15.150:0/1294429093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0009420 con 0x7f2db00a0c80 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db69a5700 1 -- 172.21.15.150:0/1294429093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00a0c80 msgr2=0x7f2db01424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db69a5700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2db00a0c80 0x7f2db01424e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2da800ccc0 tx=0x7f2da8005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da800ab40 con 0x7f2db00a0c80 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:34.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: dump: 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 5 monclient: authenticate success, global_id 14950 2024-09-18T10:12:34.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da800acc0 con 0x7f2db00a0c80 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8007340 con 0x7f2db00a0c80 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _renew_subs 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 1 -- 172.21.15.150:0/1294429093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db008faa0 con 0x7f2db00a0c80 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db49a1700 4 set_mon_vals no callback set 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 10 monclient: dump: 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.664 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _renew_subs 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 1 -- 172.21.15.150:0/1294429093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db00915d0 con 0x7f2db00a0c80 2024-09-18T10:12:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2da80076b0 con 0x7f2db00a0c80 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.649+0000 7f2db8408700 1 -- 172.21.15.150:0/1294429093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0063ed0 con 0x7f2db00a0c80 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.653+0000 7f2dae7fc700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d9c035a70 0x7f2d9c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.653+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(81..81 src has 40..81) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f2da8048de0 con 0x7f2db00a0c80 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.653+0000 7f2db61a4700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d9c035a70 0x7f2d9c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.653+0000 7f2db61a4700 10 monclient: get_auth_request con 0x7f2d9c035a70 auth_method 0 2024-09-18T10:12:34.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.653+0000 7f2db61a4700 1 --2- 172.21.15.150:0/1294429093 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d9c035a70 0x7f2d9c037f20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2da00030c0 tx=0x7f2da0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.665+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2da800ae30 con 0x7f2db00a0c80 2024-09-18T10:12:34.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.665+0000 7f2dae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.665+0000 7f2dae7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:34.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.821+0000 7f2db8408700 10 start_mon_command cmd=[{"prefix": "osd tree-from", "bucket": "osd.10"}] 2024-09-18T10:12:34.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.821+0000 7f2db8408700 10 monclient: _send_command 2 [{"prefix": "osd tree-from", "bucket": "osd.10"}] 2024-09-18T10:12:34.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.821+0000 7f2db8408700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.821+0000 7f2db8408700 1 -- 172.21.15.150:0/1294429093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree-from", "bucket": "osd.10"} v 0) v1 -- 0x7f2db0067b10 con 0x7f2db00a0c80 2024-09-18T10:12:34.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.825+0000 7f2dae7fc700 1 -- 172.21.15.150:0/1294429093 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree-from", "bucket": "osd.10"}]=-22 "osd.10" is not a bucket v81) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2da8010af0 con 0x7f2db00a0c80 2024-09-18T10:12:34.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.825+0000 7f2dae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree-from", "bucket": "osd.10"}] 2024-09-18T10:12:34.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.825+0000 7f2dae7fc700 10 monclient: _finish_command 2 = mon:22 "osd.10" is not a bucket 2024-09-18T10:12:34.833 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: "osd.10" is not a bucket 2024-09-18T10:12:34.833 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:34.833 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rm bar 2024-09-18T10:12:34.833 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:34.833 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm bar 2024-09-18T10:12:34.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: get_monmap_and_config 2024-09-18T10:12:34.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: monmap: 2024-09-18T10:12:34.929 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:34.929272+0000 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:34.929272+0000 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:34.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 1 Processor -- start 2024-09-18T10:12:34.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 1 -- start start 2024-09-18T10:12:34.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: init 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c09c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-b con 0x7f575c0710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f575c09a590 0x7f575c09cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-a con 0x7f575c09a590 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.925+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f575c09d2b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c09da40 con 0x7f575c09d2b0 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c09d6f0 con 0x7f575c09a590 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c071470 con 0x7f575c0710a0 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 10 monclient(hunting): authenticate will time out at 1037.024131s 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575a59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 msgr2=0x7f575c09c830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c09c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f575c09a590 0x7f575c09cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f575c09a590 0x7f575c09cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47362/0 (socket says 172.21.15.150:47362) 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 -- 172.21.15.150:0/957715912 learned_addr learned my addr 172.21.15.150:0/957715912 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): get_auth_request con 0x7f575c09a590 auth_method 0 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575ad9d700 1 --1- 172.21.15.150:0/957715912 >> v1:172.21.15.142:6789/0 conn(0x7f575c09d2b0 0x7f575c136650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35154/0 (socket says 172.21.15.150:35154) 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient(hunting): handle_auth_done global_id 5223 payload 274 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 -- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 msgr2=0x7f575c09c830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 --2- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c09c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 -- 172.21.15.150:0/957715912 >> v1:172.21.15.142:6789/0 conn(0x7f575c09d2b0 legacy=0x7f575c136650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 monclient: found mon.noname-a 2024-09-18T10:12:34.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:34.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 -- 172.21.15.150:0/957715912 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c09ab20 con 0x7f575c09a590 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 -- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09a590 msgr2=0x7f575c09cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5759d9b700 1 --2- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09a590 0x7f575c09cd70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f57500099b0 tx=0x7f5750009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8336a528e451c957 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 1 -- 172.21.15.150:0/957715912 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575000bb50 con 0x7f575c09a590 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 10 monclient: dump: 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.939 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 5 monclient: authenticate success, global_id 5223 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 1 -- 172.21.15.150:0/957715912 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575000bcb0 con 0x7f575c09a590 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f5760df7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:34.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.929+0000 7f575959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f575959a700 1 -- 172.21.15.150:0/957715912 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750011670 con 0x7f575c09a590 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 4 set_mon_vals no callback set 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f575959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f575959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f575959a700 10 monclient: dump: 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.941 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 10 monclient: shutdown 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09a590 msgr2=0x7f575c09cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 --2- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09a590 0x7f575c09cd70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f57500099b0 tx=0x7f5750009250 comp rx=0 tx=0).stop 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- 172.21.15.150:0/957715912 shutdown_connections 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 --2- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09a590 0x7f575c09cd70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 --2- 172.21.15.150:0/957715912 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c09c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- 172.21.15.150:0/957715912 >> 172.21.15.150:0/957715912 conn(0x7f575c068730 msgr2=0x7f575c0a1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- 172.21.15.150:0/957715912 shutdown_connections 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- 172.21.15.150:0/957715912 wait complete. 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 10 monclient: build_initial_monmap 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 10 monclient: monmap: 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:34.943 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 Processor -- start 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 1 -- start start 2024-09-18T10:12:34.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.933+0000 7f5760df7700 10 monclient: init 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c11e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-c con 0x7f575c0710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f575c09a590 0x7f575c13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-b con 0x7f575c09a590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09d2b0 0x7f575c13f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): picked mon.noname-a con 0x7f575c09d2b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c06e750 con 0x7f575c09d2b0 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c120940 con 0x7f575c09a590 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c13c9b0 con 0x7f575c0710a0 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient(hunting): authenticate will time out at 1037.032614s 2024-09-18T10:12:34.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575a59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 msgr2=0x7f575c11e030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c11e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f575c09a590 0x7f575c13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f575c09a590 0x7f575c13b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55914/0 (socket says 172.21.15.150:55914) 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 -- 172.21.15.150:0/3703202755 learned_addr learned my addr 172.21.15.150:0/3703202755 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): get_auth_request con 0x7f575c09a590 auth_method 0 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09d2b0 0x7f575c13f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): get_auth_request con 0x7f575c09d2b0 auth_method 0 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f575ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient(hunting): handle_auth_done global_id 14953 payload 824 2024-09-18T10:12:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 -- 172.21.15.150:0/3703202755 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 msgr2=0x7f575c11e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f575c0710a0 0x7f575c11e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 -- 172.21.15.150:0/3703202755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09d2b0 msgr2=0x7f575c13f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f575c09d2b0 0x7f575c13f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 monclient: found mon.noname-b 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 -- 172.21.15.150:0/3703202755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57500094f0 con 0x7f575c09a590 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 -- 172.21.15.150:0/3703202755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f575c09a590 msgr2=0x7f575c13b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5759d9b700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f575c09a590 0x7f575c13b5c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f575c0717e0 tx=0x7f5750016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750006c00 con 0x7f575c09a590 2024-09-18T10:12:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: dump: 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 5 monclient: authenticate success, global_id 14953 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _renew_subs 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5750020070 con 0x7f575c09a590 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 -- 172.21.15.150:0/3703202755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c1206c0 con 0x7f575c09a590 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5758d99700 4 set_mon_vals no callback set 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _renew_subs 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f5760df7700 1 -- 172.21.15.150:0/3703202755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c13c140 con 0x7f575c09a590 2024-09-18T10:12:34.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575001b450 con 0x7f575c09a590 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.937+0000 7f574a7fc700 10 monclient: dump: 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:34.955 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5750022030 con 0x7f575c09a590 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f574a7fc700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f573c035a70 0x7f573c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(81..81 src has 40..81) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f575004a740 con 0x7f575c09a590 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f575a59c700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f573c035a70 0x7f573c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f575a59c700 10 monclient: get_auth_request con 0x7f573c035a70 auth_method 0 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f5760df7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f5760df7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f5760df7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.941+0000 7f5760df7700 1 -- 172.21.15.150:0/3703202755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c063ed0 con 0x7f575c09a590 2024-09-18T10:12:34.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.945+0000 7f575a59c700 1 --2- 172.21.15.150:0/3703202755 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f573c035a70 0x7f573c037f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f574c003980 tx=0x7f574c006e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:34.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.949+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f575002e430 con 0x7f575c09a590 2024-09-18T10:12:34.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.949+0000 7f574a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:34.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:34.949+0000 7f574a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f5760df7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f5760df7700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f5760df7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f5760df7700 1 -- 172.21.15.150:0/3703202755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7f575c063ed0 con 0x7f575c09a590 2024-09-18T10:12:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f574a7fc700 1 -- 172.21.15.150:0/3703202755 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=-39 (39) Directory not empty v81) v1 ==== 99+0+0 (secure 0 0 0) 0x7f5750023b70 con 0x7f575c09a590 2024-09-18T10:12:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f574a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:35.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.121+0000 7f574a7fc700 10 monclient: _finish_command 2 = mon:39 (39) Directory not empty 2024-09-18T10:12:35.129 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOTEMPTY: (39) Directory not empty 2024-09-18T10:12:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:35.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush unlink host2 2024-09-18T10:12:35.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: get_monmap_and_config 2024-09-18T10:12:35.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: build_initial_monmap 2024-09-18T10:12:35.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: monmap: 2024-09-18T10:12:35.226 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:35.226 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:35.225681+0000 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:35.225681+0000 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:35.227 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:35.228 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:35.228 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:35.228 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:35.228 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 Processor -- start 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 -- start start 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: init 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:12:35.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19cc0a0120 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19cc0a0c00 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19cc09e7a0 0x7f19cc13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-a con 0x7f19cc09e7a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc09dae0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc09eba0 con 0x7f19cc0a0120 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc0a21d0 con 0x7f19cc09e7a0 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d3069700 10 monclient(hunting): authenticate will time out at 1037.319310s 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d0e05700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f19cc0a0120 0x7f19cc0a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50296/0 (socket says 172.21.15.150:50296) 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d0e05700 1 -- 172.21.15.150:0/1139560262 learned_addr learned my addr 172.21.15.150:0/1139560262 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:35.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 1 --2- 172.21.15.150:0/1139560262 >> v2:172.21.15.142:3300/0 conn(0x7f19cc09e7a0 0x7f19cc13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): get_auth_request con 0x7f19cc09e7a0 auth_method 0 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:35.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.221+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 10 monclient(hunting): handle_auth_done global_id 5229 payload 274 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 -- 172.21.15.150:0/1139560262 >> v1:172.21.15.150:6789/0 conn(0x7f19cc0a0120 legacy=0x7f19cc0a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 -- 172.21.15.150:0/1139560262 >> v1:172.21.15.142:6789/0 conn(0x7f19cc0a0c00 legacy=0x7f19cc1379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 10 monclient: _finish_hunting 0 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 monclient: found mon.noname-a 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 -- 172.21.15.150:0/1139560262 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0a1590 con 0x7f19cc09e7a0 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 -- 172.21.15.150:0/1139560262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc09e7a0 msgr2=0x7f19cc13b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d1606700 1 --2- 172.21.15.150:0/1139560262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc09e7a0 0x7f19cc13b0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009e60 tx=0x7f19c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c078332524ee0d9c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 1 -- 172.21.15.150:0/1139560262 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c400e070 con 0x7f19cc09e7a0 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:35.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: dump: 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:35.236 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d3069700 5 monclient: authenticate success, global_id 5229 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d3069700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 1 -- 172.21.15.150:0/1139560262 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4012430 con 0x7f19cc09e7a0 2024-09-18T10:12:35.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d3069700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19d3069700 4 set_mon_vals no callback set 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 1 -- 172.21.15.150:0/1139560262 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c40127d0 con 0x7f19cc09e7a0 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:35.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.225+0000 7f19cb7fe700 10 monclient: dump: 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:35.239 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: shutdown 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- 172.21.15.150:0/1139560262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc09e7a0 msgr2=0x7f19cc13b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 --2- 172.21.15.150:0/1139560262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc09e7a0 0x7f19cc13b0e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009e60 tx=0x7f19c4009780 comp rx=0 tx=0).stop 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- 172.21.15.150:0/1139560262 shutdown_connections 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 --2- 172.21.15.150:0/1139560262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc09e7a0 0x7f19cc13b0e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- 172.21.15.150:0/1139560262 >> 172.21.15.150:0/1139560262 conn(0x7f19cc0a5830 msgr2=0x7f19cc0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:35.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- 172.21.15.150:0/1139560262 shutdown_connections 2024-09-18T10:12:35.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- 172.21.15.150:0/1139560262 wait complete. 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: build_initial_monmap 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: monmap: 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.249 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 Processor -- start 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- start start 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: init 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 0x7f19cc141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-c con 0x7f19cc09e7a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc0a0120 0x7f19cc1421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-a con 0x7f19cc0a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:35.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): picked mon.noname-b con 0x7f19cc0a0c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0929c0 con 0x7f19cc0a0120 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc0a2ca0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc08fc90 con 0x7f19cc09e7a0 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d3069700 10 monclient(hunting): authenticate will time out at 1037.326453s 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d0e05700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 msgr2=0x7f19cc141ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d0e05700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 0x7f19cc141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:35.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55928/0 (socket says 172.21.15.150:55928) 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 -- 172.21.15.150:0/3045036970 learned_addr learned my addr 172.21.15.150:0/3045036970 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): get_auth_request con 0x7f19cc0a0c00 auth_method 0 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc0a0120 0x7f19cc1421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): get_auth_request con 0x7f19cc0a0120 auth_method 0 2024-09-18T10:12:35.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient(hunting): handle_auth_done global_id 14956 payload 824 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 -- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 msgr2=0x7f19cc141ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 0x7f19cc141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 -- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc0a0120 msgr2=0x7f19cc1421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:35.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc0a0120 0x7f19cc1421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient: _finish_hunting 0 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 monclient: found mon.noname-b 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.229+0000 7f19d1606700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4009a20 con 0x7f19cc0a0c00 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d1606700 1 -- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 msgr2=0x7f19cc08e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d1606700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f19c400b6c0 tx=0x7f19c4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4026430 con 0x7f19cc0a0c00 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: dump: 2024-09-18T10:12:35.254 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:35.255 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 5 monclient: authenticate success, global_id 14956 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4021070 con 0x7f19cc0a0c00 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 10 monclient: _renew_subs 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0917c0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4026820 con 0x7f19cc0a0c00 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:35.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 10 monclient: dump: 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 10 monclient: _renew_subs 2024-09-18T10:12:35.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19caffd700 4 set_mon_vals no callback set 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc0914d0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.233+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f19c4023030 con 0x7f19cc0a0c00 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d3069700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d3069700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d3069700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc063ed0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19c8ff9700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 0x7f19b8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(81..81 src has 40..81) v4 ==== 6133+0+0 (secure 0 0 0) 0x7f19c404afe0 con 0x7f19cc0a0c00 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d0e05700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 0x7f19b8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d0e05700 10 monclient: get_auth_request con 0x7f19b8035a70 auth_method 0 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.237+0000 7f19d0e05700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 0x7f19b8037f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19c0004860 tx=0x7f19c0007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:35.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.241+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f19c401bb80 con 0x7f19cc0a0c00 2024-09-18T10:12:35.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.241+0000 7f19c8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:35.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.241+0000 7f19c8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:35.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.413+0000 7f19d3069700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:35.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.413+0000 7f19d3069700 10 monclient: _send_command 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:35.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.413+0000 7f19d3069700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:35.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:35.413+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f19cc063ed0 con 0x7f19cc0a0c00 2024-09-18T10:12:36.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.348+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v82) v1 ==== 128+0+0 (secure 0 0 0) 0x7f19c401b3a0 con 0x7f19cc0a0c00 2024-09-18T10:12:36.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.348+0000 7f19c8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:36.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.348+0000 7f19c8ff9700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T10:12:36.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.456+0000 7f19d3069700 10 start_mon_command cmd=[{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:36.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.456+0000 7f19d3069700 10 monclient: _send_command 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:36.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.456+0000 7f19d3069700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:36.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:36.456+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush unlink", "name": "host2"} v 0) v1 -- 0x7f19cc063ed0 con 0x7f19cc0a0c00 2024-09-18T10:12:37.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.360+0000 7f19c8ff9700 1 -- 172.21.15.150:0/3045036970 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush unlink", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v83) v1 ==== 128+0+0 (secure 0 0 0) 0x7f19c4034030 con 0x7f19cc0a0c00 2024-09-18T10:12:37.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.360+0000 7f19c8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush unlink", "name": "host2"}] 2024-09-18T10:12:37.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.360+0000 7f19c8ff9700 10 monclient: _finish_command 3 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T10:12:37.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 msgr2=0x7f19b8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:37.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 0x7f19b8037f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19c0004860 tx=0x7f19c0007c10 comp rx=0 tx=0).stop 2024-09-18T10:12:37.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 10 monclient: shutdown 2024-09-18T10:12:37.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 msgr2=0x7f19cc08e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f19c400b6c0 tx=0x7f19c4018000 comp rx=0 tx=0).stop 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 shutdown_connections 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035a70 0x7f19b8037f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19cc0a0c00 0x7f19cc08e160 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19cc0a0120 0x7f19cc1421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 --2- 172.21.15.150:0/3045036970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19cc09e7a0 0x7f19cc141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.364+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 >> 172.21.15.150:0/3045036970 conn(0x7f19cc0a5830 msgr2=0x7f19cc09ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:37.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.368+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 shutdown_connections 2024-09-18T10:12:37.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.368+0000 7f19d3069700 1 -- 172.21.15.150:0/3045036970 wait complete. 2024-09-18T10:12:37.376 INFO:tasks.workunit.client.0.smithi150.stderr:removed item id -16 name 'host2' from crush map 2024-09-18T10:12:37.387 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket host-for-test host root=root-for-test rack=rack-for-test 2024-09-18T10:12:37.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: get_monmap_and_config 2024-09-18T10:12:37.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: build_initial_monmap 2024-09-18T10:12:37.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: monmap: 2024-09-18T10:12:37.502 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:37.502 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:37.494198+0000 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:37.494198+0000 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:37.503 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 Processor -- start 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 -- start start 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: init 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f051009acd0 0x7f0510096b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): picked mon.noname-a con 0x7f051009acd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f051009bfd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f051009cab0 0x7f0510137790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): picked mon.noname-c con 0x7f051009cab0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510095e70 con 0x7f051009bfd0 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0510096250 con 0x7f051009acd0 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0510095920 con 0x7f051009cab0 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f0515199700 10 monclient(hunting): authenticate will time out at 1039.587743s 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.488+0000 7f050f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f051009cab0 0x7f0510137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f051009cab0 0x7f0510137790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55934/0 (socket says 172.21.15.150:55934) 2024-09-18T10:12:37.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 -- 172.21.15.150:0/1805774206 learned_addr learned my addr 172.21.15.150:0/1805774206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): get_auth_request con 0x7f051009cab0 auth_method 0 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 1 --2- 172.21.15.150:0/1805774206 >> v2:172.21.15.142:3300/0 conn(0x7f051009acd0 0x7f0510096b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:37.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): get_auth_request con 0x7f051009acd0 auth_method 0 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient(hunting): handle_auth_done global_id 14959 payload 274 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 -- 172.21.15.150:0/1805774206 >> v2:172.21.15.142:3300/0 conn(0x7f051009acd0 msgr2=0x7f0510096b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 --2- 172.21.15.150:0/1805774206 >> v2:172.21.15.142:3300/0 conn(0x7f051009acd0 0x7f0510096b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 -- 172.21.15.150:0/1805774206 >> v1:172.21.15.142:6789/0 conn(0x7f051009bfd0 legacy=0x7f051009c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient: _finish_hunting 0 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 monclient: found mon.noname-c 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 -- 172.21.15.150:0/1805774206 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051009b0a0 con 0x7f051009cab0 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 -- 172.21.15.150:0/1805774206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009cab0 msgr2=0x7f0510137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050f59e700 1 --2- 172.21.15.150:0/1805774206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009cab0 0x7f0510137790 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0510067420 tx=0x7f0508009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e41b6fabfed4d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 1 -- 172.21.15.150:0/1805774206 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0508009e70 con 0x7f051009cab0 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:37.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: dump: 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:12:37.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 1 -- 172.21.15.150:0/1805774206 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050800ba40 con 0x7f051009cab0 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f0515199700 5 monclient: authenticate success, global_id 14959 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f0515199700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f0515199700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 1 -- 172.21.15.150:0/1805774206 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050800be00 con 0x7f051009cab0 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f0515199700 4 set_mon_vals no callback set 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.492+0000 7f050dd9b700 10 monclient: dump: 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:37.510 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: shutdown 2024-09-18T10:12:37.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- 172.21.15.150:0/1805774206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009cab0 msgr2=0x7f0510137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- 172.21.15.150:0/1805774206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009cab0 0x7f0510137790 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0510067420 tx=0x7f0508009250 comp rx=0 tx=0).stop 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- 172.21.15.150:0/1805774206 shutdown_connections 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- 172.21.15.150:0/1805774206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009cab0 0x7f0510137790 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- 172.21.15.150:0/1805774206 >> v2:172.21.15.142:3300/0 conn(0x7f051009acd0 0x7f0510096b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- 172.21.15.150:0/1805774206 >> 172.21.15.150:0/1805774206 conn(0x7f05100a15e0 msgr2=0x7f05100a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- 172.21.15.150:0/1805774206 shutdown_connections 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- 172.21.15.150:0/1805774206 wait complete. 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: build_initial_monmap 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: monmap: 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:37.512 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 Processor -- start 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- start start 2024-09-18T10:12:37.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: init 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 0x7f051013f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): picked mon.noname-c con 0x7f051009acd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): picked mon.noname-b con 0x7f051009bfd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 0x7f05101402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): picked mon.noname-a con 0x7f051009cab0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051008f610 con 0x7f051009cab0 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:37.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051006f530 con 0x7f051009bfd0 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05100904c0 con 0x7f051009acd0 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f0515199700 10 monclient(hunting): authenticate will time out at 1039.595286s 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050ed9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 msgr2=0x7f051013f840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55942/0 (socket says 172.21.15.150:55942) 2024-09-18T10:12:37.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 0x7f051013f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 -- 172.21.15.150:0/1007801926 learned_addr learned my addr 172.21.15.150:0/1007801926 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): get_auth_request con 0x7f051009bfd0 auth_method 0 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050f59e700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 0x7f05101402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient(hunting): handle_auth_done global_id 14962 payload 824 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 -- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 msgr2=0x7f051013f840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:37.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 0x7f051013f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 -- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 msgr2=0x7f05101402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 0x7f05101402c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 monclient: found mon.noname-b 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050e59c700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05080094f0 con 0x7f051009bfd0 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050f59e700 10 monclient: get_auth_request con 0x7f051009cab0 auth_method 0 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.496+0000 7f050f59e700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 0x7f05101402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:12:37.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f050e59c700 1 -- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 msgr2=0x7f051013fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f050e59c700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c600 tx=0x7f04f8007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80117a0 con 0x7f051009bfd0 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: dump: 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.521 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f8011900 con 0x7f051009bfd0 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 5 monclient: authenticate success, global_id 14962 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _renew_subs 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051008dce0 con 0x7f051009bfd0 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8011cc0 con 0x7f051009bfd0 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0506ffd700 10 monclient: dump: 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:37.523 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f050d59a700 4 set_mon_vals no callback set 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _renew_subs 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051008f230 con 0x7f051009bfd0 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.500+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510095b90 con 0x7f051009bfd0 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f04f8012630 con 0x7f051009bfd0 2024-09-18T10:12:37.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f0506ffd700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 0x7f04f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(83..83 src has 40..83) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f04f80485e0 con 0x7f051009bfd0 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f050ed9d700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 0x7f04f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f050ed9d700 10 monclient: get_auth_request con 0x7f04f0035a70 auth_method 0 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.504+0000 7f050ed9d700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 0x7f04f0037f20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0500009ac0 tx=0x7f050000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.512+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f04f8011e60 con 0x7f051009bfd0 2024-09-18T10:12:37.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.512+0000 7f0506ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:37.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.512+0000 7f0506ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:37.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.680+0000 7f0515199700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:37.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.680+0000 7f0515199700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:37.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.680+0000 7f0515199700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:37.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:37.680+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f051008dce0 con 0x7f051009bfd0 2024-09-18T10:12:38.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.368+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} v84) v1 ==== 251+0+0 (secure 0 0 0) 0x7f051008dce0 con 0x7f051009bfd0 2024-09-18T10:12:38.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.368+0000 7f0506ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:38.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.368+0000 7f0506ffd700 10 monclient: _finish_command 2 = system:0 added bucket host-for-test type host to location {rack=rack-for-test,root=root-for-test} 2024-09-18T10:12:38.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.472+0000 7f0515199700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:38.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.472+0000 7f0515199700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:38.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.472+0000 7f0515199700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.472+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]} v 0) v1 -- 0x7f051008dce0 con 0x7f051009bfd0 2024-09-18T10:12:38.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0506ffd700 1 -- 172.21.15.150:0/1007801926 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}]=0 bucket 'host-for-test' already exists v84) v1 ==== 200+0+0 (secure 0 0 0) 0x7f051008dce0 con 0x7f051009bfd0 2024-09-18T10:12:38.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0506ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "host-for-test", "type": "host", "args": ["root=root-for-test", "rack=rack-for-test"]}] 2024-09-18T10:12:38.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0506ffd700 10 monclient: _finish_command 3 = system:0 bucket 'host-for-test' already exists 2024-09-18T10:12:38.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 msgr2=0x7f04f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 0x7f04f0037f20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0500009ac0 tx=0x7f050000b040 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 10 monclient: shutdown 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 msgr2=0x7f051013fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c600 tx=0x7f04f8007f60 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 shutdown_connections 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f04f0035a70 0x7f04f0037f20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f051009cab0 0x7f05101402c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f051009bfd0 0x7f051013fd80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 --2- 172.21.15.150:0/1007801926 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f051009acd0 0x7f051013f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 >> 172.21.15.150:0/1007801926 conn(0x7f05100a15e0 msgr2=0x7f051006c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:38.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 shutdown_connections 2024-09-18T10:12:38.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.476+0000 7f0515199700 1 -- 172.21.15.150:0/1007801926 wait complete. 2024-09-18T10:12:38.484 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'host-for-test' already exists 2024-09-18T10:12:38.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:38.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep host-for-test 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.580+0000 7f51435d1700 10 monclient: get_monmap_and_config 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.580+0000 7f51435d1700 10 monclient: build_initial_monmap 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.580+0000 7f51435d1700 10 monclient: monmap: 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:38.587989+0000 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:38.587989+0000 2024-09-18T10:12:38.588 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:38.589 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 Processor -- start 2024-09-18T10:12:38.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 -- start start 2024-09-18T10:12:38.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient: init 2024-09-18T10:12:38.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f513c09c2e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f513c133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-b con 0x7f513c1316e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f513c134000 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c095de0 con 0x7f513c134000 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513c09af60 con 0x7f513c09c2e0 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c098580 con 0x7f513c1316e0 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f51435d1700 10 monclient(hunting): authenticate will time out at 1040.681403s 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f514136d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f513c09c2e0 0x7f513c0956d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50310/0 (socket says 172.21.15.150:50310) 2024-09-18T10:12:38.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f5140b6c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 msgr2=0x7f513c133ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:38.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 learned_addr learned my addr 172.21.15.150:0/1225253683 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:38.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f5140b6c700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f513c133ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:38.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.584+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 >> v1:172.21.15.150:6789/0 conn(0x7f513c09c2e0 legacy=0x7f513c0956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:38.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5141b6e700 1 -- 172.21.15.150:0/1225253683 >> v1:172.21.15.142:6789/0 conn(0x7f513c134000 legacy=0x7f513c1363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663729968 0 0) 0x7f513c09af60 con 0x7f513c09c2e0 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): my global_id is 14965 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5124003570 con 0x7f513c09c2e0 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784191687 0 0) 0x7f513c095de0 con 0x7f513c134000 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): my global_id is 5238 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f513c09af60 con 0x7f513c134000 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2725459827 0 0) 0x7f5124003570 con 0x7f513c09c2e0 2024-09-18T10:12:38.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 msgr2=0x7f513c133ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f513c133ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 >> v1:172.21.15.142:6789/0 conn(0x7f513c134000 legacy=0x7f513c1363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c09b520 con 0x7f513c09c2e0 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f51435d1700 5 monclient: authenticate success, global_id 14965 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f5138001860 con 0x7f513c09c2e0 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:38.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: dump: 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:38.600 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:38.601 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:38.601 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:38.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:38.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 >> v1:172.21.15.150:6789/0 conn(0x7f513c09c2e0 legacy=0x7f513c0956d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f5124004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): picked mon.b con 0x7f5124003c40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f5124004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): picked mon.c con 0x7f513c1316e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5124004f50 0x7f512400dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): picked mon.a con 0x7f5124004f50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c098580 con 0x7f5124004f50 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512400e1b0 con 0x7f5124003c40 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f512400e2f0 con 0x7f513c1316e0 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f5124004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5140b6c700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 msgr2=0x7f5124004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): get_auth_request con 0x7f5124003c40 auth_method 0 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f51435d1700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5140b6c700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f5124004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:38.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5141b6e700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5124004f50 0x7f512400dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient(hunting): handle_auth_done global_id 14965 payload 293 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 msgr2=0x7f5124004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c1316e0 0x7f5124004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5124004f50 msgr2=0x7f512400dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5124004f50 0x7f512400dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 monclient: found mon.b 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124003570 con 0x7f5124003c40 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 10 monclient: _finish_auth 0 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5141b6e700 10 monclient: get_auth_request con 0x7f5124004f50 auth_method 0 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5141b6e700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5124004f50 0x7f512400dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 msgr2=0x7f5124004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f514136d700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f5124004030 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f51380014d0 tx=0x7f5138007b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21df42fd09fe9c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138007e70 con 0x7f5124003c40 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: dump: 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.605 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f51435d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:38.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513800ce90 con 0x7f5124003c40 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f51435d1700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f51435d1700 4 set_mon_vals no callback set 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 1 -- 172.21.15.150:0/1225253683 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138005b50 con 0x7f5124003c40 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.588+0000 7f5133fff700 10 monclient: dump: 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:38.607 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 10 monclient: shutdown 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 msgr2=0x7f5124004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 --2- 172.21.15.150:0/1225253683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f5124004030 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f51380014d0 tx=0x7f5138007b40 comp rx=0 tx=0).stop 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f5141b6e700 1 -- 172.21.15.150:0/1225253683 reap_dead start 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- 172.21.15.150:0/1225253683 shutdown_connections 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- 172.21.15.150:0/1225253683 >> 172.21.15.150:0/1225253683 conn(0x7f513c0a1710 msgr2=0x7f513c0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- 172.21.15.150:0/1225253683 shutdown_connections 2024-09-18T10:12:38.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- 172.21.15.150:0/1225253683 wait complete. 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 10 monclient: build_initial_monmap 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 10 monclient: monmap: 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:38.614 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 Processor -- start 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 1 -- start start 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.592+0000 7f51435d1700 10 monclient: init 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-b con 0x7f5124003c40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f513c126640 0x7f513c126a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-a con 0x7f513c126640 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 0x7f513c094150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): picked mon.noname-c con 0x7f513c126f50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:38.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c129370 con 0x7f513c126640 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c128d20 con 0x7f5124003c40 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c128a40 con 0x7f513c126f50 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient(hunting): authenticate will time out at 1040.692347s 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5141b6e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 msgr2=0x7f513c094150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5141b6e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 0x7f513c094150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55946/0 (socket says 172.21.15.150:55946) 2024-09-18T10:12:38.616 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 -- 172.21.15.150:0/478231763 learned_addr learned my addr 172.21.15.150:0/478231763 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): get_auth_request con 0x7f5124003c40 auth_method 0 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f513c126640 0x7f513c126a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): get_auth_request con 0x7f513c126640 auth_method 0 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:38.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f5140b6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient(hunting): handle_auth_done global_id 14968 payload 824 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 -- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 msgr2=0x7f513c094150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 0x7f513c094150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 -- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f513c126640 msgr2=0x7f513c126a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f513c126640 0x7f513c126a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 monclient: found mon.noname-b 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c129000 con 0x7f5124003c40 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 -- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 msgr2=0x7f513c129cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:38.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f514136d700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5138007b80 tx=0x7f5138015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51380074f0 con 0x7f5124003c40 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: dump: 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:38.619 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 5 monclient: authenticate success, global_id 14968 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5138007650 con 0x7f5124003c40 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138012400 con 0x7f5124003c40 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _renew_subs 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c12a200 con 0x7f5124003c40 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51337fe700 4 set_mon_vals no callback set 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:38.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51317fa700 10 monclient: dump: 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _renew_subs 2024-09-18T10:12:38.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.596+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c06e7c0 con 0x7f5124003c40 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51435d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51435d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51435d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5138019030 con 0x7f5124003c40 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c06e9b0 con 0x7f5124003c40 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51317fa700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 0x7f5128037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(84..84 src has 40..84) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f513804a230 con 0x7f5124003c40 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f5140b6c700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 0x7f5128037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.600+0000 7f5140b6c700 10 monclient: get_auth_request con 0x7f5128035a70 auth_method 0 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.604+0000 7f5140b6c700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 0x7f5128037f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f512c003d00 tx=0x7f512c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.608+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5138011960 con 0x7f5124003c40 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.608+0000 7f51317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:38.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.608+0000 7f51317fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51435d1700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51435d1700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51435d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f513c063ed0 con 0x7f5124003c40 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v84) v1 ==== 56+0+1575 (secure 0 0 0) 0x7f5138017070 con 0x7f5124003c40 2024-09-18T10:12:38.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51317fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:38.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.776+0000 7f51317fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:38.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51435d1700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:38.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51435d1700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:38.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51435d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:38.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f513c0962c0 con 0x7f5124003c40 2024-09-18T10:12:38.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51317fa700 1 -- 172.21.15.150:0/478231763 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v84) v1 ==== 56+0+1575 (secure 0 0 0) 0x7f513801a380 con 0x7f5124003c40 2024-09-18T10:12:38.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51317fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:38.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.880+0000 7f51317fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 msgr2=0x7f5128037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 0x7f5128037f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f512c003d00 tx=0x7f512c00d040 comp rx=0 tx=0).stop 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 10 monclient: shutdown 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 msgr2=0x7f513c129cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5138007b80 tx=0x7f5138015000 comp rx=0 tx=0).stop 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 shutdown_connections 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5128035a70 0x7f5128037f20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f513c126f50 0x7f513c094150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f513c126640 0x7f513c126a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 --2- 172.21.15.150:0/478231763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5124003c40 0x7f513c129cc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:38.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 >> 172.21.15.150:0/478231763 conn(0x7f513c0a1710 msgr2=0x7f513c09df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:38.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 shutdown_connections 2024-09-18T10:12:38.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.884+0000 7f51435d1700 1 -- 172.21.15.150:0/478231763 wait complete. 2024-09-18T10:12:38.900 INFO:tasks.workunit.client.0.smithi150.stdout:-11 0 host host-for-test 2024-09-18T10:12:38.900 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:38.900 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep rack-for-test 2024-09-18T10:12:38.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 10 monclient: get_monmap_and_config 2024-09-18T10:12:38.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 10 monclient: build_initial_monmap 2024-09-18T10:12:38.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 10 monclient: monmap: 2024-09-18T10:12:38.995 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:38.995323+0000 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:38.995323+0000 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:38.996 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 1 Processor -- start 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 1 -- start start 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.988+0000 7fe9d2882700 10 monclient: init 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc09ac60 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9cc06f9e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9cc135710 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc09b9e0 con 0x7fe9cc135710 2024-09-18T10:12:38.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9cc09b780 con 0x7fe9cc06f9e0 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9cc09b030 con 0x7fe9cc09ac60 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d2882700 10 monclient(hunting): authenticate will time out at 1041.088585s 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cb7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe9cc06f9e0 0x7fe9cc09c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33512/0 (socket says 172.21.15.150:33512) 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59486/0 (socket says 172.21.15.150:59486) 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/38164232 learned_addr learned my addr 172.21.15.150:0/38164232 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): get_auth_request con 0x7fe9cc09ac60 auth_method 0 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:38.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/38164232 >> v1:172.21.15.150:6789/0 conn(0x7fe9cc06f9e0 legacy=0x7fe9cc09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient(hunting): handle_auth_done global_id 14971 payload 274 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9d0e1f700 1 -- 172.21.15.150:0/38164232 >> v1:172.21.15.142:6789/0 conn(0x7fe9cc135710 legacy=0x7fe9cc137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 -- 172.21.15.150:0/38164232 >> v1:172.21.15.150:6789/0 conn(0x7fe9cc06f9e0 legacy=0x7fe9cc09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 -- 172.21.15.150:0/38164232 >> v1:172.21.15.142:6789/0 conn(0x7fe9cc135710 legacy=0x7fe9cc137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9caffd700 1 -- 172.21.15.150:0/38164232 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560957487 0 0) 0x7fe9cc09b780 con 0x7fe9cc06f9e0 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 monclient: found mon.noname-c 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9cbfff700 1 -- 172.21.15.150:0/38164232 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc098270 con 0x7fe9cc09ac60 2024-09-18T10:12:39.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.992+0000 7fe9caffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9cbfff700 1 -- 172.21.15.150:0/38164232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc09ac60 msgr2=0x7fe9cc09c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9cbfff700 1 --2- 172.21.15.150:0/38164232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40099e0 tx=0x7fe9b40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d67056e3795219 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 1 -- 172.21.15.150:0/38164232 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400bba0 con 0x7fe9cc09ac60 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: dump: 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.003 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 5 monclient: authenticate success, global_id 14971 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:39.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 1 -- 172.21.15.150:0/38164232 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b400bd20 con 0x7fe9cc09ac60 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 4 set_mon_vals no callback set 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 1 -- 172.21.15.150:0/38164232 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4011670 con 0x7fe9cc09ac60 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9caffd700 10 monclient: dump: 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.005 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 10 monclient: shutdown 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 -- 172.21.15.150:0/38164232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc09ac60 msgr2=0x7fe9cc09c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 --2- 172.21.15.150:0/38164232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40099e0 tx=0x7fe9b40091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 -- 172.21.15.150:0/38164232 shutdown_connections 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 --2- 172.21.15.150:0/38164232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc09ac60 0x7fe9cc09c2e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 -- 172.21.15.150:0/38164232 >> 172.21.15.150:0/38164232 conn(0x7fe9cc0a1710 msgr2=0x7fe9cc0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 -- 172.21.15.150:0/38164232 shutdown_connections 2024-09-18T10:12:39.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:38.996+0000 7fe9d2882700 1 -- 172.21.15.150:0/38164232 wait complete. 2024-09-18T10:12:39.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient: monmap: 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 Processor -- start 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 -- start start 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient: init 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 0x7fe9cc134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc06f9e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc135710 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 0x7fe9cc08bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): picked mon.noname-a con 0x7fe9cc090170 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc094fc0 con 0x7fe9cc090170 2024-09-18T10:12:39.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cbfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 msgr2=0x7fe9cc134b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc08d980 con 0x7fe9cc135710 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc08efa0 con 0x7fe9cc06f9e0 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 0x7fe9cc134b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59494/0 (socket says 172.21.15.150:59494) 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/34871054 learned_addr learned my addr 172.21.15.150:0/34871054 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d2882700 10 monclient(hunting): authenticate will time out at 1041.098988s 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): get_auth_request con 0x7fe9cc135710 auth_method 0 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient(hunting): handle_auth_done global_id 14977 payload 824 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d0e1f700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 0x7fe9cc08bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 msgr2=0x7fe9cc134b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 0x7fe9cc134b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 msgr2=0x7fe9cc08bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 0x7fe9cc08bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:39.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b4009420 con 0x7fe9cc135710 2024-09-18T10:12:39.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d0e1f700 10 monclient: get_auth_request con 0x7fe9cc090170 auth_method 0 2024-09-18T10:12:39.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.000+0000 7fe9d0e1f700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 0x7fe9cc08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9cb7fe700 1 -- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 msgr2=0x7fe9cc1350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9cb7fe700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c7c0 tx=0x7fe9bc007bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc005e20 con 0x7fe9cc135710 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: dump: 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.015 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0070d0 con 0x7fe9cc135710 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 5 monclient: authenticate success, global_id 14977 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _renew_subs 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc08f2e0 con 0x7fe9cc135710 2024-09-18T10:12:39.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc007470 con 0x7fe9cc135710 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 10 monclient: dump: 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.017 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9ca7fc700 4 set_mon_vals no callback set 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _renew_subs 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc08d690 con 0x7fe9cc135710 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc063ed0 con 0x7fe9cc135710 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.004+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe9bc01b3b0 con 0x7fe9cc135710 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.008+0000 7fe9c3fff700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 0x7fe9ac037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.008+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(84..84 src has 40..84) v4 ==== 6610+0+0 (secure 0 0 0) 0x7fe9bc048140 con 0x7fe9cc135710 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.008+0000 7fe9cbfff700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 0x7fe9ac037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.008+0000 7fe9cbfff700 10 monclient: get_auth_request con 0x7fe9ac035a70 auth_method 0 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.008+0000 7fe9cbfff700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 0x7fe9ac037f20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40099e0 tx=0x7fe9b4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.012+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe9bc0196c0 con 0x7fe9cc135710 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.012+0000 7fe9c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.012+0000 7fe9c3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:39.047 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:12:39.047 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:12:39.047 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:12:39.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:12:39.050 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:12:39.050 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:12:39.050 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: get_monmap_and_config 2024-09-18T10:12:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: monmap: 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:39.157743+0000 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:39.157743+0000 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:39.160 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 Processor -- start 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 -- start start 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: init 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 0x7f300012f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): picked mon.noname-c con 0x7f300012ec80 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f300009bf90 0x7f300012b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): picked mon.noname-a con 0x7f300009bf90 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): picked mon.noname-b con 0x7f300009c3d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300009e470 con 0x7f300009bf90 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300009e8b0 con 0x7f300012ec80 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300009a4a0 con 0x7f300009c3d0 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.152+0000 7f300661e700 10 monclient(hunting): authenticate will time out at 1048.083992s 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f300009bf90 0x7f300012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f300009bf90 0x7f300012b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60340/0 (socket says 172.21.15.142:60340) 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 -- 172.21.15.142:0/2245292747 learned_addr learned my addr 172.21.15.142:0/2245292747 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): get_auth_request con 0x7f300009bf90 auth_method 0 2024-09-18T10:12:39.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 1 --2- 172.21.15.142:0/2245292747 >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 0x7f300012f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): get_auth_request con 0x7f300012ec80 auth_method 0 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 10 monclient(hunting): get_auth_request con 0x7f300009c3d0 auth_method 0 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 1 -- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 msgr2=0x7f300012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 1 -- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 msgr2=0x7f300012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient(hunting): handle_auth_done global_id 5247 payload 274 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 -- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 msgr2=0x7f300012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3005e1d700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 -- 172.21.15.142:0/2245292747 >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 msgr2=0x7f300012f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:39.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2245292747 >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 0x7f300012f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 monclient: found mon.noname-a 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 -- 172.21.15.142:0/2245292747 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300009ce70 con 0x7f300009bf90 2024-09-18T10:12:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300561c700 1 --2- 172.21.15.142:0/2245292747 >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 0x7f300012f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 -- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300009bf90 msgr2=0x7f300012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300009bf90 0x7f300012b770 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f30000a34d0 tx=0x7f2ff0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09107f7010318d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 1 -- 172.21.15.142:0/2245292747 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0009ea0 con 0x7f300009bf90 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: dump: 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300661e700 5 monclient: authenticate success, global_id 5247 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300661e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 1 -- 172.21.15.142:0/2245292747 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff000ba30 con 0x7f300009bf90 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300661e700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f300661e700 4 set_mon_vals no callback set 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 1 -- 172.21.15.142:0/2245292747 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff000bdd0 con 0x7f300009bf90 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.156+0000 7f2ff7fff700 10 monclient: dump: 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: shutdown 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300009bf90 msgr2=0x7f300012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300009bf90 0x7f300012b770 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f30000a34d0 tx=0x7f2ff0009250 comp rx=0 tx=0).stop 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- 172.21.15.142:0/2245292747 shutdown_connections 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009c3d0 0x7f300012bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- 172.21.15.142:0/2245292747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300009bf90 0x7f300012b770 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- 172.21.15.142:0/2245292747 >> v2:172.21.15.150:3300/0 conn(0x7f300012ec80 0x7f300012f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- 172.21.15.142:0/2245292747 >> 172.21.15.142:0/2245292747 conn(0x7f300006a2e0 msgr2=0x7f30000a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- 172.21.15.142:0/2245292747 shutdown_connections 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- 172.21.15.142:0/2245292747 wait complete. 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: monmap: 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 Processor -- start 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- start start 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: init 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): picked mon.noname-c con 0x7f300009bf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 0x7f300012a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): picked mon.noname-b con 0x7f300009c3d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 0x7f300012a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): picked mon.noname-a con 0x7f300012ec80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300006f920 con 0x7f300012ec80 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300009de10 con 0x7f300009c3d0 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3000099b90 con 0x7f300009bf90 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300661e700 10 monclient(hunting): authenticate will time out at 1048.091205s 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 0x7f300012a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51742/0 (socket says 172.21.15.142:51742) 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 -- 172.21.15.142:0/2068140584 learned_addr learned my addr 172.21.15.142:0/2068140584 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 10 monclient(hunting): get_auth_request con 0x7f300009bf90 auth_method 0 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): get_auth_request con 0x7f300012ec80 auth_method 0 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 0x7f300012a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): get_auth_request con 0x7f300009c3d0 auth_method 0 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 msgr2=0x7f3000129e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 msgr2=0x7f3000129e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3005e1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f300561c700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.160+0000 7f3004e1b700 10 monclient(hunting): handle_auth_done global_id 14983 payload 824 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 msgr2=0x7f3000129e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 msgr2=0x7f300012a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 0x7f300012a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 monclient: found mon.noname-b 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 -- 172.21.15.142:0/2068140584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc009420 con 0x7f300009c3d0 2024-09-18T10:12:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3005e1d700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 0x7f300012a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 msgr2=0x7f300012a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f3004e1b700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 0x7f300012a360 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30000a0330 tx=0x7f2ff0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0011e90 con 0x7f300009c3d0 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: dump: 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0024440 con 0x7f300009c3d0 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 5 monclient: authenticate success, global_id 14983 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0024820 con 0x7f300009c3d0 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 10 monclient: _renew_subs 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff6ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff00094f0 con 0x7f300009c3d0 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f2ff57fa700 10 monclient: dump: 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 10 monclient: _renew_subs 2024-09-18T10:12:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.164+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300012c080 con 0x7f300009c3d0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300661e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300661e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300661e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2ff001a420 con 0x7f300009c3d0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3000072a80 con 0x7f300009c3d0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f2ff57fa700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 0x7f2fe4037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300561c700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 0x7f2fe4037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.168+0000 7f300561c700 10 monclient: get_auth_request con 0x7f2fe4035b10 auth_method 0 2024-09-18T10:12:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.172+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 40..85) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f2ff002ee40 con 0x7f300009c3d0 2024-09-18T10:12:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.172+0000 7f300561c700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 0x7f2fe4037fc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0096d0 tx=0x7f2ffc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.180+0000 7fe9d2882700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:39.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.180+0000 7fe9d2882700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.180+0000 7fe9d2882700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.180+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe9cc095c50 con 0x7fe9cc135710 2024-09-18T10:12:39.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.184+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2ff002cdb0 con 0x7f300009c3d0 2024-09-18T10:12:39.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.184+0000 7f2ff57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.184+0000 7f2ff57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.188+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v85) v1 ==== 56+0+1575 (secure 0 0 0) 0x7fe9bc017070 con 0x7fe9cc135710 2024-09-18T10:12:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.188+0000 7fe9c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.188+0000 7fe9c3fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9d2882700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9d2882700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9d2882700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fe9cc08e060 con 0x7fe9cc135710 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9c3fff700 1 -- 172.21.15.150:0/34871054 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v85) v1 ==== 56+0+1575 (secure 0 0 0) 0x7fe9bc00e290 con 0x7fe9cc135710 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9c3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.284+0000 7fe9c3fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 msgr2=0x7fe9ac037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 0x7fe9ac037f20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40099e0 tx=0x7fe9b4017000 comp rx=0 tx=0).stop 2024-09-18T10:12:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 10 monclient: shutdown 2024-09-18T10:12:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 msgr2=0x7fe9cc1350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00c7c0 tx=0x7fe9bc007bf0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 shutdown_connections 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe9ac035a70 0x7fe9ac037f20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe9cc090170 0x7fe9cc08bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc135710 0x7fe9cc1350c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 --2- 172.21.15.150:0/34871054 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9cc06f9e0 0x7fe9cc134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 >> 172.21.15.150:0/34871054 conn(0x7fe9cc0a1710 msgr2=0x7fe9cc09df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 shutdown_connections 2024-09-18T10:12:39.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.288+0000 7fe9d2882700 1 -- 172.21.15.150:0/34871054 wait complete. 2024-09-18T10:12:39.302 INFO:tasks.workunit.client.0.smithi150.stdout:-12 0 rack rack-for-test 2024-09-18T10:12:39.303 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep root-for-test 2024-09-18T10:12:39.304 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:12:39.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: get_monmap_and_config 2024-09-18T10:12:39.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: monmap: 2024-09-18T10:12:39.396 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:39.396 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:39.396296+0000 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:39.396296+0000 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:39.397 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 Processor -- start 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 -- start start 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: init 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f040a0120 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f040a0c00 0x7f4f0409e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-c con 0x7f4f040a0c00 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-b con 0x7f4f041379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0409dae0 con 0x7f4f040a0120 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f040a21d0 con 0x7f4f040a0c00 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0409ddc0 con 0x7f4f041379b0 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0c267700 10 monclient(hunting): authenticate will time out at 1041.489684s 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f040a0c00 0x7f4f0409e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a003700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4f040a0120 0x7f4f040a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33514/0 (socket says 172.21.15.150:33514) 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a003700 1 -- 172.21.15.150:0/3399271959 learned_addr learned my addr 172.21.15.150:0/3399271959 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): get_auth_request con 0x7f4f040a0c00 auth_method 0 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 10 monclient(hunting): get_auth_request con 0x7f4f041379b0 auth_method 0 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 1 -- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 msgr2=0x7f4f04139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:39.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 1 -- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 msgr2=0x7f4f04139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f0a804700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient(hunting): handle_auth_done global_id 14986 payload 274 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 -- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 msgr2=0x7f4f04139d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 -- 172.21.15.150:0/3399271959 >> v1:172.21.15.150:6789/0 conn(0x7f4f040a0120 legacy=0x7f4f040a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 monclient: found mon.noname-c 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.392+0000 7f4f09802700 1 -- 172.21.15.150:0/3399271959 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040a1590 con 0x7f4f040a0c00 2024-09-18T10:12:39.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09802700 1 -- 172.21.15.150:0/3399271959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f040a0c00 msgr2=0x7f4f0409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09802700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f040a0c00 0x7f4f0409e7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80098e0 tx=0x7f4ef80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9858883c5750becd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 1 -- 172.21.15.150:0/3399271959 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8009e80 con 0x7f4f040a0c00 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: dump: 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.408 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 1 -- 172.21.15.150:0/3399271959 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef800ba70 con 0x7f4f040a0c00 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 5 monclient: authenticate success, global_id 14986 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 1 -- 172.21.15.150:0/3399271959 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800bdf0 con 0x7f4f040a0c00 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 4 set_mon_vals no callback set 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f09001700 10 monclient: dump: 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 10 monclient: shutdown 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 -- 172.21.15.150:0/3399271959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f040a0c00 msgr2=0x7f4f0409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f040a0c00 0x7f4f0409e7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80098e0 tx=0x7f4ef80091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 -- 172.21.15.150:0/3399271959 shutdown_connections 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f041379b0 0x7f4f04139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 --2- 172.21.15.150:0/3399271959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f040a0c00 0x7f4f0409e7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 -- 172.21.15.150:0/3399271959 >> 172.21.15.150:0/3399271959 conn(0x7f4f040a5830 msgr2=0x7f4f040a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 -- 172.21.15.150:0/3399271959 shutdown_connections 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 1 -- 172.21.15.150:0/3399271959 wait complete. 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.396+0000 7f4f0c267700 10 monclient: monmap: 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:39.411 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 Processor -- start 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 -- start start 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient: init 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 0x7f4f0412c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-a con 0x7f4f040a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-c con 0x7f4f040a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 0x7f4f0412d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): picked mon.noname-b con 0x7f4f041379b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0409d790 con 0x7f4f040a0120 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0412fcf0 con 0x7f4f041379b0 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0409c110 con 0x7f4f040a0c00 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0c267700 10 monclient(hunting): authenticate will time out at 1041.498126s 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 0x7f4f0412d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 0x7f4f0412c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 0x7f4f0412c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:33912/0 (socket says 172.21.15.150:33912) 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 1 -- 172.21.15.150:0/2786416160 learned_addr learned my addr 172.21.15.150:0/2786416160 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): get_auth_request con 0x7f4f040a0120 auth_method 0 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): get_auth_request con 0x7f4f041379b0 auth_method 0 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 10 monclient(hunting): get_auth_request con 0x7f4f040a0c00 auth_method 0 2024-09-18T10:12:39.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a003700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 msgr2=0x7f4f0412cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 msgr2=0x7f4f0412cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f09802700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient(hunting): handle_auth_done global_id 14989 payload 824 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 msgr2=0x7f4f0412cc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 msgr2=0x7f4f0412c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 0x7f4f0412c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 monclient: found mon.noname-b 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.400+0000 7f4f0a804700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef8009420 con 0x7f4f041379b0 2024-09-18T10:12:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.404+0000 7f300661e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.404+0000 7f300661e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.404+0000 7f300661e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.404+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f300009e020 con 0x7f300009c3d0 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0a804700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 msgr2=0x7f4f0412d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0a804700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 0x7f4f0412d170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00d650 tx=0x7f4efc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc017070 con 0x7f4f041379b0 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:39.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: dump: 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 5 monclient: authenticate success, global_id 14989 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _renew_subs 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc012800 con 0x7f4f041379b0 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0412dfd0 con 0x7f4f041379b0 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc012bc0 con 0x7f4f041379b0 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 10 monclient: dump: 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.421 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f08800700 4 set_mon_vals no callback set 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _renew_subs 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0412fac0 con 0x7f4f041379b0 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4efc013480 con 0x7f4f041379b0 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f04063ed0 con 0x7f4f041379b0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 0x7f4ee8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 40..85) v4 ==== 6610+0+0 (secure 0 0 0) 0x7f4efc0485d0 con 0x7f4f041379b0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0a003700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 0x7f4ee8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.404+0000 7f4f0a003700 10 monclient: get_auth_request con 0x7f4ee8035ac0 auth_method 0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.408+0000 7f4f0a003700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 0x7f4ee8037f70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006f40 tx=0x7f4f00005480 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.412+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4efc010870 con 0x7f4f041379b0 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.412+0000 7f4ef67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.412+0000 7f4ef67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:39.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.412+0000 7f2ff57fa700 1 -- 172.21.15.142:0/2068140584 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f2ff0015030 con 0x7f300009c3d0 2024-09-18T10:12:39.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.412+0000 7f2ff57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:39.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.412+0000 7f2ff57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:39.424 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 msgr2=0x7f2fe4037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 0x7f2fe4037fc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0096d0 tx=0x7f2ffc008040 comp rx=0 tx=0).stop 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 10 monclient: shutdown 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 msgr2=0x7f300012a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 0x7f300012a360 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f30000a0330 tx=0x7f2ff0017000 comp rx=0 tx=0).stop 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 shutdown_connections 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2fe4035b10 0x7f2fe4037fc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f300012ec80 0x7f300012a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f300009c3d0 0x7f300012a360 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 --2- 172.21.15.142:0/2068140584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f300009bf90 0x7f3000129e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 >> 172.21.15.142:0/2068140584 conn(0x7f300006a2e0 msgr2=0x7f300012df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 shutdown_connections 2024-09-18T10:12:39.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:39.416+0000 7f300661e700 1 -- 172.21.15.142:0/2068140584 wait complete. 2024-09-18T10:12:39.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4f0c267700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:39.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4f0c267700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4f0c267700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4f0412ebe0 con 0x7f4f041379b0 2024-09-18T10:12:39.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v85) v1 ==== 56+0+1575 (secure 0 0 0) 0x7f4efc010640 con 0x7f4f041379b0 2024-09-18T10:12:39.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4ef67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.585 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.580+0000 7f4ef67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:39.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4f0c267700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:12:39.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4f0c267700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4f0c267700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f4f040008d0 con 0x7f4f041379b0 2024-09-18T10:12:39.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4ef67fc700 1 -- 172.21.15.150:0/2786416160 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v85) v1 ==== 56+0+1575 (secure 0 0 0) 0x7f4efc010870 con 0x7f4f041379b0 2024-09-18T10:12:39.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4ef67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:12:39.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.676+0000 7f4ef67fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:12:39.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 msgr2=0x7f4ee8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 0x7f4ee8037f70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4f00006f40 tx=0x7f4f00005480 comp rx=0 tx=0).stop 2024-09-18T10:12:39.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 10 monclient: shutdown 2024-09-18T10:12:39.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 msgr2=0x7f4f0412d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 0x7f4f0412d170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4efc00d650 tx=0x7f4efc008040 comp rx=0 tx=0).stop 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 shutdown_connections 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4ee8035ac0 0x7f4ee8037f70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f041379b0 0x7f4f0412d170 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f040a0c00 0x7f4f0412cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 --2- 172.21.15.150:0/2786416160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f040a0120 0x7f4f0412c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 >> 172.21.15.150:0/2786416160 conn(0x7f4f040a5830 msgr2=0x7f4f0409acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 shutdown_connections 2024-09-18T10:12:39.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.680+0000 7f4f0c267700 1 -- 172.21.15.150:0/2786416160 wait complete. 2024-09-18T10:12:39.701 INFO:tasks.workunit.client.0.smithi150.stdout:-14 0 root root-for-test 2024-09-18T10:12:39.701 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm host-for-test 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: get_monmap_and_config 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: monmap: 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.797 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:39.797080+0000 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:39.797080+0000 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:39.798 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 Processor -- start 2024-09-18T10:12:39.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 -- start start 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: init 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa1809d970 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa18131750 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7faa18135250 0x7faa18137630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-a con 0x7faa18135250 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1809b0a0 con 0x7faa1809d970 2024-09-18T10:12:39.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1809ab50 con 0x7faa18131750 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1809ae30 con 0x7faa18135250 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1edd0700 10 monclient(hunting): authenticate will time out at 1041.891189s 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa0ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7faa18131750 0x7faa18133b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33526/0 (socket says 172.21.15.150:33526) 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa0ffff700 1 -- 172.21.15.150:0/3041443317 learned_addr learned my addr 172.21.15.150:0/3041443317 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 1 --2- 172.21.15.150:0/3041443317 >> v2:172.21.15.142:3300/0 conn(0x7faa18135250 0x7faa18137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): get_auth_request con 0x7faa18135250 auth_method 0 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.792+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 10 monclient(hunting): handle_auth_done global_id 5256 payload 274 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 -- 172.21.15.150:0/3041443317 >> v1:172.21.15.150:6789/0 conn(0x7faa18131750 legacy=0x7faa18133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 -- 172.21.15.150:0/3041443317 >> v1:172.21.15.142:6789/0 conn(0x7faa1809d970 legacy=0x7faa180955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 monclient: found mon.noname-a 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:39.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 -- 172.21.15.150:0/3041443317 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18095cc0 con 0x7faa18135250 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 -- 172.21.15.150:0/3041443317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 msgr2=0x7faa18137630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1d36d700 1 --2- 172.21.15.150:0/3041443317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa18137630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faa1809a410 tx=0x7faa10009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0c7e106a1f93c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 1 -- 172.21.15.150:0/3041443317 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1000e070 con 0x7faa18135250 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: dump: 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.808 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.809 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 5 monclient: authenticate success, global_id 5256 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 1 -- 172.21.15.150:0/3041443317 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10012430 con 0x7faa18135250 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 1 -- 172.21.15.150:0/3041443317 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100127d0 con 0x7faa18135250 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 4 set_mon_vals no callback set 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa0f7fe700 10 monclient: dump: 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.810 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 10 monclient: shutdown 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:39.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 -- 172.21.15.150:0/3041443317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 msgr2=0x7faa18137630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 --2- 172.21.15.150:0/3041443317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa18137630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faa1809a410 tx=0x7faa10009780 comp rx=0 tx=0).stop 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 -- 172.21.15.150:0/3041443317 shutdown_connections 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 --2- 172.21.15.150:0/3041443317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa18137630 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 -- 172.21.15.150:0/3041443317 >> 172.21.15.150:0/3041443317 conn(0x7faa180a15e0 msgr2=0x7faa180a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 -- 172.21.15.150:0/3041443317 shutdown_connections 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.796+0000 7faa1edd0700 1 -- 172.21.15.150:0/3041443317 wait complete. 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient: build_initial_monmap 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient: monmap: 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:39.812 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 Processor -- start 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 -- start start 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient: init 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:39.813 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-c con 0x7faa1809d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-b con 0x7faa18131750 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa1806d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): picked mon.noname-a con 0x7faa18135250 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa18128040 con 0x7faa18135250 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa181277a0 con 0x7faa18131750 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:39.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa18127d60 con 0x7faa1809d970 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1edd0700 10 monclient(hunting): authenticate will time out at 1041.898399s 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59514/0 (socket says 172.21.15.150:59514) 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 -- 172.21.15.150:0/1456552209 learned_addr learned my addr 172.21.15.150:0/1456552209 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): get_auth_request con 0x7faa18131750 auth_method 0 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa1806d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 10 monclient(hunting): get_auth_request con 0x7faa1809d970 auth_method 0 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): get_auth_request con 0x7faa18135250 auth_method 0 2024-09-18T10:12:39.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1d36d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 msgr2=0x7faa181400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 msgr2=0x7faa181400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa1cb6c700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient(hunting): handle_auth_done global_id 14992 payload 824 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 msgr2=0x7faa181400a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:39.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 msgr2=0x7faa1806d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa1806d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 monclient: found mon.noname-b 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.800+0000 7faa0ffff700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009a20 con 0x7faa18131750 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0ffff700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 msgr2=0x7faa181405e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0ffff700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faa0800c610 tx=0x7faa08004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa08005d40 con 0x7faa18131750 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: dump: 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.820 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 5 monclient: authenticate success, global_id 14992 2024-09-18T10:12:39.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa080070d0 con 0x7faa18131750 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _renew_subs 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa18127a80 con 0x7faa18131750 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa08007490 con 0x7faa18131750 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 10 monclient: dump: 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.822 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0effd700 4 set_mon_vals no callback set 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _renew_subs 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa18140b20 con 0x7faa18131750 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7faa0801b3b0 con 0x7faa18131750 2024-09-18T10:12:39.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa0cff9700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 0x7faa04037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.804+0000 7faa1edd0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.808+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(85..85 src has 40..85) v4 ==== 6610+0+0 (secure 0 0 0) 0x7faa080480a0 con 0x7faa18131750 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.808+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18140cb0 con 0x7faa18131750 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.808+0000 7faa1cb6c700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 0x7faa04037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.808+0000 7faa1cb6c700 10 monclient: get_auth_request con 0x7faa04035ac0 auth_method 0 2024-09-18T10:12:39.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.808+0000 7faa1cb6c700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 0x7faa04037f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7faa14006df0 tx=0x7faa14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:39.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.816+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7faa08017020 con 0x7faa18131750 2024-09-18T10:12:39.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.816+0000 7faa0cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:39.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.816+0000 7faa0cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:39.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.988+0000 7faa1edd0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:39.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.988+0000 7faa1edd0700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:39.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.988+0000 7faa1edd0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:39.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:39.988+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7faa18131530 con 0x7faa18131750 2024-09-18T10:12:40.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.392+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 removed item id -11 name 'host-for-test' from crush map v86) v1 ==== 140+0+0 (secure 0 0 0) 0x7faa0800f840 con 0x7faa18131750 2024-09-18T10:12:40.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.392+0000 7faa0cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:40.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.392+0000 7faa0cff9700 10 monclient: _finish_command 2 = system:0 removed item id -11 name 'host-for-test' from crush map 2024-09-18T10:12:40.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.500+0000 7faa1edd0700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:40.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.500+0000 7faa1edd0700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:40.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.500+0000 7faa1edd0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.500+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host-for-test"} v 0) v1 -- 0x7faa180a0f90 con 0x7faa18131750 2024-09-18T10:12:40.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa0cff9700 1 -- 172.21.15.150:0/1456552209 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host-for-test"}]=0 device 'host-for-test' does not appear in the crush map v86) v1 ==== 140+0+0 (secure 0 0 0) 0x7faa08029090 con 0x7faa18131750 2024-09-18T10:12:40.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa0cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host-for-test"}] 2024-09-18T10:12:40.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa0cff9700 10 monclient: _finish_command 3 = system:0 device 'host-for-test' does not appear in the crush map 2024-09-18T10:12:40.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 msgr2=0x7faa04037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:40.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 0x7faa04037f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7faa14006df0 tx=0x7faa14008040 comp rx=0 tx=0).stop 2024-09-18T10:12:40.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.504+0000 7faa1edd0700 10 monclient: shutdown 2024-09-18T10:12:40.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:40.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 msgr2=0x7faa181405e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faa0800c610 tx=0x7faa08004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 shutdown_connections 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faa04035ac0 0x7faa04037f70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faa18135250 0x7faa1806d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa18131750 0x7faa181405e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 --2- 172.21.15.150:0/1456552209 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faa1809d970 0x7faa181400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 >> 172.21.15.150:0/1456552209 conn(0x7faa180a15e0 msgr2=0x7faa1809c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 shutdown_connections 2024-09-18T10:12:40.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.508+0000 7faa1edd0700 1 -- 172.21.15.150:0/1456552209 wait complete. 2024-09-18T10:12:40.514 INFO:tasks.workunit.client.0.smithi150.stderr:device 'host-for-test' does not appear in the crush map 2024-09-18T10:12:40.527 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm rack-for-test 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: get_monmap_and_config 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: build_initial_monmap 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: monmap: 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:40.629890+0000 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:40.629890+0000 2024-09-18T10:12:40.630 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:40.631 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 Processor -- start 2024-09-18T10:12:40.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 -- start start 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: init 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c0809c930 0x7f3c080977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-a con 0x7f3c0809c930 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c0809cd00 0x7f3c0809d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-c con 0x7f3c0809cd00 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c08136760 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c08098f00 con 0x7f3c08136760 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:40.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c08097d00 con 0x7f3c0809c930 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c08097030 con 0x7f3c0809cd00 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0ebdc700 10 monclient(hunting): authenticate will time out at 1042.723613s 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0d179700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3c08136760 0x7f3c08138b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33528/0 (socket says 172.21.15.150:33528) 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c0d179700 1 -- 172.21.15.150:0/3170940686 learned_addr learned my addr 172.21.15.150:0/3170940686 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c07fff700 1 --2- 172.21.15.150:0/3170940686 >> v2:172.21.15.150:3300/0 conn(0x7f3c0809cd00 0x7f3c0809d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c07fff700 10 monclient(hunting): get_auth_request con 0x7f3c0809cd00 auth_method 0 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c07fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c07fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:40.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.624+0000 7f3c07fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3170940686 >> v2:172.21.15.142:3300/0 conn(0x7f3c0809c930 0x7f3c080977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): get_auth_request con 0x7f3c0809c930 auth_method 0 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient(hunting): handle_auth_done global_id 5262 payload 274 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 -- 172.21.15.150:0/3170940686 >> v2:172.21.15.150:3300/0 conn(0x7f3c0809cd00 msgr2=0x7f3c0809d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:40.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3170940686 >> v2:172.21.15.150:3300/0 conn(0x7f3c0809cd00 0x7f3c0809d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 -- 172.21.15.150:0/3170940686 >> v1:172.21.15.150:6789/0 conn(0x7f3c08136760 legacy=0x7f3c08138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient: _finish_hunting 0 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 monclient: found mon.noname-a 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:40.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 -- 172.21.15.150:0/3170940686 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0806f730 con 0x7f3c0809c930 2024-09-18T10:12:40.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 -- 172.21.15.150:0/3170940686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c0809c930 msgr2=0x7f3c080977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:40.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3170940686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c0809c930 0x7f3c080977c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3c08071200 tx=0x7f3bf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b6afb96abb1426 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:40.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 1 -- 172.21.15.150:0/3170940686 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800b8c0 con 0x7f3c0809c930 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: dump: 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:40.641 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0ebdc700 5 monclient: authenticate success, global_id 5262 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0ebdc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 1 -- 172.21.15.150:0/3170940686 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800ba20 con 0x7f3c0809c930 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0ebdc700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c0ebdc700 4 set_mon_vals no callback set 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 1 -- 172.21.15.150:0/3170940686 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800bdc0 con 0x7f3c0809c930 2024-09-18T10:12:40.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.628+0000 7f3c077fe700 10 monclient: dump: 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:40.643 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: shutdown 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3170940686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c0809c930 msgr2=0x7f3c080977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3170940686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c0809c930 0x7f3c080977c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3c08071200 tx=0x7f3bf80091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3170940686 shutdown_connections 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3170940686 >> v2:172.21.15.150:3300/0 conn(0x7f3c0809cd00 0x7f3c0809d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3170940686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c0809c930 0x7f3c080977c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3170940686 >> 172.21.15.150:0/3170940686 conn(0x7f3c08068360 msgr2=0x7f3c080a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3170940686 shutdown_connections 2024-09-18T10:12:40.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3170940686 wait complete. 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: build_initial_monmap 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: monmap: 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:40.645 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 Processor -- start 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- start start 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: init 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c0809c930 0x7f3c08135750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-c con 0x7f3c0809c930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-b con 0x7f3c0809cd00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c08136760 0x7f3c0811f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): picked mon.noname-a con 0x7f3c08136760 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0809e3f0 con 0x7f3c08136760 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0809bb30 con 0x7f3c0809cd00 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08122900 con 0x7f3c0809c930 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0ebdc700 10 monclient(hunting): authenticate will time out at 1042.730682s 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59538/0 (socket says 172.21.15.150:59538) 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 -- 172.21.15.150:0/3091418078 learned_addr learned my addr 172.21.15.150:0/3091418078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:40.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): get_auth_request con 0x7f3c0809cd00 auth_method 0 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0d179700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c08136760 0x7f3c0811f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c0809c930 0x7f3c08135750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0c978700 10 monclient(hunting): get_auth_request con 0x7f3c0809c930 auth_method 0 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0c978700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0c978700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0c978700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0d179700 10 monclient(hunting): get_auth_request con 0x7f3c08136760 auth_method 0 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0d179700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0d179700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c0d179700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient(hunting): handle_auth_done global_id 14998 payload 824 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 -- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c0809c930 msgr2=0x7f3c08135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:40.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c0809c930 0x7f3c08135750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 -- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c08136760 msgr2=0x7f3c0811f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c08136760 0x7f3c0811f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 monclient: found mon.noname-b 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.632+0000 7f3c07fff700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8009420 con 0x7f3c0809cd00 2024-09-18T10:12:40.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c07fff700 1 -- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 msgr2=0x7f3c0811f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c07fff700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00d890 tx=0x7f3bfc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc012800 con 0x7f3c0809cd00 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: dump: 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.653 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:40.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 5 monclient: authenticate success, global_id 14998 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc012980 con 0x7f3c0809cd00 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc012d40 con 0x7f3c0809cd00 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _renew_subs 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08121250 con 0x7f3c0809cd00 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c06ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 10 monclient: dump: 2024-09-18T10:12:40.655 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _renew_subs 2024-09-18T10:12:40.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c08122e30 con 0x7f3c0809cd00 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c08122a90 con 0x7f3c0809cd00 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3bfc014640 con 0x7f3c0809cd00 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 0x7f3bf4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(86..86 src has 40..86) v4 ==== 6427+0+0 (secure 0 0 0) 0x7f3bfc049200 con 0x7f3c0809cd00 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.636+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 0x7f3bf4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.640+0000 7f3c0c978700 10 monclient: get_auth_request con 0x7f3bf4035ac0 auth_method 0 2024-09-18T10:12:40.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.640+0000 7f3c0c978700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 0x7f3bf4037f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80030c0 tx=0x7f3bf8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:40.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.648+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3bfc01c590 con 0x7f3c0809cd00 2024-09-18T10:12:40.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.648+0000 7f3c04ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:40.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.648+0000 7f3c04ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:40.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.820+0000 7f3c0ebdc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:40.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.820+0000 7f3c0ebdc700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:40.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.820+0000 7f3c0ebdc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:40.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:40.820+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f3c0804bbf0 con 0x7f3c0809cd00 2024-09-18T10:12:41.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.400+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 removed item id -12 name 'rack-for-test' from crush map v87) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3bfc011ae0 con 0x7f3c0809cd00 2024-09-18T10:12:41.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.400+0000 7f3c04ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:41.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.400+0000 7f3c04ff9700 10 monclient: _finish_command 2 = system:0 removed item id -12 name 'rack-for-test' from crush map 2024-09-18T10:12:41.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c0ebdc700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:41.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c0ebdc700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:41.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c0ebdc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "rack-for-test"} v 0) v1 -- 0x7f3c08063ed0 con 0x7f3c0809cd00 2024-09-18T10:12:41.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c04ff9700 1 -- 172.21.15.150:0/3091418078 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "rack-for-test"}]=0 device 'rack-for-test' does not appear in the crush map v87) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3bfc01c360 con 0x7f3c0809cd00 2024-09-18T10:12:41.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c04ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "rack-for-test"}] 2024-09-18T10:12:41.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.508+0000 7f3c04ff9700 10 monclient: _finish_command 3 = system:0 device 'rack-for-test' does not appear in the crush map 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 msgr2=0x7f3bf4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 0x7f3bf4037f70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80030c0 tx=0x7f3bf8016000 comp rx=0 tx=0).stop 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 10 monclient: shutdown 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 msgr2=0x7f3c0811f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00d890 tx=0x7f3bfc008040 comp rx=0 tx=0).stop 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 shutdown_connections 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3bf4035ac0 0x7f3bf4037f70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c08136760 0x7f3c0811f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c0809cd00 0x7f3c0811f290 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 --2- 172.21.15.150:0/3091418078 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c0809c930 0x7f3c08135750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 >> 172.21.15.150:0/3091418078 conn(0x7f3c08068360 msgr2=0x7f3c0809a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:41.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 shutdown_connections 2024-09-18T10:12:41.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.512+0000 7f3c0ebdc700 1 -- 172.21.15.150:0/3091418078 wait complete. 2024-09-18T10:12:41.519 INFO:tasks.workunit.client.0.smithi150.stderr:device 'rack-for-test' does not appear in the crush map 2024-09-18T10:12:41.527 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm root-for-test 2024-09-18T10:12:41.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:41.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: build_initial_monmap 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: monmap: 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:41.628547+0000 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:41.628547+0000 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:41.629 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:41.630 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 Processor -- start 2024-09-18T10:12:41.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 -- start start 2024-09-18T10:12:41.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: init 2024-09-18T10:12:41.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:41.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41ac098fd0 0x7f41ac0a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-c con 0x7f41ac098fd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41ac09e530 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-b con 0x7f41ac09d2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac0993a0 con 0x7f41ac09e530 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac09db50 con 0x7f41ac098fd0 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac09d870 con 0x7f41ac09d2c0 2024-09-18T10:12:41.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41b1c9a700 10 monclient(hunting): authenticate will time out at 1043.722457s 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41ac098fd0 0x7f41ac0a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41ac098fd0 0x7f41ac0a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59542/0 (socket says 172.21.15.150:59542) 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 -- 172.21.15.150:0/1260884823 learned_addr learned my addr 172.21.15.150:0/1260884823 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): get_auth_request con 0x7f41ac098fd0 auth_method 0 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:41.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 10 monclient(hunting): get_auth_request con 0x7f41ac09d2c0 auth_method 0 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 1 -- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 1 -- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41abfff700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:41.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_done global_id 15001 payload 274 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 -- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac139a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 -- 172.21.15.150:0/1260884823 >> v1:172.21.15.142:6789/0 conn(0x7f41ac09e530 legacy=0x7f41ac0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.624+0000 7f41ab7fe700 1 -- 172.21.15.150:0/1260884823 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac09de30 con 0x7f41ac098fd0 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41ab7fe700 1 -- 172.21.15.150:0/1260884823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac098fd0 msgr2=0x7f41ac0a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41ab7fe700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac098fd0 0x7f41ac0a02f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41940099b0 tx=0x7f4194009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7d5a325766b79b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 1 -- 172.21.15.150:0/1260884823 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419400ba70 con 0x7f41ac098fd0 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: dump: 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:41.642 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 5 monclient: authenticate success, global_id 15001 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 1 -- 172.21.15.150:0/1260884823 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419400bbd0 con 0x7f41ac098fd0 2024-09-18T10:12:41.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 4 set_mon_vals no callback set 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 1 -- 172.21.15.150:0/1260884823 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194011420 con 0x7f41ac098fd0 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41aa7fc700 10 monclient: dump: 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.644 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 10 monclient: shutdown 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 -- 172.21.15.150:0/1260884823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac098fd0 msgr2=0x7f41ac0a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac098fd0 0x7f41ac0a02f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f41940099b0 tx=0x7f4194009250 comp rx=0 tx=0).stop 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 -- 172.21.15.150:0/1260884823 shutdown_connections 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/1260884823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac098fd0 0x7f41ac0a02f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 -- 172.21.15.150:0/1260884823 >> 172.21.15.150:0/1260884823 conn(0x7f41ac0a5900 msgr2=0x7f41ac0a5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 -- 172.21.15.150:0/1260884823 shutdown_connections 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 1 -- 172.21.15.150:0/1260884823 wait complete. 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 10 monclient: build_initial_monmap 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.628+0000 7f41b1c9a700 10 monclient: monmap: 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:41.646 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 Processor -- start 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 -- start start 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient: init 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f41ac098fd0 0x7f41ac141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-a con 0x7f41ac098fd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-c con 0x7f41ac09d2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): picked mon.noname-b con 0x7f41ac09e530 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:41.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac090c90 con 0x7f41ac098fd0 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0984d0 con 0x7f41ac09e530 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac0a4290 con 0x7f41ac09d2c0 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41b1c9a700 10 monclient(hunting): authenticate will time out at 1043.728679s 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59554/0 (socket says 172.21.15.150:59554) 2024-09-18T10:12:41.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 -- 172.21.15.150:0/762932079 learned_addr learned my addr 172.21.15.150:0/762932079 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): get_auth_request con 0x7f41ac09e530 auth_method 0 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f41ac098fd0 0x7f41ac141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): get_auth_request con 0x7f41ac098fd0 auth_method 0 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:41.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 10 monclient(hunting): get_auth_request con 0x7f41ac09d2c0 auth_method 0 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:41.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient(hunting): handle_auth_done global_id 15004 payload 824 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 msgr2=0x7f41ac142220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41aaffd700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f41ac098fd0 msgr2=0x7f41ac141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f41ac098fd0 0x7f41ac141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 monclient: found mon.noname-b 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41940094f0 con 0x7f41ac09e530 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 msgr2=0x7f41ac08d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.632+0000 7f41abfff700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007ef0 tx=0x7f41a4009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:41.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a400ee50 con 0x7f41ac09e530 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: dump: 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:41.652 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 5 monclient: authenticate success, global_id 15004 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _renew_subs 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4014420 con 0x7f41ac09e530 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac08f750 con 0x7f41ac09e530 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _renew_subs 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac090640 con 0x7f41ac09e530 2024-09-18T10:12:41.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40147e0 con 0x7f41ac09e530 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a9ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 10 monclient: dump: 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:41.654 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f41a4014b90 con 0x7f41ac09e530 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 0x7f418c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(87..87 src has 40..87) v4 ==== 6244+0+0 (secure 0 0 0) 0x7f41a4048e10 con 0x7f41ac09e530 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41ab7fe700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 0x7f418c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:41.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.636+0000 7f41ab7fe700 10 monclient: get_auth_request con 0x7f418c035af0 auth_method 0 2024-09-18T10:12:41.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.640+0000 7f41ab7fe700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 0x7f418c037fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f41940099b0 tx=0x7f4194016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:41.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.648+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:41.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.648+0000 7f41a37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:41.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.648+0000 7f41a37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:41.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.808+0000 7f41b1c9a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:41.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.808+0000 7f41b1c9a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:41.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.808+0000 7f41b1c9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:41.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:41.808+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:42.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.408+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 removed item id -14 name 'root-for-test' from crush map v88) v1 ==== 140+0+0 (secure 0 0 0) 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:42.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.408+0000 7f41a37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:42.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.408+0000 7f41a37fe700 10 monclient: _finish_command 2 = system:0 removed item id -14 name 'root-for-test' from crush map 2024-09-18T10:12:42.434 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:42.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.520+0000 7f41b1c9a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:42.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.520+0000 7f41b1c9a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:42.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.520+0000 7f41b1c9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.520+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "root-for-test"} v 0) v1 -- 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:42.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41a37fe700 1 -- 172.21.15.150:0/762932079 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "root-for-test"}]=0 device 'root-for-test' does not appear in the crush map v88) v1 ==== 140+0+0 (secure 0 0 0) 0x7f41ac08ec40 con 0x7f41ac09e530 2024-09-18T10:12:42.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41a37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "root-for-test"}] 2024-09-18T10:12:42.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41a37fe700 10 monclient: _finish_command 3 = system:0 device 'root-for-test' does not appear in the crush map 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 msgr2=0x7f418c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 0x7f418c037fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f41940099b0 tx=0x7f4194016000 comp rx=0 tx=0).stop 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 10 monclient: shutdown 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 msgr2=0x7f41ac08d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007ef0 tx=0x7f41a4009870 comp rx=0 tx=0).stop 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 shutdown_connections 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f418c035af0 0x7f418c037fa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41ac09e530 0x7f41ac08d080 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f41ac09d2c0 0x7f41ac142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 --2- 172.21.15.150:0/762932079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f41ac098fd0 0x7f41ac141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 >> 172.21.15.150:0/762932079 conn(0x7f41ac0a5900 msgr2=0x7f41ac09e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 shutdown_connections 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.524+0000 7f41b1c9a700 1 -- 172.21.15.150:0/762932079 wait complete. 2024-09-18T10:12:42.532 INFO:tasks.workunit.client.0.smithi150.stderr:device 'root-for-test' does not appear in the crush map 2024-09-18T10:12:42.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule create-simple foo-rule foo host firstn 2024-09-18T10:12:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: get_monmap_and_config 2024-09-18T10:12:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: build_initial_monmap 2024-09-18T10:12:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: monmap: 2024-09-18T10:12:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:42.544805+0000 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:42.544805+0000 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:42.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 Processor -- start 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 -- start start 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: init 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 0x7f5998098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-a con 0x7f599809a7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5998099220 0x7f599809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5998099220 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f599809c320 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599809abc0 con 0x7f599809c320 2024-09-18T10:12:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599809d950 con 0x7f599809a7f0 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f599809db90 con 0x7f5998099220 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5997fff700 10 monclient(hunting): authenticate will time out at 1051.470358s 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 0x7f5998098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59977fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f599809c320 0x7f599813d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37510/0 (socket says 172.21.15.142:37510) 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 0x7f5998098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53028/0 (socket says 172.21.15.142:53028) 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59977fe700 1 -- 172.21.15.142:0/3157979388 learned_addr learned my addr 172.21.15.142:0/3157979388 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): get_auth_request con 0x7f599809a7f0 auth_method 0 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 --2- 172.21.15.142:0/3157979388 >> v2:172.21.15.150:3300/0 conn(0x7f5998099220 0x7f599809bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): get_auth_request con 0x7f5998099220 auth_method 0 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient(hunting): handle_auth_done global_id 15007 payload 274 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 -- 172.21.15.142:0/3157979388 >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 msgr2=0x7f5998098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 --2- 172.21.15.142:0/3157979388 >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 0x7f5998098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 -- 172.21.15.142:0/3157979388 >> v1:172.21.15.142:6789/0 conn(0x7f599809c320 legacy=0x7f599813d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient: _finish_hunting 0 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 monclient: found mon.noname-c 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.540+0000 7f59967fc700 1 -- 172.21.15.142:0/3157979388 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998099a30 con 0x7f5998099220 2024-09-18T10:12:42.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f59967fc700 1 -- 172.21.15.142:0/3157979388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 msgr2=0x7f599809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:42.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f59967fc700 1 --2- 172.21.15.142:0/3157979388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599809bde0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f599809d6b0 tx=0x7f5988009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60cde0755591ea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 1 -- 172.21.15.142:0/3157979388 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800e070 con 0x7f5998099220 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: dump: 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.558 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 1 -- 172.21.15.142:0/3157979388 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598800bd70 con 0x7f5998099220 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 5 monclient: authenticate success, global_id 15007 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 1 -- 172.21.15.142:0/3157979388 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988012680 con 0x7f5998099220 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 4 set_mon_vals no callback set 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5995ffb700 10 monclient: dump: 2024-09-18T10:12:42.559 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 10 monclient: shutdown 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 -- 172.21.15.142:0/3157979388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 msgr2=0x7f599809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 --2- 172.21.15.142:0/3157979388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599809bde0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f599809d6b0 tx=0x7f5988009780 comp rx=0 tx=0).stop 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 -- 172.21.15.142:0/3157979388 shutdown_connections 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 --2- 172.21.15.142:0/3157979388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599809bde0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 --2- 172.21.15.142:0/3157979388 >> v2:172.21.15.142:3300/0 conn(0x7f599809a7f0 0x7f5998098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 -- 172.21.15.142:0/3157979388 >> 172.21.15.142:0/3157979388 conn(0x7f59980a1fe0 msgr2=0x7f5998070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 -- 172.21.15.142:0/3157979388 shutdown_connections 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.544+0000 7f5997fff700 1 -- 172.21.15.142:0/3157979388 wait complete. 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient: build_initial_monmap 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient: monmap: 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:42.561 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 Processor -- start 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 -- start start 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient: init 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5998099220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:42.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-c con 0x7f599809a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f599809c320 0x7f599806ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): picked mon.noname-a con 0x7f599809c320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599806c950 con 0x7f599809c320 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5998094330 con 0x7f5998099220 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599806c670 con 0x7f599809a7f0 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5997fff700 10 monclient(hunting): authenticate will time out at 1051.478485s 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40664/0 (socket says 172.21.15.142:40664) 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:52206/0 (socket says 172.21.15.142:52206) 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 -- 172.21.15.142:0/1906428569 learned_addr learned my addr 172.21.15.142:0/1906428569 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 10 monclient(hunting): get_auth_request con 0x7f599809a7f0 auth_method 0 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): get_auth_request con 0x7f5998099220 auth_method 0 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f599809c320 0x7f599806ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 msgr2=0x7f599806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 msgr2=0x7f599806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:42.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): get_auth_request con 0x7f599809c320 auth_method 0 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59967fc700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f59977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient(hunting): handle_auth_done global_id 15010 payload 824 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 msgr2=0x7f599806a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f599809c320 msgr2=0x7f599806ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f599809c320 0x7f599806ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:42.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 monclient: found mon.noname-b 2024-09-18T10:12:42.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:42.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.548+0000 7f5996ffd700 1 -- 172.21.15.142:0/1906428569 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5988009a20 con 0x7f5998099220 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5996ffd700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 msgr2=0x7f599806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5996ffd700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f598c00ea50 tx=0x7f598c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c0095d0 con 0x7f5998099220 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: dump: 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: _finish_auth 0 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 5 monclient: authenticate success, global_id 15010 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _renew_subs 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599806cc30 con 0x7f5998099220 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598c00ca00 con 0x7f5998099220 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:42.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c00cdc0 con 0x7f5998099220 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f597effd700 10 monclient: dump: 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f59957fa700 4 set_mon_vals no callback set 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _renew_subs 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59980939e0 con 0x7f5998099220 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.552+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998094840 con 0x7f5998099220 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f598c013680 con 0x7f5998099220 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f597effd700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 0x7f5984037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(88..88 src has 40..88) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f598c048bf0 con 0x7f5998099220 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f59967fc700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 0x7f5984037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f59967fc700 10 monclient: get_auth_request con 0x7f5984035ac0 auth_method 0 2024-09-18T10:12:42.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.556+0000 7f59967fc700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 0x7f5984037f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59880099a0 tx=0x7f5988018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.564+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f598c023ad0 con 0x7f5998099220 2024-09-18T10:12:42.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.564+0000 7f597effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.564+0000 7f597effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: get_monmap_and_config 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: build_initial_monmap 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: monmap: 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.648 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:42.648155+0000 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:42.648155+0000 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:42.649 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 Processor -- start 2024-09-18T10:12:42.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 -- start start 2024-09-18T10:12:42.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: init 2024-09-18T10:12:42.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:42.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95901328b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9590098070 0x7f9590137700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-a con 0x7f9590098070 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f959009db20 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959009d6c0 con 0x7f95901328b0 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900970c0 con 0x7f959009db20 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9590133390 con 0x7f9590098070 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9597a9c700 10 monclient(hunting): authenticate will time out at 1044.743052s 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9596039700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f959009db20 0x7f9590139c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33538/0 (socket says 172.21.15.150:33538) 2024-09-18T10:12:42.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9596039700 1 -- 172.21.15.150:0/440616947 learned_addr learned my addr 172.21.15.150:0/440616947 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 1 --2- 172.21.15.150:0/440616947 >> v2:172.21.15.142:3300/0 conn(0x7f9590098070 0x7f9590137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): get_auth_request con 0x7f9590098070 auth_method 0 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient(hunting): handle_auth_done global_id 5277 payload 274 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 1 -- 172.21.15.150:0/440616947 >> v1:172.21.15.150:6789/0 conn(0x7f959009db20 legacy=0x7f9590139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:42.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 1 -- 172.21.15.150:0/440616947 >> v1:172.21.15.142:6789/0 conn(0x7f95901328b0 legacy=0x7f9590132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient: _finish_hunting 0 2024-09-18T10:12:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 1 monclient: found mon.noname-a 2024-09-18T10:12:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.644+0000 7f9595037700 1 -- 172.21.15.150:0/440616947 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590096de0 con 0x7f9590098070 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9595037700 1 -- 172.21.15.150:0/440616947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9590098070 msgr2=0x7f9590137700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9595037700 1 --2- 172.21.15.150:0/440616947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9590098070 0x7f9590137700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9590097250 tx=0x7f95800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d376f67b01403d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 1 -- 172.21.15.150:0/440616947 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000b9a0 con 0x7f9590098070 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: dump: 2024-09-18T10:12:42.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: _finish_auth 0 2024-09-18T10:12:42.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 5 monclient: authenticate success, global_id 5277 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 1 -- 172.21.15.150:0/440616947 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000bb00 con 0x7f9590098070 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 4 set_mon_vals no callback set 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 1 -- 172.21.15.150:0/440616947 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000bea0 con 0x7f9590098070 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:42.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9594836700 10 monclient: dump: 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 10 monclient: shutdown 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:42.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 1 -- 172.21.15.150:0/440616947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9590098070 msgr2=0x7f9590137700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.648+0000 7f9597a9c700 1 --2- 172.21.15.150:0/440616947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9590098070 0x7f9590137700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9590097250 tx=0x7f95800091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- 172.21.15.150:0/440616947 shutdown_connections 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 --2- 172.21.15.150:0/440616947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9590098070 0x7f9590137700 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- 172.21.15.150:0/440616947 >> 172.21.15.150:0/440616947 conn(0x7f95900a1cc0 msgr2=0x7f95900a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- 172.21.15.150:0/440616947 shutdown_connections 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- 172.21.15.150:0/440616947 wait complete. 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient: build_initial_monmap 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient: monmap: 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:42.661 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:42.662 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.662 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:42.662 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:42.662 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:42.662 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 Processor -- start 2024-09-18T10:12:42.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- start start 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient: init 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-c con 0x7f9590098070 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-a con 0x7f959009db20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): picked mon.noname-b con 0x7f95901328b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959011e500 con 0x7f959009db20 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959009e000 con 0x7f95901328b0 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590142910 con 0x7f9590098070 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9597a9c700 10 monclient(hunting): authenticate will time out at 1044.751365s 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9596039700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:33984/0 (socket says 172.21.15.150:33984) 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 1 -- 172.21.15.150:0/1469308529 learned_addr learned my addr 172.21.15.150:0/1469308529 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): get_auth_request con 0x7f959009db20 auth_method 0 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9596039700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 msgr2=0x7f959011c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9596039700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 msgr2=0x7f959011c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 10 monclient(hunting): get_auth_request con 0x7f9590098070 auth_method 0 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9596039700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9596039700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:42.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595037700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 msgr2=0x7f9590141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 msgr2=0x7f9590141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.652+0000 7f9595838700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 10 monclient(hunting): handle_auth_done global_id 5280 payload 824 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 msgr2=0x7f9590141a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 msgr2=0x7f959011c3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 10 monclient: _finish_hunting 0 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 monclient: found mon.noname-a 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:42.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009420 con 0x7f959009db20 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 msgr2=0x7f9590131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9595037700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f958000b0e0 tx=0x7f9580016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580006af0 con 0x7f959009db20 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: dump: 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.671 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 5 monclient: authenticate success, global_id 5280 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9580020070 con 0x7f959009db20 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 10 monclient: _renew_subs 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959011eb50 con 0x7f959009db20 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958001b6a0 con 0x7f959009db20 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:42.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9585ffb700 10 monclient: dump: 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9587fff700 4 set_mon_vals no callback set 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 10 monclient: _renew_subs 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.656+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959011fd20 con 0x7f959009db20 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9580022030 con 0x7f959009db20 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9597a9c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9597a9c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9597a9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959009d8c0 con 0x7f959009db20 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9585ffb700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 0x7f9574037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(88..88 src has 40..88) v4 ==== 6085+0+0 (secure 0 0 0) 0x7f958004a740 con 0x7f959009db20 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9595838700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 0x7f9574037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:42.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9595838700 10 monclient: get_auth_request con 0x7f9574035ac0 auth_method 0 2024-09-18T10:12:42.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.660+0000 7f9595838700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 0x7f9574037f70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f958c007cb0 tx=0x7f958c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:42.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.668+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f958001c450 con 0x7f959009db20 2024-09-18T10:12:42.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.668+0000 7f9585ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:42.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.668+0000 7f9585ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f5997fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f5997fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f5997fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5998072a80 con 0x7f5998099220 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f597effd700 1 -- 172.21.15.142:0/1906428569 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f598c021020 con 0x7f5998099220 2024-09-18T10:12:42.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f597effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:42.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.792+0000 7f597effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:42.800 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 msgr2=0x7f5984037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 0x7f5984037f70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f59880099a0 tx=0x7f5988018000 comp rx=0 tx=0).stop 2024-09-18T10:12:42.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 10 monclient: shutdown 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 msgr2=0x7f599806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f598c00ea50 tx=0x7f598c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 shutdown_connections 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5984035ac0 0x7f5984037f70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f599809c320 0x7f599806ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f599809a7f0 0x7f599806a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 --2- 172.21.15.142:0/1906428569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5998099220 0x7f599806a2b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 >> 172.21.15.142:0/1906428569 conn(0x7f59980a1fe0 msgr2=0x7f599809f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 shutdown_connections 2024-09-18T10:12:42.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:42.796+0000 7f5997fff700 1 -- 172.21.15.142:0/1906428569 wait complete. 2024-09-18T10:12:42.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.836+0000 7f9597a9c700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:42.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.836+0000 7f9597a9c700 10 monclient: _send_command 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:42.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.836+0000 7f9597a9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:42.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:42.836+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f959009d8c0 con 0x7f959009db20 2024-09-18T10:12:43.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.420+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 v89) v1 ==== 145+0+0 (secure 0 0 0) 0x7f958001a3d0 con 0x7f959009db20 2024-09-18T10:12:43.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.420+0000 7f9585ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:43.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.420+0000 7f9585ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:43.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9597a9c700 10 start_mon_command cmd=[{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:43.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9597a9c700 10 monclient: _send_command 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:43.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9597a9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:43.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"} v 0) v1 -- 0x7f959009d8c0 con 0x7f959009db20 2024-09-18T10:12:43.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9585ffb700 1 -- 172.21.15.150:0/1469308529 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}]=0 rule foo-rule already exists v89) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9580010380 con 0x7f959009db20 2024-09-18T10:12:43.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9585ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule create-simple", "name": "foo-rule", "root": "foo", "type": "host", "mode": "firstn"}] 2024-09-18T10:12:43.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.524+0000 7f9585ffb700 10 monclient: _finish_command 3 = system:0 rule foo-rule already exists 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 msgr2=0x7f9574037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 0x7f9574037f70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f958c007cb0 tx=0x7f958c009040 comp rx=0 tx=0).stop 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 10 monclient: shutdown 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 msgr2=0x7f9590131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f958000b0e0 tx=0x7f9580016000 comp rx=0 tx=0).stop 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 shutdown_connections 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9574035ac0 0x7f9574037f70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95901328b0 0x7f959011c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f959009db20 0x7f9590131910 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 --2- 172.21.15.150:0/1469308529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9590098070 0x7f9590141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 >> 172.21.15.150:0/1469308529 conn(0x7f95900a1cc0 msgr2=0x7f9590099810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 shutdown_connections 2024-09-18T10:12:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.528+0000 7f9597a9c700 1 -- 172.21.15.150:0/1469308529 wait complete. 2024-09-18T10:12:43.534 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo-rule already exists 2024-09-18T10:12:43.543 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush rm foo 2024-09-18T10:12:43.543 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:12:43.543 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm foo 2024-09-18T10:12:43.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 10 monclient: get_monmap_and_config 2024-09-18T10:12:43.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 10 monclient: build_initial_monmap 2024-09-18T10:12:43.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 10 monclient: monmap: 2024-09-18T10:12:43.643 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:43.643308+0000 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:43.643308+0000 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:43.644 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 1 Processor -- start 2024-09-18T10:12:43.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 1 -- start start 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.636+0000 7fb2f3c17700 10 monclient: init 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb2ec09d970 0x7fb2ec0955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec09d970 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2ec098ac0 0x7fb2ec098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec098ac0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2ec0993d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec09b0a0 con 0x7fb2ec0993d0 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec09ab50 con 0x7fb2ec09d970 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec09ae30 con 0x7fb2ec098ac0 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient(hunting): authenticate will time out at 1045.736699s 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2ec098ac0 0x7fb2ec098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2ec098ac0 0x7fb2ec098e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59568/0 (socket says 172.21.15.150:59568) 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 -- 172.21.15.150:0/4199024370 learned_addr learned my addr 172.21.15.150:0/4199024370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): get_auth_request con 0x7fb2ec098ac0 auth_method 0 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:43.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f19b3700 1 --2- 172.21.15.150:0/4199024370 >> v2:172.21.15.142:3300/0 conn(0x7fb2ec09d970 0x7fb2ec0955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f19b3700 10 monclient(hunting): get_auth_request con 0x7fb2ec09d970 auth_method 0 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f19b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f19b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f19b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient(hunting): handle_auth_done global_id 15013 payload 274 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 -- 172.21.15.150:0/4199024370 >> v2:172.21.15.142:3300/0 conn(0x7fb2ec09d970 msgr2=0x7fb2ec0955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/4199024370 >> v2:172.21.15.142:3300/0 conn(0x7fb2ec09d970 0x7fb2ec0955b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 -- 172.21.15.150:0/4199024370 >> v1:172.21.15.142:6789/0 conn(0x7fb2ec0993d0 legacy=0x7fb2ec135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient: _finish_hunting 0 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 monclient: found mon.noname-c 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 -- 172.21.15.150:0/4199024370 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec09dd40 con 0x7fb2ec098ac0 2024-09-18T10:12:43.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 -- 172.21.15.150:0/4199024370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 msgr2=0x7fb2ec098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:43.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/4199024370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec098e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec09a410 tx=0x7fb2e8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=971901d6792db209 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:43.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 1 -- 172.21.15.150:0/4199024370 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800ba70 con 0x7fb2ec098ac0 2024-09-18T10:12:43.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:43.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: dump: 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: _finish_auth 0 2024-09-18T10:12:43.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 1 -- 172.21.15.150:0/4199024370 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e800bbd0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 5 monclient: authenticate success, global_id 15013 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 1 -- 172.21.15.150:0/4199024370 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8011420 con 0x7fb2ec098ac0 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f3c17700 4 set_mon_vals no callback set 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.640+0000 7fb2f09b1700 10 monclient: dump: 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.655 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 10 monclient: shutdown 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- 172.21.15.150:0/4199024370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 msgr2=0x7fb2ec098e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 --2- 172.21.15.150:0/4199024370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec098e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec09a410 tx=0x7fb2e8009250 comp rx=0 tx=0).stop 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- 172.21.15.150:0/4199024370 shutdown_connections 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 --2- 172.21.15.150:0/4199024370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec098e90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 --2- 172.21.15.150:0/4199024370 >> v2:172.21.15.142:3300/0 conn(0x7fb2ec09d970 0x7fb2ec0955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- 172.21.15.150:0/4199024370 >> 172.21.15.150:0/4199024370 conn(0x7fb2ec0a15e0 msgr2=0x7fb2ec0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- 172.21.15.150:0/4199024370 shutdown_connections 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- 172.21.15.150:0/4199024370 wait complete. 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 10 monclient: build_initial_monmap 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 10 monclient: monmap: 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:43.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 Processor -- start 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 1 -- start start 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.644+0000 7fb2f3c17700 10 monclient: init 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec13ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-b con 0x7fb2ec098ac0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec0993d0 0x7fb2ec094450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec0993d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 0x7fb2ec08b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec09d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:43.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec08d090 con 0x7fb2ec0993d0 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec08d6e0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec098450 con 0x7fb2ec09d970 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient(hunting): authenticate will time out at 1045.744352s 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec13ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec13ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59578/0 (socket says 172.21.15.150:59578) 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 -- 172.21.15.150:0/2229232864 learned_addr learned my addr 172.21.15.150:0/2229232864 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): get_auth_request con 0x7fb2ec098ac0 auth_method 0 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 0x7fb2ec08b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec0993d0 0x7fb2ec094450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 10 monclient(hunting): get_auth_request con 0x7fb2ec09d970 auth_method 0 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 10 monclient(hunting): get_auth_request con 0x7fb2ec0993d0 auth_method 0 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient(hunting): handle_auth_done global_id 15016 payload 824 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 -- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 msgr2=0x7fb2ec08b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 1 -- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 msgr2=0x7fb2ec08b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 1 -- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 msgr2=0x7fb2ec08b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 0x7fb2ec08b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f21b4700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 0x7fb2ec08b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2ec09d970 0x7fb2ec08b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 -- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec0993d0 msgr2=0x7fb2ec094450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec0993d0 0x7fb2ec094450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient: _finish_hunting 0 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 monclient: found mon.noname-b 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:43.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 -- 172.21.15.150:0/2229232864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80094f0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec0993d0 0x7fb2ec094450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:43.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 -- 172.21.15.150:0/2229232864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 msgr2=0x7fb2ec13ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:43.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f19b3700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2ec098ac0 0x7fb2ec13ea10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc007fa0 tx=0x7fb2dc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:43.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc011940 con 0x7fb2ec098ac0 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: dump: 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:43.664 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 5 monclient: authenticate success, global_id 15016 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _renew_subs 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2dc011aa0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 -- 172.21.15.150:0/2229232864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec08e640 con 0x7fb2ec098ac0 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc011e40 con 0x7fb2ec098ac0 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:43.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e1ffb700 10 monclient: dump: 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2e3fff700 4 set_mon_vals no callback set 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _renew_subs 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.648+0000 7fb2f3c17700 1 -- 172.21.15.150:0/2229232864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec08cb10 con 0x7fb2ec098ac0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb2dc010650 con 0x7fb2ec098ac0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2e1ffb700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb2d0035ac0 0x7fb2d0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f3c17700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f3c17700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f3c17700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f3c17700 1 -- 172.21.15.150:0/2229232864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec063ed0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(89..89 src has 40..89) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fb2dc04aa60 con 0x7fb2ec098ac0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb2d0035ac0 0x7fb2d0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.652+0000 7fb2f11b2700 10 monclient: get_auth_request con 0x7fb2d0035ac0 auth_method 0 2024-09-18T10:12:43.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.656+0000 7fb2f11b2700 1 --2- 172.21.15.150:0/2229232864 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb2d0035ac0 0x7fb2d0037f70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec09a410 tx=0x7fb2e8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:43.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.660+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb2dc020070 con 0x7fb2ec098ac0 2024-09-18T10:12:43.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.660+0000 7fb2e1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:43.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.660+0000 7fb2e1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:43.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2f3c17700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:43.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2f3c17700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:43.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2f3c17700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:43.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2f3c17700 1 -- 172.21.15.150:0/2229232864 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7fb2ec0008d0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2e1ffb700 1 -- 172.21.15.150:0/2229232864 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=-16 (16) Device or resource busy v89) v1 ==== 103+0+0 (secure 0 0 0) 0x7fb2dc023ae0 con 0x7fb2ec098ac0 2024-09-18T10:12:43.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2e1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:43.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.832+0000 7fb2e1ffb700 10 monclient: _finish_command 2 = mon:16 (16) Device or resource busy 2024-09-18T10:12:43.841 INFO:tasks.workunit.client.0.smithi150.stderr:Error EBUSY: (16) Device or resource busy 2024-09-18T10:12:43.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:12:43.845 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rule rm foo-rule 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.940+0000 7f6ec436e700 10 monclient: get_monmap_and_config 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.940+0000 7f6ec436e700 10 monclient: build_initial_monmap 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.940+0000 7f6ec436e700 10 monclient: monmap: 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:43.947761+0000 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:43.947761+0000 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:43.948 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:43.949 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:43.949 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:43.949 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:43.949 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 Processor -- start 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 -- start start 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient: init 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-18T10:12:43.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ebc09e6d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-c con 0x7f6ebc0a09a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-a con 0x7f6ebc0a0da0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc09e220 con 0x7f6ebc09e6d0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc09ee30 con 0x7f6ebc0a0da0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc09eaa0 con 0x7f6ebc0a09a0 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec436e700 10 monclient(hunting): authenticate will time out at 1046.041400s 2024-09-18T10:12:43.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59584/0 (socket says 172.21.15.150:59584) 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 learned_addr learned my addr 172.21.15.150:0/1263726428 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a09a0 auth_method 0 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a0da0 auth_method 0 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:43.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.944+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:43.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:43.948+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> v1:172.21.15.142:6789/0 conn(0x7f6ebc09e6d0 legacy=0x7f6ebc0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a09a0 auth_method 0 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:44.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a0da0 auth_method 0 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:44.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.148+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:44.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:44.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a09a0 auth_method 0 2024-09-18T10:12:44.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:44.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a0da0 auth_method 0 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:44.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:44.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:44.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:44.552+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a09a0 auth_method 0 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:45.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ebc0a0da0 auth_method 0 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:45.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:45.356+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:45.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:45.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: get_monmap_and_config 2024-09-18T10:12:45.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: build_initial_monmap 2024-09-18T10:12:45.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: monmap: 2024-09-18T10:12:45.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:45.928649+0000 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:45.928649+0000 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:45.930 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 Processor -- start 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 -- start start 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: init 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:12:45.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-a con 0x7f7d9809d870 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-b con 0x7f7d980989c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d98099300 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d98069c70 con 0x7f7d98099300 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9806f9d0 con 0x7f7d9809d870 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9809dc40 con 0x7f7d980989c0 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9e583700 10 monclient(hunting): authenticate will time out at 1054.854137s 2024-09-18T10:12:45.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9cb20700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7d98099300 0x7f7d98135670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37524/0 (socket says 172.21.15.142:37524) 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d9cb20700 1 -- 172.21.15.142:0/3614182308 learned_addr learned my addr 172.21.15.142:0/3614182308 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d9809d870 auth_method 0 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d980989c0 auth_method 0 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:45.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.924+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:45.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:45.928+0000 7f7d9cb20700 1 -- 172.21.15.142:0/3614182308 >> v1:172.21.15.142:6789/0 conn(0x7f7d98099300 legacy=0x7f7d98135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d9809d870 auth_method 0 2024-09-18T10:12:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:46.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d980989c0 auth_method 0 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:46.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:46.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.128+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:46.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d9809d870 auth_method 0 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d980989c0 auth_method 0 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:46.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:46.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:46.532+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:46.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): tick 2024-09-18T10:12:46.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 msgr2=0x7f6ebc137a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.150:3300/0 conn(0x7f6ebc0a09a0 0x7f6ebc137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 -- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 msgr2=0x7f6ebc139f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 --2- 172.21.15.150:0/1263726428 >> v2:172.21.15.142:3300/0 conn(0x7f6ebc0a0da0 0x7f6ebc139f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 -- 172.21.15.150:0/1263726428 >> v1:172.21.15.142:6789/0 conn(0x7f6ebc09e6d0 legacy=0x7f6ebc0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient: _add_conns ranks=[2,1] 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6ea0003780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ea00062b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 -- 172.21.15.150:0/1263726428 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc09d930 con 0x7f6ea00062b0 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:46.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 1 -- 172.21.15.150:0/1263726428 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc09ee30 con 0x7f6ea0003780 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6eb3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea0003780 auth_method 0 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:46.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.944+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:46.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:46.948+0000 7f6ec1909700 1 -- 172.21.15.150:0/1263726428 >> v1:172.21.15.150:6789/0 conn(0x7f6ea00062b0 legacy=0x7f6ea0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:47.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea0003780 auth_method 0 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:47.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.148+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:47.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:47.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:47.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d980989c0 auth_method 0 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d9809d870 auth_method 0 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:47.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:47.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:47.336+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea0003780 auth_method 0 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:47.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:47.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:47.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:47.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:47.552+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea0003780 auth_method 0 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:48.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.356+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:12:48.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666776437 0 0) 0x7f6ebc09d930 con 0x7f6ea00062b0 2024-09-18T10:12:48.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): my global_id is 15019 2024-09-18T10:12:48.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea8003570 con 0x7f6ea00062b0 2024-09-18T10:12:48.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469544812 0 0) 0x7f6ea8003570 con 0x7f6ea00062b0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 msgr2=0x7f6ea0005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea0003780 0x7f6ea0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea000b110 con 0x7f6ea00062b0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f6eb800c680 con 0x7f6ea00062b0 2024-09-18T10:12:48.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec436e700 5 monclient: authenticate success, global_id 15019 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec436e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: dump: 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.888 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 >> v1:172.21.15.150:6789/0 conn(0x7f6ea00062b0 legacy=0x7f6ea0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec290b700 1 -- 172.21.15.150:0/1263726428 reap_dead start 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ea8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): picked mon.b con 0x7f6ea8003b00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ea8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): picked mon.c con 0x7f6ea8004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ea800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): picked mon.a con 0x7f6ea8005370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc09ee30 con 0x7f6ea8005370 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea800c340 con 0x7f6ea8003b00 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea800c500 con 0x7f6ea8004810 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec1108700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec436e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ea8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea8003b00 auth_method 0 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.876+0000 7f6ec210a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ea8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ea8004810 auth_method 0 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ea800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ea8005370 auth_method 0 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1909700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient(hunting): handle_auth_done global_id 15019 payload 293 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 msgr2=0x7f6ea8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ea8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 msgr2=0x7f6ea800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ea800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 monclient: found mon.b 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea8003570 con 0x7f6ea8003b00 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 msgr2=0x7f6ea8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec210a700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ea8003ef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6eac007690 tx=0x7f6eac009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2ba8d20803f7da server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac001510 con 0x7f6ea8003b00 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: dump: 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.897 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac001690 con 0x7f6ea8003b00 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec436e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec436e700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 1 -- 172.21.15.150:0/1263726428 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac001a50 con 0x7f6ea8003b00 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec436e700 4 set_mon_vals no callback set 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.880+0000 7f6ec1108700 10 monclient: dump: 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 10 monclient: shutdown 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 msgr2=0x7f6ea8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ea8003ef0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6eac007690 tx=0x7f6eac009040 comp rx=0 tx=0).stop 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- 172.21.15.150:0/1263726428 shutdown_connections 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ea800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ea8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 --2- 172.21.15.150:0/1263726428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ea8003ef0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- 172.21.15.150:0/1263726428 >> 172.21.15.150:0/1263726428 conn(0x7f6ebc0a58a0 msgr2=0x7f6ebc0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- 172.21.15.150:0/1263726428 shutdown_connections 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- 172.21.15.150:0/1263726428 wait complete. 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 10 monclient: build_initial_monmap 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 10 monclient: monmap: 2024-09-18T10:12:48.900 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 Processor -- start 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 1 -- start start 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.884+0000 7f6ec436e700 10 monclient: init 2024-09-18T10:12:48.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-b con 0x7f6ea8003b00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-c con 0x7f6ea8004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ebc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): picked mon.noname-a con 0x7f6ea8005370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc12c780 con 0x7f6ea8005370 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc12bee0 con 0x7f6ea8003b00 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc12c1c0 con 0x7f6ea8004810 2024-09-18T10:12:48.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec436e700 10 monclient(hunting): authenticate will time out at 1050.984645s 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59640/0 (socket says 172.21.15.150:59640) 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 1 -- 172.21.15.150:0/926783625 learned_addr learned my addr 172.21.15.150:0/926783625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): get_auth_request con 0x7f6ea8003b00 auth_method 0 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ebc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 10 monclient(hunting): get_auth_request con 0x7f6ea8004810 auth_method 0 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): get_auth_request con 0x7f6ea8005370 auth_method 0 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 msgr2=0x7f6ebc131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 msgr2=0x7f6ebc131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec1909700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:48.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient(hunting): handle_auth_done global_id 5289 payload 824 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 msgr2=0x7f6ebc131cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 msgr2=0x7f6ebc141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 monclient: found mon.noname-a 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec290b700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc12c4a0 con 0x7f6ea8005370 2024-09-18T10:12:48.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.888+0000 7f6ec210a700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:48.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.892+0000 7f6ec290b700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 msgr2=0x7f6ebc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:48.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.892+0000 7f6ec290b700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ebc06dc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc09e660 tx=0x7f6eb400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883075503 0 0) 0x7f7d98069c70 con 0x7f7d98099300 2024-09-18T10:12:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): my global_id is 5292 2024-09-18T10:12:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d84003570 con 0x7f7d98099300 2024-09-18T10:12:48.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4021460 con 0x7f6ea8005370 2024-09-18T10:12:48.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:48.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: dump: 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb400f040 con 0x7f6ea8005370 2024-09-18T10:12:48.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 5 monclient: authenticate success, global_id 5289 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4021810 con 0x7f6ea8005370 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 10 monclient: _renew_subs 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc141f80 con 0x7f6ea8005370 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec0907700 4 set_mon_vals no callback set 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6eb27fc700 10 monclient: dump: 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.919 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 10 monclient: _renew_subs 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.908+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc06fa10 con 0x7f6ea8005370 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.912+0000 7f6ec436e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.912+0000 7f6ec436e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.912+0000 7f6ec436e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.912+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc09dbf0 con 0x7f6ea8005370 2024-09-18T10:12:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.916+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6eb4003040 con 0x7f6ea8005370 2024-09-18T10:12:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.916+0000 7f6eb27fc700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 0x7f6ea0038980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.916+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(89..89 src has 40..89) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f6eb404a650 con 0x7f6ea8005370 2024-09-18T10:12:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.916+0000 7f6ec210a700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 0x7f6ea0038980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.916+0000 7f6ec210a700 10 monclient: get_auth_request con 0x7f6ea00364d0 auth_method 0 2024-09-18T10:12:48.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.920+0000 7f6ec210a700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 0x7f6ea0038980 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6eac007ba0 tx=0x7f6eac01b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 474114934 0 0) 0x7f7d84003570 con 0x7f7d98099300 2024-09-18T10:12:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d98098dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d98098dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 msgr2=0x7f7d980954b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 --2- 172.21.15.142:0/3614182308 >> v2:172.21.15.142:3300/0 conn(0x7f7d9809d870 0x7f7d980954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9806fc20 con 0x7f7d98099300 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f7d90001b20 con 0x7f7d98099300 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: dump: 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.932 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 >> v1:172.21.15.142:6789/0 conn(0x7f7d98099300 legacy=0x7f7d98135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 0x7f7d84004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): picked mon.b con 0x7f7d84003c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d84004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): picked mon.c con 0x7f7d980989c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84004e90 0x7f7d8400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): picked mon.a con 0x7f7d84004e90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9809dc40 con 0x7f7d84004e90 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d84005670 con 0x7f7d84003c00 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8400bfd0 con 0x7f7d980989c0 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d977fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:48.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 0x7f7d84004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84004e90 0x7f7d8400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d84003c00 auth_method 0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d84004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request con 0x7f7d84004e90 auth_method 0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d980989c0 auth_method 0 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient(hunting): handle_auth_done global_id 5292 payload 293 2024-09-18T10:12:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 msgr2=0x7f7d84004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d980989c0 0x7f7d84004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84004e90 msgr2=0x7f7d8400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84004e90 0x7f7d8400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 monclient: found mon.b 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84003570 con 0x7f7d84003c00 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d97fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9e583700 5 monclient: authenticate success, global_id 5292 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9e583700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.908+0000 7f7d9cb20700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84004e90 0x7f7d8400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d97fff700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d84004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d97fff700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 0x7f7d84004030 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7d88011cf0 tx=0x7f7d88008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8365d9e71a1e50d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88011660 con 0x7f7d84003c00 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: dump: 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d88015a90 con 0x7f7d84003c00 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 1 -- 172.21.15.142:0/3614182308 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88015e50 con 0x7f7d84003c00 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d9e583700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d9e583700 4 set_mon_vals no callback set 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.912+0000 7f7d977fe700 10 monclient: dump: 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:48.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 10 monclient: shutdown 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d84004030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 --2- 172.21.15.142:0/3614182308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d84003c00 0x7f7d84004030 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7d88011cf0 tx=0x7f7d88008040 comp rx=0 tx=0).stop 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9cb20700 1 -- 172.21.15.142:0/3614182308 reap_dead start 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- 172.21.15.142:0/3614182308 shutdown_connections 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- 172.21.15.142:0/3614182308 >> 172.21.15.142:0/3614182308 conn(0x7f7d980a14e0 msgr2=0x7f7d980a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- 172.21.15.142:0/3614182308 shutdown_connections 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- 172.21.15.142:0/3614182308 wait complete. 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 10 monclient: build_initial_monmap 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 10 monclient: monmap: 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:48.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 Processor -- start 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 1 -- start start 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.916+0000 7f7d9e583700 10 monclient: init 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-a con 0x7f7d84003c00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d9808daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-c con 0x7f7d98141ed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:48.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d9808dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): picked mon.noname-b con 0x7f7d981422a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98091700 con 0x7f7d84003c00 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98097b80 con 0x7f7d981422a0 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98096eb0 con 0x7f7d98141ed0 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9e583700 10 monclient(hunting): authenticate will time out at 1057.848619s 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d9808daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53110/0 (socket says 172.21.15.142:53110) 2024-09-18T10:12:48.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 learned_addr learned my addr 172.21.15.142:0/468674693 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d84003c00 auth_method 0 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request con 0x7f7d98141ed0 auth_method 0 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d9808dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:48.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request con 0x7f7d981422a0 auth_method 0 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 msgr2=0x7f7d9808daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 msgr2=0x7f7d9808daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d9808daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d9808daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:48.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient(hunting): handle_auth_done global_id 5295 payload 824 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 msgr2=0x7f7d9808daa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d9808daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 msgr2=0x7f7d9808dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d9808dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 monclient: found mon.noname-a 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.920+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980901c0 con 0x7f7d84003c00 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d98141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d880154c0 tx=0x7f7d88020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8801dc70 con 0x7f7d84003c00 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: dump: 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.947 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:48.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 5 monclient: authenticate success, global_id 5295 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8801ddd0 con 0x7f7d84003c00 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _renew_subs 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9808fb00 con 0x7f7d84003c00 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88027650 con 0x7f7d84003c00 2024-09-18T10:12:48.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 10 monclient: dump: 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:48.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _renew_subs 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9808f5f0 con 0x7f7d84003c00 2024-09-18T10:12:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d96ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:48.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.924+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6eb401f960 con 0x7f6ea8005370 2024-09-18T10:12:48.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.924+0000 7f6eb27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:48.924+0000 7f6eb27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:48.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d980696a0 con 0x7f7d84003c00 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7d88027a40 con 0x7f7d84003c00 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 0x7f7d78037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.924+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(89..89 src has 40..89) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f7d8804a120 con 0x7f7d84003c00 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.928+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 0x7f7d78037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.928+0000 7f7d8ffff700 10 monclient: get_auth_request con 0x7f7d78035b10 auth_method 0 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.928+0000 7f7d8ffff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 0x7f7d78037fc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7d9808f280 tx=0x7f7d80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.932+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7d88025530 con 0x7f7d84003c00 2024-09-18T10:12:48.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.932+0000 7f7d94ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:48.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:48.932+0000 7f7d94ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:49.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.096+0000 7f6ec436e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.096+0000 7f6ec436e700 10 monclient: _send_command 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.096+0000 7f6ec436e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:49.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.096+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f6ebc063ed0 con 0x7f6ea8005370 2024-09-18T10:12:49.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9e583700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:49.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9e583700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:49.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9e583700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:49.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d98067950 con 0x7f7d84003c00 2024-09-18T10:12:49.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d98141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d98141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d880154c0 tx=0x7f7d88020000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7d880154c0 tx=0x7f7d88020000 comp rx=0 tx=0).stop 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:49.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 msgr2=0x7f7d98141990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d84003c00 0x7f7d98141990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 0x7f7d7803fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): picked mon.a con 0x7f7d7803d9e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d780426c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): picked mon.c con 0x7f7d98141ed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d78045100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): picked mon.b con 0x7f7d981422a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78045a80 con 0x7f7d7803d9e0 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78045bc0 con 0x7f7d981422a0 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d78045d50 con 0x7f7d98141ed0 2024-09-18T10:12:49.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d94ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 0x7f7d7803fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d7803d9e0 auth_method 0 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d780426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request con 0x7f7d98141ed0 auth_method 0 2024-09-18T10:12:49.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:49.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d78045100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d9cb20700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:49.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): get_auth_request con 0x7f7d981422a0 auth_method 0 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:49.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient(hunting): handle_auth_done global_id 5295 payload 293 2024-09-18T10:12:49.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 msgr2=0x7f7d780426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d780426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 msgr2=0x7f7d78045100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d78045100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:49.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 monclient: found mon.a 2024-09-18T10:12:49.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:49.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:49.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f7d78045ee0 con 0x7f7d7803d9e0 2024-09-18T10:12:49.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:49.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:49.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d98067950 con 0x7f7d7803d9e0 2024-09-18T10:12:49.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:49.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.152+0000 7f7d97fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:49.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d97fff700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 msgr2=0x7f7d7803fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:49.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d97fff700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 0x7f7d7803fdc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7d880154c0 tx=0x7f7d88008080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88026420 con 0x7f7d7803d9e0 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: dump: 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:49.176 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:49.177 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d88030310 con 0x7f7d7803d9e0 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d96ffd700 4 set_mon_vals no callback set 2024-09-18T10:12:49.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7d88030ce0 con 0x7f7d7803d9e0 2024-09-18T10:12:49.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 1 -- 172.21.15.142:0/468674693 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f7d8804d610 con 0x7f7d7803d9e0 2024-09-18T10:12:49.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:49.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.156+0000 7f7d94ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:49.179 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":48,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:49.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 msgr2=0x7f7d78037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 0x7f7d78037fc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7d9808f280 tx=0x7f7d80008040 comp rx=0 tx=0).stop 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 10 monclient: shutdown 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 msgr2=0x7f7d7803fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 0x7f7d7803fdc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7d880154c0 tx=0x7f7d88008080 comp rx=0 tx=0).stop 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 shutdown_connections 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d981422a0 0x7f7d78045100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d98141ed0 0x7f7d780426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d7803d9e0 0x7f7d7803fdc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 --2- 172.21.15.142:0/468674693 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d78035b10 0x7f7d78037fc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 >> 172.21.15.142:0/468674693 conn(0x7f7d980a14e0 msgr2=0x7f7d98095520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:49.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 shutdown_connections 2024-09-18T10:12:49.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:49.160+0000 7f7d9e583700 1 -- 172.21.15.142:0/468674693 wait complete. 2024-09-18T10:12:49.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.208+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 v90) v1 ==== 85+0+0 (secure 0 0 0) 0x7f6eb4028340 con 0x7f6ea8005370 2024-09-18T10:12:49.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.208+0000 7f6eb27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.208+0000 7f6eb27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:49.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 10 start_mon_command cmd=[{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 10 monclient: _send_command 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:49.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rule rm", "name": "foo-rule"} v 0) v1 -- 0x7f6ebc12c300 con 0x7f6ea8005370 2024-09-18T10:12:49.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6eb27fc700 1 -- 172.21.15.150:0/926783625 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rule rm", "name": "foo-rule"}]=0 rule foo-rule does not exist v90) v1 ==== 113+0+0 (secure 0 0 0) 0x7f6ebc12c300 con 0x7f6ea8005370 2024-09-18T10:12:49.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6eb27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rule rm", "name": "foo-rule"}] 2024-09-18T10:12:49.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6eb27fc700 10 monclient: _finish_command 3 = system:0 rule foo-rule does not exist 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 msgr2=0x7f6ea0038980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 0x7f6ea0038980 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6eac007ba0 tx=0x7f6eac01b000 comp rx=0 tx=0).stop 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 10 monclient: shutdown 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 msgr2=0x7f6ebc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ebc06dc90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc09e660 tx=0x7f6eb400b040 comp rx=0 tx=0).stop 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 shutdown_connections 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6ea00364d0 0x7f6ea0038980 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ea8005370 0x7f6ebc06dc90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6ea8004810 0x7f6ebc131cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 --2- 172.21.15.150:0/926783625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ea8003b00 0x7f6ebc141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 >> 172.21.15.150:0/926783625 conn(0x7f6ebc0a58a0 msgr2=0x7f6ebc0a4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 shutdown_connections 2024-09-18T10:12:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.316+0000 7f6ec436e700 1 -- 172.21.15.150:0/926783625 wait complete. 2024-09-18T10:12:49.325 INFO:tasks.workunit.client.0.smithi150.stderr:rule foo-rule does not exist 2024-09-18T10:12:49.334 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm bar 2024-09-18T10:12:49.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.428+0000 7fd34e0e8700 10 monclient: get_monmap_and_config 2024-09-18T10:12:49.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.428+0000 7fd34e0e8700 10 monclient: build_initial_monmap 2024-09-18T10:12:49.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.428+0000 7fd34e0e8700 10 monclient: monmap: 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:49.435418+0000 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:49.435418+0000 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:49.436 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 Processor -- start 2024-09-18T10:12:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 -- start start 2024-09-18T10:12:49.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient: init 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-b con 0x7fd3480a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3480a04f0 0x7fd3480a0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-c con 0x7fd3480a04f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3481369a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34809dae0 con 0x7fd3481369a0 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3480a21d0 con 0x7fd3480a04f0 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34809ddc0 con 0x7fd3480a0120 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd34e0e8700 10 monclient(hunting): authenticate will time out at 1051.529058s 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3480a04f0 0x7fd3480a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd3480a04f0 0x7fd3480a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37204/0 (socket says 172.21.15.150:37204) 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 -- 172.21.15.150:0/145402008 learned_addr learned my addr 172.21.15.150:0/145402008 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): get_auth_request con 0x7fd3480a04f0 auth_method 0 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 10 monclient(hunting): get_auth_request con 0x7fd3480a0120 auth_method 0 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:49.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 1 -- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 msgr2=0x7fd34809e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 1 -- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 msgr2=0x7fd34809e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd3477fe700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient(hunting): handle_auth_done global_id 15028 payload 274 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 -- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 msgr2=0x7fd34809e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 -- 172.21.15.150:0/145402008 >> v1:172.21.15.142:6789/0 conn(0x7fd3481369a0 legacy=0x7fd348138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 monclient: found mon.noname-c 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.432+0000 7fd346ffd700 1 -- 172.21.15.150:0/145402008 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480a1590 con 0x7fd3480a04f0 2024-09-18T10:12:49.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd346ffd700 1 -- 172.21.15.150:0/145402008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 msgr2=0x7fd3480a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:49.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd346ffd700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd3480a0920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0099b0 tx=0x7fd33c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64204ac9f79cab57 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:49.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 1 -- 172.21.15.150:0/145402008 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00ba70 con 0x7fd3480a04f0 2024-09-18T10:12:49.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:49.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:49.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: dump: 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.453 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 1 -- 172.21.15.150:0/145402008 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c00bbd0 con 0x7fd3480a04f0 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:49.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 5 monclient: authenticate success, global_id 15028 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 4 set_mon_vals no callback set 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 1 -- 172.21.15.150:0/145402008 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c011420 con 0x7fd3480a04f0 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd3467fc700 10 monclient: dump: 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:49.455 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:49.456 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: shutdown 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- 172.21.15.150:0/145402008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 msgr2=0x7fd3480a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd3480a0920 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd33c0099b0 tx=0x7fd33c009250 comp rx=0 tx=0).stop 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- 172.21.15.150:0/145402008 shutdown_connections 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd3480a0920 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/145402008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3480a0120 0x7fd34809e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- 172.21.15.150:0/145402008 >> 172.21.15.150:0/145402008 conn(0x7fd3480a5830 msgr2=0x7fd3480a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- 172.21.15.150:0/145402008 shutdown_connections 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- 172.21.15.150:0/145402008 wait complete. 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: build_initial_monmap 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: monmap: 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:49.458 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:49.459 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:49.459 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:49.459 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.459 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:49.459 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 Processor -- start 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- start start 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: init 2024-09-18T10:12:49.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3480a0120 0x7fd348141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-a con 0x7fd3480a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-b con 0x7fd3480a04f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:49.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): picked mon.noname-c con 0x7fd3481369a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd348130170 con 0x7fd3480a0120 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34812e030 con 0x7fd3480a04f0 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34809d720 con 0x7fd3481369a0 2024-09-18T10:12:49.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.436+0000 7fd34e0e8700 10 monclient(hunting): authenticate will time out at 1051.535970s 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37218/0 (socket says 172.21.15.150:37218) 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 -- 172.21.15.150:0/1221945827 learned_addr learned my addr 172.21.15.150:0/1221945827 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): get_auth_request con 0x7fd3480a04f0 auth_method 0 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3480a0120 0x7fd348141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 10 monclient(hunting): get_auth_request con 0x7fd3481369a0 auth_method 0 2024-09-18T10:12:49.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): get_auth_request con 0x7fd3480a0120 auth_method 0 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 msgr2=0x7fd3481428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:12:49.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 msgr2=0x7fd3481428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd3477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient(hunting): handle_auth_done global_id 15031 payload 824 2024-09-18T10:12:49.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd347fff700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 msgr2=0x7fd3481428a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3480a0120 msgr2=0x7fd348141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3480a0120 0x7fd348141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 monclient: found mon.noname-b 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:49.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0094f0 con 0x7fd3480a04f0 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 msgr2=0x7fd348142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd346ffd700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c011420 con 0x7fd3480a04f0 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: dump: 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.468 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 5 monclient: authenticate success, global_id 15031 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c020070 con 0x7fd3480a04f0 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 10 monclient: _renew_subs 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34809bff0 con 0x7fd3480a04f0 2024-09-18T10:12:49.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c01b6c0 con 0x7fd3480a04f0 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd345ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd32f7fe700 10 monclient: dump: 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:49.470 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:49.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 10 monclient: _renew_subs 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.440+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd348130db0 con 0x7fd3480a04f0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd34e0e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd34e0e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd34e0e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34809d8b0 con 0x7fd3480a04f0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd33c022030 con 0x7fd3480a04f0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd32f7fe700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 0x7fd334037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(90..90 src has 40..90) v4 ==== 6085+0+0 (secure 0 0 0) 0x7fd33c04a540 con 0x7fd3480a04f0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd3477fe700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 0x7fd334037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:49.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.444+0000 7fd3477fe700 10 monclient: get_auth_request con 0x7fd334035ac0 auth_method 0 2024-09-18T10:12:49.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.448+0000 7fd3477fe700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 0x7fd334037f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd338006f30 tx=0x7fd338008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:49.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.456+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd33c006a90 con 0x7fd3480a04f0 2024-09-18T10:12:49.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.456+0000 7fd32f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:49.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.456+0000 7fd32f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:49.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.624+0000 7fd34e0e8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:49.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.624+0000 7fd34e0e8700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:49.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.624+0000 7fd34e0e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:49.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:49.624+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fd34812fc40 con 0x7fd3480a04f0 2024-09-18T10:12:50.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.212+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 removed item id -19 name 'bar' from crush map v91) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd33c01ab80 con 0x7fd3480a04f0 2024-09-18T10:12:50.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.212+0000 7fd32f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:50.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.212+0000 7fd32f7fe700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'bar' from crush map 2024-09-18T10:12:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.320+0000 7fd34e0e8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.320+0000 7fd34e0e8700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.320+0000 7fd34e0e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.320+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "bar"} v 0) v1 -- 0x7fd34812fc40 con 0x7fd3480a04f0 2024-09-18T10:12:50.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd32f7fe700 1 -- 172.21.15.150:0/1221945827 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "bar"}]=0 device 'bar' does not appear in the crush map v91) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd33c01ad60 con 0x7fd3480a04f0 2024-09-18T10:12:50.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd32f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "bar"}] 2024-09-18T10:12:50.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd32f7fe700 10 monclient: _finish_command 3 = system:0 device 'bar' does not appear in the crush map 2024-09-18T10:12:50.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 msgr2=0x7fd334037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:50.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 0x7fd334037f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd338006f30 tx=0x7fd338008040 comp rx=0 tx=0).stop 2024-09-18T10:12:50.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 10 monclient: shutdown 2024-09-18T10:12:50.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 msgr2=0x7fd348142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c016000 comp rx=0 tx=0).stop 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 shutdown_connections 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd334035ac0 0x7fd334037f70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3481369a0 0x7fd3481428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3480a04f0 0x7fd348142360 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 --2- 172.21.15.150:0/1221945827 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3480a0120 0x7fd348141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 >> 172.21.15.150:0/1221945827 conn(0x7fd3480a5830 msgr2=0x7fd34809ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 shutdown_connections 2024-09-18T10:12:50.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.324+0000 7fd34e0e8700 1 -- 172.21.15.150:0/1221945827 wait complete. 2024-09-18T10:12:50.332 INFO:tasks.workunit.client.0.smithi150.stderr:device 'bar' does not appear in the crush map 2024-09-18T10:12:50.340 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm foo 2024-09-18T10:12:50.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: build_initial_monmap 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: monmap: 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:50.444646+0000 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:50.444646+0000 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:50.445 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:50.446 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:50.446 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:50.446 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 Processor -- start 2024-09-18T10:12:50.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 -- start start 2024-09-18T10:12:50.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: init 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f820c09cc90 0x7f820c0a0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): picked mon.noname-c con 0x7f820c09cc90 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f820c09e6d0 0x7f820c0a0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): picked mon.noname-a con 0x7f820c09e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): picked mon.noname-b con 0x7f820c0a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c06fd00 con 0x7f820c09e6d0 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820c0a0e80 con 0x7f820c09cc90 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c09dcf0 con 0x7f820c0a0a80 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f821080a700 10 monclient(hunting): authenticate will time out at 1052.538102s 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f820c09cc90 0x7f820c0a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f820c09cc90 0x7f820c0a0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37230/0 (socket says 172.21.15.150:37230) 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 -- 172.21.15.150:0/2213832514 learned_addr learned my addr 172.21.15.150:0/2213832514 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): get_auth_request con 0x7f820c09cc90 auth_method 0 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 1 --2- 172.21.15.150:0/2213832514 >> v2:172.21.15.142:3300/0 conn(0x7f820c09e6d0 0x7f820c0a0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): get_auth_request con 0x7f820c09e6d0 auth_method 0 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 10 monclient(hunting): get_auth_request con 0x7f820c0a0a80 auth_method 0 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient(hunting): handle_auth_done global_id 15034 payload 274 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 1 -- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 msgr2=0x7f820c132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 1 -- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 msgr2=0x7f820c132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 -- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 msgr2=0x7f820c132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820bfff700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 -- 172.21.15.150:0/2213832514 >> v2:172.21.15.142:3300/0 conn(0x7f820c09e6d0 msgr2=0x7f820c0a0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 --2- 172.21.15.150:0/2213832514 >> v2:172.21.15.142:3300/0 conn(0x7f820c09e6d0 0x7f820c0a0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:50.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:50.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:50.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.440+0000 7f820b7fe700 1 -- 172.21.15.150:0/2213832514 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820c0a64b0 con 0x7f820c09cc90 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820b7fe700 1 -- 172.21.15.150:0/2213832514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 msgr2=0x7f820c0a0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820b7fe700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c0a0000 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f820c068bf0 tx=0x7f8200009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e368437d16b03488 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 1 -- 172.21.15.150:0/2213832514 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200009e70 con 0x7f820c09cc90 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: dump: 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.459 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 1 -- 172.21.15.150:0/2213832514 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000ba70 con 0x7f820c09cc90 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 1 -- 172.21.15.150:0/2213832514 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000be10 con 0x7f820c09cc90 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:50.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f820a7fc700 10 monclient: dump: 2024-09-18T10:12:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.461 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 5 monclient: authenticate success, global_id 15034 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 4 set_mon_vals no callback set 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 10 monclient: shutdown 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:50.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 -- 172.21.15.150:0/2213832514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 msgr2=0x7f820c0a0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:50.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c0a0000 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f820c068bf0 tx=0x7f8200009250 comp rx=0 tx=0).stop 2024-09-18T10:12:50.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 -- 172.21.15.150:0/2213832514 shutdown_connections 2024-09-18T10:12:50.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c0a0a80 0x7f820c132760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 --2- 172.21.15.150:0/2213832514 >> v2:172.21.15.142:3300/0 conn(0x7f820c09e6d0 0x7f820c0a0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 --2- 172.21.15.150:0/2213832514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c0a0000 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 -- 172.21.15.150:0/2213832514 >> 172.21.15.150:0/2213832514 conn(0x7f820c0683b0 msgr2=0x7f820c0a5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 -- 172.21.15.150:0/2213832514 shutdown_connections 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 1 -- 172.21.15.150:0/2213832514 wait complete. 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 10 monclient: build_initial_monmap 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.444+0000 7f821080a700 10 monclient: monmap: 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.464 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 Processor -- start 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 -- start start 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient: init 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): picked mon.noname-b con 0x7f820c09cc90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): picked mon.noname-c con 0x7f820c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c0a0a80 0x7f820c12c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): picked mon.noname-a con 0x7f820c0a0a80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c06acd0 con 0x7f820c0a0a80 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820c12fe70 con 0x7f820c09cc90 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820c12dd80 con 0x7f820c09e6d0 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f821080a700 10 monclient(hunting): authenticate will time out at 1052.544745s 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37234/0 (socket says 172.21.15.150:37234) 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 -- 172.21.15.150:0/3250935535 learned_addr learned my addr 172.21.15.150:0/3250935535 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): get_auth_request con 0x7f820c09cc90 auth_method 0 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c0a0a80 0x7f820c12c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): get_auth_request con 0x7f820c0a0a80 auth_method 0 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 10 monclient(hunting): get_auth_request con 0x7f820c09e6d0 auth_method 0 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 msgr2=0x7f820c09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 msgr2=0x7f820c09bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:50.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820affd700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient(hunting): handle_auth_done global_id 15037 payload 824 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 msgr2=0x7f820c09bbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c0a0a80 msgr2=0x7f820c12c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c0a0a80 0x7f820c12c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82000094f0 con 0x7f820c09cc90 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 msgr2=0x7f820c09b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:50.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f820b7fe700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8200009290 tx=0x7f8200017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.448+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200011e20 con 0x7f820c09cc90 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: dump: 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:50.470 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 5 monclient: authenticate success, global_id 15037 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _renew_subs 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8200000c90 con 0x7f820c09cc90 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820c12f8b0 con 0x7f820c09cc90 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820001a880 con 0x7f820c09cc90 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:50.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 10 monclient: dump: 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f8209ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _renew_subs 2024-09-18T10:12:50.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820c12e950 con 0x7f820c09cc90 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820c09eb80 con 0x7f820c09cc90 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f820001ac70 con 0x7f820c09cc90 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 0x7f81f8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(91..91 src has 40..91) v4 ==== 5956+0+0 (secure 0 0 0) 0x7f820004a020 con 0x7f820c09cc90 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f820affd700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 0x7f81f8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:50.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.452+0000 7f820affd700 10 monclient: get_auth_request con 0x7f81f8035ac0 auth_method 0 2024-09-18T10:12:50.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.456+0000 7f820affd700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 0x7f81f8037f70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f820c09e080 tx=0x7f81fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:50.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.464+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f820002c9a0 con 0x7f820c09cc90 2024-09-18T10:12:50.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.464+0000 7f81f37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:50.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.464+0000 7f81f37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.632+0000 7f821080a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.632+0000 7f821080a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.632+0000 7f821080a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:50.632+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f820c04bbf0 con 0x7f820c09cc90 2024-09-18T10:12:51.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.232+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8200015070 con 0x7f820c09cc90 2024-09-18T10:12:51.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.232+0000 7f81f37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:51.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.232+0000 7f81f37fe700 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-18T10:12:51.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.340+0000 7f821080a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:51.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.340+0000 7f821080a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:51.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.340+0000 7f821080a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.340+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f820c0689b0 con 0x7f820c09cc90 2024-09-18T10:12:51.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f81f37fe700 1 -- 172.21.15.150:0/3250935535 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8200022390 con 0x7f820c09cc90 2024-09-18T10:12:51.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f81f37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:12:51.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f81f37fe700 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-18T10:12:51.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 msgr2=0x7f81f8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:51.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 0x7f81f8037f70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f820c09e080 tx=0x7f81fc008040 comp rx=0 tx=0).stop 2024-09-18T10:12:51.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 10 monclient: shutdown 2024-09-18T10:12:51.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 msgr2=0x7f820c09b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8200009290 tx=0x7f8200017000 comp rx=0 tx=0).stop 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 shutdown_connections 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f81f8035ac0 0x7f81f8037f70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f820c0a0a80 0x7f820c12c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f820c09e6d0 0x7f820c09bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 --2- 172.21.15.150:0/3250935535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f820c09cc90 0x7f820c09b6a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 >> 172.21.15.150:0/3250935535 conn(0x7f820c0683b0 msgr2=0x7f820c06e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 shutdown_connections 2024-09-18T10:12:51.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.344+0000 7f821080a700 1 -- 172.21.15.150:0/3250935535 wait complete. 2024-09-18T10:12:51.352 INFO:tasks.workunit.client.0.smithi150.stderr:device 'foo' does not appear in the crush map 2024-09-18T10:12:51.366 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.10 host2 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 10 monclient: get_monmap_and_config 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 10 monclient: build_initial_monmap 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 10 monclient: monmap: 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:51.469842+0000 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:51.469842+0000 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.470 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 1 Processor -- start 2024-09-18T10:12:51.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 1 -- start start 2024-09-18T10:12:51.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.464+0000 7fb98ca7e700 10 monclient: init 2024-09-18T10:12:51.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:51.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:12:51.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb98812e460 0x7fb98812eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-c con 0x7fb98812e460 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb98806eb50 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb988133790 0x7fb988135b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-a con 0x7fb988133790 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98806be40 con 0x7fb98806eb50 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98806f660 con 0x7fb988133790 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98806c630 con 0x7fb98812e460 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb98ca7e700 10 monclient(hunting): authenticate will time out at 1053.564955s 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb98812e460 0x7fb98812eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb98812e460 0x7fb98812eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37236/0 (socket says 172.21.15.150:37236) 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 -- 172.21.15.150:0/559765762 learned_addr learned my addr 172.21.15.150:0/559765762 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): get_auth_request con 0x7fb98812e460 auth_method 0 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb987fff700 1 --2- 172.21.15.150:0/559765762 >> v2:172.21.15.142:3300/0 conn(0x7fb988133790 0x7fb988135b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb987fff700 10 monclient(hunting): get_auth_request con 0x7fb988133790 auth_method 0 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb987fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:51.476 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb987fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb987fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient(hunting): handle_auth_done global_id 15040 payload 274 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 -- 172.21.15.150:0/559765762 >> v2:172.21.15.142:3300/0 conn(0x7fb988133790 msgr2=0x7fb988135b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 --2- 172.21.15.150:0/559765762 >> v2:172.21.15.142:3300/0 conn(0x7fb988133790 0x7fb988135b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 -- 172.21.15.150:0/559765762 >> v1:172.21.15.142:6789/0 conn(0x7fb98806eb50 legacy=0x7fb98806ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 -- 172.21.15.150:0/559765762 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98806c070 con 0x7fb98812e460 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 -- 172.21.15.150:0/559765762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb98812e460 msgr2=0x7fb98812eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.468+0000 7fb9877fe700 1 --2- 172.21.15.150:0/559765762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb98812e460 0x7fb98812eaa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ab50 tx=0x7fb97800a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d14d2976fc8b55d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 1 -- 172.21.15.150:0/559765762 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800cc30 con 0x7fb98812e460 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: dump: 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:51.480 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 5 monclient: authenticate success, global_id 15040 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 1 -- 172.21.15.150:0/559765762 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97800cd90 con 0x7fb98812e460 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:51.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 1 -- 172.21.15.150:0/559765762 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978018640 con 0x7fb98812e460 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 4 set_mon_vals no callback set 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb986ffd700 10 monclient: dump: 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.482 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 10 monclient: shutdown 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- 172.21.15.150:0/559765762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb98812e460 msgr2=0x7fb98812eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/559765762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb98812e460 0x7fb98812eaa0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ab50 tx=0x7fb97800a3f0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- 172.21.15.150:0/559765762 shutdown_connections 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/559765762 >> v2:172.21.15.142:3300/0 conn(0x7fb988133790 0x7fb988135b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/559765762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb98812e460 0x7fb98812eaa0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- 172.21.15.150:0/559765762 >> 172.21.15.150:0/559765762 conn(0x7fb988068590 msgr2=0x7fb9880a1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- 172.21.15.150:0/559765762 shutdown_connections 2024-09-18T10:12:51.483 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- 172.21.15.150:0/559765762 wait complete. 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 10 monclient: build_initial_monmap 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 10 monclient: monmap: 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 Processor -- start 2024-09-18T10:12:51.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 1 -- start start 2024-09-18T10:12:51.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.472+0000 7fb98ca7e700 10 monclient: init 2024-09-18T10:12:51.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:51.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:51.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb98806eb50 0x7fb98809b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-a con 0x7fb98806eb50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 0x7fb988118290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-c con 0x7fb98812e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): picked mon.noname-b con 0x7fb988133790 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98811b2e0 con 0x7fb98806eb50 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98809d7d0 con 0x7fb988133790 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98806d010 con 0x7fb98812e460 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient(hunting): authenticate will time out at 1053.572968s 2024-09-18T10:12:51.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37240/0 (socket says 172.21.15.150:37240) 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 -- 172.21.15.150:0/797325840 learned_addr learned my addr 172.21.15.150:0/797325840 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): get_auth_request con 0x7fb988133790 auth_method 0 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb9877fe700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb98806eb50 0x7fb98809b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb9877fe700 10 monclient(hunting): get_auth_request con 0x7fb98806eb50 auth_method 0 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97effd700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 0x7fb988118290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb9877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:51.490 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb9877fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb9877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient(hunting): handle_auth_done global_id 15043 payload 824 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 -- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 msgr2=0x7fb988118290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 0x7fb988118290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 -- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb98806eb50 msgr2=0x7fb98809b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb98806eb50 0x7fb98809b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 monclient: found mon.noname-b 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97800a690 con 0x7fb988133790 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97effd700 10 monclient: get_auth_request con 0x7fb98812e460 auth_method 0 2024-09-18T10:12:51.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97effd700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 0x7fb988118290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 -- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 msgr2=0x7fb9881187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb987fff700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb980009fa0 tx=0x7fb980009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980014420 con 0x7fb988133790 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 10 monclient: dump: 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.492 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 10 monclient: _finish_auth 0 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 5 monclient: authenticate success, global_id 15043 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient: _renew_subs 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980014580 con 0x7fb988133790 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.476+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98811b8a0 con 0x7fb988133790 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:51.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980014940 con 0x7fb988133790 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 10 monclient: dump: 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:51.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 10 monclient: _renew_subs 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb9867fc700 4 set_mon_vals no callback set 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb988119540 con 0x7fb988133790 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb980014cb0 con 0x7fb988133790 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98804bbf0 con 0x7fb988133790 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 0x7fb968037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(92..92 src has 40..92) v4 ==== 5827+0+0 (secure 0 0 0) 0x7fb9800490c0 con 0x7fb988133790 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb9877fe700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 0x7fb968037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:51.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.480+0000 7fb9877fe700 10 monclient: get_auth_request con 0x7fb968035ac0 auth_method 0 2024-09-18T10:12:51.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.484+0000 7fb9877fe700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 0x7fb968037f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ab50 tx=0x7fb978019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:51.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.488+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb9800126a0 con 0x7fb988133790 2024-09-18T10:12:51.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.488+0000 7fb97ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:51.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.488+0000 7fb97ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.652+0000 7fb98ca7e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.652+0000 7fb98ca7e700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.652+0000 7fb98ca7e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:51.652+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"} v 0) v1 -- 0x7fb988063ed0 con 0x7fb988133790 2024-09-18T10:12:52.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:52.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.244+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}]=0 removed item id 10 name 'osd.10' from crush map v93) v1 ==== 146+0+0 (secure 0 0 0) 0x7fb9800122e0 con 0x7fb988133790 2024-09-18T10:12:52.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.244+0000 7fb97ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:52.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.244+0000 7fb97ffff700 10 monclient: _finish_command 2 = system:0 removed item id 10 name 'osd.10' from crush map 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.288+0000 7f39c1673700 10 monclient: get_monmap_and_config 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.288+0000 7f39c1673700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.288+0000 7f39c1673700 10 monclient: monmap: 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:52.295188+0000 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:52.295188+0000 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:52.296 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:52.297 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:52.297 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:52.297 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:52.297 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 Processor -- start 2024-09-18T10:12:52.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 -- start start 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient: init 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39bc070ef0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-b con 0x7f39bc09a380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f39bc09df70 0x7f39bc139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-a con 0x7f39bc09df70 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc09e720 con 0x7f39bc070ef0 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39bc06fc50 con 0x7f39bc09df70 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc09e9e0 con 0x7f39bc09a380 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c1673700 10 monclient(hunting): authenticate will time out at 1061.220715s 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44972/0 (socket says 172.21.15.142:44972) 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 -- 172.21.15.142:0/3486145919 learned_addr learned my addr 172.21.15.142:0/3486145919 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 10 monclient(hunting): get_auth_request con 0x7f39bc09a380 auth_method 0 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 -- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 msgr2=0x7f39bc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 -- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 msgr2=0x7f39bc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39b37fe700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 --2- 172.21.15.142:0/3486145919 >> v2:172.21.15.142:3300/0 conn(0x7f39bc09df70 0x7f39bc139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): get_auth_request con 0x7f39bc09df70 auth_method 0 2024-09-18T10:12:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39bbfff700 1 -- 172.21.15.142:0/3486145919 >> v1:172.21.15.142:6789/0 conn(0x7f39bc070ef0 legacy=0x7f39bc0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient(hunting): handle_auth_done global_id 5313 payload 274 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 -- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 msgr2=0x7f39bc1376a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 -- 172.21.15.142:0/3486145919 >> v1:172.21.15.142:6789/0 conn(0x7f39bc070ef0 legacy=0x7f39bc0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39bb7fe700 1 -- 172.21.15.142:0/3486145919 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139535882 0 0) 0x7f39bc09e720 con 0x7f39bc070ef0 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39c0e72700 1 -- 172.21.15.142:0/3486145919 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc09ecc0 con 0x7f39bc09df70 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.292+0000 7f39bb7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c0e72700 1 -- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 msgr2=0x7f39bc139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c0e72700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc139bf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009ee0 tx=0x7f39b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529e4e00e3ae6aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 1 -- 172.21.15.142:0/3486145919 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b400e070 con 0x7f39bc09df70 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: dump: 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.305 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 1 -- 172.21.15.142:0/3486145919 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b400be60 con 0x7f39bc09df70 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 5 monclient: authenticate success, global_id 5313 2024-09-18T10:12:52.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 4 set_mon_vals no callback set 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 1 -- 172.21.15.142:0/3486145919 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b4012680 con 0x7f39bc09df70 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39bb7fe700 10 monclient: dump: 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 10 monclient: shutdown 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 -- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 msgr2=0x7f39bc139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc139bf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f39b4009ee0 tx=0x7f39b4009780 comp rx=0 tx=0).stop 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 -- 172.21.15.142:0/3486145919 shutdown_connections 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc09a380 0x7f39bc1376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 --2- 172.21.15.142:0/3486145919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc139bf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 -- 172.21.15.142:0/3486145919 >> 172.21.15.142:0/3486145919 conn(0x7f39bc06a2e0 msgr2=0x7f39bc0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 -- 172.21.15.142:0/3486145919 shutdown_connections 2024-09-18T10:12:52.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.296+0000 7f39c1673700 1 -- 172.21.15.142:0/3486145919 wait complete. 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient: monmap: 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:52.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 Processor -- start 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 -- start start 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient: init 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-c con 0x7f39bc070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 0x7f39bc08d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-b con 0x7f39bc09a380 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc08db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): picked mon.noname-a con 0x7f39bc09df70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc09e470 con 0x7f39bc09df70 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0a2310 con 0x7f39bc09a380 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc090700 con 0x7f39bc070ef0 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c1673700 10 monclient(hunting): authenticate will time out at 1061.229654s 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc08db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44980/0 (socket says 172.21.15.142:44980) 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 -- 172.21.15.142:0/692507479 learned_addr learned my addr 172.21.15.142:0/692507479 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 10 monclient(hunting): get_auth_request con 0x7f39bc070ef0 auth_method 0 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): get_auth_request con 0x7f39bc09df70 auth_method 0 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 0x7f39bc08d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): get_auth_request con 0x7f39bc09a380 auth_method 0 2024-09-18T10:12:52.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 msgr2=0x7f39bc08d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 msgr2=0x7f39bc08d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39bbfff700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient(hunting): handle_auth_done global_id 5319 payload 824 2024-09-18T10:12:52.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 msgr2=0x7f39bc08d090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 msgr2=0x7f39bc08d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 0x7f39bc08d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39c0e72700 1 -- 172.21.15.142:0/692507479 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4009a20 con 0x7f39bc09df70 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.300+0000 7f39b37fe700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 0x7f39bc08d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c0e72700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 msgr2=0x7f39bc08db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c0e72700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc08db10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f39b4016000 tx=0x7f39b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b401e630 con 0x7f39bc09df70 2024-09-18T10:12:52.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: dump: 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.319 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 5 monclient: authenticate success, global_id 5319 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _renew_subs 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b401e790 con 0x7f39bc09df70 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc090420 con 0x7f39bc09df70 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b401eb50 con 0x7f39bc09df70 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39b8ff9700 10 monclient: dump: 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39baffd700 4 set_mon_vals no callback set 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _renew_subs 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc095400 con 0x7f39bc09df70 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.304+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc072a80 con 0x7f39bc09df70 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.308+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f39b4024070 con 0x7f39bc09df70 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.308+0000 7f39b8ff9700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 0x7f399c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.308+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 40..93) v4 ==== 5805+0+0 (secure 0 0 0) 0x7f39b404b2a0 con 0x7f39bc09df70 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.308+0000 7f39bbfff700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 0x7f399c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.308+0000 7f39bbfff700 10 monclient: get_auth_request con 0x7f399c035ac0 auth_method 0 2024-09-18T10:12:52.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.312+0000 7f39bbfff700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 0x7f399c037f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0015e0 tx=0x7f39ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.316+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f39b4011990 con 0x7f39bc09df70 2024-09-18T10:12:52.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.316+0000 7f39b8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.316+0000 7f39b8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:52.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.356+0000 7fb98ca7e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:52.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.356+0000 7fb98ca7e700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:52.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.356+0000 7fb98ca7e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:52.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.356+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"} v 0) v1 -- 0x7fb98811a320 con 0x7fb988133790 2024-09-18T10:12:52.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb97ffff700 1 -- 172.21.15.150:0/797325840 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}]=0 device 'osd.10' does not appear in the crush map v93) v1 ==== 147+0+0 (secure 0 0 0) 0x7fb9800124c0 con 0x7fb988133790 2024-09-18T10:12:52.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb97ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.10", "ancestor": "host2"}] 2024-09-18T10:12:52.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb97ffff700 10 monclient: _finish_command 3 = system:0 device 'osd.10' does not appear in the crush map 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 msgr2=0x7fb968037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 0x7fb968037f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ab50 tx=0x7fb978019000 comp rx=0 tx=0).stop 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 10 monclient: shutdown 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 msgr2=0x7fb9881187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb980009fa0 tx=0x7fb980009900 comp rx=0 tx=0).stop 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 shutdown_connections 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb968035ac0 0x7fb968037f70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb988133790 0x7fb9881187d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb98812e460 0x7fb988118290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 --2- 172.21.15.150:0/797325840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb98806eb50 0x7fb98809b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 >> 172.21.15.150:0/797325840 conn(0x7fb988068590 msgr2=0x7fb9880a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 shutdown_connections 2024-09-18T10:12:52.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.360+0000 7fb98ca7e700 1 -- 172.21.15.150:0/797325840 wait complete. 2024-09-18T10:12:52.368 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.10' does not appear in the crush map 2024-09-18T10:12:52.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm host2 2024-09-18T10:12:52.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:52.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 10 monclient: monmap: 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:52.482350+0000 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:52.482350+0000 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:52.483 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 1 Processor -- start 2024-09-18T10:12:52.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 1 -- start start 2024-09-18T10:12:52.484 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.476+0000 7fc20969a700 10 monclient: init 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc20409e6d0 0x7fc2040a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-c con 0x7fc20409e6d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2040a07d0 0x7fc2040a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-b con 0x7fc2040a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc204136a10 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040a1110 con 0x7fc204136a10 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20409ee30 con 0x7fc20409e6d0 2024-09-18T10:12:52.486 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20409e220 con 0x7fc2040a07d0 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc20969a700 10 monclient(hunting): authenticate will time out at 1054.576603s 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc20409e6d0 0x7fc2040a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc20409e6d0 0x7fc2040a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37246/0 (socket says 172.21.15.150:37246) 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 -- 172.21.15.150:0/3224868596 learned_addr learned my addr 172.21.15.150:0/3224868596 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): get_auth_request con 0x7fc20409e6d0 auth_method 0 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.487 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc1fa7fc700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2040a07d0 0x7fc2040a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc1fa7fc700 10 monclient(hunting): get_auth_request con 0x7fc2040a07d0 auth_method 0 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc1fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc1fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc1fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient(hunting): handle_auth_done global_id 15049 payload 274 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 -- 172.21.15.150:0/3224868596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2040a07d0 msgr2=0x7fc2040a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2040a07d0 0x7fc2040a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 -- 172.21.15.150:0/3224868596 >> v1:172.21.15.142:6789/0 conn(0x7fc204136a10 legacy=0x7fc204138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 monclient: found mon.noname-c 2024-09-18T10:12:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:52.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 -- 172.21.15.150:0/3224868596 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20409eaa0 con 0x7fc20409e6d0 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 -- 172.21.15.150:0/3224868596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20409e6d0 msgr2=0x7fc2040a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.480+0000 7fc202ffd700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20409e6d0 0x7fc2040a0290 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40099e0 tx=0x7fc1f4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac7cf025cb2781f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 1 -- 172.21.15.150:0/3224868596 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f400ba30 con 0x7fc20409e6d0 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: dump: 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.494 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 5 monclient: authenticate success, global_id 15049 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 1 -- 172.21.15.150:0/3224868596 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f400bb90 con 0x7fc20409e6d0 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 1 -- 172.21.15.150:0/3224868596 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4011420 con 0x7fc20409e6d0 2024-09-18T10:12:52.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 4 set_mon_vals no callback set 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2027fc700 10 monclient: dump: 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.496 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: shutdown 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- 172.21.15.150:0/3224868596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20409e6d0 msgr2=0x7fc2040a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20409e6d0 0x7fc2040a0290 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40099e0 tx=0x7fc1f4009250 comp rx=0 tx=0).stop 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- 172.21.15.150:0/3224868596 shutdown_connections 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2040a07d0 0x7fc2040a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- 172.21.15.150:0/3224868596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20409e6d0 0x7fc2040a0290 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- 172.21.15.150:0/3224868596 >> 172.21.15.150:0/3224868596 conn(0x7fc2040a58a0 msgr2=0x7fc2040a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- 172.21.15.150:0/3224868596 shutdown_connections 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- 172.21.15.150:0/3224868596 wait complete. 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: monmap: 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:52.498 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 Processor -- start 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- start start 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: init 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 0x7fc20412c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-a con 0x7fc2040a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:52.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-c con 0x7fc204136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): picked mon.noname-b con 0x7fc20412cb10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc204130930 con 0x7fc2040a07d0 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20409bec0 con 0x7fc20412cb10 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20412f9e0 con 0x7fc204136a10 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc20969a700 10 monclient(hunting): authenticate will time out at 1054.583367s 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 0x7fc20412c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37258/0 (socket says 172.21.15.150:37258) 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): get_auth_request con 0x7fc2040a07d0 auth_method 0 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 1 -- 172.21.15.150:0/1014432329 learned_addr learned my addr 172.21.15.150:0/1014432329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): get_auth_request con 0x7fc20412cb10 auth_method 0 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 10 monclient(hunting): get_auth_request con 0x7fc204136a10 auth_method 0 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc202ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 msgr2=0x7fc20412c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 msgr2=0x7fc20412c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc2037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.484+0000 7fc1fa7fc700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 10 monclient(hunting): handle_auth_done global_id 5322 payload 824 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 msgr2=0x7fc20412c5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 msgr2=0x7fc20412cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f40094f0 con 0x7fc2040a07d0 2024-09-18T10:12:52.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc2037fe700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:52.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 msgr2=0x7fc20412c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc202ffd700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 0x7fc20412c090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4002000 tx=0x7fc1f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4011420 con 0x7fc2040a07d0 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: dump: 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.506 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 5 monclient: authenticate success, global_id 5322 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4020070 con 0x7fc2040a07d0 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 10 monclient: _renew_subs 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20412d480 con 0x7fc2040a07d0 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f401b450 con 0x7fc2040a07d0 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc1fb7fe700 10 monclient: dump: 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc201ffb700 4 set_mon_vals no callback set 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 10 monclient: _renew_subs 2024-09-18T10:12:52.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.488+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20412e7a0 con 0x7fc2040a07d0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc1f4022030 con 0x7fc2040a07d0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc20969a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc20969a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc20969a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20412f810 con 0x7fc2040a07d0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc1fb7fe700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 0x7fc1e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 40..93) v4 ==== 5805+0+0 (secure 0 0 0) 0x7fc1f404a5b0 con 0x7fc2040a07d0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc1fa7fc700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 0x7fc1e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.492+0000 7fc1fa7fc700 10 monclient: get_auth_request con 0x7fc1e4035ac0 auth_method 0 2024-09-18T10:12:52.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.496+0000 7fc1fa7fc700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 0x7fc1e4037f70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.504+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc1f401b9d0 con 0x7fc2040a07d0 2024-09-18T10:12:52.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.504+0000 7fc1fb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.504+0000 7fc1fb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:52.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.540+0000 7f39c1673700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:52.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.540+0000 7f39c1673700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:52.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.540+0000 7f39c1673700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.540+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f39bc0678a0 con 0x7f39bc09df70 2024-09-18T10:12:52.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39b8ff9700 1 -- 172.21.15.142:0/692507479 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f39b4011390 con 0x7f39bc09df70 2024-09-18T10:12:52.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39b8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:52.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39b8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:52.549 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 msgr2=0x7f399c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 0x7f399c037f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0015e0 tx=0x7f39ac009040 comp rx=0 tx=0).stop 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 10 monclient: shutdown 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 msgr2=0x7f39bc08db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc08db10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f39b4016000 tx=0x7f39b4019000 comp rx=0 tx=0).stop 2024-09-18T10:12:52.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 shutdown_connections 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f399c035ac0 0x7f399c037f70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39bc09df70 0x7f39bc08db10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39bc09a380 0x7f39bc08d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 --2- 172.21.15.142:0/692507479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39bc070ef0 0x7f39bc08d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 >> 172.21.15.142:0/692507479 conn(0x7f39bc06a2e0 msgr2=0x7f39bc09b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 shutdown_connections 2024-09-18T10:12:52.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.544+0000 7f39c1673700 1 -- 172.21.15.142:0/692507479 wait complete. 2024-09-18T10:12:52.565 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:12:52.565 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:12:52.566 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:12:52.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 10 monclient: get_monmap_and_config 2024-09-18T10:12:52.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 10 monclient: monmap: 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:52.670829+0000 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:52.670829+0000 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:52.672 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 1 Processor -- start 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 1 -- start start 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.664+0000 7f4deed58700 10 monclient: init 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4de806f3b0 0x7f4de80989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-a con 0x7f4de806f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4de8098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4de8099320 0x7f4de8136d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-c con 0x7f4de8099320 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de809c330 con 0x7f4de8098f00 2024-09-18T10:12:52.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de809b030 con 0x7f4de806f3b0 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de809c530 con 0x7f4de8099320 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4deed58700 10 monclient(hunting): authenticate will time out at 1061.596261s 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4ded555700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4de8098f00 0x7f4de8133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51462/0 (socket says 172.21.15.142:51462) 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4ded555700 1 -- 172.21.15.142:0/2468920153 learned_addr learned my addr 172.21.15.142:0/2468920153 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2468920153 >> v2:172.21.15.142:3300/0 conn(0x7f4de806f3b0 0x7f4de80989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): get_auth_request con 0x7f4de806f3b0 auth_method 0 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 1 --2- 172.21.15.142:0/2468920153 >> v2:172.21.15.150:3300/0 conn(0x7f4de8099320 0x7f4de8136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): get_auth_request con 0x7f4de8099320 auth_method 0 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dee557700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient(hunting): handle_auth_done global_id 5325 payload 274 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 -- 172.21.15.142:0/2468920153 >> v2:172.21.15.150:3300/0 conn(0x7f4de8099320 msgr2=0x7f4de8136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2468920153 >> v2:172.21.15.150:3300/0 conn(0x7f4de8099320 0x7f4de8136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 -- 172.21.15.142:0/2468920153 >> v1:172.21.15.142:6789/0 conn(0x7f4de8098f00 legacy=0x7f4de8133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 -- 172.21.15.142:0/2468920153 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de809c0f0 con 0x7f4de806f3b0 2024-09-18T10:12:52.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.672+0000 7fc20969a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:52.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.672+0000 7fc20969a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:52.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.672+0000 7fc20969a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:52.672+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7fc20412f810 con 0x7fc2040a07d0 2024-09-18T10:12:52.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 -- 172.21.15.142:0/2468920153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806f3b0 msgr2=0x7f4de80989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.668+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2468920153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806f3b0 0x7f4de80989c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4de809a7f0 tx=0x7f4ddc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6078b03da9493 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 1 -- 172.21.15.142:0/2468920153 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc011840 con 0x7f4de806f3b0 2024-09-18T10:12:52.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: dump: 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 1 -- 172.21.15.142:0/2468920153 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc0119a0 con 0x7f4de806f3b0 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 5 monclient: authenticate success, global_id 5325 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 1 -- 172.21.15.142:0/2468920153 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc011d40 con 0x7f4de806f3b0 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 4 set_mon_vals no callback set 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4decd54700 10 monclient: dump: 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.686 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 10 monclient: shutdown 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 -- 172.21.15.142:0/2468920153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806f3b0 msgr2=0x7f4de80989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 --2- 172.21.15.142:0/2468920153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806f3b0 0x7f4de80989c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4de809a7f0 tx=0x7f4ddc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 -- 172.21.15.142:0/2468920153 shutdown_connections 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 --2- 172.21.15.142:0/2468920153 >> v2:172.21.15.150:3300/0 conn(0x7f4de8099320 0x7f4de8136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 --2- 172.21.15.142:0/2468920153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806f3b0 0x7f4de80989c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 -- 172.21.15.142:0/2468920153 >> 172.21.15.142:0/2468920153 conn(0x7f4de8070ef0 msgr2=0x7f4de80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 -- 172.21.15.142:0/2468920153 shutdown_connections 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 -- 172.21.15.142:0/2468920153 wait complete. 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 10 monclient: monmap: 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.688 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.672+0000 7f4deed58700 1 Processor -- start 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 -- start start 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient: init 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-c con 0x7f4de806f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 0x7f4de813e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-a con 0x7f4de8098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de8099320 0x7f4de813ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): picked mon.noname-b con 0x7f4de8099320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de808e7d0 con 0x7f4de8098f00 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de806fa20 con 0x7f4de8099320 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8097300 con 0x7f4de806f3b0 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4deed58700 10 monclient(hunting): authenticate will time out at 1061.605296s 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 0x7f4de813e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44986/0 (socket says 172.21.15.142:44986) 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): get_auth_request con 0x7f4de8098f00 auth_method 0 2024-09-18T10:12:52.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 learned_addr learned my addr 172.21.15.142:0/2416611702 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 10 monclient(hunting): get_auth_request con 0x7f4de806f3b0 auth_method 0 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 msgr2=0x7f4de808f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 msgr2=0x7f4de808f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:52.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dee557700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de8099320 0x7f4de813ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dee557700 10 monclient(hunting): get_auth_request con 0x7f4de8099320 auth_method 0 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dee557700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dee557700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dee557700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient(hunting): handle_auth_done global_id 5328 payload 824 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 msgr2=0x7f4de808f420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4de806f3b0 0x7f4de808f420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de8099320 msgr2=0x7f4de813ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4de8099320 0x7f4de813ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.676+0000 7f4ded555700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc009420 con 0x7f4de8098f00 2024-09-18T10:12:52.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4ded555700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 msgr2=0x7f4de813e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4ded555700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 0x7f4de813e940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800cbe0 tx=0x7f4dd80053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd800ab70 con 0x7f4de8098f00 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: dump: 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.696 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 5 monclient: authenticate success, global_id 5328 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd800acf0 con 0x7f4de8098f00 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 monclient: _renew_subs 2024-09-18T10:12:52.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8007340 con 0x7f4de8098f00 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de7fff700 4 set_mon_vals no callback set 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de808c650 con 0x7f4de8098f00 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 10 monclient: dump: 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.698 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 monclient: _renew_subs 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de813f3c0 con 0x7f4de8098f00 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 0x7f4de806d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de808e320 con 0x7f4de806d380 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 0x7f4de806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient: get_auth_request con 0x7f4de806d380 auth_method 0 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 10 monclient(hunting): handle_auth_done global_id 5331 payload 824 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4de808e4b0 con 0x7f4de806d380 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4dd80076f0 con 0x7f4de8098f00 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4dd4035ac0 0x7f4dd4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 40..93) v4 ==== 5805+0+0 (secure 0 0 0) 0x7f4dd8048cf0 con 0x7f4de8098f00 2024-09-18T10:12:52.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dee557700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4dd4035ac0 0x7f4dd4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.680+0000 7f4dee557700 10 monclient: get_auth_request con 0x7f4dd4035ac0 auth_method 0 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 msgr2=0x7f4de806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 0x7f4de806d750 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4de80952e0 tx=0x7f4ddc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc00bd00 con 0x7f4de806d380 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 10 monclient: dump: 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.701 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4ddc00be60 con 0x7f4de806d380 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 msgr2=0x7f4de806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 0x7f4de806d750 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4de80952e0 tx=0x7f4ddc016000 comp rx=0 tx=0).stop 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 msgr2=0x7f4de806d750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4de5ffb700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806d380 0x7f4de806d750 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.684+0000 7f4dee557700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4dd4035ac0 0x7f4dd4037f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009d40 tx=0x7f4de0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4deed58700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:52.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4deed58700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 0x7f4de808aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4deed58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80648a0 con 0x7f4de806dc90 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 0x7f4de808aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient: get_auth_request con 0x7f4de806dc90 auth_method 0 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.692+0000 7f4dedd56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dedd56700 10 monclient(hunting): handle_auth_done global_id 5334 payload 824 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4de8072a80 con 0x7f4de806dc90 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dedd56700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 msgr2=0x7f4de808aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dedd56700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 0x7f4de808aba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc026000 tx=0x7f4ddc01f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc01cc40 con 0x7f4de806dc90 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 10 monclient: dump: 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.706 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f4ddc00baf0 con 0x7f4de806dc90 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 msgr2=0x7f4de808aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 0x7f4de808aba0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc026000 tx=0x7f4ddc01f040 comp rx=0 tx=0).stop 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 msgr2=0x7f4de808aba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4de5ffb700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de806dc90 0x7f4de808aba0 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc026000 tx=0x7f4ddc01f040 comp rx=0 tx=0).stop 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:52.707 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:52.708 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:52.709 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:52.710 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:52.711 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.712 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.714 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:52.715 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:52.718 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:52.719 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:52.720 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:52.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4dd4035ac0 msgr2=0x7f4dd4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4dd4035ac0 0x7f4dd4037f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009d40 tx=0x7f4de0008040 comp rx=0 tx=0).stop 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 10 monclient: shutdown 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 msgr2=0x7f4de813e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 --2- 172.21.15.142:0/2416611702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4de8098f00 0x7f4de813e940 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800cbe0 tx=0x7f4dd80053c0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dee557700 1 -- 172.21.15.142:0/2416611702 reap_dead start 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4dee557700 1 -- 172.21.15.142:0/2416611702 reap_dead start 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 shutdown_connections 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 >> 172.21.15.142:0/2416611702 conn(0x7f4de8070ef0 msgr2=0x7f4de80a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 shutdown_connections 2024-09-18T10:12:52.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.696+0000 7f4deed58700 1 -- 172.21.15.142:0/2416611702 wait complete. 2024-09-18T10:12:52.723 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:52.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:12:52.723 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:12:52.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: get_monmap_and_config 2024-09-18T10:12:52.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: monmap: 2024-09-18T10:12:52.832 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.833 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.833 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:52.830174+0000 2024-09-18T10:12:52.833 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:52.830174+0000 2024-09-18T10:12:52.834 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.834 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.834 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:52.834 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 1 Processor -- start 2024-09-18T10:12:52.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 1 -- start start 2024-09-18T10:12:52.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: init 2024-09-18T10:12:52.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 0x7fa7c40954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.824+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-c con 0x7fa7c409d870 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7c40989c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7c40994d0 0x7fa7c4136680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-a con 0x7fa7c40994d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c4069c70 con 0x7fa7c40989c0 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c406f9d0 con 0x7fa7c40994d0 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c409dc40 con 0x7fa7c409d870 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient(hunting): authenticate will time out at 1061.756289s 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7c40994d0 0x7fa7c4136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7c40994d0 0x7fa7c4136680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45806/0 (socket says 172.21.15.142:45806) 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c37fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa7c40989c0 0x7fa7c4098dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51466/0 (socket says 172.21.15.142:51466) 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1460897017 learned_addr learned my addr 172.21.15.142:0/1460897017 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): get_auth_request con 0x7fa7c40994d0 auth_method 0 2024-09-18T10:12:52.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1460897017 >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 0x7fa7c40954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request con 0x7fa7c409d870 auth_method 0 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_done global_id 5337 payload 274 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1460897017 >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 msgr2=0x7fa7c40954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1460897017 >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 0x7fa7c40954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1460897017 >> v1:172.21.15.142:6789/0 conn(0x7fa7c40989c0 legacy=0x7fa7c4098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1460897017 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c406fc20 con 0x7fa7c40994d0 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1460897017 >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 0x7fa7c40954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1460897017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c40994d0 msgr2=0x7fa7c4136680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1460897017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c40994d0 0x7fa7c4136680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4076e90 tx=0x7fa7b400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d31fc6e44c781c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 1 -- 172.21.15.142:0/1460897017 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400de60 con 0x7fa7c40994d0 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: dump: 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.847 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 1 -- 172.21.15.142:0/1460897017 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4010470 con 0x7fa7c40994d0 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 5 monclient: authenticate success, global_id 5337 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 1 -- 172.21.15.142:0/1460897017 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4010810 con 0x7fa7c40994d0 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7ca58e700 4 set_mon_vals no callback set 2024-09-18T10:12:52.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.828+0000 7fa7c2ffd700 10 monclient: dump: 2024-09-18T10:12:52.850 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.853 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: shutdown 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1460897017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c40994d0 msgr2=0x7fa7c4136680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1460897017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c40994d0 0x7fa7c4136680 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4076e90 tx=0x7fa7b400b4f0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1460897017 shutdown_connections 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1460897017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c40994d0 0x7fa7c4136680 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1460897017 >> v2:172.21.15.150:3300/0 conn(0x7fa7c409d870 0x7fa7c40954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1460897017 >> 172.21.15.142:0/1460897017 conn(0x7fa7c40a14e0 msgr2=0x7fa7c40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:52.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1460897017 shutdown_connections 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1460897017 wait complete. 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: build_initial_monmap 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: monmap: 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 Processor -- start 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- start start 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: init 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c40989c0 0x7fa7c408dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-b con 0x7fa7c40989c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 0x7fa7c408e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-c con 0x7fa7c40994d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): picked mon.noname-a con 0x7fa7c409d870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c408fb20 con 0x7fa7c409d870 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4096f80 con 0x7fa7c40989c0 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4092380 con 0x7fa7c40994d0 2024-09-18T10:12:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7ca58e700 10 monclient(hunting): authenticate will time out at 1061.763518s 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 0x7fa7c408e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45822/0 (socket says 172.21.15.142:45822) 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 learned_addr learned my addr 172.21.15.142:0/1386358379 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): get_auth_request con 0x7fa7c409d870 auth_method 0 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c37fe700 10 monclient(hunting): get_auth_request con 0x7fa7c40994d0 auth_method 0 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c40989c0 0x7fa7c408dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.832+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request con 0x7fa7c40989c0 auth_method 0 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c37fe700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 msgr2=0x7fa7c408e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c37fe700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 msgr2=0x7fa7c408e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c37fe700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 0x7fa7c408e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c37fe700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 0x7fa7c408e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 10 monclient(hunting): handle_auth_done global_id 5340 payload 824 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 msgr2=0x7fa7c408e220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c40994d0 0x7fa7c408e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c40989c0 msgr2=0x7fa7c408dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7c40989c0 0x7fa7c408dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 monclient: found mon.noname-a 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b400b790 con 0x7fa7c409d870 2024-09-18T10:12:52.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 msgr2=0x7fa7c408e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:52.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40030c0 tx=0x7fa7b401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:52.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4006a90 con 0x7fa7c409d870 2024-09-18T10:12:52.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: dump: 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.864 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: _finish_auth 0 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 5 monclient: authenticate success, global_id 5340 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4019950 con 0x7fa7c409d870 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 10 monclient: _renew_subs 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c40910d0 con 0x7fa7c409d870 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4019d10 con 0x7fa7c409d870 2024-09-18T10:12:52.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7bbfff700 10 monclient: dump: 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:52.866 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7c27fc700 4 set_mon_vals no callback set 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 10 monclient: _renew_subs 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.836+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c4090170 con 0x7fa7c409d870 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7ca58e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4091550 con 0x7fa7c408eca0 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 10 monclient: get_auth_request con 0x7fa7c408eca0 auth_method 0 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c8b2b700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40030c0 tx=0x7fa7b401a000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fa7b4022600 mgrmap(e 5) v1 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa7b4022600 con 0x7fa7c409d870 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7bbfff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7a4035ac0 0x7fa7a4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:52.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c37fe700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7a4035ac0 0x7fa7a4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:52.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.840+0000 7fa7c37fe700 10 monclient: get_auth_request con 0x7fa7a4035ac0 auth_method 0 2024-09-18T10:12:52.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:52.844+0000 7fa7c37fe700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7a4035ac0 0x7fa7a4037f70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7c409d750 tx=0x7fa7bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 10 monclient: get_auth_request con 0x7fa7c408eca0 auth_method 0 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:53.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:53.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:53.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:53.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.044+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:12:53.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.252+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 removed item id -16 name 'host2' from crush map v94) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc1f402ee60 con 0x7fc2040a07d0 2024-09-18T10:12:53.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.252+0000 7fc1fb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:53.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.252+0000 7fc1fb7fe700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'host2' from crush map 2024-09-18T10:12:53.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc20969a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:53.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc20969a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:53.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc20969a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:53.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "host2"} v 0) v1 -- 0x7fc20412f810 con 0x7fc2040a07d0 2024-09-18T10:12:53.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc1fb7fe700 1 -- 172.21.15.150:0/1014432329 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "host2"}]=0 device 'host2' does not appear in the crush map v94) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc1f402ee60 con 0x7fc2040a07d0 2024-09-18T10:12:53.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc1fb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "host2"}] 2024-09-18T10:12:53.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.360+0000 7fc1fb7fe700 10 monclient: _finish_command 3 = system:0 device 'host2' does not appear in the crush map 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 msgr2=0x7fc1e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 0x7fc1e4037f70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec008040 comp rx=0 tx=0).stop 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 10 monclient: shutdown 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 msgr2=0x7fc20412c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 0x7fc20412c090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4002000 tx=0x7fc1f4016000 comp rx=0 tx=0).stop 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 shutdown_connections 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1e4035ac0 0x7fc1e4037f70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20412cb10 0x7fc20412cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc204136a10 0x7fc20412c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 --2- 172.21.15.150:0/1014432329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2040a07d0 0x7fc20412c090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 >> 172.21.15.150:0/1014432329 conn(0x7fc2040a58a0 msgr2=0x7fc2040a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 shutdown_connections 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.364+0000 7fc20969a700 1 -- 172.21.15.150:0/1014432329 wait complete. 2024-09-18T10:12:53.371 INFO:tasks.workunit.client.0.smithi150.stderr:device 'host2' does not appear in the crush map 2024-09-18T10:12:53.380 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket foo host 2024-09-18T10:12:53.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 10 monclient: get_auth_request con 0x7fa7c408eca0 auth_method 0 2024-09-18T10:12:53.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:53.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:12:53.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:53.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:53.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.448+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:12:53.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: get_monmap_and_config 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: build_initial_monmap 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: monmap: 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:53.481403+0000 2024-09-18T10:12:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:53.481403+0000 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:53.483 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:53.484 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 1 Processor -- start 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 1 -- start start 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: init 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-b con 0x7fb67809e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb67809e740 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.476+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb678135590 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6780a00e0 con 0x7fb678135590 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67806d770 con 0x7fb67809e740 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67809f220 con 0x7fb67809e370 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 10 monclient(hunting): authenticate will time out at 1055.576126s 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:54982/0 (socket says 172.21.15.150:54982) 2024-09-18T10:12:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 -- 172.21.15.150:0/1247131854 learned_addr learned my addr 172.21.15.150:0/1247131854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request con 0x7fb67809e370 auth_method 0 2024-09-18T10:12:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 msgr2=0x7fb67806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 msgr2=0x7fb67806e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 >> v1:172.21.15.142:6789/0 conn(0x7fb678135590 legacy=0x7fb678137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67c9d0700 1 -- 172.21.15.150:0/1247131854 >> v1:172.21.15.150:6789/0 conn(0x7fb67809e740 legacy=0x7fb67809eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376166447 0 0) 0x7fb6780a00e0 con 0x7fb678135590 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): my global_id is 5343 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664003570 con 0x7fb678135590 2024-09-18T10:12:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314651833 0 0) 0x7fb67806d770 con 0x7fb67809e740 2024-09-18T10:12:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): my global_id is 15067 2024-09-18T10:12:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6780a00e0 con 0x7fb67809e740 2024-09-18T10:12:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110136319 0 0) 0x7fb6780a00e0 con 0x7fb67809e740 2024-09-18T10:12:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 msgr2=0x7fb67806e760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb67806e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 >> v1:172.21.15.142:6789/0 conn(0x7fb678135590 legacy=0x7fb678137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67809fe50 con 0x7fb67809e740 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: _finish_auth 0 2024-09-18T10:12:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 5 monclient: authenticate success, global_id 15067 2024-09-18T10:12:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb67f435700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fb668001890 con 0x7fb67809e740 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: dump: 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:53.517 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 >> v1:172.21.15.150:6789/0 conn(0x7fb67809e740 legacy=0x7fb67809eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb664003c70 0x7fb664004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): picked mon.a con 0x7fb664003c70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb664004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): picked mon.c con 0x7fb67809e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 0x7fb6640052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): picked mon.b con 0x7fb664004e70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.480+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb674001630 con 0x7fb664003c70 2024-09-18T10:12:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67809f220 con 0x7fb664004e70 2024-09-18T10:12:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66400bd90 con 0x7fb67809e370 2024-09-18T10:12:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67f435700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 0x7fb6640052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): get_auth_request con 0x7fb664004e70 auth_method 0 2024-09-18T10:12:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb664003c70 0x7fb664004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb664004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request con 0x7fb664003c70 auth_method 0 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): get_auth_request con 0x7fb67809e370 auth_method 0 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient(hunting): handle_auth_done global_id 15067 payload 293 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 msgr2=0x7fb664004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb67809e370 0x7fb664004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb664003c70 msgr2=0x7fb664004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb664003c70 0x7fb664004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient: _finish_hunting 0 2024-09-18T10:12:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 monclient: found mon.b 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6780a00e0 con 0x7fb664004e70 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 10 monclient: _finish_auth 0 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 msgr2=0x7fb6640052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 0x7fb6640052e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c5c0 tx=0x7fb674007ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72476cf39567d65a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400ce10 con 0x7fb664004e70 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: dump: 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6740070d0 con 0x7fb664004e70 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67f435700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67f435700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb67f435700 4 set_mon_vals no callback set 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 1 -- 172.21.15.150:0/1247131854 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674007490 con 0x7fb664004e70 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.484+0000 7fb66ffff700 10 monclient: dump: 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: shutdown 2024-09-18T10:12:53.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 msgr2=0x7fb6640052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 --2- 172.21.15.150:0/1247131854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb664004e70 0x7fb6640052e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c5c0 tx=0x7fb674007ad0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 -- 172.21.15.150:0/1247131854 reap_dead start 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- 172.21.15.150:0/1247131854 shutdown_connections 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- 172.21.15.150:0/1247131854 >> 172.21.15.150:0/1247131854 conn(0x7fb678071850 msgr2=0x7fb6780a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- 172.21.15.150:0/1247131854 shutdown_connections 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- 172.21.15.150:0/1247131854 wait complete. 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: build_initial_monmap 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: monmap: 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:53.531 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 Processor -- start 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- start start 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: init 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 0x7fb67811d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-a con 0x7fb67811d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-b con 0x7fb67811d910 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): picked mon.noname-c con 0x7fb678143a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678120430 con 0x7fb67811d000 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67811e220 con 0x7fb67811d910 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb678120960 con 0x7fb678143a20 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67f435700 10 monclient(hunting): authenticate will time out at 1055.586791s 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 0x7fb67811d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37266/0 (socket says 172.21.15.150:37266) 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 -- 172.21.15.150:0/4038258337 learned_addr learned my addr 172.21.15.150:0/4038258337 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): get_auth_request con 0x7fb67811d910 auth_method 0 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request con 0x7fb67811d000 auth_method 0 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:53.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:53.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:53.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:53.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 10 monclient(hunting): get_auth_request con 0x7fb678143a20 auth_method 0 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 msgr2=0x7fb678145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 msgr2=0x7fb678145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:12:53.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d9d2700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient(hunting): handle_auth_done global_id 15070 payload 824 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 msgr2=0x7fb678145e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 msgr2=0x7fb67811d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 0x7fb67811d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient: _finish_hunting 0 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 monclient: found mon.noname-b 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67c9d0700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67806ef10 con 0x7fb67811d910 2024-09-18T10:12:53.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.488+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 0x7fb67811d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67c9d0700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 msgr2=0x7fb67811dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67c9d0700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb6680070c0 tx=0x7fb6680079d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668013810 con 0x7fb67811d910 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: dump: 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.539 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb668013970 con 0x7fb67811d910 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 5 monclient: authenticate success, global_id 15070 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 10 monclient: _renew_subs 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67809dc20 con 0x7fb67811d910 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668013d10 con 0x7fb67811d910 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66d7fa700 10 monclient: dump: 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:53.541 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 10 monclient: _renew_subs 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67811f3d0 con 0x7fb67811d910 2024-09-18T10:12:53.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.492+0000 7fb66f7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb66800f5f0 con 0x7fb67811d910 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67f435700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67f435700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67f435700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67811e920 con 0x7fb67811d910 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb66d7fa700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 0x7fb65c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(94..94 src has 40..94) v4 ==== 5670+0+0 (secure 0 0 0) 0x7fb668048550 con 0x7fb67811d910 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 0x7fb65c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67d1d1700 10 monclient: get_auth_request con 0x7fb65c035ac0 auth_method 0 2024-09-18T10:12:53.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.496+0000 7fb67d1d1700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 0x7fb65c037f70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb670009410 tx=0x7fb6700091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:53.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.504+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb668019880 con 0x7fb67811d910 2024-09-18T10:12:53.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.504+0000 7fb66d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:53.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.504+0000 7fb66d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.668+0000 7fb67f435700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.668+0000 7fb67f435700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.668+0000 7fb67f435700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:53.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:53.668+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fb67809c740 con 0x7fb67811d910 2024-09-18T10:12:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:53.672+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(93..93 src has 40..93) v4 ==== 5805+0+0 (secure 0 0 0) 0x7fa7b4014070 con 0x7fa7c409d870 2024-09-18T10:12:54.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient: get_auth_request con 0x7fa7c408eca0 auth_method 0 2024-09-18T10:12:54.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:54.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_done global_id 44099 payload 824 2024-09-18T10:12:54.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7c40696a0 con 0x7fa7c408eca0 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.252+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac015d00 con 0x7fa7c408eca0 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 10 monclient: dump: 2024-09-18T10:12:54.260 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa7ac01c070 con 0x7fa7c408eca0 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).stop 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 msgr2=0x7fa7c406d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:54.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.256+0000 7fa7bbfff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c408eca0 0x7fa7c406d410 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac006fd0 tx=0x7fa7ac008040 comp rx=0 tx=0).stop 2024-09-18T10:12:54.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.260+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 added bucket foo type host to crush map v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb6680170a0 con 0x7fb67811d910 2024-09-18T10:12:54.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.260+0000 7fb66d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:54.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.260+0000 7fb66d7fa700 10 monclient: _finish_command 2 = system:0 added bucket foo type host to crush map 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7ca58e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 0x7fa7c406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7ca58e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4064950 con 0x7fa7c406d950 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 0x7fa7c406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient: get_auth_request con 0x7fa7c406d950 auth_method 0 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 10 monclient(hunting): handle_auth_done global_id 44102 payload 824 2024-09-18T10:12:54.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.264+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7c4067950 con 0x7fa7c406d950 2024-09-18T10:12:54.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7c3fff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 msgr2=0x7fa7c406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7c3fff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 0x7fa7c406dd50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac022000 tx=0x7fa7ac020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac015e60 con 0x7fa7c406d950 2024-09-18T10:12:54.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 10 monclient: dump: 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fa7ac00c050 con 0x7fa7c406d950 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 msgr2=0x7fa7c406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 0x7fa7c406dd50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac022000 tx=0x7fa7ac020000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 msgr2=0x7fa7c406dd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7bbfff700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7c406d950 0x7fa7c406dd50 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac022000 tx=0x7fa7ac020000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:54.277 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.278 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.279 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:54.280 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.281 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:54.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:54.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.288 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.288 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:54.288 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:54.289 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:54.291 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:54.292 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7a4035ac0 msgr2=0x7fa7a4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7a4035ac0 0x7fa7a4037f70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7c409d750 tx=0x7fa7bc009040 comp rx=0 tx=0).stop 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 10 monclient: shutdown 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 msgr2=0x7fa7c408e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7c8b2b700 1 -- 172.21.15.142:0/1386358379 reap_dead start 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40030c0 tx=0x7fa7b401a000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 shutdown_connections 2024-09-18T10:12:54.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 --2- 172.21.15.142:0/1386358379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7c409d870 0x7fa7c408e760 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 >> 172.21.15.142:0/1386358379 conn(0x7fa7c40a14e0 msgr2=0x7fa7c4095a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 shutdown_connections 2024-09-18T10:12:54.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.268+0000 7fa7ca58e700 1 -- 172.21.15.142:0/1386358379 wait complete. 2024-09-18T10:12:54.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:54.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:12:54.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:12:54.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.368+0000 7fb67f435700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:54.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.368+0000 7fb67f435700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:54.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.368+0000 7fb67f435700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.368+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "foo", "type": "host"} v 0) v1 -- 0x7fb678063ed0 con 0x7fb67811d910 2024-09-18T10:12:54.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb66d7fa700 1 -- 172.21.15.150:0/4038258337 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}]=0 bucket 'foo' already exists v95) v1 ==== 126+0+0 (secure 0 0 0) 0x7fb668019880 con 0x7fb67811d910 2024-09-18T10:12:54.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb66d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "foo", "type": "host"}] 2024-09-18T10:12:54.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb66d7fa700 10 monclient: _finish_command 3 = system:0 bucket 'foo' already exists 2024-09-18T10:12:54.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 msgr2=0x7fb65c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 0x7fb65c037f70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb670009410 tx=0x7fb6700091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 10 monclient: shutdown 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 msgr2=0x7fb67811dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb6680070c0 tx=0x7fb6680079d0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 shutdown_connections 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb65c035ac0 0x7fb65c037f70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb678143a20 0x7fb678145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb67811d910 0x7fb67811dce0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 --2- 172.21.15.150:0/4038258337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67811d000 0x7fb67811d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 >> 172.21.15.150:0/4038258337 conn(0x7fb678071850 msgr2=0x7fb6780a2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 shutdown_connections 2024-09-18T10:12:54.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.372+0000 7fb67f435700 1 -- 172.21.15.150:0/4038258337 wait complete. 2024-09-18T10:12:54.381 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'foo' already exists 2024-09-18T10:12:54.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush move foo root=default rack=localrack 2024-09-18T10:12:54.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 10 monclient: get_monmap_and_config 2024-09-18T10:12:54.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 10 monclient: monmap: 2024-09-18T10:12:54.407 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:54.407 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:54.406662+0000 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:54.406662+0000 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:54.408 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 1 Processor -- start 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 1 -- start start 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.400+0000 7fb404319700 10 monclient: init 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3fc09d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb3fc0954b0 0x7fb3fc0990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): picked mon.noname-a con 0x7fb3fc0954b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:54.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3fc134670 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc069c70 con 0x7fb3fc134670 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0958b0 con 0x7fb3fc09d870 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc06f9d0 con 0x7fb3fc0954b0 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb404319700 10 monclient(hunting): authenticate will time out at 1063.332146s 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb3fc0954b0 0x7fb3fc0990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb3fc0954b0 0x7fb3fc0990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45826/0 (socket says 172.21.15.142:45826) 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 -- 172.21.15.142:0/623373305 learned_addr learned my addr 172.21.15.142:0/623373305 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): get_auth_request con 0x7fb3fc0954b0 auth_method 0 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4028b6700 1 -- 172.21.15.142:0/623373305 >> v1:172.21.15.142:6789/0 conn(0x7fb3fc134670 legacy=0x7fb3fc136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4020b5700 1 -- 172.21.15.142:0/623373305 >> v1:172.21.15.150:6789/0 conn(0x7fb3fc09d870 legacy=0x7fb3fc0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient(hunting): handle_auth_done global_id 5349 payload 274 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 -- 172.21.15.142:0/623373305 >> v1:172.21.15.150:6789/0 conn(0x7fb3fc09d870 legacy=0x7fb3fc0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 -- 172.21.15.142:0/623373305 >> v1:172.21.15.142:6789/0 conn(0x7fb3fc134670 legacy=0x7fb3fc136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 monclient: found mon.noname-a 2024-09-18T10:12:54.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4010b3700 1 -- 172.21.15.142:0/623373305 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758942235 0 0) 0x7fb3fc069c70 con 0x7fb3fc134670 2024-09-18T10:12:54.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:54.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4018b4700 1 -- 172.21.15.142:0/623373305 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc09dc40 con 0x7fb3fc0954b0 2024-09-18T10:12:54.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.404+0000 7fb4010b3700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4018b4700 1 -- 172.21.15.142:0/623373305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc0954b0 msgr2=0x7fb3fc0990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4018b4700 1 --2- 172.21.15.142:0/623373305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc0990d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000bd50 tx=0x7fb3f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a99342cb5fd8ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 1 -- 172.21.15.142:0/623373305 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0010680 con 0x7fb3fc0954b0 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: dump: 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.416 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 1 -- 172.21.15.142:0/623373305 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f00107e0 con 0x7fb3fc0954b0 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 5 monclient: authenticate success, global_id 5349 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 1 -- 172.21.15.142:0/623373305 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0010ba0 con 0x7fb3fc0954b0 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 4 set_mon_vals no callback set 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb4010b3700 10 monclient: dump: 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.418 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 10 monclient: shutdown 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 -- 172.21.15.142:0/623373305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc0954b0 msgr2=0x7fb3fc0990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 --2- 172.21.15.142:0/623373305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc0990d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000bd50 tx=0x7fb3f000b4f0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 -- 172.21.15.142:0/623373305 shutdown_connections 2024-09-18T10:12:54.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 --2- 172.21.15.142:0/623373305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc0990d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 -- 172.21.15.142:0/623373305 >> 172.21.15.142:0/623373305 conn(0x7fb3fc0a14e0 msgr2=0x7fb3fc0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 -- 172.21.15.142:0/623373305 shutdown_connections 2024-09-18T10:12:54.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.408+0000 7fb404319700 1 -- 172.21.15.142:0/623373305 wait complete. 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient: monmap: 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:54.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 Processor -- start 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 -- start start 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient: init 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:54.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): picked mon.noname-b con 0x7fb3fc0954b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3fc09d870 0x7fb3fc06abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): picked mon.noname-c con 0x7fb3fc09d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 0x7fb3fc06b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): picked mon.noname-a con 0x7fb3fc134670 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc06e120 con 0x7fb3fc134670 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc06f320 con 0x7fb3fc0954b0 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc06e750 con 0x7fb3fc09d870 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb404319700 10 monclient(hunting): authenticate will time out at 1063.342593s 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 0x7fb3fc06b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3fc09d870 0x7fb3fc06abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 0x7fb3fc06b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45832/0 (socket says 172.21.15.142:45832) 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 learned_addr learned my addr 172.21.15.142:0/942049082 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): get_auth_request con 0x7fb3fc134670 auth_method 0 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): get_auth_request con 0x7fb3fc09d870 auth_method 0 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): get_auth_request con 0x7fb3fc0954b0 auth_method 0 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4018b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient(hunting): handle_auth_done global_id 5355 payload 824 2024-09-18T10:12:54.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3fc09d870 msgr2=0x7fb3fc06abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3fc09d870 0x7fb3fc06abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc0954b0 msgr2=0x7fb3fc06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc0954b0 0x7fb3fc06ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 monclient: found mon.noname-a 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.412+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f000b790 con 0x7fb3fc134670 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 msgr2=0x7fb3fc06b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4028b6700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 0x7fb3fc06b100 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c940 tx=0x7fb3f4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40070f0 con 0x7fb3fc134670 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: dump: 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 5 monclient: authenticate success, global_id 5355 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4007250 con 0x7fb3fc134670 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 monclient: _renew_subs 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc097b80 con 0x7fb3fc134670 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4007610 con 0x7fb3fc134670 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 10 monclient: dump: 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.433 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4008b2700 4 set_mon_vals no callback set 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 monclient: _renew_subs 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc06c630 con 0x7fb3fc134670 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 0x7fb3fc12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc064950 con 0x7fb3fc06b640 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4020b5700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 0x7fb3fc12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4020b5700 10 monclient: get_auth_request con 0x7fb3fc06b640 auth_method 0 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb3f401b400 con 0x7fb3fc134670 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4020b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4020b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.416+0000 7fb4020b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3e0035ac0 0x7fb3e0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 10 monclient(hunting): handle_auth_done global_id 15079 payload 824 2024-09-18T10:12:54.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(95..95 src has 40..95) v4 ==== 5697+0+0 (secure 0 0 0) 0x7fb3f4048400 con 0x7fb3fc134670 2024-09-18T10:12:54.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3fc0696a0 con 0x7fb3fc06b640 2024-09-18T10:12:54.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4018b4700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3e0035ac0 0x7fb3e0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4018b4700 10 monclient: get_auth_request con 0x7fb3e0035ac0 auth_method 0 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 msgr2=0x7fb3fc12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4020b5700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 0x7fb3fc12e840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800f7b0 tx=0x7fb3f8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f80074f0 con 0x7fb3fc06b640 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 10 monclient: dump: 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.437 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb4018b4700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3e0035ac0 0x7fb3e0037f70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000bd50 tx=0x7fb3f001b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb3f8007650 con 0x7fb3fc06b640 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:54.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 msgr2=0x7fb3fc12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 0x7fb3fc12e840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800f7b0 tx=0x7fb3f8009040 comp rx=0 tx=0).stop 2024-09-18T10:12:54.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 msgr2=0x7fb3fc12e840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:54.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.420+0000 7fb3ee7fc700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc06b640 0x7fb3fc12e840 secure :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800f7b0 tx=0x7fb3f8009040 comp rx=0 tx=0).stop 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb404319700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb404319700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 0x7fb3fc12f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb404319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc067950 con 0x7fb3fc12ed80 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 0x7fb3fc12f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient: get_auth_request con 0x7fb3fc12ed80 auth_method 0 2024-09-18T10:12:54.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 10 monclient(hunting): handle_auth_done global_id 15082 payload 824 2024-09-18T10:12:54.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.432+0000 7fb4020b5700 1 -- 172.21.15.142:0/942049082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3fc06ba10 con 0x7fb3fc12ed80 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb4020b5700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 msgr2=0x7fb3fc12f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb4020b5700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 0x7fb3fc12f180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800f7b0 tx=0x7fb3f8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8021620 con 0x7fb3fc12ed80 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:54.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 10 monclient: dump: 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.443 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fb3f8021780 con 0x7fb3fc12ed80 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 msgr2=0x7fb3fc12f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 0x7fb3fc12f180 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800f7b0 tx=0x7fb3f8015000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 msgr2=0x7fb3fc12f180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:54.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb3ee7fc700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3fc12ed80 0x7fb3fc12f180 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:12:54.445 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:54.448 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:54.449 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:54.454 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:54.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:54.456 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:54.457 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3e0035ac0 msgr2=0x7fb3e0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3e0035ac0 0x7fb3e0037f70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000bd50 tx=0x7fb3f001b000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 10 monclient: shutdown 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 msgr2=0x7fb3fc06b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 --2- 172.21.15.142:0/942049082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3fc134670 0x7fb3fc06b100 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400c940 tx=0x7fb3f4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 reap_dead start 2024-09-18T10:12:54.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb4028b6700 1 -- 172.21.15.142:0/942049082 reap_dead start 2024-09-18T10:12:54.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 shutdown_connections 2024-09-18T10:12:54.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 >> 172.21.15.142:0/942049082 conn(0x7fb3fc0a14e0 msgr2=0x7fb3fc133bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 shutdown_connections 2024-09-18T10:12:54.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.436+0000 7fb404319700 1 -- 172.21.15.142:0/942049082 wait complete. 2024-09-18T10:12:54.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:54.465 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:12:54.466 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:12:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: get_monmap_and_config 2024-09-18T10:12:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: monmap: 2024-09-18T10:12:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:54.492 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:54.492195+0000 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:54.492195+0000 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:54.493 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 Processor -- start 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 -- start start 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: init 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0290 0x7fd63409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-b con 0x7fd6340a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6340a0660 0x7fd6340a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-c con 0x7fd6340a0660 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd634136a10 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63409e360 con 0x7fd634136a10 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63409ef80 con 0x7fd6340a0660 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6340a1fb0 con 0x7fd6340a0290 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd63945a700 10 monclient(hunting): authenticate will time out at 1056.585751s 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6337fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd634136a10 0x7fd634138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34440/0 (socket says 172.21.15.150:34440) 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6337fe700 1 -- 172.21.15.150:0/2535374504 learned_addr learned my addr 172.21.15.150:0/2535374504 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:54.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 --2- 172.21.15.150:0/2535374504 >> v2:172.21.15.150:3300/0 conn(0x7fd6340a0660 0x7fd6340a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): get_auth_request con 0x7fd6340a0660 auth_method 0 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0290 0x7fd63409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): get_auth_request con 0x7fd6340a0290 auth_method 0 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd632ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient(hunting): handle_auth_done global_id 15085 payload 274 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 -- 172.21.15.150:0/2535374504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0290 msgr2=0x7fd63409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0290 0x7fd63409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 -- 172.21.15.150:0/2535374504 >> v1:172.21.15.150:6789/0 conn(0x7fd634136a10 legacy=0x7fd634138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 monclient: found mon.noname-c 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.488+0000 7fd6327fc700 1 -- 172.21.15.150:0/2535374504 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63409da50 con 0x7fd6340a0660 2024-09-18T10:12:54.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd6327fc700 1 -- 172.21.15.150:0/2535374504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6340a0660 msgr2=0x7fd6340a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd6327fc700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6340a0660 0x7fd6340a0a90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd6280098e0 tx=0x7fd6280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a615a96297f31586 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 1 -- 172.21.15.150:0/2535374504 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800b850 con 0x7fd6340a0660 2024-09-18T10:12:54.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: dump: 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.504 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 1 -- 172.21.15.150:0/2535374504 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62800b9b0 con 0x7fd6340a0660 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 5 monclient: authenticate success, global_id 15085 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 1 -- 172.21.15.150:0/2535374504 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800bd50 con 0x7fd6340a0660 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 4 set_mon_vals no callback set 2024-09-18T10:12:54.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd631ffb700 10 monclient: dump: 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 10 monclient: shutdown 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- 172.21.15.150:0/2535374504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6340a0660 msgr2=0x7fd6340a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6340a0660 0x7fd6340a0a90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd6280098e0 tx=0x7fd6280091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- 172.21.15.150:0/2535374504 shutdown_connections 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6340a0660 0x7fd6340a0a90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 --2- 172.21.15.150:0/2535374504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0290 0x7fd63409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- 172.21.15.150:0/2535374504 >> 172.21.15.150:0/2535374504 conn(0x7fd6340a58a0 msgr2=0x7fd6340a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- 172.21.15.150:0/2535374504 shutdown_connections 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- 172.21.15.150:0/2535374504 wait complete. 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 10 monclient: monmap: 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:54.507 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 Processor -- start 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 1 -- start start 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.492+0000 7fd63945a700 10 monclient: init 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6340a0290 0x7fd634130720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-a con 0x7fd6340a0290 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0660 0x7fd63412c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-c con 0x7fd6340a0660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): picked mon.noname-b con 0x7fd634136a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63412f650 con 0x7fd6340a0290 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd634130c60 con 0x7fd634136a10 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63412e130 con 0x7fd6340a0660 2024-09-18T10:12:54.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd63945a700 10 monclient(hunting): authenticate will time out at 1056.592876s 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37282/0 (socket says 172.21.15.150:37282) 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 -- 172.21.15.150:0/862253459 learned_addr learned my addr 172.21.15.150:0/862253459 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): get_auth_request con 0x7fd634136a10 auth_method 0 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6327fc700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0660 0x7fd63412c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd632ffd700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6340a0290 0x7fd634130720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd632ffd700 10 monclient(hunting): get_auth_request con 0x7fd6340a0290 auth_method 0 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd632ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6327fc700 10 monclient(hunting): get_auth_request con 0x7fd6340a0660 auth_method 0 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6327fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient(hunting): handle_auth_done global_id 15088 payload 824 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 -- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0660 msgr2=0x7fd63412c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0660 0x7fd63412c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 -- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6340a0290 msgr2=0x7fd634130720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6340a0290 0x7fd634130720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 monclient: found mon.noname-b 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628009420 con 0x7fd634136a10 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 -- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 msgr2=0x7fd63412ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.496+0000 7fd6337fe700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00cbf0 tx=0x7fd62c0053d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c00abd0 con 0x7fd634136a10 2024-09-18T10:12:54.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:54.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: dump: 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 5 monclient: authenticate success, global_id 15088 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _renew_subs 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c00ad50 con 0x7fd634136a10 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd634130020 con 0x7fd634136a10 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c007340 con 0x7fd634136a10 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 10 monclient: dump: 2024-09-18T10:12:54.516 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _renew_subs 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6341303b0 con 0x7fd634136a10 2024-09-18T10:12:54.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd6317fa700 4 set_mon_vals no callback set 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd62c007730 con 0x7fd634136a10 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63412fe20 con 0x7fd634136a10 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 0x7fd620037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(95..95 src has 40..95) v4 ==== 5697+0+0 (secure 0 0 0) 0x7fd62c048cd0 con 0x7fd634136a10 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd632ffd700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 0x7fd620037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.500+0000 7fd632ffd700 10 monclient: get_auth_request con 0x7fd620035ac0 auth_method 0 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.504+0000 7fd632ffd700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 0x7fd620037f70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd624009d40 tx=0x7fd624008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.508+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd62c01a960 con 0x7fd634136a10 2024-09-18T10:12:54.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.508+0000 7fd61affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.508+0000 7fd61affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:54.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 10 monclient: get_monmap_and_config 2024-09-18T10:12:54.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 10 monclient: monmap: 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:54.578806+0000 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:54.578806+0000 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:54.580 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:54.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 1 Processor -- start 2024-09-18T10:12:54.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 1 -- start start 2024-09-18T10:12:54.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.572+0000 7fc67a7aa700 10 monclient: init 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc674132030 0x7fc674132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-c con 0x7fc674132030 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc67409ece0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 0x7fc67409b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-a con 0x7fc67409f0e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc674132940 con 0x7fc67409ece0 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6740a1a90 con 0x7fc67409f0e0 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67409f560 con 0x7fc674132030 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc67a7aa700 10 monclient(hunting): authenticate will time out at 1063.504306s 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 0x7fc67409b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 0x7fc67409b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45848/0 (socket says 172.21.15.142:45848) 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 1 -- 172.21.15.142:0/1237968578 learned_addr learned my addr 172.21.15.142:0/1237968578 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 10 monclient(hunting): get_auth_request con 0x7fc67409f0e0 auth_method 0 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc679fa9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 --2- 172.21.15.142:0/1237968578 >> v2:172.21.15.150:3300/0 conn(0x7fc674132030 0x7fc674132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): get_auth_request con 0x7fc674132030 auth_method 0 2024-09-18T10:12:54.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient(hunting): handle_auth_done global_id 15091 payload 274 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 -- 172.21.15.142:0/1237968578 >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 msgr2=0x7fc67409b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 --2- 172.21.15.142:0/1237968578 >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 0x7fc67409b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 -- 172.21.15.142:0/1237968578 >> v1:172.21.15.142:6789/0 conn(0x7fc67409ece0 legacy=0x7fc67409ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 monclient: found mon.noname-c 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.576+0000 7fc6797a8700 1 -- 172.21.15.142:0/1237968578 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740a17b0 con 0x7fc674132030 2024-09-18T10:12:54.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc6797a8700 1 -- 172.21.15.142:0/1237968578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 msgr2=0x7fc674132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc6797a8700 1 --2- 172.21.15.142:0/1237968578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674132400 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc67406a4d0 tx=0x7fc6700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3c1f99958ceca83 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 1 -- 172.21.15.142:0/1237968578 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000b8c0 con 0x7fc674132030 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: dump: 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.593 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 1 -- 172.21.15.142:0/1237968578 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67000ba20 con 0x7fc674132030 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 5 monclient: authenticate success, global_id 15091 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 1 -- 172.21.15.142:0/1237968578 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000bdc0 con 0x7fc674132030 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 4 set_mon_vals no callback set 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc66bfff700 10 monclient: dump: 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 10 monclient: shutdown 2024-09-18T10:12:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 -- 172.21.15.142:0/1237968578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 msgr2=0x7fc674132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/1237968578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674132400 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc67406a4d0 tx=0x7fc6700091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 -- 172.21.15.142:0/1237968578 shutdown_connections 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/1237968578 >> v2:172.21.15.142:3300/0 conn(0x7fc67409f0e0 0x7fc67409b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/1237968578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674132400 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 -- 172.21.15.142:0/1237968578 >> 172.21.15.142:0/1237968578 conn(0x7fc6740a5700 msgr2=0x7fc6740a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 -- 172.21.15.142:0/1237968578 shutdown_connections 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 1 -- 172.21.15.142:0/1237968578 wait complete. 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 10 monclient: build_initial_monmap 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.580+0000 7fc67a7aa700 10 monclient: monmap: 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:54.596 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 Processor -- start 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 -- start start 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient: init 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:54.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-c con 0x7fc67409ece0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 0x7fc6741311c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-a con 0x7fc67409f0e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): picked mon.noname-b con 0x7fc674132030 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67411cdc0 con 0x7fc67409f0e0 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67411e2e0 con 0x7fc674132030 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:54.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67411e5c0 con 0x7fc67409ece0 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc67a7aa700 10 monclient(hunting): authenticate will time out at 1063.513496s 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 0x7fc6741311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45052/0 (socket says 172.21.15.142:45052) 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 -- 172.21.15.142:0/3154633109 learned_addr learned my addr 172.21.15.142:0/3154633109 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): get_auth_request con 0x7fc67409ece0 auth_method 0 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): get_auth_request con 0x7fc67409f0e0 auth_method 0 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): get_auth_request con 0x7fc674132030 auth_method 0 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc679fa9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient(hunting): handle_auth_done global_id 44114 payload 824 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 -- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 msgr2=0x7fc674131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 -- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 msgr2=0x7fc6741311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 0x7fc6741311c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient: _finish_hunting 0 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 monclient: found mon.noname-c 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc6797a8700 1 -- 172.21.15.142:0/3154633109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670009420 con 0x7fc67409ece0 2024-09-18T10:12:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.584+0000 7fc678fa7700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 0x7fc6741311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6797a8700 1 -- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 msgr2=0x7fc67406ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6797a8700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a1bd0 tx=0x7fc670016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670011b00 con 0x7fc67409ece0 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: dump: 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670020070 con 0x7fc67409ece0 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 5 monclient: authenticate success, global_id 44114 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _renew_subs 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67411e8a0 con 0x7fc67409ece0 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67001b450 con 0x7fc67409ece0 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 10 monclient: dump: 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc66b7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _renew_subs 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67411c230 con 0x7fc67409ece0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc670022030 con 0x7fc67409ece0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6740a4890 con 0x7fc67409ece0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 0x7fc658037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(95..95 src has 40..95) v4 ==== 5697+0+0 (secure 0 0 0) 0x7fc67004a390 con 0x7fc67409ece0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc678fa7700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 0x7fc658037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.588+0000 7fc678fa7700 10 monclient: get_auth_request con 0x7fc658035ac0 auth_method 0 2024-09-18T10:12:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.592+0000 7fc678fa7700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 0x7fc658037f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc674063df0 tx=0x7fc664009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.596+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc6740a4890 con 0x7fc67409ece0 2024-09-18T10:12:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.596+0000 7fc6697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.596+0000 7fc6697fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:54.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.680+0000 7fd63945a700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:54.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.680+0000 7fd63945a700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:54.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.680+0000 7fd63945a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:54.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:54.680+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7fd63412f7c0 con 0x7fd634136a10 2024-09-18T10:12:54.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.820+0000 7fc67a7aa700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:12:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.820+0000 7fc67a7aa700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:12:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.820+0000 7fc67a7aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:54.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.820+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc6740a4890 con 0x7fc67409ece0 2024-09-18T10:12:54.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc6697fa700 1 -- 172.21.15.142:0/3154633109 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc6740a4890 con 0x7fc67409ece0 2024-09-18T10:12:54.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc6697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:12:54.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc6697fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=4258951575,config=3140514009,health=1017346500,kv=4261215878,logm=2580080816}) 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=4118288878,config=3140514009,health=1585574141,kv=4261215878,logm=1025341790}) 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=4258951575,config=3140514009,health=1017346500,kv=4261215878,logm=2580080816}) 2024-09-18T10:12:54.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=8,logm=67} crc {auth=650357090,config=3140514009,health=1017346500,kv=1576577393,logm=14924602}) 2024-09-18T10:12:54.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669106721}) 2024-09-18T10:12:54.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711109386}) 2024-09-18T10:12:54.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 msgr2=0x7fc658037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 0x7fc658037f70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc674063df0 tx=0x7fc664009060 comp rx=0 tx=0).stop 2024-09-18T10:12:54.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669106721}) 2024-09-18T10:12:54.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069481875}) 2024-09-18T10:12:54.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 10 monclient: shutdown 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 msgr2=0x7fc67406ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc6740a1bd0 tx=0x7fc670016000 comp rx=0 tx=0).stop 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 shutdown_connections 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc658035ac0 0x7fc658037f70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc674132030 0x7fc674131700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67409f0e0 0x7fc6741311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 --2- 172.21.15.142:0/3154633109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67409ece0 0x7fc67406ea00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 >> 172.21.15.142:0/3154633109 conn(0x7fc6740a5700 msgr2=0x7fc6740a4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 shutdown_connections 2024-09-18T10:12:54.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:54.824+0000 7fc67a7aa700 1 -- 172.21.15.142:0/3154633109 wait complete. 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289355886}) 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=980928251}) 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289355886}) 2024-09-18T10:12:54.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3243181081}) 2024-09-18T10:12:54.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153820768}) 2024-09-18T10:12:54.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284515154}) 2024-09-18T10:12:54.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153820768}) 2024-09-18T10:12:54.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.828+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671490538}) 2024-09-18T10:12:54.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258724487}) 2024-09-18T10:12:54.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136818658}) 2024-09-18T10:12:54.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258724487}) 2024-09-18T10:12:54.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529595371}) 2024-09-18T10:12:54.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179267163}) 2024-09-18T10:12:54.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823101482}) 2024-09-18T10:12:54.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179267163}) 2024-09-18T10:12:54.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186290950}) 2024-09-18T10:12:54.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050452728}) 2024-09-18T10:12:54.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394777772}) 2024-09-18T10:12:54.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050452728}) 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2595244882}) 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666583592}) 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3025824482}) 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666583592}) 2024-09-18T10:12:54.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.832+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214841411}) 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498644720}) 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629370354}) 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498644720}) 2024-09-18T10:12:54.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647975496}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922395016}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1788290962}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922395016}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93334744}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605619027}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978258224}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605619027}) 2024-09-18T10:12:54.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068600207}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125334935}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760692713}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125334935}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.836+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288582904}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360219244}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236089228}) 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360219244}) 2024-09-18T10:12:54.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259909081}) 2024-09-18T10:12:54.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:12:54.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3} crc {logm=783016843,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=144029212,mon_config_key=620546692,monmap=3505902565}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=4,monmap=3,osd_metadata=4} crc {logm=997956045,mdsmap=1531151339,mgr=3316658069,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2088455921,mon_config_key=2836485344,monmap=3505902565,osd_metadata=373314451}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3} crc {logm=783016843,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=144029212,mon_config_key=620546692,monmap=3505902565}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=3} crc {logm=3828373579,mdsmap=1531151339,mgr=3895449980,mgr_command_descs=47663135,mgrstat=144029212,mon_config_key=620546692,monmap=3505902565,osd_metadata=445674298}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=1440571423,osd_pg_creating=2497657979,osd_snap=454132730,osdmap=590736122}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=1440571423,osd_pg_creating=2497657979,osd_snap=454132730,osdmap=2323148413}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=91} crc {osd_metadata=1440571423,osd_pg_creating=2497657979,osd_snap=454132730,osdmap=590736122}) 2024-09-18T10:12:54.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.840+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=92} crc {osd_metadata=361655855,osd_pg_creating=2497657979,osd_snap=454132730,osdmap=1155307434}) 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1667650687}) 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=8} crc {osdmap=3979648248}) 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1667650687}) 2024-09-18T10:12:54.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:12:54.844+0000 7fa622eb6700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=7} crc {osdmap=140807442}) 2024-09-18T10:12:55.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.272+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v96) v1 ==== 204+0+0 (secure 0 0 0) 0x7fd62c010300 con 0x7fd634136a10 2024-09-18T10:12:55.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.276+0000 7fd61affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:55.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.276+0000 7fd61affd700 10 monclient: _finish_command 2 = system:0 moved item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T10:12:55.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd63945a700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:55.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd63945a700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:55.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd63945a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]} v 0) v1 -- 0x7fd63409d6c0 con 0x7fd634136a10 2024-09-18T10:12:55.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd61affd700 1 -- 172.21.15.150:0/862253459 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}]=0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map v96) v1 ==== 214+0+0 (secure 0 0 0) 0x7fd62c01a360 con 0x7fd634136a10 2024-09-18T10:12:55.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd61affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "foo", "args": ["root=default", "rack=localrack"]}] 2024-09-18T10:12:55.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.384+0000 7fd61affd700 10 monclient: _finish_command 3 = system:0 no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 msgr2=0x7fd620037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 0x7fd620037f70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd624009d40 tx=0x7fd624008040 comp rx=0 tx=0).stop 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 10 monclient: shutdown 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 msgr2=0x7fd63412ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00cbf0 tx=0x7fd62c0053d0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 shutdown_connections 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd620035ac0 0x7fd620037f70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd634136a10 0x7fd63412ce70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6340a0660 0x7fd63412c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 --2- 172.21.15.150:0/862253459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6340a0290 0x7fd634130720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 >> 172.21.15.150:0/862253459 conn(0x7fd6340a58a0 msgr2=0x7fd63409abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 shutdown_connections 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.388+0000 7fd63945a700 1 -- 172.21.15.150:0/862253459 wait complete. 2024-09-18T10:12:55.395 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -10 name 'foo' to location {rack=localrack,root=default} in crush map 2024-09-18T10:12:55.406 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush create-or-move osd.8 1.0 root=default 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: get_monmap_and_config 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: build_initial_monmap 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: monmap: 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:55.509990+0000 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:55.509990+0000 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:55.510 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:55.511 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 1 Processor -- start 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 1 -- start start 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: init 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1dc0a5fb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 0x7fe1dc09e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc09fb20 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc0a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc09df10 con 0x7fe1dc0a5fb0 2024-09-18T10:12:55.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.504+0000 7fe1e451b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc09d1e0 con 0x7fe1dc09fb20 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc09dca0 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 10 monclient(hunting): authenticate will time out at 1057.604030s 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 0x7fe1dc09e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:55018/0 (socket says 172.21.15.150:55018) 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 0x7fe1dc09e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:36132/0 (socket says 172.21.15.150:36132) 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/1851217814 learned_addr learned my addr 172.21.15.150:0/1851217814 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): get_auth_request con 0x7fe1dc0a09a0 auth_method 0 2024-09-18T10:12:55.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 10 monclient(hunting): get_auth_request con 0x7fe1dc09fb20 auth_method 0 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e1ab6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_done global_id 44117 payload 274 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/1851217814 >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 msgr2=0x7fe1dc09e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 --2- 172.21.15.150:0/1851217814 >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 0x7fe1dc09e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/1851217814 >> v1:172.21.15.142:6789/0 conn(0x7fe1dc0a5fb0 legacy=0x7fe1dc0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:55.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient: _finish_hunting 0 2024-09-18T10:12:55.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 monclient: found mon.noname-b 2024-09-18T10:12:55.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:55.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:12:55.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/1851217814 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0a6380 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/1851217814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 msgr2=0x7fe1dc139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e2ab8700 1 --2- 172.21.15.150:0/1851217814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc06fe50 tx=0x7fe1d4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d009fc7d3f33fde server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 1 -- 172.21.15.150:0/1851217814 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400e070 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: dump: 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.527 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: _finish_auth 0 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 1 -- 172.21.15.150:0/1851217814 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400bc60 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 5 monclient: authenticate success, global_id 44117 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 1 -- 172.21.15.150:0/1851217814 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40092e0 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e451b700 4 set_mon_vals no callback set 2024-09-18T10:12:55.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.508+0000 7fe1e12b5700 10 monclient: dump: 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 10 monclient: shutdown 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- 172.21.15.150:0/1851217814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 msgr2=0x7fe1dc139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 --2- 172.21.15.150:0/1851217814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc06fe50 tx=0x7fe1d4009840 comp rx=0 tx=0).stop 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- 172.21.15.150:0/1851217814 shutdown_connections 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 --2- 172.21.15.150:0/1851217814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a09a0 0x7fe1dc139a30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 --2- 172.21.15.150:0/1851217814 >> v2:172.21.15.142:3300/0 conn(0x7fe1dc09fb20 0x7fe1dc09e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- 172.21.15.150:0/1851217814 >> 172.21.15.150:0/1851217814 conn(0x7fe1dc071ae0 msgr2=0x7fe1dc0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- 172.21.15.150:0/1851217814 shutdown_connections 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- 172.21.15.150:0/1851217814 wait complete. 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 10 monclient: build_initial_monmap 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 10 monclient: monmap: 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:55.530 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 Processor -- start 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 1 -- start start 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.512+0000 7fe1e451b700 10 monclient: init 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1dc0a09a0 0x7fe1dc08ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc0a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a5fb0 0x7fe1dc08e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-c con 0x7fe1dc0a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc090380 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc08fd30 con 0x7fe1dc0a09a0 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc091550 con 0x7fe1dc090380 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc08f820 con 0x7fe1dc0a5fb0 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e451b700 10 monclient(hunting): authenticate will time out at 1057.612709s 2024-09-18T10:12:55.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37298/0 (socket says 172.21.15.150:37298) 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/2720911518 learned_addr learned my addr 172.21.15.150:0/2720911518 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): get_auth_request con 0x7fe1dc090380 auth_method 0 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a5fb0 0x7fe1dc08e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1dc0a09a0 0x7fe1dc08ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): get_auth_request con 0x7fe1dc0a5fb0 auth_method 0 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): get_auth_request con 0x7fe1dc0a09a0 auth_method 0 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e1ab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e22b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient(hunting): handle_auth_done global_id 15097 payload 824 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a5fb0 msgr2=0x7fe1dc08e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a5fb0 0x7fe1dc08e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1dc0a09a0 msgr2=0x7fe1dc08ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1dc0a09a0 0x7fe1dc08ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient: _finish_hunting 0 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 monclient: found mon.noname-b 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.516+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4009ae0 con 0x7fe1dc090380 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e2ab8700 1 -- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 msgr2=0x7fe1dc08e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e2ab8700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4003340 tx=0x7fe1d4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4007000 con 0x7fe1dc090380 2024-09-18T10:12:55.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: dump: 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 5 monclient: authenticate success, global_id 15097 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d4007b30 con 0x7fe1dc090380 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4007160 con 0x7fe1dc090380 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 10 monclient: dump: 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.537 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e0ab4700 4 set_mon_vals no callback set 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _renew_subs 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc09c0d0 con 0x7fe1dc090380 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _renew_subs 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc099280 con 0x7fe1dc090380 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe1d401d420 con 0x7fe1dc090380 2024-09-18T10:12:55.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 0x7fe1c0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(96..96 src has 40..96) v4 ==== 5994+0+0 (secure 0 0 0) 0x7fe1d404a530 con 0x7fe1dc090380 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e22b7700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 0x7fe1c0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.520+0000 7fe1e22b7700 10 monclient: get_auth_request con 0x7fe1c0035ac0 auth_method 0 2024-09-18T10:12:55.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.524+0000 7fe1e22b7700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 0x7fe1c0037f70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0003a30 tx=0x7fe1d0007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.532+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:55.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.532+0000 7fe1ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.540 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.532+0000 7fe1ce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:55.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.704+0000 7fe1e451b700 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:55.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.704+0000 7fe1e451b700 10 monclient: _send_command 2 [{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:55.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.704+0000 7fe1e451b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:55.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:55.704+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:55.848 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:12:55.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:55.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: get_monmap_and_config 2024-09-18T10:12:55.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: build_initial_monmap 2024-09-18T10:12:55.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: monmap: 2024-09-18T10:12:55.958 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:55.958 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:55.957595+0000 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:55.957595+0000 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:55.959 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 Processor -- start 2024-09-18T10:12:55.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 -- start start 2024-09-18T10:12:55.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: init 2024-09-18T10:12:55.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:55.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 0x7f20b0095440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-a con 0x7f20b009d9a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20b009b6f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 0x7f20b0136610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-b con 0x7f20b0099060 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20b009b080 con 0x7f20b009b6f0 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b009ad10 con 0x7f20b009d9a0 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b009a000 con 0x7f20b0099060 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b7ff7700 10 monclient(hunting): authenticate will time out at 1064.883291s 2024-09-18T10:12:55.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 0x7f20b0095440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 0x7f20b0136610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 0x7f20b0095440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45870/0 (socket says 172.21.15.142:45870) 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 1 -- 172.21.15.142:0/2953333756 learned_addr learned my addr 172.21.15.142:0/2953333756 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): get_auth_request con 0x7f20b009d9a0 auth_method 0 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): get_auth_request con 0x7f20b0099060 auth_method 0 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.952+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 10 monclient(hunting): handle_auth_done global_id 44123 payload 274 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 -- 172.21.15.142:0/2953333756 >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 msgr2=0x7f20b0095440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 --2- 172.21.15.142:0/2953333756 >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 0x7f20b0095440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 -- 172.21.15.142:0/2953333756 >> v1:172.21.15.142:6789/0 conn(0x7f20b009b6f0 legacy=0x7f20b0098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 10 monclient: _finish_hunting 0 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 monclient: found mon.noname-b 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:12:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 -- 172.21.15.142:0/2953333756 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b009aa30 con 0x7f20b0099060 2024-09-18T10:12:55.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 -- 172.21.15.142:0/2953333756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 msgr2=0x7f20b0136610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:55.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b6594700 1 --2- 172.21.15.142:0/2953333756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 0x7f20b0136610 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20a00098e0 tx=0x7f20a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6db7c1f7bf6170d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 1 -- 172.21.15.142:0/2953333756 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000b850 con 0x7f20b0099060 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: dump: 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: _finish_auth 0 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 5 monclient: authenticate success, global_id 44123 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 1 -- 172.21.15.142:0/2953333756 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a000b9b0 con 0x7f20b0099060 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 4 set_mon_vals no callback set 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 1 -- 172.21.15.142:0/2953333756 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000bd50 con 0x7f20b0099060 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b4d91700 10 monclient: dump: 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:55.970 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 10 monclient: shutdown 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 -- 172.21.15.142:0/2953333756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 msgr2=0x7f20b0136610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/2953333756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 0x7f20b0136610 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20a00098e0 tx=0x7f20a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 -- 172.21.15.142:0/2953333756 shutdown_connections 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/2953333756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b0099060 0x7f20b0136610 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/2953333756 >> v2:172.21.15.142:3300/0 conn(0x7f20b009d9a0 0x7f20b0095440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 -- 172.21.15.142:0/2953333756 >> 172.21.15.142:0/2953333756 conn(0x7f20b00a1470 msgr2=0x7f20b00a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 -- 172.21.15.142:0/2953333756 shutdown_connections 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.956+0000 7f20b7ff7700 1 -- 172.21.15.142:0/2953333756 wait complete. 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient: build_initial_monmap 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient: monmap: 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:55.972 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 Processor -- start 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 -- start start 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient: init 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 0x7f20b013d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-a con 0x7f20b0099060 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:55.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-c con 0x7f20b009d9a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b0089b20 0x7f20b0089ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): picked mon.noname-b con 0x7f20b0089b20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b008d730 con 0x7f20b0099060 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b006f8d0 con 0x7f20b0089b20 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b008ba90 con 0x7f20b009d9a0 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b7ff7700 10 monclient(hunting): authenticate will time out at 1064.889431s 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 0x7f20b013d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45068/0 (socket says 172.21.15.142:45068) 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 1 -- 172.21.15.142:0/4214673726 learned_addr learned my addr 172.21.15.142:0/4214673726 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): get_auth_request con 0x7f20b009d9a0 auth_method 0 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): get_auth_request con 0x7f20b0099060 auth_method 0 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b0089b20 0x7f20b0089ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): get_auth_request con 0x7f20b0089b20 auth_method 0 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b6594700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient(hunting): handle_auth_done global_id 5376 payload 824 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 -- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 msgr2=0x7f20b013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 -- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b0089b20 msgr2=0x7f20b0089ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b0089b20 0x7f20b0089ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient: _finish_hunting 0 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 monclient: found mon.noname-a 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5d93700 1 -- 172.21.15.142:0/4214673726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a0009420 con 0x7f20b0099060 2024-09-18T10:12:55.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.960+0000 7f20b5592700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b5d93700 1 -- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 msgr2=0x7f20b013d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b5d93700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 0x7f20b013d8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f20a4006d40 tx=0x7f20a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4013890 con 0x7f20b0099060 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: dump: 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.979 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 5 monclient: authenticate success, global_id 5376 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a40139f0 con 0x7f20b0099060 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:55.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4013d70 con 0x7f20b0099060 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _renew_subs 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b008b580 con 0x7f20b0099060 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 10 monclient: dump: 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:55.981 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20af7fe700 4 set_mon_vals no callback set 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _renew_subs 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b008e2d0 con 0x7f20b0099060 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b00696a0 con 0x7f20b0099060 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f20a40125e0 con 0x7f20b0099060 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 0x7f209c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:55.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.964+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(96..96 src has 40..96) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f20a4048890 con 0x7f20b0099060 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.968+0000 7f20b5592700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 0x7f209c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.968+0000 7f20b5592700 10 monclient: get_auth_request con 0x7f209c035ac0 auth_method 0 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.968+0000 7f20b5592700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 0x7f209c037f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20a8007fd0 tx=0x7f20a8002760 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.972+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f20a4023ad0 con 0x7f20b0099060 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.972+0000 7f20adffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:55.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:55.972+0000 7f20adffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:56.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20b7ff7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:56.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20b7ff7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:56.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20b7ff7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:56.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f20b0099b60 con 0x7f20b0099060 2024-09-18T10:12:56.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20adffb700 1 -- 172.21.15.142:0/4214673726 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f20a40232f0 con 0x7f20b0099060 2024-09-18T10:12:56.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20adffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:56.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.200+0000 7f20adffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:56.207 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:56.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 msgr2=0x7f209c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 0x7f209c037f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20a8007fd0 tx=0x7f20a8002760 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 10 monclient: shutdown 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 msgr2=0x7f20b013d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 0x7f20b013d8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f20a4006d40 tx=0x7f20a4009250 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 shutdown_connections 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f209c035ac0 0x7f209c037f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20b0089b20 0x7f20b0089ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f20b009d9a0 0x7f20b013de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 --2- 172.21.15.142:0/4214673726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20b0099060 0x7f20b013d8e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 >> 172.21.15.142:0/4214673726 conn(0x7f20b00a1470 msgr2=0x7f20b0096190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 shutdown_connections 2024-09-18T10:12:56.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.204+0000 7f20b7ff7700 1 -- 172.21.15.142:0/4214673726 wait complete. 2024-09-18T10:12:56.227 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:12:56.227 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:12:56.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:12:56.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:12:56.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.280+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updating item name 'osd.8' weight 1 at location {root=default} to crush map v97) v1 ==== 212+0+0 (secure 0 0 0) 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:56.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.280+0000 7fe1ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:56.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.280+0000 7fe1ce7fc700 10 monclient: _finish_command 2 = system:0 create-or-move updating item name 'osd.8' weight 1 at location {root=default} to crush map 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: get_monmap_and_config 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: monmap: 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:56.338149+0000 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:56.338149+0000 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:56.340 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 Processor -- start 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 -- start start 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: init 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 0x7f3b2c09e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c09cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 0x7f3b2c0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c0a02e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b2c0a0bf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c09d1a0 con 0x7f3b2c0a0bf0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c09e980 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c09d780 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b30ce5700 10 monclient(hunting): authenticate will time out at 1065.263761s 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b29d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 0x7f3b2c0a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.332+0000 7f3b2a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 0x7f3b2c09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 0x7f3b2c0a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45886/0 (socket says 172.21.15.142:45886) 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 1 -- 172.21.15.142:0/2905493068 learned_addr learned my addr 172.21.15.142:0/2905493068 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a02e0 auth_method 0 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request con 0x7f3b2c09cdd0 auth_method 0 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_done global_id 44129 payload 274 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 -- 172.21.15.142:0/2905493068 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 msgr2=0x7f3b2c0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/2905493068 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 0x7f3b2c0a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 -- 172.21.15.142:0/2905493068 >> v1:172.21.15.142:6789/0 conn(0x7f3b2c0a0bf0 legacy=0x7f3b2c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 monclient: found mon.noname-b 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 -- 172.21.15.142:0/2905493068 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c09dd40 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 -- 172.21.15.142:0/2905493068 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 msgr2=0x7f3b2c09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/2905493068 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 0x7f3b2c09e440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b200098e0 tx=0x7f3b200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347a23b0ab925ba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 1 -- 172.21.15.142:0/2905493068 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000b850 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: dump: 2024-09-18T10:12:56.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.354 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.354 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.354 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.354 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 1 -- 172.21.15.142:0/2905493068 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2000b9b0 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 5 monclient: authenticate success, global_id 44129 2024-09-18T10:12:56.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 1 -- 172.21.15.142:0/2905493068 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000bd50 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 4 set_mon_vals no callback set 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b2959a700 10 monclient: dump: 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 10 monclient: shutdown 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 1 -- 172.21.15.142:0/2905493068 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 msgr2=0x7f3b2c09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.336+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/2905493068 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 0x7f3b2c09e440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b200098e0 tx=0x7f3b200091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- 172.21.15.142:0/2905493068 shutdown_connections 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/2905493068 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a02e0 0x7f3b2c0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/2905493068 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c09cdd0 0x7f3b2c09e440 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- 172.21.15.142:0/2905493068 >> 172.21.15.142:0/2905493068 conn(0x7f3b2c071680 msgr2=0x7f3b2c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- 172.21.15.142:0/2905493068 shutdown_connections 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- 172.21.15.142:0/2905493068 wait complete. 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient: monmap: 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:56.358 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 Processor -- start 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- start start 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient: init 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:12:56.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c09cdd0 0x7f3b2c141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c09cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c0a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c0a0bf0 0x7f3b2c08cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c0a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0a4030 con 0x7f3b2c0a0bf0 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c08edc0 con 0x7f3b2c09cdd0 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c091570 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b30ce5700 10 monclient(hunting): authenticate will time out at 1065.270467s 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45090/0 (socket says 172.21.15.142:45090) 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 -- 172.21.15.142:0/960989816 learned_addr learned my addr 172.21.15.142:0/960989816 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a02e0 auth_method 0 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c0a0bf0 0x7f3b2c08cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c09cdd0 0x7f3b2c141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a0bf0 auth_method 0 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request con 0x7f3b2c09cdd0 auth_method 0 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_done global_id 44132 payload 824 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c09cdd0 msgr2=0x7f3b2c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c09cdd0 0x7f3b2c141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c0a0bf0 msgr2=0x7f3b2c08cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c0a0bf0 0x7f3b2c08cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 monclient: found mon.noname-c 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b29d9b700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20009420 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.340+0000 7f3b2ad9d700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c0a0bf0 0x7f3b2c08cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b29d9b700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 msgr2=0x7f3b2c1422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b29d9b700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00ea20 tx=0x7f3b1c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00c850 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:12:56.364 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c00c9f0 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 5 monclient: authenticate success, global_id 44132 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 monclient: _renew_subs 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c08e8b0 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c00cdb0 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.366 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1bfff700 4 set_mon_vals no callback set 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 monclient: _renew_subs 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c098400 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 0x7f3b2c08d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0648c0 con 0x7f3b2c08d2b0 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 0x7f3b2c08d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient: get_auth_request con 0x7f3b2c08d2b0 auth_method 0 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_done global_id 5385 payload 824 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b2a59c700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b2c090860 con 0x7f3b2c08d2b0 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.344+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3b1c013680 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035af0 0x7f3b0c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7f3b1c048e30 con 0x7f3b2c0a02e0 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b2ad9d700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035af0 0x7f3b0c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b2ad9d700 10 monclient: get_auth_request con 0x7f3b0c035af0 auth_method 0 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b2a59c700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 msgr2=0x7f3b2c08d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 0x7f3b2c08d680 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b200098e0 tx=0x7f3b20016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20021070 con 0x7f3b2c08d2b0 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:12:56.372 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3b2001c450 con 0x7f3b2c08d2b0 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 msgr2=0x7f3b2c08d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 0x7f3b2c08d680 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b200098e0 tx=0x7f3b20016000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 msgr2=0x7f3b2c08d680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b1a7fc700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08d2b0 0x7f3b2c08d680 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.348+0000 7f3b2ad9d700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035af0 0x7f3b0c037fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007fd0 tx=0x7f3b24002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b30ce5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 0x7f3b2c06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b30ce5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c0905e0 con 0x7f3b2c08dbc0 2024-09-18T10:12:56.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 0x7f3b2c06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient: get_auth_request con 0x7f3b2c08dbc0 auth_method 0 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 10 monclient(hunting): handle_auth_done global_id 5388 payload 824 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.360+0000 7f3b2a59c700 1 -- 172.21.15.142:0/960989816 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3b2c0696a0 con 0x7f3b2c08dbc0 2024-09-18T10:12:56.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b2a59c700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 msgr2=0x7f3b2c06cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b2a59c700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 0x7f3b2c06cbc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0721b0 tx=0x7f3b2001e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20025d40 con 0x7f3b2c08dbc0 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.376 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f3b20011cd0 con 0x7f3b2c08dbc0 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 msgr2=0x7f3b2c06cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 0x7f3b2c06cbc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0721b0 tx=0x7f3b2001e070 comp rx=0 tx=0).stop 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 msgr2=0x7f3b2c06cbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b1a7fc700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c08dbc0 0x7f3b2c06cbc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:56.377 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:56.378 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.379 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:56.380 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:56.381 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:56.382 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.386 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.387 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.388 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.389 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:56.390 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035af0 msgr2=0x7f3b0c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035af0 0x7f3b0c037fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b24007fd0 tx=0x7f3b24002620 comp rx=0 tx=0).stop 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 10 monclient: shutdown 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 msgr2=0x7f3b2c1422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00ea20 tx=0x7f3b1c009250 comp rx=0 tx=0).stop 2024-09-18T10:12:56.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b2ad9d700 1 -- 172.21.15.142:0/960989816 reap_dead start 2024-09-18T10:12:56.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 shutdown_connections 2024-09-18T10:12:56.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 --2- 172.21.15.142:0/960989816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a02e0 0x7f3b2c1422a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 >> 172.21.15.142:0/960989816 conn(0x7f3b2c071680 msgr2=0x7f3b2c09eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 shutdown_connections 2024-09-18T10:12:56.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.364+0000 7f3b30ce5700 1 -- 172.21.15.142:0/960989816 wait complete. 2024-09-18T10:12:56.393 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:56.393 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:12:56.394 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:12:56.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1e451b700 10 start_mon_command cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:56.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1e451b700 10 monclient: _send_command 3 [{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:56.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1e451b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:56.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]} v 0) v1 -- 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:56.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1ce7fc700 1 -- 172.21.15.150:0/2720911518 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}]=0 create-or-move updated item name 'osd.8' weight 1 at location {root=default} to crush map v97) v1 ==== 211+0+0 (secure 0 0 0) 0x7fe1dc0919e0 con 0x7fe1dc090380 2024-09-18T10:12:56.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1ce7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush create-or-move", "id": 8, "weight": 1.0, "args": ["root=default"]}] 2024-09-18T10:12:56.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.392+0000 7fe1ce7fc700 10 monclient: _finish_command 3 = system:0 create-or-move updated item name 'osd.8' weight 1 at location {root=default} to crush map 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 msgr2=0x7fe1c0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 0x7fe1c0037f70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0003a30 tx=0x7fe1d0007ea0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 10 monclient: shutdown 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 msgr2=0x7fe1dc08e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4003340 tx=0x7fe1d4018000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 shutdown_connections 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe1c0035ac0 0x7fe1c0037f70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1dc090380 0x7fe1dc08e950 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1dc0a5fb0 0x7fe1dc08e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 --2- 172.21.15.150:0/2720911518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1dc0a09a0 0x7fe1dc08ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 >> 172.21.15.150:0/2720911518 conn(0x7fe1dc071ae0 msgr2=0x7fe1dc09a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 shutdown_connections 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.396+0000 7fe1e451b700 1 -- 172.21.15.150:0/2720911518 wait complete. 2024-09-18T10:12:56.403 INFO:tasks.workunit.client.0.smithi150.stderr:create-or-move updated item name 'osd.8' weight 1 at location {root=default} to crush map 2024-09-18T10:12:56.412 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush move osd.8 host=foo 2024-09-18T10:12:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: get_monmap_and_config 2024-09-18T10:12:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: monmap: 2024-09-18T10:12:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:12:56.508200+0000 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:12:56.508200+0000 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:56.510 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: get_monmap_and_config 2024-09-18T10:12:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: monmap: 2024-09-18T10:12:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:56.508234+0000 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:56.508234+0000 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 Processor -- start 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 -- start start 2024-09-18T10:12:56.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: init 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff34c09b0f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff34c09b4c0 0x7ff34c097e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): picked mon.noname-c con 0x7ff34c09b4c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff34c133870 0x7ff34c135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): picked mon.noname-a con 0x7ff34c133870 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c099e80 con 0x7ff34c09b0f0 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c09a470 con 0x7ff34c133870 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c098ae0 con 0x7ff34c09b4c0 2024-09-18T10:12:56.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351853700 10 monclient(hunting): authenticate will time out at 1065.434304s 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff350851700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff34c09b0f0 0x7ff34c0976f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51542/0 (socket says 172.21.15.142:51542) 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff350851700 1 -- 172.21.15.142:0/3379895672 learned_addr learned my addr 172.21.15.142:0/3379895672 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 --2- 172.21.15.142:0/3379895672 >> v2:172.21.15.150:3300/0 conn(0x7ff34c09b4c0 0x7ff34c097e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): get_auth_request con 0x7ff34c09b4c0 auth_method 0 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351052700 1 --2- 172.21.15.142:0/3379895672 >> v2:172.21.15.142:3300/0 conn(0x7ff34c133870 0x7ff34c135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351052700 10 monclient(hunting): get_auth_request con 0x7ff34c133870 auth_method 0 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351052700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351052700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff351052700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient(hunting): handle_auth_done global_id 15106 payload 274 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 -- 172.21.15.142:0/3379895672 >> v2:172.21.15.142:3300/0 conn(0x7ff34c133870 msgr2=0x7ff34c135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 --2- 172.21.15.142:0/3379895672 >> v2:172.21.15.142:3300/0 conn(0x7ff34c133870 0x7ff34c135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 -- 172.21.15.142:0/3379895672 >> v1:172.21.15.142:6789/0 conn(0x7ff34c09b0f0 legacy=0x7ff34c0976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 monclient: found mon.noname-c 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.504+0000 7ff34bfff700 1 -- 172.21.15.142:0/3379895672 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c098340 con 0x7ff34c09b4c0 2024-09-18T10:12:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34bfff700 1 -- 172.21.15.142:0/3379895672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 msgr2=0x7ff34c097e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34bfff700 1 --2- 172.21.15.142:0/3379895672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c097e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff34c095380 tx=0x7ff3380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed708691376dcb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 1 -- 172.21.15.142:0/3379895672 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33800b8c0 con 0x7ff34c09b4c0 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: dump: 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 5 monclient: authenticate success, global_id 15106 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 1 -- 172.21.15.142:0/3379895672 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33800ba20 con 0x7ff34c09b4c0 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 4 set_mon_vals no callback set 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 1 -- 172.21.15.142:0/3379895672 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33800bdc0 con 0x7ff34c09b4c0 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff34b7fe700 10 monclient: dump: 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 10 monclient: shutdown 2024-09-18T10:12:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- 172.21.15.142:0/3379895672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 msgr2=0x7ff34c097e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 --2- 172.21.15.142:0/3379895672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c097e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff34c095380 tx=0x7ff3380091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- 172.21.15.142:0/3379895672 shutdown_connections 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 --2- 172.21.15.142:0/3379895672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c097e00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 --2- 172.21.15.142:0/3379895672 >> v2:172.21.15.142:3300/0 conn(0x7ff34c133870 0x7ff34c135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- 172.21.15.142:0/3379895672 >> 172.21.15.142:0/3379895672 conn(0x7ff34c070ef0 msgr2=0x7ff34c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- 172.21.15.142:0/3379895672 shutdown_connections 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- 172.21.15.142:0/3379895672 wait complete. 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 10 monclient: monmap: 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 Processor -- start 2024-09-18T10:12:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 1 -- start start 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.508+0000 7ff351853700 10 monclient: init 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 0x7ff34c12e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): picked mon.noname-c con 0x7ff34c09b0f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): picked mon.noname-b con 0x7ff34c09b4c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c133870 0x7ff34c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): picked mon.noname-a con 0x7ff34c133870 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c128e40 con 0x7ff34c133870 2024-09-18T10:12:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c128a20 con 0x7ff34c09b4c0 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c1266e0 con 0x7ff34c09b0f0 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient(hunting): authenticate will time out at 1065.440189s 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 0x7ff34c12e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 0x7ff34c12e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45100/0 (socket says 172.21.15.142:45100) 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c06da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:48584/0 (socket says 172.21.15.142:48584) 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 1 -- 172.21.15.142:0/2442705084 learned_addr learned my addr 172.21.15.142:0/2442705084 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): get_auth_request con 0x7ff34c09b0f0 auth_method 0 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c133870 0x7ff34c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): get_auth_request con 0x7ff34c09b4c0 auth_method 0 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): get_auth_request con 0x7ff34c133870 auth_method 0 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient(hunting): handle_auth_done global_id 15112 payload 824 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 msgr2=0x7ff34c12e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 0x7ff34c12e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c133870 msgr2=0x7ff34c06df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c133870 0x7ff34c06df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 monclient: found mon.noname-b 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff338009420 con 0x7ff34c09b4c0 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff350851700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c09b0f0 0x7ff34c12e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:56.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351052700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c133870 0x7ff34c06df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 msgr2=0x7ff34c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34bfff700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c06da10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff34c098600 tx=0x7ff338016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338011b40 con 0x7ff34c09b4c0 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: dump: 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.542 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 5 monclient: authenticate success, global_id 15112 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _renew_subs 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338020070 con 0x7ff34c09b4c0 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c06f810 con 0x7ff34c09b4c0 2024-09-18T10:12:56.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33801b450 con 0x7ff34c09b4c0 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff348ff9700 10 monclient: dump: 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.544 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff34affd700 4 set_mon_vals no callback set 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _renew_subs 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.512+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c128460 con 0x7ff34c09b4c0 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351853700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351853700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 0x7ff34c02ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0648a0 con 0x7ff34c06e490 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 0x7ff34c02ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient: get_auth_request con 0x7ff34c06e490 auth_method 0 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff338022030 con 0x7ff34c09b4c0 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 10 monclient(hunting): handle_auth_done global_id 44141 payload 824 2024-09-18T10:12:56.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff350851700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff34c06e860 con 0x7ff34c06e490 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff348ff9700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff330035b10 0x7ff330037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7ff33804a450 con 0x7ff34c09b4c0 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351052700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff330035b10 0x7ff330037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.516+0000 7ff351052700 10 monclient: get_auth_request con 0x7ff330035b10 auth_method 0 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff350851700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 msgr2=0x7ff34c02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff350851700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 0x7ff34c02ae80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff3400073d0 tx=0x7ff340004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34000fb70 con 0x7ff34c06e490 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 10 monclient: dump: 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.547 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff34000fcd0 con 0x7ff34c06e490 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:56.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 msgr2=0x7ff34c02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 0x7ff34c02ae80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff3400073d0 tx=0x7ff340004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:56.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 msgr2=0x7ff34c02ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff348ff9700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06e490 0x7ff34c02ae80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.520+0000 7ff351052700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff330035b10 0x7ff330037fc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff34400afd0 tx=0x7ff34400f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 Processor -- start 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 -- start start 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: init 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03a00a0120 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f03a00a0c00 0x7f03a009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-a con 0x7f03a00a0c00 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03a01379b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a009dae0 con 0x7f03a01379b0 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a00a1590 con 0x7f03a00a0120 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a00a21d0 con 0x7f03a00a0c00 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a76df700 10 monclient(hunting): authenticate will time out at 1058.601630s 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a547b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f03a00a0120 0x7f03a00a04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34444/0 (socket says 172.21.15.150:34444) 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a547b700 1 -- 172.21.15.150:0/2898920446 learned_addr learned my addr 172.21.15.150:0/2898920446 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 1 --2- 172.21.15.150:0/2898920446 >> v2:172.21.15.142:3300/0 conn(0x7f03a00a0c00 0x7f03a009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): get_auth_request con 0x7f03a00a0c00 auth_method 0 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.504+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a547b700 1 -- 172.21.15.150:0/2898920446 >> v1:172.21.15.150:6789/0 conn(0x7f03a00a0120 legacy=0x7f03a00a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a5c7c700 1 -- 172.21.15.150:0/2898920446 >> v1:172.21.15.142:6789/0 conn(0x7f03a01379b0 legacy=0x7f03a0139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 1 -- 172.21.15.150:0/2898920446 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130355616 0 0) 0x7f03a00a1590 con 0x7f03a00a0120 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient(hunting): my global_id is 15109 2024-09-18T10:12:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 1 -- 172.21.15.150:0/2898920446 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f038c003570 con 0x7f03a00a0120 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_done global_id 5391 payload 274 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 -- 172.21.15.150:0/2898920446 >> v1:172.21.15.150:6789/0 conn(0x7f03a00a0120 legacy=0x7f03a00a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 -- 172.21.15.150:0/2898920446 >> v1:172.21.15.142:6789/0 conn(0x7f03a01379b0 legacy=0x7f03a0139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 monclient: found mon.noname-a 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 -- 172.21.15.150:0/2898920446 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a00a1da0 con 0x7f03a00a0c00 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 -- 172.21.15.150:0/2898920446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0c00 msgr2=0x7f03a009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a4c7a700 1 --2- 172.21.15.150:0/2898920446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0c00 0x7f03a009e7a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f039c0099e0 tx=0x7f039c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cec51384a1c483 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 1 -- 172.21.15.150:0/2898920446 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c00ba70 con 0x7f03a00a0c00 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: dump: 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a76df700 5 monclient: authenticate success, global_id 5391 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a76df700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 1 -- 172.21.15.150:0/2898920446 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c00bbd0 con 0x7f03a00a0c00 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a76df700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f03a76df700 4 set_mon_vals no callback set 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 1 -- 172.21.15.150:0/2898920446 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c011420 con 0x7f03a00a0c00 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.508+0000 7f0397fff700 10 monclient: dump: 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:56.555 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: shutdown 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- 172.21.15.150:0/2898920446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0c00 msgr2=0x7f03a009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 --2- 172.21.15.150:0/2898920446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0c00 0x7f03a009e7a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f039c0099e0 tx=0x7f039c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- 172.21.15.150:0/2898920446 shutdown_connections 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 --2- 172.21.15.150:0/2898920446 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0c00 0x7f03a009e7a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- 172.21.15.150:0/2898920446 >> 172.21.15.150:0/2898920446 conn(0x7f03a00a5830 msgr2=0x7f03a00a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- 172.21.15.150:0/2898920446 shutdown_connections 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- 172.21.15.150:0/2898920446 wait complete. 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: monmap: 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.557 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 Processor -- start 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- start start 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: init 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0120 0x7f03a012c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-a con 0x7f03a00a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 0x7f03a012c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-b con 0x7f03a00a0c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 0x7f03a012cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): picked mon.noname-c con 0x7f03a01379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a012ea10 con 0x7f03a00a0120 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a009d720 con 0x7f03a00a0c00 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a009be60 con 0x7f03a01379b0 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a76df700 10 monclient(hunting): authenticate will time out at 1058.611012s 2024-09-18T10:12:56.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 0x7f03a012c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0120 0x7f03a012c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 0x7f03a012c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37312/0 (socket says 172.21.15.150:37312) 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 0x7f03a012cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 1 -- 172.21.15.150:0/1181836612 learned_addr learned my addr 172.21.15.150:0/1181836612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): get_auth_request con 0x7f03a00a0c00 auth_method 0 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): get_auth_request con 0x7f03a00a0120 auth_method 0 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): get_auth_request con 0x7f03a01379b0 auth_method 0 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a547b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a4c7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 -- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 msgr2=0x7f03a012c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 0x7f03a012c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 -- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0120 msgr2=0x7f03a012c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0120 0x7f03a012c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 monclient: found mon.noname-c 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.512+0000 7f03a5c7c700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c009420 con 0x7f03a01379b0 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a5c7c700 1 -- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 msgr2=0x7f03a012cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a5c7c700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 0x7f03a012cf30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f039800c8e0 tx=0x7f03980018c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980070f0 con 0x7f03a01379b0 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: dump: 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.563 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398007250 con 0x7f03a01379b0 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 5 monclient: authenticate success, global_id 44138 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 10 monclient: _renew_subs 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a012f590 con 0x7f03a01379b0 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398007610 con 0x7f03a01379b0 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03957fa700 10 monclient: dump: 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03977fe700 4 set_mon_vals no callback set 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 10 monclient: _renew_subs 2024-09-18T10:12:56.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.516+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a0130250 con 0x7f03a01379b0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f039801a360 con 0x7f03a01379b0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03957fa700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 0x7f0384037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7f0398047dd0 con 0x7f03a01379b0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a76df700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a76df700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a76df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a0063ed0 con 0x7f03a01379b0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a547b700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 0x7f0384037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.520+0000 7f03a547b700 10 monclient: get_auth_request con 0x7f0384035ac0 auth_method 0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.524+0000 7f03a547b700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 0x7f0384037f70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f03900058d0 tx=0x7f0390007bb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.532+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0398017080 con 0x7f03a01379b0 2024-09-18T10:12:56.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.532+0000 7f03957fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.532+0000 7f03957fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:56.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.528+0000 7ff351853700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:56.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.528+0000 7ff351853700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 0x7ff34c05ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.528+0000 7ff351853700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.528+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c128650 con 0x7ff34c05e890 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 0x7ff34c05ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient: get_auth_request con 0x7ff34c05e890 auth_method 0 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 1 -- 172.21.15.142:0/2442705084 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff34c1300c0 con 0x7ff34c05e890 2024-09-18T10:12:56.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 msgr2=0x7ff34c05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff350851700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 0x7ff34c05ec90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff340007970 tx=0x7ff340018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34001c9a0 con 0x7ff34c05e890 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 10 monclient: dump: 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.570 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7ff34001cb00 con 0x7ff34c05e890 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 msgr2=0x7ff34c05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.532+0000 7ff348ff9700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 0x7ff34c05ec90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff340007970 tx=0x7ff340018000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff348ff9700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 msgr2=0x7ff34c05ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff348ff9700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c05e890 0x7ff34c05ec90 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff340007970 tx=0x7ff340018000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.571 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:56.572 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:56.573 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:56.574 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:56.575 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.576 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.577 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.578 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:56.579 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.580 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.581 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.583 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:56.584 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:56.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff330035b10 msgr2=0x7ff330037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff330035b10 0x7ff330037fc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff34400afd0 tx=0x7ff34400f040 comp rx=0 tx=0).stop 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 10 monclient: shutdown 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 msgr2=0x7ff34c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 --2- 172.21.15.142:0/2442705084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c09b4c0 0x7ff34c06da10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff34c098600 tx=0x7ff338016000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351052700 1 -- 172.21.15.142:0/2442705084 reap_dead start 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351052700 1 -- 172.21.15.142:0/2442705084 reap_dead start 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 shutdown_connections 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 >> 172.21.15.142:0/2442705084 conn(0x7ff34c070ef0 msgr2=0x7ff34c131cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 shutdown_connections 2024-09-18T10:12:56.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.536+0000 7ff351853700 1 -- 172.21.15.142:0/2442705084 wait complete. 2024-09-18T10:12:56.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:56.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:12:56.587 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:12:56.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.700+0000 7f03a76df700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:12:56.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.700+0000 7f03a76df700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:12:56.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.700+0000 7f03a76df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:56.700+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]} v 0) v1 -- 0x7f03a0067b10 con 0x7f03a01379b0 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 10 monclient: get_monmap_and_config 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 10 monclient: monmap: 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:56.705376+0000 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:56.705376+0000 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:56.707 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:56.708 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:56.708 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 1 Processor -- start 2024-09-18T10:12:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 1 -- start start 2024-09-18T10:12:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.700+0000 7ff91ef6b700 10 monclient: init 2024-09-18T10:12:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 0x7ff9180676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-b con 0x7ff918096f80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff918097350 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff918134610 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff918097fb0 con 0x7ff918097350 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91809e460 con 0x7ff918134610 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91809db70 con 0x7ff918096f80 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient(hunting): authenticate will time out at 1065.632495s 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 0x7ff9180676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91d768700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff918097350 0x7ff91809b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51554/0 (socket says 172.21.15.142:51554) 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): get_auth_request con 0x7ff918096f80 auth_method 0 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91d768700 1 -- 172.21.15.142:0/1720237212 learned_addr learned my addr 172.21.15.142:0/1720237212 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient(hunting): handle_auth_done global_id 44147 payload 274 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 -- 172.21.15.142:0/1720237212 >> v1:172.21.15.150:6789/0 conn(0x7ff918134610 legacy=0x7ff9181369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 -- 172.21.15.142:0/1720237212 >> v1:172.21.15.142:6789/0 conn(0x7ff918097350 legacy=0x7ff91809b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 monclient: found mon.noname-b 2024-09-18T10:12:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 -- 172.21.15.142:0/1720237212 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918097cd0 con 0x7ff918096f80 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 -- 172.21.15.142:0/1720237212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 msgr2=0x7ff9180676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91df69700 1 --2- 172.21.15.142:0/1720237212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 0x7ff9180676a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9180a2dc0 tx=0x7ff9140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d70162415bb15877 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 1 -- 172.21.15.142:0/1720237212 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91400ba70 con 0x7ff918096f80 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: dump: 2024-09-18T10:12:56.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 5 monclient: authenticate success, global_id 44147 2024-09-18T10:12:56.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 1 -- 172.21.15.142:0/1720237212 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91400bbd0 con 0x7ff918096f80 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91ef6b700 4 set_mon_vals no callback set 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 1 -- 172.21.15.142:0/1720237212 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914011420 con 0x7ff918096f80 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.704+0000 7ff91cf67700 10 monclient: dump: 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.717 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 10 monclient: shutdown 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- 172.21.15.142:0/1720237212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 msgr2=0x7ff9180676a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/1720237212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 0x7ff9180676a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff9180a2dc0 tx=0x7ff9140091a0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- 172.21.15.142:0/1720237212 shutdown_connections 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/1720237212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918096f80 0x7ff9180676a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- 172.21.15.142:0/1720237212 >> 172.21.15.142:0/1720237212 conn(0x7ff9180717a0 msgr2=0x7ff9180a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- 172.21.15.142:0/1720237212 shutdown_connections 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- 172.21.15.142:0/1720237212 wait complete. 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 10 monclient: monmap: 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:56.719 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:56.720 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 Processor -- start 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 1 -- start start 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.708+0000 7ff91ef6b700 10 monclient: init 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff918096f80 0x7ff91813c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-b con 0x7ff918096f80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 0x7ff918133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-a con 0x7ff918097350 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918134610 0x7ff91813f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): picked mon.noname-c con 0x7ff918134610 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181387a0 con 0x7ff918097350 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9180a0630 con 0x7ff918096f80 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91812cdb0 con 0x7ff918134610 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91ef6b700 10 monclient(hunting): authenticate will time out at 1065.640632s 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 0x7ff918133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918134610 0x7ff91813f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff918096f80 0x7ff91813c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 0x7ff918133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45950/0 (socket says 172.21.15.142:45950) 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff918096f80 0x7ff91813c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:48600/0 (socket says 172.21.15.142:48600) 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 -- 172.21.15.142:0/3346487585 learned_addr learned my addr 172.21.15.142:0/3346487585 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): get_auth_request con 0x7ff918097350 auth_method 0 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): get_auth_request con 0x7ff918134610 auth_method 0 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): get_auth_request con 0x7ff918096f80 auth_method 0 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient(hunting): handle_auth_done global_id 5406 payload 824 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918134610 msgr2=0x7ff91813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918134610 0x7ff91813f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff918096f80 msgr2=0x7ff91813c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff918096f80 0x7ff91813c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 monclient: found mon.noname-a 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914009420 con 0x7ff918097350 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91e76a700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff918134610 0x7ff91813f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.712+0000 7ff91d768700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 msgr2=0x7ff918133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91d768700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 0x7ff918133bb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff90800c5e0 tx=0x7ff908004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908005d40 con 0x7ff918097350 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: dump: 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.730 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 5 monclient: authenticate success, global_id 5406 2024-09-18T10:12:56.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9080070d0 con 0x7ff918097350 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 monclient: _renew_subs 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91812d090 con 0x7ff918097350 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908007490 con 0x7ff918097350 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90ffff700 4 set_mon_vals no callback set 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 10 monclient: dump: 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.732 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 monclient: _renew_subs 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9181392e0 con 0x7ff918097350 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 0x7ff91813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918064680 con 0x7ff91813cd50 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff90801b3b0 con 0x7ff918097350 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 0x7ff91813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient: get_auth_request con 0x7ff91813cd50 auth_method 0 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff8fc035af0 0x7ff8fc037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7ff908047cd0 con 0x7ff918097350 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 10 monclient(hunting): handle_auth_done global_id 15121 payload 824 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91df69700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff918072a80 con 0x7ff91813cd50 2024-09-18T10:12:56.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91e76a700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff8fc035af0 0x7ff8fc037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.716+0000 7ff91e76a700 10 monclient: get_auth_request con 0x7ff8fc035af0 auth_method 0 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff91df69700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 msgr2=0x7ff91813fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff91df69700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 0x7ff91813fca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff9180a2dc0 tx=0x7ff914016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914021070 con 0x7ff91813cd50 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 10 monclient: dump: 2024-09-18T10:12:56.736 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff91e76a700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff8fc035af0 0x7ff8fc037fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff918139190 tx=0x7ff910002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff91401c660 con 0x7ff91813cd50 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 msgr2=0x7ff91813fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 0x7ff91813fca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff9180a2dc0 tx=0x7ff914016000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 msgr2=0x7ff91813fca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.720+0000 7ff90dffb700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff91813cd50 0x7ff91813fca0 secure :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff9180a2dc0 tx=0x7ff914016000 comp rx=0 tx=0).stop 2024-09-18T10:12:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.732+0000 7ff91ef6b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:12:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.732+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 0x7ff9181401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.732+0000 7ff91ef6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.732+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91809dcc0 con 0x7ff9181340f0 2024-09-18T10:12:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.732+0000 7ff91df69700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 0x7ff9181401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient: get_auth_request con 0x7ff9181340f0 auth_method 0 2024-09-18T10:12:56.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 10 monclient(hunting): handle_auth_done global_id 15124 payload 824 2024-09-18T10:12:56.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 1 -- 172.21.15.142:0/3346487585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff918068360 con 0x7ff9181340f0 2024-09-18T10:12:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 msgr2=0x7ff9181401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff91df69700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 0x7ff9181401e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff9140091e0 tx=0x7ff91401e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914025c50 con 0x7ff9181340f0 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff90dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff90dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.736+0000 7ff90dffb700 10 monclient: dump: 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7ff914025db0 con 0x7ff9181340f0 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 msgr2=0x7ff9181401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 0x7ff9181401e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff9140091e0 tx=0x7ff91401e070 comp rx=0 tx=0).stop 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 msgr2=0x7ff9181401e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff90dffb700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9181340f0 0x7ff9181401e0 secure :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff9140091e0 tx=0x7ff91401e070 comp rx=0 tx=0).stop 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:12:56.748 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 50, 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:12:56.749 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.750 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:12:56.751 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:12:56.752 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:12:56.753 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.754 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.755 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:12:56.756 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.759 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:12:56.760 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff8fc035af0 msgr2=0x7ff8fc037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff8fc035af0 0x7ff8fc037fa0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff918139190 tx=0x7ff910002620 comp rx=0 tx=0).stop 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 10 monclient: shutdown 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 msgr2=0x7ff918133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 --2- 172.21.15.142:0/3346487585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff918097350 0x7ff918133bb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff90800c5e0 tx=0x7ff908004a80 comp rx=0 tx=0).stop 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91e76a700 1 -- 172.21.15.142:0/3346487585 reap_dead start 2024-09-18T10:12:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91e76a700 1 -- 172.21.15.142:0/3346487585 reap_dead start 2024-09-18T10:12:56.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 shutdown_connections 2024-09-18T10:12:56.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 >> 172.21.15.142:0/3346487585 conn(0x7ff9180717a0 msgr2=0x7ff91809c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 shutdown_connections 2024-09-18T10:12:56.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.740+0000 7ff91ef6b700 1 -- 172.21.15.142:0/3346487585 wait complete. 2024-09-18T10:12:56.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:12:56.763 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:12:56.763 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:12:56.763 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:12:56.764 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:12:56.764 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:12:56.764 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:12:56.783 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:12:56.783 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:12:56.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:12:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: get_monmap_and_config 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: monmap: 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:12:56.889923+0000 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:12:56.889923+0000 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:12:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:12:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:12:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:12:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:12:56.892 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 Processor -- start 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 -- start start 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: init 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48a406f0a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f48a409f360 0x7f48a4137370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f48a409f360 addr v2:172.21.15.142:3300/0 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48a41378b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a409ef00 con 0x7f48a41378b0 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a406a2e0 con 0x7f48a406f0a0 2024-09-18T10:12:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a406a000 con 0x7f48a409f360 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.884+0000 7f48a3fff700 10 monclient(hunting): authenticate will time out at 1065.815999s 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a37fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f48a41378b0 legacy=0x7f48a4139c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a27fc700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f48a409f360 msgr2=0x7f48a4137370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a27fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f48a409f360 0x7f48a4137370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a2ffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f48a406f0a0 0x7f48a406f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:34058/0 (socket says 172.21.15.142:34058) 2024-09-18T10:12:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a2ffd700 1 -- 172.21.15.142:0/2791773571 learned_addr learned my addr 172.21.15.142:0/2791773571 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:12:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a2ffd700 1 -- 172.21.15.142:0/2791773571 >> v1:172.21.15.150:6789/0 conn(0x7f48a406f0a0 legacy=0x7f48a406f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023098837 0 0) 0x7f48a406a2e0 con 0x7f48a406f0a0 2024-09-18T10:12:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 10 monclient(hunting): my global_id is 15127 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c003570 con 0x7f48a406f0a0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645555635 0 0) 0x7f488c003570 con 0x7f48a406f0a0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 >> v2:172.21.15.142:3300/0 conn(0x7f48a409f360 msgr2=0x7f48a4137370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 --2- 172.21.15.142:0/2791773571 >> v2:172.21.15.142:3300/0 conn(0x7f48a409f360 0x7f48a4137370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 >> v1:172.21.15.142:6789/0 conn(0x7f48a41378b0 legacy=0x7f48a4139c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.888+0000 7f48a1ffb700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a406fb80 con 0x7f48a406f0a0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a3fff700 5 monclient: authenticate success, global_id 15127 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f48940017a0 con 0x7f48a406f0a0 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: dump: 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 >> v1:172.21.15.150:6789/0 conn(0x7f48a406f0a0 legacy=0x7f48a406f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f488c003a60 0x7f488c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): picked mon.a con 0x7f488c003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c004810 0x7f488c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): picked mon.b con 0x7f488c004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 0x7f488c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): picked mon.c con 0x7f488c0053c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a406a000 con 0x7f488c003a60 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c00c380 con 0x7f488c004810 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c00c540 con 0x7f488c0053c0 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a3fff700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a2ffd700 1 -- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f488c003a60 msgr2=0x7f488c003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a2ffd700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f488c003a60 0x7f488c003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 0x7f488c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c004810 0x7f488c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): get_auth_request con 0x7f488c0053c0 auth_method 0 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): get_auth_request con 0x7f488c004810 auth_method 0 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a27fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient(hunting): handle_auth_done global_id 15127 payload 293 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 -- 172.21.15.142:0/2791773571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c004810 msgr2=0x7f488c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c004810 0x7f488c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 -- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f488c003a60 msgr2=0x7f488c003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f488c003a60 0x7f488c003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 monclient: found mon.c 2024-09-18T10:12:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 -- 172.21.15.142:0/2791773571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c003570 con 0x7f488c0053c0 2024-09-18T10:12:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 -- 172.21.15.142:0/2791773571 reap_dead start 2024-09-18T10:12:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 -- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 msgr2=0x7f488c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.892+0000 7f48a37fe700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 0x7f488c00baa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f489c003d50 tx=0x7f489c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63eb56404f2b5d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c009920 con 0x7f488c0053c0 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: dump: 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:12:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c015470 con 0x7f488c0053c0 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 1 -- 172.21.15.142:0/2791773571 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c015810 con 0x7f488c0053c0 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 4 set_mon_vals no callback set 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a1ffb700 10 monclient: dump: 2024-09-18T10:12:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: shutdown 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 msgr2=0x7f488c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 0x7f488c00baa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f489c003d50 tx=0x7f489c0095f0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- 172.21.15.142:0/2791773571 shutdown_connections 2024-09-18T10:12:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 --2- 172.21.15.142:0/2791773571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f488c0053c0 0x7f488c00baa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- 172.21.15.142:0/2791773571 >> 172.21.15.142:0/2791773571 conn(0x7f48a4096a50 msgr2=0x7f48a4071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- 172.21.15.142:0/2791773571 shutdown_connections 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- 172.21.15.142:0/2791773571 wait complete. 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: build_initial_monmap 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: monmap: 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:12:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 Processor -- start 2024-09-18T10:12:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 -- start start 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: init 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.896+0000 7f48a3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0053c0 0x7f48a411eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f488c0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:12:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 0x7f48a411f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f48a411f430 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f48a411fd40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a4120a20 con 0x7f48a411f430 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a409f840 con 0x7f488c0053c0 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a409ae40 con 0x7f48a411fd40 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient(hunting): authenticate will time out at 1065.828130s 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a27fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 msgr2=0x7f48a411f800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a27fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 0x7f48a411f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45154/0 (socket says 172.21.15.142:45154) 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 -- 172.21.15.142:0/594753411 learned_addr learned my addr 172.21.15.142:0/594753411 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0053c0 0x7f48a411eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): get_auth_request con 0x7f48a411fd40 auth_method 0 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): get_auth_request con 0x7f488c0053c0 auth_method 0 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 -- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0053c0 msgr2=0x7f48a411eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0053c0 0x7f48a411eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 -- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 msgr2=0x7f48a411f800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 0x7f48a411f800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:56.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 monclient: found mon.noname-c 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 -- 172.21.15.142:0/594753411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4121670 con 0x7f48a411fd40 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 -- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 msgr2=0x7f48a4147d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f489c009630 tx=0x7f489c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a37fe700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f489c009630 tx=0x7f489c015000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f489c00c850 config(0 keys) v1 2024-09-18T10:12:56.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c01d070 con 0x7f48a411fd40 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 10 monclient: dump: 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.926 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 10 monclient: _finish_auth 0 2024-09-18T10:12:56.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489c00c850 con 0x7f48a411fd40 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 5 monclient: authenticate success, global_id 44153 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f488affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient: _renew_subs 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a41224c0 con 0x7f48a411fd40 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient: _renew_subs 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a17fa700 4 set_mon_vals no callback set 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.900+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a4137530 con 0x7f48a411fd40 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c00cbf0 con 0x7f48a411fd40 2024-09-18T10:12:56.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 10 monclient: dump: 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.929 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:12:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:12:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:12:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:12:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:12:56.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f489c020070 con 0x7f48a411fd40 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:56.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4120380 con 0x7f48a411fd40 2024-09-18T10:12:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f488affd700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 0x7f4890037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:12:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a2ffd700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 0x7f4890037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:12:56.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.904+0000 7f48a2ffd700 10 monclient: get_auth_request con 0x7f4890035af0 auth_method 0 2024-09-18T10:12:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.908+0000 7f48a2ffd700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 0x7f4890037fa0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4894001ed0 tx=0x7f48a406a930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:12:56.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:56.912+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7f489c049b50 con 0x7f48a411fd40 2024-09-18T10:12:57.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.728+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f489c012920 con 0x7f48a411fd40 2024-09-18T10:12:57.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.728+0000 7f488affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:12:57.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.728+0000 7f488affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:12:57.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f48a3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:12:57.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f48a3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:57.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f48a3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:12:57.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48a4072a80 con 0x7f48a411fd40 2024-09-18T10:12:57.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f488affd700 1 -- 172.21.15.142:0/594753411 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f489c029300 con 0x7f48a411fd40 2024-09-18T10:12:57.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f488affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:12:57.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.952+0000 7f488affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:12:57.960 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 msgr2=0x7f4890037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 0x7f4890037fa0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4894001ed0 tx=0x7f48a406a930 comp rx=0 tx=0).stop 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 10 monclient: shutdown 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 msgr2=0x7f48a4147d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f489c009630 tx=0x7f489c015000 comp rx=0 tx=0).stop 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 shutdown_connections 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4890035af0 0x7f4890037fa0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48a411fd40 0x7f48a4147d50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48a411f430 0x7f48a411f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:57.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 --2- 172.21.15.142:0/594753411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f488c0053c0 0x7f48a411eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:12:57.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 >> 172.21.15.142:0/594753411 conn(0x7f48a4096a50 msgr2=0x7f48a409c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:12:57.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 shutdown_connections 2024-09-18T10:12:57.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:12:57.956+0000 7f48a3fff700 1 -- 172.21.15.142:0/594753411 wait complete. 2024-09-18T10:12:59.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:59.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:12:59.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:59.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:12:59.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:12:59.516+0000 7f038bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:00.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:00.512+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:00.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:00.512+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:00.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:01.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.076+0000 7f29e6efb700 10 monclient: get_monmap_and_config 2024-09-18T10:13:01.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.076+0000 7f29e6efb700 10 monclient: build_initial_monmap 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.076+0000 7f29e6efb700 10 monclient: monmap: 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:01.083574+0000 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:01.083574+0000 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:01.085 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:01.086 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:01.086 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 Processor -- start 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 -- start start 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient: init 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-b con 0x7f29e009cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f29e00a02e0 0x7f29e00a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-c con 0x7f29e00a02e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f29e00a0bf0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e009d1a0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e009e980 con 0x7f29e00a02e0 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e009d780 con 0x7f29e009cdd0 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e6efb700 10 monclient(hunting): authenticate will time out at 1070.009492s 2024-09-18T10:13:01.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45170/0 (socket says 172.21.15.142:45170) 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 -- 172.21.15.142:0/2174556226 learned_addr learned my addr 172.21.15.142:0/2174556226 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): get_auth_request con 0x7f29e009cdd0 auth_method 0 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 1 --2- 172.21.15.142:0/2174556226 >> v2:172.21.15.150:3300/0 conn(0x7f29e00a02e0 0x7f29e00a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): get_auth_request con 0x7f29e00a02e0 auth_method 0 2024-09-18T10:13:01.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient(hunting): handle_auth_done global_id 44156 payload 274 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 -- 172.21.15.142:0/2174556226 >> v2:172.21.15.150:3300/0 conn(0x7f29e00a02e0 msgr2=0x7f29e00a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 --2- 172.21.15.142:0/2174556226 >> v2:172.21.15.150:3300/0 conn(0x7f29e00a02e0 0x7f29e00a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 -- 172.21.15.142:0/2174556226 >> v1:172.21.15.150:6789/0 conn(0x7f29e00a0bf0 legacy=0x7f29e0138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient: _finish_hunting 0 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 monclient: found mon.noname-b 2024-09-18T10:13:01.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:01.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.080+0000 7f29e4c97700 1 -- 172.21.15.142:0/2174556226 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e009dd40 con 0x7f29e009cdd0 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e4c97700 1 -- 172.21.15.142:0/2174556226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 msgr2=0x7f29e009e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e4c97700 1 --2- 172.21.15.142:0/2174556226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29c80099b0 tx=0x7f29c8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=370a3801a39db0b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 1 -- 172.21.15.142:0/2174556226 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8009ea0 con 0x7f29e009cdd0 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:01.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: dump: 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:01.099 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:01.100 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:01.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:01.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 5 monclient: authenticate success, global_id 44156 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 1 -- 172.21.15.142:0/2174556226 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c800ba70 con 0x7f29e009cdd0 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 4 set_mon_vals no callback set 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 1 -- 172.21.15.142:0/2174556226 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c800be30 con 0x7f29e009cdd0 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29df7fe700 10 monclient: dump: 2024-09-18T10:13:01.101 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 10 monclient: shutdown 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 -- 172.21.15.142:0/2174556226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 msgr2=0x7f29e009e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 --2- 172.21.15.142:0/2174556226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29c80099b0 tx=0x7f29c8009250 comp rx=0 tx=0).stop 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 -- 172.21.15.142:0/2174556226 shutdown_connections 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 --2- 172.21.15.142:0/2174556226 >> v2:172.21.15.150:3300/0 conn(0x7f29e00a02e0 0x7f29e00a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 --2- 172.21.15.142:0/2174556226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e009cdd0 0x7f29e009e440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 -- 172.21.15.142:0/2174556226 >> 172.21.15.142:0/2174556226 conn(0x7f29e0071680 msgr2=0x7f29e00a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 -- 172.21.15.142:0/2174556226 shutdown_connections 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 1 -- 172.21.15.142:0/2174556226 wait complete. 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 10 monclient: build_initial_monmap 2024-09-18T10:13:01.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.084+0000 7f29e6efb700 10 monclient: monmap: 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:01.104 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 Processor -- start 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 -- start start 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient: init 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29e009cdd0 0x7f29e0141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-b con 0x7f29e009cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 0x7f29e01422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-a con 0x7f29e00a02e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): picked mon.noname-c con 0x7f29e00a0bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:01.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00900c0 con 0x7f29e00a02e0 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0090640 con 0x7f29e009cdd0 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e009c070 con 0x7f29e00a0bf0 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e6efb700 10 monclient(hunting): authenticate will time out at 1070.017746s 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29dffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 msgr2=0x7f29e01422a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29dffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 0x7f29e01422a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45178/0 (socket says 172.21.15.142:45178) 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 -- 172.21.15.142:0/428360669 learned_addr learned my addr 172.21.15.142:0/428360669 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:01.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): get_auth_request con 0x7f29e00a0bf0 auth_method 0 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29e009cdd0 0x7f29e0141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): get_auth_request con 0x7f29e009cdd0 auth_method 0 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:01.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e4c97700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient(hunting): handle_auth_done global_id 44159 payload 824 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 -- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29e009cdd0 msgr2=0x7f29e0141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29e009cdd0 0x7f29e0141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 -- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 msgr2=0x7f29e01422a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 0x7f29e01422a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient: _finish_hunting 0 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 monclient: found mon.noname-c 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:01.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.088+0000 7f29e5498700 1 -- 172.21.15.142:0/428360669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c80094f0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e5498700 1 -- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 msgr2=0x7f29e008ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e5498700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c5e0 tx=0x7f29d8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d80117a0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: dump: 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.109 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8011900 con 0x7f29e00a0bf0 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 5 monclient: authenticate success, global_id 44159 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8011cc0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:01.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 10 monclient: dump: 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29deffd700 4 set_mon_vals no callback set 2024-09-18T10:13:01.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _renew_subs 2024-09-18T10:13:01.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e0091510 con 0x7f29e00a0bf0 2024-09-18T10:13:01.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _renew_subs 2024-09-18T10:13:01.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e008f400 con 0x7f29e00a0bf0 2024-09-18T10:13:01.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:01.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e00696a0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f29d8012630 con 0x7f29e00a0bf0 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 0x7f29c0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7f29d80484e0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.092+0000 7f29e4c97700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 0x7f29c0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:01.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.096+0000 7f29e4c97700 10 monclient: get_auth_request con 0x7f29c0035a70 auth_method 0 2024-09-18T10:13:01.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.096+0000 7f29e4c97700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 0x7f29c0037f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29c80099b0 tx=0x7f29c8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:01.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.104+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f29d80108c0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.104+0000 7f29dcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:01.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.104+0000 7f29dcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:01.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.324+0000 7f29e6efb700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.324+0000 7f29e6efb700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.324+0000 7f29e6efb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.324+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f29e00678c0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.324+0000 7f29dcff9700 1 -- 172.21.15.142:0/428360669 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f29d80102e0 con 0x7f29e00a0bf0 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29dcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:01.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29dcff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:01.333 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:01.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 msgr2=0x7f29c0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:01.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 0x7f29c0037f20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29c80099b0 tx=0x7f29c8016000 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 10 monclient: shutdown 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 msgr2=0x7f29e008ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c5e0 tx=0x7f29d8007f60 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 shutdown_connections 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f29c0035a70 0x7f29c0037f20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29e00a0bf0 0x7f29e008ce60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29e00a02e0 0x7f29e01422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 --2- 172.21.15.142:0/428360669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29e009cdd0 0x7f29e0141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 >> 172.21.15.142:0/428360669 conn(0x7f29e0071680 msgr2=0x7f29e00a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 shutdown_connections 2024-09-18T10:13:01.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:01.328+0000 7f29e6efb700 1 -- 172.21.15.142:0/428360669 wait complete. 2024-09-18T10:13:01.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:01.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:01.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:01.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:02.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:02.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:02.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:02.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:03.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:03.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:03.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:03.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:04.346 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:04.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.448+0000 7fbf65ebf700 10 monclient: get_monmap_and_config 2024-09-18T10:13:04.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.448+0000 7fbf65ebf700 10 monclient: build_initial_monmap 2024-09-18T10:13:04.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.448+0000 7fbf65ebf700 10 monclient: monmap: 2024-09-18T10:13:04.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:04.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:04.455544+0000 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:04.455544+0000 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:04.457 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:04.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 Processor -- start 2024-09-18T10:13:04.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 -- start start 2024-09-18T10:13:04.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient: init 2024-09-18T10:13:04.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-a con 0x7fbf60132b80 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf60132070 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf60137230 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf60132470 con 0x7fbf60137230 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf60096ca0 con 0x7fbf60132070 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf60096f80 con 0x7fbf60132b80 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf65ebf700 10 monclient(hunting): authenticate will time out at 1073.381298s 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf64ebd700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 msgr2=0x7fbf600a2670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:04.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf64ebd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:04.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf656be700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fbf60137230 legacy=0x7fbf60139610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:04.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf5ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbf60132070 0x7fbf60098790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37428/0 (socket says 172.21.15.142:37428) 2024-09-18T10:13:04.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.452+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 learned_addr learned my addr 172.21.15.142:0/4292901517 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:04.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.456+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.150:6789/0 conn(0x7fbf60132070 legacy=0x7fbf60098790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:04.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:04.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.656+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 msgr2=0x7fbf600a2670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:04.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.656+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.142:6789/0 conn(0x7fbf60137230 legacy=0x7fbf60139610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:04.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:04.656+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:05.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.060+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 msgr2=0x7fbf600a2670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:05.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.060+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.142:6789/0 conn(0x7fbf60137230 legacy=0x7fbf60139610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:05.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.060+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:05.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:05.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:05.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:05.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:05.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.864+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.142:6789/0 conn(0x7fbf60137230 legacy=0x7fbf60139610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:05.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.864+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 msgr2=0x7fbf600a2670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:05.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:05.864+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:13:06.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:06.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:06.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:06.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:07.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): tick 2024-09-18T10:13:07.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 monclient(hunting): continuing hunt 2024-09-18T10:13:07.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 msgr2=0x7fbf600a2670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf60132b80 0x7fbf600a2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.150:6789/0 conn(0x7fbf60132070 legacy=0x7fbf60098790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.142:6789/0 conn(0x7fbf60137230 legacy=0x7fbf60139610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient: _add_conns ranks=[4,2] 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fbf40003790 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fbf40006290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf60096f80 con 0x7fbf40003790 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40008e00 con 0x7fbf40006290 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5e7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40003790 auth_method 0 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:07.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.456+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:07.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:07.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:07.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:07.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:07.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40003790 auth_method 0 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:07.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:07.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:07.660+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40003790 auth_method 0 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:08.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:08.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.064+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:08.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:08.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:08.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:08.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:08.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40003790 auth_method 0 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:08.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:08.868+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:13:09.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:09.516+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:09.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:09.516+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:09.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:09.516+0000 7f038bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:10.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): tick 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 monclient(hunting): continuing hunt 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 msgr2=0x7fbf40005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:10.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.150:3300/0 conn(0x7fbf40003790 0x7fbf40005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:10.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T10:13:10.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 reap_dead start 2024-09-18T10:13:10.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:10.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fbf40006290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:10.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 0x7fbf40009200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fbf40003bf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf40009740 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf60098ea0 con 0x7fbf40009740 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf60096f80 con 0x7fbf40003bf0 2024-09-18T10:13:10.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40008e00 con 0x7fbf40006290 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5e7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 msgr2=0x7fbf40009200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:10.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:10.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 0x7fbf40009200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.452+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:10.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.456+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.150:6789/0 conn(0x7fbf40009740 legacy=0x7fbf4000bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:10.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:10.520+0000 7f038bfff700 10 monclient: tick 2024-09-18T10:13:10.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:10.520+0000 7f038bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 msgr2=0x7fbf40009200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 0x7fbf40009200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:10.656+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf5ffff700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 msgr2=0x7fbf40009200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 0x7fbf40009200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:11.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:11.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:11.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:11.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.060+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.312+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585400237 0 0) 0x7fbf60098ea0 con 0x7fbf40009740 2024-09-18T10:13:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.312+0000 7fbf5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.312+0000 7fbf5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.312+0000 7fbf5f7fe700 10 monclient(hunting): my global_id is 15139 2024-09-18T10:13:11.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.312+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf4c003570 con 0x7fbf40009740 2024-09-18T10:13:11.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771629057 0 0) 0x7fbf4c003570 con 0x7fbf40009740 2024-09-18T10:13:11.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf40008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf40008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 msgr2=0x7fbf40009200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 --2- 172.21.15.142:0/4292901517 >> v2:172.21.15.142:3300/0 conn(0x7fbf40003bf0 0x7fbf40009200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60132470 con 0x7fbf40009740 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fbf50004010 con 0x7fbf40009740 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:11.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: dump: 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 >> v1:172.21.15.150:6789/0 conn(0x7fbf40009740 legacy=0x7fbf4000bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf4c003a60 0x7fbf4c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): picked mon.a con 0x7fbf4c003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf4c004810 0x7fbf4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): picked mon.b con 0x7fbf4c004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf4c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): picked mon.c con 0x7fbf40006290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60096f80 con 0x7fbf4c003a60 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf64ebd700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf4c003a60 msgr2=0x7fbf4c003e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40008e00 con 0x7fbf4c004810 2024-09-18T10:13:11.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c00bc70 con 0x7fbf40006290 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf4c003a60 0x7fbf4c003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf4c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf4c004810 0x7fbf4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): get_auth_request con 0x7fbf40006290 auth_method 0 2024-09-18T10:13:11.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf4c004810 auth_method 0 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:11.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:11.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient(hunting): handle_auth_done global_id 15139 payload 293 2024-09-18T10:13:11.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf4c004810 msgr2=0x7fbf4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf4c004810 0x7fbf4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf4c003a60 msgr2=0x7fbf4c003e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf4c003a60 0x7fbf4c003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 monclient: found mon.c 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c003570 con 0x7fbf40006290 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf656be700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf65ebf700 5 monclient: authenticate success, global_id 15139 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf65ebf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.320+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf4c004810 0x7fbf4c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:11.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf4c005270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf656be700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf4c005270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf601327f0 tx=0x7fbf500078b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38eed1ad2a55225a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf50007bc0 con 0x7fbf40006290 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: dump: 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.341 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.343 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50007150 con 0x7fbf40006290 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 1 -- 172.21.15.142:0/4292901517 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5000cbc0 con 0x7fbf40006290 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 4 set_mon_vals no callback set 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf5f7fe700 10 monclient: dump: 2024-09-18T10:13:11.345 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.345 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.345 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.345 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.345 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 10 monclient: shutdown 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 -- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 msgr2=0x7fbf4c005270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/4292901517 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf40006290 0x7fbf4c005270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbf601327f0 tx=0x7fbf500078b0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf656be700 1 -- 172.21.15.142:0/4292901517 reap_dead start 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 -- 172.21.15.142:0/4292901517 shutdown_connections 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 -- 172.21.15.142:0/4292901517 >> 172.21.15.142:0/4292901517 conn(0x7fbf60071700 msgr2=0x7fbf600a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 -- 172.21.15.142:0/4292901517 shutdown_connections 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.324+0000 7fbf65ebf700 1 -- 172.21.15.142:0/4292901517 wait complete. 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient: monmap: 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.347 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 Processor -- start 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 -- start start 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient: init 2024-09-18T10:13:11.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 0x7fbf601366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-a con 0x7fbf60141810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-c con 0x7fbf60141be0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6012a420 0x7fbf6012a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): picked mon.noname-b con 0x7fbf6012a420 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60135710 con 0x7fbf60141810 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6012f7f0 con 0x7fbf6012a420 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60135430 con 0x7fbf60141be0 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf65ebf700 10 monclient(hunting): authenticate will time out at 1080.259560s 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf64ebd700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 msgr2=0x7fbf601366b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf64ebd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 0x7fbf601366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf656be700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6012a420 0x7fbf6012a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54052/0 (socket says 172.21.15.142:54052) 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 1 -- 172.21.15.142:0/1845739303 learned_addr learned my addr 172.21.15.142:0/1845739303 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request con 0x7fbf60141be0 auth_method 0 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.328+0000 7fbf5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): get_auth_request con 0x7fbf6012a420 auth_method 0 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf656be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient(hunting): handle_auth_done global_id 44162 payload 824 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 -- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6012a420 msgr2=0x7fbf6012a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6012a420 0x7fbf6012a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 -- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 msgr2=0x7fbf601366b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 0x7fbf601366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 monclient: found mon.noname-c 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 -- 172.21.15.142:0/1845739303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf601359f0 con 0x7fbf60141be0 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 -- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 msgr2=0x7fbf60141fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5ffff700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbf58015380 tx=0x7fbf58009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58017640 con 0x7fbf60141be0 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: dump: 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.354 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580177a0 con 0x7fbf60141be0 2024-09-18T10:13:11.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58017b20 con 0x7fbf60141be0 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5cff9700 10 monclient: dump: 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.356 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 5 monclient: authenticate success, global_id 44162 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 10 monclient: _renew_subs 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf5effd700 4 set_mon_vals no callback set 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf601346b0 con 0x7fbf60141be0 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 10 monclient: _renew_subs 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.332+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf60142690 con 0x7fbf60141be0 2024-09-18T10:13:11.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf65ebf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf65ebf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf65ebf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbf580293f0 con 0x7fbf60141be0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf60072a80 con 0x7fbf60141be0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf5cff9700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 0x7fbf400390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(97..97 src has 40..97) v4 ==== 6015+0+0 (secure 0 0 0) 0x7fbf58049f20 con 0x7fbf60141be0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 0x7fbf400390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.336+0000 7fbf64ebd700 10 monclient: get_auth_request con 0x7fbf40036c40 auth_method 0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.340+0000 7fbf64ebd700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 0x7fbf400390f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009540 tx=0x7fbf48005590 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.344+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbf5801dd50 con 0x7fbf60141be0 2024-09-18T10:13:11.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.344+0000 7fbf5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.344+0000 7fbf5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:11.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.384+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}]=0 moved item id 8 name 'osd.8' to location {host=foo} in crush map v98) v1 ==== 165+0+0 (secure 0 0 0) 0x7f0398011360 con 0x7f03a01379b0 2024-09-18T10:13:11.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.384+0000 7f03957fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:13:11.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.384+0000 7f03957fa700 10 monclient: _finish_command 2 = system:0 moved item id 8 name 'osd.8' to location {host=foo} in crush map 2024-09-18T10:13:11.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03a76df700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:13:11.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03a76df700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:13:11.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03a76df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]} v 0) v1 -- 0x7f03a012d470 con 0x7f03a01379b0 2024-09-18T10:13:11.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03957fa700 1 -- 172.21.15.150:0/1181836612 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}]=0 no need to move item id 8 name 'osd.8' to location {host=foo} in crush map v98) v1 ==== 175+0+0 (secure 0 0 0) 0x7f039800e290 con 0x7f03a01379b0 2024-09-18T10:13:11.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03957fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "osd.8", "args": ["host=foo"]}] 2024-09-18T10:13:11.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.492+0000 7f03957fa700 10 monclient: _finish_command 3 = system:0 no need to move item id 8 name 'osd.8' to location {host=foo} in crush map 2024-09-18T10:13:11.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 msgr2=0x7f0384037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 0x7f0384037f70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f03900058d0 tx=0x7f0390007bb0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 10 monclient: shutdown 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 msgr2=0x7f03a012cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 0x7f03a012cf30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f039800c8e0 tx=0x7f03980018c0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 shutdown_connections 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0384035ac0 0x7f0384037f70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03a01379b0 0x7f03a012cf30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03a00a0c00 0x7f03a012c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 --2- 172.21.15.150:0/1181836612 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03a00a0120 0x7f03a012c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 >> 172.21.15.150:0/1181836612 conn(0x7f03a00a5830 msgr2=0x7f03a009aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 shutdown_connections 2024-09-18T10:13:11.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.496+0000 7f03a76df700 1 -- 172.21.15.150:0/1181836612 wait complete. 2024-09-18T10:13:11.503 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id 8 name 'osd.8' to location {host=foo} in crush map 2024-09-18T10:13:11.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd find osd.8 2024-09-18T10:13:11.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep host 2024-09-18T10:13:11.512 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep foo 2024-09-18T10:13:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.576+0000 7fbf65ebf700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.576+0000 7fbf65ebf700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.576+0000 7fbf65ebf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.576+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbf6012fe80 con 0x7fbf60141be0 2024-09-18T10:13:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf5cff9700 1 -- 172.21.15.142:0/1845739303 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fbf5801d780 con 0x7fbf60141be0 2024-09-18T10:13:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:11.586 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":52,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 msgr2=0x7fbf400390f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 0x7fbf400390f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009540 tx=0x7fbf48005590 comp rx=0 tx=0).stop 2024-09-18T10:13:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 10 monclient: shutdown 2024-09-18T10:13:11.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 msgr2=0x7fbf60141fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbf58015380 tx=0x7fbf58009040 comp rx=0 tx=0).stop 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 shutdown_connections 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf40036c40 0x7fbf400390f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6012a420 0x7fbf6012a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf60141be0 0x7fbf60141fb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 --2- 172.21.15.142:0/1845739303 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf60141810 0x7fbf601366b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 >> 172.21.15.142:0/1845739303 conn(0x7fbf60071700 msgr2=0x7fbf6012b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 shutdown_connections 2024-09-18T10:13:11.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.580+0000 7fbf65ebf700 1 -- 172.21.15.142:0/1845739303 wait complete. 2024-09-18T10:13:11.599 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:13:11.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:13:11.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:13:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: get_monmap_and_config 2024-09-18T10:13:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: monmap: 2024-09-18T10:13:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:11.609820+0000 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:11.609820+0000 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:11.612 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 Processor -- start 2024-09-18T10:13:11.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 -- start start 2024-09-18T10:13:11.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: init 2024-09-18T10:13:11.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15100a6090 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f151009e460 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f151009d0f0 0x7f151013ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): picked mon.noname-a con 0x7f151009d0f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15100a6460 con 0x7f15100a6090 2024-09-18T10:13:11.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15100a1190 con 0x7f151009e460 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151009d530 con 0x7f151009d0f0 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f151773c700 10 monclient(hunting): authenticate will time out at 1073.703390s 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f1515cd9700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f151009d0f0 msgr2=0x7f151013ab20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f15154d8700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f15100a6090 legacy=0x7f15100a0370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f1515cd9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f151009d0f0 0x7f151013ab20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f1514cd7700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f151009e460 0x7f15100a0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44610/0 (socket says 172.21.15.150:44610) 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.604+0000 7f1514cd7700 1 -- 172.21.15.150:0/2423915870 learned_addr learned my addr 172.21.15.150:0/2423915870 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 1 -- 172.21.15.150:0/2423915870 >> v1:172.21.15.150:6789/0 conn(0x7f151009e460 legacy=0x7f15100a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240921644 0 0) 0x7f15100a1190 con 0x7f151009e460 2024-09-18T10:13:11.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): my global_id is 15145 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14fc003570 con 0x7f151009e460 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201860831 0 0) 0x7f14fc003570 con 0x7f151009e460 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 >> v2:172.21.15.142:3300/0 conn(0x7f151009d0f0 msgr2=0x7f151013ab20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 --2- 172.21.15.150:0/2423915870 >> v2:172.21.15.142:3300/0 conn(0x7f151009d0f0 0x7f151013ab20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 >> v1:172.21.15.142:6789/0 conn(0x7f15100a6090 legacy=0x7f15100a0370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151009fd00 con 0x7f151009e460 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1500001860 con 0x7f151009e460 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f151773c700 5 monclient: authenticate success, global_id 15145 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f151773c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: dump: 2024-09-18T10:13:11.639 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.640 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 >> v1:172.21.15.150:6789/0 conn(0x7f151009e460 legacy=0x7f15100a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14fc003a60 0x7f14fc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): picked mon.a con 0x7f14fc003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc004810 0x7f14fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): picked mon.c con 0x7f14fc004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 0x7f14fc00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): picked mon.b con 0x7f14fc0053f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151009d530 con 0x7f14fc003a60 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc00c3c0 con 0x7f14fc0053f0 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14fc00c580 con 0x7f14fc004810 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1507fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f151773c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f15154d8700 1 -- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14fc003a60 msgr2=0x7f14fc003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f15154d8700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14fc003a60 0x7f14fc003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 0x7f14fc00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): get_auth_request con 0x7f14fc0053f0 auth_method 0 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc004810 0x7f14fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 10 monclient(hunting): get_auth_request con 0x7f14fc004810 auth_method 0 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient(hunting): handle_auth_done global_id 15145 payload 293 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 -- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc004810 msgr2=0x7f14fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc004810 0x7f14fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 -- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14fc003a60 msgr2=0x7f14fc003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f14fc003a60 0x7f14fc003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 monclient: found mon.b 2024-09-18T10:13:11.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 -- 172.21.15.150:0/2423915870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14fc003570 con 0x7f14fc0053f0 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1514cd7700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc004810 0x7f14fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.608+0000 7f1515cd9700 1 -- 172.21.15.150:0/2423915870 reap_dead start 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1515cd9700 1 -- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 msgr2=0x7f14fc00bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1515cd9700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 0x7f14fc00bae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1508009a50 tx=0x7f15080090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e16c9ab3bc924bc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508009e90 con 0x7f14fc0053f0 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: dump: 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1508009340 con 0x7f14fc0053f0 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 4 set_mon_vals no callback set 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 1 -- 172.21.15.150:0/2423915870 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508003e60 con 0x7f14fc0053f0 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f1507fff700 10 monclient: dump: 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.648 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 10 monclient: shutdown 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 msgr2=0x7f14fc00bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 0x7f14fc00bae0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1508009a50 tx=0x7f15080090f0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- 172.21.15.150:0/2423915870 shutdown_connections 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 --2- 172.21.15.150:0/2423915870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14fc0053f0 0x7f14fc00bae0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- 172.21.15.150:0/2423915870 >> 172.21.15.150:0/2423915870 conn(0x7f1510071830 msgr2=0x7f15100a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- 172.21.15.150:0/2423915870 shutdown_connections 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- 172.21.15.150:0/2423915870 wait complete. 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 10 monclient: monmap: 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:11.650 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 Processor -- start 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 1 -- start start 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.612+0000 7f151773c700 10 monclient: init 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 0x7f151008cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): picked mon.noname-c con 0x7f14fc0053f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 0x7f151008d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): picked mon.noname-a con 0x7f151008d4d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 0x7f1510147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): picked mon.noname-b con 0x7f151008dde0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510091510 con 0x7f151008d4d0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151009c120 con 0x7f151008dde0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1510090940 con 0x7f14fc0053f0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f151773c700 10 monclient(hunting): authenticate will time out at 1073.713249s 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1514cd7700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 msgr2=0x7f151008d8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1514cd7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 0x7f151008d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 0x7f151008cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 0x7f1510147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 0x7f151008cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:36412/0 (socket says 172.21.15.150:36412) 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 1 -- 172.21.15.150:0/4085765861 learned_addr learned my addr 172.21.15.150:0/4085765861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:11.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): get_auth_request con 0x7f14fc0053f0 auth_method 0 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): get_auth_request con 0x7f151008dde0 auth_method 0 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f15154d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient(hunting): handle_auth_done global_id 15148 payload 824 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 -- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 msgr2=0x7f151008cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 0x7f151008cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 -- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 msgr2=0x7f151008d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 0x7f151008d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 monclient: found mon.noname-b 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.616+0000 7f1515cd9700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100905d0 con 0x7f151008dde0 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f1515cd9700 1 -- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 msgr2=0x7f1510147bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f1515cd9700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 0x7f1510147bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1510068eb0 tx=0x7f1508015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508022070 con 0x7f151008dde0 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: dump: 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.655 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 5 monclient: authenticate success, global_id 15148 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _renew_subs 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150800bbc0 con 0x7f151008dde0 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151008f390 con 0x7f151008dde0 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150801d430 con 0x7f151008dde0 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f1506ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 10 monclient: dump: 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.657 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _renew_subs 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510098430 con 0x7f151008dde0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f150801d7e0 con 0x7f151008dde0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510090760 con 0x7f151008dde0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 0x7f14f4037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(98..98 src has 40..98) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f1508049c30 con 0x7f151008dde0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15154d8700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 0x7f14f4037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.620+0000 7f15154d8700 10 monclient: get_auth_request con 0x7f14f4035aa0 auth_method 0 2024-09-18T10:13:11.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.624+0000 7f15154d8700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 0x7f14f4037f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f150c0062a0 tx=0x7f150c0097b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.628+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f15080105a0 con 0x7f151008dde0 2024-09-18T10:13:11.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.628+0000 7f15057fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.628+0000 7f15057fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:11.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: get_monmap_and_config 2024-09-18T10:13:11.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: monmap: 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:11.704370+0000 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:11.704370+0000 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:11.706 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:11.707 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:11.707 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:11.707 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:11.707 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 Processor -- start 2024-09-18T10:13:11.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 -- start start 2024-09-18T10:13:11.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: init 2024-09-18T10:13:11.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T10:13:11.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 0x7fc66c0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-a con 0x7fc66c09abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:11.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-b con 0x7fc66c09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc66c09c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66c09dc90 con 0x7fc66c09c410 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66c09b500 con 0x7fc66c09abd0 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c09afa0 con 0x7fc66c09b740 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc6709fd700 10 monclient(hunting): authenticate will time out at 1080.629791s 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc66bfff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fc66c09c410 legacy=0x7fc66c136680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc66b7fe700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 msgr2=0x7fc66c0960e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc66b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 0x7fc66c0960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54062/0 (socket says 172.21.15.142:54062) 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 -- 172.21.15.142:0/1471361338 learned_addr learned my addr 172.21.15.142:0/1471361338 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): get_auth_request con 0x7fc66c09b740 auth_method 0 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient(hunting): handle_auth_done global_id 44168 payload 274 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 -- 172.21.15.142:0/1471361338 >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 msgr2=0x7fc66c0960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 --2- 172.21.15.142:0/1471361338 >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 0x7fc66c0960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 -- 172.21.15.142:0/1471361338 >> v1:172.21.15.142:6789/0 conn(0x7fc66c09c410 legacy=0x7fc66c136680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 monclient: found mon.noname-b 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.700+0000 7fc663fff700 1 -- 172.21.15.142:0/1471361338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66c06f810 con 0x7fc66c09b740 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc663fff700 1 -- 172.21.15.142:0/1471361338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 msgr2=0x7fc66c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc663fff700 1 --2- 172.21.15.142:0/1471361338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6540099b0 tx=0x7fc654009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32781a960c53286d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 1 -- 172.21.15.142:0/1471361338 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400bb50 con 0x7fc66c09b740 2024-09-18T10:13:11.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: dump: 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 1 -- 172.21.15.142:0/1471361338 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65400bcb0 con 0x7fc66c09b740 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 5 monclient: authenticate success, global_id 44168 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 1 -- 172.21.15.142:0/1471361338 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc654011670 con 0x7fc66c09b740 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 4 set_mon_vals no callback set 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc66affd700 10 monclient: dump: 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.716 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 10 monclient: shutdown 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 -- 172.21.15.142:0/1471361338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 msgr2=0x7fc66c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1471361338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc6540099b0 tx=0x7fc654009250 comp rx=0 tx=0).stop 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 -- 172.21.15.142:0/1471361338 shutdown_connections 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1471361338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09b740 0x7fc66c09bed0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1471361338 >> v2:172.21.15.142:3300/0 conn(0x7fc66c09abd0 0x7fc66c0960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 -- 172.21.15.142:0/1471361338 >> 172.21.15.142:0/1471361338 conn(0x7fc66c070ef0 msgr2=0x7fc66c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 -- 172.21.15.142:0/1471361338 shutdown_connections 2024-09-18T10:13:11.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 1 -- 172.21.15.142:0/1471361338 wait complete. 2024-09-18T10:13:11.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.704+0000 7fc6709fd700 10 monclient: monmap: 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 Processor -- start 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 -- start start 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient: init 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66c09abd0 0x7fc66c13f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-b con 0x7fc66c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc66c09b740 0x7fc66c13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-a con 0x7fc66c09b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 0x7fc66c1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): picked mon.noname-c con 0x7fc66c09c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c08f8a0 con 0x7fc66c09b740 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c09d240 con 0x7fc66c09abd0 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c09ff00 con 0x7fc66c09c410 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc6709fd700 10 monclient(hunting): authenticate will time out at 1080.637916s 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc663fff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc66c09b740 msgr2=0x7fc66c13fc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc663fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc66c09b740 0x7fc66c13fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 0x7fc66c1401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 0x7fc66c1401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54076/0 (socket says 172.21.15.142:54076) 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 learned_addr learned my addr 172.21.15.142:0/1187873378 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): get_auth_request con 0x7fc66c09c410 auth_method 0 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66b7fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66c09abd0 0x7fc66c13f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66b7fe700 10 monclient(hunting): get_auth_request con 0x7fc66c09abd0 auth_method 0 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient(hunting): handle_auth_done global_id 44171 payload 824 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66c09abd0 msgr2=0x7fc66c13f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc66c09abd0 0x7fc66c13f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc66c09b740 msgr2=0x7fc66c13fc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc66c09b740 0x7fc66c13fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 monclient: found mon.noname-c 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.708+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6540094f0 con 0x7fc66c09c410 2024-09-18T10:13:11.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 msgr2=0x7fc66c1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66bfff700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 0x7fc66c1401c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc664009fc0 tx=0x7fc664009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664016430 con 0x7fc66c09c410 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: dump: 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.729 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664016590 con 0x7fc66c09c410 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 5 monclient: authenticate success, global_id 44171 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 monclient: _renew_subs 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66c0905a0 con 0x7fc66c09c410 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664016740 con 0x7fc66c09c410 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 10 monclient: dump: 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.731 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 monclient: _renew_subs 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66a7fc700 4 set_mon_vals no callback set 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66c08d720 con 0x7fc66c09c410 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc664012020 con 0x7fc66c09c410 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 0x7fc66c0883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0648a0 con 0x7fc66c087fe0 2024-09-18T10:13:11.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 0x7fc66c0883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient: get_auth_request con 0x7fc66c087fe0 auth_method 0 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc650035a70 0x7fc650037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(98..98 src has 40..98) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fc664048750 con 0x7fc66c09c410 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_done global_id 44174 payload 824 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc66b7fe700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc66c072a80 con 0x7fc66c087fe0 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc663fff700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc650035a70 0x7fc650037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.712+0000 7fc663fff700 10 monclient: get_auth_request con 0x7fc650035a70 auth_method 0 2024-09-18T10:13:11.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc66b7fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 msgr2=0x7fc66c0883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc66b7fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 0x7fc66c0883b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0039d0 tx=0x7fc65c004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c003e70 con 0x7fc66c087fe0 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 10 monclient: dump: 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.734 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc663fff700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc650035a70 0x7fc650037f20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc6540099b0 tx=0x7fc654016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc65c004050 con 0x7fc66c087fe0 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 msgr2=0x7fc66c0883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 0x7fc66c0883b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0039d0 tx=0x7fc65c004cd0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 msgr2=0x7fc66c0883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:11.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.716+0000 7fc6637fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c087fe0 0x7fc66c0883b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6709fd700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 0x7fc66c08bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6709fd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66c0678a0 con 0x7fc66c0888f0 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 0x7fc66c08bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient: get_auth_request con 0x7fc66c0888f0 auth_method 0 2024-09-18T10:13:11.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 10 monclient(hunting): handle_auth_done global_id 44177 payload 824 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 1 -- 172.21.15.142:0/1187873378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc66c08e2f0 con 0x7fc66c0888f0 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 msgr2=0x7fc66c08bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc66b7fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 0x7fc66c08bc70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc65c006ae0 tx=0x7fc65c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c012e90 con 0x7fc66c0888f0 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 10 monclient: dump: 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7fc65c003c60 con 0x7fc66c0888f0 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:11.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 msgr2=0x7fc66c08bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 0x7fc66c08bc70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc65c006ae0 tx=0x7fc65c014000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 msgr2=0x7fc66c08bc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.728+0000 7fc6637fe700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c0888f0 0x7fc66c08bc70 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc65c006ae0 tx=0x7fc65c014000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 52, 2024-09-18T10:13:11.740 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.741 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.742 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:11.743 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.744 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:11.745 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.746 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.747 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:11.748 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:11.749 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.750 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.751 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:11.752 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc650035a70 msgr2=0x7fc650037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc650035a70 0x7fc650037f20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc6540099b0 tx=0x7fc654016000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 10 monclient: shutdown 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 msgr2=0x7fc66c1401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 --2- 172.21.15.142:0/1187873378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc66c09c410 0x7fc66c1401c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc664009fc0 tx=0x7fc664009730 comp rx=0 tx=0).stop 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 reap_dead start 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc66bfff700 1 -- 172.21.15.142:0/1187873378 reap_dead start 2024-09-18T10:13:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 shutdown_connections 2024-09-18T10:13:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 >> 172.21.15.142:0/1187873378 conn(0x7fc66c070ef0 msgr2=0x7fc66c09def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 shutdown_connections 2024-09-18T10:13:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.732+0000 7fc6709fd700 1 -- 172.21.15.142:0/1187873378 wait complete. 2024-09-18T10:13:11.755 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:13:11.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:13:11.755 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:13:11.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f151773c700 10 start_mon_command cmd=[{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f151773c700 10 monclient: _send_command 2 [{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f151773c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd find", "id": 8} v 0) v1 -- 0x7f151006d3c0 con 0x7f151008dde0 2024-09-18T10:13:11.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd find", "id": 8}]=0 v98) v1 ==== 65+0+226 (secure 0 0 0) 0x7f1508025b20 con 0x7f151008dde0 2024-09-18T10:13:11.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f15057fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.796+0000 7f15057fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:11.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: get_monmap_and_config 2024-09-18T10:13:11.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: monmap: 2024-09-18T10:13:11.861 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:11.860794+0000 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:11.860794+0000 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:11.862 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:11.863 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 Processor -- start 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 -- start start 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: init 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c680999e0 0x7f3c680a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-c con 0x7f3c680999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c6809c410 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c68137910 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:11.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c680a20c0 con 0x7f3c68137910 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c680a1c80 con 0x7f3c6809c410 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6809a000 con 0x7f3c680999e0 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c6ca45700 10 monclient(hunting): authenticate will time out at 1080.787081s 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c67fff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f3c68137910 legacy=0x7f3c68139cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c680999e0 0x7f3c680a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c66ffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3c6809c410 0x7f3c680a06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37448/0 (socket says 172.21.15.142:37448) 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c66ffd700 1 -- 172.21.15.142:0/573601153 learned_addr learned my addr 172.21.15.142:0/573601153 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): get_auth_request con 0x7f3c680999e0 auth_method 0 2024-09-18T10:13:11.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.856+0000 7f3c677fe700 10 monclient(hunting): handle_auth_done global_id 15154 payload 274 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 -- 172.21.15.142:0/573601153 >> v1:172.21.15.150:6789/0 conn(0x7f3c6809c410 legacy=0x7f3c680a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 -- 172.21.15.142:0/573601153 >> v1:172.21.15.142:6789/0 conn(0x7f3c68137910 legacy=0x7f3c68139cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 monclient: found mon.noname-c 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 -- 172.21.15.142:0/573601153 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6809a580 con 0x7f3c680999e0 2024-09-18T10:13:11.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 -- 172.21.15.142:0/573601153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c680999e0 msgr2=0x7f3c680a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c677fe700 1 --2- 172.21.15.142:0/573601153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c680999e0 0x7f3c680a0180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3c580098e0 tx=0x7f3c580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a5b2acd65aad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 1 -- 172.21.15.142:0/573601153 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5800ba70 con 0x7f3c680999e0 2024-09-18T10:13:11.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: dump: 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.874 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 5 monclient: authenticate success, global_id 15154 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 1 -- 172.21.15.142:0/573601153 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5800bbd0 con 0x7f3c680999e0 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:11.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 4 set_mon_vals no callback set 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 1 -- 172.21.15.142:0/573601153 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58011420 con 0x7f3c680999e0 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c667fc700 10 monclient: dump: 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.876 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 10 monclient: shutdown 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 1 -- 172.21.15.142:0/573601153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c680999e0 msgr2=0x7f3c680a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.860+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/573601153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c680999e0 0x7f3c680a0180 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3c580098e0 tx=0x7f3c580091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- 172.21.15.142:0/573601153 shutdown_connections 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/573601153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c680999e0 0x7f3c680a0180 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- 172.21.15.142:0/573601153 >> 172.21.15.142:0/573601153 conn(0x7f3c680a5790 msgr2=0x7f3c680a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- 172.21.15.142:0/573601153 shutdown_connections 2024-09-18T10:13:11.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- 172.21.15.142:0/573601153 wait complete. 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient: build_initial_monmap 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient: monmap: 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 Processor -- start 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- start start 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient: init 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c680999e0 0x7f3c6812c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-a con 0x7f3c680999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 0x7f3c6812c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-c con 0x7f3c6809c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c68137910 0x7f3c6812cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): picked mon.noname-b con 0x7f3c68137910 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:11.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680a4c20 con 0x7f3c680999e0 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6812f980 con 0x7f3c68137910 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681376b0 con 0x7f3c6809c410 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c6ca45700 10 monclient(hunting): authenticate will time out at 1080.794508s 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c677fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c680999e0 msgr2=0x7f3c6812c280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c677fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c680999e0 0x7f3c6812c280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:11.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 0x7f3c6812c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 0x7f3c6812c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54116/0 (socket says 172.21.15.142:54116) 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 -- 172.21.15.142:0/431941356 learned_addr learned my addr 172.21.15.142:0/431941356 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): get_auth_request con 0x7f3c6809c410 auth_method 0 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c68137910 0x7f3c6812cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): get_auth_request con 0x7f3c68137910 auth_method 0 2024-09-18T10:13:11.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient(hunting): handle_auth_done global_id 44180 payload 824 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c68137910 msgr2=0x7f3c6812cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c68137910 0x7f3c6812cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c680999e0 msgr2=0x7f3c6812c280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c680999e0 0x7f3c6812c280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 monclient: found mon.noname-c 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:11.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.864+0000 7f3c66ffd700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c58009420 con 0x7f3c6809c410 2024-09-18T10:13:11.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c66ffd700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 msgr2=0x7f3c6812c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c66ffd700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 0x7f3c6812c7c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00c610 tx=0x7f3c5c004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c005d40 con 0x7f3c6809c410 2024-09-18T10:13:11.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: dump: 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:11.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c0070d0 con 0x7f3c6809c410 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 5 monclient: authenticate success, global_id 44180 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 monclient: _renew_subs 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6812f6a0 con 0x7f3c6809c410 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c007490 con 0x7f3c6809c410 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:11.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c4f7fe700 10 monclient: dump: 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c657fa700 4 set_mon_vals no callback set 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 monclient: _renew_subs 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6812e2e0 con 0x7f3c6809c410 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 0x7f3c6806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.868+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c680647f0 con 0x7f3c6812d240 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 0x7f3c6806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient: get_auth_request con 0x7f3c6812d240 auth_method 0 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3c5c01b3b0 con 0x7f3c6809c410 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c54035a70 0x7f3c54037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(98..98 src has 40..98) v4 ==== 6247+0+0 (secure 0 0 0) 0x7f3c5c047ec0 con 0x7f3c6809c410 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 10 monclient(hunting): handle_auth_done global_id 15160 payload 824 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c68072a80 con 0x7f3c6812d240 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c67fff700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c54035a70 0x7f3c54037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c67fff700 10 monclient: get_auth_request con 0x7f3c54035a70 auth_method 0 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 msgr2=0x7f3c6806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c677fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 0x7f3c6806dad0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3c580098e0 tx=0x7f3c58016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58006c50 con 0x7f3c6812d240 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c4f7fe700 10 monclient: dump: 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.892 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.872+0000 7f3c67fff700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c54035a70 0x7f3c54037f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3c68130430 tx=0x7f3c600096a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3c58011e70 con 0x7f3c6812d240 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 msgr2=0x7f3c6806dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 0x7f3c6806dad0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3c580098e0 tx=0x7f3c58016000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 msgr2=0x7f3c6806dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:11.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.876+0000 7f3c4f7fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812d240 0x7f3c6806dad0 secure :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3c580098e0 tx=0x7f3c58016000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c6ca45700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 0x7f3c6806e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c6ca45700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6812faf0 con 0x7f3c6812e7b0 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 0x7f3c6806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient: get_auth_request con 0x7f3c6812e7b0 auth_method 0 2024-09-18T10:13:11.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.884+0000 7f3c677fe700 10 monclient(hunting): handle_auth_done global_id 15163 payload 824 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c677fe700 1 -- 172.21.15.142:0/431941356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3c68067800 con 0x7f3c6812e7b0 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c677fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 msgr2=0x7f3c6806e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c677fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 0x7f3c6806e010 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3c68064360 tx=0x7f3c5801e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5802a070 con 0x7f3c6812e7b0 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:11.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 10 monclient: dump: 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:11.898 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f3c5801c440 con 0x7f3c6812e7b0 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 msgr2=0x7f3c6806e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 0x7f3c6806e010 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3c68064360 tx=0x7f3c5801e040 comp rx=0 tx=0).stop 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 msgr2=0x7f3c6806e010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c4f7fe700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6812e7b0 0x7f3c6806e010 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.899 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 52, 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.900 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.901 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.902 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:11.903 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:11.904 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.905 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:11.906 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:11.906 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.906 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.906 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.906 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.907 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.908 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.910 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.912 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.913 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:11.914 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:11.915 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c54035a70 msgr2=0x7f3c54037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c54035a70 0x7f3c54037f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3c68130430 tx=0x7f3c600096a0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 10 monclient: shutdown 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 msgr2=0x7f3c6812c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c6ca45700 1 --2- 172.21.15.142:0/431941356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c6809c410 0x7f3c6812c7c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c00c610 tx=0x7f3c5c004a80 comp rx=0 tx=0).stop 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.888+0000 7f3c67fff700 1 -- 172.21.15.142:0/431941356 reap_dead start 2024-09-18T10:13:11.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.892+0000 7f3c67fff700 1 -- 172.21.15.142:0/431941356 reap_dead start 2024-09-18T10:13:11.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.892+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 shutdown_connections 2024-09-18T10:13:11.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.892+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 >> 172.21.15.142:0/431941356 conn(0x7f3c680a5790 msgr2=0x7f3c680a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.892+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 shutdown_connections 2024-09-18T10:13:11.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:11.892+0000 7f3c6ca45700 1 -- 172.21.15.142:0/431941356 wait complete. 2024-09-18T10:13:11.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:13:11.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f151773c700 10 start_mon_command cmd=[{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f151773c700 10 monclient: _send_command 3 [{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f151773c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd find", "id": 8} v 0) v1 -- 0x7f1510063ed0 con 0x7f151008dde0 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f15057fa700 1 -- 172.21.15.150:0/4085765861 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd find", "id": 8}]=0 v98) v1 ==== 65+0+226 (secure 0 0 0) 0x7f1508025d00 con 0x7f151008dde0 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f15057fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd find", "id": 8}] 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.892+0000 7f15057fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 msgr2=0x7f14f4037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 0x7f14f4037f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f150c0062a0 tx=0x7f150c0097b0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 10 monclient: shutdown 2024-09-18T10:13:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 msgr2=0x7f1510147bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 0x7f1510147bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1510068eb0 tx=0x7f1508015000 comp rx=0 tx=0).stop 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 shutdown_connections 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f14f4035aa0 0x7f14f4037f50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f151008dde0 0x7f1510147bf0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f151008d4d0 0x7f151008d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 --2- 172.21.15.150:0/4085765861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f14fc0053f0 0x7f151008cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 >> 172.21.15.150:0/4085765861 conn(0x7f1510071830 msgr2=0x7f15100a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 shutdown_connections 2024-09-18T10:13:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:11.896+0000 7f151773c700 1 -- 172.21.15.150:0/4085765861 wait complete. 2024-09-18T10:13:11.922 INFO:tasks.workunit.client.0.smithi150.stdout: "host": "foo", 2024-09-18T10:13:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.8 2024-09-18T10:13:12.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.004+0000 7fca1350c700 10 monclient: get_monmap_and_config 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.004+0000 7fca1350c700 10 monclient: build_initial_monmap 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.004+0000 7fca1350c700 10 monclient: monmap: 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:12.011407+0000 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:12.011407+0000 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:12.013 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 Processor -- start 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 -- start start 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient: init 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca0c06aca0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:12.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 0x7fca0c137850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-a con 0x7fca0c09e530 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-b con 0x7fca0c09e950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c068dd0 con 0x7fca0c06aca0 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c06b8a0 con 0x7fca0c09e530 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c09f830 con 0x7fca0c09e950 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:12.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1350c700 10 monclient(hunting): authenticate will time out at 1074.105533s 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca11d09700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 msgr2=0x7fca0c137850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca11d09700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 0x7fca0c137850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca1250a700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fca0c06aca0 legacy=0x7fca0c09de20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:36414/0 (socket says 172.21.15.150:36414) 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 -- 172.21.15.150:0/1802068553 learned_addr learned my addr 172.21.15.150:0/1802068553 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): get_auth_request con 0x7fca0c09e950 auth_method 0 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:12.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient(hunting): handle_auth_done global_id 44183 payload 274 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 -- 172.21.15.150:0/1802068553 >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 msgr2=0x7fca0c137850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 --2- 172.21.15.150:0/1802068553 >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 0x7fca0c137850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 -- 172.21.15.150:0/1802068553 >> v1:172.21.15.142:6789/0 conn(0x7fca0c06aca0 legacy=0x7fca0c09de20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 monclient: found mon.noname-b 2024-09-18T10:13:12.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:12.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:12.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.008+0000 7fca12d0b700 1 -- 172.21.15.150:0/1802068553 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c069f50 con 0x7fca0c09e950 2024-09-18T10:13:12.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca12d0b700 1 -- 172.21.15.150:0/1802068553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 msgr2=0x7fca0c139da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca12d0b700 1 --2- 172.21.15.150:0/1802068553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca040099b0 tx=0x7fca04009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55be5068ac2a57e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 1 -- 172.21.15.150:0/1802068553 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400bb50 con 0x7fca0c09e950 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: dump: 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.023 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: _finish_auth 0 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 5 monclient: authenticate success, global_id 44183 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 1 -- 172.21.15.150:0/1802068553 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0400bcb0 con 0x7fca0c09e950 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:12.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 1 -- 172.21.15.150:0/1802068553 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011650 con 0x7fca0c09e950 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 4 set_mon_vals no callback set 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca11508700 10 monclient: dump: 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.025 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 10 monclient: shutdown 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- 172.21.15.150:0/1802068553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 msgr2=0x7fca0c139da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 --2- 172.21.15.150:0/1802068553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fca040099b0 tx=0x7fca04009250 comp rx=0 tx=0).stop 2024-09-18T10:13:12.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- 172.21.15.150:0/1802068553 shutdown_connections 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 --2- 172.21.15.150:0/1802068553 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c09e950 0x7fca0c139da0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 --2- 172.21.15.150:0/1802068553 >> v2:172.21.15.142:3300/0 conn(0x7fca0c09e530 0x7fca0c137850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- 172.21.15.150:0/1802068553 >> 172.21.15.150:0/1802068553 conn(0x7fca0c0a3430 msgr2=0x7fca0c0a3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- 172.21.15.150:0/1802068553 shutdown_connections 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- 172.21.15.150:0/1802068553 wait complete. 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 10 monclient: build_initial_monmap 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 10 monclient: monmap: 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:12.027 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 Processor -- start 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 1 -- start start 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.012+0000 7fca1350c700 10 monclient: init 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c06aca0 0x7fca0c1203e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-c con 0x7fca0c06aca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:12.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 0x7fca0c13d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-a con 0x7fca0c09e530 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): picked mon.noname-b con 0x7fca0c09e950 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c06b4d0 con 0x7fca0c09e530 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c06cde0 con 0x7fca0c09e950 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1229b0 con 0x7fca0c06aca0 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient(hunting): authenticate will time out at 1074.112322s 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca11d09700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 msgr2=0x7fca0c13d910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca11d09700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 0x7fca0c13d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:12.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33052/0 (socket says 172.21.15.150:33052) 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 -- 172.21.15.150:0/1828575870 learned_addr learned my addr 172.21.15.150:0/1828575870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): get_auth_request con 0x7fca0c09e950 auth_method 0 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c06aca0 0x7fca0c1203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): get_auth_request con 0x7fca0c06aca0 auth_method 0 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:12.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1250a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient(hunting): handle_auth_done global_id 15166 payload 824 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 -- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c06aca0 msgr2=0x7fca0c1203e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c06aca0 0x7fca0c1203e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 -- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 msgr2=0x7fca0c13d910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 0x7fca0c13d910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 monclient: found mon.noname-b 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040094f0 con 0x7fca0c09e950 2024-09-18T10:13:12.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 -- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 msgr2=0x7fca0c1417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:12.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca12d0b700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fca04009290 tx=0x7fca04016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011e80 con 0x7fca0c09e950 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: dump: 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.034 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 5 monclient: authenticate success, global_id 15166 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0401b890 con 0x7fca0c09e950 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient: _renew_subs 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c122c90 con 0x7fca0c09e950 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0401bc10 con 0x7fca0c09e950 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:12.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca027fc700 10 monclient: dump: 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.016+0000 7fca10d07700 4 set_mon_vals no callback set 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 10 monclient: _renew_subs 2024-09-18T10:13:12.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c13eea0 con 0x7fca0c09e950 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c123050 con 0x7fca0c09e950 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fca040193b0 con 0x7fca0c09e950 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca027fc700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 0x7fc9f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(98..98 src has 40..98) v4 ==== 6247+0+0 (secure 0 0 0) 0x7fca0404ac60 con 0x7fca0c09e950 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1250a700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 0x7fc9f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.020+0000 7fca1250a700 10 monclient: get_auth_request con 0x7fc9f0035a70 auth_method 0 2024-09-18T10:13:12.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.024+0000 7fca1250a700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 0x7fc9f0037f20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc003980 tx=0x7fc9fc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.028+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fca0402cda0 con 0x7fca0c09e950 2024-09-18T10:13:12.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.028+0000 7fca027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:12.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.028+0000 7fca027fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:12.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.196+0000 7fca1350c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.196+0000 7fca1350c700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.196+0000 7fca1350c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.196+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.8"} v 0) v1 -- 0x7fca0c06db10 con 0x7fca0c09e950 2024-09-18T10:13:12.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.384+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.8"}]=0 removed item id 8 name 'osd.8' from crush map v99) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca0402c7a0 con 0x7fca0c09e950 2024-09-18T10:13:12.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.384+0000 7fca027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.384+0000 7fca027fc700 10 monclient: _finish_command 2 = system:0 removed item id 8 name 'osd.8' from crush map 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca1350c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca1350c700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca1350c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.8"} v 0) v1 -- 0x7fca0c04bbf0 con 0x7fca0c09e950 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca027fc700 1 -- 172.21.15.150:0/1828575870 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.8"}]=0 device 'osd.8' does not appear in the crush map v99) v1 ==== 124+0+0 (secure 0 0 0) 0x7fca0402c7a0 con 0x7fca0c09e950 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca027fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.8"}] 2024-09-18T10:13:12.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.496+0000 7fca027fc700 10 monclient: _finish_command 3 = system:0 device 'osd.8' does not appear in the crush map 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 msgr2=0x7fc9f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 0x7fc9f0037f20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc003980 tx=0x7fc9fc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 10 monclient: shutdown 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 msgr2=0x7fca0c1417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fca04009290 tx=0x7fca04016000 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 shutdown_connections 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9f0035a70 0x7fc9f0037f20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0c09e950 0x7fca0c1417b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0c09e530 0x7fca0c13d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 --2- 172.21.15.150:0/1828575870 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca0c06aca0 0x7fca0c1203e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 >> 172.21.15.150:0/1828575870 conn(0x7fca0c0a3430 msgr2=0x7fca0c06ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:12.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 shutdown_connections 2024-09-18T10:13:12.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.500+0000 7fca1350c700 1 -- 172.21.15.150:0/1828575870 wait complete. 2024-09-18T10:13:12.506 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.8' does not appear in the crush map 2024-09-18T10:13:12.514 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.10 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: get_monmap_and_config 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: build_initial_monmap 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: monmap: 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:12.613593+0000 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:12.613593+0000 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:12.614 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:12.615 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 Processor -- start 2024-09-18T10:13:12.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 -- start start 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: init 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 0x7f5b1809d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-a con 0x7f5b180a6090 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1809e460 0x7f5b180a0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-c con 0x7f5b1809e460 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:12.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b180a08b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180a6460 con 0x7f5b180a08b0 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1809fd00 con 0x7f5b180a6090 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b180a6850 con 0x7f5b1809e460 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1cdd9700 10 monclient(hunting): authenticate will time out at 1074.707562s 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1659c700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 msgr2=0x7f5b1809d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b16d9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f5b180a08b0 legacy=0x7f5b18138b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:12.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b1659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 0x7f5b1809d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1809e460 0x7f5b180a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b1809e460 0x7f5b180a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33066/0 (socket says 172.21.15.150:33066) 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 1 -- 172.21.15.150:0/2193835644 learned_addr learned my addr 172.21.15.150:0/2193835644 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 10 monclient(hunting): get_auth_request con 0x7f5b1809e460 auth_method 0 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.608+0000 7f5b15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:12.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_done global_id 15169 payload 274 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 -- 172.21.15.150:0/2193835644 >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 msgr2=0x7f5b1809d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/2193835644 >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 0x7f5b1809d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 -- 172.21.15.150:0/2193835644 >> v1:172.21.15.142:6789/0 conn(0x7f5b180a08b0 legacy=0x7f5b18138b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 monclient: found mon.noname-c 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 -- 172.21.15.150:0/2193835644 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1809d7a0 con 0x7f5b1809e460 2024-09-18T10:13:12.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 -- 172.21.15.150:0/2193835644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1809e460 msgr2=0x7f5b180a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:12.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/2193835644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1809e460 0x7f5b180a0370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b18068d70 tx=0x7f5b00009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6259e270ecdcb325 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 1 -- 172.21.15.150:0/2193835644 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0000ba40 con 0x7f5b1809e460 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: dump: 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.626 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: _finish_auth 0 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 5 monclient: authenticate success, global_id 15169 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 1 -- 172.21.15.150:0/2193835644 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0000bba0 con 0x7f5b1809e460 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:12.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 4 set_mon_vals no callback set 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 1 -- 172.21.15.150:0/2193835644 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00011420 con 0x7f5b1809e460 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1559a700 10 monclient: dump: 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.628 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 10 monclient: shutdown 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/2193835644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1809e460 msgr2=0x7f5b180a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/2193835644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1809e460 0x7f5b180a0370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b18068d70 tx=0x7f5b00009250 comp rx=0 tx=0).stop 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/2193835644 shutdown_connections 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/2193835644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b1809e460 0x7f5b180a0370 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/2193835644 >> v2:172.21.15.142:3300/0 conn(0x7f5b180a6090 0x7f5b1809d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.612+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/2193835644 >> 172.21.15.150:0/2193835644 conn(0x7f5b18071830 msgr2=0x7f5b180a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/2193835644 shutdown_connections 2024-09-18T10:13:12.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/2193835644 wait complete. 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient: build_initial_monmap 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient: monmap: 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 Processor -- start 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- start start 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient: init 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 0x7f5b18141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-a con 0x7f5b1809e460 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-b con 0x7f5b180a08b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 0x7f5b1808d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): picked mon.noname-c con 0x7f5b180a6090 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180a44c0 con 0x7f5b1809e460 2024-09-18T10:13:12.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b18090b10 con 0x7f5b180a08b0 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b18098480 con 0x7f5b180a6090 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1cdd9700 10 monclient(hunting): authenticate will time out at 1074.714683s 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1659c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 msgr2=0x7f5b18141df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b1659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 0x7f5b18141df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33072/0 (socket says 172.21.15.150:33072) 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 -- 172.21.15.150:0/201726640 learned_addr learned my addr 172.21.15.150:0/201726640 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): get_auth_request con 0x7f5b180a08b0 auth_method 0 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:12.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b16d9d700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 0x7f5b1808d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient(hunting): handle_auth_done global_id 15172 payload 824 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 -- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 msgr2=0x7f5b1808d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 0x7f5b1808d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 -- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 msgr2=0x7f5b18141df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 0x7f5b18141df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 monclient: found mon.noname-b 2024-09-18T10:13:12.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b15d9b700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b000094f0 con 0x7f5b180a08b0 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b16d9d700 10 monclient: get_auth_request con 0x7f5b180a6090 auth_method 0 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.616+0000 7f5b16d9d700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 0x7f5b1808d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b15d9b700 1 -- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 msgr2=0x7f5b18142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:12.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5b180a1850 tx=0x7f5b00017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00006b60 con 0x7f5b180a08b0 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b15d9b700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5b180a1850 tx=0x7f5b00017000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f5b0001a9f0 mon_map magic: 0 v1 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: dump: 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.637 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 5 monclient: authenticate success, global_id 15172 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0001a630 con 0x7f5b180a08b0 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _renew_subs 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b18090830 con 0x7f5b180a08b0 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b14d99700 4 set_mon_vals no callback set 2024-09-18T10:13:12.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _renew_subs 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b18091c60 con 0x7f5b180a08b0 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0001a9f0 con 0x7f5b180a08b0 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 10 monclient: dump: 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:12.639 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5b0002c3b0 con 0x7f5b180a08b0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1808ed90 con 0x7f5b180a08b0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b0e7fc700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 0x7f5af8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.620+0000 7f5b1659c700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 0x7f5af8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.624+0000 7f5b1659c700 10 monclient: get_auth_request con 0x7f5af8035a70 auth_method 0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.624+0000 7f5b1659c700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 0x7f5af8037f20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b08006fd0 tx=0x7f5b08006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:12.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:12.628+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(99..99 src has 40..99) v4 ==== 6197+0+0 (secure 0 0 0) 0x7f5b0004a1d0 con 0x7f5b180a08b0 2024-09-18T10:13:13.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.244+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5b000225a0 con 0x7f5b180a08b0 2024-09-18T10:13:13.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.244+0000 7f5b0e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:13.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.244+0000 7f5b0e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:13.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b1cdd9700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b1cdd9700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b1cdd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.10"} v 0) v1 -- 0x7f5b1809da80 con 0x7f5b180a08b0 2024-09-18T10:13:13.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.10"}]=0 device 'osd.10' does not appear in the crush map v100) v1 ==== 126+0+0 (secure 0 0 0) 0x7f5b00028020 con 0x7f5b180a08b0 2024-09-18T10:13:13.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b0e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.424+0000 7f5b0e7fc700 10 monclient: _finish_command 2 = system:0 device 'osd.10' does not appear in the crush map 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b1cdd9700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b1cdd9700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b1cdd9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.10"} v 0) v1 -- 0x7f5b18063ed0 con 0x7f5b180a08b0 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b0e7fc700 1 -- 172.21.15.150:0/201726640 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.10"}]=0 device 'osd.10' does not appear in the crush map v100) v1 ==== 126+0+0 (secure 0 0 0) 0x7f5b00021b00 con 0x7f5b180a08b0 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b0e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.10"}] 2024-09-18T10:13:13.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.520+0000 7f5b0e7fc700 10 monclient: _finish_command 3 = system:0 device 'osd.10' does not appear in the crush map 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 msgr2=0x7f5af8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 0x7f5af8037f20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5b08006fd0 tx=0x7f5b08006e10 comp rx=0 tx=0).stop 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 10 monclient: shutdown 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 msgr2=0x7f5b18142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5b180a1850 tx=0x7f5b00017000 comp rx=0 tx=0).stop 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 shutdown_connections 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5af8035a70 0x7f5af8037f20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b180a6090 0x7f5b1808d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b180a08b0 0x7f5b18142330 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 --2- 172.21.15.150:0/201726640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b1809e460 0x7f5b18141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 >> 172.21.15.150:0/201726640 conn(0x7f5b18071830 msgr2=0x7f5b1809e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:13.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 shutdown_connections 2024-09-18T10:13:13.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.524+0000 7f5b1cdd9700 1 -- 172.21.15.150:0/201726640 wait complete. 2024-09-18T10:13:13.530 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.10' does not appear in the crush map 2024-09-18T10:13:13.537 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm foo 2024-09-18T10:13:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: get_monmap_and_config 2024-09-18T10:13:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: build_initial_monmap 2024-09-18T10:13:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: monmap: 2024-09-18T10:13:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:13.636794+0000 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:13.636794+0000 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 Processor -- start 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 -- start start 2024-09-18T10:13:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: init 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f083409d970 0x7f08340955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-b con 0x7f083409d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 0x7f0834098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-a con 0x7f0834098ac0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08340993d0 0x7f0834135770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-c con 0x7f08340993d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083409b0a0 con 0x7f0834098ac0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083409ab50 con 0x7f08340993d0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083409ae30 con 0x7f083409d970 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f083b94f700 10 monclient(hunting): authenticate will time out at 1075.730039s 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08340993d0 0x7f0834135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0838eea700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 msgr2=0x7f0834098e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0838eea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 0x7f0834098e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08340993d0 0x7f0834135770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33082/0 (socket says 172.21.15.150:33082) 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 -- 172.21.15.150:0/2557618885 learned_addr learned my addr 172.21.15.150:0/2557618885 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): get_auth_request con 0x7f08340993d0 auth_method 0 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f08396eb700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f083409d970 0x7f08340955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f08396eb700 10 monclient(hunting): get_auth_request con 0x7f083409d970 auth_method 0 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f08396eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f08396eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f08396eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient(hunting): handle_auth_done global_id 15175 payload 274 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 -- 172.21.15.150:0/2557618885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f083409d970 msgr2=0x7f08340955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f083409d970 0x7f08340955b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 -- 172.21.15.150:0/2557618885 >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 msgr2=0x7f0834098e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 --2- 172.21.15.150:0/2557618885 >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 0x7f0834098e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient: _finish_hunting 0 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 monclient: found mon.noname-c 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.632+0000 7f0839eec700 1 -- 172.21.15.150:0/2557618885 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083409dd40 con 0x7f08340993d0 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0839eec700 1 -- 172.21.15.150:0/2557618885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08340993d0 msgr2=0x7f0834135770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:13.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0839eec700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08340993d0 0x7f0834135770 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f083409bab0 tx=0x7f082c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ac9aa70f09d5b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 1 -- 172.21.15.150:0/2557618885 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c010070 con 0x7f08340993d0 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: dump: 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:13.647 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 5 monclient: authenticate success, global_id 15175 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 1 -- 172.21.15.150:0/2557618885 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c00de90 con 0x7f08340993d0 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 1 -- 172.21.15.150:0/2557618885 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c009610 con 0x7f08340993d0 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 4 set_mon_vals no callback set 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:13.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f0827fff700 10 monclient: dump: 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: shutdown 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:13.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- 172.21.15.150:0/2557618885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08340993d0 msgr2=0x7f0834135770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08340993d0 0x7f0834135770 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f083409bab0 tx=0x7f082c00ba50 comp rx=0 tx=0).stop 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- 172.21.15.150:0/2557618885 shutdown_connections 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08340993d0 0x7f0834135770 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- 172.21.15.150:0/2557618885 >> v2:172.21.15.142:3300/0 conn(0x7f0834098ac0 0x7f0834098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- 172.21.15.150:0/2557618885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f083409d970 0x7f08340955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- 172.21.15.150:0/2557618885 >> 172.21.15.150:0/2557618885 conn(0x7f08340a15e0 msgr2=0x7f08340a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- 172.21.15.150:0/2557618885 shutdown_connections 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- 172.21.15.150:0/2557618885 wait complete. 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: build_initial_monmap 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: monmap: 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:13.650 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 Processor -- start 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- start start 2024-09-18T10:13:13.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: init 2024-09-18T10:13:13.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:13.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:13.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 0x7f0834141860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-a con 0x7f0834098ac0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 0x7f0834141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-c con 0x7f08340993d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): picked mon.noname-b con 0x7f083409d970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0834098450 con 0x7f0834098ac0 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0834097700 con 0x7f083409d970 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083408f9d0 con 0x7f08340993d0 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:13.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.636+0000 7f083b94f700 10 monclient(hunting): authenticate will time out at 1075.735960s 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08396eb700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 msgr2=0x7f0834141860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08396eb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 0x7f0834141860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33096/0 (socket says 172.21.15.150:33096) 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 -- 172.21.15.150:0/2975791004 learned_addr learned my addr 172.21.15.150:0/2975791004 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): get_auth_request con 0x7f083409d970 auth_method 0 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:13.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 0x7f0834141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 10 monclient(hunting): get_auth_request con 0x7f08340993d0 auth_method 0 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient(hunting): handle_auth_done global_id 15178 payload 824 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 -- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 msgr2=0x7f0834141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 0x7f0834141da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 -- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 msgr2=0x7f0834141860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 0x7f0834141860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient: _finish_hunting 0 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 monclient: found mon.noname-b 2024-09-18T10:13:13.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c00bcf0 con 0x7f083409d970 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0838eea700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 0x7f0834141da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 -- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 msgr2=0x7f08341422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0839eec700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f082c00ba90 tx=0x7f082c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c00ae30 con 0x7f083409d970 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: dump: 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.659 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 5 monclient: authenticate success, global_id 15178 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c01e440 con 0x7f083409d970 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c01e800 con 0x7f083409d970 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 10 monclient: _renew_subs 2024-09-18T10:13:13.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f0826ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0834092830 con 0x7f083409d970 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f08257fa700 10 monclient: dump: 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:13.661 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 10 monclient: _renew_subs 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.640+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0834090640 con 0x7f083409d970 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f083b94f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f083b94f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f083b94f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0834063ed0 con 0x7f083409d970 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f082c003040 con 0x7f083409d970 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f08257fa700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 0x7f0818037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(100..100 src has 40..100) v4 ==== 5994+0+0 (secure 0 0 0) 0x7f082c04a230 con 0x7f083409d970 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f08396eb700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 0x7f0818037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:13.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.644+0000 7f08396eb700 10 monclient: get_auth_request con 0x7f0818035aa0 auth_method 0 2024-09-18T10:13:13.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.648+0000 7f08396eb700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 0x7f0818037f50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0830006fd0 tx=0x7f0830009530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:13.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.652+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f082c025dd0 con 0x7f083409d970 2024-09-18T10:13:13.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.652+0000 7f08257fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:13.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.652+0000 7f08257fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:13.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.820+0000 7f083b94f700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:13.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.820+0000 7f083b94f700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:13.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.820+0000 7f083b94f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:13.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:13.820+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f0834091930 con 0x7f083409d970 2024-09-18T10:13:14.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.404+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 removed item id -10 name 'foo' from crush map v101) v1 ==== 120+0+0 (secure 0 0 0) 0x7f082c022090 con 0x7f083409d970 2024-09-18T10:13:14.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.404+0000 7f08257fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:14.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.404+0000 7f08257fa700 10 monclient: _finish_command 2 = system:0 removed item id -10 name 'foo' from crush map 2024-09-18T10:13:14.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:14.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:14.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "foo"} v 0) v1 -- 0x7f08340008d0 con 0x7f083409d970 2024-09-18T10:13:14.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f08257fa700 1 -- 172.21.15.150:0/2975791004 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "foo"}]=0 device 'foo' does not appear in the crush map v101) v1 ==== 120+0+0 (secure 0 0 0) 0x7f082c02c020 con 0x7f083409d970 2024-09-18T10:13:14.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f08257fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "foo"}] 2024-09-18T10:13:14.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f08257fa700 10 monclient: _finish_command 3 = system:0 device 'foo' does not appear in the crush map 2024-09-18T10:13:14.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 msgr2=0x7f0818037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:14.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 0x7f0818037f50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0830006fd0 tx=0x7f0830009530 comp rx=0 tx=0).stop 2024-09-18T10:13:14.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 10 monclient: shutdown 2024-09-18T10:13:14.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:14.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 msgr2=0x7f08341422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f082c00ba90 tx=0x7f082c01a000 comp rx=0 tx=0).stop 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 shutdown_connections 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0818035aa0 0x7f0818037f50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f083409d970 0x7f08341422e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08340993d0 0x7f0834141da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.512+0000 7f083b94f700 1 --2- 172.21.15.150:0/2975791004 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0834098ac0 0x7f0834141860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.516+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 >> 172.21.15.150:0/2975791004 conn(0x7f08340a15e0 msgr2=0x7f083409a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.516+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 shutdown_connections 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.516+0000 7f083b94f700 1 -- 172.21.15.150:0/2975791004 wait complete. 2024-09-18T10:13:14.521 INFO:tasks.workunit.client.0.smithi150.stderr:device 'foo' does not appear in the crush map 2024-09-18T10:13:14.530 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph osd create 2024-09-18T10:13:14.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.616+0000 7fb59b83e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:14.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.616+0000 7fb59b83e700 10 monclient: build_initial_monmap 2024-09-18T10:13:14.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.616+0000 7fb59b83e700 10 monclient: monmap: 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:14.622494+0000 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:14.622494+0000 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.616+0000 7fb59b83e700 1 Processor -- start 2024-09-18T10:13:14.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- start start 2024-09-18T10:13:14.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: init 2024-09-18T10:13:14.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:14.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:13:14.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb594096a70 0x7fb5940999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-c con 0x7fb594096a70 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:14.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 0x7fb594097260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-a con 0x7fb594096e40 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5940977a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb594098880 con 0x7fb5940977a0 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59409e120 con 0x7fb594096e40 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb594096a70 0x7fb5940999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb594098eb0 con 0x7fb594096a70 2024-09-18T10:13:14.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:14.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient(hunting): authenticate will time out at 1076.717659s 2024-09-18T10:13:14.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb594096a70 0x7fb5940999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33106/0 (socket says 172.21.15.150:33106) 2024-09-18T10:13:14.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 -- 172.21.15.150:0/734917069 learned_addr learned my addr 172.21.15.150:0/734917069 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:14.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): get_auth_request con 0x7fb594096a70 auth_method 0 2024-09-18T10:13:14.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb599ddb700 1 -- 172.21.15.150:0/734917069 >> v1:172.21.15.142:6789/0 conn(0x7fb5940977a0 legacy=0x7fb594135530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb598dd9700 1 -- 172.21.15.150:0/734917069 >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 msgr2=0x7fb594097260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb598dd9700 1 --2- 172.21.15.150:0/734917069 >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 0x7fb594097260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 1 -- 172.21.15.150:0/734917069 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:14.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient(hunting): handle_auth_done global_id 15181 payload 274 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 -- 172.21.15.150:0/734917069 >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 msgr2=0x7fb594097260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 --2- 172.21.15.150:0/734917069 >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 0x7fb594097260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 -- 172.21.15.150:0/734917069 >> v1:172.21.15.142:6789/0 conn(0x7fb5940977a0 legacy=0x7fb594135530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient: _finish_hunting 0 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 monclient: found mon.noname-c 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 -- 172.21.15.150:0/734917069 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940a2c10 con 0x7fb594096a70 2024-09-18T10:13:14.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 -- 172.21.15.150:0/734917069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb594096a70 msgr2=0x7fb5940999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb5995da700 1 --2- 172.21.15.150:0/734917069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb594096a70 0x7fb5940999c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb594064e50 tx=0x7fb590009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386b9ca860463f61 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 1 -- 172.21.15.150:0/734917069 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59000ba70 con 0x7fb594096a70 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: dump: 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.639 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: _finish_auth 0 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 5 monclient: authenticate success, global_id 15181 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 1 -- 172.21.15.150:0/734917069 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59000bbd0 con 0x7fb594096a70 2024-09-18T10:13:14.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 4 set_mon_vals no callback set 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 1 -- 172.21.15.150:0/734917069 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb590011420 con 0x7fb594096a70 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb58bfff700 10 monclient: dump: 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.641 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: shutdown 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- 172.21.15.150:0/734917069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb594096a70 msgr2=0x7fb5940999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 --2- 172.21.15.150:0/734917069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb594096a70 0x7fb5940999c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb594064e50 tx=0x7fb590009250 comp rx=0 tx=0).stop 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- 172.21.15.150:0/734917069 shutdown_connections 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 --2- 172.21.15.150:0/734917069 >> v2:172.21.15.142:3300/0 conn(0x7fb594096e40 0x7fb594097260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 --2- 172.21.15.150:0/734917069 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb594096a70 0x7fb5940999c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- 172.21.15.150:0/734917069 >> 172.21.15.150:0/734917069 conn(0x7fb594068460 msgr2=0x7fb5940a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- 172.21.15.150:0/734917069 shutdown_connections 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- 172.21.15.150:0/734917069 wait complete. 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: build_initial_monmap 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: monmap: 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:14.643 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 Processor -- start 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 1 -- start start 2024-09-18T10:13:14.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.624+0000 7fb59b83e700 10 monclient: init 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 0x7fb59409c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-a con 0x7fb594096a70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb594096e40 0x7fb59411ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-c con 0x7fb594096e40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): picked mon.noname-b con 0x7fb5940977a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59411c330 con 0x7fb594096a70 2024-09-18T10:13:14.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594097cb0 con 0x7fb5940977a0 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5940a0680 con 0x7fb594096e40 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient(hunting): authenticate will time out at 1076.724529s 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5995da700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 msgr2=0x7fb59409c740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5995da700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 0x7fb59409c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33108/0 (socket says 172.21.15.150:33108) 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 -- 172.21.15.150:0/2046056165 learned_addr learned my addr 172.21.15.150:0/2046056165 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:14.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): get_auth_request con 0x7fb5940977a0 auth_method 0 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb594096e40 0x7fb59411ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): get_auth_request con 0x7fb594096e40 auth_method 0 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:14.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb598dd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient(hunting): handle_auth_done global_id 15184 payload 824 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 -- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb594096e40 msgr2=0x7fb59411ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb594096e40 0x7fb59411ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 -- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 msgr2=0x7fb59409c740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 0x7fb59409c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient: _finish_hunting 0 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 monclient: found mon.noname-b 2024-09-18T10:13:14.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5900094f0 con 0x7fb5940977a0 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 -- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 msgr2=0x7fb59411b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb599ddb700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00bb10 tx=0x7fb58c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00cb10 con 0x7fb5940977a0 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: dump: 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:14.649 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c00cc90 con 0x7fb5940977a0 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 5 monclient: authenticate success, global_id 15184 2024-09-18T10:13:14.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c00ce40 con 0x7fb5940977a0 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _renew_subs 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59411d2b0 con 0x7fb5940977a0 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb5897fa700 10 monclient: dump: 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:14.651 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb58b7fe700 4 set_mon_vals no callback set 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _renew_subs 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.628+0000 7fb59b83e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59411c560 con 0x7fb5940977a0 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb59b83e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:14.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb59b83e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb59b83e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594063ed0 con 0x7fb5940977a0 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb58c012020 con 0x7fb5940977a0 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb5897fa700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 0x7fb578037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(101..101 src has 40..101) v4 ==== 5841+0+0 (secure 0 0 0) 0x7fb58c048650 con 0x7fb5940977a0 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb5995da700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 0x7fb578037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:14.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.632+0000 7fb5995da700 10 monclient: get_auth_request con 0x7fb578035a70 auth_method 0 2024-09-18T10:13:14.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.636+0000 7fb5995da700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 0x7fb578037f20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb594064e50 tx=0x7fb590016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:14.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:14.640+0000 7fb599ddb700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00bb10 tx=0x7fb58c0097d0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fb58c0153e0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:13:15.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.472+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb58c0153e0 con 0x7fb5940977a0 2024-09-18T10:13:15.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.472+0000 7fb5897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:15.474 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.472+0000 7fb5897fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:15.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.648+0000 7fb59b83e700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:15.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.648+0000 7fb59b83e700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T10:13:15.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.648+0000 7fb59b83e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:15.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.648+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fb59406fc90 con 0x7fb5940977a0 2024-09-18T10:13:15.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.900+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v102) v1 ==== 58+0+3 (secure 0 0 0) 0x7fb58c014340 con 0x7fb5940977a0 2024-09-18T10:13:15.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.900+0000 7fb5897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T10:13:15.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:15.900+0000 7fb5897fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:16.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.016+0000 7fb59b83e700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:16.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.016+0000 7fb59b83e700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:13:16.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.016+0000 7fb59b83e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:16.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.016+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fb59409d300 con 0x7fb5940977a0 2024-09-18T10:13:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.899+0000 7fb5897fa700 1 -- 172.21.15.150:0/2046056165 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v103) v1 ==== 58+0+3 (secure 0 0 0) 0x7fb58c01d300 con 0x7fb5940977a0 2024-09-18T10:13:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.899+0000 7fb5897fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T10:13:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.899+0000 7fb5897fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 msgr2=0x7fb578037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 0x7fb578037f20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb594064e50 tx=0x7fb590016000 comp rx=0 tx=0).stop 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 10 monclient: shutdown 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 msgr2=0x7fb59411b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb58c00bb10 tx=0x7fb58c0097d0 comp rx=0 tx=0).stop 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 shutdown_connections 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb578035a70 0x7fb578037f20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5940977a0 0x7fb59411b1c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb594096e40 0x7fb59411ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 --2- 172.21.15.150:0/2046056165 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb594096a70 0x7fb59409c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 >> 172.21.15.150:0/2046056165 conn(0x7fb594068460 msgr2=0x7fb5940a0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 shutdown_connections 2024-09-18T10:13:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:16.903+0000 7fb59b83e700 1 -- 172.21.15.150:0/2046056165 wait complete. 2024-09-18T10:13:16.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ o3=12 2024-09-18T10:13:16.922 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 12 123 root=default 2024-09-18T10:13:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: get_monmap_and_config 2024-09-18T10:13:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: build_initial_monmap 2024-09-18T10:13:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: monmap: 2024-09-18T10:13:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:17.033144+0000 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:17.033144+0000 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:17.034 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 Processor -- start 2024-09-18T10:13:17.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 -- start start 2024-09-18T10:13:17.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: init 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 0x7f79240a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-a con 0x7f79240a2810 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79240a1550 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79240a1950 0x7f7924132e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-c con 0x7f79240a1950 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79240697a0 con 0x7f79240a1550 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792406a240 con 0x7f79240a2810 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792409d170 con 0x7f79240a1950 2024-09-18T10:13:17.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7929d9b700 10 monclient(hunting): authenticate will time out at 1079.127391s 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7923fff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f79240a1550 legacy=0x7f7924132760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79240a1950 0x7f7924132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7928d99700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 msgr2=0x7f79240a1010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79240a1950 0x7f7924132e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33114/0 (socket says 172.21.15.150:33114) 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 1 -- 172.21.15.150:0/3589559953 learned_addr learned my addr 172.21.15.150:0/3589559953 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): get_auth_request con 0x7f79240a1950 auth_method 0 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f7928d99700 1 --2- 172.21.15.150:0/3589559953 >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 0x7f79240a1010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:17.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.031+0000 7f792959a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 10 monclient(hunting): handle_auth_done global_id 15187 payload 274 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 -- 172.21.15.150:0/3589559953 >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 msgr2=0x7f79240a1010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 --2- 172.21.15.150:0/3589559953 >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 0x7f79240a1010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 -- 172.21.15.150:0/3589559953 >> v1:172.21.15.142:6789/0 conn(0x7f79240a1550 legacy=0x7f7924132760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 10 monclient: _finish_hunting 0 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 monclient: found mon.noname-c 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 -- 172.21.15.150:0/3589559953 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f792409cec0 con 0x7f79240a1950 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 -- 172.21.15.150:0/3589559953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1950 msgr2=0x7f7924132e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f792959a700 1 --2- 172.21.15.150:0/3589559953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1950 0x7f7924132e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7924064ca0 tx=0x7f791c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d65c62d93d7404b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 1 -- 172.21.15.150:0/3589559953 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00e070 con 0x7f79240a1950 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: dump: 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:17.044 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 1 -- 172.21.15.150:0/3589559953 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c00bc60 con 0x7f79240a1950 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 5 monclient: authenticate success, global_id 15187 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 1 -- 172.21.15.150:0/3589559953 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c012430 con 0x7f79240a1950 2024-09-18T10:13:17.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 4 set_mon_vals no callback set 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f79237fe700 10 monclient: dump: 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 10 monclient: shutdown 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 1 -- 172.21.15.150:0/3589559953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1950 msgr2=0x7f7924132e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.035+0000 7f7929d9b700 1 --2- 172.21.15.150:0/3589559953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1950 0x7f7924132e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7924064ca0 tx=0x7f791c009840 comp rx=0 tx=0).stop 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- 172.21.15.150:0/3589559953 shutdown_connections 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 --2- 172.21.15.150:0/3589559953 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1950 0x7f7924132e70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 --2- 172.21.15.150:0/3589559953 >> v2:172.21.15.142:3300/0 conn(0x7f79240a2810 0x7f79240a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- 172.21.15.150:0/3589559953 >> 172.21.15.150:0/3589559953 conn(0x7f79240a60b0 msgr2=0x7f79240a2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- 172.21.15.150:0/3589559953 shutdown_connections 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- 172.21.15.150:0/3589559953 wait complete. 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient: build_initial_monmap 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient: monmap: 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:17.047 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 Processor -- start 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- start start 2024-09-18T10:13:17.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient: init 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-b con 0x7f79240a1550 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79240a1950 0x7f792412f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-c con 0x7f79240a1950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 0x7f792412f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): picked mon.noname-a con 0x7f79240a2810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792406efa0 con 0x7f79240a2810 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79241325f0 con 0x7f79240a1550 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:17.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7924131960 con 0x7f79240a1950 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7929d9b700 10 monclient(hunting): authenticate will time out at 1079.133627s 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f792959a700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 msgr2=0x7f792412f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f792959a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 0x7f792412f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33118/0 (socket says 172.21.15.150:33118) 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 -- 172.21.15.150:0/1723424441 learned_addr learned my addr 172.21.15.150:0/1723424441 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): get_auth_request con 0x7f79240a1550 auth_method 0 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:17.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79240a1950 0x7f792412f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): get_auth_request con 0x7f79240a1950 auth_method 0 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7923fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient(hunting): handle_auth_done global_id 15190 payload 824 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 -- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79240a1950 msgr2=0x7f792412f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79240a1950 0x7f792412f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 -- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 msgr2=0x7f792412f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:17.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 0x7f792412f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:17.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient: _finish_hunting 0 2024-09-18T10:13:17.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 monclient: found mon.noname-b 2024-09-18T10:13:17.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:17.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.039+0000 7f7928d99700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c009ae0 con 0x7f79240a1550 2024-09-18T10:13:17.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7928d99700 1 -- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 msgr2=0x7f7924142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:17.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7928d99700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f791800b810 tx=0x7f7918009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:17.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918003e20 con 0x7f79240a1550 2024-09-18T10:13:17.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: dump: 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 5 monclient: authenticate success, global_id 15190 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _renew_subs 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7924132820 con 0x7f79240a1550 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _renew_subs 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7924130d60 con 0x7f79240a1550 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7918006150 con 0x7f79240a1550 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918006510 con 0x7f79240a1550 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:17.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 10 monclient: dump: 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f79227fc700 4 set_mon_vals no callback set 2024-09-18T10:13:17.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79040050b0 con 0x7f79240a1550 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f791801b3b0 con 0x7f79240a1550 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 0x7f7908037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.043+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(103..103 src has 40..103) v4 ==== 6071+0+0 (secure 0 0 0) 0x7f7918047e30 con 0x7f79240a1550 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.047+0000 7f7923fff700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 0x7f7908037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.047+0000 7f7923fff700 10 monclient: get_auth_request con 0x7f7908035a70 auth_method 0 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.047+0000 7f7923fff700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 0x7f7908037f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f79241309f0 tx=0x7f79100091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:17.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.055+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f791800e680 con 0x7f79240a1550 2024-09-18T10:13:17.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.055+0000 7f7920ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:17.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.055+0000 7f7920ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:17.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.223+0000 7f7929d9b700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:17.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.223+0000 7f7929d9b700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:17.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.223+0000 7f7929d9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:17.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.223+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f7904005360 con 0x7f79240a1550 2024-09-18T10:13:17.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.903+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}]=0 add item id 12 name 'osd.12' weight 123 at location {root=default} to crush map v104) v1 ==== 193+0+0 (secure 0 0 0) 0x7f7918017070 con 0x7f79240a1550 2024-09-18T10:13:17.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.903+0000 7f7920ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:17.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:17.903+0000 7f7920ff9700 10 monclient: _finish_command 2 = system:0 add item id 12 name 'osd.12' weight 123 at location {root=default} to crush map 2024-09-18T10:13:18.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7929d9b700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:18.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7929d9b700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:18.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7929d9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]} v 0) v1 -- 0x7f7904005aa0 con 0x7f79240a1550 2024-09-18T10:13:18.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7920ff9700 1 -- 172.21.15.150:0/1723424441 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}]=0 set item id 12 name 'osd.12' weight 123 at location {root=default}: no change v104) v1 ==== 191+0+0 (secure 0 0 0) 0x7f791800d290 con 0x7f79240a1550 2024-09-18T10:13:18.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7920ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 12, "weight": 123.0, "args": ["root=default"]}] 2024-09-18T10:13:18.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.011+0000 7f7920ff9700 10 monclient: _finish_command 3 = system:0 set item id 12 name 'osd.12' weight 123 at location {root=default}: no change 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 msgr2=0x7f7908037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 0x7f7908037f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f79241309f0 tx=0x7f79100091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 10 monclient: shutdown 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 msgr2=0x7f7924142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f791800b810 tx=0x7f7918009220 comp rx=0 tx=0).stop 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 shutdown_connections 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7908035a70 0x7f7908037f20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79240a2810 0x7f792412f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79240a1950 0x7f792412f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 --2- 172.21.15.150:0/1723424441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79240a1550 0x7f7924142010 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 >> 172.21.15.150:0/1723424441 conn(0x7f79240a60b0 msgr2=0x7f792406d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:18.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 shutdown_connections 2024-09-18T10:13:18.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.015+0000 7f7929d9b700 1 -- 172.21.15.150:0/1723424441 wait complete. 2024-09-18T10:13:18.019 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 12 name 'osd.12' weight 123 at location {root=default}: no change 2024-09-18T10:13:18.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:18.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.12 2024-09-18T10:13:18.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 123 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.119+0000 7f82b27ec700 10 monclient: get_monmap_and_config 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.119+0000 7f82b27ec700 10 monclient: build_initial_monmap 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.119+0000 7f82b27ec700 10 monclient: monmap: 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:18.123561+0000 2024-09-18T10:13:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:18.123561+0000 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:18.126 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 Processor -- start 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 -- start start 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient: init 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82ac06ea90 0x7f82ac09c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-c con 0x7f82ac06ea90 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82ac06f1d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 0x7f82ac139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-a con 0x7f82ac137730 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac09d430 con 0x7f82ac06f1d0 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac098e60 con 0x7f82ac137730 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac098b80 con 0x7f82ac06ea90 2024-09-18T10:13:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b27ec700 10 monclient(hunting): authenticate will time out at 1080.216992s 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b0d89700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 msgr2=0x7f82ac139b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82ab7fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f82ac06f1d0 legacy=0x7f82ac09c750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82b0d89700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 0x7f82ac139b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82ac06ea90 0x7f82ac09c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f82ac06ea90 0x7f82ac09c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33132/0 (socket says 172.21.15.150:33132) 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 -- 172.21.15.150:0/1243883764 learned_addr learned my addr 172.21.15.150:0/1243883764 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): get_auth_request con 0x7f82ac06ea90 auth_method 0 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient(hunting): handle_auth_done global_id 15193 payload 274 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 -- 172.21.15.150:0/1243883764 >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 msgr2=0x7f82ac139b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 --2- 172.21.15.150:0/1243883764 >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 0x7f82ac139b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 -- 172.21.15.150:0/1243883764 >> v1:172.21.15.142:6789/0 conn(0x7f82ac06f1d0 legacy=0x7f82ac09c750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 monclient: found mon.noname-c 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.123+0000 7f82abfff700 1 -- 172.21.15.150:0/1243883764 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac099b70 con 0x7f82ac06ea90 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82abfff700 1 -- 172.21.15.150:0/1243883764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac06ea90 msgr2=0x7f82ac09c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82abfff700 1 --2- 172.21.15.150:0/1243883764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac06ea90 0x7f82ac09c210 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f82ac09f0b0 tx=0x7f82a0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8009b6edfb6cb807 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 1 -- 172.21.15.150:0/1243883764 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a000ba70 con 0x7f82ac06ea90 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: dump: 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:18.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: _finish_auth 0 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 5 monclient: authenticate success, global_id 15193 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 1 -- 172.21.15.150:0/1243883764 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a000bbd0 con 0x7f82ac06ea90 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 4 set_mon_vals no callback set 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 1 -- 172.21.15.150:0/1243883764 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0011420 con 0x7f82ac06ea90 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:18.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82aaffd700 10 monclient: dump: 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: shutdown 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- 172.21.15.150:0/1243883764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac06ea90 msgr2=0x7f82ac09c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1243883764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac06ea90 0x7f82ac09c210 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f82ac09f0b0 tx=0x7f82a0009250 comp rx=0 tx=0).stop 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- 172.21.15.150:0/1243883764 shutdown_connections 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1243883764 >> v2:172.21.15.142:3300/0 conn(0x7f82ac137730 0x7f82ac139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1243883764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac06ea90 0x7f82ac09c210 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- 172.21.15.150:0/1243883764 >> 172.21.15.150:0/1243883764 conn(0x7f82ac0a1710 msgr2=0x7f82ac0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- 172.21.15.150:0/1243883764 shutdown_connections 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- 172.21.15.150:0/1243883764 wait complete. 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: build_initial_monmap 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: monmap: 2024-09-18T10:13:18.137 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:18.138 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 Processor -- start 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- start start 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: init 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82ac06ea90 0x7f82ac136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-c con 0x7f82ac06ea90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 0x7f82ac0940b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-a con 0x7f82ac06f1d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): picked mon.noname-b con 0x7f82ac137730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac06f630 con 0x7f82ac06f1d0 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac09ffc0 con 0x7f82ac137730 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac090fe0 con 0x7f82ac06ea90 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b27ec700 10 monclient(hunting): authenticate will time out at 1080.223608s 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82ab7fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 msgr2=0x7f82ac0940b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82ab7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 0x7f82ac0940b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33140/0 (socket says 172.21.15.150:33140) 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 1 -- 172.21.15.150:0/1534027997 learned_addr learned my addr 172.21.15.150:0/1534027997 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:18.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 10 monclient(hunting): get_auth_request con 0x7f82ac137730 auth_method 0 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.127+0000 7f82b0d89700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82ac06ea90 0x7f82ac136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): get_auth_request con 0x7f82ac06ea90 auth_method 0 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:18.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient(hunting): handle_auth_done global_id 15196 payload 824 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 -- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82ac06ea90 msgr2=0x7f82ac136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82ac06ea90 0x7f82ac136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 -- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 msgr2=0x7f82ac0940b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 0x7f82ac0940b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient: _finish_hunting 0 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 monclient: found mon.noname-b 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00094f0 con 0x7f82ac137730 2024-09-18T10:13:18.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 -- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 msgr2=0x7f82ac08dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f82a400bb10 tx=0x7f82a40097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b0d89700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f82a400bb10 tx=0x7f82a40097d0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f82a400cbd0 config(0 keys) v1 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a400ca50 con 0x7f82ac137730 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 10 monclient: dump: 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:18.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 5 monclient: authenticate success, global_id 15196 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 10 monclient: _renew_subs 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a400cbd0 con 0x7f82ac137730 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac136ef0 con 0x7f82ac137730 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f8293fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 10 monclient: _renew_subs 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac0912d0 con 0x7f82ac137730 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.131+0000 7f82a9ffb700 4 set_mon_vals no callback set 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a400cd80 con 0x7f82ac137730 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 10 monclient: dump: 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:18.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f82a4012020 con 0x7f82ac137730 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82b27ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82b27ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82b27ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac063ed0 con 0x7f82ac137730 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f8293fff700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 0x7f8298037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82abfff700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 0x7f8298037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.135+0000 7f82abfff700 10 monclient: get_auth_request con 0x7f8298035a70 auth_method 0 2024-09-18T10:13:18.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.139+0000 7f82abfff700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 0x7f8298037f20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f82ac09f0b0 tx=0x7f82a0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:18.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.143+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(104..104 src has 40..104) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f82a4048610 con 0x7f82ac137730 2024-09-18T10:13:18.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.755+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f82a401d300 con 0x7f82ac137730 2024-09-18T10:13:18.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.755+0000 7f8293fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:18.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.755+0000 7f8293fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:18.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f82b27ec700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:18.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f82b27ec700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:18.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f82b27ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:18.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f82ac06fce0 con 0x7f82ac137730 2024-09-18T10:13:18.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v104) v1 ==== 56+0+1242 (secure 0 0 0) 0x7f82ac06fce0 con 0x7f82ac137730 2024-09-18T10:13:18.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f8293fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:18.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:18.923+0000 7f8293fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:19.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f82b27ec700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:19.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f82b27ec700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:19.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f82b27ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f82ac06fce0 con 0x7f82ac137730 2024-09-18T10:13:19.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f8293fff700 1 -- 172.21.15.150:0/1534027997 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v104) v1 ==== 56+0+1242 (secure 0 0 0) 0x7f82ac06fce0 con 0x7f82ac137730 2024-09-18T10:13:19.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f8293fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:19.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.015+0000 7f8293fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 msgr2=0x7f8298037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 0x7f8298037f20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f82ac09f0b0 tx=0x7f82a0016000 comp rx=0 tx=0).stop 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 10 monclient: shutdown 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 msgr2=0x7f82ac08dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f82a400bb10 tx=0x7f82a40097d0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 shutdown_connections 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8298035a70 0x7f8298037f20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82ac137730 0x7f82ac08dcc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82ac06f1d0 0x7f82ac0940b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 --2- 172.21.15.150:0/1534027997 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82ac06ea90 0x7f82ac136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 >> 172.21.15.150:0/1534027997 conn(0x7f82ac0a1710 msgr2=0x7f82ac09de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:19.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 shutdown_connections 2024-09-18T10:13:19.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.019+0000 7f82b27ec700 1 -- 172.21.15.150:0/1534027997 wait complete. 2024-09-18T10:13:19.029 INFO:tasks.workunit.client.0.smithi150.stdout: 12 123.00000 osd.12 down 1.00000 1.00000 2024-09-18T10:13:19.030 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush reweight osd.12 113 2024-09-18T10:13:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: get_monmap_and_config 2024-09-18T10:13:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: build_initial_monmap 2024-09-18T10:13:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: monmap: 2024-09-18T10:13:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:19.133252+0000 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:19.133252+0000 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:19.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 Processor -- start 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 -- start start 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: init 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe37009b320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3700992f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe37009b6f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:19.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe37009a560 con 0x7fe3700992f0 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe370099090 con 0x7fe37009b6f0 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37009dd80 con 0x7fe37009b320 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe376c3c700 10 monclient(hunting): authenticate will time out at 1081.226757s 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe36ffff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fe3700992f0 legacy=0x7fe37012ece0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3751d9700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe37009b6f0 0x7fe370135ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52660/0 (socket says 172.21.15.150:52660) 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3751d9700 1 -- 172.21.15.150:0/1007796282 learned_addr learned my addr 172.21.15.150:0/1007796282 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 1 --2- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): get_auth_request con 0x7fe37009b320 auth_method 0 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:19.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient(hunting): handle_auth_done global_id 44204 payload 274 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 1 -- 172.21.15.150:0/1007796282 >> v1:172.21.15.150:6789/0 conn(0x7fe37009b6f0 legacy=0x7fe370135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 1 -- 172.21.15.150:0/1007796282 >> v1:172.21.15.142:6789/0 conn(0x7fe3700992f0 legacy=0x7fe37012ece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient: _finish_hunting 0 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 1 monclient: found mon.noname-b 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:19.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.131+0000 7fe3749d8700 1 -- 172.21.15.150:0/1007796282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37012f3f0 con 0x7fe37009b320 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe3749d8700 1 -- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 msgr2=0x7fe37012e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe3749d8700 1 --2- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe364009910 tx=0x7fe3640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c467d97fea6dda8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 1 -- 172.21.15.150:0/1007796282 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400ba70 con 0x7fe37009b320 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: dump: 2024-09-18T10:13:19.142 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:19.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 5 monclient: authenticate success, global_id 44204 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 1 -- 172.21.15.150:0/1007796282 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36400bbd0 con 0x7fe37009b320 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 1 -- 172.21.15.150:0/1007796282 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364011420 con 0x7fe37009b320 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe36f7fe700 10 monclient: dump: 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:19.144 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 4 set_mon_vals no callback set 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 10 monclient: shutdown 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 1 -- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 msgr2=0x7fe37012e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 1 --2- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe364009910 tx=0x7fe3640091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 1 -- 172.21.15.150:0/1007796282 shutdown_connections 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 1 --2- 172.21.15.150:0/1007796282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e7a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.135+0000 7fe376c3c700 1 -- 172.21.15.150:0/1007796282 >> 172.21.15.150:0/1007796282 conn(0x7fe3700a15e0 msgr2=0x7fe3700a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- 172.21.15.150:0/1007796282 shutdown_connections 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- 172.21.15.150:0/1007796282 wait complete. 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient: build_initial_monmap 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient: monmap: 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 Processor -- start 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- start start 2024-09-18T10:13:19.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient: init 2024-09-18T10:13:19.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe3700992f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-c con 0x7fe37009b320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 0x7fe37013f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): picked mon.noname-a con 0x7fe37009b6f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe370129e30 con 0x7fe37009b6f0 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:19.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3701298f0 con 0x7fe3700992f0 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe370128b30 con 0x7fe37009b320 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe376c3c700 10 monclient(hunting): authenticate will time out at 1081.234614s 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43048/0 (socket says 172.21.15.150:43048) 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 -- 172.21.15.150:0/71912212 learned_addr learned my addr 172.21.15.150:0/71912212 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): get_auth_request con 0x7fe3700992f0 auth_method 0 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:19.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3751d9700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 msgr2=0x7fe37013f8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3751d9700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 0x7fe37013f8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 10 monclient(hunting): get_auth_request con 0x7fe37009b320 auth_method 0 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:19.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient(hunting): handle_auth_done global_id 15199 payload 824 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 msgr2=0x7fe37012e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 msgr2=0x7fe37013f8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 0x7fe37013f8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient: _finish_hunting 0 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 monclient: found mon.noname-b 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe3749d8700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe364009420 con 0x7fe3700992f0 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.139+0000 7fe36ffff700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe3749d8700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 msgr2=0x7fe37013c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe3749d8700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe37009e000 tx=0x7fe364016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:19.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36401b680 con 0x7fe3700992f0 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: dump: 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:19.155 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 5 monclient: authenticate success, global_id 15199 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36401b7e0 con 0x7fe3700992f0 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36401bb60 con 0x7fe3700992f0 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 10 monclient: _renew_subs 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe37012af00 con 0x7fe3700992f0 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36e7fc700 4 set_mon_vals no callback set 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:19.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe36cff9700 10 monclient: dump: 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 10 monclient: _renew_subs 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.143+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe370129220 con 0x7fe3700992f0 2024-09-18T10:13:19.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe3640193b0 con 0x7fe3700992f0 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe376c3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe376c3c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe376c3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe370129410 con 0x7fe3700992f0 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36cff9700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 0x7fe35c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(104..104 src has 40..104) v4 ==== 6093+0+0 (secure 0 0 0) 0x7fe36404a2a0 con 0x7fe3700992f0 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36ffff700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 0x7fe35c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:19.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36ffff700 10 monclient: get_auth_request con 0x7fe35c035a70 auth_method 0 2024-09-18T10:13:19.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.147+0000 7fe36ffff700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 0x7fe35c037f20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe360006fd0 tx=0x7fe360004a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:19.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.155+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe364024350 con 0x7fe3700992f0 2024-09-18T10:13:19.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.155+0000 7fe36cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:19.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.155+0000 7fe36cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.319+0000 7fe376c3c700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.319+0000 7fe376c3c700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.319+0000 7fe376c3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.319+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0} v 0) v1 -- 0x7fe370129410 con 0x7fe3700992f0 2024-09-18T10:13:19.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.923+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}]=0 reweighted item id 12 name 'osd.12' to 113 in crush map v105) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe364024350 con 0x7fe3700992f0 2024-09-18T10:13:19.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.923+0000 7fe36cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:19.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:19.923+0000 7fe36cff9700 10 monclient: _finish_command 2 = system:0 reweighted item id 12 name 'osd.12' to 113 in crush map 2024-09-18T10:13:20.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.031+0000 7fe376c3c700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:20.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.031+0000 7fe376c3c700 10 monclient: _send_command 3 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:20.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.031+0000 7fe376c3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:20.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.031+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0} v 0) v1 -- 0x7fe370129410 con 0x7fe3700992f0 2024-09-18T10:13:20.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.903+0000 7fe36cff9700 1 -- 172.21.15.150:0/71912212 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}]=0 reweighted item id 12 name 'osd.12' to 113 in crush map v106) v1 ==== 156+0+0 (secure 0 0 0) 0x7fe364024350 con 0x7fe3700992f0 2024-09-18T10:13:20.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.903+0000 7fe36cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 113.0}] 2024-09-18T10:13:20.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.903+0000 7fe36cff9700 10 monclient: _finish_command 3 = system:0 reweighted item id 12 name 'osd.12' to 113 in crush map 2024-09-18T10:13:20.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 msgr2=0x7fe35c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:20.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 0x7fe35c037f20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe360006fd0 tx=0x7fe360004a10 comp rx=0 tx=0).stop 2024-09-18T10:13:20.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 10 monclient: shutdown 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 msgr2=0x7fe37013c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe37009e000 tx=0x7fe364016000 comp rx=0 tx=0).stop 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 shutdown_connections 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe35c035a70 0x7fe35c037f20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe37009b6f0 0x7fe37013f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe37009b320 0x7fe37012e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 --2- 172.21.15.150:0/71912212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3700992f0 0x7fe37013c540 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 >> 172.21.15.150:0/71912212 conn(0x7fe3700a15e0 msgr2=0x7fe37006a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 shutdown_connections 2024-09-18T10:13:20.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:20.907+0000 7fe376c3c700 1 -- 172.21.15.150:0/71912212 wait complete. 2024-09-18T10:13:20.914 INFO:tasks.workunit.client.0.smithi150.stderr:reweighted item id 12 name 'osd.12' to 113 in crush map 2024-09-18T10:13:20.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush reweight osd.12 123456 2024-09-18T10:13:20.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:13:20.927 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush reweight osd.12 123456 2024-09-18T10:13:21.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: get_monmap_and_config 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: monmap: 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:21.025474+0000 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:21.025474+0000 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:21.026 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:21.027 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:21.027 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 Processor -- start 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 -- start start 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: init 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 0x7f10140a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): picked mon.noname-a con 0x7f101409e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10140a07d0 0x7f10140a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): picked mon.noname-c con 0x7f10140a07d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f1014138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): picked mon.noname-b con 0x7f1014136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101409ee30 con 0x7f101409e6d0 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10140a1110 con 0x7f10140a07d0 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101409e220 con 0x7f1014136a10 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1019577700 10 monclient(hunting): authenticate will time out at 1083.119105s 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1012ffd700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 msgr2=0x7f10140a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f1012ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 0x7f10140a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10140a07d0 0x7f10140a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10140a07d0 0x7f10140a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43056/0 (socket says 172.21.15.150:43056) 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 -- 172.21.15.150:0/3749188565 learned_addr learned my addr 172.21.15.150:0/3749188565 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): get_auth_request con 0x7f10140a07d0 auth_method 0 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f1014138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): get_auth_request con 0x7f1014136a10 auth_method 0 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient(hunting): handle_auth_done global_id 15202 payload 274 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 -- 172.21.15.150:0/3749188565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 msgr2=0x7f1014138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f1014138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 -- 172.21.15.150:0/3749188565 >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 msgr2=0x7f10140a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 --2- 172.21.15.150:0/3749188565 >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 0x7f10140a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 monclient: found mon.noname-c 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.023+0000 7f10127fc700 1 -- 172.21.15.150:0/3749188565 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101409eaa0 con 0x7f10140a07d0 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f10127fc700 1 -- 172.21.15.150:0/3749188565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10140a07d0 msgr2=0x7f10140a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f10127fc700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10140a07d0 0x7f10140a0bd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1014069dd0 tx=0x7f0ffc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37b5d94aa8c2ed20 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 1 -- 172.21.15.150:0/3749188565 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc009e70 con 0x7f10140a07d0 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:21.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: dump: 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 1 -- 172.21.15.150:0/3749188565 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc00ba70 con 0x7f10140a07d0 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 1 -- 172.21.15.150:0/3749188565 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc00be30 con 0x7f10140a07d0 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 5 monclient: authenticate success, global_id 15202 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 4 set_mon_vals no callback set 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1011ffb700 10 monclient: dump: 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.037 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 10 monclient: shutdown 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 -- 172.21.15.150:0/3749188565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10140a07d0 msgr2=0x7f10140a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10140a07d0 0x7f10140a0bd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1014069dd0 tx=0x7f0ffc009250 comp rx=0 tx=0).stop 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 -- 172.21.15.150:0/3749188565 shutdown_connections 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f1014138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 --2- 172.21.15.150:0/3749188565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10140a07d0 0x7f10140a0bd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 --2- 172.21.15.150:0/3749188565 >> v2:172.21.15.142:3300/0 conn(0x7f101409e6d0 0x7f10140a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 -- 172.21.15.150:0/3749188565 >> 172.21.15.150:0/3749188565 conn(0x7f10140a58a0 msgr2=0x7f10140a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 -- 172.21.15.150:0/3749188565 shutdown_connections 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 1 -- 172.21.15.150:0/3749188565 wait complete. 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.027+0000 7f1019577700 10 monclient: monmap: 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:21.039 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 Processor -- start 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 -- start start 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient: init 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101409e6d0 0x7f10141420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): picked mon.noname-b con 0x7f101409e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10140a07d0 0x7f1014142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): picked mon.noname-a con 0x7f10140a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f101412c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): picked mon.noname-c con 0x7f1014136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:21.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101409bf20 con 0x7f10140a07d0 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101412e1c0 con 0x7f101409e6d0 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140a1340 con 0x7f1014136a10 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1019577700 10 monclient(hunting): authenticate will time out at 1083.125532s 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101409e6d0 0x7f10141420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f10127fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10140a07d0 msgr2=0x7f1014142600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f10127fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10140a07d0 0x7f1014142600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101409e6d0 0x7f10141420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43060/0 (socket says 172.21.15.150:43060) 2024-09-18T10:13:21.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 -- 172.21.15.150:0/1973834232 learned_addr learned my addr 172.21.15.150:0/1973834232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): get_auth_request con 0x7f101409e6d0 auth_method 0 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient(hunting): handle_auth_done global_id 15205 payload 824 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f10137fe700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f101412c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 -- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 msgr2=0x7f101412c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f101412c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 -- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10140a07d0 msgr2=0x7f1014142600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:21.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10140a07d0 0x7f1014142600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 monclient: found mon.noname-b 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f1012ffd700 1 -- 172.21.15.150:0/1973834232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ffc0094f0 con 0x7f101409e6d0 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f10137fe700 10 monclient: get_auth_request con 0x7f1014136a10 auth_method 0 2024-09-18T10:13:21.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.031+0000 7f10137fe700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1014136a10 0x7f101412c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1012ffd700 1 -- 172.21.15.150:0/1973834232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101409e6d0 msgr2=0x7f10141420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1012ffd700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f101409e6d0 0x7f10141420c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f100400b810 tx=0x7f1004009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004015070 con 0x7f101409e6d0 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:21.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: dump: 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1004006150 con 0x7f101409e6d0 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 5 monclient: authenticate success, global_id 15205 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _renew_subs 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 1 -- 172.21.15.150:0/1973834232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1014130220 con 0x7f101409e6d0 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004006510 con 0x7f101409e6d0 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 10 monclient: dump: 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.047 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f10117fa700 4 set_mon_vals no callback set 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _renew_subs 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 1 -- 172.21.15.150:0/1973834232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101412fc60 con 0x7f101409e6d0 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f1019577700 1 -- 172.21.15.150:0/1973834232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101412fe80 con 0x7f101409e6d0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1004010440 con 0x7f101409e6d0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ff4035a70 0x7f0ff4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(106..106 src has 40..106) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f1004048510 con 0x7f101409e6d0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f10127fc700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ff4035a70 0x7f0ff4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.035+0000 7f10127fc700 10 monclient: get_auth_request con 0x7f0ff4035a70 auth_method 0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.039+0000 7f10127fc700 1 --2- 172.21.15.150:0/1973834232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ff4035a70 0x7f0ff4037f20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009470 tx=0x7f0ffc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.043+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f10040106f0 con 0x7f101409e6d0 2024-09-18T10:13:21.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.043+0000 7f100affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.043+0000 7f100affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f1019577700 10 start_mon_command cmd=[{"prefix": "osd crush reweight", "name": "osd.12", "weight": 123456.0}] 2024-09-18T10:13:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f1019577700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 123456.0}] 2024-09-18T10:13:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f1019577700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f1019577700 1 -- 172.21.15.150:0/1973834232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight", "name": "osd.12", "weight": 123456.0} v 0) v1 -- 0x7f101409bb60 con 0x7f101409e6d0 2024-09-18T10:13:21.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f100affd700 1 -- 172.21.15.150:0/1973834232 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight", "name": "osd.12", "weight": 123456.0}]=-75 (75) Value too large for defined data type v106) v1 ==== 146+0+0 (secure 0 0 0) 0x7f1004018780 con 0x7f101409e6d0 2024-09-18T10:13:21.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f100affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight", "name": "osd.12", "weight": 123456.0}] 2024-09-18T10:13:21.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.215+0000 7f100affd700 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-18T10:13:21.219 INFO:tasks.workunit.client.0.smithi150.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-18T10:13:21.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:13:21.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:21.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.12 2024-09-18T10:13:21.222 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 113 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: get_monmap_and_config 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: monmap: 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:21.312846+0000 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:21.312846+0000 2024-09-18T10:13:21.313 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:21.314 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 Processor -- start 2024-09-18T10:13:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 -- start start 2024-09-18T10:13:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: init 2024-09-18T10:13:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f059009ac60 0x7f059009c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): picked mon.noname-c con 0x7f059009ac60 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f059006f9e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 0x7f0590137af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): picked mon.noname-a con 0x7f0590135710 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059009b9e0 con 0x7f059006f9e0 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059009b780 con 0x7f0590135710 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059009b030 con 0x7f059009ac60 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f0597382700 10 monclient(hunting): authenticate will time out at 1083.405963s 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f059009ac60 0x7f059009c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059591f700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 msgr2=0x7f0590137af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f059009ac60 0x7f059009c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43062/0 (socket says 172.21.15.150:43062) 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059491d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f059006f9e0 legacy=0x7f059009c820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 -- 172.21.15.150:0/2396418078 learned_addr learned my addr 172.21.15.150:0/2396418078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): get_auth_request con 0x7f059009ac60 auth_method 0 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059591f700 1 --2- 172.21.15.150:0/2396418078 >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 0x7f0590137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient(hunting): handle_auth_done global_id 15208 payload 274 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 -- 172.21.15.150:0/2396418078 >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 msgr2=0x7f0590137af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 --2- 172.21.15.150:0/2396418078 >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 0x7f0590137af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 -- 172.21.15.150:0/2396418078 >> v1:172.21.15.142:6789/0 conn(0x7f059006f9e0 legacy=0x7f059009c820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 monclient: found mon.noname-c 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.311+0000 7f059511e700 1 -- 172.21.15.150:0/2396418078 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0590098270 con 0x7f059009ac60 2024-09-18T10:13:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f059511e700 1 -- 172.21.15.150:0/2396418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 msgr2=0x7f059009c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f059511e700 1 --2- 172.21.15.150:0/2396418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f059009c2e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f05900a25c0 tx=0x7f058c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c323015951de968 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 1 -- 172.21.15.150:0/2396418078 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00ba70 con 0x7f059009ac60 2024-09-18T10:13:21.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: dump: 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.322 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 1 -- 172.21.15.150:0/2396418078 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c00bbd0 con 0x7f059009ac60 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 1 -- 172.21.15.150:0/2396418078 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c011420 con 0x7f059009ac60 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 5 monclient: authenticate success, global_id 15208 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 4 set_mon_vals no callback set 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0587fff700 10 monclient: dump: 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 10 monclient: shutdown 2024-09-18T10:13:21.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 -- 172.21.15.150:0/2396418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 msgr2=0x7f059009c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 --2- 172.21.15.150:0/2396418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f059009c2e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f05900a25c0 tx=0x7f058c009250 comp rx=0 tx=0).stop 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 -- 172.21.15.150:0/2396418078 shutdown_connections 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 --2- 172.21.15.150:0/2396418078 >> v2:172.21.15.142:3300/0 conn(0x7f0590135710 0x7f0590137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 --2- 172.21.15.150:0/2396418078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f059009c2e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 -- 172.21.15.150:0/2396418078 >> 172.21.15.150:0/2396418078 conn(0x7f05900a1710 msgr2=0x7f05900a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 -- 172.21.15.150:0/2396418078 shutdown_connections 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 1 -- 172.21.15.150:0/2396418078 wait complete. 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.315+0000 7f0597382700 10 monclient: monmap: 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.325 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 Processor -- start 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 -- start start 2024-09-18T10:13:21.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient: init 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f059006f9e0 0x7f0590134720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): picked mon.noname-c con 0x7f059006f9e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): picked mon.noname-b con 0x7f059009ac60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 0x7f05901351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): picked mon.noname-a con 0x7f0590135710 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059008d580 con 0x7f0590135710 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0590098480 con 0x7f059009ac60 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059006ee50 con 0x7f059006f9e0 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f0597382700 10 monclient(hunting): authenticate will time out at 1083.414211s 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059591f700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 msgr2=0x7f05901351a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059591f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 0x7f05901351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43078/0 (socket says 172.21.15.150:43078) 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 -- 172.21.15.150:0/3499397030 learned_addr learned my addr 172.21.15.150:0/3499397030 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): get_auth_request con 0x7f059009ac60 auth_method 0 2024-09-18T10:13:21.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f059006f9e0 0x7f0590134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): get_auth_request con 0x7f059006f9e0 auth_method 0 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059511e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient(hunting): handle_auth_done global_id 15211 payload 824 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 -- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f059006f9e0 msgr2=0x7f0590134720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f059006f9e0 0x7f0590134720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 -- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 msgr2=0x7f05901351a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 0x7f05901351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 monclient: found mon.noname-b 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.319+0000 7f059491d700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0094f0 con 0x7f059009ac60 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f059491d700 1 -- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 msgr2=0x7f0590134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f059491d700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f058000b810 tx=0x7f05800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580003d50 con 0x7f059009ac60 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: dump: 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.335 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800050f0 con 0x7f059009ac60 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 5 monclient: authenticate success, global_id 15211 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _renew_subs 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059008f020 con 0x7f059009ac60 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800054b0 con 0x7f059009ac60 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 10 monclient: dump: 2024-09-18T10:13:21.336 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05877fe700 4 set_mon_vals no callback set 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _renew_subs 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0590094fa0 con 0x7f059009ac60 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f058001a400 con 0x7f059009ac60 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0590063ed0 con 0x7f059009ac60 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 0x7f0574037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(106..106 src has 40..106) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f0580048470 con 0x7f059009ac60 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f059511e700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 0x7f0574037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.323+0000 7f059511e700 10 monclient: get_auth_request con 0x7f0574035a70 auth_method 0 2024-09-18T10:13:21.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.327+0000 7f059511e700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 0x7f0574037f20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f05900a25c0 tx=0x7f058c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.331+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0580019540 con 0x7f059009ac60 2024-09-18T10:13:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.331+0000 7f05857fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.331+0000 7f05857fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:21.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f0597382700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f0597382700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f0597382700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f059008f790 con 0x7f059009ac60 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v106) v1 ==== 56+0+1242 (secure 0 0 0) 0x7f0580019360 con 0x7f059009ac60 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f05857fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:21.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.499+0000 7f05857fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.595+0000 7f0597382700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:21.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f05900008d0 con 0x7f059009ac60 2024-09-18T10:13:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f05857fa700 1 -- 172.21.15.150:0/3499397030 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v106) v1 ==== 56+0+1242 (secure 0 0 0) 0x7f0580019540 con 0x7f059009ac60 2024-09-18T10:13:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f05857fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:21.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f05857fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 msgr2=0x7f0574037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 0x7f0574037f20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f05900a25c0 tx=0x7f058c016000 comp rx=0 tx=0).stop 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 10 monclient: shutdown 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 msgr2=0x7f0590134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f058000b810 tx=0x7f05800091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 shutdown_connections 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0574035a70 0x7f0574037f20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0590135710 0x7f05901351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f059009ac60 0x7f0590134c60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 --2- 172.21.15.150:0/3499397030 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f059006f9e0 0x7f0590134720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 >> 172.21.15.150:0/3499397030 conn(0x7f05900a1710 msgr2=0x7f059009df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.599+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 shutdown_connections 2024-09-18T10:13:21.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.603+0000 7f0597382700 1 -- 172.21.15.150:0/3499397030 wait complete. 2024-09-18T10:13:21.614 INFO:tasks.workunit.client.0.smithi150.stdout: 12 113.00000 osd.12 down 1.00000 1.00000 2024-09-18T10:13:21.614 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.12 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: get_monmap_and_config 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: monmap: 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:21.712664+0000 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:21.712664+0000 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:21.713 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:21.714 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:21.714 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:21.714 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:21.714 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 Processor -- start 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 -- start start 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: init 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 0x7f092809e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.716 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): picked mon.noname-b con 0x7f092809e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f092806cbc0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 0x7f092809ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): picked mon.noname-a con 0x7f09280699a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f092806af30 con 0x7f092806cbc0 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f092809f1c0 con 0x7f09280699a0 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092806b630 con 0x7f092809e370 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f0930397700 10 monclient(hunting): authenticate will time out at 1083.806225s 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e934700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 msgr2=0x7f092809ec80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e934700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 0x7f092809ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.717 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092d932700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f092806cbc0 0x7f0928132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52664/0 (socket says 172.21.15.150:52664) 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092d932700 1 -- 172.21.15.150:0/1863398967 learned_addr learned my addr 172.21.15.150:0/1863398967 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 --2- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 0x7f092809e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): get_auth_request con 0x7f092809e370 auth_method 0 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient(hunting): handle_auth_done global_id 44216 payload 274 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 -- 172.21.15.150:0/1863398967 >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 msgr2=0x7f092809ec80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 --2- 172.21.15.150:0/1863398967 >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 0x7f092809ec80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 -- 172.21.15.150:0/1863398967 >> v1:172.21.15.150:6789/0 conn(0x7f092806cbc0 legacy=0x7f0928132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 monclient: found mon.noname-b 2024-09-18T10:13:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:21.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.711+0000 7f092e133700 1 -- 172.21.15.150:0/1863398967 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092806ac50 con 0x7f092809e370 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092e133700 1 -- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 msgr2=0x7f092809e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092e133700 1 --2- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 0x7f092809e740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f09200098e0 tx=0x7f09200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b883d8a8ce20434 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 1 -- 172.21.15.150:0/1863398967 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000ba70 con 0x7f092809e370 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: dump: 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.723 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 5 monclient: authenticate success, global_id 44216 2024-09-18T10:13:21.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 1 -- 172.21.15.150:0/1863398967 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092000bbf0 con 0x7f092809e370 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 4 set_mon_vals no callback set 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 1 -- 172.21.15.150:0/1863398967 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920011420 con 0x7f092809e370 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f092d131700 10 monclient: dump: 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.725 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 10 monclient: shutdown 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 -- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 msgr2=0x7f092809e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 --2- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 0x7f092809e740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f09200098e0 tx=0x7f09200091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 -- 172.21.15.150:0/1863398967 shutdown_connections 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 --2- 172.21.15.150:0/1863398967 >> v2:172.21.15.142:3300/0 conn(0x7f09280699a0 0x7f092809ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 --2- 172.21.15.150:0/1863398967 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f092809e370 0x7f092809e740 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 -- 172.21.15.150:0/1863398967 >> 172.21.15.150:0/1863398967 conn(0x7f0928071850 msgr2=0x7f09280a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 -- 172.21.15.150:0/1863398967 shutdown_connections 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 1 -- 172.21.15.150:0/1863398967 wait complete. 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 10 monclient: build_initial_monmap 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.715+0000 7f0930397700 10 monclient: monmap: 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:21.727 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 Processor -- start 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 -- start start 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient: init 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09280699a0 0x7f09281322a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): picked mon.noname-c con 0x7f09280699a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 0x7f092811bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): picked mon.noname-a con 0x7f092806cbc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): picked mon.noname-b con 0x7f092809e370 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:21.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092806d6e0 con 0x7f092806cbc0 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092811f6a0 con 0x7f092809e370 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092811d900 con 0x7f09280699a0 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f0930397700 10 monclient(hunting): authenticate will time out at 1083.813087s 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092d932700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 msgr2=0x7f092811bca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092d932700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 0x7f092811bca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43086/0 (socket says 172.21.15.150:43086) 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 -- 172.21.15.150:0/2752803176 learned_addr learned my addr 172.21.15.150:0/2752803176 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:21.729 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): get_auth_request con 0x7f092809e370 auth_method 0 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09280699a0 0x7f09281322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): get_auth_request con 0x7f09280699a0 auth_method 0 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:21.730 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient(hunting): handle_auth_done global_id 15214 payload 824 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 -- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09280699a0 msgr2=0x7f09281322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09280699a0 0x7f09281322a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 -- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 msgr2=0x7f092811bca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 0x7f092811bca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient: _finish_hunting 0 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 monclient: found mon.noname-b 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.731 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.719+0000 7f092e934700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920009420 con 0x7f092809e370 2024-09-18T10:13:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f092e934700 1 -- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 msgr2=0x7f092811c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:21.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f092e934700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fc0 tx=0x7f0918009730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800ce20 con 0x7f092809e370 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: dump: 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.734 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 5 monclient: authenticate success, global_id 15214 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _renew_subs 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092811f3c0 con 0x7f092809e370 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0918016430 con 0x7f092809e370 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180165e0 con 0x7f092809e370 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:21.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 10 monclient: dump: 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:21.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0927fff700 4 set_mon_vals no callback set 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _renew_subs 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092811e4d0 con 0x7f092809e370 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092811f810 con 0x7f092809e370 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0918012020 con 0x7f092809e370 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 0x7f0914037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(106..106 src has 40..106) v4 ==== 6093+0+0 (secure 0 0 0) 0x7f0918048a50 con 0x7f092809e370 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f092e133700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 0x7f0914037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.723+0000 7f092e133700 10 monclient: get_auth_request con 0x7f0914035a70 auth_method 0 2024-09-18T10:13:21.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.727+0000 7f092e133700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 0x7f0914037f20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f09200098e0 tx=0x7f0920016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.731+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f09180069d0 con 0x7f092809e370 2024-09-18T10:13:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.731+0000 7f09267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:21.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.731+0000 7f09267fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.899+0000 7f0930397700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.899+0000 7f0930397700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.899+0000 7f0930397700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:21.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.899+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f0928063ed0 con 0x7f092809e370 2024-09-18T10:13:21.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.947+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 removed item id 12 name 'osd.12' from crush map v107) v1 ==== 125+0+0 (secure 0 0 0) 0x7f09180072f0 con 0x7f092809e370 2024-09-18T10:13:21.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.947+0000 7f09267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:21.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:21.947+0000 7f09267fc700 10 monclient: _finish_command 2 = system:0 removed item id 12 name 'osd.12' from crush map 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.055+0000 7f0930397700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.055+0000 7f0930397700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.055+0000 7f0930397700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.055+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.12"} v 0) v1 -- 0x7f09280008d0 con 0x7f092809e370 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f09267fc700 1 -- 172.21.15.150:0/2752803176 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.12"}]=0 device 'osd.12' does not appear in the crush map v107) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0918025090 con 0x7f092809e370 2024-09-18T10:13:22.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f09267fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.12"}] 2024-09-18T10:13:22.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f09267fc700 10 monclient: _finish_command 3 = system:0 device 'osd.12' does not appear in the crush map 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 msgr2=0x7f0914037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 0x7f0914037f20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f09200098e0 tx=0x7f0920016000 comp rx=0 tx=0).stop 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 10 monclient: shutdown 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 msgr2=0x7f092811c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fc0 tx=0x7f0918009730 comp rx=0 tx=0).stop 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 shutdown_connections 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0914035a70 0x7f0914037f20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f092809e370 0x7f092811c1e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f092806cbc0 0x7f092811bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 --2- 172.21.15.150:0/2752803176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09280699a0 0x7f09281322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 >> 172.21.15.150:0/2752803176 conn(0x7f0928071850 msgr2=0x7f092809ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:22.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 shutdown_connections 2024-09-18T10:13:22.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.059+0000 7f0930397700 1 -- 172.21.15.150:0/2752803176 wait complete. 2024-09-18T10:13:22.064 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.12' does not appear in the crush map 2024-09-18T10:13:22.072 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm osd.12 2024-09-18T10:13:22.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: get_monmap_and_config 2024-09-18T10:13:22.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: build_initial_monmap 2024-09-18T10:13:22.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: monmap: 2024-09-18T10:13:22.168 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:22.168 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:22.168 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:22.166338+0000 2024-09-18T10:13:22.168 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:22.166338+0000 2024-09-18T10:13:22.168 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:22.169 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:22.170 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 Processor -- start 2024-09-18T10:13:22.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 -- start start 2024-09-18T10:13:22.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: init 2024-09-18T10:13:22.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:22.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:13:22.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 0x7fbfe409c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-a con 0x7fbfe409ac60 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfe406f9e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-b con 0x7fbfe4135710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe409b9e0 con 0x7fbfe406f9e0 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:22.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe409b780 con 0x7fbfe409ac60 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe409b030 con 0x7fbfe4135710 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfeba78700 10 monclient(hunting): authenticate will time out at 1084.259718s 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfe9013700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fbfe406f9e0 legacy=0x7fbfe409c820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:22.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfe9814700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 msgr2=0x7fbfe409c2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:22.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.163+0000 7fbfe9814700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 0x7fbfe409c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:22.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:22.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:35304/0 (socket says 172.21.15.150:35304) 2024-09-18T10:13:22.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 -- 172.21.15.150:0/4096906087 learned_addr learned my addr 172.21.15.150:0/4096906087 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:22.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): get_auth_request con 0x7fbfe4135710 auth_method 0 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:22.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient(hunting): handle_auth_done global_id 44222 payload 274 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 -- 172.21.15.150:0/4096906087 >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 msgr2=0x7fbfe409c2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 --2- 172.21.15.150:0/4096906087 >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 0x7fbfe409c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 -- 172.21.15.150:0/4096906087 >> v1:172.21.15.142:6789/0 conn(0x7fbfe406f9e0 legacy=0x7fbfe409c820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient: _finish_hunting 0 2024-09-18T10:13:22.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 monclient: found mon.noname-b 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 -- 172.21.15.150:0/4096906087 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4098270 con 0x7fbfe4135710 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 -- 172.21.15.150:0/4096906087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 msgr2=0x7fbfe4137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfea015700 1 --2- 172.21.15.150:0/4096906087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0099b0 tx=0x7fbfdc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c4a102c13b67bf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:22.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 1 -- 172.21.15.150:0/4096906087 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00bb50 con 0x7fbfe4135710 2024-09-18T10:13:22.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:22.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: dump: 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:22.192 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: _finish_auth 0 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 1 -- 172.21.15.150:0/4096906087 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00bcb0 con 0x7fbfe4135710 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 1 -- 172.21.15.150:0/4096906087 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc011670 con 0x7fbfe4135710 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfe8812700 10 monclient: dump: 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.193 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfeba78700 5 monclient: authenticate success, global_id 44222 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfeba78700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.167+0000 7fbfeba78700 4 set_mon_vals no callback set 2024-09-18T10:13:22.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: shutdown 2024-09-18T10:13:22.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:22.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- 172.21.15.150:0/4096906087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 msgr2=0x7fbfe4137af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:22.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- 172.21.15.150:0/4096906087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0099b0 tx=0x7fbfdc009250 comp rx=0 tx=0).stop 2024-09-18T10:13:22.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- 172.21.15.150:0/4096906087 shutdown_connections 2024-09-18T10:13:22.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- 172.21.15.150:0/4096906087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe4135710 0x7fbfe4137af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- 172.21.15.150:0/4096906087 >> v2:172.21.15.142:3300/0 conn(0x7fbfe409ac60 0x7fbfe409c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- 172.21.15.150:0/4096906087 >> 172.21.15.150:0/4096906087 conn(0x7fbfe40a1710 msgr2=0x7fbfe40a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- 172.21.15.150:0/4096906087 shutdown_connections 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- 172.21.15.150:0/4096906087 wait complete. 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: build_initial_monmap 2024-09-18T10:13:22.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: monmap: 2024-09-18T10:13:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:22.197 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 Processor -- start 2024-09-18T10:13:22.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- start start 2024-09-18T10:13:22.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: init 2024-09-18T10:13:22.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:22.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:22.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe406f9e0 0x7fbfe4098a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-c con 0x7fbfe406f9e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:22.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-b con 0x7fbfe409ac60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:22.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 0x7fbfe40994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): picked mon.noname-a con 0x7fbfe4135710 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:22.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:22.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4132930 con 0x7fbfe4135710 2024-09-18T10:13:22.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:22.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe409aa50 con 0x7fbfe409ac60 2024-09-18T10:13:22.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:22.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe412c580 con 0x7fbfe406f9e0 2024-09-18T10:13:22.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:22.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfeba78700 10 monclient(hunting): authenticate will time out at 1084.266126s 2024-09-18T10:13:22.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfea015700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 msgr2=0x7fbfe40994d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:22.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:22.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfea015700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 0x7fbfe40994d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:22.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43098/0 (socket says 172.21.15.150:43098) 2024-09-18T10:13:22.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 -- 172.21.15.150:0/3347431556 learned_addr learned my addr 172.21.15.150:0/3347431556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:22.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): get_auth_request con 0x7fbfe409ac60 auth_method 0 2024-09-18T10:13:22.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:22.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:22.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:22.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe406f9e0 0x7fbfe4098a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:22.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:22.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): get_auth_request con 0x7fbfe406f9e0 auth_method 0 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:22.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:22.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:22.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9814700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:22.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient(hunting): handle_auth_done global_id 15217 payload 824 2024-09-18T10:13:22.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 -- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe406f9e0 msgr2=0x7fbfe4098a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:22.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe406f9e0 0x7fbfe4098a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 -- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 msgr2=0x7fbfe40994d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:22.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 0x7fbfe40994d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:22.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient: _finish_hunting 0 2024-09-18T10:13:22.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 monclient: found mon.noname-b 2024-09-18T10:13:22.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:22.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.171+0000 7fbfe9013700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc0094f0 con 0x7fbfe409ac60 2024-09-18T10:13:22.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe9013700 1 -- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 msgr2=0x7fbfe4098f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:22.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe9013700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400bb80 tx=0x7fbfd40097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:22.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400cad0 con 0x7fbfe409ac60 2024-09-18T10:13:22.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:22.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:22.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: dump: 2024-09-18T10:13:22.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:22.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:22.210 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.210 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:22.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:22.211 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:22.211 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:22.211 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:22.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:22.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 5 monclient: authenticate success, global_id 15217 2024-09-18T10:13:22.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd400cc30 con 0x7fbfe409ac60 2024-09-18T10:13:22.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _renew_subs 2024-09-18T10:13:22.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe412c860 con 0x7fbfe409ac60 2024-09-18T10:13:22.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:22.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400ce20 con 0x7fbfe409ac60 2024-09-18T10:13:22.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:22.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:22.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe37fe700 4 set_mon_vals no callback set 2024-09-18T10:13:22.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 10 monclient: dump: 2024-09-18T10:13:22.214 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:22.214 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:22.214 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.214 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:22.215 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _renew_subs 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe4132360 con 0x7fbfe409ac60 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe406cdd0 con 0x7fbfe409ac60 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbfd4012020 con 0x7fbfe409ac60 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 0x7fbfc4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(107..107 src has 40..107) v4 ==== 6071+0+0 (secure 0 0 0) 0x7fbfd4048760 con 0x7fbfe409ac60 2024-09-18T10:13:22.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe9814700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 0x7fbfc4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:22.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.175+0000 7fbfe9814700 10 monclient: get_auth_request con 0x7fbfc4035a70 auth_method 0 2024-09-18T10:13:22.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.179+0000 7fbfe9814700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 0x7fbfc4037f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc003980 tx=0x7fbfcc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:22.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.183+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbfd401d900 con 0x7fbfe409ac60 2024-09-18T10:13:22.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.183+0000 7fbfe1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:22.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.183+0000 7fbfe1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:22.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.355+0000 7fbfeba78700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:22.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.359+0000 7fbfeba78700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:22.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.359+0000 7fbfeba78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:22.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.359+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7fbfe4063ed0 con 0x7fbfe409ac60 2024-09-18T10:13:22.361 INFO:tasks.ceph.mon.b.smithi150.stderr:2024-09-18T10:13:22.359+0000 7f65bb623700 -1 mon.b@1(leader).osd e107 definitely_dead 0 2024-09-18T10:13:22.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.959+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 removed osd.12 v108) v1 ==== 87+0+0 (secure 0 0 0) 0x7fbfd4014340 con 0x7fbfe409ac60 2024-09-18T10:13:22.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.959+0000 7fbfe1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:22.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:22.959+0000 7fbfe1ffb700 10 monclient: _finish_command 2 = system:0 removed osd.12 2024-09-18T10:13:23.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfeba78700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:23.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfeba78700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:23.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfeba78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.12"]} v 0) v1 -- 0x7fbfe412c6c0 con 0x7fbfe409ac60 2024-09-18T10:13:23.074 INFO:tasks.ceph.mon.b.smithi150.stderr:2024-09-18T10:13:23.071+0000 7f65bb623700 -1 mon.b@1(leader).osd e108 definitely_dead 0 2024-09-18T10:13:23.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfe1ffb700 1 -- 172.21.15.150:0/3347431556 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.12"]}]=0 osd.12 does not exist. v108) v1 ==== 96+0+0 (secure 0 0 0) 0x7fbfe412c6c0 con 0x7fbfe409ac60 2024-09-18T10:13:23.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfe1ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.12"]}] 2024-09-18T10:13:23.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.071+0000 7fbfe1ffb700 10 monclient: _finish_command 3 = system:0 osd.12 does not exist. 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 msgr2=0x7fbfc4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 0x7fbfc4037f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc003980 tx=0x7fbfcc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 10 monclient: shutdown 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 msgr2=0x7fbfe4098f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:23.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400bb80 tx=0x7fbfd40097f0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 shutdown_connections 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbfc4035a70 0x7fbfc4037f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfe4135710 0x7fbfe40994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfe409ac60 0x7fbfe4098f90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 --2- 172.21.15.150:0/3347431556 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbfe406f9e0 0x7fbfe4098a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 >> 172.21.15.150:0/3347431556 conn(0x7fbfe40a1710 msgr2=0x7fbfe4134f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 shutdown_connections 2024-09-18T10:13:23.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.075+0000 7fbfeba78700 1 -- 172.21.15.150:0/3347431556 wait complete. 2024-09-18T10:13:23.079 INFO:tasks.workunit.client.0.smithi150.stderr:osd.12 does not exist. 2024-09-18T10:13:23.087 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph osd create 2024-09-18T10:13:23.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.175+0000 7f40e5f42700 10 monclient: get_monmap_and_config 2024-09-18T10:13:23.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.175+0000 7f40e5f42700 10 monclient: build_initial_monmap 2024-09-18T10:13:23.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.175+0000 7f40e5f42700 10 monclient: monmap: 2024-09-18T10:13:23.211 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:23.211 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.211 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:23.179749+0000 2024-09-18T10:13:23.211 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:23.179749+0000 2024-09-18T10:13:23.212 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:23.212 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.212 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:23.212 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:23.212 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:23.213 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:23.213 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:23.213 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 Processor -- start 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 -- start start 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient: init 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40e00a39f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40e00a3dc0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f40e0138890 0x7f40e013ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-a con 0x7f40e0138890 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e006cf00 con 0x7f40e00a3dc0 2024-09-18T10:13:23.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e006a490 con 0x7f40e00a39f0 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e006bf90 con 0x7f40e0138890 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40e5f42700 10 monclient(hunting): authenticate will time out at 1085.273088s 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40d6ffd700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f40e00a3dc0 legacy=0x7f40e00a41e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40df7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f40e00a39f0 0x7f40e006c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52674/0 (socket says 172.21.15.150:52674) 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40df7fe700 1 -- 172.21.15.150:0/2189841915 learned_addr learned my addr 172.21.15.150:0/2189841915 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 >> v2:172.21.15.142:3300/0 conn(0x7f40e0138890 msgr2=0x7f40e013ac70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40dffff700 1 --2- 172.21.15.150:0/2189841915 >> v2:172.21.15.142:3300/0 conn(0x7f40e0138890 0x7f40e013ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40df7fe700 1 -- 172.21.15.150:0/2189841915 >> v1:172.21.15.150:6789/0 conn(0x7f40e00a39f0 legacy=0x7f40e006c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228762802 0 0) 0x7f40e006a490 con 0x7f40e00a39f0 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:23.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40deffd700 10 monclient(hunting): my global_id is 15220 2024-09-18T10:13:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.179+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40cc003570 con 0x7f40e00a39f0 2024-09-18T10:13:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544820606 0 0) 0x7f40cc003570 con 0x7f40e00a39f0 2024-09-18T10:13:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 >> v2:172.21.15.142:3300/0 conn(0x7f40e0138890 msgr2=0x7f40e013ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:23.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 --2- 172.21.15.150:0/2189841915 >> v2:172.21.15.142:3300/0 conn(0x7f40e0138890 0x7f40e013ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 >> v1:172.21.15.142:6789/0 conn(0x7f40e00a3dc0 legacy=0x7f40e00a41e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:23.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e006b660 con 0x7f40e00a39f0 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: _finish_auth 0 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40e5f42700 5 monclient: authenticate success, global_id 15220 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f40d00017a0 con 0x7f40e00a39f0 2024-09-18T10:13:23.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40e5f42700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: dump: 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.243 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:23.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 >> v1:172.21.15.150:6789/0 conn(0x7f40e00a39f0 legacy=0x7f40e006c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40cc003a60 0x7f40cc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): picked mon.a con 0x7f40cc003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40cc004810 0x7f40cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): picked mon.c con 0x7f40cc004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:23.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40cc00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): picked mon.b con 0x7f40cc0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e006bf90 con 0x7f40cc003a60 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc00c380 con 0x7f40cc0053c0 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40cc00c540 con 0x7f40cc004810 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40deffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:23.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40e5f42700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40df7fe700 1 -- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40cc003a60 msgr2=0x7f40cc003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40df7fe700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40cc003a60 0x7f40cc003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40cc004810 0x7f40cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40cc00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): get_auth_request con 0x7f40cc004810 auth_method 0 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): get_auth_request con 0x7f40cc0053c0 auth_method 0 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:23.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient(hunting): handle_auth_done global_id 15220 payload 293 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40cc004810 msgr2=0x7f40cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40cc004810 0x7f40cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40cc003a60 msgr2=0x7f40cc003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40cc003a60 0x7f40cc003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 monclient: found mon.b 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:23.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc003570 con 0x7f40cc0053c0 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 10 monclient: _finish_auth 0 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 reap_dead start 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 -- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 msgr2=0x7f40cc00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.183+0000 7f40dffff700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40cc00baa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009f90 tx=0x7f40d80095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5047de03fb6498c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d8009900 con 0x7f40cc0053c0 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: dump: 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.249 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d800be70 con 0x7f40cc0053c0 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 4 set_mon_vals no callback set 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 1 -- 172.21.15.150:0/2189841915 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d80156a0 con 0x7f40cc0053c0 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:23.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40deffd700 10 monclient: dump: 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: shutdown 2024-09-18T10:13:23.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 msgr2=0x7f40cc00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40cc00baa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009f90 tx=0x7f40d80095d0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- 172.21.15.150:0/2189841915 shutdown_connections 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 --2- 172.21.15.150:0/2189841915 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40cc00baa0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- 172.21.15.150:0/2189841915 >> 172.21.15.150:0/2189841915 conn(0x7f40e00a6300 msgr2=0x7f40e00710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- 172.21.15.150:0/2189841915 shutdown_connections 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- 172.21.15.150:0/2189841915 wait complete. 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: build_initial_monmap 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: monmap: 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:23.252 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 Processor -- start 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- start start 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: init 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-b con 0x7f40cc0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:23.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40e011f450 0x7f40e011f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-c con 0x7f40e011f450 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 0x7f40e0147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): picked mon.noname-a con 0x7f40e011fd60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0122780 con 0x7f40e011fd60 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00a0120 con 0x7f40cc0053c0 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e00a1400 con 0x7f40e011f450 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40e5f42700 10 monclient(hunting): authenticate will time out at 1085.283467s 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:23.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40dffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 msgr2=0x7f40e0147d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43110/0 (socket says 172.21.15.150:43110) 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40dffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 0x7f40e0147d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 1 -- 172.21.15.150:0/1015647245 learned_addr learned my addr 172.21.15.150:0/1015647245 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): get_auth_request con 0x7f40cc0053c0 auth_method 0 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40d6ffd700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40e011f450 0x7f40e011f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:23.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40d6ffd700 10 monclient(hunting): get_auth_request con 0x7f40e011f450 auth_method 0 2024-09-18T10:13:23.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:23.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:23.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.187+0000 7f40d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:23.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:23.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 10 monclient(hunting): handle_auth_done global_id 15223 payload 824 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 -- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40e011f450 msgr2=0x7f40e011f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40e011f450 0x7f40e011f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 -- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 msgr2=0x7f40e0147d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:23.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 0x7f40e0147d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e01215e0 con 0x7f40cc0053c0 2024-09-18T10:13:23.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 -- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 msgr2=0x7f40e011ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000f60 tx=0x7f40d00010c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0007570 con 0x7f40cc0053c0 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40df7fe700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000f60 tx=0x7f40d00010c0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f40d00076f0 config(0 keys) v1 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:23.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 10 monclient: dump: 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:23.260 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:23.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d00076f0 con 0x7f40cc0053c0 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 5 monclient: authenticate success, global_id 15223 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 10 monclient: _renew_subs 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e01224a0 con 0x7f40cc0053c0 2024-09-18T10:13:23.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40d7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40de7fc700 4 set_mon_vals no callback set 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 10 monclient: _renew_subs 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.191+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e0138520 con 0x7f40cc0053c0 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40e5f42700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:23.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40e5f42700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:23.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40e5f42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:23.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0004a80 con 0x7f40cc0053c0 2024-09-18T10:13:23.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e0063ed0 con 0x7f40cc0053c0 2024-09-18T10:13:23.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:23.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 10 monclient: dump: 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:23.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f40d0014420 con 0x7f40cc0053c0 2024-09-18T10:13:23.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d7fff700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 0x7f40c0037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:23.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d6ffd700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 0x7f40c0037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:23.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.195+0000 7f40d6ffd700 10 monclient: get_auth_request con 0x7f40c0035aa0 auth_method 0 2024-09-18T10:13:23.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.199+0000 7f40d6ffd700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 0x7f40c0037f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009220 tx=0x7f40c80090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:23.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:23.203+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(108..108 src has 40..108) v4 ==== 6123+0+0 (secure 0 0 0) 0x7f40d0048860 con 0x7f40cc0053c0 2024-09-18T10:13:24.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.019+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f40d0004e30 con 0x7f40cc0053c0 2024-09-18T10:13:24.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.019+0000 7f40d7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:24.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.019+0000 7f40d7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:24.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.203+0000 7f40e5f42700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:24.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.203+0000 7f40e5f42700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T10:13:24.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.203+0000 7f40e5f42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:24.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.203+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f40e00a5880 con 0x7f40cc0053c0 2024-09-18T10:13:24.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.967+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v109) v1 ==== 58+0+3 (secure 0 0 0) 0x7f40d000a760 con 0x7f40cc0053c0 2024-09-18T10:13:24.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.967+0000 7f40d7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T10:13:24.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:24.967+0000 7f40d7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:25.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.087+0000 7f40e5f42700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:25.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.087+0000 7f40e5f42700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:13:25.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.087+0000 7f40e5f42700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:25.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.087+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7f40e00008d0 con 0x7f40cc0053c0 2024-09-18T10:13:25.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.903+0000 7f40d7fff700 1 -- 172.21.15.150:0/1015647245 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v110) v1 ==== 58+0+3 (secure 0 0 0) 0x7f40d000a940 con 0x7f40cc0053c0 2024-09-18T10:13:25.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.903+0000 7f40d7fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T10:13:25.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.903+0000 7f40d7fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 msgr2=0x7f40c0037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 0x7f40c0037f50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009220 tx=0x7f40c80090f0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 10 monclient: shutdown 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 msgr2=0x7f40e011ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000f60 tx=0x7f40d00010c0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 shutdown_connections 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f40c0035aa0 0x7f40c0037f50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40e011fd60 0x7f40e0147d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40e011f450 0x7f40e011f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 --2- 172.21.15.150:0/1015647245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc0053c0 0x7f40e011ef10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:25.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 >> 172.21.15.150:0/1015647245 conn(0x7f40e00a6300 msgr2=0x7f40e006ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:25.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 shutdown_connections 2024-09-18T10:13:25.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:25.907+0000 7f40e5f42700 1 -- 172.21.15.150:0/1015647245 wait complete. 2024-09-18T10:13:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:+ o4=13 2024-09-18T10:13:25.923 INFO:tasks.workunit.client.0.smithi150.stderr:++ ceph osd create 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: get_monmap_and_config 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: build_initial_monmap 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: monmap: 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:26.021659+0000 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:26.021659+0000 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.022 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:26.023 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:26.023 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:26.023 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:26.023 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:26.023 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 Processor -- start 2024-09-18T10:13:26.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 -- start start 2024-09-18T10:13:26.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: init 2024-09-18T10:13:26.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:26.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe15c09c2e0 0x7fe15c0956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-a con 0x7fe15c09c2e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe15c098a50 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe15c099560 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c0984f0 con 0x7fe15c098a50 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c09af60 con 0x7fe15c099560 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c09b520 con 0x7fe15c09c2e0 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe163d2c700 10 monclient(hunting): authenticate will time out at 1088.114868s 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe1622c9700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe15c099560 0x7fe15c136710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52678/0 (socket says 172.21.15.150:52678) 2024-09-18T10:13:26.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe1622c9700 1 -- 172.21.15.150:0/2188705230 learned_addr learned my addr 172.21.15.150:0/2188705230 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:26.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe161ac8700 1 -- 172.21.15.150:0/2188705230 >> v2:172.21.15.142:3300/0 conn(0x7fe15c09c2e0 msgr2=0x7fe15c0956d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:26.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe1612c7700 1 -- 172.21.15.150:0/2188705230 >> v1:172.21.15.142:6789/0 conn(0x7fe15c098a50 legacy=0x7fe15c098e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:26.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.019+0000 7fe161ac8700 1 --2- 172.21.15.150:0/2188705230 >> v2:172.21.15.142:3300/0 conn(0x7fe15c09c2e0 0x7fe15c0956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1622c9700 1 -- 172.21.15.150:0/2188705230 >> v1:172.21.15.150:6789/0 conn(0x7fe15c099560 legacy=0x7fe15c136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922085738 0 0) 0x7fe15c09af60 con 0x7fe15c099560 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): my global_id is 15226 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003570 con 0x7fe15c099560 2024-09-18T10:13:26.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852757424 0 0) 0x7fe148003570 con 0x7fe15c099560 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 >> v2:172.21.15.142:3300/0 conn(0x7fe15c09c2e0 msgr2=0x7fe15c0956d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 --2- 172.21.15.150:0/2188705230 >> v2:172.21.15.142:3300/0 conn(0x7fe15c09c2e0 0x7fe15c0956d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 >> v1:172.21.15.142:6789/0 conn(0x7fe15c098a50 legacy=0x7fe15c098e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: _finish_hunting 0 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c09c6b0 con 0x7fe15c099560 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: _finish_auth 0 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe163d2c700 5 monclient: authenticate success, global_id 15226 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fe154001b20 con 0x7fe15c099560 2024-09-18T10:13:26.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: dump: 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 >> v1:172.21.15.150:6789/0 conn(0x7fe15c099560 legacy=0x7fe15c136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe148003a60 0x7fe148003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): picked mon.a con 0x7fe148003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): picked mon.b con 0x7fe148004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1480051c0 0x7fe148010070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): picked mon.c con 0x7fe1480051c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c09b520 con 0x7fe148003a60 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480107f0 con 0x7fe148004810 2024-09-18T10:13:26.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1480109b0 con 0x7fe1480051c0 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe160ac6700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe163d2c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe161ac8700 1 -- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe148003a60 msgr2=0x7fe148003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe161ac8700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe148003a60 0x7fe148003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): get_auth_request con 0x7fe148004810 auth_method 0 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:26.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient(hunting): handle_auth_done global_id 15226 payload 293 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 -- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1480051c0 msgr2=0x7fe148010070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1622c9700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1480051c0 0x7fe148010070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1480051c0 0x7fe148010070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 -- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe148003a60 msgr2=0x7fe148003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe148003a60 0x7fe148003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient: _finish_hunting 0 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 monclient: found mon.b 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 1 -- 172.21.15.150:0/2188705230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe148003570 con 0x7fe148004810 2024-09-18T10:13:26.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 10 monclient: _finish_auth 0 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1612c7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1622c9700 10 monclient: get_auth_request con 0x7fe1480051c0 auth_method 0 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1622c9700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1480051c0 0x7fe148010070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.023+0000 7fe1622c9700 1 -- 172.21.15.150:0/2188705230 reap_dead start 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe1612c7700 1 -- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 msgr2=0x7fe148004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe148004c80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe158003980 tx=0x7fe1580090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=657a5ceb72ed0ee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe158009e00 con 0x7fe148004810 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: dump: 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.035 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe158003c10 con 0x7fe148004810 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 1 -- 172.21.15.150:0/2188705230 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1580050f0 con 0x7fe148004810 2024-09-18T10:13:26.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 4 set_mon_vals no callback set 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe160ac6700 10 monclient: dump: 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:26.037 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: shutdown 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 msgr2=0x7fe148004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe148004c80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe158003980 tx=0x7fe1580090f0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- 172.21.15.150:0/2188705230 shutdown_connections 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2188705230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe148004c80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- 172.21.15.150:0/2188705230 >> 172.21.15.150:0/2188705230 conn(0x7fe15c0a1710 msgr2=0x7fe15c0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- 172.21.15.150:0/2188705230 shutdown_connections 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- 172.21.15.150:0/2188705230 wait complete. 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: build_initial_monmap 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: monmap: 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:26.043 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 Processor -- start 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- start start 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: init 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-b con 0x7fe148004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe15c08bd40 0x7fe15c089c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-c con 0x7fe15c08bd40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 0x7fe15c08a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): picked mon.noname-a con 0x7fe15c08a140 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:26.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0942f0 con 0x7fe15c08a140 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c08cf80 con 0x7fe148004810 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c094c40 con 0x7fe15c08bd40 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe163d2c700 10 monclient(hunting): authenticate will time out at 1088.123730s 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43140/0 (socket says 172.21.15.150:43140) 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe1622c9700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 msgr2=0x7fe15c08a5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 1 -- 172.21.15.150:0/2512608421 learned_addr learned my addr 172.21.15.150:0/2512608421 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): get_auth_request con 0x7fe148004810 auth_method 0 2024-09-18T10:13:26.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe1622c9700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 0x7fe15c08a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.027+0000 7fe161ac8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe15c08bd40 0x7fe15c089c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1612c7700 10 monclient(hunting): get_auth_request con 0x7fe15c08bd40 auth_method 0 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1612c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1612c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1612c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 10 monclient(hunting): handle_auth_done global_id 15229 payload 824 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 -- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe15c08bd40 msgr2=0x7fe15c089c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:26.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe15c08bd40 0x7fe15c089c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 -- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 msgr2=0x7fe15c08a5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 0x7fe15c08a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 10 monclient: _finish_hunting 0 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 monclient: found mon.noname-b 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0951f0 con 0x7fe148004810 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 -- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 msgr2=0x7fe15c0896c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe161ac8700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00b7e0 tx=0x7fe14c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c005110 con 0x7fe148004810 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:26.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: dump: 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:26.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c005270 con 0x7fe148004810 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 5 monclient: authenticate success, global_id 15229 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 10 monclient: _renew_subs 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c08aaf0 con 0x7fe148004810 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c005630 con 0x7fe148004810 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe151ffb700 10 monclient: dump: 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:26.049 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 10 monclient: _renew_subs 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c097310 con 0x7fe148004810 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.031+0000 7fe1537fe700 4 set_mon_vals no callback set 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe163d2c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe163d2c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe163d2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c08cde0 con 0x7fe148004810 2024-09-18T10:13:26.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe14c019460 con 0x7fe148004810 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe151ffb700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 0x7fe140037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(110..110 src has 40..110) v4 ==== 6242+0+0 (secure 0 0 0) 0x7fe14c048770 con 0x7fe148004810 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 0x7fe140037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe1612c7700 10 monclient: get_auth_request con 0x7fe140035aa0 auth_method 0 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.035+0000 7fe1612c7700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 0x7fe140037f50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe158009130 tx=0x7fe158015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:26.051 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.043+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe15c08cde0 con 0x7fe148004810 2024-09-18T10:13:26.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.043+0000 7fe151ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:26.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.043+0000 7fe151ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:26.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.219+0000 7fe163d2c700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:26.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.223+0000 7fe163d2c700 10 monclient: _send_command 2 [{"prefix": "osd create"}] 2024-09-18T10:13:26.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.223+0000 7fe163d2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:26.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.223+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fe15c063ed0 con 0x7fe148004810 2024-09-18T10:13:26.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.979+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd create"}]=0 v111) v1 ==== 58+0+3 (secure 0 0 0) 0x7fe14c018a10 con 0x7fe148004810 2024-09-18T10:13:26.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.979+0000 7fe151ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create"}] 2024-09-18T10:13:26.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:26.979+0000 7fe151ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:27.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.099+0000 7fe163d2c700 10 start_mon_command cmd=[{"prefix": "osd create"}] 2024-09-18T10:13:27.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.099+0000 7fe163d2c700 10 monclient: _send_command 3 [{"prefix": "osd create"}] 2024-09-18T10:13:27.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.099+0000 7fe163d2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:27.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.099+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create"} v 0) v1 -- 0x7fe15c08cd60 con 0x7fe148004810 2024-09-18T10:13:27.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.983+0000 7fe151ffb700 1 -- 172.21.15.150:0/2512608421 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd create"}]=0 v112) v1 ==== 58+0+3 (secure 0 0 0) 0x7fe14c018830 con 0x7fe148004810 2024-09-18T10:13:27.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.983+0000 7fe151ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create"}] 2024-09-18T10:13:27.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.983+0000 7fe151ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:27.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 msgr2=0x7fe140037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:27.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 0x7fe140037f50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe158009130 tx=0x7fe158015000 comp rx=0 tx=0).stop 2024-09-18T10:13:27.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 10 monclient: shutdown 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 msgr2=0x7fe15c0896c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00b7e0 tx=0x7fe14c009310 comp rx=0 tx=0).stop 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 shutdown_connections 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe140035aa0 0x7fe140037f50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe15c08a140 0x7fe15c08a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe15c08bd40 0x7fe15c089c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 --2- 172.21.15.150:0/2512608421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe148004810 0x7fe15c0896c0 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00b7e0 tx=0x7fe14c009310 comp rx=0 tx=0).stop 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 >> 172.21.15.150:0/2512608421 conn(0x7fe15c0a1710 msgr2=0x7fe15c0958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 shutdown_connections 2024-09-18T10:13:27.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:27.991+0000 7fe163d2c700 1 -- 172.21.15.150:0/2512608421 wait complete. 2024-09-18T10:13:28.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ o5=15 2024-09-18T10:13:28.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 13 123 root=default host=foobaz 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.103+0000 7fd69ec06700 10 monclient: get_monmap_and_config 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.103+0000 7fd69ec06700 10 monclient: build_initial_monmap 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.103+0000 7fd69ec06700 10 monclient: monmap: 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:28.109 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:28.107584+0000 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:28.107584+0000 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:28.110 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:28.111 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 Processor -- start 2024-09-18T10:13:28.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 -- start start 2024-09-18T10:13:28.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient: init 2024-09-18T10:13:28.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:28.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T10:13:28.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6980a2960 0x7fd698096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-c con 0x7fd6980a2960 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:28.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 0x7fd698096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-a con 0x7fd69809e4e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6980974f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6980978f0 con 0x7fd6980974f0 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69809c060 con 0x7fd69809e4e0 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69809dc10 con 0x7fd6980a2960 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6980a2960 0x7fd698096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69ec06700 10 monclient(hunting): authenticate will time out at 1090.202739s 2024-09-18T10:13:28.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd697fff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 msgr2=0x7fd698096fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd697fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 0x7fd698096fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6980a2960 0x7fd698096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43142/0 (socket says 172.21.15.150:43142) 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 -- 172.21.15.150:0/3524005337 learned_addr learned my addr 172.21.15.150:0/3524005337 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): get_auth_request con 0x7fd6980a2960 auth_method 0 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient(hunting): handle_auth_done global_id 15232 payload 274 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 -- 172.21.15.150:0/3524005337 >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 msgr2=0x7fd698096fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:28.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 --2- 172.21.15.150:0/3524005337 >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 0x7fd698096fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 -- 172.21.15.150:0/3524005337 >> v1:172.21.15.150:6789/0 conn(0x7fd6980974f0 legacy=0x7fd698135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient: _finish_hunting 0 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 monclient: found mon.noname-c 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.107+0000 7fd69c9a2700 1 -- 172.21.15.150:0/3524005337 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698099930 con 0x7fd6980a2960 2024-09-18T10:13:28.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69c9a2700 1 -- 172.21.15.150:0/3524005337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 msgr2=0x7fd698096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:28.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69c9a2700 1 --2- 172.21.15.150:0/3524005337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd698096a70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd69809f980 tx=0x7fd68400b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eb751194f7f37af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:28.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 1 -- 172.21.15.150:0/3524005337 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684010470 con 0x7fd6980a2960 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: dump: 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:28.120 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 5 monclient: authenticate success, global_id 15232 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 1 -- 172.21.15.150:0/3524005337 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6840105d0 con 0x7fd6980a2960 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 4 set_mon_vals no callback set 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 1 -- 172.21.15.150:0/3524005337 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684010970 con 0x7fd6980a2960 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd6977fe700 10 monclient: dump: 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:28.121 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 10 monclient: shutdown 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 -- 172.21.15.150:0/3524005337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 msgr2=0x7fd698096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 --2- 172.21.15.150:0/3524005337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd698096a70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd69809f980 tx=0x7fd68400b4f0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 -- 172.21.15.150:0/3524005337 shutdown_connections 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 --2- 172.21.15.150:0/3524005337 >> v2:172.21.15.142:3300/0 conn(0x7fd69809e4e0 0x7fd698096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 --2- 172.21.15.150:0/3524005337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd698096a70 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 -- 172.21.15.150:0/3524005337 >> 172.21.15.150:0/3524005337 conn(0x7fd698068360 msgr2=0x7fd6980a1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 -- 172.21.15.150:0/3524005337 shutdown_connections 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 1 -- 172.21.15.150:0/3524005337 wait complete. 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.111+0000 7fd69ec06700 10 monclient: build_initial_monmap 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient: monmap: 2024-09-18T10:13:28.123 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 Processor -- start 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 -- start start 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient: init 2024-09-18T10:13:28.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 0x7fd698139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-a con 0x7fd6980974f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd69809e4e0 0x7fd698139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-c con 0x7fd69809e4e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): picked mon.noname-b con 0x7fd6980a2960 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69811eda0 con 0x7fd6980974f0 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69811e7e0 con 0x7fd6980a2960 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:28.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69811c2c0 con 0x7fd69809e4e0 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69ec06700 10 monclient(hunting): authenticate will time out at 1090.209699s 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43148/0 (socket says 172.21.15.150:43148) 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69c9a2700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 msgr2=0x7fd698139db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 -- 172.21.15.150:0/4276104863 learned_addr learned my addr 172.21.15.150:0/4276104863 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): get_auth_request con 0x7fd6980a2960 auth_method 0 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69c9a2700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 0x7fd698139db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:28.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd69809e4e0 0x7fd698139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): get_auth_request con 0x7fd69809e4e0 auth_method 0 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd697fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient(hunting): handle_auth_done global_id 15235 payload 824 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 -- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd69809e4e0 msgr2=0x7fd698139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd69809e4e0 0x7fd698139460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 -- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 msgr2=0x7fd698139db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:28.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 0x7fd698139db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:28.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient: _finish_hunting 0 2024-09-18T10:13:28.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 monclient: found mon.noname-b 2024-09-18T10:13:28.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:28.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.115+0000 7fd69d1a3700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68400b790 con 0x7fd6980a2960 2024-09-18T10:13:28.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69d1a3700 1 -- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 msgr2=0x7fd69813d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:28.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69d1a3700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd69000bc30 tx=0x7fd690009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:28.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69000cb10 con 0x7fd6980a2960 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: dump: 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:28.131 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69000cc90 con 0x7fd6980a2960 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 5 monclient: authenticate success, global_id 15235 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _renew_subs 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69811e500 con 0x7fd6980a2960 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd690014660 con 0x7fd6980a2960 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd696ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:28.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 10 monclient: dump: 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _renew_subs 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd698139b40 con 0x7fd6980a2960 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69811ebe0 con 0x7fd6980a2960 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd690014a50 con 0x7fd6980a2960 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 0x7fd67c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(112..112 src has 40..112) v4 ==== 6480+0+0 (secure 0 0 0) 0x7fd690048a20 con 0x7fd6980a2960 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69c9a2700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 0x7fd67c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.119+0000 7fd69c9a2700 10 monclient: get_auth_request con 0x7fd67c035ac0 auth_method 0 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.123+0000 7fd69c9a2700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 0x7fd67c037f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd69809f980 tx=0x7fd68401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.123+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.127+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd6900073e0 con 0x7fd6980a2960 2024-09-18T10:13:28.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.127+0000 7fd694ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:28.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.127+0000 7fd694ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:28.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.299+0000 7fd69ec06700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:28.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.299+0000 7fd69ec06700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:28.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.299+0000 7fd69ec06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:28.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.299+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fd698063ed0 con 0x7fd6980a2960 2024-09-18T10:13:29.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.999+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 13 name 'osd.13' weight 123 at location {host=foobaz,root=default} to crush map v113) v1 ==== 220+0+0 (secure 0 0 0) 0x7fd69001b920 con 0x7fd6980a2960 2024-09-18T10:13:29.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.999+0000 7fd694ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:28.999+0000 7fd694ff9700 10 monclient: _finish_command 2 = system:0 add item id 13 name 'osd.13' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-18T10:13:29.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.103+0000 7fd69ec06700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.103+0000 7fd69ec06700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.103+0000 7fd69ec06700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.103+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7fd698071930 con 0x7fd6980a2960 2024-09-18T10:13:29.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd694ff9700 1 -- 172.21.15.150:0/4276104863 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 13 name 'osd.13' weight 123 at location {host=foobaz,root=default}: no change v113) v1 ==== 218+0+0 (secure 0 0 0) 0x7fd69001bd40 con 0x7fd6980a2960 2024-09-18T10:13:29.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd694ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 13, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd694ff9700 10 monclient: _finish_command 3 = system:0 set item id 13 name 'osd.13' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 msgr2=0x7fd67c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 0x7fd67c037f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd69809f980 tx=0x7fd68401a000 comp rx=0 tx=0).stop 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 10 monclient: shutdown 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 msgr2=0x7fd69813d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd69000bc30 tx=0x7fd690009820 comp rx=0 tx=0).stop 2024-09-18T10:13:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 shutdown_connections 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd67c035ac0 0x7fd67c037f70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6980a2960 0x7fd69813d320 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd69809e4e0 0x7fd698139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 --2- 172.21.15.150:0/4276104863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6980974f0 0x7fd698139db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 >> 172.21.15.150:0/4276104863 conn(0x7fd698068360 msgr2=0x7fd69809c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 shutdown_connections 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.107+0000 7fd69ec06700 1 -- 172.21.15.150:0/4276104863 wait complete. 2024-09-18T10:13:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 13 name 'osd.13' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T10:13:29.122 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add 15 123 root=default host=foobaz 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.219+0000 7f48f8cbf700 10 monclient: get_monmap_and_config 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.219+0000 7f48f8cbf700 10 monclient: build_initial_monmap 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.219+0000 7f48f8cbf700 10 monclient: monmap: 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:29.223810+0000 2024-09-18T10:13:29.224 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:29.223810+0000 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:29.225 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 Processor -- start 2024-09-18T10:13:29.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 -- start start 2024-09-18T10:13:29.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient: init 2024-09-18T10:13:29.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48f409acd0 0x7f48f4096b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-c con 0x7f48f409acd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f409c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-b con 0x7f48f409bfd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48f409c8e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:29.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f4095e70 con 0x7f48f409c8e0 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f4096250 con 0x7f48f409acd0 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f4095920 con 0x7f48f409bfd0 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f8cbf700 10 monclient(hunting): authenticate will time out at 1091.317338s 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48f409acd0 0x7f48f4096b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f2d9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f48f409c8e0 0x7f48f4136780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53596/0 (socket says 172.21.15.150:53596) 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f2d9d700 1 -- 172.21.15.150:0/529865814 learned_addr learned my addr 172.21.15.150:0/529865814 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): get_auth_request con 0x7f48f409acd0 auth_method 0 2024-09-18T10:13:29.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f1d9b700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f409c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f1d9b700 10 monclient(hunting): get_auth_request con 0x7f48f409bfd0 auth_method 0 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient(hunting): handle_auth_done global_id 15238 payload 274 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 -- 172.21.15.150:0/529865814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 msgr2=0x7f48f409c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f409c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 -- 172.21.15.150:0/529865814 >> v1:172.21.15.150:6789/0 conn(0x7f48f409c8e0 legacy=0x7f48f4136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 monclient: found mon.noname-c 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.223+0000 7f48f259c700 1 -- 172.21.15.150:0/529865814 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f409b0a0 con 0x7f48f409acd0 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f259c700 1 -- 172.21.15.150:0/529865814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409acd0 msgr2=0x7f48f4096b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:29.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f259c700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409acd0 0x7f48f4096b40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1b4b73558001970 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 1 -- 172.21.15.150:0/529865814 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8009e80 con 0x7f48f409acd0 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: dump: 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.237 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: _finish_auth 0 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 5 monclient: authenticate success, global_id 15238 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 1 -- 172.21.15.150:0/529865814 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e800ba70 con 0x7f48f409acd0 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:29.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 4 set_mon_vals no callback set 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 1 -- 172.21.15.150:0/529865814 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e800bdf0 con 0x7f48f409acd0 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f159a700 10 monclient: dump: 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.239 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: shutdown 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- 172.21.15.150:0/529865814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409acd0 msgr2=0x7f48f4096b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409acd0 0x7f48f4096b40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e80091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- 172.21.15.150:0/529865814 shutdown_connections 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f409c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/529865814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409acd0 0x7f48f4096b40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- 172.21.15.150:0/529865814 >> 172.21.15.150:0/529865814 conn(0x7f48f40a15e0 msgr2=0x7f48f40a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- 172.21.15.150:0/529865814 shutdown_connections 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- 172.21.15.150:0/529865814 wait complete. 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: build_initial_monmap 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: monmap: 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 Processor -- start 2024-09-18T10:13:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- start start 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: init 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 0x7f48f413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-a con 0x7f48f409acd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f413ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-c con 0x7f48f409bfd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): picked mon.noname-b con 0x7f48f409c8e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f408f470 con 0x7f48f409acd0 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f408faa0 con 0x7f48f409c8e0 2024-09-18T10:13:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f408de50 con 0x7f48f409bfd0 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f8cbf700 10 monclient(hunting): authenticate will time out at 1091.323514s 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f259c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 msgr2=0x7f48f413fa70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 0x7f48f413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40354/0 (socket says 172.21.15.150:40354) 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 1 -- 172.21.15.150:0/707761096 learned_addr learned my addr 172.21.15.150:0/707761096 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): get_auth_request con 0x7f48f409c8e0 auth_method 0 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:29.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f1d9b700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.227+0000 7f48f1d9b700 10 monclient(hunting): get_auth_request con 0x7f48f409bfd0 auth_method 0 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 10 monclient(hunting): handle_auth_done global_id 15241 payload 824 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 -- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 msgr2=0x7f48f413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:29.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 -- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 msgr2=0x7f48f413fa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 0x7f48f413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 monclient: found mon.noname-b 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009420 con 0x7f48f409c8e0 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 -- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 msgr2=0x7f48f408bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f2d9d700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00cd20 tx=0x7f48ec005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec00ce20 con 0x7f48f409c8e0 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:29.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: dump: 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:29.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 5 monclient: authenticate success, global_id 15241 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec00ad00 con 0x7f48f409c8e0 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 10 monclient: _renew_subs 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f40979c0 con 0x7f48f409c8e0 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec007340 con 0x7f48f409c8e0 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e3fff700 4 set_mon_vals no callback set 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48e27fc700 10 monclient: dump: 2024-09-18T10:13:29.247 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 10 monclient: _renew_subs 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.231+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f4090700 con 0x7f48f409c8e0 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f8cbf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:29.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f8cbf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:29.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f8cbf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f409cd40 con 0x7f48f409c8e0 2024-09-18T10:13:29.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f48ec0076f0 con 0x7f48f409c8e0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48e27fc700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 0x7f48d4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(113..113 src has 40..113) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f48ec048f50 con 0x7f48f409c8e0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f259c700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 0x7f48d4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.235+0000 7f48f259c700 10 monclient: get_auth_request con 0x7f48d4035a70 auth_method 0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.239+0000 7f48f259c700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 0x7f48d4037f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.243+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f48ec0079a0 con 0x7f48f409c8e0 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.243+0000 7f48e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:29.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.243+0000 7f48e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.407+0000 7f48f8cbf700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.407+0000 7f48f8cbf700 10 monclient: _send_command 2 [{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.407+0000 7f48f8cbf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:29.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:29.407+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f48f4063ed0 con 0x7f48f409c8e0 2024-09-18T10:13:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.003+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 add item id 15 name 'osd.15' weight 123 at location {host=foobaz,root=default} to crush map v114) v1 ==== 220+0+0 (secure 0 0 0) 0x7f48ec010300 con 0x7f48f409c8e0 2024-09-18T10:13:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.003+0000 7f48e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:30.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.003+0000 7f48e27fc700 10 monclient: _finish_command 2 = system:0 add item id 15 name 'osd.15' weight 123 at location {host=foobaz,root=default} to crush map 2024-09-18T10:13:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48f8cbf700 10 start_mon_command cmd=[{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:30.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48f8cbf700 10 monclient: _send_command 3 [{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:30.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48f8cbf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]} v 0) v1 -- 0x7f48f4067b10 con 0x7f48f409c8e0 2024-09-18T10:13:30.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48e27fc700 1 -- 172.21.15.150:0/707761096 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}]=0 set item id 15 name 'osd.15' weight 123 at location {host=foobaz,root=default}: no change v114) v1 ==== 218+0+0 (secure 0 0 0) 0x7f48ec0290c0 con 0x7f48f409c8e0 2024-09-18T10:13:30.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48e27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add", "id": 15, "weight": 123.0, "args": ["root=default", "host=foobaz"]}] 2024-09-18T10:13:30.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.111+0000 7f48e27fc700 10 monclient: _finish_command 3 = system:0 set item id 15 name 'osd.15' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 msgr2=0x7f48d4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 0x7f48d4037f20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f48e80098e0 tx=0x7f48e8016000 comp rx=0 tx=0).stop 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 10 monclient: shutdown 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 msgr2=0x7f48f408bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00cd20 tx=0x7f48ec005400 comp rx=0 tx=0).stop 2024-09-18T10:13:30.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 shutdown_connections 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48d4035a70 0x7f48d4037f20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48f409c8e0 0x7f48f408bf20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48f409bfd0 0x7f48f413ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 --2- 172.21.15.150:0/707761096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48f409acd0 0x7f48f413fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 >> 172.21.15.150:0/707761096 conn(0x7f48f40a15e0 msgr2=0x7f48f406c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 shutdown_connections 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.115+0000 7f48f8cbf700 1 -- 172.21.15.150:0/707761096 wait complete. 2024-09-18T10:13:30.119 INFO:tasks.workunit.client.0.smithi150.stderr:set item id 15 name 'osd.15' weight 123 at location {host=foobaz,root=default}: no change 2024-09-18T10:13:30.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:30.128 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.13 2024-09-18T10:13:30.128 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 123 2024-09-18T10:13:30.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: get_monmap_and_config 2024-09-18T10:13:30.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: build_initial_monmap 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: monmap: 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:30.222218+0000 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:30.222218+0000 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:30.223 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:30.224 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:30.224 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:30.224 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:30.224 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 Processor -- start 2024-09-18T10:13:30.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 -- start start 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: init 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3eb809c2e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 0x7f3eb8133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-b con 0x7f3eb81316e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3eb8134000 0x7f3eb81363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-c con 0x7f3eb8134000 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb8095de0 con 0x7f3eb809c2e0 2024-09-18T10:13:30.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb8098580 con 0x7f3eb8134000 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb809af60 con 0x7f3eb81316e0 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.219+0000 7f3ec0523700 10 monclient(hunting): authenticate will time out at 1092.315943s 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebe2bf700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f3eb809c2e0 legacy=0x7f3eb80956d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3eb8134000 0x7f3eb81363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3eb8134000 0x7f3eb81363e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40368/0 (socket says 172.21.15.150:40368) 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 0x7f3eb8133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 -- 172.21.15.150:0/2311232586 learned_addr learned my addr 172.21.15.150:0/2311232586 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): get_auth_request con 0x7f3eb8134000 auth_method 0 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): get_auth_request con 0x7f3eb81316e0 auth_method 0 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:30.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient(hunting): handle_auth_done global_id 15244 payload 274 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 -- 172.21.15.150:0/2311232586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 msgr2=0x7f3eb8133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 0x7f3eb8133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 -- 172.21.15.150:0/2311232586 >> v1:172.21.15.142:6789/0 conn(0x7f3eb809c2e0 legacy=0x7f3eb80956d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient: _finish_hunting 0 2024-09-18T10:13:30.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 monclient: found mon.noname-c 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 -- 172.21.15.150:0/2311232586 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb809b520 con 0x7f3eb8134000 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebdabe700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 0x7f3eb8133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 -- 172.21.15.150:0/2311232586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb8134000 msgr2=0x7f3eb81363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebeac0700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb8134000 0x7f3eb81363e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009f20 tx=0x7f3eb0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24d57495e05cdccb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.223+0000 7f3ebd2bd700 1 -- 172.21.15.150:0/2311232586 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000e070 con 0x7f3eb8134000 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:30.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: dump: 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.235 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: _finish_auth 0 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 5 monclient: authenticate success, global_id 15244 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 1 -- 172.21.15.150:0/2311232586 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000bbe0 con 0x7f3eb8134000 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:30.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 1 -- 172.21.15.150:0/2311232586 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00092e0 con 0x7f3eb8134000 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 4 set_mon_vals no callback set 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ebd2bd700 10 monclient: dump: 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.237 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: shutdown 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- 172.21.15.150:0/2311232586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb8134000 msgr2=0x7f3eb81363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb8134000 0x7f3eb81363e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0009f20 tx=0x7f3eb0009840 comp rx=0 tx=0).stop 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- 172.21.15.150:0/2311232586 shutdown_connections 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb8134000 0x7f3eb81363e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2311232586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb81316e0 0x7f3eb8133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- 172.21.15.150:0/2311232586 >> 172.21.15.150:0/2311232586 conn(0x7f3eb80a1710 msgr2=0x7f3eb80a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:30.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- 172.21.15.150:0/2311232586 shutdown_connections 2024-09-18T10:13:30.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- 172.21.15.150:0/2311232586 wait complete. 2024-09-18T10:13:30.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: build_initial_monmap 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: monmap: 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:30.243 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 Processor -- start 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 -- start start 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: init 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:30.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.227+0000 7f3ec0523700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb809c2e0 0x7f3eb80940c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-c con 0x7f3eb809c2e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-b con 0x7f3eb81316e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 0x7f3eb8094b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): picked mon.noname-a con 0x7f3eb8134000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb81291e0 con 0x7f3eb8134000 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80951d0 con 0x7f3eb81316e0 2024-09-18T10:13:30.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb8128670 con 0x7f3eb809c2e0 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient(hunting): authenticate will time out at 1092.324194s 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40370/0 (socket says 172.21.15.150:40370) 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebeac0700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 msgr2=0x7f3eb8094b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 -- 172.21.15.150:0/2745682217 learned_addr learned my addr 172.21.15.150:0/2745682217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): get_auth_request con 0x7f3eb81316e0 auth_method 0 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebeac0700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 0x7f3eb8094b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebe2bf700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb809c2e0 0x7f3eb80940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebe2bf700 10 monclient(hunting): get_auth_request con 0x7f3eb809c2e0 auth_method 0 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebe2bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebe2bf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebe2bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient(hunting): handle_auth_done global_id 15247 payload 824 2024-09-18T10:13:30.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 -- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb809c2e0 msgr2=0x7f3eb80940c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb809c2e0 0x7f3eb80940c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 -- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 msgr2=0x7f3eb8094b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 0x7f3eb8094b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 monclient: found mon.noname-b 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009420 con 0x7f3eb81316e0 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 -- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 msgr2=0x7f3eb8094600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebdabe700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800ec80 tx=0x7f3ea80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800cbe0 con 0x7f3eb81316e0 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: dump: 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.249 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800cd40 con 0x7f3eb81316e0 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80136a0 con 0x7f3eb81316e0 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 5 monclient: authenticate success, global_id 15247 2024-09-18T10:13:30.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ebcabc700 4 set_mon_vals no callback set 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3eae7fc700 10 monclient: dump: 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.251 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient: _renew_subs 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb0009ae0 con 0x7f3eb81316e0 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient: _renew_subs 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.231+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb806edf0 con 0x7f3eb81316e0 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3ea8013a50 con 0x7f3eb81316e0 2024-09-18T10:13:30.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3eae7fc700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 0x7f3e9c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(114..114 src has 40..114) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f3ea8049330 con 0x7f3eb81316e0 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ebe2bf700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 0x7f3e9c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ec0523700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ec0523700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ec0523700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb8063ed0 con 0x7f3eb81316e0 2024-09-18T10:13:30.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.235+0000 7f3ebe2bf700 10 monclient: get_auth_request con 0x7f3e9c035a70 auth_method 0 2024-09-18T10:13:30.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.239+0000 7f3ebe2bf700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 0x7f3e9c037f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4003980 tx=0x7f3eb4004e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.243+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3ea80123d0 con 0x7f3eb81316e0 2024-09-18T10:13:30.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.243+0000 7f3eae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.243+0000 7f3eae7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:30.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3ec0523700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:30.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3ec0523700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3ec0523700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3eb8096330 con 0x7f3eb81316e0 2024-09-18T10:13:30.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f3ea8024e20 con 0x7f3eb81316e0 2024-09-18T10:13:30.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3eae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.407+0000 7f3eae7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:30.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:30.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f3eb80008d0 con 0x7f3eb81316e0 2024-09-18T10:13:30.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3eae7fc700 1 -- 172.21.15.150:0/2745682217 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f3ea801b820 con 0x7f3eb81316e0 2024-09-18T10:13:30.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3eae7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3eae7fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 msgr2=0x7f3e9c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 0x7f3e9c037f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4003980 tx=0x7f3eb4004e80 comp rx=0 tx=0).stop 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 10 monclient: shutdown 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 msgr2=0x7f3eb8094600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800ec80 tx=0x7f3ea80091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 shutdown_connections 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3e9c035a70 0x7f3e9c037f20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3eb8134000 0x7f3eb8094b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3eb81316e0 0x7f3eb8094600 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 --2- 172.21.15.150:0/2745682217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3eb809c2e0 0x7f3eb80940c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 >> 172.21.15.150:0/2745682217 conn(0x7f3eb80a1710 msgr2=0x7f3eb809cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:30.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 shutdown_connections 2024-09-18T10:13:30.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.507+0000 7f3ec0523700 1 -- 172.21.15.150:0/2745682217 wait complete. 2024-09-18T10:13:30.520 INFO:tasks.workunit.client.0.smithi150.stdout: 13 123.00000 osd.13 down 1.00000 1.00000 2024-09-18T10:13:30.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:30.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.15 2024-09-18T10:13:30.522 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 123 2024-09-18T10:13:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: get_monmap_and_config 2024-09-18T10:13:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: build_initial_monmap 2024-09-18T10:13:30.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: monmap: 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:30.605769+0000 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:30.605769+0000 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:30.607 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:30.608 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:30.608 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 Processor -- start 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 -- start start 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: init 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f118809cf10 0x7f11880a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-a con 0x7f118809cf10 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11880a0750 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11880a0b50 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118809e6e0 con 0x7f11880a0b50 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118809d2e0 con 0x7f11880a0750 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118809e2e0 con 0x7f118809cf10 2024-09-18T10:13:30.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.603+0000 7f118fb02700 10 monclient(hunting): authenticate will time out at 1092.699836s 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118e09f700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f11880a0b50 legacy=0x7f118813a0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d89e700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f118809cf10 msgr2=0x7f11880a0210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d89e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f118809cf10 0x7f11880a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f11880a0750 0x7f1188136990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53602/0 (socket says 172.21.15.150:53602) 2024-09-18T10:13:30.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 learned_addr learned my addr 172.21.15.150:0/2319691588 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 >> v1:172.21.15.150:6789/0 conn(0x7f11880a0750 legacy=0x7f1188136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099626299 0 0) 0x7f118809d2e0 con 0x7f11880a0750 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): my global_id is 15250 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003570 con 0x7f11880a0750 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3814817849 0 0) 0x7f1174003570 con 0x7f11880a0750 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 >> v2:172.21.15.142:3300/0 conn(0x7f118809cf10 msgr2=0x7f11880a0210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 --2- 172.21.15.150:0/2319691588 >> v2:172.21.15.142:3300/0 conn(0x7f118809cf10 0x7f11880a0210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 >> v1:172.21.15.142:6789/0 conn(0x7f11880a0b50 legacy=0x7f118813a0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:30.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11880a1f90 con 0x7f11880a0750 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: _finish_auth 0 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1184001860 con 0x7f11880a0750 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118fb02700 5 monclient: authenticate success, global_id 15250 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118fb02700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: dump: 2024-09-18T10:13:30.618 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:30.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 >> v1:172.21.15.150:6789/0 conn(0x7f11880a0750 legacy=0x7f1188136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1174003b30 0x7f1174003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): picked mon.c con 0x7f1174003b30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1174004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): picked mon.b con 0x7f1174004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1174005410 0x7f117400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): picked mon.a con 0x7f1174005410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118809e2e0 con 0x7f1174005410 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117400c3a0 con 0x7f1174004810 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117400c560 con 0x7f1174003b30 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118c89c700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118fb02700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1174004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): get_auth_request con 0x7f1174004810 auth_method 0 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118e09f700 1 -- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1174005410 msgr2=0x7f117400bac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118e09f700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1174005410 0x7f117400bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d89e700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1174003b30 0x7f1174003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.607+0000 7f118d09d700 10 monclient(hunting): handle_auth_done global_id 15250 payload 293 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1174003b30 msgr2=0x7f1174003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1174003b30 0x7f1174003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1174005410 msgr2=0x7f117400bac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1174005410 0x7f117400bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:30.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 monclient: found mon.b 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174003570 con 0x7f1174004810 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 10 monclient: _finish_auth 0 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118e09f700 1 -- 172.21.15.150:0/2319691588 reap_dead start 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:30.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d89e700 10 monclient: get_auth_request con 0x7f1174003b30 auth_method 0 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d89e700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1174003b30 0x7f1174003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 -- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 msgr2=0x7f1174004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118d09d700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1174004c80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f11840014d0 tx=0x7f118806a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54defff72572c00d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118400ce60 con 0x7f1174004810 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: dump: 2024-09-18T10:13:30.624 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184005920 con 0x7f1174004810 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 4 set_mon_vals no callback set 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 1 -- 172.21.15.150:0/2319691588 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184005ce0 con 0x7f1174004810 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118c89c700 10 monclient: dump: 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.626 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 10 monclient: shutdown 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 -- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 msgr2=0x7f1174004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1174004c80 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f11840014d0 tx=0x7f118806a230 comp rx=0 tx=0).stop 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 -- 172.21.15.150:0/2319691588 shutdown_connections 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 --2- 172.21.15.150:0/2319691588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1174004c80 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 -- 172.21.15.150:0/2319691588 >> 172.21.15.150:0/2319691588 conn(0x7f11880a5820 msgr2=0x7f11880a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 -- 172.21.15.150:0/2319691588 shutdown_connections 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.611+0000 7f118fb02700 1 -- 172.21.15.150:0/2319691588 wait complete. 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient: build_initial_monmap 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient: monmap: 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:30.628 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:30.629 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 Processor -- start 2024-09-18T10:13:30.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 -- start start 2024-09-18T10:13:30.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient: init 2024-09-18T10:13:30.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-b con 0x7f1174004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1188141fa0 0x7f1188142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-c con 0x7f1188141fa0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 0x7f118808e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): picked mon.noname-a con 0x7f118808df90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880a4fc0 con 0x7f118808df90 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880928e0 con 0x7f1174004810 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11881428b0 con 0x7f1188141fa0 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118fb02700 10 monclient(hunting): authenticate will time out at 1092.709307s 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118e09f700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 msgr2=0x7f118808e410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40402/0 (socket says 172.21.15.150:40402) 2024-09-18T10:13:30.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 -- 172.21.15.150:0/3171902726 learned_addr learned my addr 172.21.15.150:0/3171902726 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): get_auth_request con 0x7f1174004810 auth_method 0 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118e09f700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 0x7f118808e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d09d700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1188141fa0 0x7f1188142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d09d700 10 monclient(hunting): get_auth_request con 0x7f1188141fa0 auth_method 0 2024-09-18T10:13:30.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d09d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d09d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d09d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient(hunting): handle_auth_done global_id 15253 payload 824 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 -- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1188141fa0 msgr2=0x7f1188142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1188141fa0 0x7f1188142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 -- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 msgr2=0x7f118808e410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 0x7f118808e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient: _finish_hunting 0 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 monclient: found mon.noname-b 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.615+0000 7f118d89e700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1188090080 con 0x7f1174004810 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118d89e700 1 -- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 msgr2=0x7f1188141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118d89e700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f117800c830 tx=0x7f1178008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178003d50 con 0x7f1174004810 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: dump: 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.638 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11780050f0 con 0x7f1174004810 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 5 monclient: authenticate success, global_id 15253 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _renew_subs 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11780054b0 con 0x7f1174004810 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118808fbe0 con 0x7f1174004810 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117ffff700 4 set_mon_vals no callback set 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 10 monclient: dump: 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.640 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _renew_subs 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1188091910 con 0x7f1174004810 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f117801b460 con 0x7f1174004810 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 0x7f116c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(114..114 src has 40..114) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f1178048950 con 0x7f1174004810 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118d09d700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 0x7f116c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.619+0000 7f118d09d700 10 monclient: get_auth_request con 0x7f116c035a70 auth_method 0 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.623+0000 7f118d09d700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 0x7f116c037f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f118808f770 tx=0x7f1184012000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.631+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.631+0000 7f117dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:30.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.631+0000 7f117dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:30.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.795+0000 7f118fb02700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:30.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.795+0000 7f118fb02700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.795+0000 7f118fb02700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.795+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.799+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.799+0000 7f117dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.799+0000 7f117dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:30.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f117dffb700 1 -- 172.21.15.150:0/3171902726 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v114) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f1188090800 con 0x7f1174004810 2024-09-18T10:13:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f117dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:30.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f117dffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 msgr2=0x7f116c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 0x7f116c037f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f118808f770 tx=0x7f1184012000 comp rx=0 tx=0).stop 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 10 monclient: shutdown 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 msgr2=0x7f1188141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f117800c830 tx=0x7f1178008040 comp rx=0 tx=0).stop 2024-09-18T10:13:30.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 shutdown_connections 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f116c035a70 0x7f116c037f20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f118808df90 0x7f118808e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1188141fa0 0x7f1188142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.899+0000 7f118fb02700 1 --2- 172.21.15.150:0/3171902726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1174004810 0x7f1188141a60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.903+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 >> 172.21.15.150:0/3171902726 conn(0x7f11880a5820 msgr2=0x7f11880a3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.903+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 shutdown_connections 2024-09-18T10:13:30.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:30.903+0000 7f118fb02700 1 -- 172.21.15.150:0/3171902726 wait complete. 2024-09-18T10:13:30.915 INFO:tasks.workunit.client.0.smithi150.stdout: 15 123.00000 osd.15 down 1.00000 1.00000 2024-09-18T10:13:30.915 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush reweight-subtree foobaz 155 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: get_monmap_and_config 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: build_initial_monmap 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: monmap: 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:31.017 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:31.016864+0000 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:31.016864+0000 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:31.018 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 Processor -- start 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 -- start start 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: init 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b2c09e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:31.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 0x7f3b2c137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c0a09a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 0x7f3b2c139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c0a0da0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c09e220 con 0x7f3b2c09e6d0 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c09ee30 con 0x7f3b2c0a09a0 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c09eaa0 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b30bf7700 10 monclient(hunting): authenticate will time out at 1093.110789s 2024-09-18T10:13:31.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b29d9b700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 msgr2=0x7f3b2c137a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b29d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 0x7f3b2c137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2a59c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3b2c09e6d0 0x7f3b2c0a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53614/0 (socket says 172.21.15.150:53614) 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2a59c700 1 -- 172.21.15.150:0/3130117696 learned_addr learned my addr 172.21.15.150:0/3130117696 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 0x7f3b2c139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a0da0 auth_method 0 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_done global_id 44252 payload 274 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/3130117696 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 msgr2=0x7f3b2c137a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:31.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/3130117696 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 0x7f3b2c137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/3130117696 >> v1:172.21.15.150:6789/0 conn(0x7f3b2c09e6d0 legacy=0x7f3b2c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 monclient: found mon.noname-b 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:31.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.015+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/3130117696 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c09d930 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 msgr2=0x7f3b2c139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 0x7f3b2c139f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009e60 tx=0x7f3b24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153f9d54b1e65105 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 1 -- 172.21.15.150:0/3130117696 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400e070 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: dump: 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.029 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: _finish_auth 0 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b30bf7700 5 monclient: authenticate success, global_id 44252 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 1 -- 172.21.15.150:0/3130117696 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24012430 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b30bf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b30bf7700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b30bf7700 4 set_mon_vals no callback set 2024-09-18T10:13:31.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 1 -- 172.21.15.150:0/3130117696 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240127d0 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.019+0000 7f3b2959a700 10 monclient: dump: 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:31.031 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: shutdown 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 msgr2=0x7f3b2c139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 0x7f3b2c139f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009e60 tx=0x7f3b24009780 comp rx=0 tx=0).stop 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- 172.21.15.150:0/3130117696 shutdown_connections 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/3130117696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a0da0 0x7f3b2c139f70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/3130117696 >> v2:172.21.15.142:3300/0 conn(0x7f3b2c0a09a0 0x7f3b2c137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- 172.21.15.150:0/3130117696 >> 172.21.15.150:0/3130117696 conn(0x7f3b2c0a58a0 msgr2=0x7f3b2c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- 172.21.15.150:0/3130117696 shutdown_connections 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- 172.21.15.150:0/3130117696 wait complete. 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: build_initial_monmap 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: monmap: 2024-09-18T10:13:31.032 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 Processor -- start 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- start start 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: init 2024-09-18T10:13:31.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 0x7f3b2c08df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c09e6d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a09a0 0x7f3b2c08e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c0a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c0a0da0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c09bf00 con 0x7f3b2c09e6d0 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c092550 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c091630 con 0x7f3b2c0a09a0 2024-09-18T10:13:31.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b30bf7700 10 monclient(hunting): authenticate will time out at 1093.118925s 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2a59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 msgr2=0x7f3b2c08df10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 0x7f3b2c08df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40418/0 (socket says 172.21.15.150:40418) 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a09a0 0x7f3b2c08e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/4043274760 learned_addr learned my addr 172.21.15.150:0/4043274760 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a0da0 auth_method 0 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:31.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request con 0x7f3b2c0a09a0 auth_method 0 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient(hunting): handle_auth_done global_id 15256 payload 824 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a09a0 msgr2=0x7f3b2c08e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:31.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a09a0 0x7f3b2c08e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 msgr2=0x7f3b2c08df10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 0x7f3b2c08df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 monclient: found mon.noname-b 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:31.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.023+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24009a20 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b2ad9d700 1 -- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 msgr2=0x7f3b2c08e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b2ad9d700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b24018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24026430 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:31.040 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24021070 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 5 monclient: authenticate success, global_id 15256 2024-09-18T10:13:31.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 10 monclient: _renew_subs 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c091350 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b240267e0 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 10 monclient: dump: 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:31.042 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b28d99700 4 set_mon_vals no callback set 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 10 monclient: _renew_subs 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c08f820 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.027+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3b24023030 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b1a7fc700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 0x7f3b0c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:31.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b30bf7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b30bf7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b30bf7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(114..114 src has 58..114) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f3b2404ae60 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c091e10 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b2a59c700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 0x7f3b0c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b2a59c700 10 monclient: get_auth_request con 0x7f3b0c035a70 auth_method 0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.031+0000 7f3b2a59c700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 0x7f3b0c037f20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c004860 tx=0x7f3b1c007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.035+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3b2401bb80 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.035+0000 7f3b1a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:31.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.035+0000 7f3b1a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:31.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.191+0000 7f3b30bf7700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:31.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.191+0000 7f3b30bf7700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:31.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.191+0000 7f3b30bf7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:31.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:31.191+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f3b2c09dbf0 con 0x7f3b2c0a0da0 2024-09-18T10:13:31.919 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:13:31.919 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:13:31.919 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:13:31.920 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:13:31.922 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:13:31.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:13:31.923 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:32.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.019+0000 7f8928e03700 10 monclient: get_monmap_and_config 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.019+0000 7f8928e03700 10 monclient: build_initial_monmap 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.019+0000 7f8928e03700 10 monclient: monmap: 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:32.027208+0000 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:32.027208+0000 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:32.029 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:32.030 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:32.030 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:32.030 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 Processor -- start 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 -- start start 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient: init 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-a con 0x7f892406eec0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f892406f810 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f89241375b0 0x7f8924139990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-c con 0x7f89241375b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8924099760 con 0x7f892406f810 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:32.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8924098b90 con 0x7f892406eec0 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f892409d5e0 con 0x7f89241375b0 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8928e03700 10 monclient(hunting): authenticate will time out at 1100.953377s 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44960/0 (socket says 172.21.15.142:44960) 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 -- 172.21.15.142:0/4179800216 learned_addr learned my addr 172.21.15.142:0/4179800216 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 10 monclient(hunting): get_auth_request con 0x7f892406eec0 auth_method 0 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:32.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 --2- 172.21.15.142:0/4179800216 >> v2:172.21.15.150:3300/0 conn(0x7f89241375b0 0x7f8924139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 -- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 msgr2=0x7f892409c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): get_auth_request con 0x7f89241375b0 auth_method 0 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 -- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 msgr2=0x7f892409c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 --2- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f89237fe700 1 --2- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient(hunting): handle_auth_done global_id 15259 payload 274 2024-09-18T10:13:32.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 -- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 msgr2=0x7f892409c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 --2- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 -- 172.21.15.142:0/4179800216 >> v1:172.21.15.142:6789/0 conn(0x7f892406f810 legacy=0x7f892409c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 monclient: found mon.noname-c 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:32.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.023+0000 7f8923fff700 1 -- 172.21.15.142:0/4179800216 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892406f290 con 0x7f89241375b0 2024-09-18T10:13:32.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.027+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v115) v1 ==== 168+0+0 (secure 0 0 0) 0x7f3b2401b3a0 con 0x7f3b2c0a0da0 2024-09-18T10:13:32.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.027+0000 7f3b1a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:32.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.027+0000 7f3b1a7fc700 10 monclient: _finish_command 2 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8923fff700 1 -- 172.21.15.142:0/4179800216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 msgr2=0x7f8924139990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8923fff700 1 --2- 172.21.15.142:0/4179800216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924139990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f892406fc50 tx=0x7f891c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffe22d3cb4afcf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 1 -- 172.21.15.142:0/4179800216 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c00e070 con 0x7f89241375b0 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: dump: 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 5 monclient: authenticate success, global_id 15259 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 1 -- 172.21.15.142:0/4179800216 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00be40 con 0x7f89241375b0 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 1 -- 172.21.15.142:0/4179800216 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c012660 con 0x7f89241375b0 2024-09-18T10:13:32.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 4 set_mon_vals no callback set 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f89227fc700 10 monclient: dump: 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:32.045 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 10 monclient: shutdown 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 -- 172.21.15.142:0/4179800216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 msgr2=0x7f8924139990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 --2- 172.21.15.142:0/4179800216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924139990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f892406fc50 tx=0x7f891c009780 comp rx=0 tx=0).stop 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 -- 172.21.15.142:0/4179800216 shutdown_connections 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 --2- 172.21.15.142:0/4179800216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924139990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 --2- 172.21.15.142:0/4179800216 >> v2:172.21.15.142:3300/0 conn(0x7f892406eec0 0x7f892409c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 -- 172.21.15.142:0/4179800216 >> 172.21.15.142:0/4179800216 conn(0x7f89240a13f0 msgr2=0x7f89240a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 -- 172.21.15.142:0/4179800216 shutdown_connections 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 1 -- 172.21.15.142:0/4179800216 wait complete. 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 10 monclient: build_initial_monmap 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.027+0000 7f8928e03700 10 monclient: monmap: 2024-09-18T10:13:32.046 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 Processor -- start 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 -- start start 2024-09-18T10:13:32.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient: init 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-a con 0x7f892406eec0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 0x7f8924129530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-c con 0x7f892406f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924129a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): picked mon.noname-b con 0x7f89241375b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892412d530 con 0x7f892406eec0 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892409d3d0 con 0x7f89241375b0 2024-09-18T10:13:32.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8924097810 con 0x7f892406f810 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8928e03700 10 monclient(hunting): authenticate will time out at 1100.961143s 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 0x7f8924129530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44970/0 (socket says 172.21.15.142:44970) 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): get_auth_request con 0x7f892406f810 auth_method 0 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 -- 172.21.15.142:0/1834564385 learned_addr learned my addr 172.21.15.142:0/1834564385 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:32.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 10 monclient(hunting): get_auth_request con 0x7f892406eec0 auth_method 0 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924129a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 msgr2=0x7f8924128ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 msgr2=0x7f8924128ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): get_auth_request con 0x7f89241375b0 auth_method 0 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:32.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f89237fe700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8923fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient(hunting): handle_auth_done global_id 44258 payload 824 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 msgr2=0x7f8924129a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924129a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 msgr2=0x7f8924128ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 monclient: found mon.noname-c 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:32.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.031+0000 7f8922ffd700 1 -- 172.21.15.142:0/1834564385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c009a20 con 0x7f892406f810 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8922ffd700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 msgr2=0x7f8924129530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8922ffd700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 0x7f8924129530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f891800c510 tx=0x7f8918007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89180117a0 con 0x7f892406f810 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: dump: 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:32.052 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 5 monclient: authenticate success, global_id 44258 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8918011920 con 0x7f892406f810 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918011ce0 con 0x7f892406f810 2024-09-18T10:13:32.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _renew_subs 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892412a8c0 con 0x7f892406f810 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8921ffb700 4 set_mon_vals no callback set 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 10 monclient: dump: 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:32.054 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _renew_subs 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89241322e0 con 0x7f892406f810 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8918012630 con 0x7f892406f810 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8924072a80 con 0x7f892406f810 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 0x7f8910037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:32.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(115..115 src has 58..115) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f89180487a0 con 0x7f892406f810 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f89237fe700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 0x7f8910037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.035+0000 7f89237fe700 10 monclient: get_auth_request con 0x7f8910035ac0 auth_method 0 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.039+0000 7f89237fe700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 0x7f8910037f70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8914009240 tx=0x7f8914008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.043+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8918011e40 con 0x7f892406f810 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.043+0000 7f890b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:32.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.043+0000 7f890b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:32.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.131+0000 7f3b30bf7700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:32.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.131+0000 7f3b30bf7700 10 monclient: _send_command 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:32.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.131+0000 7f3b30bf7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:32.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:32.131+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0} v 0) v1 -- 0x7f3b2c09dbf0 con 0x7f3b2c0a0da0 2024-09-18T10:13:32.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.271+0000 7f8928e03700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.271+0000 7f8928e03700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.271+0000 7f8928e03700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.271+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f892412c8a0 con 0x7f892406f810 2024-09-18T10:13:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f890b7fe700 1 -- 172.21.15.142:0/1834564385 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f89180102e0 con 0x7f892406f810 2024-09-18T10:13:32.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f890b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:32.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f890b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:32.281 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":52,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:32.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 msgr2=0x7f8910037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:32.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 0x7f8910037f70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8914009240 tx=0x7f8914008040 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 10 monclient: shutdown 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 msgr2=0x7f8924129530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 0x7f8924129530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f891800c510 tx=0x7f8918007ea0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 shutdown_connections 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8910035ac0 0x7f8910037f70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89241375b0 0x7f8924129a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f892406f810 0x7f8924129530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 --2- 172.21.15.142:0/1834564385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f892406eec0 0x7f8924128ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 >> 172.21.15.142:0/1834564385 conn(0x7f89240a13f0 msgr2=0x7f8924099910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:32.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 shutdown_connections 2024-09-18T10:13:32.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:32.275+0000 7f8928e03700 1 -- 172.21.15.142:0/1834564385 wait complete. 2024-09-18T10:13:33.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.551+0000 7f3b1a7fc700 1 -- 172.21.15.150:0/4043274760 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}]=0 reweighted subtree id -10 name 'foobaz' to 155 in crush map v116) v1 ==== 168+0+0 (secure 0 0 0) 0x7f3b2401b580 con 0x7f3b2c0a0da0 2024-09-18T10:13:33.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.551+0000 7f3b1a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 155.0}] 2024-09-18T10:13:33.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.551+0000 7f3b1a7fc700 10 monclient: _finish_command 3 = system:0 reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T10:13:33.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 msgr2=0x7f3b0c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 0x7f3b0c037f20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c004860 tx=0x7f3b1c007c10 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 10 monclient: shutdown 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 msgr2=0x7f3b2c08e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b24018000 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 shutdown_connections 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3b0c035a70 0x7f3b0c037f20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b2c0a0da0 0x7f3b2c08e990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b2c0a09a0 0x7f3b2c08e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 --2- 172.21.15.150:0/4043274760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b2c09e6d0 0x7f3b2c08df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 >> 172.21.15.150:0/4043274760 conn(0x7f3b2c0a58a0 msgr2=0x7f3b2c09a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 shutdown_connections 2024-09-18T10:13:33.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.555+0000 7f3b30bf7700 1 -- 172.21.15.150:0/4043274760 wait complete. 2024-09-18T10:13:33.560 INFO:tasks.workunit.client.0.smithi150.stderr:reweighted subtree id -10 name 'foobaz' to 155 in crush map 2024-09-18T10:13:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush reweight-subtree foobaz 123456 2024-09-18T10:13:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:13:33.573 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush reweight-subtree foobaz 123456 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: get_monmap_and_config 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: build_initial_monmap 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: monmap: 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:33.680852+0000 2024-09-18T10:13:33.681 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:33.680852+0000 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:33.682 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 Processor -- start 2024-09-18T10:13:33.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 -- start start 2024-09-18T10:13:33.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: init 2024-09-18T10:13:33.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:33.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:13:33.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f322c06d890 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:33.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-a con 0x7f322c12ef20 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 0x7f322c136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-b con 0x7f322c12f830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c06ea00 con 0x7f322c06d890 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c06e720 con 0x7f322c12ef20 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c06e440 con 0x7f322c12f830 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.679+0000 7f32332f6700 10 monclient(hunting): authenticate will time out at 1095.775806s 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 0x7f322c136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231092700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f322c06d890 0x7f322c096c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33024/0 (socket says 172.21.15.150:33024) 2024-09-18T10:13:33.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231092700 1 -- 172.21.15.150:0/4254010707 learned_addr learned my addr 172.21.15.150:0/4254010707 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 10 monclient(hunting): get_auth_request con 0x7f322c12ef20 auth_method 0 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): get_auth_request con 0x7f322c12f830 auth_method 0 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 1 -- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 msgr2=0x7f322c12f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 1 -- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 msgr2=0x7f322c12f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 1 --2- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:33.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3230891700 1 --2- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient(hunting): handle_auth_done global_id 44261 payload 274 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 -- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 msgr2=0x7f322c12f2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 --2- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 -- 172.21.15.150:0/4254010707 >> v1:172.21.15.142:6789/0 conn(0x7f322c06d890 legacy=0x7f322c096c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient: _finish_hunting 0 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 monclient: found mon.noname-b 2024-09-18T10:13:33.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:33.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:33.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 -- 172.21.15.150:0/4254010707 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c06caa0 con 0x7f322c12f830 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 -- 172.21.15.150:0/4254010707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 msgr2=0x7f322c136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3231893700 1 --2- 172.21.15.150:0/4254010707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 0x7f322c136420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3224009e60 tx=0x7f3224009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=357c329383c8d34c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 1 -- 172.21.15.150:0/4254010707 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400e070 con 0x7f322c12f830 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: dump: 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.693 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 1 -- 172.21.15.150:0/4254010707 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322400be40 con 0x7f322c12f830 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 1 -- 172.21.15.150:0/4254010707 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224012660 con 0x7f322c12f830 2024-09-18T10:13:33.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.683+0000 7f3223fff700 10 monclient: dump: 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 5 monclient: authenticate success, global_id 44261 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 4 set_mon_vals no callback set 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: shutdown 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- 172.21.15.150:0/4254010707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 msgr2=0x7f322c136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- 172.21.15.150:0/4254010707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 0x7f322c136420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3224009e60 tx=0x7f3224009780 comp rx=0 tx=0).stop 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- 172.21.15.150:0/4254010707 shutdown_connections 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- 172.21.15.150:0/4254010707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12f830 0x7f322c136420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- 172.21.15.150:0/4254010707 >> v2:172.21.15.142:3300/0 conn(0x7f322c12ef20 0x7f322c12f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- 172.21.15.150:0/4254010707 >> 172.21.15.150:0/4254010707 conn(0x7f322c0a31e0 msgr2=0x7f322c09f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- 172.21.15.150:0/4254010707 shutdown_connections 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- 172.21.15.150:0/4254010707 wait complete. 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: build_initial_monmap 2024-09-18T10:13:33.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: monmap: 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 Processor -- start 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- start start 2024-09-18T10:13:33.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: init 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c06d890 0x7f322c12ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-b con 0x7f322c06d890 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12ef20 0x7f322c1198b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-c con 0x7f322c12ef20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 0x7f322c119df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): picked mon.noname-a con 0x7f322c12f830 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c06df20 con 0x7f322c12f830 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0a20e0 con 0x7f322c06d890 2024-09-18T10:13:33.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1306e0 con 0x7f322c12ef20 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f32332f6700 10 monclient(hunting): authenticate will time out at 1095.782245s 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c06d890 0x7f322c12ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c06d890 0x7f322c12ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40434/0 (socket says 172.21.15.150:40434) 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 -- 172.21.15.150:0/3472567234 learned_addr learned my addr 172.21.15.150:0/3472567234 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): get_auth_request con 0x7f322c06d890 auth_method 0 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12ef20 0x7f322c1198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 0x7f322c119df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:33.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 10 monclient(hunting): get_auth_request con 0x7f322c12ef20 auth_method 0 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 10 monclient(hunting): get_auth_request con 0x7f322c12f830 auth_method 0 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient(hunting): handle_auth_done global_id 15265 payload 824 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 1 -- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 msgr2=0x7f322c119df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 1 -- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 msgr2=0x7f322c119df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 0x7f322c119df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:33.700 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 -- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12ef20 msgr2=0x7f322c1198b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12ef20 0x7f322c1198b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 -- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 msgr2=0x7f322c119df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231893700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 0x7f322c119df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f322c12f830 0x7f322c119df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient: _finish_hunting 0 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 monclient: found mon.noname-b 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:33.701 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3231092700 1 -- 172.21.15.150:0/3472567234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009a20 con 0x7f322c06d890 2024-09-18T10:13:33.702 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.687+0000 7f3230891700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f322c12ef20 0x7f322c1198b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:33.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f3231092700 1 -- 172.21.15.150:0/3472567234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c06d890 msgr2=0x7f322c12ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:33.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f3231092700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f322c06d890 0x7f322c12ddc0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f322800c5e0 tx=0x7f3228004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:33.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228005d40 con 0x7f322c06d890 2024-09-18T10:13:33.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: dump: 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 5 monclient: authenticate success, global_id 15265 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _renew_subs 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32280070d0 con 0x7f322c06d890 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 1 -- 172.21.15.150:0/3472567234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c11d1e0 con 0x7f322c06d890 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228007490 con 0x7f322c06d890 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32237fe700 4 set_mon_vals no callback set 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32217fa700 10 monclient: dump: 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.706 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _renew_subs 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 1 -- 172.21.15.150:0/3472567234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c11cf00 con 0x7f322c06d890 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:33.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.691+0000 7f32332f6700 1 -- 172.21.15.150:0/3472567234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c11d3d0 con 0x7f322c06d890 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f322801b3b0 con 0x7f322c06d890 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f32217fa700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3210035ac0 0x7f3210037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(116..116 src has 58..116) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f3228048140 con 0x7f322c06d890 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f3230891700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3210035ac0 0x7f3210037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f3230891700 10 monclient: get_auth_request con 0x7f3210035ac0 auth_method 0 2024-09-18T10:13:33.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.695+0000 7f3230891700 1 --2- 172.21.15.150:0/3472567234 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3210035ac0 0x7f3210037f70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f321c00b310 tx=0x7f321c00b540 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:33.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.703+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f322800f860 con 0x7f322c06d890 2024-09-18T10:13:33.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.703+0000 7f32217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:33.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.703+0000 7f32217fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:33.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32332f6700 10 start_mon_command cmd=[{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T10:13:33.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32332f6700 10 monclient: _send_command 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T10:13:33.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32332f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32332f6700 1 -- 172.21.15.150:0/3472567234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0} v 0) v1 -- 0x7f322c12fcb0 con 0x7f322c06d890 2024-09-18T10:13:33.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32217fa700 1 -- 172.21.15.150:0/3472567234 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}]=-75 (75) Value too large for defined data type v116) v1 ==== 154+0+0 (secure 0 0 0) 0x7f3228017020 con 0x7f322c06d890 2024-09-18T10:13:33.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush reweight-subtree", "name": "foobaz", "weight": 123456.0}] 2024-09-18T10:13:33.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.875+0000 7f32217fa700 10 monclient: _finish_command 2 = mon:75 (75) Value too large for defined data type 2024-09-18T10:13:33.881 INFO:tasks.workunit.client.0.smithi150.stderr:Error EOVERFLOW: (75) Value too large for defined data type 2024-09-18T10:13:33.884 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:13:33.885 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:33.885 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.13 2024-09-18T10:13:33.885 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 155 2024-09-18T10:13:33.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.971+0000 7fa725ffe700 10 monclient: get_monmap_and_config 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.971+0000 7fa725ffe700 10 monclient: build_initial_monmap 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.971+0000 7fa725ffe700 10 monclient: monmap: 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:33.975227+0000 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:33.975227+0000 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:33.976 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:33.977 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:33.977 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.971+0000 7fa725ffe700 1 Processor -- start 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 1 -- start start 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient: init 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 0x7fa7200a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-b con 0x7fa72009cf10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7200a0750 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7200a0b20 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72009e6e0 con 0x7fa7200a0750 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72009d2e0 con 0x7fa7200a0b20 2024-09-18T10:13:33.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72009e2e0 con 0x7fa72009cf10 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa725ffe700 10 monclient(hunting): authenticate will time out at 1096.068754s 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa7200a0b20 0x7fa72013a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53626/0 (socket says 172.21.15.150:53626) 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71ffff700 1 -- 172.21.15.150:0/1986372712 learned_addr learned my addr 172.21.15.150:0/1986372712 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 1 --2- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 0x7fa7200a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): get_auth_request con 0x7fa72009cf10 auth_method 0 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:33.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_done global_id 44267 payload 274 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 1 -- 172.21.15.150:0/1986372712 >> v1:172.21.15.150:6789/0 conn(0x7fa7200a0b20 legacy=0x7fa72013a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 1 -- 172.21.15.150:0/1986372712 >> v1:172.21.15.142:6789/0 conn(0x7fa7200a0750 legacy=0x7fa720136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:33.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.975+0000 7fa71f7fe700 1 -- 172.21.15.150:0/1986372712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7200a1f90 con 0x7fa72009cf10 2024-09-18T10:13:33.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71f7fe700 1 -- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 msgr2=0x7fa7200a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71f7fe700 1 --2- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 0x7fa7200a0210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa710009910 tx=0x7fa7100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f872afc4811c2d48 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 1 -- 172.21.15.150:0/1986372712 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71000ba70 con 0x7fa72009cf10 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.983 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 1 -- 172.21.15.150:0/1986372712 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71000bbd0 con 0x7fa72009cf10 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 5 monclient: authenticate success, global_id 44267 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 1 -- 172.21.15.150:0/1986372712 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710011420 con 0x7fa72009cf10 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 4 set_mon_vals no callback set 2024-09-18T10:13:33.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa71e7fc700 10 monclient: dump: 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.985 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 10 monclient: shutdown 2024-09-18T10:13:33.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 msgr2=0x7fa7200a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 --2- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 0x7fa7200a0210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa710009910 tx=0x7fa7100091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- 172.21.15.150:0/1986372712 shutdown_connections 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 --2- 172.21.15.150:0/1986372712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa72009cf10 0x7fa7200a0210 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- 172.21.15.150:0/1986372712 >> 172.21.15.150:0/1986372712 conn(0x7fa7200a5820 msgr2=0x7fa7200a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- 172.21.15.150:0/1986372712 shutdown_connections 2024-09-18T10:13:33.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- 172.21.15.150:0/1986372712 wait complete. 2024-09-18T10:13:33.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 10 monclient: build_initial_monmap 2024-09-18T10:13:33.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 10 monclient: monmap: 2024-09-18T10:13:33.991 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 Processor -- start 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 1 -- start start 2024-09-18T10:13:33.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.979+0000 7fa725ffe700 10 monclient: init 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-b con 0x7fa72009cf10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7200a0750 0x7fa720142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-a con 0x7fa7200a0750 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7200a0b20 0x7fa72008e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): picked mon.noname-c con 0x7fa7200a0b20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200a5120 con 0x7fa7200a0750 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa720091aa0 con 0x7fa72009cf10 2024-09-18T10:13:33.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200917c0 con 0x7fa7200a0b20 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient(hunting): authenticate will time out at 1096.076658s 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40440/0 (socket says 172.21.15.150:40440) 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 -- 172.21.15.150:0/3250834494 learned_addr learned my addr 172.21.15.150:0/3250834494 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): get_auth_request con 0x7fa72009cf10 auth_method 0 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:33.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71ffff700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7200a0b20 0x7fa72008e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71effd700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7200a0750 0x7fa720142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71ffff700 10 monclient(hunting): get_auth_request con 0x7fa7200a0b20 auth_method 0 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71effd700 10 monclient(hunting): get_auth_request con 0x7fa7200a0750 auth_method 0 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:33.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient(hunting): handle_auth_done global_id 15268 payload 824 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 -- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7200a0b20 msgr2=0x7fa72008e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7200a0b20 0x7fa72008e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 -- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7200a0750 msgr2=0x7fa720142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7200a0750 0x7fa720142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa710009420 con 0x7fa72009cf10 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 -- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 msgr2=0x7fa720141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71f7fe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7100093f0 tx=0x7fa710016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:33.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71001b680 con 0x7fa72009cf10 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: dump: 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.997 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 5 monclient: authenticate success, global_id 15268 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71001b7e0 con 0x7fa72009cf10 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _renew_subs 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa720091db0 con 0x7fa72009cf10 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71001bb80 con 0x7fa72009cf10 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa7077fe700 10 monclient: dump: 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:33.998 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa71dffb700 4 set_mon_vals no callback set 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _renew_subs 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.983+0000 7fa725ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:33.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa720092b50 con 0x7fa72009cf10 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa725ffe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa725ffe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa725ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa720063ed0 con 0x7fa72009cf10 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa71001c450 con 0x7fa72009cf10 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa7077fe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 0x7fa70c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(116..116 src has 58..116) v4 ==== 6686+0+0 (secure 0 0 0) 0x7fa71004a420 con 0x7fa72009cf10 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa71effd700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 0x7fa70c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.987+0000 7fa71effd700 10 monclient: get_auth_request con 0x7fa70c035af0 auth_method 0 2024-09-18T10:13:34.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.991+0000 7fa71effd700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 0x7fa70c037fa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa714003870 tx=0x7fa714009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.995+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa7100236c0 con 0x7fa72009cf10 2024-09-18T10:13:34.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.995+0000 7fa7077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:33.995+0000 7fa7077fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:34.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.159+0000 7fa725ffe700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:34.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.159+0000 7fa725ffe700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.159+0000 7fa725ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.159+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa720063ed0 con 0x7fa72009cf10 2024-09-18T10:13:34.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.163+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v116) v1 ==== 56+0+1518 (secure 0 0 0) 0x7fa710023300 con 0x7fa72009cf10 2024-09-18T10:13:34.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.163+0000 7fa7077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.163+0000 7fa7077fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:34.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa725ffe700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:34.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa725ffe700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa725ffe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7fa720063ed0 con 0x7fa72009cf10 2024-09-18T10:13:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa7077fe700 1 -- 172.21.15.150:0/3250834494 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v116) v1 ==== 56+0+1518 (secure 0 0 0) 0x7fa7100234e0 con 0x7fa72009cf10 2024-09-18T10:13:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa7077fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.255+0000 7fa7077fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:34.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 msgr2=0x7fa70c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 0x7fa70c037fa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa714003870 tx=0x7fa714009040 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 10 monclient: shutdown 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 msgr2=0x7fa720141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7100093f0 tx=0x7fa710016000 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 shutdown_connections 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa70c035af0 0x7fa70c037fa0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7200a0b20 0x7fa72008e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7200a0750 0x7fa720142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 --2- 172.21.15.150:0/3250834494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa72009cf10 0x7fa720141c00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 >> 172.21.15.150:0/3250834494 conn(0x7fa7200a5820 msgr2=0x7fa7200a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:34.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 shutdown_connections 2024-09-18T10:13:34.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.259+0000 7fa725ffe700 1 -- 172.21.15.150:0/3250834494 wait complete. 2024-09-18T10:13:34.272 INFO:tasks.workunit.client.0.smithi150.stdout: 13 155.00000 osd.13 down 1.00000 1.00000 2024-09-18T10:13:34.272 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd tree 2024-09-18T10:13:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep osd.15 2024-09-18T10:13:34.273 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 155 2024-09-18T10:13:34.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 10 monclient: get_monmap_and_config 2024-09-18T10:13:34.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 10 monclient: build_initial_monmap 2024-09-18T10:13:34.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 10 monclient: monmap: 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:34.367210+0000 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:34.367210+0000 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:34.368 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 1 Processor -- start 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 1 -- start start 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.363+0000 7f53c33b7700 10 monclient: init 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:13:34.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc09c830 0x7f53bc09cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-b con 0x7f53bc09c830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53bc09dc60 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53bc097d90 0x7f53bc09d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-c con 0x7f53bc097d90 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0970d0 con 0x7f53bc09dc60 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc096df0 con 0x7f53bc097d90 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc09d680 con 0x7f53bc09c830 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c33b7700 10 monclient(hunting): authenticate will time out at 1096.460629s 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53bc097d90 0x7f53bc09d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53bc097d90 0x7f53bc09d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40446/0 (socket says 172.21.15.150:40446) 2024-09-18T10:13:34.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 -- 172.21.15.150:0/630562194 learned_addr learned my addr 172.21.15.150:0/630562194 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:34.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): get_auth_request con 0x7f53bc097d90 auth_method 0 2024-09-18T10:13:34.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc09c830 0x7f53bc09cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): get_auth_request con 0x7f53bc09c830 auth_method 0 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient(hunting): handle_auth_done global_id 15271 payload 274 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 -- 172.21.15.150:0/630562194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc09c830 msgr2=0x7f53bc09cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc09c830 0x7f53bc09cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 -- 172.21.15.150:0/630562194 >> v1:172.21.15.142:6789/0 conn(0x7f53bc09dc60 legacy=0x7f53bc132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient: _finish_hunting 0 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 monclient: found mon.noname-c 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.367+0000 7f53c1954700 1 -- 172.21.15.150:0/630562194 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc096b10 con 0x7f53bc097d90 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 1 -- 172.21.15.150:0/630562194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc097d90 msgr2=0x7f53bc09d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc097d90 0x7f53bc09d140 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f53b80099b0 tx=0x7f53b8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38974af247167ef2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 1 -- 172.21.15.150:0/630562194 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8009ea0 con 0x7f53bc097d90 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: dump: 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.380 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 1 -- 172.21.15.150:0/630562194 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b800ba70 con 0x7f53bc097d90 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 5 monclient: authenticate success, global_id 15271 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:34.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 1 -- 172.21.15.150:0/630562194 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b800be30 con 0x7f53bc097d90 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 4 set_mon_vals no callback set 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53b3fff700 10 monclient: dump: 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.382 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: shutdown 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- 172.21.15.150:0/630562194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc097d90 msgr2=0x7f53bc09d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc097d90 0x7f53bc09d140 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f53b80099b0 tx=0x7f53b8009250 comp rx=0 tx=0).stop 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- 172.21.15.150:0/630562194 shutdown_connections 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc097d90 0x7f53bc09d140 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- 172.21.15.150:0/630562194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc09c830 0x7f53bc09cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- 172.21.15.150:0/630562194 >> 172.21.15.150:0/630562194 conn(0x7f53bc068400 msgr2=0x7f53bc0a1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- 172.21.15.150:0/630562194 shutdown_connections 2024-09-18T10:13:34.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- 172.21.15.150:0/630562194 wait complete. 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: build_initial_monmap 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: monmap: 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 Processor -- start 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- start start 2024-09-18T10:13:34.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: init 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 0x7f53bc132650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-c con 0x7f53bc097d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-b con 0x7f53bc09c830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): picked mon.noname-a con 0x7f53bc09dc60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1227a0 con 0x7f53bc09dc60 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc122dd0 con 0x7f53bc09c830 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1214f0 con 0x7f53bc097d90 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c33b7700 10 monclient(hunting): authenticate will time out at 1096.467507s 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40456/0 (socket says 172.21.15.150:40456) 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 1 -- 172.21.15.150:0/489078919 learned_addr learned my addr 172.21.15.150:0/489078919 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 10 monclient(hunting): get_auth_request con 0x7f53bc09c830 auth_method 0 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c0952700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1153700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 0x7f53bc132650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 10 monclient(hunting): get_auth_request con 0x7f53bc09dc60 auth_method 0 2024-09-18T10:13:34.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.371+0000 7f53c1954700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): get_auth_request con 0x7f53bc097d90 auth_method 0 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1954700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 msgr2=0x7f53bc11f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:13:34.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1954700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 msgr2=0x7f53bc11f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1954700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1954700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient(hunting): handle_auth_done global_id 15274 payload 824 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 msgr2=0x7f53bc132650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 0x7f53bc132650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 msgr2=0x7f53bc11f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient: _finish_hunting 0 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 monclient: found mon.noname-b 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b80094f0 con 0x7f53bc09c830 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c1153700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 0x7f53bc132650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:34.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 msgr2=0x7f53bc11f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c0952700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c600 tx=0x7f53ac007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0117a0 con 0x7f53bc09c830 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: dump: 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.389 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac011900 con 0x7f53bc09c830 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac011cc0 con 0x7f53bc09c830 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 5 monclient: authenticate success, global_id 15274 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b17fa700 10 monclient: dump: 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.390 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 10 monclient: _renew_subs 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53b37fe700 4 set_mon_vals no callback set 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc0a1930 con 0x7f53bc09c830 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 10 monclient: _renew_subs 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.375+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc142540 con 0x7f53bc09c830 2024-09-18T10:13:34.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c33b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c33b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c33b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc063ed0 con 0x7f53bc09c830 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f53ac012630 con 0x7f53bc09c830 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53b17fa700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 0x7f53a0037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(116..116 src has 58..116) v4 ==== 6686+0+0 (secure 0 0 0) 0x7f53ac048860 con 0x7f53bc09c830 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c1153700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 0x7f53a0037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.379+0000 7f53c1153700 10 monclient: get_auth_request con 0x7f53a0035b10 auth_method 0 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.383+0000 7f53c1153700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 0x7f53a0037fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e950 tx=0x7f53b400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.387+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f53ac022250 con 0x7f53bc09c830 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.387+0000 7f53b17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.387+0000 7f53b17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:34.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.551+0000 7f53c33b7700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.551+0000 7f53c33b7700 10 monclient: _send_command 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.551+0000 7f53c33b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.551+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f53bc09e670 con 0x7f53bc09c830 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.555+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v116) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f53ac0102e0 con 0x7f53bc09c830 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.555+0000 7f53b17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.555+0000 7f53b17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.647+0000 7f53c33b7700 10 start_mon_command cmd=[{"prefix": "osd tree"}] 2024-09-18T10:13:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.647+0000 7f53c33b7700 10 monclient: _send_command 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.647+0000 7f53c33b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.647+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd tree"} v 0) v1 -- 0x7f53bc09da40 con 0x7f53bc09c830 2024-09-18T10:13:34.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53b17fa700 1 -- 172.21.15.150:0/489078919 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd tree"}]=0 v116) v1 ==== 56+0+1518 (secure 0 0 0) 0x7f53ac010930 con 0x7f53bc09c830 2024-09-18T10:13:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53b17fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd tree"}] 2024-09-18T10:13:34.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53b17fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 msgr2=0x7f53a0037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 0x7f53a0037fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e950 tx=0x7f53b400b040 comp rx=0 tx=0).stop 2024-09-18T10:13:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 10 monclient: shutdown 2024-09-18T10:13:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:34.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 msgr2=0x7f53bc11f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00c600 tx=0x7f53ac007f60 comp rx=0 tx=0).stop 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 shutdown_connections 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f53a0035b10 0x7f53a0037fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53bc09dc60 0x7f53bc11f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53bc09c830 0x7f53bc11f260 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 --2- 172.21.15.150:0/489078919 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53bc097d90 0x7f53bc132650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 >> 172.21.15.150:0/489078919 conn(0x7f53bc068400 msgr2=0x7f53bc09a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 shutdown_connections 2024-09-18T10:13:34.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.651+0000 7f53c33b7700 1 -- 172.21.15.150:0/489078919 wait complete. 2024-09-18T10:13:34.665 INFO:tasks.workunit.client.0.smithi150.stdout: 15 155.00000 osd.15 down 1.00000 1.00000 2024-09-18T10:13:34.665 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.13 2024-09-18T10:13:34.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.755+0000 7ff0382e7700 10 monclient: get_monmap_and_config 2024-09-18T10:13:34.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.755+0000 7ff0382e7700 10 monclient: build_initial_monmap 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.755+0000 7ff0382e7700 10 monclient: monmap: 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:34.758983+0000 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:34.758983+0000 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:34.760 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 Processor -- start 2024-09-18T10:13:34.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 -- start start 2024-09-18T10:13:34.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient: init 2024-09-18T10:13:34.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:34.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff03009e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 0x7ff030137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-a con 0x7ff0300a09a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0300a0da0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03009e220 con 0x7ff0300a0da0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03009eaa0 con 0x7ff03009e6d0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03009ee30 con 0x7ff0300a09a0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff0382e7700 10 monclient(hunting): authenticate will time out at 1096.853146s 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff036083700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff03009e6d0 0x7ff0300a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53638/0 (socket says 172.21.15.150:53638) 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 learned_addr learned my addr 172.21.15.150:0/3579569292 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 1 --2- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 0x7ff030137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 10 monclient(hunting): get_auth_request con 0x7ff0300a09a0 auth_method 0 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 1 -- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 msgr2=0x7ff030137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 1 -- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 msgr2=0x7ff030137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 1 --2- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 0x7ff030137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:34.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035882700 1 --2- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 0x7ff030137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 >> v1:172.21.15.150:6789/0 conn(0x7ff03009e6d0 legacy=0x7ff0300a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721659768 0 0) 0x7ff03009eaa0 con 0x7ff03009e6d0 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035081700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035081700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035081700 10 monclient(hunting): my global_id is 15277 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff01c003570 con 0x7ff03009e6d0 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.759+0000 7ff036884700 1 -- 172.21.15.150:0/3579569292 >> v1:172.21.15.142:6789/0 conn(0x7ff0300a0da0 legacy=0x7ff030139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 675622342 0 0) 0x7ff01c003570 con 0x7ff03009e6d0 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 msgr2=0x7ff030137a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 --2- 172.21.15.150:0/3579569292 >> v2:172.21.15.142:3300/0 conn(0x7ff0300a09a0 0x7ff030137a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 >> v1:172.21.15.142:6789/0 conn(0x7ff0300a0da0 legacy=0x7ff030139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: _finish_hunting 0 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03009d930 con 0x7ff03009e6d0 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: _finish_auth 0 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 5 monclient: authenticate success, global_id 15277 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff02c001860 con 0x7ff03009e6d0 2024-09-18T10:13:34.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: dump: 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:34.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 >> v1:172.21.15.150:6789/0 conn(0x7ff03009e6d0 legacy=0x7ff0300a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 0x7ff01c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): picked mon.b con 0x7ff01c003b80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff01c004810 0x7ff01c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): picked mon.a con 0x7ff01c004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c005410 0x7ff01c00bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): picked mon.c con 0x7ff01c005410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03009ee30 con 0x7ff01c004810 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c00c3e0 con 0x7ff01c003b80 2024-09-18T10:13:34.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c00c5a0 con 0x7ff01c005410 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 0x7ff01c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): get_auth_request con 0x7ff01c003b80 auth_method 0 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:34.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035882700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff01c004810 0x7ff01c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c005410 0x7ff01c00bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 10 monclient(hunting): get_auth_request con 0x7ff01c005410 auth_method 0 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035882700 10 monclient(hunting): get_auth_request con 0x7ff01c004810 auth_method 0 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035882700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035882700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035882700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient(hunting): handle_auth_done global_id 15277 payload 293 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c005410 msgr2=0x7ff01c00bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c005410 0x7ff01c00bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff01c004810 msgr2=0x7ff01c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff01c004810 0x7ff01c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient: _finish_hunting 0 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 monclient: found mon.b 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036884700 1 -- 172.21.15.150:0/3579569292 reap_dead start 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c003570 con 0x7ff01c003b80 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 10 monclient: _finish_auth 0 2024-09-18T10:13:34.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 -- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 msgr2=0x7ff01c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff036083700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 0x7ff01c003f90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0014d0 tx=0x7ff0300a36a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117ded9e7eff1ee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c005920 con 0x7ff01c003b80 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: dump: 2024-09-18T10:13:34.778 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:34.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c005aa0 con 0x7ff01c003b80 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 4 set_mon_vals no callback set 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 1 -- 172.21.15.150:0/3579569292 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c005e60 con 0x7ff01c003b80 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff035081700 10 monclient: dump: 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.780 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 10 monclient: shutdown 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 1 -- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 msgr2=0x7ff01c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.763+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 0x7ff01c003f90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0014d0 tx=0x7ff0300a36a0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- 172.21.15.150:0/3579569292 shutdown_connections 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3579569292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff01c003b80 0x7ff01c003f90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- 172.21.15.150:0/3579569292 >> 172.21.15.150:0/3579569292 conn(0x7ff0300a58a0 msgr2=0x7ff0300a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- 172.21.15.150:0/3579569292 shutdown_connections 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- 172.21.15.150:0/3579569292 wait complete. 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient: build_initial_monmap 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient: monmap: 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:34.782 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 Processor -- start 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- start start 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient: init 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c003b80 0x7ff030141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-c con 0x7ff01c003b80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-b con 0x7ff030142250 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): picked mon.noname-a con 0x7ff030142620 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03009c180 con 0x7ff030142620 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.783 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff030092780 con 0x7ff030142250 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff030091bd0 con 0x7ff01c003b80 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff0382e7700 10 monclient(hunting): authenticate will time out at 1096.862612s 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40472/0 (socket says 172.21.15.150:40472) 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 -- 172.21.15.150:0/3935834862 learned_addr learned my addr 172.21.15.150:0/3935834862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): get_auth_request con 0x7ff030142250 auth_method 0 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c003b80 0x7ff030141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): get_auth_request con 0x7ff01c003b80 auth_method 0 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 10 monclient(hunting): get_auth_request con 0x7ff030142620 auth_method 0 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:34.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036083700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient(hunting): handle_auth_done global_id 15280 payload 824 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 msgr2=0x7ff03008e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 msgr2=0x7ff03008e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c003b80 msgr2=0x7ff030141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c003b80 0x7ff030141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 msgr2=0x7ff03008e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff036884700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient: _finish_hunting 0 2024-09-18T10:13:34.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 monclient: found mon.noname-b 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.767+0000 7ff035882700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff030091860 con 0x7ff030142250 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff035882700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 msgr2=0x7ff03008e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff035882700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff02000f6c0 tx=0x7ff020008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020014880 con 0x7ff030142250 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: dump: 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.787 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 5 monclient: authenticate success, global_id 15280 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _renew_subs 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0200149e0 con 0x7ff030142250 2024-09-18T10:13:34.788 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0300912d0 con 0x7ff030142250 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020014d80 con 0x7ff030142250 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 10 monclient: dump: 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:34.789 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff034880700 4 set_mon_vals no callback set 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _renew_subs 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff030090620 con 0x7ff030142250 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03009dbf0 con 0x7ff030142250 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff0200135d0 con 0x7ff030142250 2024-09-18T10:13:34.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 0x7ff014037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(116..116 src has 58..116) v4 ==== 6686+0+0 (secure 0 0 0) 0x7ff020048d60 con 0x7ff030142250 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.771+0000 7ff036083700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 0x7ff014037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.775+0000 7ff036083700 10 monclient: get_auth_request con 0x7ff014035ac0 auth_method 0 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.775+0000 7ff036083700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 0x7ff014037f70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff02c000c00 tx=0x7ff02c014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.783+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff02004e020 con 0x7ff030142250 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.783+0000 7ff0267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:34.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.783+0000 7ff0267fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:34.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.947+0000 7ff0382e7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.947+0000 7ff0382e7700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.947+0000 7ff0382e7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:34.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:34.947+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.13"} v 0) v1 -- 0x7ff0300919f0 con 0x7ff030142250 2024-09-18T10:13:35.296 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: get_monmap_and_config 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: build_initial_monmap 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: monmap: 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:35.407243+0000 2024-09-18T10:13:35.408 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:35.407243+0000 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:35.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 Processor -- start 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 -- start start 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: init 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb13c09aae0 0x7fb13c096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:35.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-c con 0x7fb13c09aae0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb13c0967a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb13c09c4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c09aeb0 con 0x7fb13c09c4f0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c06f6d0 con 0x7fb13c0967a0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c0955a0 con 0x7fb13c09aae0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb142ad3700 10 monclient(hunting): authenticate will time out at 1104.332824s 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb1422d2700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb13c09c4f0 0x7fb13c1375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51302/0 (socket says 172.21.15.142:51302) 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb1422d2700 1 -- 172.21.15.142:0/3421829892 learned_addr learned my addr 172.21.15.142:0/3421829892 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 1 --2- 172.21.15.142:0/3421829892 >> v2:172.21.15.150:3300/0 conn(0x7fb13c09aae0 0x7fb13c096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:35.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): get_auth_request con 0x7fb13c09aae0 auth_method 0 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient(hunting): handle_auth_done global_id 15283 payload 274 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 1 -- 172.21.15.142:0/3421829892 >> v1:172.21.15.150:6789/0 conn(0x7fb13c0967a0 legacy=0x7fb13c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 1 -- 172.21.15.142:0/3421829892 >> v1:172.21.15.142:6789/0 conn(0x7fb13c09c4f0 legacy=0x7fb13c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient: _finish_hunting 0 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 1 monclient: found mon.noname-c 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:35.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:35.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.403+0000 7fb141ad1700 1 -- 172.21.15.142:0/3421829892 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c095b60 con 0x7fb13c09aae0 2024-09-18T10:13:35.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 -- 172.21.15.142:0/3421829892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09aae0 msgr2=0x7fb13c096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:35.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 --2- 172.21.15.142:0/3421829892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09aae0 0x7fb13c096260 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb1380098e0 tx=0x7fb1380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2a5424a14b3bb6a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:35.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 1 -- 172.21.15.142:0/3421829892 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13800ba70 con 0x7fb13c09aae0 2024-09-18T10:13:35.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: dump: 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:35.417 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: _finish_auth 0 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 5 monclient: authenticate success, global_id 15283 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 1 -- 172.21.15.142:0/3421829892 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13800bbd0 con 0x7fb13c09aae0 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 1 -- 172.21.15.142:0/3421829892 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138011420 con 0x7fb13c09aae0 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 4 set_mon_vals no callback set 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:35.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb140acf700 10 monclient: dump: 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: shutdown 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:35.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- 172.21.15.142:0/3421829892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09aae0 msgr2=0x7fb13c096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 --2- 172.21.15.142:0/3421829892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09aae0 0x7fb13c096260 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb1380098e0 tx=0x7fb1380091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- 172.21.15.142:0/3421829892 shutdown_connections 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 --2- 172.21.15.142:0/3421829892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09aae0 0x7fb13c096260 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- 172.21.15.142:0/3421829892 >> 172.21.15.142:0/3421829892 conn(0x7fb13c0a1fe0 msgr2=0x7fb13c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- 172.21.15.142:0/3421829892 shutdown_connections 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- 172.21.15.142:0/3421829892 wait complete. 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: build_initial_monmap 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: monmap: 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:35.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.421 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:35.421 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:35.421 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:35.421 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 Processor -- start 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- start start 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: init 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-c con 0x7fb13c0967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:35.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-a con 0x7fb13c09aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09c4f0 0x7fb13c08bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): picked mon.noname-b con 0x7fb13c09c4f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c08f460 con 0x7fb13c09aae0 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c06f8d0 con 0x7fb13c09c4f0 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c06ee70 con 0x7fb13c0967a0 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb142ad3700 10 monclient(hunting): authenticate will time out at 1104.341880s 2024-09-18T10:13:35.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38868/0 (socket says 172.21.15.142:38868) 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 -- 172.21.15.142:0/1277284088 learned_addr learned my addr 172.21.15.142:0/1277284088 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): get_auth_request con 0x7fb13c0967a0 auth_method 0 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 10 monclient(hunting): get_auth_request con 0x7fb13c09aae0 auth_method 0 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:35.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09c4f0 0x7fb13c08bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): get_auth_request con 0x7fb13c09c4f0 auth_method 0 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 msgr2=0x7fb13c13fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 msgr2=0x7fb13c13fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1412d0700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:35.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb1422d2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient(hunting): handle_auth_done global_id 44282 payload 824 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09c4f0 msgr2=0x7fb13c08bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09c4f0 0x7fb13c08bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 msgr2=0x7fb13c13fdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient: _finish_hunting 0 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 monclient: found mon.noname-c 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.411+0000 7fb141ad1700 1 -- 172.21.15.142:0/1277284088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138009420 con 0x7fb13c0967a0 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb141ad1700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 msgr2=0x7fb13c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:35.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb141ad1700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1380030c0 tx=0x7fb138016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138011420 con 0x7fb13c0967a0 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: dump: 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:35.431 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13801b680 con 0x7fb13c0967a0 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 5 monclient: authenticate success, global_id 44282 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13801ba00 con 0x7fb13c0967a0 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 10 monclient: dump: 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:35.432 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _renew_subs 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb133fff700 4 set_mon_vals no callback set 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c08d650 con 0x7fb13c0967a0 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _renew_subs 2024-09-18T10:13:35.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c08e220 con 0x7fb13c0967a0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c072a80 con 0x7fb13c0967a0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb13801c450 con 0x7fb13c0967a0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 0x7fb120037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.415+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(116..116 src has 58..116) v4 ==== 6686+0+0 (secure 0 0 0) 0x7fb13804a640 con 0x7fb13c0967a0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.419+0000 7fb1412d0700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 0x7fb120037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:35.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.419+0000 7fb1412d0700 10 monclient: get_auth_request con 0x7fb120035af0 auth_method 0 2024-09-18T10:13:35.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.419+0000 7fb1412d0700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 0x7fb120037fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb12c007cb0 tx=0x7fb12c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:35.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.427+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb138018950 con 0x7fb13c0967a0 2024-09-18T10:13:35.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.427+0000 7fb131ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:35.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.427+0000 7fb131ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:35.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb142ad3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:35.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb142ad3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:35.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb142ad3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:35.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb13c072a80 con 0x7fb13c0967a0 2024-09-18T10:13:35.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb131ffb700 1 -- 172.21.15.142:0/1277284088 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fb13802a020 con 0x7fb13c0967a0 2024-09-18T10:13:35.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb131ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:35.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.643+0000 7fb131ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:35.652 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":52,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 msgr2=0x7fb120037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 0x7fb120037fa0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb12c007cb0 tx=0x7fb12c009040 comp rx=0 tx=0).stop 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 10 monclient: shutdown 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 msgr2=0x7fb13c13f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb1380030c0 tx=0x7fb138016000 comp rx=0 tx=0).stop 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 shutdown_connections 2024-09-18T10:13:35.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb120035af0 0x7fb120037fa0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb13c09c4f0 0x7fb13c08bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb13c09aae0 0x7fb13c13fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 --2- 172.21.15.142:0/1277284088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb13c0967a0 0x7fb13c13f880 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 >> 172.21.15.142:0/1277284088 conn(0x7fb13c0a1fe0 msgr2=0x7fb13c06c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 shutdown_connections 2024-09-18T10:13:35.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:35.647+0000 7fb142ad3700 1 -- 172.21.15.142:0/1277284088 wait complete. 2024-09-18T10:13:35.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.855+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.13"}]=0 removed item id 13 name 'osd.13' from crush map v117) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff020018070 con 0x7ff030142250 2024-09-18T10:13:35.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.855+0000 7ff0267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:35.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.855+0000 7ff0267fc700 10 monclient: _finish_command 2 = system:0 removed item id 13 name 'osd.13' from crush map 2024-09-18T10:13:35.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0382e7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0382e7700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0382e7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.13"} v 0) v1 -- 0x7ff030063ed0 con 0x7ff030142250 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0267fc700 1 -- 172.21.15.150:0/3935834862 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.13"}]=0 device 'osd.13' does not appear in the crush map v118) v1 ==== 126+0+0 (secure 0 0 0) 0x7ff020023660 con 0x7ff030142250 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0267fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.13"}] 2024-09-18T10:13:35.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.967+0000 7ff0267fc700 10 monclient: _finish_command 3 = system:0 device 'osd.13' does not appear in the crush map 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 msgr2=0x7ff014037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 0x7ff014037f70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff02c000c00 tx=0x7ff02c014000 comp rx=0 tx=0).stop 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 10 monclient: shutdown 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 msgr2=0x7ff03008e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:35.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff02000f6c0 tx=0x7ff020008040 comp rx=0 tx=0).stop 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 shutdown_connections 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff014035ac0 0x7ff014037f70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff030142620 0x7ff03008e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff030142250 0x7ff03008e0d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 --2- 172.21.15.150:0/3935834862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff01c003b80 0x7ff030141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 >> 172.21.15.150:0/3935834862 conn(0x7ff0300a58a0 msgr2=0x7ff03009ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 shutdown_connections 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:35.971+0000 7ff0382e7700 1 -- 172.21.15.150:0/3935834862 wait complete. 2024-09-18T10:13:35.974 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.13' does not appear in the crush map 2024-09-18T10:13:35.982 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm osd.15 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: get_monmap_and_config 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: build_initial_monmap 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: monmap: 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:36.085 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:36.084689+0000 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:36.084689+0000 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:36.086 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 Processor -- start 2024-09-18T10:13:36.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 -- start start 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: init 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a0c09cf90 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a0c09e480 0x7f0a0c0a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-c con 0x7f0a0c09e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c0990a0 0x7f0a0c139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-b con 0x7f0a0c0990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c09e060 con 0x7f0a0c09cf90 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0c0a66c0 con 0x7f0a0c09e480 2024-09-18T10:13:36.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c09d750 con 0x7f0a0c0990a0 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a133c4700 10 monclient(hunting): authenticate will time out at 1098.178343s 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a0c09e480 0x7f0a0c0a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a0c09e480 0x7f0a0c0a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40478/0 (socket says 172.21.15.150:40478) 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 learned_addr learned my addr 172.21.15.150:0/1479279747 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): get_auth_request con 0x7f0a0c09e480 auth_method 0 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:36.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a11961700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c0990a0 0x7f0a0c139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a11961700 10 monclient(hunting): get_auth_request con 0x7f0a0c0990a0 auth_method 0 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a11961700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a11961700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a11961700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient(hunting): handle_auth_done global_id 15289 payload 274 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c0990a0 msgr2=0x7f0a0c139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c0990a0 0x7f0a0c139a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> v1:172.21.15.142:6789/0 conn(0x7f0a0c09cf90 legacy=0x7f0a0c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient: _finish_hunting 0 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 monclient: found mon.noname-c 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.083+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c09d360 con 0x7f0a0c09e480 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c0a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c0a09a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0a000099b0 tx=0x7f0a00009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8dd328020cc2c0e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c0a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c0a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c0a09a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0a000099b0 tx=0x7f0a00009250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c0a09a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0a000099b0 tx=0x7f0a00009250 comp rx=0 tx=0).stop 2024-09-18T10:13:36.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c0a09a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c0a09a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient: _add_conns ranks=[1,3] 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09f4003550 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 1 --2- 172.21.15.150:0/1479279747 >> v2:172.21.15.142:3300/0 conn(0x7f09f40074d0 0x7f09f4009980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f09f40074d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f400a250 con 0x7f09f4003550 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f400b410 con 0x7f09f40074d0 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a0bfff700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:36.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> v2:172.21.15.142:3300/0 conn(0x7f09f40074d0 0x7f09f4009980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): get_auth_request con 0x7f09f40074d0 auth_method 0 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient(hunting): handle_auth_done global_id 15289 payload 293 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> v1:172.21.15.150:6789/0 conn(0x7f09f4003550 legacy=0x7f09f40059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient: _finish_hunting 0 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 monclient: found mon.noname-a 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:36.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.087+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f400b640 con 0x7f09f40074d0 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a1095f700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 msgr2=0x7f09f4009980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a1095f700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f09f4009980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a00015000 tx=0x7f0a00016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17774b78d41b15b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0000ba70 con 0x7f09f40074d0 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: dump: 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.097 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: _finish_auth 0 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 5 monclient: authenticate success, global_id 15289 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0001d4e0 con 0x7f09f40074d0 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 1 -- 172.21.15.150:0/1479279747 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0001d8c0 con 0x7f09f40074d0 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:36.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 4 set_mon_vals no callback set 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a0bfff700 10 monclient: dump: 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:36.099 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 10 monclient: shutdown 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 1 -- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 msgr2=0x7f09f4009980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.091+0000 7f0a133c4700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f09f4009980 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a00015000 tx=0x7f0a00016000 comp rx=0 tx=0).stop 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a11961700 1 -- 172.21.15.150:0/1479279747 reap_dead start 2024-09-18T10:13:36.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 -- 172.21.15.150:0/1479279747 shutdown_connections 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c0a09a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 --2- 172.21.15.150:0/1479279747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f09f4009980 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 -- 172.21.15.150:0/1479279747 >> 172.21.15.150:0/1479279747 conn(0x7f0a0c0a58a0 msgr2=0x7f0a0c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 -- 172.21.15.150:0/1479279747 shutdown_connections 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 -- 172.21.15.150:0/1479279747 wait complete. 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 10 monclient: build_initial_monmap 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 10 monclient: monmap: 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:36.104 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 Processor -- start 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 1 -- start start 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.095+0000 7f0a133c4700 10 monclient: init 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-a con 0x7f09f40074d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:36.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-b con 0x7f0a0c09e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c142560 0x7f0a0c08d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): picked mon.noname-c con 0x7f0a0c142560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0a4210 con 0x7f09f40074d0 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c091870 con 0x7f0a0c09e480 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c08eb20 con 0x7f0a0c142560 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient(hunting): authenticate will time out at 1098.192300s 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40490/0 (socket says 172.21.15.150:40490) 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 -- 172.21.15.150:0/2727406730 learned_addr learned my addr 172.21.15.150:0/2727406730 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): get_auth_request con 0x7f0a0c09e480 auth_method 0 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 10 monclient(hunting): get_auth_request con 0x7f09f40074d0 auth_method 0 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11961700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c142560 0x7f0a0c08d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11961700 10 monclient(hunting): get_auth_request con 0x7f0a0c142560 auth_method 0 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11961700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11961700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11961700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 msgr2=0x7f0a0c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient(hunting): handle_auth_done global_id 15292 payload 824 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 msgr2=0x7f0a0c141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c142560 msgr2=0x7f0a0c08d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a11160700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c142560 0x7f0a0c08d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 msgr2=0x7f0a0c141ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient: _finish_hunting 0 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 monclient: found mon.noname-b 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c08efc0 con 0x7f0a0c09e480 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:36.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a1095f700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0a000090b0 tx=0x7f0a0001f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:36.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0001d8c0 con 0x7f0a0c09e480 2024-09-18T10:13:36.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:36.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:36.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: dump: 2024-09-18T10:13:36.111 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:36.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 5 monclient: authenticate success, global_id 15292 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a00026480 con 0x7f0a0c09e480 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _renew_subs 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c0905e0 con 0x7f0a0c09e480 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00026860 con 0x7f0a0c09e480 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a097fa700 10 monclient: dump: 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a0affd700 4 set_mon_vals no callback set 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _renew_subs 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.099+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c0905e0 con 0x7f0a0c09e480 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0a00026bd0 con 0x7f0a0c09e480 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a097fa700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 0x7f09f8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a133c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a133c4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a133c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(118..118 src has 58..118) v4 ==== 6664+0+0 (secure 0 0 0) 0x7f0a0004b840 con 0x7f0a0c09e480 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c0905e0 con 0x7f0a0c09e480 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a11160700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 0x7f09f8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.103+0000 7f0a11160700 10 monclient: get_auth_request con 0x7f09f8035ac0 auth_method 0 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.107+0000 7f0a11160700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 0x7f09f8037f70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0a04007fd0 tx=0x7f0a04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.111+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0a0001a420 con 0x7f0a0c09e480 2024-09-18T10:13:36.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.111+0000 7f0a097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:36.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.111+0000 7f0a097fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:36.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.275+0000 7f0a133c4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:36.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.275+0000 7f0a133c4700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:36.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.275+0000 7f0a133c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:36.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.275+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.15"} v 0) v1 -- 0x7f0a0c0905e0 con 0x7f0a0c09e480 2024-09-18T10:13:36.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.923+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.15"}]=0 removed item id 15 name 'osd.15' from crush map v119) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0a00010380 con 0x7f0a0c09e480 2024-09-18T10:13:36.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.923+0000 7f0a097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:36.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:36.923+0000 7f0a097fa700 10 monclient: _finish_command 2 = system:0 removed item id 15 name 'osd.15' from crush map 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a133c4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a133c4700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a133c4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "osd.15"} v 0) v1 -- 0x7f0a0c142960 con 0x7f0a0c09e480 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a097fa700 1 -- 172.21.15.150:0/2727406730 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "osd.15"}]=0 device 'osd.15' does not appear in the crush map v119) v1 ==== 126+0+0 (secure 0 0 0) 0x7f0a00046090 con 0x7f0a0c09e480 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a097fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "osd.15"}] 2024-09-18T10:13:37.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.031+0000 7f0a097fa700 10 monclient: _finish_command 3 = system:0 device 'osd.15' does not appear in the crush map 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 msgr2=0x7f09f8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 0x7f09f8037f70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0a04007fd0 tx=0x7f0a04009040 comp rx=0 tx=0).stop 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 10 monclient: shutdown 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 msgr2=0x7f0a0c142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0a000090b0 tx=0x7f0a0001f040 comp rx=0 tx=0).stop 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 shutdown_connections 2024-09-18T10:13:37.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f09f8035ac0 0x7f09f8037f70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a0c142560 0x7f0a0c08d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a0c09e480 0x7f0a0c142020 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 --2- 172.21.15.150:0/2727406730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09f40074d0 0x7f0a0c141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 >> 172.21.15.150:0/2727406730 conn(0x7f0a0c0a58a0 msgr2=0x7f0a0c0a2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 shutdown_connections 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.035+0000 7f0a133c4700 1 -- 172.21.15.150:0/2727406730 wait complete. 2024-09-18T10:13:37.039 INFO:tasks.workunit.client.0.smithi150.stderr:device 'osd.15' does not appear in the crush map 2024-09-18T10:13:37.047 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm osd.13 2024-09-18T10:13:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: get_monmap_and_config 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: build_initial_monmap 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: monmap: 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:37.148375+0000 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:37.148375+0000 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:37.150 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 Processor -- start 2024-09-18T10:13:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 -- start start 2024-09-18T10:13:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: init 2024-09-18T10:13:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f1c09cf90 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f1c09e480 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 0x7f5f1c13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c0990a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c09e060 con 0x7f5f1c09e480 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c0a10b0 con 0x7f5f1c09cf90 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1c0a66c0 con 0x7f5f1c0990a0 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f22ac9700 10 monclient(hunting): authenticate will time out at 1099.242013s 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f20865700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5f1c09cf90 0x7f5f1c0a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53646/0 (socket says 172.21.15.150:53646) 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f20865700 1 -- 172.21.15.150:0/1493301606 learned_addr learned my addr 172.21.15.150:0/1493301606 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 1 --2- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 0x7f5f1c13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 10 monclient(hunting): get_auth_request con 0x7f5f1c0990a0 auth_method 0 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 1 -- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 msgr2=0x7f5f1c13aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 1 -- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 msgr2=0x7f5f1c13aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 1 --2- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 0x7f5f1c13aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.147+0000 7f5f21066700 1 --2- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 0x7f5f1c13aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 1 -- 172.21.15.150:0/1493301606 >> v1:172.21.15.150:6789/0 conn(0x7f5f1c09cf90 legacy=0x7f5f1c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119667741 0 0) 0x7f5f1c0a10b0 con 0x7f5f1c09cf90 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): my global_id is 15295 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f04003570 con 0x7f5f1c09cf90 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1493301606 >> v1:172.21.15.142:6789/0 conn(0x7f5f1c09e480 legacy=0x7f5f1c0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783297054 0 0) 0x7f5f04003570 con 0x7f5f1c09cf90 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 msgr2=0x7f5f1c13aa40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 --2- 172.21.15.150:0/1493301606 >> v2:172.21.15.142:3300/0 conn(0x7f5f1c0990a0 0x7f5f1c13aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 >> v1:172.21.15.142:6789/0 conn(0x7f5f1c09e480 legacy=0x7f5f1c0a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c09d750 con 0x7f5f1c09cf90 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f22ac9700 5 monclient: authenticate success, global_id 15295 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f5f0c001860 con 0x7f5f1c09cf90 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f22ac9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: dump: 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:13:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 >> v1:172.21.15.150:6789/0 conn(0x7f5f1c09cf90 legacy=0x7f5f1c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f04003b80 0x7f5f04003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): picked mon.a con 0x7f5f04003b80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 0x7f5f04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): picked mon.b con 0x7f5f04004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04005410 0x7f5f0400bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): picked mon.c con 0x7f5f04005410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0400c3b0 con 0x7f5f04003b80 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0400c520 con 0x7f5f04004810 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0400c6e0 con 0x7f5f04005410 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f22ac9700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 0x7f5f04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): get_auth_request con 0x7f5f04004810 auth_method 0 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f04003b80 0x7f5f04003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 10 monclient(hunting): get_auth_request con 0x7f5f04003b80 auth_method 0 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f20865700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f21066700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04005410 0x7f5f0400bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_done global_id 15295 payload 293 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04005410 msgr2=0x7f5f0400bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04005410 0x7f5f0400bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f04003b80 msgr2=0x7f5f04003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f04003b80 0x7f5f04003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 monclient: found mon.b 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1493301606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f04003570 con 0x7f5f04004810 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 10 monclient: _finish_auth 0 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f1bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f21066700 10 monclient: get_auth_request con 0x7f5f04005410 auth_method 0 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f21066700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04005410 0x7f5f0400bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.151+0000 7f5f21066700 1 -- 172.21.15.150:0/1493301606 reap_dead start 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 msgr2=0x7f5f04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 0x7f5f04004c80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5f10001bc0 tx=0x7f5f10007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37999d17adcb6d45 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f100070d0 con 0x7f5f04004810 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:37.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: dump: 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f10007250 con 0x7f5f04004810 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 4 set_mon_vals no callback set 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 1 -- 172.21.15.150:0/1493301606 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10007610 con 0x7f5f04004810 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1b7fe700 10 monclient: dump: 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.173 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: shutdown 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 msgr2=0x7f5f04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 0x7f5f04004c80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5f10001bc0 tx=0x7f5f10007d10 comp rx=0 tx=0).stop 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1493301606 shutdown_connections 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1493301606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f04004810 0x7f5f04004c80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1493301606 >> 172.21.15.150:0/1493301606 conn(0x7f5f1c0a58a0 msgr2=0x7f5f1c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1493301606 shutdown_connections 2024-09-18T10:13:37.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1493301606 wait complete. 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: build_initial_monmap 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: monmap: 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 Processor -- start 2024-09-18T10:13:37.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- start start 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: init 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04004810 0x7f5f1c08cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-c con 0x7f5f04004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-b con 0x7f5f1c08d4c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f1c08ddd0 0x7f5f1c147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): picked mon.noname-a con 0x7f5f1c08ddd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c08e920 con 0x7f5f1c08ddd0 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c091650 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c09be80 con 0x7f5f04004810 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f22ac9700 10 monclient(hunting): authenticate will time out at 1099.251734s 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40538/0 (socket says 172.21.15.150:40538) 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1722090183 learned_addr learned my addr 172.21.15.150:0/1722090183 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): get_auth_request con 0x7f5f1c08d4c0 auth_method 0 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:37.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.155+0000 7f5f20865700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04004810 0x7f5f1c08cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f20865700 10 monclient(hunting): get_auth_request con 0x7f5f04004810 auth_method 0 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f20865700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f20865700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f20865700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f21066700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f1c08ddd0 0x7f5f1c147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f21066700 10 monclient(hunting): get_auth_request con 0x7f5f1c08ddd0 auth_method 0 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f21066700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f21066700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f21066700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 10 monclient(hunting): handle_auth_done global_id 15298 payload 824 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04004810 msgr2=0x7f5f1c08cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04004810 0x7f5f1c08cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f1c08ddd0 msgr2=0x7f5f1c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f1c08ddd0 0x7f5f1c147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:37.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 monclient: found mon.noname-b 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c08e200 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 -- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 msgr2=0x7f5f1c08d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1bfff700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000c3c0 tx=0x7f5f10015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10005e00 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: dump: 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.179 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 5 monclient: authenticate success, global_id 15298 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1000c910 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 10 monclient: _renew_subs 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c08eda0 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1000ccd0 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f18ff9700 10 monclient: dump: 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:37.181 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f1a7fc700 4 set_mon_vals no callback set 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 10 monclient: _renew_subs 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.159+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c08f460 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5f1001e640 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f22ac9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f22ac9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f22ac9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c063ed0 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f18ff9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 0x7f5f08037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(119..119 src has 58..119) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f5f10049f10 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f20865700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 0x7f5f08037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:37.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f20865700 10 monclient: get_auth_request con 0x7f5f08035ac0 auth_method 0 2024-09-18T10:13:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.163+0000 7f5f20865700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 0x7f5f08037f70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c001ed0 tx=0x7f5f0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.171+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5f1001e8f0 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.171+0000 7f5f18ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:37.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.171+0000 7f5f18ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:37.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.339+0000 7f5f22ac9700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:37.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.339+0000 7f5f22ac9700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:37.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.339+0000 7f5f22ac9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:37.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.339+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.13"]} v 0) v1 -- 0x7f5f1c090560 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.343 INFO:tasks.ceph.mon.b.smithi150.stderr:2024-09-18T10:13:37.339+0000 7f65bb623700 -1 mon.b@1(leader).osd e119 definitely_dead 0 2024-09-18T10:13:37.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.939+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.13"]}]=0 removed osd.13 v120) v1 ==== 87+0+0 (secure 0 0 0) 0x7f5f10017070 con 0x7f5f1c08d4c0 2024-09-18T10:13:37.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.939+0000 7f5f18ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:37.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:37.939+0000 7f5f18ff9700 10 monclient: _finish_command 2 = system:0 removed osd.13 2024-09-18T10:13:38.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f22ac9700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:38.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f22ac9700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:38.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f22ac9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.13"]} v 0) v1 -- 0x7f5f1c09d360 con 0x7f5f1c08d4c0 2024-09-18T10:13:38.055 INFO:tasks.ceph.mon.b.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f65bb623700 -1 mon.b@1(leader).osd e120 definitely_dead 0 2024-09-18T10:13:38.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f18ff9700 1 -- 172.21.15.150:0/1722090183 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.13"]}]=0 osd.13 does not exist. v120) v1 ==== 96+0+0 (secure 0 0 0) 0x7f5f10011360 con 0x7f5f1c08d4c0 2024-09-18T10:13:38.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f18ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.13"]}] 2024-09-18T10:13:38.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.051+0000 7f5f18ff9700 10 monclient: _finish_command 3 = system:0 osd.13 does not exist. 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 msgr2=0x7f5f08037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 0x7f5f08037f70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c001ed0 tx=0x7f5f0c009040 comp rx=0 tx=0).stop 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 10 monclient: shutdown 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 msgr2=0x7f5f1c08d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000c3c0 tx=0x7f5f10015000 comp rx=0 tx=0).stop 2024-09-18T10:13:38.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 shutdown_connections 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5f08035ac0 0x7f5f08037f70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f1c08ddd0 0x7f5f1c147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f1c08d4c0 0x7f5f1c08d890 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 --2- 172.21.15.150:0/1722090183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f04004810 0x7f5f1c08cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 >> 172.21.15.150:0/1722090183 conn(0x7f5f1c0a58a0 msgr2=0x7f5f1c09b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 shutdown_connections 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.055+0000 7f5f22ac9700 1 -- 172.21.15.150:0/1722090183 wait complete. 2024-09-18T10:13:38.059 INFO:tasks.workunit.client.0.smithi150.stderr:osd.13 does not exist. 2024-09-18T10:13:38.067 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm osd.15 2024-09-18T10:13:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: get_monmap_and_config 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: build_initial_monmap 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: monmap: 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:38.169399+0000 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:38.169399+0000 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:38.171 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:38.171 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 Processor -- start 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 -- start start 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: init 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f7809cf10 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-a con 0x7f0f780a0920 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 0x7f0f78139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-b con 0x7f0f780a0d20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7809e6e0 con 0x7f0f7809cf10 2024-09-18T10:13:38.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7809e2e0 con 0x7f0f780a0920 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7809dc20 con 0x7f0f780a0d20 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f7d424700 10 monclient(hunting): authenticate will time out at 1100.263226s 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f76ffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0f7809cf10 0x7f0f780a0210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:53650/0 (socket says 172.21.15.150:53650) 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f76ffd700 1 -- 172.21.15.150:0/1310271303 learned_addr learned my addr 172.21.15.150:0/1310271303 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 1 --2- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 0x7f0f78139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 1 --2- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): get_auth_request con 0x7f0f780a0d20 auth_method 0 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 10 monclient(hunting): get_auth_request con 0x7f0f780a0920 auth_method 0 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 1 -- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 msgr2=0x7f0f781379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 1 -- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 msgr2=0x7f0f781379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.167+0000 7f0f767fc700 1 --2- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f767fc700 1 --2- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 10 monclient(hunting): handle_auth_done global_id 44294 payload 274 2024-09-18T10:13:38.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 -- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 msgr2=0x7f0f781379a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 --2- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 -- 172.21.15.150:0/1310271303 >> v1:172.21.15.150:6789/0 conn(0x7f0f7809cf10 legacy=0x7f0f780a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 -- 172.21.15.150:0/1310271303 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7809d2e0 con 0x7f0f780a0d20 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 -- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 msgr2=0x7f0f78139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f777fe700 1 --2- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 0x7f0f78139ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d079e1a9a1b85dd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 1 -- 172.21.15.150:0/1310271303 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7000e070 con 0x7f0f780a0d20 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:38.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: dump: 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:38.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 1 -- 172.21.15.150:0/1310271303 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7000be40 con 0x7f0f780a0d20 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 1 -- 172.21.15.150:0/1310271303 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70012680 con 0x7f0f780a0d20 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f75ffb700 10 monclient: dump: 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.188 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f7d424700 5 monclient: authenticate success, global_id 44294 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f7d424700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.171+0000 7f0f7d424700 4 set_mon_vals no callback set 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: shutdown 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 msgr2=0x7f0f78139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 0x7f0f78139ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009e60 tx=0x7f0f70009780 comp rx=0 tx=0).stop 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- 172.21.15.150:0/1310271303 shutdown_connections 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- 172.21.15.150:0/1310271303 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0d20 0x7f0f78139ef0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- 172.21.15.150:0/1310271303 >> v2:172.21.15.142:3300/0 conn(0x7f0f780a0920 0x7f0f781379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- 172.21.15.150:0/1310271303 >> 172.21.15.150:0/1310271303 conn(0x7f0f780a5820 msgr2=0x7f0f780a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:38.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- 172.21.15.150:0/1310271303 shutdown_connections 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- 172.21.15.150:0/1310271303 wait complete. 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: build_initial_monmap 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: monmap: 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 Processor -- start 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- start start 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: init 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-a con 0x7f0f7809cf10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0920 0x7f0f7808e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-c con 0x7f0f780a0920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): picked mon.noname-b con 0x7f0f780a0d20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7809be30 con 0x7f0f7809cf10 2024-09-18T10:13:38.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780915f0 con 0x7f0f780a0d20 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780903b0 con 0x7f0f780a0920 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f7d424700 10 monclient(hunting): authenticate will time out at 1100.271719s 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40552/0 (socket says 172.21.15.150:40552) 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 1 -- 172.21.15.150:0/3291209976 learned_addr learned my addr 172.21.15.150:0/3291209976 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 10 monclient(hunting): get_auth_request con 0x7f0f780a0d20 auth_method 0 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.175+0000 7f0f777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0920 0x7f0f7808e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): get_auth_request con 0x7f0f780a0920 auth_method 0 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request con 0x7f0f7809cf10 auth_method 0 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient(hunting): handle_auth_done global_id 15301 payload 824 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0920 msgr2=0x7f0f7808e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0920 0x7f0f7808e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 msgr2=0x7f0f7808dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 msgr2=0x7f0f7808dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 msgr2=0x7f0f7808dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f76ffd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:38.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:38.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009a20 con 0x7f0f780a0d20 2024-09-18T10:13:38.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 msgr2=0x7f0f7808ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:38.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f777fe700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0f7809dd60 tx=0x7f0f70018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7001e640 con 0x7f0f780a0d20 2024-09-18T10:13:38.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:38.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: dump: 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: _finish_auth 0 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:38.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 5 monclient: authenticate success, global_id 15301 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7001e7a0 con 0x7f0f780a0d20 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 10 monclient: _renew_subs 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f78091310 con 0x7f0f780a0d20 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7001eb60 con 0x7f0f780a0d20 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f5effd700 10 monclient: dump: 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.199 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 10 monclient: _renew_subs 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f78091980 con 0x7f0f780a0d20 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.179+0000 7f0f74ff9700 4 set_mon_vals no callback set 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0f70024070 con 0x7f0f780a0d20 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f7d424700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f7d424700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f7d424700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f5effd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 0x7f0f64037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7808f7a0 con 0x7f0f780a0d20 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f76ffd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 0x7f0f64037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(120..120 src has 58..120) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f0f7004b980 con 0x7f0f780a0d20 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.183+0000 7f0f76ffd700 10 monclient: get_auth_request con 0x7f0f64035ac0 auth_method 0 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.187+0000 7f0f76ffd700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 0x7f0f64037f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006df0 tx=0x7f0f6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.191+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0f7001b420 con 0x7f0f780a0d20 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.191+0000 7f0f5effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.191+0000 7f0f5effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:38.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.367+0000 7f0f7d424700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:38.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.367+0000 7f0f7d424700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:38.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.367+0000 7f0f7d424700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:38.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:38.367+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.15"]} v 0) v1 -- 0x7f0f78091760 con 0x7f0f780a0d20 2024-09-18T10:13:38.371 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:38.367+0000 7fc32aa98700 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-18T10:13:38.667 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:38.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: get_monmap_and_config 2024-09-18T10:13:38.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: build_initial_monmap 2024-09-18T10:13:38.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: monmap: 2024-09-18T10:13:38.777 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:38.776444+0000 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:38.776444+0000 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:38.778 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:38.779 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 Processor -- start 2024-09-18T10:13:38.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 -- start start 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: init 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 0x7f71dc0a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-c con 0x7f71dc0a5f90 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-b con 0x7f71dc0990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71dc09e360 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71dc09d980 con 0x7f71dc09e360 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71dc09d780 con 0x7f71dc0a5f90 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc09fc70 con 0x7f71dc0990b0 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71e09e4700 10 monclient(hunting): authenticate will time out at 1107.702747s 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38876/0 (socket says 172.21.15.142:38876) 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3806040210 learned_addr learned my addr 172.21.15.142:0/3806040210 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): get_auth_request con 0x7f71dc0990b0 auth_method 0 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71da59c700 1 --2- 172.21.15.142:0/3806040210 >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 0x7f71dc0a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_done global_id 44300 payload 274 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3806040210 >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 msgr2=0x7f71dc0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 --2- 172.21.15.142:0/3806040210 >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 0x7f71dc0a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3806040210 >> v1:172.21.15.150:6789/0 conn(0x7f71dc09e360 legacy=0x7f71dc138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 monclient: found mon.noname-b 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3806040210 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0a6360 con 0x7f71dc0990b0 2024-09-18T10:13:38.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71da59c700 10 monclient: get_auth_request con 0x7f71dc0a5f90 auth_method 0 2024-09-18T10:13:38.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.775+0000 7f71da59c700 1 --2- 172.21.15.142:0/3806040210 >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 0x7f71dc0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3806040210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 msgr2=0x7f71dc0a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d9d9b700 1 --2- 172.21.15.142:0/3806040210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0099b0 tx=0x7f71cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1271fa8ca828a6af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 1 -- 172.21.15.142:0/3806040210 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc00bb50 con 0x7f71dc0990b0 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: dump: 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.792 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: _finish_auth 0 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 5 monclient: authenticate success, global_id 44300 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 1 -- 172.21.15.142:0/3806040210 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc00bcb0 con 0x7f71dc0990b0 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:38.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 4 set_mon_vals no callback set 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 1 -- 172.21.15.142:0/3806040210 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc011670 con 0x7f71dc0990b0 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71d959a700 10 monclient: dump: 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.794 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 10 monclient: shutdown 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 -- 172.21.15.142:0/3806040210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 msgr2=0x7f71dc0a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3806040210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0099b0 tx=0x7f71cc009250 comp rx=0 tx=0).stop 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 -- 172.21.15.142:0/3806040210 shutdown_connections 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3806040210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0990b0 0x7f71dc0a07b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3806040210 >> v2:172.21.15.150:3300/0 conn(0x7f71dc0a5f90 0x7f71dc0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 -- 172.21.15.142:0/3806040210 >> 172.21.15.142:0/3806040210 conn(0x7f71dc071680 msgr2=0x7f71dc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 -- 172.21.15.142:0/3806040210 shutdown_connections 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 -- 172.21.15.142:0/3806040210 wait complete. 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 10 monclient: build_initial_monmap 2024-09-18T10:13:38.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 10 monclient: monmap: 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.779+0000 7f71e09e4700 1 Processor -- start 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 -- start start 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient: init 2024-09-18T10:13:38.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc09e360 0x7f71dc08cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-b con 0x7f71dc09e360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-c con 0x7f71dc0a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 0x7f71dc147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): picked mon.noname-a con 0x7f71dc08da30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc090a40 con 0x7f71dc08da30 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0984b0 con 0x7f71dc09e360 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:38.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71dc0a43c0 con 0x7f71dc0a5f90 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71e09e4700 10 monclient(hunting): authenticate will time out at 1107.708936s 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 0x7f71dc147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38892/0 (socket says 172.21.15.142:38892) 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3930372028 learned_addr learned my addr 172.21.15.142:0/3930372028 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 0x7f71dc147af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39626/0 (socket says 172.21.15.142:39626) 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): get_auth_request con 0x7f71dc0a5f90 auth_method 0 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): get_auth_request con 0x7f71dc08da30 auth_method 0 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc09e360 0x7f71dc08cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): get_auth_request con 0x7f71dc09e360 auth_method 0 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:38.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71dad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71da59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient(hunting): handle_auth_done global_id 44303 payload 824 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc09e360 msgr2=0x7f71dc08cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc09e360 0x7f71dc08cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 msgr2=0x7f71dc147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:38.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 0x7f71dc147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 monclient: found mon.noname-c 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.783+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3930372028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc0094f0 con 0x7f71dc0a5f90 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d9d9b700 1 -- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 msgr2=0x7f71dc08d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d9d9b700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f71cc006fd0 tx=0x7f71cc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc006a90 con 0x7f71dc0a5f90 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: dump: 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:38.801 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 5 monclient: authenticate success, global_id 44303 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc020070 con 0x7f71dc0a5f90 2024-09-18T10:13:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _renew_subs 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc011e80 con 0x7f71dc0a5f90 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71dc08edd0 con 0x7f71dc0a5f90 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d8d99700 4 set_mon_vals no callback set 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 10 monclient: dump: 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _renew_subs 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71dc08e8c0 con 0x7f71dc0a5f90 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc08f580 con 0x7f71dc0a5f90 2024-09-18T10:13:38.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f71cc022030 con 0x7f71dc0a5f90 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 0x7f71bc037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(120..120 src has 58..120) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f71cc04a830 con 0x7f71dc0a5f90 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71da59c700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 0x7f71bc037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.787+0000 7f71da59c700 10 monclient: get_auth_request con 0x7f71bc035af0 auth_method 0 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.791+0000 7f71da59c700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 0x7f71bc037fa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f71c400adc0 tx=0x7f71c4006f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.799+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f71cc01b660 con 0x7f71dc0a5f90 2024-09-18T10:13:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.799+0000 7f71d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:38.799+0000 7f71d27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71e09e4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:39.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71e09e4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:39.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71e09e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f71dc08f580 con 0x7f71dc0a5f90 2024-09-18T10:13:39.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71d27fc700 1 -- 172.21.15.142:0/3930372028 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f71cc02cd90 con 0x7f71dc0a5f90 2024-09-18T10:13:39.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:39.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.023+0000 7f71d27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:39.028 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:39.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 msgr2=0x7f71bc037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 0x7f71bc037fa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f71c400adc0 tx=0x7f71c4006f90 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 10 monclient: shutdown 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 msgr2=0x7f71dc08d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f71cc006fd0 tx=0x7f71cc016000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 shutdown_connections 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71bc035af0 0x7f71bc037fa0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71dc08da30 0x7f71dc147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71dc0a5f90 0x7f71dc08d4f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 --2- 172.21.15.142:0/3930372028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71dc09e360 0x7f71dc08cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 >> 172.21.15.142:0/3930372028 conn(0x7f71dc071680 msgr2=0x7f71dc09e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 shutdown_connections 2024-09-18T10:13:39.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.027+0000 7f71e09e4700 1 -- 172.21.15.142:0/3930372028 wait complete. 2024-09-18T10:13:39.042 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:13:39.042 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:13:39.042 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:13:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: get_monmap_and_config 2024-09-18T10:13:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: monmap: 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:39.148725+0000 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:39.148725+0000 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:39.150 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 Processor -- start 2024-09-18T10:13:39.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 -- start start 2024-09-18T10:13:39.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: init 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f90b006f3b0 0x7f90b00989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-a con 0x7f90b006f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-b con 0x7f90b0098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 0x7f90b0135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-c con 0x7f90b0133660 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b009b030 con 0x7f90b006f3b0 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b0099830 con 0x7f90b0133660 2024-09-18T10:13:39.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b009c330 con 0x7f90b0098f00 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b654c700 10 monclient(hunting): authenticate will time out at 1108.074168s 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f90b006f3b0 0x7f90b00989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f90b006f3b0 0x7f90b00989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39642/0 (socket says 172.21.15.142:39642) 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38906/0 (socket says 172.21.15.142:38906) 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 -- 172.21.15.142:0/228336479 learned_addr learned my addr 172.21.15.142:0/228336479 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): get_auth_request con 0x7f90b006f3b0 auth_method 0 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/228336479 >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 0x7f90b0135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): get_auth_request con 0x7f90b0098f00 auth_method 0 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): get_auth_request con 0x7f90b0133660 auth_method 0 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient(hunting): handle_auth_done global_id 54102 payload 274 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 -- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 msgr2=0x7f90b00992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 -- 172.21.15.142:0/228336479 >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 msgr2=0x7f90b0135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 --2- 172.21.15.142:0/228336479 >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 0x7f90b0135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 monclient: found mon.noname-a 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:39.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b554a700 1 -- 172.21.15.142:0/228336479 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b009c530 con 0x7f90b006f3b0 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/228336479 >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 0x7f90b0135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.147+0000 7f90b4d49700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b554a700 1 -- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b006f3b0 msgr2=0x7f90b00989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b554a700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b006f3b0 0x7f90b00989c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f90b006fd30 tx=0x7f90ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7470031afa363e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 1 -- 172.21.15.142:0/228336479 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00b850 con 0x7f90b006f3b0 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: dump: 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.159 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 5 monclient: authenticate success, global_id 54102 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 1 -- 172.21.15.142:0/228336479 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac00b9b0 con 0x7f90b006f3b0 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 1 -- 172.21.15.142:0/228336479 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00bd50 con 0x7f90b006f3b0 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 4 set_mon_vals no callback set 2024-09-18T10:13:39.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90a7fff700 10 monclient: dump: 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.161 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 10 monclient: shutdown 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 -- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b006f3b0 msgr2=0x7f90b00989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b006f3b0 0x7f90b00989c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f90b006fd30 tx=0x7f90ac009250 comp rx=0 tx=0).stop 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 -- 172.21.15.142:0/228336479 shutdown_connections 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 --2- 172.21.15.142:0/228336479 >> v2:172.21.15.150:3300/0 conn(0x7f90b0133660 0x7f90b0135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b0098f00 0x7f90b00992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 --2- 172.21.15.142:0/228336479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b006f3b0 0x7f90b00989c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 -- 172.21.15.142:0/228336479 >> 172.21.15.142:0/228336479 conn(0x7f90b0070ef0 msgr2=0x7f90b00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 -- 172.21.15.142:0/228336479 shutdown_connections 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 1 -- 172.21.15.142:0/228336479 wait complete. 2024-09-18T10:13:39.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.151+0000 7f90b654c700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient: monmap: 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 Processor -- start 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 -- start start 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient: init 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 0x7f90b013f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-c con 0x7f90b006f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0098f00 0x7f90b0131ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-a con 0x7f90b0098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0133660 0x7f90b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): picked mon.noname-b con 0x7f90b0133660 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b006fa20 con 0x7f90b0098f00 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b009ed40 con 0x7f90b0133660 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0127f60 con 0x7f90b006f3b0 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 0x7f90b013f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b654c700 10 monclient(hunting): authenticate will time out at 1108.082386s 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0098f00 0x7f90b0131ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 0x7f90b013f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38912/0 (socket says 172.21.15.142:38912) 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 -- 172.21.15.142:0/2588147140 learned_addr learned my addr 172.21.15.142:0/2588147140 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): get_auth_request con 0x7f90b006f3b0 auth_method 0 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): get_auth_request con 0x7f90b0098f00 auth_method 0 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0133660 0x7f90b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 10 monclient(hunting): get_auth_request con 0x7f90b0133660 auth_method 0 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient(hunting): handle_auth_done global_id 44309 payload 824 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0133660 msgr2=0x7f90b006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0133660 0x7f90b006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0098f00 msgr2=0x7f90b0131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0098f00 0x7f90b0131ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b554a700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a0009420 con 0x7f90b006f3b0 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.155+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90b0133660 0x7f90b006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b554a700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 msgr2=0x7f90b013f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b554a700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 0x7f90b013f3e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006fd0 tx=0x7f90ac017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac011d00 con 0x7f90b006f3b0 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: dump: 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.174 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac011e80 con 0x7f90b006f3b0 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 5 monclient: authenticate success, global_id 44309 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 monclient: _renew_subs 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a8009a20 con 0x7f90b006f3b0 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac024690 con 0x7f90b006f3b0 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 10 monclient: dump: 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a77fe700 4 set_mon_vals no callback set 2024-09-18T10:13:39.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 monclient: _renew_subs 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac0094f0 con 0x7f90b006f3b0 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 0x7f90b006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b00648a0 con 0x7f90b013f920 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f90ac02d440 con 0x7f90b006f3b0 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 0x7f90b006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient: get_auth_request con 0x7f90b013f920 auth_method 0 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 10 monclient(hunting): handle_auth_done global_id 54108 payload 824 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9094035ac0 0x7f9094037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b4d49700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90b0072a80 con 0x7f90b013f920 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(120..120 src has 58..120) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f90ac04a240 con 0x7f90b006f3b0 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9094035ac0 0x7f9094037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.159+0000 7f90b5d4b700 10 monclient: get_auth_request con 0x7f9094035ac0 auth_method 0 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90b4d49700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 msgr2=0x7f90b006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90b4d49700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 0x7f90b006d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90a0011bf0 tx=0x7f90a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a001d480 con 0x7f90b013f920 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:39.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 10 monclient: dump: 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f90a001d5e0 con 0x7f90b013f920 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 msgr2=0x7f90b006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 0x7f90b006d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90a0011bf0 tx=0x7f90a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 msgr2=0x7f90b006d900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90a57fa700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b013f920 0x7f90b006d900 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f90a0011bf0 tx=0x7f90a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.163+0000 7f90b5d4b700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9094035ac0 0x7f9094037f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f90b0128660 tx=0x7f90a800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b654c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b654c700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 0x7f90b006de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b654c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0140210 con 0x7f90b0132010 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 0x7f90b006de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient: get_auth_request con 0x7f90b0132010 auth_method 0 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-18T10:13:39.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.175+0000 7f90b4d49700 1 -- 172.21.15.142:0/2588147140 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f90b0126d20 con 0x7f90b0132010 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b4d49700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 msgr2=0x7f90b006de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b4d49700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 0x7f90b006de40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f90a0010010 tx=0x7f90a001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0025040 con 0x7f90b0132010 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 10 monclient: dump: 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f90a001c3e0 con 0x7f90b0132010 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 msgr2=0x7f90b006de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 0x7f90b006de40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f90a0010010 tx=0x7f90a001d000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 msgr2=0x7f90b006de40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90a57fa700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f90b0132010 0x7f90b006de40 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f90a0010010 tx=0x7f90a001d000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:39.185 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.186 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.187 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:39.188 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.189 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.192 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.193 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.194 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.195 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.196 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.197 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:39.198 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.199 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.200 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:39.200 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9094035ac0 msgr2=0x7f9094037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9094035ac0 0x7f9094037f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f90b0128660 tx=0x7f90a800b040 comp rx=0 tx=0).stop 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 10 monclient: shutdown 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 msgr2=0x7f90b013f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 --2- 172.21.15.142:0/2588147140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f90b006f3b0 0x7f90b013f3e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006fd0 tx=0x7f90ac017000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b5d4b700 1 -- 172.21.15.142:0/2588147140 reap_dead start 2024-09-18T10:13:39.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b5d4b700 1 -- 172.21.15.142:0/2588147140 reap_dead start 2024-09-18T10:13:39.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 shutdown_connections 2024-09-18T10:13:39.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 >> 172.21.15.142:0/2588147140 conn(0x7f90b0070ef0 msgr2=0x7f90b0132880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 shutdown_connections 2024-09-18T10:13:39.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.179+0000 7f90b654c700 1 -- 172.21.15.142:0/2588147140 wait complete. 2024-09-18T10:13:39.202 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:39.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:13:39.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:13:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 10 monclient: get_monmap_and_config 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 10 monclient: monmap: 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:39.309976+0000 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:39.309976+0000 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:39.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 1 Processor -- start 2024-09-18T10:13:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 1 -- start start 2024-09-18T10:13:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.307+0000 7f18a2af9700 10 monclient: init 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f189c0999e0 0x7f189c0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-a con 0x7f189c0999e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f189c09c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-b con 0x7f189c137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189c0a20c0 con 0x7f189c09c410 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f189c09a000 con 0x7f189c0999e0 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0a1c80 con 0x7f189c137910 2024-09-18T10:13:39.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a2af9700 10 monclient(hunting): authenticate will time out at 1108.237093s 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f189c0999e0 0x7f189c0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38916/0 (socket says 172.21.15.142:38916) 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): get_auth_request con 0x7f189c0999e0 auth_method 0 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 -- 172.21.15.142:0/3204903291 learned_addr learned my addr 172.21.15.142:0/3204903291 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): get_auth_request con 0x7f189c137910 auth_method 0 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient(hunting): handle_auth_done global_id 44312 payload 274 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 -- 172.21.15.142:0/3204903291 >> v2:172.21.15.142:3300/0 conn(0x7f189c0999e0 msgr2=0x7f189c0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3204903291 >> v2:172.21.15.142:3300/0 conn(0x7f189c0999e0 0x7f189c0a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 -- 172.21.15.142:0/3204903291 >> v1:172.21.15.142:6789/0 conn(0x7f189c09c410 legacy=0x7f189c0a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 monclient: found mon.noname-b 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.311+0000 7f18a22f8700 1 -- 172.21.15.142:0/3204903291 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c09a580 con 0x7f189c137910 2024-09-18T10:13:39.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a22f8700 1 -- 172.21.15.142:0/3204903291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 msgr2=0x7f189c139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3204903291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1894009f20 tx=0x7f1894009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150f895809c612b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 1 -- 172.21.15.142:0/3204903291 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189400e070 con 0x7f189c137910 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: dump: 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.320 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 5 monclient: authenticate success, global_id 44312 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 1 -- 172.21.15.142:0/3204903291 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189400bbe0 con 0x7f189c137910 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 1 -- 172.21.15.142:0/3204903291 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18940092e0 con 0x7f189c137910 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 4 set_mon_vals no callback set 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a0af5700 10 monclient: dump: 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 10 monclient: shutdown 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 -- 172.21.15.142:0/3204903291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 msgr2=0x7f189c139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3204903291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1894009f20 tx=0x7f1894009840 comp rx=0 tx=0).stop 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 -- 172.21.15.142:0/3204903291 shutdown_connections 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3204903291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c137910 0x7f189c139cf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3204903291 >> v2:172.21.15.142:3300/0 conn(0x7f189c0999e0 0x7f189c0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 -- 172.21.15.142:0/3204903291 >> 172.21.15.142:0/3204903291 conn(0x7f189c0a5790 msgr2=0x7f189c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 -- 172.21.15.142:0/3204903291 shutdown_connections 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 1 -- 172.21.15.142:0/3204903291 wait complete. 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.315+0000 7f18a2af9700 10 monclient: monmap: 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:39.323 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:39.324 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 Processor -- start 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 -- start start 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient: init 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 0x7f189c093f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-c con 0x7f189c0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c09c410 0x7f189c06f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-b con 0x7f189c09c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f189c137910 0x7f189c06afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): picked mon.noname-a con 0x7f189c137910 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c06e630 con 0x7f189c137910 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c06eb60 con 0x7f189c09c410 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0994c0 con 0x7f189c0999e0 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a2af9700 10 monclient(hunting): authenticate will time out at 1108.245718s 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 0x7f189c093f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f189c137910 0x7f189c06afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 0x7f189c093f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38920/0 (socket says 172.21.15.142:38920) 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 -- 172.21.15.142:0/3672824495 learned_addr learned my addr 172.21.15.142:0/3672824495 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): get_auth_request con 0x7f189c0999e0 auth_method 0 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): get_auth_request con 0x7f189c137910 auth_method 0 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c09c410 0x7f189c06f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): get_auth_request con 0x7f189c09c410 auth_method 0 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a12f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient(hunting): handle_auth_done global_id 44315 payload 824 2024-09-18T10:13:39.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c09c410 msgr2=0x7f189c06f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f189c09c410 0x7f189c06f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f189c137910 msgr2=0x7f189c06afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f189c137910 0x7f189c06afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a1af7700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894009ae0 con 0x7f189c0999e0 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.319+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f189c137910 0x7f189c06afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a1af7700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 msgr2=0x7f189c093f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a1af7700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 0x7f189c093f10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f189800ec80 tx=0x7f18980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18980096b0 con 0x7f189c0999e0 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: dump: 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189800ca60 con 0x7f189c0999e0 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 5 monclient: authenticate success, global_id 44315 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 monclient: _renew_subs 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189c136ac0 con 0x7f189c0999e0 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189800ce20 con 0x7f189c0999e0 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 10 monclient: dump: 2024-09-18T10:13:39.335 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1893fff700 4 set_mon_vals no callback set 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 monclient: _renew_subs 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c06d300 con 0x7f189c0999e0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1898013680 con 0x7f189c0999e0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1880035af0 0x7f1880037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(120..120 src has 58..120) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f1898048df0 con 0x7f189c0999e0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 0x7f189c06b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c0647f0 con 0x7f189c06b510 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 0x7f189c06b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient: get_auth_request con 0x7f189c06b510 auth_method 0 2024-09-18T10:13:39.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a12f6700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1880035af0 0x7f1880037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a12f6700 10 monclient: get_auth_request con 0x7f1880035af0 auth_method 0 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 10 monclient(hunting): handle_auth_done global_id 44318 payload 824 2024-09-18T10:13:39.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.323+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f189c06be20 con 0x7f189c06b510 2024-09-18T10:13:39.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 msgr2=0x7f189c06b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 0x7f189c06b8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f189c0a1710 tx=0x7f1894017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189401f620 con 0x7f189c06b510 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 10 monclient: dump: 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.340 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f18a12f6700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1880035af0 0x7f1880037fa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f189c06d970 tx=0x7f188c007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1894027420 con 0x7f189c06b510 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 msgr2=0x7f189c06b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 0x7f189c06b8e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f189c0a1710 tx=0x7f1894017000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 msgr2=0x7f189c06b8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.327+0000 7f1891ffb700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c06b510 0x7f189c06b8e0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f189c0a1710 tx=0x7f1894017000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a2af9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:39.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 0x7f189c097db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a2af9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c06e970 con 0x7f189c0979e0 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 0x7f189c097db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient: get_auth_request con 0x7f189c0979e0 auth_method 0 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 10 monclient(hunting): handle_auth_done global_id 44321 payload 824 2024-09-18T10:13:39.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f189c072a80 con 0x7f189c0979e0 2024-09-18T10:13:39.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 msgr2=0x7f189c097db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.339+0000 7f18a22f8700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 0x7f189c097db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f189c06e910 tx=0x7f1894021070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189401fc20 con 0x7f189c0979e0 2024-09-18T10:13:39.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 10 monclient: dump: 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.345 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f189401e410 con 0x7f189c0979e0 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 msgr2=0x7f189c097db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 0x7f189c097db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f189c06e910 tx=0x7f1894021070 comp rx=0 tx=0).stop 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 msgr2=0x7f189c097db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f1891ffb700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0979e0 0x7f189c097db0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f189c06e910 tx=0x7f1894021070 comp rx=0 tx=0).stop 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:13:39.347 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.348 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.349 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:39.350 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.351 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.353 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:39.354 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:39.355 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.356 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.357 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:13:39.358 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.359 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:39.359 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:39.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1880035af0 msgr2=0x7f1880037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1880035af0 0x7f1880037fa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f189c06d970 tx=0x7f188c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 10 monclient: shutdown 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 msgr2=0x7f189c093f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 --2- 172.21.15.142:0/3672824495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f189c0999e0 0x7f189c093f10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f189800ec80 tx=0x7f18980091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 reap_dead start 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a22f8700 1 -- 172.21.15.142:0/3672824495 reap_dead start 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 shutdown_connections 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 >> 172.21.15.142:0/3672824495 conn(0x7f189c0a5790 msgr2=0x7f189c096690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 shutdown_connections 2024-09-18T10:13:39.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.343+0000 7f18a2af9700 1 -- 172.21.15.142:0/3672824495 wait complete. 2024-09-18T10:13:39.362 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:39.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:13:39.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:13:39.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.375+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.15"]}]=0 removed osd.15 v121) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0f700263d0 con 0x7f0f780a0d20 2024-09-18T10:13:39.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.375+0000 7f0f5effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:39.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.375+0000 7f0f5effd700 10 monclient: _finish_command 2 = system:0 removed osd.15 2024-09-18T10:13:39.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.471+0000 7f941359e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:39.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.471+0000 7f941359e700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.471+0000 7f941359e700 10 monclient: monmap: 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:39.475601+0000 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:39.475601+0000 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:39.477 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:39.478 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:39.478 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 Processor -- start 2024-09-18T10:13:39.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 -- start start 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient: init 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f941409c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9414134880 0x7f9414136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): picked mon.noname-a con 0x7f9414134880 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 0x7f9414139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): picked mon.noname-b con 0x7f94141371a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140a6620 con 0x7f941409c2c0 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941409ab70 con 0x7f9414134880 2024-09-18T10:13:39.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94140a1c70 con 0x7f94141371a0 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941359e700 10 monclient(hunting): authenticate will time out at 1108.401106s 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 0x7f9414139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9414134880 0x7f9414136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941259c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f941409c2c0 0x7f94140999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37222/0 (socket says 172.21.15.142:37222) 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f941259c700 1 -- 172.21.15.142:0/1067524906 learned_addr learned my addr 172.21.15.142:0/1067524906 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): get_auth_request con 0x7f9414134880 auth_method 0 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): get_auth_request con 0x7f94141371a0 auth_method 0 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient(hunting): handle_auth_done global_id 54120 payload 274 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 -- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 msgr2=0x7f9414139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 0x7f9414139580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 -- 172.21.15.142:0/1067524906 >> v1:172.21.15.150:6789/0 conn(0x7f941409c2c0 legacy=0x7f94140999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 monclient: found mon.noname-a 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9411d9b700 1 -- 172.21.15.142:0/1067524906 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941409c690 con 0x7f9414134880 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.475+0000 7f9412d9d700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 0x7f9414139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 -- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9414134880 msgr2=0x7f9414136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9414134880 0x7f9414136c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0098e0 tx=0x7f93fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1295c6f104192961 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.479+0000 7f0f7d424700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:39.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.479+0000 7f0f7d424700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:39.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.479+0000 7f0f7d424700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.485 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.479+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["osd.15"]} v 0) v1 -- 0x7f0f78091760 con 0x7f0f780a0d20 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 1 -- 172.21.15.142:0/1067524906 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00b850 con 0x7f9414134880 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: dump: 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 5 monclient: authenticate success, global_id 54120 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 1 -- 172.21.15.142:0/1067524906 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc00b9b0 con 0x7f9414134880 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 4 set_mon_vals no callback set 2024-09-18T10:13:39.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 1 -- 172.21.15.142:0/1067524906 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00bd50 con 0x7f9414134880 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941159a700 10 monclient: dump: 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: shutdown 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9414134880 msgr2=0x7f9414136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9414134880 0x7f9414136c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0098e0 tx=0x7f93fc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- 172.21.15.142:0/1067524906 shutdown_connections 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94141371a0 0x7f9414139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- 172.21.15.142:0/1067524906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9414134880 0x7f9414136c60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- 172.21.15.142:0/1067524906 >> 172.21.15.142:0/1067524906 conn(0x7f94140a5720 msgr2=0x7f94140a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- 172.21.15.142:0/1067524906 shutdown_connections 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- 172.21.15.142:0/1067524906 wait complete. 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: monmap: 2024-09-18T10:13:39.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 Processor -- start 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- start start 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: init 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f941409c2c0 0x7f941412b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): picked mon.noname-a con 0x7f941409c2c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 0x7f941406c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): picked mon.noname-c con 0x7f9414134880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94141371a0 0x7f941406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): picked mon.noname-b con 0x7f94141371a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941406f8f0 con 0x7f941409c2c0 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941406ec80 con 0x7f94141371a0 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941412ab30 con 0x7f9414134880 2024-09-18T10:13:39.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941359e700 10 monclient(hunting): authenticate will time out at 1108.406845s 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 0x7f941406c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f941409c2c0 0x7f941412b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 0x7f941406c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38956/0 (socket says 172.21.15.142:38956) 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 -- 172.21.15.142:0/2657927669 learned_addr learned my addr 172.21.15.142:0/2657927669 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): get_auth_request con 0x7f9414134880 auth_method 0 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): get_auth_request con 0x7f941409c2c0 auth_method 0 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94141371a0 0x7f941406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 10 monclient(hunting): get_auth_request con 0x7f94141371a0 auth_method 0 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient(hunting): handle_auth_done global_id 44327 payload 824 2024-09-18T10:13:39.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94141371a0 msgr2=0x7f941406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94141371a0 0x7f941406c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f941409c2c0 msgr2=0x7f941412b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f941409c2c0 0x7f941412b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9411d9b700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c009a20 con 0x7f9414134880 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.479+0000 7f9412d9d700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94141371a0 0x7f941406c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f9411d9b700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 msgr2=0x7f941406c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f9411d9b700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 0x7f941406c4b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0093f0 tx=0x7f93fc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc011bf0 con 0x7f9414134880 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: dump: 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 5 monclient: authenticate success, global_id 44327 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 monclient: _renew_subs 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc020070 con 0x7f9414134880 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93fc009420 con 0x7f9414134880 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc01b450 con 0x7f9414134880 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 10 monclient: dump: 2024-09-18T10:13:39.502 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f9410d99700 4 set_mon_vals no callback set 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 monclient: _renew_subs 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941412aec0 con 0x7f9414134880 2024-09-18T10:13:39.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f93fc022030 con 0x7f9414134880 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 0x7f941402ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414064750 con 0x7f941406cf30 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.483+0000 7f940a7fc700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93f4035ac0 0x7f93f4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 0x7f941402ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(121..121 src has 58..121) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f93fc04a510 con 0x7f9414134880 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient: get_auth_request con 0x7f941406cf30 auth_method 0 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f9412d9d700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93f4035ac0 0x7f93f4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f9412d9d700 10 monclient: get_auth_request con 0x7f93f4035ac0 auth_method 0 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 10 monclient(hunting): handle_auth_done global_id 15319 payload 824 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f941406e780 con 0x7f941406cf30 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 msgr2=0x7f941402ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f941259c700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 0x7f941402ae80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f940400eac0 tx=0x7f9404007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f9412d9d700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93f4035ac0 0x7f93f4037f70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f941412b830 tx=0x7f940c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940400cc40 con 0x7f941406cf30 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f940a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f940a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:39.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.487+0000 7f940a7fc700 10 monclient: dump: 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.506 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f940400cda0 con 0x7f941406cf30 2024-09-18T10:13:39.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 msgr2=0x7f941402ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 0x7f941402ae80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f940400eac0 tx=0x7f9404007f90 comp rx=0 tx=0).stop 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 msgr2=0x7f941402ae80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.491+0000 7f940a7fc700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941406cf30 0x7f941402ae80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941359e700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 0x7f941405ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94140a6620 con 0x7f941405e890 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 0x7f941405ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient: get_auth_request con 0x7f941405e890 auth_method 0 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.499+0000 7f941259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941259c700 10 monclient(hunting): handle_auth_done global_id 15322 payload 824 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941259c700 1 -- 172.21.15.142:0/2657927669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9414134630 con 0x7f941405e890 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941259c700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 msgr2=0x7f941405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941259c700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 0x7f941405ec90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9404007360 tx=0x7f9404016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404029400 con 0x7f941405e890 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 10 monclient: dump: 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f9404014480 con 0x7f941405e890 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 msgr2=0x7f941405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 0x7f941405ec90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9404007360 tx=0x7f9404016000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 msgr2=0x7f941405ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f940a7fc700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f941405e890 0x7f941405ec90 secure :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9404007360 tx=0x7f9404016000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:39.512 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.513 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.514 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:39.515 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:39.516 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:39.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.518 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:39.519 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:39.521 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:39.522 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:39.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.524 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:39.525 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93f4035ac0 msgr2=0x7f93f4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93f4035ac0 0x7f93f4037f70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f941412b830 tx=0x7f940c00b040 comp rx=0 tx=0).stop 2024-09-18T10:13:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 10 monclient: shutdown 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 msgr2=0x7f941406c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f941359e700 1 --2- 172.21.15.142:0/2657927669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9414134880 0x7f941406c4b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0093f0 tx=0x7f93fc016000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f9412d9d700 1 -- 172.21.15.142:0/2657927669 reap_dead start 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.503+0000 7f9412d9d700 1 -- 172.21.15.142:0/2657927669 reap_dead start 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.507+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 shutdown_connections 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.507+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 >> 172.21.15.142:0/2657927669 conn(0x7f94140a5720 msgr2=0x7f9414136450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.507+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 shutdown_connections 2024-09-18T10:13:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.507+0000 7f941359e700 1 -- 172.21.15.142:0/2657927669 wait complete. 2024-09-18T10:13:39.528 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:39.507+0000 7fc32aa98700 -1 mon.a@0(leader).osd e121 definitely_dead 0 2024-09-18T10:13:39.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:39.529 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:13:39.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:13:39.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f5effd700 1 -- 172.21.15.150:0/3291209976 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["osd.15"]}]=0 osd.15 does not exist. v121) v1 ==== 96+0+0 (secure 0 0 0) 0x7f0f700265b0 con 0x7f0f780a0d20 2024-09-18T10:13:39.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f5effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["osd.15"]}] 2024-09-18T10:13:39.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f5effd700 10 monclient: _finish_command 3 = system:0 osd.15 does not exist. 2024-09-18T10:13:39.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 msgr2=0x7f0f64037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 0x7f0f64037f70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006df0 tx=0x7f0f6c008040 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 10 monclient: shutdown 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 msgr2=0x7f0f7808ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0f7809dd60 tx=0x7f0f70018000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 shutdown_connections 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f64035ac0 0x7f0f64037f70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f780a0d20 0x7f0f7808ea40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0f780a0920 0x7f0f7808e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 --2- 172.21.15.150:0/3291209976 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0f7809cf10 0x7f0f7808dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.507+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 >> 172.21.15.150:0/3291209976 conn(0x7f0f780a5820 msgr2=0x7f0f7809b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.511+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 shutdown_connections 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.511+0000 7f0f7d424700 1 -- 172.21.15.150:0/3291209976 wait complete. 2024-09-18T10:13:39.531 INFO:tasks.workunit.client.0.smithi150.stderr:osd.15 does not exist. 2024-09-18T10:13:39.532 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd set-require-min-compat-client luminous 2024-09-18T10:13:39.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:39.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: monmap: 2024-09-18T10:13:39.625 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:39.625 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:39.625133+0000 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:39.625133+0000 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:39.626 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 Processor -- start 2024-09-18T10:13:39.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 -- start start 2024-09-18T10:13:39.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: init 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 0x7fd8280999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-a con 0x7fd828096a70 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd828096e40 0x7fd828097250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-c con 0x7fd828096e40 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd828097790 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd828098880 con 0x7fd828097790 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd82809e120 con 0x7fd828096a70 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd828098eb0 con 0x7fd828096e40 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.623+0000 7fd82dc9e700 10 monclient(hunting): authenticate will time out at 1101.719928s 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd828096e40 0x7fd828097250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd828096e40 0x7fd828097250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33286/0 (socket says 172.21.15.150:33286) 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 -- 172.21.15.150:0/2866205078 learned_addr learned my addr 172.21.15.150:0/2866205078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): get_auth_request con 0x7fd828096e40 auth_method 0 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 1 --2- 172.21.15.150:0/2866205078 >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 0x7fd8280999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 10 monclient(hunting): get_auth_request con 0x7fd828096a70 auth_method 0 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient(hunting): handle_auth_done global_id 15325 payload 274 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 -- 172.21.15.150:0/2866205078 >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 msgr2=0x7fd8280999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 --2- 172.21.15.150:0/2866205078 >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 0x7fd8280999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 -- 172.21.15.150:0/2866205078 >> v1:172.21.15.142:6789/0 conn(0x7fd828097790 legacy=0x7fd828135530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:39.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 -- 172.21.15.150:0/2866205078 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8280a2c10 con 0x7fd828096e40 2024-09-18T10:13:39.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8277fe700 1 --2- 172.21.15.150:0/2866205078 >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 0x7fd8280999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:39.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 -- 172.21.15.150:0/2866205078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096e40 msgr2=0x7fd828097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd826ffd700 1 --2- 172.21.15.150:0/2866205078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096e40 0x7fd828097250 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd82806f6e0 tx=0x7fd81c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ec0634cb685456 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 1 -- 172.21.15.150:0/2866205078 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00ba70 con 0x7fd828096e40 2024-09-18T10:13:39.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: dump: 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd82dc9e700 5 monclient: authenticate success, global_id 15325 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 1 -- 172.21.15.150:0/2866205078 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c00bbd0 con 0x7fd828096e40 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd82dc9e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd82dc9e700 4 set_mon_vals no callback set 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 1 -- 172.21.15.150:0/2866205078 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c011420 con 0x7fd828096e40 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.627+0000 7fd8267fc700 10 monclient: dump: 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: shutdown 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- 172.21.15.150:0/2866205078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096e40 msgr2=0x7fd828097250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/2866205078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096e40 0x7fd828097250 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd82806f6e0 tx=0x7fd81c009250 comp rx=0 tx=0).stop 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- 172.21.15.150:0/2866205078 shutdown_connections 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/2866205078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096e40 0x7fd828097250 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/2866205078 >> v2:172.21.15.142:3300/0 conn(0x7fd828096a70 0x7fd8280999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- 172.21.15.150:0/2866205078 >> 172.21.15.150:0/2866205078 conn(0x7fd828068460 msgr2=0x7fd8280a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- 172.21.15.150:0/2866205078 shutdown_connections 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- 172.21.15.150:0/2866205078 wait complete. 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: monmap: 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:39.639 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 Processor -- start 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- start start 2024-09-18T10:13:39.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: init 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: get_monmap_and_config 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: monmap: 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:39.639965+0000 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:39.639965+0000 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:39.644 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 Processor -- start 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 -- start start 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: init 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c840a61a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4c84132490 0x7f4c841373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-c con 0x7f4c84132490 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:39.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c841328c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8409dcd0 con 0x7f4c841328c0 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8409fb40 con 0x7f4c840a61a0 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8409cdd0 con 0x7f4c84132490 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8b882700 10 monclient(hunting): authenticate will time out at 1108.565648s 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c89e1f700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4c841328c0 0x7f4c84139910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51318/0 (socket says 172.21.15.142:51318) 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c89e1f700 1 -- 172.21.15.142:0/3087788480 learned_addr learned my addr 172.21.15.142:0/3087788480 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c8961e700 1 --1- 172.21.15.142:0/3087788480 >> v1:172.21.15.150:6789/0 conn(0x7f4c840a61a0 0x7f4c8409f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37234/0 (socket says 172.21.15.142:37234) 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c88e1d700 1 --2- 172.21.15.142:0/3087788480 >> v2:172.21.15.150:3300/0 conn(0x7f4c84132490 0x7f4c841373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c88e1d700 10 monclient(hunting): get_auth_request con 0x7f4c84132490 auth_method 0 2024-09-18T10:13:39.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c88e1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c88e1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.639+0000 7f4c88e1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c89e1f700 1 -- 172.21.15.142:0/3087788480 >> v1:172.21.15.142:6789/0 conn(0x7f4c841328c0 legacy=0x7f4c84139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c8961e700 1 -- 172.21.15.142:0/3087788480 >> v1:172.21.15.150:6789/0 conn(0x7f4c840a61a0 legacy=0x7f4c8409f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_done global_id 15331 payload 274 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 -- 172.21.15.142:0/3087788480 >> v1:172.21.15.150:6789/0 conn(0x7f4c840a61a0 legacy=0x7f4c8409f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 -- 172.21.15.142:0/3087788480 >> v1:172.21.15.142:6789/0 conn(0x7f4c841328c0 legacy=0x7f4c84139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c7bfff700 1 -- 172.21.15.142:0/3087788480 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881166586 0 0) 0x7f4c8409dcd0 con 0x7f4c841328c0 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 -- 172.21.15.142:0/3087788480 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8409d6a0 con 0x7f4c84132490 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c7bfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:13:39.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 -- 172.21.15.142:0/3087788480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c84132490 msgr2=0x7f4c841373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.643+0000 7f4c88e1d700 1 --2- 172.21.15.142:0/3087788480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c84132490 0x7f4c841373c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4c740099e0 tx=0x7f4c740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c440c27bde5c346 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-b con 0x7fd828096a70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd828096e40 0x7fd828132630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-a con 0x7fd828096e40 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd828097790 0x7fd82811e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): picked mon.noname-c con 0x7fd828097790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8280a0650 con 0x7fd828096e40 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82811f460 con 0x7fd828096a70 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd828140630 con 0x7fd828097790 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.631+0000 7fd82dc9e700 10 monclient(hunting): authenticate will time out at 1101.727999s 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33298/0 (socket says 172.21.15.150:33298) 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 -- 172.21.15.150:0/582574601 learned_addr learned my addr 172.21.15.150:0/582574601 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): get_auth_request con 0x7fd828096a70 auth_method 0 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd828096e40 0x7fd828132630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd828097790 0x7fd82811e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): get_auth_request con 0x7fd828096e40 auth_method 0 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): get_auth_request con 0x7fd828097790 auth_method 0 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd827fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd826ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient(hunting): handle_auth_done global_id 15328 payload 824 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 -- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd828097790 msgr2=0x7fd82811e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd828097790 0x7fd82811e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 -- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd828096e40 msgr2=0x7fd828132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd828096e40 0x7fd828132630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0094f0 con 0x7fd828096a70 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 -- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 msgr2=0x7fd82809c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8277fe700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd81800e8d0 tx=0x7fd818008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8180093b0 con 0x7fd828096a70 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: dump: 2024-09-18T10:13:39.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818013620 con 0x7fd828096a70 2024-09-18T10:13:39.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 5 monclient: authenticate success, global_id 15328 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 10 monclient: _renew_subs 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd828097cb0 con 0x7fd828096a70 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8180139c0 con 0x7fd828096a70 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd80f7fe700 10 monclient: dump: 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.659 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd8257fa700 4 set_mon_vals no callback set 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 10 monclient: _renew_subs 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.635+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82809d310 con 0x7fd828096a70 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd82dc9e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd82dc9e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd82dc9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd828063ed0 con 0x7fd828096a70 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd818012370 con 0x7fd828096a70 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd80f7fe700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 0x7fd814037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(121..121 src has 58..121) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fd8180487a0 con 0x7fd828096a70 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd826ffd700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 0x7fd814037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.639+0000 7fd826ffd700 10 monclient: get_auth_request con 0x7fd814035ac0 auth_method 0 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.643+0000 7fd826ffd700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 0x7fd814037f70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd82806f6e0 tx=0x7fd81c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.647+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd81800f040 con 0x7fd828096a70 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.647+0000 7fd80f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.647+0000 7fd80f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 1 -- 172.21.15.142:0/3087788480 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7400bba0 con 0x7f4c84132490 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: dump: 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.666 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c8b882700 5 monclient: authenticate success, global_id 15331 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c8b882700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 1 -- 172.21.15.142:0/3087788480 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7400bd20 con 0x7f4c84132490 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c8b882700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:39.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c8b882700 4 set_mon_vals no callback set 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 1 -- 172.21.15.142:0/3087788480 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74011670 con 0x7f4c84132490 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.647+0000 7f4c7bfff700 10 monclient: dump: 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.668 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: shutdown 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- 172.21.15.142:0/3087788480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c84132490 msgr2=0x7f4c841373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 --2- 172.21.15.142:0/3087788480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c84132490 0x7f4c841373c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4c740099e0 tx=0x7f4c740091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- 172.21.15.142:0/3087788480 shutdown_connections 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 --2- 172.21.15.142:0/3087788480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c84132490 0x7f4c841373c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- 172.21.15.142:0/3087788480 >> 172.21.15.142:0/3087788480 conn(0x7f4c84071680 msgr2=0x7f4c840a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- 172.21.15.142:0/3087788480 shutdown_connections 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- 172.21.15.142:0/3087788480 wait complete. 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: build_initial_monmap 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: monmap: 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:39.670 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 Processor -- start 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- start start 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: init 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c840a61a0 0x7f4c8406e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-b con 0x7f4c840a61a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-c con 0x7f4c84132490 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c841328c0 0x7f4c8412cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): picked mon.noname-a con 0x7f4c841328c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:39.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8409d000 con 0x7f4c841328c0 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c840a55f0 con 0x7f4c840a61a0 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c841317f0 con 0x7f4c84132490 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8b882700 10 monclient(hunting): authenticate will time out at 1108.578559s 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38966/0 (socket says 172.21.15.142:38966) 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 -- 172.21.15.142:0/24212146 learned_addr learned my addr 172.21.15.142:0/24212146 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:39.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c841328c0 0x7f4c8412cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): get_auth_request con 0x7f4c84132490 auth_method 0 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c840a61a0 0x7f4c8406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): get_auth_request con 0x7f4c840a61a0 auth_method 0 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): get_auth_request con 0x7f4c841328c0 auth_method 0 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c8961e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c89e1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient(hunting): handle_auth_done global_id 44333 payload 824 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 -- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c840a61a0 msgr2=0x7f4c8406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c840a61a0 0x7f4c8406e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 -- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c841328c0 msgr2=0x7f4c8412cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c841328c0 0x7f4c8412cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 monclient: found mon.noname-c 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.651+0000 7f4c88e1d700 1 -- 172.21.15.142:0/24212146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74009420 con 0x7f4c84132490 2024-09-18T10:13:39.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c88e1d700 1 -- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 msgr2=0x7f4c8412c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c88e1d700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fa0 tx=0x7f4c74017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7401f070 con 0x7f4c84132490 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: dump: 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.676 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 5 monclient: authenticate success, global_id 44333 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74011630 con 0x7f4c84132490 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _renew_subs 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8412e8a0 con 0x7f4c84132490 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7401a880 con 0x7f4c84132490 2024-09-18T10:13:39.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c797fa700 10 monclient: dump: 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c7b7fe700 4 set_mon_vals no callback set 2024-09-18T10:13:39.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _renew_subs 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8412d460 con 0x7f4c84132490 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.655+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c840696a0 con 0x7f4c84132490 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.659+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4c7401abf0 con 0x7f4c84132490 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.659+0000 7f4c797fa700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 0x7f4c68037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.659+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(121..121 src has 58..121) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f4c7404a230 con 0x7f4c84132490 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.659+0000 7f4c8961e700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 0x7f4c68037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.659+0000 7f4c8961e700 10 monclient: get_auth_request con 0x7f4c68035ac0 auth_method 0 2024-09-18T10:13:39.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.663+0000 7f4c8961e700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 0x7f4c68037f70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4c80007fd0 tx=0x7f4c80007d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:39.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.663+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4c74006cb0 con 0x7f4c84132490 2024-09-18T10:13:39.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.663+0000 7f4c797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:39.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.663+0000 7f4c797fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.815+0000 7fd82dc9e700 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.815+0000 7fd82dc9e700 10 monclient: _send_command 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:39.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.815+0000 7fd82dc9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:39.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:39.815+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fd82806fc90 con 0x7fd828096a70 2024-09-18T10:13:39.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.887+0000 7f4c8b882700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:13:39.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.887+0000 7f4c8b882700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:13:39.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.887+0000 7f4c8b882700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:39.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.887+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f4c8409fd30 con 0x7f4c84132490 2024-09-18T10:13:39.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.979+0000 7f4c797fa700 1 -- 172.21.15.142:0/24212146 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f4c74015080 con 0x7f4c84132490 2024-09-18T10:13:39.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.979+0000 7f4c797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:13:39.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.979+0000 7f4c797fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 msgr2=0x7f4c68037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 0x7f4c68037f70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4c80007fd0 tx=0x7f4c80007d90 comp rx=0 tx=0).stop 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 10 monclient: shutdown 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 msgr2=0x7f4c8412c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fa0 tx=0x7f4c74017000 comp rx=0 tx=0).stop 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 shutdown_connections 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c68035ac0 0x7f4c68037f70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c841328c0 0x7f4c8412cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c84132490 0x7f4c8412c9e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 --2- 172.21.15.142:0/24212146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c840a61a0 0x7f4c8406e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 >> 172.21.15.142:0/24212146 conn(0x7f4c84071680 msgr2=0x7f4c8412bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 shutdown_connections 2024-09-18T10:13:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:39.983+0000 7f4c8b882700 1 -- 172.21.15.142:0/24212146 wait complete. 2024-09-18T10:13:40.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:13:40.249 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=8,logm=66} crc {auth=3868739983,config=3140514009,health=3887269300,kv=4261215878,logm=2661688066}) 2024-09-18T10:13:40.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=9,kv=8,logm=72} crc {auth=1852987327,config=3140514009,health=71310588,kv=4261215878,logm=52497662}) 2024-09-18T10:13:40.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=12,kv=8,logm=66} crc {auth=3868739983,config=3140514009,health=3887269300,kv=4261215878,logm=2661688066}) 2024-09-18T10:13:40.250 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=10,kv=8,logm=69} crc {auth=2853851040,config=3140514009,health=1281764270,kv=4261215878,logm=3561492874}) 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285976010}) 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476144100}) 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285976010}) 2024-09-18T10:13:40.252 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.247+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785652229}) 2024-09-18T10:13:40.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128870288}) 2024-09-18T10:13:40.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945723610}) 2024-09-18T10:13:40.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128870288}) 2024-09-18T10:13:40.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857577242}) 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897207402}) 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168073100}) 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897207402}) 2024-09-18T10:13:40.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.291+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3916831989}) 2024-09-18T10:13:40.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394989803}) 2024-09-18T10:13:40.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071185976}) 2024-09-18T10:13:40.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394989803}) 2024-09-18T10:13:40.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3133832471}) 2024-09-18T10:13:40.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871764001}) 2024-09-18T10:13:40.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939033127}) 2024-09-18T10:13:40.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871764001}) 2024-09-18T10:13:40.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.295+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709469628}) 2024-09-18T10:13:40.301 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.301 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125144470}) 2024-09-18T10:13:40.302 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2126115267}) 2024-09-18T10:13:40.302 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.302 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125144470}) 2024-09-18T10:13:40.302 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056677915}) 2024-09-18T10:13:40.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704282720}) 2024-09-18T10:13:40.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409441100}) 2024-09-18T10:13:40.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704282720}) 2024-09-18T10:13:40.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.299+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1954834156}) 2024-09-18T10:13:40.305 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215751060}) 2024-09-18T10:13:40.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16817929}) 2024-09-18T10:13:40.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215751060}) 2024-09-18T10:13:40.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=565825912}) 2024-09-18T10:13:40.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34457966}) 2024-09-18T10:13:40.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876385796}) 2024-09-18T10:13:40.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34457966}) 2024-09-18T10:13:40.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.303+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827134425}) 2024-09-18T10:13:40.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1669403722}) 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403691254}) 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1669403722}) 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485941558}) 2024-09-18T10:13:40.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122552715}) 2024-09-18T10:13:40.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642218241}) 2024-09-18T10:13:40.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122552715}) 2024-09-18T10:13:40.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1877840324}) 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442107835}) 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1374839477}) 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442107835}) 2024-09-18T10:13:40.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400501754}) 2024-09-18T10:13:40.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.307+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054354167}) 2024-09-18T10:13:40.313 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574458196}) 2024-09-18T10:13:40.313 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.313 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054354167}) 2024-09-18T10:13:40.313 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319109752}) 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909308191}) 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315485154}) 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909308191}) 2024-09-18T10:13:40.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1391135471}) 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099635567}) 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536071667}) 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4099635567}) 2024-09-18T10:13:40.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.311+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126329797}) 2024-09-18T10:13:40.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860658112}) 2024-09-18T10:13:40.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021580918}) 2024-09-18T10:13:40.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860658112}) 2024-09-18T10:13:40.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=95227542}) 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419573235}) 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945734064}) 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419573235}) 2024-09-18T10:13:40.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2592839333}) 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=27} crc {logm=2610858372,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1203638120,mon_config_key=620546692,monmap=3505902565,osd_metadata=4002491971,osd_pg_creating=3025569654,osd_snap=2673976383,osdmap=76696602}) 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=27} crc {logm=785416936,mdsmap=1531151339,mgr=1666935248,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=545520180,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=3025569654,osd_snap=3823785925,osdmap=76696602}) 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=27} crc {logm=2610858372,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1203638120,mon_config_key=620546692,monmap=3505902565,osd_metadata=4002491971,osd_pg_creating=3025569654,osd_snap=2673976383,osdmap=76696602}) 2024-09-18T10:13:40.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.315+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=25} crc {logm=2575582521,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3494099322,mon_config_key=620546692,monmap=3505902565,osd_metadata=964887042,osd_pg_creating=3025569654,osd_snap=454132730,osdmap=1623500395}) 2024-09-18T10:13:40.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=84} crc {osdmap=2189253693}) 2024-09-18T10:13:40.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=79} crc {osdmap=2443825077}) 2024-09-18T10:13:40.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:13:40.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=84} crc {osdmap=2189253693}) 2024-09-18T10:13:40.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:13:40.323+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=85} crc {osdmap=3085599021}) 2024-09-18T10:13:40.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.375+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v122) v1 ==== 145+0+0 (secure 0 0 0) 0x7fd818022af0 con 0x7fd828096a70 2024-09-18T10:13:40.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.375+0000 7fd80f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:40.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.375+0000 7fd80f7fe700 10 monclient: _finish_command 2 = system:0 set require_min_compat_client to luminous 2024-09-18T10:13:40.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.487+0000 7fd82dc9e700 10 start_mon_command cmd=[{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:40.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.487+0000 7fd82dc9e700 10 monclient: _send_command 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:40.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.487+0000 7fd82dc9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:40.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:40.487+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "luminous"} v 0) v1 -- 0x7fd82811f870 con 0x7fd828096a70 2024-09-18T10:13:41.005 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:13:41.006 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: monmap: 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.118 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:41.117422+0000 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:41.117422+0000 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.119 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 Processor -- start 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 -- start start 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: init 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f214409cdd0 0x7f214409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): picked mon.noname-a con 0x7f214409cdd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:41.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21440a02e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21440a0df0 0x7f2144139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): picked mon.noname-c con 0x7f21440a0df0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214409da60 con 0x7f21440a02e0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214409d1a0 con 0x7f214409cdd0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214409e980 con 0x7f21440a0df0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2149475700 10 monclient(hunting): authenticate will time out at 1110.043656s 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.115+0000 7f2142ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f214409cdd0 0x7f214409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f214409cdd0 0x7f214409e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39742/0 (socket says 172.21.15.142:39742) 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 -- 172.21.15.142:0/642639961 learned_addr learned my addr 172.21.15.142:0/642639961 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): get_auth_request con 0x7f214409cdd0 auth_method 0 2024-09-18T10:13:41.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 1 --2- 172.21.15.142:0/642639961 >> v2:172.21.15.150:3300/0 conn(0x7f21440a0df0 0x7f2144139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): get_auth_request con 0x7f21440a0df0 auth_method 0 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient(hunting): handle_auth_done global_id 54138 payload 274 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 -- 172.21.15.142:0/642639961 >> v2:172.21.15.150:3300/0 conn(0x7f21440a0df0 msgr2=0x7f2144139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 --2- 172.21.15.142:0/642639961 >> v2:172.21.15.150:3300/0 conn(0x7f21440a0df0 0x7f2144139a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 -- 172.21.15.142:0/642639961 >> v1:172.21.15.150:6789/0 conn(0x7f21440a02e0 legacy=0x7f21440a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 -- 172.21.15.142:0/642639961 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214409d780 con 0x7f214409cdd0 2024-09-18T10:13:41.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 -- 172.21.15.142:0/642639961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 msgr2=0x7f214409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2142ffd700 1 --2- 172.21.15.142:0/642639961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214409e440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f214409dbf0 tx=0x7f2134009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55adb06ea5aac3d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 1 -- 172.21.15.142:0/642639961 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134009e70 con 0x7f214409cdd0 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: dump: 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2149475700 5 monclient: authenticate success, global_id 54138 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2149475700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 1 -- 172.21.15.142:0/642639961 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213400ba40 con 0x7f214409cdd0 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2149475700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f2149475700 4 set_mon_vals no callback set 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 1 -- 172.21.15.142:0/642639961 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400be00 con 0x7f214409cdd0 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.119+0000 7f21427fc700 10 monclient: dump: 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.133 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: shutdown 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- 172.21.15.142:0/642639961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 msgr2=0x7f214409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- 172.21.15.142:0/642639961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214409e440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f214409dbf0 tx=0x7f2134009250 comp rx=0 tx=0).stop 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- 172.21.15.142:0/642639961 shutdown_connections 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- 172.21.15.142:0/642639961 >> v2:172.21.15.150:3300/0 conn(0x7f21440a0df0 0x7f2144139a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- 172.21.15.142:0/642639961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214409e440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- 172.21.15.142:0/642639961 >> 172.21.15.142:0/642639961 conn(0x7f2144071680 msgr2=0x7f21440a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- 172.21.15.142:0/642639961 shutdown_connections 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- 172.21.15.142:0/642639961 wait complete. 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: monmap: 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.135 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 Processor -- start 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- start start 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: init 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:41.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): picked mon.noname-a con 0x7f214409cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21440a02e0 0x7f214408d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): picked mon.noname-c con 0x7f21440a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 0x7f214408dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): picked mon.noname-b con 0x7f21440a0df0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2144091690 con 0x7f214409cdd0 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440906e0 con 0x7f21440a0df0 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214408ede0 con 0x7f21440a02e0 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2149475700 10 monclient(hunting): authenticate will time out at 1110.051801s 2024-09-18T10:13:41.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.123+0000 7f2142ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39756/0 (socket says 172.21.15.142:39756) 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 -- 172.21.15.142:0/2606815045 learned_addr learned my addr 172.21.15.142:0/2606815045 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): get_auth_request con 0x7f214409cdd0 auth_method 0 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21440a02e0 0x7f214408d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): get_auth_request con 0x7f21440a02e0 auth_method 0 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 0x7f214408dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): get_auth_request con 0x7f21440a0df0 auth_method 0 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient(hunting): handle_auth_done global_id 54141 payload 824 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 -- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21440a02e0 msgr2=0x7f214408d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21440a02e0 0x7f214408d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 -- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 msgr2=0x7f214408dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 0x7f214408dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 -- 172.21.15.142:0/2606815045 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21340094f0 con 0x7f214409cdd0 2024-09-18T10:13:41.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f21437fe700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 0x7f214408dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:41.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 -- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 msgr2=0x7f214408d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2142ffd700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2134006fd0 tx=0x7f2134017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213401f070 con 0x7f214409cdd0 2024-09-18T10:13:41.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: dump: 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.143 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 5 monclient: authenticate success, global_id 54141 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 10 monclient: _renew_subs 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134006be0 con 0x7f214409cdd0 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2144090400 con 0x7f214409cdd0 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134011b10 con 0x7f214409cdd0 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f213b7fe700 10 monclient: dump: 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2141ffb700 4 set_mon_vals no callback set 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 10 monclient: _renew_subs 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.127+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214408f4a0 con 0x7f214409cdd0 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f2149475700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f2149475700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f2149475700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214409c130 con 0x7f214409cdd0 2024-09-18T10:13:41.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2134023070 con 0x7f214409cdd0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f213b7fe700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 0x7f2124037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(122..122 src has 58..122) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f2134049cd0 con 0x7f214409cdd0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f213a7fc700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 0x7f2124037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.131+0000 7f213a7fc700 10 monclient: get_auth_request con 0x7f2124035ac0 auth_method 0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.135+0000 7f213a7fc700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 0x7f2124037f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2144064050 tx=0x7f212c007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.139+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2134023550 con 0x7f214409cdd0 2024-09-18T10:13:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.139+0000 7f213b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.139+0000 7f213b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.363+0000 7f2149475700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:41.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.363+0000 7f2149475700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:41.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.363+0000 7f2149475700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.363+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f214406da30 con 0x7f214409cdd0 2024-09-18T10:13:41.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f213b7fe700 1 -- 172.21.15.142:0/2606815045 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f213402a020 con 0x7f214409cdd0 2024-09-18T10:13:41.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f213b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:41.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f213b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:41.369 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:41.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 msgr2=0x7f2124037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 0x7f2124037f70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2144064050 tx=0x7f212c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 10 monclient: shutdown 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 msgr2=0x7f214408d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2134006fd0 tx=0x7f2134017000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 shutdown_connections 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2124035ac0 0x7f2124037f70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21440a0df0 0x7f214408dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21440a02e0 0x7f214408d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 --2- 172.21.15.142:0/2606815045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f214409cdd0 0x7f214408d050 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 >> 172.21.15.142:0/2606815045 conn(0x7f2144071680 msgr2=0x7f214406abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 shutdown_connections 2024-09-18T10:13:41.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.367+0000 7f2149475700 1 -- 172.21.15.142:0/2606815045 wait complete. 2024-09-18T10:13:41.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:13:41.383 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:13:41.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:13:41.383 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:13:41.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.395+0000 7fd80f7fe700 1 -- 172.21.15.150:0/582574601 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "luminous"}]=0 set require_min_compat_client to luminous v123) v1 ==== 145+0+0 (secure 0 0 0) 0x7fd818020070 con 0x7fd828096a70 2024-09-18T10:13:41.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.395+0000 7fd80f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd set-require-min-compat-client", "version": "luminous"}] 2024-09-18T10:13:41.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.395+0000 7fd80f7fe700 10 monclient: _finish_command 3 = system:0 set require_min_compat_client to luminous 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 msgr2=0x7fd814037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 0x7fd814037f70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd82806f6e0 tx=0x7fd81c016000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 10 monclient: shutdown 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 msgr2=0x7fd82809c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd81800e8d0 tx=0x7fd818008040 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 shutdown_connections 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd814035ac0 0x7fd814037f70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd828097790 0x7fd82811e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd828096e40 0x7fd828132630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 --2- 172.21.15.150:0/582574601 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd828096a70 0x7fd82809c7d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 >> 172.21.15.150:0/582574601 conn(0x7fd828068460 msgr2=0x7fd82809aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 shutdown_connections 2024-09-18T10:13:41.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.399+0000 7fd82dc9e700 1 -- 172.21.15.150:0/582574601 wait complete. 2024-09-18T10:13:41.403 INFO:tasks.workunit.client.0.smithi150.stderr:set require_min_compat_client to luminous 2024-09-18T10:13:41.416 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set dump 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 10 monclient: monmap: 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:41.478213+0000 2024-09-18T10:13:41.479 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:41.478213+0000 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 1 Processor -- start 2024-09-18T10:13:41.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 1 -- start start 2024-09-18T10:13:41.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.475+0000 7f204f59e700 10 monclient: init 2024-09-18T10:13:41.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:13:41.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f205012cb80 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f205009e760 0x7f2050134ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f205009e760 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2050135530 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205009d530 con 0x7f205012cb80 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205009ddf0 con 0x7f2050135530 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205009d340 con 0x7f205009e760 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204e59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f205012cb80 0x7f205012cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51330/0 (socket says 172.21.15.142:51330) 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204e59c700 1 -- 172.21.15.142:0/3475465009 learned_addr learned my addr 172.21.15.142:0/3475465009 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204f59e700 10 monclient(hunting): authenticate will time out at 1110.405511s 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f2045d9b700 1 --2- 172.21.15.142:0/3475465009 >> v2:172.21.15.142:3300/0 conn(0x7f205009e760 0x7f2050134ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f2045d9b700 10 monclient(hunting): get_auth_request con 0x7f205009e760 auth_method 0 2024-09-18T10:13:41.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f2045d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f2045d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f2045d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204e59c700 1 -- 172.21.15.142:0/3475465009 >> v1:172.21.15.142:6789/0 conn(0x7f205012cb80 legacy=0x7f205012cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.479+0000 7f204ed9d700 1 -- 172.21.15.142:0/3475465009 >> v1:172.21.15.150:6789/0 conn(0x7f2050135530 legacy=0x7f2050137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 1 -- 172.21.15.142:0/3475465009 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749055542 0 0) 0x7f205009ddf0 con 0x7f2050135530 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient(hunting): my global_id is 15343 2024-09-18T10:13:41.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 1 -- 172.21.15.142:0/3475465009 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c003570 con 0x7f2050135530 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient(hunting): handle_auth_done global_id 54144 payload 274 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 -- 172.21.15.142:0/3475465009 >> v1:172.21.15.150:6789/0 conn(0x7f2050135530 legacy=0x7f2050137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 -- 172.21.15.142:0/3475465009 >> v1:172.21.15.142:6789/0 conn(0x7f205012cb80 legacy=0x7f205012cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 -- 172.21.15.142:0/3475465009 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205012c730 con 0x7f205009e760 2024-09-18T10:13:41.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 -- 172.21.15.142:0/3475465009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205009e760 msgr2=0x7f2050134ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f2045d9b700 1 --2- 172.21.15.142:0/3475465009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205009e760 0x7f2050134ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2038009a10 tx=0x7f20380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f521213b4f98db47 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 1 -- 172.21.15.142:0/3475465009 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203800ba30 con 0x7f205009e760 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: dump: 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.488 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 1 -- 172.21.15.142:0/3475465009 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203800bb90 con 0x7f205009e760 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 1 -- 172.21.15.142:0/3475465009 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038011420 con 0x7f205009e760 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204f59e700 5 monclient: authenticate success, global_id 54144 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204f59e700 4 set_mon_vals no callback set 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.483+0000 7f204dd9b700 10 monclient: dump: 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.489 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.490 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: shutdown 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- 172.21.15.142:0/3475465009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205009e760 msgr2=0x7f2050134ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 --2- 172.21.15.142:0/3475465009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205009e760 0x7f2050134ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2038009a10 tx=0x7f20380091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- 172.21.15.142:0/3475465009 shutdown_connections 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 --2- 172.21.15.142:0/3475465009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205009e760 0x7f2050134ff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- 172.21.15.142:0/3475465009 >> 172.21.15.142:0/3475465009 conn(0x7f20500a2390 msgr2=0x7f20500a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- 172.21.15.142:0/3475465009 shutdown_connections 2024-09-18T10:13:41.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- 172.21.15.142:0/3475465009 wait complete. 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: monmap: 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 Processor -- start 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- start start 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: init 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205009e760 0x7f205006ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f205009e760 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 0x7f205006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f205012cb80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2050135530 0x7f205006d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f2050135530 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205012eb80 con 0x7f2050135530 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205012db90 con 0x7f205009e760 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205012b3c0 con 0x7f205012cb80 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204f59e700 10 monclient(hunting): authenticate will time out at 1110.415666s 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 0x7f205006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2050135530 0x7f205006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 0x7f205006cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38988/0 (socket says 172.21.15.142:38988) 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2050135530 0x7f205006d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39774/0 (socket says 172.21.15.142:39774) 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 1 -- 172.21.15.142:0/4038531482 learned_addr learned my addr 172.21.15.142:0/4038531482 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 10 monclient(hunting): get_auth_request con 0x7f205012cb80 auth_method 0 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f2045d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 10 monclient(hunting): get_auth_request con 0x7f2050135530 auth_method 0 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.487+0000 7f204e59c700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205009e760 0x7f205006ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): get_auth_request con 0x7f205009e760 auth_method 0 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient(hunting): handle_auth_done global_id 44339 payload 824 2024-09-18T10:13:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205009e760 msgr2=0x7f205006ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f205009e760 0x7f205006ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2050135530 msgr2=0x7f205006d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2050135530 0x7f205006d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 monclient: found mon.noname-c 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2038009420 con 0x7f205012cb80 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 msgr2=0x7f205006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2045d9b700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 0x7f205006cf50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2038009fd0 tx=0x7f2038017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203801f070 con 0x7f205012cb80 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: dump: 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 5 monclient: authenticate success, global_id 44339 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 10 monclient: _renew_subs 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205012d8b0 con 0x7f205012cb80 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2038006b10 con 0x7f205012cb80 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203800ba30 con 0x7f205012cb80 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f2046ffd700 10 monclient: dump: 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.504 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 10 monclient: _renew_subs 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204d59a700 4 set_mon_vals no callback set 2024-09-18T10:13:41.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.491+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205006f8d0 con 0x7f205012cb80 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2038023070 con 0x7f205012cb80 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204f59e700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 0x7f2050127460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2050064950 con 0x7f205006d9d0 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 0x7f2050127460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2030035ac0 0x7f2030037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient: get_auth_request con 0x7f205006d9d0 auth_method 0 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f2038049ea0 con 0x7f205012cb80 2024-09-18T10:13:41.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204ed9d700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2030035ac0 0x7f2030037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 10 monclient(hunting): handle_auth_done global_id 54153 payload 824 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f205006ef40 con 0x7f205006d9d0 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204ed9d700 10 monclient: get_auth_request con 0x7f2030035ac0 auth_method 0 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 msgr2=0x7f2050127460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f204e59c700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 0x7f2050127460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f204000f780 tx=0x7f2040009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040005d20 con 0x7f205006d9d0 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.495+0000 7f2046ffd700 10 monclient: dump: 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.507 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f204ed9d700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2030035ac0 0x7f2030037f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2048006fd0 tx=0x7f2048007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2040005e80 con 0x7f205006d9d0 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 msgr2=0x7f2050127460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 0x7f2050127460 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f204000f780 tx=0x7f2040009040 comp rx=0 tx=0).stop 2024-09-18T10:13:41.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 msgr2=0x7f2050127460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.499+0000 7f2046ffd700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f205006d9d0 0x7f2050127460 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f204000f780 tx=0x7f2040009040 comp rx=0 tx=0).stop 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204f59e700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 0x7f2050127da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205012ddd0 con 0x7f20501279a0 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 0x7f2050127da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient: get_auth_request con 0x7f20501279a0 auth_method 0 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-18T10:13:41.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.507+0000 7f204e59c700 1 -- 172.21.15.142:0/4038531482 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f205012df80 con 0x7f20501279a0 2024-09-18T10:13:41.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204e59c700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 msgr2=0x7f2050127da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204e59c700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 0x7f2050127da0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f204000f780 tx=0x7f2040018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040013d20 con 0x7f20501279a0 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 10 monclient: dump: 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.514 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f2040013e80 con 0x7f20501279a0 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 msgr2=0x7f2050127da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 0x7f2050127da0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f204000f780 tx=0x7f2040018000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 msgr2=0x7f2050127da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f2046ffd700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f20501279a0 0x7f2050127da0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:41.516 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.517 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.518 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.519 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.520 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:41.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.523 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.524 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.525 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.526 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:41.528 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2030035ac0 msgr2=0x7f2030037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2030035ac0 0x7f2030037f70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2048006fd0 tx=0x7f2048007ec0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 10 monclient: shutdown 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 msgr2=0x7f205006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 --2- 172.21.15.142:0/4038531482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f205012cb80 0x7f205006cf50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2038009fd0 tx=0x7f2038017000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204ed9d700 1 -- 172.21.15.142:0/4038531482 reap_dead start 2024-09-18T10:13:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204ed9d700 1 -- 172.21.15.142:0/4038531482 reap_dead start 2024-09-18T10:13:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 shutdown_connections 2024-09-18T10:13:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 >> 172.21.15.142:0/4038531482 conn(0x7f20500a2390 msgr2=0x7f20500a1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 shutdown_connections 2024-09-18T10:13:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.511+0000 7f204f59e700 1 -- 172.21.15.142:0/4038531482 wait complete. 2024-09-18T10:13:41.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:41.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:13:41.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:13:41.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: monmap: 2024-09-18T10:13:41.543 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:41.543 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:41.516243+0000 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:41.516243+0000 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.544 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 Processor -- start 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 -- start start 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: init 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdd80710a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efdd809cdc0 0x7efdd81356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-c con 0x7efdd809cdc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd809d1e0 0x7efdd8137c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-b con 0x7efdd809d1e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd809a280 con 0x7efdd80710a0 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd80995b0 con 0x7efdd809cdc0 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd806f700 con 0x7efdd809d1e0 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efde05e7700 10 monclient(hunting): authenticate will time out at 1103.611156s 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efdd809cdc0 0x7efdd81356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdde383700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7efdd80710a0 0x7efdd809c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:59088/0 (socket says 172.21.15.150:59088) 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdde383700 1 -- 172.21.15.150:0/2205238536 learned_addr learned my addr 172.21.15.150:0/2205238536 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:41.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 --2- 172.21.15.150:0/2205238536 >> v2:172.21.15.150:3300/0 conn(0x7efdd809cdc0 0x7efdd81356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33304/0 (socket says 172.21.15.150:33304) 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): get_auth_request con 0x7efdd809cdc0 auth_method 0 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd809d1e0 0x7efdd8137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): get_auth_request con 0x7efdd809d1e0 auth_method 0 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient(hunting): handle_auth_done global_id 15349 payload 274 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 -- 172.21.15.150:0/2205238536 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd809d1e0 msgr2=0x7efdd8137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd809d1e0 0x7efdd8137c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 -- 172.21.15.150:0/2205238536 >> v1:172.21.15.142:6789/0 conn(0x7efdd80710a0 legacy=0x7efdd809c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 monclient: found mon.noname-c 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.515+0000 7efdddb82700 1 -- 172.21.15.150:0/2205238536 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd809d850 con 0x7efdd809cdc0 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efdddb82700 1 -- 172.21.15.150:0/2205238536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809cdc0 msgr2=0x7efdd81356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efdddb82700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809cdc0 0x7efdd81356e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098e0 tx=0x7efdc80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1261d13e5e59cc13 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 1 -- 172.21.15.150:0/2205238536 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc800b850 con 0x7efdd809cdc0 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: dump: 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.549 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 1 -- 172.21.15.150:0/2205238536 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc800b9b0 con 0x7efdd809cdc0 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 5 monclient: authenticate success, global_id 15349 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 4 set_mon_vals no callback set 2024-09-18T10:13:41.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 1 -- 172.21.15.150:0/2205238536 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc800bd50 con 0x7efdd809cdc0 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efddd381700 10 monclient: dump: 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.551 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 10 monclient: shutdown 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 -- 172.21.15.150:0/2205238536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809cdc0 msgr2=0x7efdd81356e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809cdc0 0x7efdd81356e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efdc80098e0 tx=0x7efdc80091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 -- 172.21.15.150:0/2205238536 shutdown_connections 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd809d1e0 0x7efdd8137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 --2- 172.21.15.150:0/2205238536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809cdc0 0x7efdd81356e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.519+0000 7efde05e7700 1 -- 172.21.15.150:0/2205238536 >> 172.21.15.150:0/2205238536 conn(0x7efdd80a1cc0 msgr2=0x7efdd80a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- 172.21.15.150:0/2205238536 shutdown_connections 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- 172.21.15.150:0/2205238536 wait complete. 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient: monmap: 2024-09-18T10:13:41.552 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 Processor -- start 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- start start 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient: init 2024-09-18T10:13:41.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-c con 0x7efdd80710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd809cdc0 0x7efdd813b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-a con 0x7efdd809cdc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809d1e0 0x7efdd813f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): picked mon.noname-b con 0x7efdd809d1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd8121180 con 0x7efdd809cdc0 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd8120bc0 con 0x7efdd809d1e0 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd811e730 con 0x7efdd80710a0 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efde05e7700 10 monclient(hunting): authenticate will time out at 1103.618799s 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:38942/0 (socket says 172.21.15.150:38942) 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 -- 172.21.15.150:0/2099275758 learned_addr learned my addr 172.21.15.150:0/2099275758 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): get_auth_request con 0x7efdd80710a0 auth_method 0 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809d1e0 0x7efdd813f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd809cdc0 0x7efdd813b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): get_auth_request con 0x7efdd809cdc0 auth_method 0 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): get_auth_request con 0x7efdd809d1e0 auth_method 0 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdddb82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.557 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efddeb84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient(hunting): handle_auth_done global_id 44345 payload 824 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 -- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809d1e0 msgr2=0x7efdd813f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809d1e0 0x7efdd813f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 -- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd809cdc0 msgr2=0x7efdd813b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd809cdc0 0x7efdd813b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 monclient: found mon.noname-c 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.523+0000 7efdde383700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8009420 con 0x7efdd80710a0 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdde383700 1 -- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 msgr2=0x7efdd813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.558 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdde383700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b610 tx=0x7efdd4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4011840 con 0x7efdd80710a0 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: dump: 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.559 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd40119a0 con 0x7efdd80710a0 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 5 monclient: authenticate success, global_id 44345 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _renew_subs 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4011d60 con 0x7efdd80710a0 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd81208e0 con 0x7efdd80710a0 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdcffff700 4 set_mon_vals no callback set 2024-09-18T10:13:41.560 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efdce7fc700 10 monclient: dump: 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _renew_subs 2024-09-18T10:13:41.561 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd813bf20 con 0x7efdd80710a0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.527+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd8063ed0 con 0x7efdd80710a0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efdd4010590 con 0x7efdd80710a0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdce7fc700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 0x7efdbc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7efdd4048ad0 con 0x7efdd80710a0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdddb82700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 0x7efdbc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdddb82700 10 monclient: get_auth_request con 0x7efdbc035ac0 auth_method 0 2024-09-18T10:13:41.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.531+0000 7efdddb82700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 0x7efdbc037f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efdc80030c0 tx=0x7efdc8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.543+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7efdd400f8c0 con 0x7efdd80710a0 2024-09-18T10:13:41.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.543+0000 7efdce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.543+0000 7efdce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.635+0000 7f9f22521700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.635+0000 7f9f22521700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.635+0000 7f9f22521700 10 monclient: monmap: 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:41.639518+0000 2024-09-18T10:13:41.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:41.639518+0000 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 Processor -- start 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 -- start start 2024-09-18T10:13:41.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient: init 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f1c09c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-b con 0x7f9f1c134880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f1c1371a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c0a6620 con 0x7f9f1c1371a0 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c0a1c70 con 0x7f9f1c09c2c0 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c09ab70 con 0x7f9f1c134880 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f22521700 10 monclient(hunting): authenticate will time out at 1110.565435s 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f21d20700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9f1c1371a0 0x7f9f1c139580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51338/0 (socket says 172.21.15.142:51338) 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39000/0 (socket says 172.21.15.142:39000) 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f21d20700 1 -- 172.21.15.142:0/2522951835 learned_addr learned my addr 172.21.15.142:0/2522951835 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): get_auth_request con 0x7f9f1c134880 auth_method 0 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_done global_id 44348 payload 274 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 -- 172.21.15.142:0/2522951835 >> v1:172.21.15.150:6789/0 conn(0x7f9f1c09c2c0 legacy=0x7f9f1c0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 -- 172.21.15.142:0/2522951835 >> v1:172.21.15.142:6789/0 conn(0x7f9f1c1371a0 legacy=0x7f9f1c139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 monclient: found mon.noname-b 2024-09-18T10:13:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.639+0000 7f9f20d1e700 1 -- 172.21.15.142:0/2522951835 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c09c690 con 0x7f9f1c134880 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f20d1e700 1 -- 172.21.15.142:0/2522951835 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 msgr2=0x7f9f1c136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f20d1e700 1 --2- 172.21.15.142:0/2522951835 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009910 tx=0x7f9f100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d7026ba0669f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 1 -- 172.21.15.142:0/2522951835 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000ba70 con 0x7f9f1c134880 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: dump: 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 5 monclient: authenticate success, global_id 44348 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 1 -- 172.21.15.142:0/2522951835 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1000bbd0 con 0x7f9f1c134880 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 4 set_mon_vals no callback set 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 1 -- 172.21.15.142:0/2522951835 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10011420 con 0x7f9f1c134880 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f0ffff700 10 monclient: dump: 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 10 monclient: shutdown 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 -- 172.21.15.142:0/2522951835 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 msgr2=0x7f9f1c136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 --2- 172.21.15.142:0/2522951835 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009910 tx=0x7f9f100091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 -- 172.21.15.142:0/2522951835 shutdown_connections 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 --2- 172.21.15.142:0/2522951835 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c134880 0x7f9f1c136c60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 -- 172.21.15.142:0/2522951835 >> 172.21.15.142:0/2522951835 conn(0x7f9f1c0a5720 msgr2=0x7f9f1c0a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 -- 172.21.15.142:0/2522951835 shutdown_connections 2024-09-18T10:13:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 1 -- 172.21.15.142:0/2522951835 wait complete. 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.643+0000 7f9f22521700 10 monclient: monmap: 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 Processor -- start 2024-09-18T10:13:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 -- start start 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient: init 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f1c09c2c0 0x7f9f1c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-b con 0x7f9f1c09c2c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-a con 0x7f9f1c134880 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c1371a0 0x7f9f1c06b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): picked mon.noname-c con 0x7f9f1c1371a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c06fa80 con 0x7f9f1c134880 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c06ce30 con 0x7f9f1c09c2c0 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c06d550 con 0x7f9f1c1371a0 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f22521700 10 monclient(hunting): authenticate will time out at 1110.573365s 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c1371a0 0x7f9f1c06b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39798/0 (socket says 172.21.15.142:39798) 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 -- 172.21.15.142:0/3412890102 learned_addr learned my addr 172.21.15.142:0/3412890102 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): get_auth_request con 0x7f9f1c134880 auth_method 0 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): get_auth_request con 0x7f9f1c1371a0 auth_method 0 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f1c09c2c0 0x7f9f1c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): get_auth_request con 0x7f9f1c09c2c0 auth_method 0 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f21d20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient(hunting): handle_auth_done global_id 54162 payload 824 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c1371a0 msgr2=0x7f9f1c06b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c1371a0 0x7f9f1c06b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f1c09c2c0 msgr2=0x7f9f1c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f1c09c2c0 0x7f9f1c06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.647+0000 7f9f20d1e700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009420 con 0x7f9f1c134880 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f20d1e700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 msgr2=0x7f9f1c06b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f20d1e700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0a1e00 tx=0x7f9f10016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10011420 con 0x7f9f1c134880 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: dump: 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 5 monclient: authenticate success, global_id 54162 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 monclient: _renew_subs 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c13a990 con 0x7f9f1c134880 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10020070 con 0x7f9f1c134880 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1001b680 con 0x7f9f1c134880 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 10 monclient: dump: 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0f7fe700 4 set_mon_vals no callback set 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 monclient: _renew_subs 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c06e880 con 0x7f9f1c134880 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 0x7f9f1c09cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c06eec0 con 0x7f9f1c06bd30 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 0x7f9f1c09cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient: get_auth_request con 0x7f9f1c06bd30 auth_method 0 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9f10022030 con 0x7f9f1c134880 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 10 monclient(hunting): handle_auth_done global_id 44354 payload 824 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f00035ac0 0x7f9f00037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f2151f700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f1c06c100 con 0x7f9f1c06bd30 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f9f1004a6b0 con 0x7f9f1c134880 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f21d20700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f00035ac0 0x7f9f00037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.651+0000 7f9f21d20700 10 monclient: get_auth_request con 0x7f9f00035ac0 auth_method 0 2024-09-18T10:13:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f2151f700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 msgr2=0x7f9f1c09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f2151f700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 0x7f9f1c09cb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f180026a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1800d710 con 0x7f9f1c06bd30 2024-09-18T10:13:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 10 monclient: dump: 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9f1800d870 con 0x7f9f1c06bd30 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 msgr2=0x7f9f1c09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 0x7f9f1c09cb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f180026a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 msgr2=0x7f9f1c09cb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f0d7fa700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c06bd30 0x7f9f1c09cb00 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f180026a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.655+0000 7f9f21d20700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f00035ac0 0x7f9f00037f70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c06a3e0 tx=0x7f9f14007f70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f22521700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f22521700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 0x7f9f1c09d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f22521700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c064750 con 0x7f9f1c09d040 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 0x7f9f1c09d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient: get_auth_request con 0x7f9f1c09d040 auth_method 0 2024-09-18T10:13:41.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 10 monclient(hunting): handle_auth_done global_id 44357 payload 824 2024-09-18T10:13:41.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.667+0000 7f9f2151f700 1 -- 172.21.15.142:0/3412890102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f1c0a6620 con 0x7f9f1c09d040 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f2151f700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 msgr2=0x7f9f1c09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f2151f700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 0x7f9f1c09d440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f18019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18021950 con 0x7f9f1c09d040 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 10 monclient: dump: 2024-09-18T10:13:41.675 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f9f18021ab0 con 0x7f9f1c09d040 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:41.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 msgr2=0x7f9f1c09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 0x7f9f1c09d440 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f18019000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 msgr2=0x7f9f1c09d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f0d7fa700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f1c09d040 0x7f9f1c09d440 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800fb00 tx=0x7f9f18019000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:41.677 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.678 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.679 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:41.680 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:41.682 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.684 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.685 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.686 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.687 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.688 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:41.689 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:41.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f00035ac0 msgr2=0x7f9f00037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f00035ac0 0x7f9f00037f70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c06a3e0 tx=0x7f9f14007f70 comp rx=0 tx=0).stop 2024-09-18T10:13:41.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 10 monclient: shutdown 2024-09-18T10:13:41.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 msgr2=0x7f9f1c06b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0a1e00 tx=0x7f9f10016000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f21d20700 1 -- 172.21.15.142:0/3412890102 reap_dead start 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 shutdown_connections 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 --2- 172.21.15.142:0/3412890102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f1c134880 0x7f9f1c06b2b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 >> 172.21.15.142:0/3412890102 conn(0x7f9f1c0a5720 msgr2=0x7f9f1c0a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 shutdown_connections 2024-09-18T10:13:41.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.671+0000 7f9f22521700 1 -- 172.21.15.142:0/3412890102 wait complete. 2024-09-18T10:13:41.693 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:41.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:13:41.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:13:41.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efde05e7700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efde05e7700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efde05e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7efdd8071930 con 0x7efdd80710a0 2024-09-18T10:13:41.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v123) v1 ==== 73+0+3 (secure 0 0 0) 0x7efdd4015090 con 0x7efdd80710a0 2024-09-18T10:13:41.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efdce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.715+0000 7efdce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:41.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.799+0000 7fb559d1a700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.799+0000 7fb559d1a700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.799+0000 7fb559d1a700 10 monclient: monmap: 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:41.802915+0000 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:41.802915+0000 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.804 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.805 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 Processor -- start 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 -- start start 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient: init 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5540975d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:41.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb554132500 0x7fb5541348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-a con 0x7fb554132500 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb554134e20 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5540979a0 con 0x7fb554134e20 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb554098860 con 0x7fb5540975d0 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55409dc00 con 0x7fb554132500 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559d1a700 10 monclient(hunting): authenticate will time out at 1110.728752s 2024-09-18T10:13:41.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559519700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb554134e20 0x7fb554137200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51344/0 (socket says 172.21.15.142:51344) 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559519700 1 -- 172.21.15.142:0/2022634846 learned_addr learned my addr 172.21.15.142:0/2022634846 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb558d18700 1 --1- 172.21.15.142:0/2022634846 >> v1:172.21.15.150:6789/0 conn(0x7fb5540975d0 0x7fb554099cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:37266/0 (socket says 172.21.15.142:37266) 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 1 --2- 172.21.15.142:0/2022634846 >> v2:172.21.15.142:3300/0 conn(0x7fb554132500 0x7fb5541348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): get_auth_request con 0x7fb554132500 auth_method 0 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb558d18700 1 -- 172.21.15.142:0/2022634846 >> v1:172.21.15.150:6789/0 conn(0x7fb5540975d0 legacy=0x7fb554099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb559519700 1 -- 172.21.15.142:0/2022634846 >> v1:172.21.15.142:6789/0 conn(0x7fb554134e20 legacy=0x7fb554137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb5537fe700 1 -- 172.21.15.142:0/2022634846 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189869379 0 0) 0x7fb554098860 con 0x7fb5540975d0 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb5537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb5537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb5537fe700 10 monclient(hunting): my global_id is 15358 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb5537fe700 1 -- 172.21.15.142:0/2022634846 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb540003570 con 0x7fb5540975d0 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 10 monclient(hunting): handle_auth_done global_id 54165 payload 274 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.803+0000 7fb553fff700 1 -- 172.21.15.142:0/2022634846 >> v1:172.21.15.150:6789/0 conn(0x7fb5540975d0 legacy=0x7fb554099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 1 -- 172.21.15.142:0/2022634846 >> v1:172.21.15.142:6789/0 conn(0x7fb554134e20 legacy=0x7fb554137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 1 -- 172.21.15.142:0/2022634846 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554098b60 con 0x7fb554132500 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 1 -- 172.21.15.142:0/2022634846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 msgr2=0x7fb5541348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb553fff700 1 --2- 172.21.15.142:0/2022634846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb5541348e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0099e0 tx=0x7fb53c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3ef668184c17e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 1 -- 172.21.15.142:0/2022634846 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c00ba70 con 0x7fb554132500 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: dump: 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.813 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 1 -- 172.21.15.142:0/2022634846 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c00bbd0 con 0x7fb554132500 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 5 monclient: authenticate success, global_id 54165 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 1 -- 172.21.15.142:0/2022634846 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c011420 con 0x7fb554132500 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 4 set_mon_vals no callback set 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb5537fe700 10 monclient: dump: 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.815 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 10 monclient: shutdown 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 -- 172.21.15.142:0/2022634846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 msgr2=0x7fb5541348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 --2- 172.21.15.142:0/2022634846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb5541348e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0099e0 tx=0x7fb53c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 -- 172.21.15.142:0/2022634846 shutdown_connections 2024-09-18T10:13:41.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 --2- 172.21.15.142:0/2022634846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb5541348e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 -- 172.21.15.142:0/2022634846 >> 172.21.15.142:0/2022634846 conn(0x7fb554071ae0 msgr2=0x7fb5540a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 -- 172.21.15.142:0/2022634846 shutdown_connections 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.807+0000 7fb559d1a700 1 -- 172.21.15.142:0/2022634846 wait complete. 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient: monmap: 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.817 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.818 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.818 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.818 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efde05e7700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efde05e7700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efde05e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7efdd8140410 con 0x7efdd80710a0 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efdce7fc700 1 -- 172.21.15.150:0/2099275758 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v123) v1 ==== 73+0+3 (secure 0 0 0) 0x7efdd400f2e0 con 0x7efdd80710a0 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efdce7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.819+0000 7efdce7fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:41.825 INFO:tasks.workunit.client.0.smithi150.stdout:{} 2024-09-18T10:13:41.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 Processor -- start 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 -- start start 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient: init 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5540975d0 0x7fb554130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-c con 0x7fb5540975d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb55406b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-a con 0x7fb554132500 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb554134e20 0x7fb55406bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): picked mon.noname-b con 0x7fb554134e20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5540a1800 con 0x7fb554132500 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55406ee50 con 0x7fb554134e20 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55406d8a0 con 0x7fb5540975d0 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559d1a700 10 monclient(hunting): authenticate will time out at 1110.738339s 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb55406b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5540975d0 0x7fb554130e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb55406b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39816/0 (socket says 172.21.15.142:39816) 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 -- 172.21.15.142:0/489444704 learned_addr learned my addr 172.21.15.142:0/489444704 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): get_auth_request con 0x7fb554132500 auth_method 0 2024-09-18T10:13:41.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): get_auth_request con 0x7fb5540975d0 auth_method 0 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb554134e20 0x7fb55406bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): get_auth_request con 0x7fb554134e20 auth_method 0 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb559519700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient(hunting): handle_auth_done global_id 54171 payload 824 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5540975d0 msgr2=0x7fb554130e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5540975d0 0x7fb554130e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb554134e20 msgr2=0x7fb55406bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb554134e20 0x7fb55406bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.811+0000 7fb553fff700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c009420 con 0x7fb554132500 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb553fff700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 msgr2=0x7fb55406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb553fff700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb55406b7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb5540a2180 tx=0x7fb53c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c006af0 con 0x7fb554132500 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: dump: 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.831 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c01a450 con 0x7fb554132500 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 5 monclient: authenticate success, global_id 54171 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 monclient: _renew_subs 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c01a7f0 con 0x7fb554132500 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55406dce0 con 0x7fb554132500 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb552ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 10 monclient: dump: 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.833 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 monclient: _renew_subs 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb554126ac0 con 0x7fb554132500 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 0x7fb55406c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb554064830 con 0x7fb55406df60 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb53c01abe0 con 0x7fb554132500 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 0x7fb55406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient: get_auth_request con 0x7fb55406df60 auth_method 0 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.815+0000 7fb550ff9700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb534035ac0 0x7fb534037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fb53c04a2b0 con 0x7fb554132500 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb558d18700 10 monclient(hunting): handle_auth_done global_id 15364 payload 824 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb558d18700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55406f070 con 0x7fb55406df60 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb559519700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb534035ac0 0x7fb534037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb559519700 10 monclient: get_auth_request con 0x7fb534035ac0 auth_method 0 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb558d18700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 msgr2=0x7fb55406c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb558d18700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 0x7fb55406c240 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb544001850 tx=0x7fb544001d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb559519700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb534035ac0 0x7fb534037f70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0058d0 tx=0x7fb54c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440133f0 con 0x7fb55406df60 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 10 monclient: dump: 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.836 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb544013550 con 0x7fb55406df60 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 msgr2=0x7fb55406c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 0x7fb55406c240 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb544001850 tx=0x7fb544001d30 comp rx=0 tx=0).stop 2024-09-18T10:13:41.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 msgr2=0x7fb55406c240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.819+0000 7fb550ff9700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55406df60 0x7fb55406c240 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 msgr2=0x7efdbc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 0x7efdbc037f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efdc80030c0 tx=0x7efdc8016000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 10 monclient: shutdown 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 msgr2=0x7efdd813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b610 tx=0x7efdd4009120 comp rx=0 tx=0).stop 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 shutdown_connections 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdbc035ac0 0x7efdbc037f70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdd809d1e0 0x7efdd813f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdd809cdc0 0x7efdd813b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 --2- 172.21.15.150:0/2099275758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdd80710a0 0x7efdd813c190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 >> 172.21.15.150:0/2099275758 conn(0x7efdd80a1cc0 msgr2=0x7efdd80a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 shutdown_connections 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.823+0000 7efde05e7700 1 -- 172.21.15.150:0/2099275758 wait complete. 2024-09-18T10:13:41.840 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:13:41.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb559d1a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:41.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb559d1a700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 0x7fb55405e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb559d1a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55402b280 con 0x7fb55402ae80 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 0x7fb55405e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient: get_auth_request con 0x7fb55402ae80 auth_method 0 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 10 monclient(hunting): handle_auth_done global_id 15367 payload 824 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.835+0000 7fb558d18700 1 -- 172.21.15.142:0/489444704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb554136950 con 0x7fb55402ae80 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb558d18700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 msgr2=0x7fb55405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb558d18700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 0x7fb55405e890 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb544007bd0 tx=0x7fb544015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544020470 con 0x7fb55402ae80 2024-09-18T10:13:41.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:41.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 10 monclient: dump: 2024-09-18T10:13:41.844 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fb54400fc10 con 0x7fb55402ae80 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:41.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 msgr2=0x7fb55405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 0x7fb55405e890 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb544007bd0 tx=0x7fb544015000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 msgr2=0x7fb55405e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb550ff9700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb55402ae80 0x7fb55405e890 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:41.846 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 56, 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.847 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.848 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:41.849 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:41.850 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.851 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.854 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:41.855 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:41.857 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.858 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:41.859 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:41.860 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:41.860 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:41.860 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:41.860 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:41.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb534035ac0 msgr2=0x7fb534037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb534035ac0 0x7fb534037f70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0058d0 tx=0x7fb54c009040 comp rx=0 tx=0).stop 2024-09-18T10:13:41.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 10 monclient: shutdown 2024-09-18T10:13:41.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 msgr2=0x7fb55406b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 --2- 172.21.15.142:0/489444704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb554132500 0x7fb55406b7c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb5540a2180 tx=0x7fb53c017000 comp rx=0 tx=0).stop 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559519700 1 -- 172.21.15.142:0/489444704 reap_dead start 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559519700 1 -- 172.21.15.142:0/489444704 reap_dead start 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 shutdown_connections 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 >> 172.21.15.142:0/489444704 conn(0x7fb554071ae0 msgr2=0x7fb554131b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 shutdown_connections 2024-09-18T10:13:41.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.839+0000 7fb559d1a700 1 -- 172.21.15.142:0/489444704 wait complete. 2024-09-18T10:13:41.862 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:13:41.863 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:13:41.863 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:13:41.863 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:13:41.863 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:13:41.863 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:13:41.863 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:13:41.878 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:13:41.879 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:13:41.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: monmap: 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:41.936784+0000 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:41.936784+0000 2024-09-18T10:13:41.937 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.938 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 Processor -- start 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 -- start start 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: init 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1000a6090 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10009d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): picked mon.noname-b con 0x7ff10009e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff1000a0a80 0x7ff100139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): picked mon.noname-a con 0x7ff1000a0a80 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1000a6460 con 0x7ff1000a6090 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10009fd00 con 0x7ff1000a0a80 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000a6850 con 0x7ff10009e460 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff108429700 10 monclient(hunting): authenticate will time out at 1104.030653s 2024-09-18T10:13:41.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1059c4700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 msgr2=0x7ff10009d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1059c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10009d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1061c5700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff1000a6090 0x7ff1000a0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48620/0 (socket says 172.21.15.150:48620) 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1061c5700 1 -- 172.21.15.150:0/714145833 learned_addr learned my addr 172.21.15.150:0/714145833 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 --2- 172.21.15.150:0/714145833 >> v2:172.21.15.142:3300/0 conn(0x7ff1000a0a80 0x7ff100139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): get_auth_request con 0x7ff1000a0a80 auth_method 0 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient(hunting): handle_auth_done global_id 54174 payload 274 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 -- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 msgr2=0x7ff10009d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 --2- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10009d0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 -- 172.21.15.150:0/714145833 >> v1:172.21.15.150:6789/0 conn(0x7ff1000a6090 legacy=0x7ff1000a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.935+0000 7ff1069c6700 1 -- 172.21.15.150:0/714145833 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10009d7a0 con 0x7ff1000a0a80 2024-09-18T10:13:41.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1069c6700 1 -- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a0a80 msgr2=0x7ff100139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1069c6700 1 --2- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a0a80 0x7ff100139b10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e60 tx=0x7ff0f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307729a5fcc2b0ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 1 -- 172.21.15.150:0/714145833 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f800e070 con 0x7ff1000a0a80 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: dump: 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.951 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 5 monclient: authenticate success, global_id 54174 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 1 -- 172.21.15.150:0/714145833 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8012430 con 0x7ff1000a0a80 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 4 set_mon_vals no callback set 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 1 -- 172.21.15.150:0/714145833 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80127d0 con 0x7ff1000a0a80 2024-09-18T10:13:41.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff1051c3700 10 monclient: dump: 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.953 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 10 monclient: shutdown 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 -- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a0a80 msgr2=0x7ff100139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 --2- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a0a80 0x7ff100139b10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009e60 tx=0x7ff0f8009780 comp rx=0 tx=0).stop 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 -- 172.21.15.150:0/714145833 shutdown_connections 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 --2- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a0a80 0x7ff100139b10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 --2- 172.21.15.150:0/714145833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10009d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 -- 172.21.15.150:0/714145833 >> 172.21.15.150:0/714145833 conn(0x7ff100071830 msgr2=0x7ff1000a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:41.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 -- 172.21.15.150:0/714145833 shutdown_connections 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.939+0000 7ff108429700 1 -- 172.21.15.150:0/714145833 wait complete. 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient: monmap: 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:41.955 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 Processor -- start 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 -- start start 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient: init 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10008cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): picked mon.noname-c con 0x7ff10009e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): picked mon.noname-b con 0x7ff1000a0a80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a6090 0x7ff10008d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): picked mon.noname-a con 0x7ff1000a6090 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10008ed70 con 0x7ff1000a6090 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff100090590 con 0x7ff1000a0a80 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10008e860 con 0x7ff10009e460 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff108429700 10 monclient(hunting): authenticate will time out at 1104.037233s 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1061c5700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 msgr2=0x7ff10008cc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1061c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10008cc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33326/0 (socket says 172.21.15.150:33326) 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 -- 172.21.15.150:0/3612394285 learned_addr learned my addr 172.21.15.150:0/3612394285 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:41.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): get_auth_request con 0x7ff1000a0a80 auth_method 0 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a6090 0x7ff10008d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): get_auth_request con 0x7ff1000a6090 auth_method 0 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1069c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient(hunting): handle_auth_done global_id 15370 payload 824 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 -- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 msgr2=0x7ff10008cc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10008cc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 -- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a6090 msgr2=0x7ff10008d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a6090 0x7ff10008d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 monclient: found mon.noname-b 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.943+0000 7ff1059c4700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8009a20 con 0x7ff1000a0a80 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff1059c4700 1 -- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 msgr2=0x7ff10008d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff1059c4700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b7e0 tx=0x7ff0fc009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc003e20 con 0x7ff1000a0a80 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: dump: 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.960 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 5 monclient: authenticate success, global_id 15370 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _renew_subs 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc006150 con 0x7ff1000a0a80 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff100098330 con 0x7ff1000a0a80 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff1049c2700 4 set_mon_vals no callback set 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _renew_subs 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:41.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff100090330 con 0x7ff1000a0a80 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc006510 con 0x7ff1000a0a80 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 10 monclient: dump: 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.962 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:41.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:41.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10006d3c0 con 0x7ff1000a0a80 2024-09-18T10:13:41.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff0fc0103c0 con 0x7ff1000a0a80 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 0x7ff0e4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7ff0fc0482a0 con 0x7ff1000a0a80 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff1061c5700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 0x7ff0e4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.947+0000 7ff1061c5700 10 monclient: get_auth_request con 0x7ff0e4035a70 auth_method 0 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.951+0000 7ff1061c5700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 0x7ff0e4037f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4004860 tx=0x7ff0f4007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.955+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff0fc00e8c0 con 0x7ff1000a0a80 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.955+0000 7ff0f27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:41.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:41.955+0000 7ff0f27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:41.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:41.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 10 monclient: build_initial_monmap 2024-09-18T10:13:41.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 10 monclient: monmap: 2024-09-18T10:13:41.987 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:41.986612+0000 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:41.986612+0000 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:41.988 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:41.989 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 1 Processor -- start 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 1 -- start start 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.983+0000 7fd36359e700 10 monclient: init 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd3640a1cf0 0x7fd36412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd3640a1cf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd36412f810 0x7fd36409f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd36412f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:41.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 0x7fd3641320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd3640a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36409d990 con 0x7fd3640a1cf0 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36409dc70 con 0x7fd36412f810 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3640a20c0 con 0x7fd3640a02d0 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36359e700 10 monclient(hunting): authenticate will time out at 1110.912544s 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd362d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 msgr2=0x7fd3641320e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd362d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 0x7fd3641320e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd3640a1cf0 0x7fd36412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd3640a1cf0 0x7fd36412f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39828/0 (socket says 172.21.15.142:39828) 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 -- 172.21.15.142:0/2912059810 learned_addr learned my addr 172.21.15.142:0/2912059810 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): get_auth_request con 0x7fd3640a1cf0 auth_method 0 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 1 --2- 172.21.15.142:0/2912059810 >> v2:172.21.15.150:3300/0 conn(0x7fd36412f810 0x7fd36409f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): get_auth_request con 0x7fd36412f810 auth_method 0 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:41.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient(hunting): handle_auth_done global_id 54180 payload 274 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 -- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 msgr2=0x7fd3641320e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 --2- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 0x7fd3641320e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 -- 172.21.15.142:0/2912059810 >> v2:172.21.15.150:3300/0 conn(0x7fd36412f810 msgr2=0x7fd36409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 --2- 172.21.15.142:0/2912059810 >> v2:172.21.15.150:3300/0 conn(0x7fd36412f810 0x7fd36409f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 monclient: found mon.noname-a 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 -- 172.21.15.142:0/2912059810 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36409f9c0 con 0x7fd3640a1cf0 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 -- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 msgr2=0x7fd36412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.987+0000 7fd36259c700 1 --2- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36412f030 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd36409ca80 tx=0x7fd354009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b8748682f4e365 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 1 -- 172.21.15.142:0/2912059810 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd354009e70 con 0x7fd3640a1cf0 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: dump: 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.997 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: _finish_auth 0 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 1 -- 172.21.15.142:0/2912059810 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35400ba70 con 0x7fd3640a1cf0 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:41.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 1 -- 172.21.15.142:0/2912059810 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400be10 con 0x7fd3640a1cf0 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 5 monclient: authenticate success, global_id 54180 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 4 set_mon_vals no callback set 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36159a700 10 monclient: dump: 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:41.999 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:42.000 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:42.000 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:42.000 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:42.000 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:42.000 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: shutdown 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 msgr2=0x7fd36412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36412f030 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd36409ca80 tx=0x7fd354009250 comp rx=0 tx=0).stop 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- 172.21.15.142:0/2912059810 shutdown_connections 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd3640a02d0 0x7fd3641320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- 172.21.15.142:0/2912059810 >> v2:172.21.15.150:3300/0 conn(0x7fd36412f810 0x7fd36409f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- 172.21.15.142:0/2912059810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36412f030 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- 172.21.15.142:0/2912059810 >> 172.21.15.142:0/2912059810 conn(0x7fd364071820 msgr2=0x7fd3640a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:42.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- 172.21.15.142:0/2912059810 shutdown_connections 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- 172.21.15.142:0/2912059810 wait complete. 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: build_initial_monmap 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: monmap: 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:42.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 Processor -- start 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- start start 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: init 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 0x7fd36409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd3640a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd3640a1cf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 0x7fd36409be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd36412f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:42.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36406b970 con 0x7fd3640a1cf0 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36406cff0 con 0x7fd3640a02d0 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36406fa10 con 0x7fd36412f810 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd36359e700 10 monclient(hunting): authenticate will time out at 1110.919860s 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd362d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 msgr2=0x7fd36409be30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd362d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 0x7fd36409be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.991+0000 7fd361d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39840/0 (socket says 172.21.15.142:39840) 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 -- 172.21.15.142:0/3548023972 learned_addr learned my addr 172.21.15.142:0/3548023972 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): get_auth_request con 0x7fd3640a1cf0 auth_method 0 2024-09-18T10:13:42.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 0x7fd36409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 10 monclient(hunting): get_auth_request con 0x7fd3640a02d0 auth_method 0 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient(hunting): handle_auth_done global_id 54183 payload 824 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 -- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 msgr2=0x7fd36409be30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:42.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 0x7fd36409be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 -- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 msgr2=0x7fd36409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 0x7fd36409b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 monclient: found mon.noname-a 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 -- 172.21.15.142:0/3548023972 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3540094f0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36259c700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 0x7fd36409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 -- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 msgr2=0x7fd36409b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd361d9b700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00ec80 tx=0x7fd34c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00cbe0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:42.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: dump: 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c00cd40 con 0x7fd3640a1cf0 2024-09-18T10:13:42.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 5 monclient: authenticate success, global_id 54183 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 10 monclient: _renew_subs 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36409cdc0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0136c0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35a7fc700 10 monclient: dump: 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:42.012 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd35bfff700 4 set_mon_vals no callback set 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 10 monclient: _renew_subs 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.995+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36406af90 con 0x7fd3640a1cf0 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:42.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd364072a80 con 0x7fd3640a1cf0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd34c013a70 con 0x7fd3640a1cf0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd35a7fc700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 0x7fd344037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fd34c048ee0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36259c700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 0x7fd344037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:41.999+0000 7fd36259c700 10 monclient: get_auth_request con 0x7fd344035ac0 auth_method 0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.003+0000 7fd36259c700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 0x7fd344037f70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd36409ca80 tx=0x7fd354017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.007+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd34c0192c0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.007+0000 7fd35a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:42.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.007+0000 7fd35a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:42.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:42.123+0000 7ff108429700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:42.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:42.123+0000 7ff108429700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:42.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:42.123+0000 7ff108429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:42.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:42.123+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff10009da80 con 0x7ff1000a0a80 2024-09-18T10:13:42.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd36359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:42.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd36359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:42.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd36359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:42.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd3640a55d0 con 0x7fd3640a1cf0 2024-09-18T10:13:42.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd35a7fc700 1 -- 172.21.15.142:0/3548023972 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd34c04a070 con 0x7fd3640a1cf0 2024-09-18T10:13:42.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd35a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:42.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.231+0000 7fd35a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:42.235 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 msgr2=0x7fd344037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 0x7fd344037f70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd36409ca80 tx=0x7fd354017000 comp rx=0 tx=0).stop 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 10 monclient: shutdown 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 msgr2=0x7fd36409b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00ec80 tx=0x7fd34c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 shutdown_connections 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd344035ac0 0x7fd344037f70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd36412f810 0x7fd36409be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd3640a1cf0 0x7fd36409b8f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 --2- 172.21.15.142:0/3548023972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3640a02d0 0x7fd36409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 >> 172.21.15.142:0/3548023972 conn(0x7fd364071820 msgr2=0x7fd36412eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:42.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 shutdown_connections 2024-09-18T10:13:42.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:42.235+0000 7fd36359e700 1 -- 172.21.15.142:0/3548023972 wait complete. 2024-09-18T10:13:44.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:44.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:44.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:44.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:44.943+0000 7ff0f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:45.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:45.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: get_monmap_and_config 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: build_initial_monmap 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: monmap: 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:45.359829+0000 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:45.359829+0000 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:45.361 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:45.362 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:45.362 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:45.362 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 Processor -- start 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 -- start start 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: init 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:13:45.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f072809cdd0 0x7f072809e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): picked mon.noname-a con 0x7f072809cdd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 0x7f07280a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): picked mon.noname-b con 0x7f07280a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07280a0bf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072809d1a0 con 0x7f07280a0bf0 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072809e980 con 0x7f072809cdd0 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072809d780 con 0x7f07280a02e0 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072d251700 10 monclient(hunting): authenticate will time out at 1114.285726s 2024-09-18T10:13:45.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072659c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 msgr2=0x7f07280a06b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:45.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f072809cdd0 0x7f072809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 0x7f07280a06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f072809cdd0 0x7f072809e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51314/0 (socket says 172.21.15.142:51314) 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 -- 172.21.15.142:0/1951631677 learned_addr learned my addr 172.21.15.142:0/1951631677 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): get_auth_request con 0x7f072809cdd0 auth_method 0 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f072759e700 1 --1- 172.21.15.142:0/1951631677 >> v1:172.21.15.142:6789/0 conn(0x7f07280a0bf0 0x7f0728138a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:42886/0 (socket says 172.21.15.142:42886) 2024-09-18T10:13:45.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient(hunting): handle_auth_done global_id 54186 payload 274 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 -- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 msgr2=0x7f07280a06b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 --2- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 0x7f07280a06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 -- 172.21.15.142:0/1951631677 >> v1:172.21.15.142:6789/0 conn(0x7f07280a0bf0 legacy=0x7f0728138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 monclient: found mon.noname-a 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.359+0000 7f0726d9d700 1 -- 172.21.15.142:0/1951631677 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072809dd40 con 0x7f072809cdd0 2024-09-18T10:13:45.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0726d9d700 1 -- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f072809cdd0 msgr2=0x7f072809e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0726d9d700 1 --2- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f072809cdd0 0x7f072809e440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0718009f20 tx=0x7f0718009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134a3adf8f95b88b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 1 -- 172.21.15.142:0/1951631677 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071800e070 con 0x7f072809cdd0 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: dump: 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.372 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: _finish_auth 0 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 5 monclient: authenticate success, global_id 54186 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 1 -- 172.21.15.142:0/1951631677 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071800bc90 con 0x7f072809cdd0 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 4 set_mon_vals no callback set 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 1 -- 172.21.15.142:0/1951631677 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718012660 con 0x7f072809cdd0 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:45.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f0725d9b700 10 monclient: dump: 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 10 monclient: shutdown 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:45.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f072809cdd0 msgr2=0x7f072809e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 --2- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f072809cdd0 0x7f072809e440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0718009f20 tx=0x7f0718009840 comp rx=0 tx=0).stop 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- 172.21.15.142:0/1951631677 shutdown_connections 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 --2- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f07280a02e0 0x7f07280a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 --2- 172.21.15.142:0/1951631677 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f072809cdd0 0x7f072809e440 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- 172.21.15.142:0/1951631677 >> 172.21.15.142:0/1951631677 conn(0x7f0728071680 msgr2=0x7f07280a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- 172.21.15.142:0/1951631677 shutdown_connections 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- 172.21.15.142:0/1951631677 wait complete. 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 10 monclient: build_initial_monmap 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 10 monclient: monmap: 2024-09-18T10:13:45.375 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:45.376 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 Processor -- start 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 1 -- start start 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.363+0000 7f072d251700 10 monclient: init 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07280a02e0 0x7f0728141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): picked mon.noname-b con 0x7f07280a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): picked mon.noname-a con 0x7f07280a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:45.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 0x7f072808d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): picked mon.noname-c con 0x7f072808cf40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0728091780 con 0x7f07280a0bf0 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072809c1e0 con 0x7f07280a02e0 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0728090b20 con 0x7f072808cf40 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072d251700 10 monclient(hunting): authenticate will time out at 1114.292467s 2024-09-18T10:13:45.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072759e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 msgr2=0x7f072808d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 0x7f072808d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51326/0 (socket says 172.21.15.142:51326) 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 -- 172.21.15.142:0/3852498641 learned_addr learned my addr 172.21.15.142:0/3852498641 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): get_auth_request con 0x7f07280a0bf0 auth_method 0 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07280a02e0 0x7f0728141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:45.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): get_auth_request con 0x7f07280a02e0 auth_method 0 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f0726d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient(hunting): handle_auth_done global_id 54189 payload 824 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 -- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 msgr2=0x7f072808d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 0x7f072808d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 -- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07280a02e0 msgr2=0x7f0728141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07280a02e0 0x7f0728141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 monclient: found mon.noname-a 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 -- 172.21.15.142:0/3852498641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718009ae0 con 0x7f07280a0bf0 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 -- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 msgr2=0x7f07281422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.367+0000 7f072659c700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f071000b7e0 tx=0x7f0710009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710003e20 con 0x7f07280a0bf0 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: dump: 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:45.383 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: _finish_auth 0 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0710006150 con 0x7f07280a0bf0 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 5 monclient: authenticate success, global_id 54189 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _renew_subs 2024-09-18T10:13:45.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0728098470 con 0x7f07280a0bf0 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710006510 con 0x7f07280a0bf0 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072559a700 4 set_mon_vals no callback set 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 10 monclient: dump: 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:45.385 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _renew_subs 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072808ea30 con 0x7f07280a0bf0 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07280696a0 con 0x7f07280a0bf0 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f07100103c0 con 0x7f07280a0bf0 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 0x7f0708037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:45.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f0710048280 con 0x7f07280a0bf0 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f0726d9d700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 0x7f0708037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.371+0000 7f0726d9d700 10 monclient: get_auth_request con 0x7f0708035a70 auth_method 0 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.375+0000 7f0726d9d700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 0x7f0708037f20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0718009f20 tx=0x7f0718017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.379+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f071000e8a0 con 0x7f07280a0bf0 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.379+0000 7f071effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.379+0000 7f071effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f072d251700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f072d251700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f072d251700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f072808f600 con 0x7f07280a0bf0 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f071effd700 1 -- 172.21.15.142:0/3852498641 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f0710017080 con 0x7f07280a0bf0 2024-09-18T10:13:45.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f071effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:45.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.599+0000 7f071effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:45.604 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 msgr2=0x7f0708037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 0x7f0708037f20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0718009f20 tx=0x7f0718017000 comp rx=0 tx=0).stop 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 10 monclient: shutdown 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 msgr2=0x7f07281422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:45.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f071000b7e0 tx=0x7f0710009220 comp rx=0 tx=0).stop 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 shutdown_connections 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0708035a70 0x7f0708037f20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f072808cf40 0x7f072808d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f07280a0bf0 0x7f07281422a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 --2- 172.21.15.142:0/3852498641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07280a02e0 0x7f0728141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 >> 172.21.15.142:0/3852498641 conn(0x7f0728071680 msgr2=0x7f07280a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 shutdown_connections 2024-09-18T10:13:45.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:45.603+0000 7f072d251700 1 -- 172.21.15.142:0/3852498641 wait complete. 2024-09-18T10:13:45.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:45.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:45.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:45.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:46.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:46.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:46.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:46.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:47.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:47.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:47.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:47.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:48.618 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:48.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:48.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: build_initial_monmap 2024-09-18T10:13:48.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: monmap: 2024-09-18T10:13:48.730 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:48.729058+0000 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:48.729058+0000 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.731 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:48.732 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:48.732 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:48.732 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:48.732 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:48.732 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 Processor -- start 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 -- start start 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: init 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe85809aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 0x7fe858134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe858132570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe858134e90 0x7fe858137270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe858134e90 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85809aeb0 con 0x7fe85809aae0 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85806f6d0 con 0x7fe858134e90 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8580955a0 con 0x7fe858132570 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe85759e700 10 monclient(hunting): authenticate will time out at 1117.654565s 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe855d9b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 msgr2=0x7fe858134950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe855d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 0x7fe858134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe858134e90 0x7fe858137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe858134e90 0x7fe858137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51330/0 (socket says 172.21.15.142:51330) 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 -- 172.21.15.142:0/3777889889 learned_addr learned my addr 172.21.15.142:0/3777889889 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:48.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): get_auth_request con 0x7fe858134e90 auth_method 0 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient(hunting): handle_auth_done global_id 54192 payload 274 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 -- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 msgr2=0x7fe858134950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 --2- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 0x7fe858134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 -- 172.21.15.142:0/3777889889 >> v1:172.21.15.150:6789/0 conn(0x7fe85809aae0 legacy=0x7fe858096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 monclient: found mon.noname-a 2024-09-18T10:13:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.727+0000 7fe856d9d700 1 -- 172.21.15.142:0/3777889889 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe858095b60 con 0x7fe858134e90 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe856d9d700 1 -- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 msgr2=0x7fe858137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe856d9d700 1 --2- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe858137270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8500099b0 tx=0x7fe850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1233f575d9a16f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 1 -- 172.21.15.142:0/3777889889 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85000bb50 con 0x7fe858134e90 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: dump: 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: _finish_auth 0 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 1 -- 172.21.15.142:0/3777889889 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85000bcb0 con 0x7fe858134e90 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 5 monclient: authenticate success, global_id 54192 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 1 -- 172.21.15.142:0/3777889889 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850011670 con 0x7fe858134e90 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 4 set_mon_vals no callback set 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85559a700 10 monclient: dump: 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 10 monclient: shutdown 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 -- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 msgr2=0x7fe858137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 --2- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe858137270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8500099b0 tx=0x7fe850009250 comp rx=0 tx=0).stop 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 -- 172.21.15.142:0/3777889889 shutdown_connections 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 --2- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe858137270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 --2- 172.21.15.142:0/3777889889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe858132570 0x7fe858134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 -- 172.21.15.142:0/3777889889 >> 172.21.15.142:0/3777889889 conn(0x7fe8580a1fe0 msgr2=0x7fe858070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 -- 172.21.15.142:0/3777889889 shutdown_connections 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.731+0000 7fe85759e700 1 -- 172.21.15.142:0/3777889889 wait complete. 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient: build_initial_monmap 2024-09-18T10:13:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient: monmap: 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 Processor -- start 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 -- start start 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient: init 2024-09-18T10:13:48.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 0x7fe85806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-c con 0x7fe85809aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 0x7fe85806ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-b con 0x7fe858132570 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): picked mon.noname-a con 0x7fe858134e90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8581316b0 con 0x7fe858134e90 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85806f8d0 con 0x7fe858132570 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:48.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe858134ac0 con 0x7fe85809aae0 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85759e700 10 monclient(hunting): authenticate will time out at 1117.662013s 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85659c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 msgr2=0x7fe85806d8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe85659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 0x7fe85806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51340/0 (socket says 172.21.15.142:51340) 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 -- 172.21.15.142:0/3807020422 learned_addr learned my addr 172.21.15.142:0/3807020422 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): get_auth_request con 0x7fe858134e90 auth_method 0 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 0x7fe85806ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:48.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): get_auth_request con 0x7fe858132570 auth_method 0 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient(hunting): handle_auth_done global_id 54195 payload 824 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 -- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 msgr2=0x7fe85806d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 0x7fe85806d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 -- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 msgr2=0x7fe85806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:48.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 0x7fe85806ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 monclient: found mon.noname-a 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe856d9d700 1 -- 172.21.15.142:0/3807020422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8500094f0 con 0x7fe858134e90 2024-09-18T10:13:48.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.735+0000 7fe855d9b700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 0x7fe85806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe856d9d700 1 -- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 msgr2=0x7fe85806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe856d9d700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe8500117e0 tx=0x7fe850016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85001b450 con 0x7fe858134e90 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: dump: 2024-09-18T10:13:48.754 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 5 monclient: authenticate success, global_id 54195 2024-09-18T10:13:48.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe850020070 con 0x7fe858134e90 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _renew_subs 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe858128350 con 0x7fe858134e90 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85001b7e0 con 0x7fe858134e90 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe8467fc700 10 monclient: dump: 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.756 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe847fff700 4 set_mon_vals no callback set 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _renew_subs 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe858128d30 con 0x7fe858134e90 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.739+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe858072a80 con 0x7fe858134e90 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.743+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe850022030 con 0x7fe858134e90 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.743+0000 7fe8467fc700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 0x7fe838037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.743+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(123..123 src has 58..123) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fe85004ab20 con 0x7fe858134e90 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.743+0000 7fe85659c700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 0x7fe838037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.743+0000 7fe85659c700 10 monclient: get_auth_request con 0x7fe838035aa0 auth_method 0 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.747+0000 7fe85659c700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 0x7fe838037f50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe84c007fd0 tx=0x7fe84c007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.747+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe85002cb70 con 0x7fe858134e90 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.747+0000 7fe8467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.747+0000 7fe8467fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:48.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:48.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:48.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:48.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:48.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.971+0000 7fe85759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:48.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.971+0000 7fe85759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:48.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.971+0000 7fe85759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:48.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.971+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe858072a80 con 0x7fe858134e90 2024-09-18T10:13:48.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe8467fc700 1 -- 172.21.15.142:0/3807020422 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fe85002cb70 con 0x7fe858134e90 2024-09-18T10:13:48.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe8467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:48.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe8467fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:48.977 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:48.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 msgr2=0x7fe838037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:48.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 0x7fe838037f50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe84c007fd0 tx=0x7fe84c007b00 comp rx=0 tx=0).stop 2024-09-18T10:13:48.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 10 monclient: shutdown 2024-09-18T10:13:48.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 msgr2=0x7fe85806e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe8500117e0 tx=0x7fe850016000 comp rx=0 tx=0).stop 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 shutdown_connections 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe838035aa0 0x7fe838037f50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe858134e90 0x7fe85806e330 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe858132570 0x7fe85806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 --2- 172.21.15.142:0/3807020422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe85809aae0 0x7fe85806d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 >> 172.21.15.142:0/3807020422 conn(0x7fe8580a1fe0 msgr2=0x7fe8581368b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 shutdown_connections 2024-09-18T10:13:48.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:48.975+0000 7fe85759e700 1 -- 172.21.15.142:0/3807020422 wait complete. 2024-09-18T10:13:49.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:49.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:49.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:49.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:50.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:50.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:50.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:50.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:51.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:51.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:51.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:51.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:51.992 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:13:52.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: get_monmap_and_config 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: build_initial_monmap 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: monmap: 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:52.105418+0000 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:52.105418+0000 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:52.107 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:52.108 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:52.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 Processor -- start 2024-09-18T10:13:52.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 -- start start 2024-09-18T10:13:52.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: init 2024-09-18T10:13:52.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:52.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:13:52.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f338809abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:52.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3388097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:52.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-c con 0x7f338809db80 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33880958c0 con 0x7f3388097880 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3388095ef0 con 0x7f338809abd0 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338806fa60 con 0x7f338809db80 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338f6a0700 10 monclient(hunting): authenticate will time out at 1121.030898s 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338cc3b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3388097880 0x7f338809c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44600/0 (socket says 172.21.15.142:44600) 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 learned_addr learned my addr 172.21.15.142:0/3945987492 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:52.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 10 monclient(hunting): get_auth_request con 0x7f338809db80 auth_method 0 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:52.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.103+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:52.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.107+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 >> v1:172.21.15.142:6789/0 conn(0x7f3388097880 legacy=0x7f338809c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:52.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.107+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v1:172.21.15.150:6789/0 conn(0x7f338809abd0 legacy=0x7f338809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 10 monclient(hunting): get_auth_request con 0x7f338809db80 auth_method 0 2024-09-18T10:13:52.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:52.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.307+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 10 monclient(hunting): get_auth_request con 0x7f338809db80 auth_method 0 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:52.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:52.711+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:52.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:52.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:52.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:52.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:53.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 10 monclient(hunting): get_auth_request con 0x7f338809db80 auth_method 0 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:53.515+0000 7f338dc3d700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:13:53.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:53.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:53.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:53.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:54.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:54.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:54.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:54.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:54.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:54.943+0000 7ff0f17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:55.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): tick 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 monclient(hunting): continuing hunt 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 msgr2=0x7f33881386a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.150:3300/0 conn(0x7f338809db80 0x7f33881386a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 -- 172.21.15.142:0/3945987492 >> v1:172.21.15.150:6789/0 conn(0x7f338809abd0 legacy=0x7f338809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 -- 172.21.15.142:0/3945987492 >> v1:172.21.15.142:6789/0 conn(0x7f3388097880 legacy=0x7f338809c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient: _add_conns ranks=[3,2] 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): picked mon.noname-a con 0x7f336c003810 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): picked mon.noname-b con 0x7f336c006360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:55.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 -- 172.21.15.142:0/3945987492 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336c008ef0 con 0x7f336c003810 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 1 -- 172.21.15.142:0/3945987492 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c009060 con 0x7f336c006360 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f337effd700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 msgr2=0x7f336c0087f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338cc3b700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 10 monclient(hunting): get_auth_request con 0x7f336c003810 auth_method 0 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:55.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.103+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:55.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 msgr2=0x7f336c0087f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:55.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338cc3b700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:55.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:55.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 10 monclient(hunting): get_auth_request con 0x7f336c003810 auth_method 0 2024-09-18T10:13:55.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:55.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.307+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 msgr2=0x7f336c0087f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338cc3b700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 10 monclient(hunting): get_auth_request con 0x7f336c003810 auth_method 0 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 msgr2=0x7f336c005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:13:55.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:55.711+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:13:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:55.943+0000 7ff0f17fa700 10 monclient: tick 2024-09-18T10:13:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:55.943+0000 7ff0f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.427+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+0 (secure 0 0 0) 0x7ff0fc017080 con 0x7ff1000a0a80 2024-09-18T10:13:56.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.427+0000 7ff0f27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:56.430 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.427+0000 7ff0f27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:56.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338cc3b700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 msgr2=0x7f336c0087f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338cc3b700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> v2:172.21.15.142:3300/0 conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): get_auth_request con 0x7f336c003810 auth_method 0 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient(hunting): handle_auth_done global_id 54198 payload 274 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 msgr2=0x7f336c0087f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f336c006360 0x7f336c0087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 monclient: found mon.noname-a 2024-09-18T10:13:56.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338809afa0 con 0x7f336c003810 2024-09-18T10:13:56.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 msgr2=0x7f336c005c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.515+0000 7f338d43c700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f336c005c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3384007bd0 tx=0x7f3384009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ed740c0f091a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 1 -- 172.21.15.142:0/3945987492 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33840073b0 con 0x7f336c003810 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: dump: 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.521 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 1 -- 172.21.15.142:0/3945987492 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384007510 con 0x7f336c003810 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 5 monclient: authenticate success, global_id 54198 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 1 -- 172.21.15.142:0/3945987492 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384005a20 con 0x7f336c003810 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 4 set_mon_vals no callback set 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f337ffff700 10 monclient: dump: 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.523 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.524 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 10 monclient: shutdown 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 -- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 msgr2=0x7f336c005c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f336c005c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3384007bd0 tx=0x7f3384009040 comp rx=0 tx=0).stop 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338dc3d700 1 -- 172.21.15.142:0/3945987492 reap_dead start 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 -- 172.21.15.142:0/3945987492 shutdown_connections 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 --2- 172.21.15.142:0/3945987492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f336c005c80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 -- 172.21.15.142:0/3945987492 >> 172.21.15.142:0/3945987492 conn(0x7f33880a14e0 msgr2=0x7f33880a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 -- 172.21.15.142:0/3945987492 shutdown_connections 2024-09-18T10:13:56.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 -- 172.21.15.142:0/3945987492 wait complete. 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 10 monclient: monmap: 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.519+0000 7f338f6a0700 1 Processor -- start 2024-09-18T10:13:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 -- start start 2024-09-18T10:13:56.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient: init 2024-09-18T10:13:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:13:56.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-a con 0x7f336c003810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 0x7f338808be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-c con 0x7f338808ba30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f338808c3a0 0x7f3388145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): picked mon.noname-b con 0x7f338808c3a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338806ea90 con 0x7f336c003810 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338808da50 con 0x7f338808c3a0 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3388097c90 con 0x7f338808ba30 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338f6a0700 10 monclient(hunting): authenticate will time out at 1125.449452s 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338cc3b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 msgr2=0x7f338808be60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338cc3b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 0x7f338808be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47056/0 (socket says 172.21.15.142:47056) 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 -- 172.21.15.142:0/7333955 learned_addr learned my addr 172.21.15.142:0/7333955 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): get_auth_request con 0x7f336c003810 auth_method 0 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f338808c3a0 0x7f3388145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): get_auth_request con 0x7f338808c3a0 auth_method 0 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338dc3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient(hunting): handle_auth_done global_id 54201 payload 824 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 -- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 msgr2=0x7f338808be60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 0x7f338808be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 -- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f338808c3a0 msgr2=0x7f3388145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f338808c3a0 0x7f3388145e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 monclient: found mon.noname-a 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.523+0000 7f338d43c700 1 -- 172.21.15.142:0/7333955 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338400f040 con 0x7f336c003810 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338d43c700 1 -- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 msgr2=0x7f338808e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338d43c700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f336c009030 tx=0x7f338401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384011980 con 0x7f336c003810 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: dump: 2024-09-18T10:13:56.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384011b00 con 0x7f336c003810 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 5 monclient: authenticate success, global_id 54201 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384011ec0 con 0x7f336c003810 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _renew_subs 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337f7fe700 4 set_mon_vals no callback set 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338808f050 con 0x7f336c003810 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 10 monclient: dump: 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _renew_subs 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338808c810 con 0x7f336c003810 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33880696a0 con 0x7f336c003810 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3384015680 con 0x7f336c003810 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 0x7f336c038ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.527+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(124..124 src has 58..124) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f338404a420 con 0x7f336c003810 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.531+0000 7f338cc3b700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 0x7f336c038ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.531+0000 7f338cc3b700 10 monclient: get_auth_request con 0x7f336c036a30 auth_method 0 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.531+0000 7f338cc3b700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 0x7f336c038ee0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f338808d030 tx=0x7f3378007b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.535+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3384015930 con 0x7f336c003810 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.535+0000 7f337d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.535+0000 7f337d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:56.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff108429700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff108429700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff108429700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7ff100063ed0 con 0x7ff1000a0a80 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff0f27fc700 1 -- 172.21.15.150:0/3612394285 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v124) v1 ==== 71+0+0 (secure 0 0 0) 0x7ff0fc00d2f0 con 0x7ff1000a0a80 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff0f27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:13:56.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.543+0000 7ff0f27fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 msgr2=0x7ff0e4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 0x7ff0e4037f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4004860 tx=0x7ff0f4007c10 comp rx=0 tx=0).stop 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 10 monclient: shutdown 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 msgr2=0x7ff10008d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00b7e0 tx=0x7ff0fc009220 comp rx=0 tx=0).stop 2024-09-18T10:13:56.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 shutdown_connections 2024-09-18T10:13:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff0e4035a70 0x7ff0e4037f20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1000a6090 0x7ff10008d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1000a0a80 0x7ff10008d1b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 --2- 172.21.15.150:0/3612394285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff10009e460 0x7ff10008cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 >> 172.21.15.150:0/3612394285 conn(0x7ff100071830 msgr2=0x7ff1000a4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 shutdown_connections 2024-09-18T10:13:56.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.547+0000 7ff108429700 1 -- 172.21.15.150:0/3612394285 wait complete. 2024-09-18T10:13:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T10:13:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:13:56.560 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.655+0000 7f61bed00700 10 monclient: get_monmap_and_config 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.655+0000 7f61bed00700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.655+0000 7f61bed00700 10 monclient: monmap: 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:56.659660+0000 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:56.659660+0000 2024-09-18T10:13:56.660 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 Processor -- start 2024-09-18T10:13:56.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 -- start start 2024-09-18T10:13:56.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient: init 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61b80a0290 0x7f61b809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-c con 0x7f61b80a0290 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61b80a0660 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f61b8137a20 0x7f61b8139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-a con 0x7f61b8137a20 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b809e360 con 0x7f61b80a0660 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b809ef80 con 0x7f61b8137a20 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80a1fb0 con 0x7f61b80a0290 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bed00700 10 monclient(hunting): authenticate will time out at 1118.754398s 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61b80a0290 0x7f61b809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61b7fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f61b80a0660 0x7f61b80a0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:36928/0 (socket says 172.21.15.150:36928) 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): get_auth_request con 0x7f61b80a0290 auth_method 0 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61b7fff700 1 -- 172.21.15.150:0/1958138804 learned_addr learned my addr 172.21.15.150:0/1958138804 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 1 --2- 172.21.15.150:0/1958138804 >> v2:172.21.15.142:3300/0 conn(0x7f61b8137a20 0x7f61b8139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): get_auth_request con 0x7f61b8137a20 auth_method 0 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bd29d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient(hunting): handle_auth_done global_id 15388 payload 274 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 -- 172.21.15.150:0/1958138804 >> v2:172.21.15.142:3300/0 conn(0x7f61b8137a20 msgr2=0x7f61b8139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 --2- 172.21.15.150:0/1958138804 >> v2:172.21.15.142:3300/0 conn(0x7f61b8137a20 0x7f61b8139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 -- 172.21.15.150:0/1958138804 >> v1:172.21.15.150:6789/0 conn(0x7f61b80a0660 legacy=0x7f61b80a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 monclient: found mon.noname-c 2024-09-18T10:13:56.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.659+0000 7f61bca9c700 1 -- 172.21.15.150:0/1958138804 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b809da50 con 0x7f61b80a0290 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bca9c700 1 -- 172.21.15.150:0/1958138804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 msgr2=0x7f61b809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bca9c700 1 --2- 172.21.15.150:0/1958138804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b809ea40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f61b80691e0 tx=0x7f61a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10217145c6a1e26f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 1 -- 172.21.15.150:0/1958138804 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8011840 con 0x7f61b80a0290 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: dump: 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.669 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 5 monclient: authenticate success, global_id 15388 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 1 -- 172.21.15.150:0/1958138804 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a80119a0 con 0x7f61b80a0290 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 1 -- 172.21.15.150:0/1958138804 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8011d40 con 0x7f61b80a0290 2024-09-18T10:13:56.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 4 set_mon_vals no callback set 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61b77fe700 10 monclient: dump: 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.671 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 10 monclient: shutdown 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 -- 172.21.15.150:0/1958138804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 msgr2=0x7f61b809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 --2- 172.21.15.150:0/1958138804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b809ea40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f61b80691e0 tx=0x7f61a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 -- 172.21.15.150:0/1958138804 shutdown_connections 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 --2- 172.21.15.150:0/1958138804 >> v2:172.21.15.142:3300/0 conn(0x7f61b8137a20 0x7f61b8139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 --2- 172.21.15.150:0/1958138804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b809ea40 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 -- 172.21.15.150:0/1958138804 >> 172.21.15.150:0/1958138804 conn(0x7f61b80a58a0 msgr2=0x7f61b80a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 -- 172.21.15.150:0/1958138804 shutdown_connections 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.663+0000 7f61bed00700 1 -- 172.21.15.150:0/1958138804 wait complete. 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient: monmap: 2024-09-18T10:13:56.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:56.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 Processor -- start 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 -- start start 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient: init 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b8141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-b con 0x7f61b80a0290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b80a0660 0x7f61b81423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-a con 0x7f61b80a0660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:56.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8137a20 0x7f61b8142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): picked mon.noname-c con 0x7f61b8137a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b812e0c0 con 0x7f61b80a0660 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8130190 con 0x7f61b80a0290 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b809c030 con 0x7f61b8137a20 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bed00700 10 monclient(hunting): authenticate will time out at 1118.762452s 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bd29d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8137a20 msgr2=0x7f61b8142910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bd29d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8137a20 0x7f61b8142910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b8141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b8141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53124/0 (socket says 172.21.15.150:53124) 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 -- 172.21.15.150:0/736509150 learned_addr learned my addr 172.21.15.150:0/736509150 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): get_auth_request con 0x7f61b80a0290 auth_method 0 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b80a0660 0x7f61b81423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): get_auth_request con 0x7f61b80a0660 auth_method 0 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient(hunting): handle_auth_done global_id 15391 payload 824 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 -- 172.21.15.150:0/736509150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8137a20 msgr2=0x7f61b8142910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61b8137a20 0x7f61b8142910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 -- 172.21.15.150:0/736509150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b80a0660 msgr2=0x7f61b81423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61b80a0660 0x7f61b81423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 monclient: found mon.noname-b 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.667+0000 7f61bca9c700 1 -- 172.21.15.150:0/736509150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8009420 con 0x7f61b80a0290 2024-09-18T10:13:56.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bca9c700 1 -- 172.21.15.150:0/736509150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 msgr2=0x7f61b8141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bca9c700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61b80a0290 0x7f61b8141e90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f61a8009f50 tx=0x7f61a8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800bb40 con 0x7f61b80a0290 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: dump: 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.681 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8020070 con 0x7f61b80a0290 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 5 monclient: authenticate success, global_id 15391 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a801b450 con 0x7f61b80a0290 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _renew_subs 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 1 -- 172.21.15.150:0/736509150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b812ec50 con 0x7f61b80a0290 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b6ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 10 monclient: dump: 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _renew_subs 2024-09-18T10:13:56.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 1 -- 172.21.15.150:0/736509150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b812e7d0 con 0x7f61b80a0290 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61bed00700 1 -- 172.21.15.150:0/736509150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b809d6c0 con 0x7f61b80a0290 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f61a8022030 con 0x7f61b80a0290 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f61a4035a70 0x7f61a4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(124..124 src has 58..124) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f61a804a710 con 0x7f61b80a0290 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b7fff700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f61a4035a70 0x7f61a4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.671+0000 7f61b7fff700 10 monclient: get_auth_request con 0x7f61a4035a70 auth_method 0 2024-09-18T10:13:56.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.675+0000 7f61b7fff700 1 --2- 172.21.15.150:0/736509150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f61a4035a70 0x7f61a4037f20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f61b81379f0 tx=0x7f61ac009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.683+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f61a801b800 con 0x7f61b80a0290 2024-09-18T10:13:56.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.683+0000 7f61b4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.683+0000 7f61b4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:56.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:13:56.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f338808f850 con 0x7f336c003810 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f337d7fa700 1 -- 172.21.15.142:0/7333955 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f338808f850 con 0x7f336c003810 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f337d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f337d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:56.762 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:13:56.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 msgr2=0x7f336c038ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 0x7f336c038ee0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f338808d030 tx=0x7f3378007b70 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 10 monclient: shutdown 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 msgr2=0x7f338808e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f336c009030 tx=0x7f338401c000 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 shutdown_connections 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f336c036a30 0x7f336c038ee0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f338808c3a0 0x7f3388145e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f338808ba30 0x7f338808be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 --2- 172.21.15.142:0/7333955 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f336c003810 0x7f338808e8e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 >> 172.21.15.142:0/7333955 conn(0x7f33880a14e0 msgr2=0x7f33880a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 shutdown_connections 2024-09-18T10:13:56.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.759+0000 7f338f6a0700 1 -- 172.21.15.142:0/7333955 wait complete. 2024-09-18T10:13:56.776 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:13:56.776 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:13:56.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:13:56.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.851+0000 7f61bed00700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:13:56.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.851+0000 7f61bed00700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:13:56.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.851+0000 7f61bed00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.851+0000 7f61bed00700 1 -- 172.21.15.150:0/736509150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f61b809d6c0 con 0x7f61b80a0290 2024-09-18T10:13:56.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.855+0000 7f61b4ff9700 1 -- 172.21.15.150:0/736509150 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 pool 'fooset' not found v124) v1 ==== 152+0+0 (secure 0 0 0) 0x7f61a8024b70 con 0x7f61b80a0290 2024-09-18T10:13:56.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.855+0000 7f61b4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:13:56.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.855+0000 7f61b4ff9700 10 monclient: _finish_command 2 = mon:2 pool 'fooset' not found 2024-09-18T10:13:56.858 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOENT: pool 'fooset' not found 2024-09-18T10:13:56.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:13:56.862 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create fooset 8 2024-09-18T10:13:56.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: get_monmap_and_config 2024-09-18T10:13:56.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: monmap: 2024-09-18T10:13:56.882 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:56.882 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:56.881379+0000 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:56.881379+0000 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:56.883 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:56.884 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:56.884 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 Processor -- start 2024-09-18T10:13:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 -- start start 2024-09-18T10:13:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: init 2024-09-18T10:13:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:13:56.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbf6809abd0 0x7fbf680960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-a con 0x7fbf6809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbf6809b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf6809c5e0 0x7fbf68137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-c con 0x7fbf6809c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6809dc90 con 0x7fbf6809b740 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6809b500 con 0x7fbf6809abd0 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6809afa0 con 0x7fbf6809c5e0 2024-09-18T10:13:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6759e700 10 monclient(hunting): authenticate will time out at 1125.806988s 2024-09-18T10:13:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbf6809abd0 0x7fbf680960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf65d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbf6809b740 0x7fbf6809bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44610/0 (socket says 172.21.15.142:44610) 2024-09-18T10:13:56.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbf6809abd0 0x7fbf680960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47072/0 (socket says 172.21.15.142:47072) 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf65d9b700 1 -- 172.21.15.142:0/1310292988 learned_addr learned my addr 172.21.15.142:0/1310292988 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): get_auth_request con 0x7fbf6809abd0 auth_method 0 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 1 --2- 172.21.15.142:0/1310292988 >> v2:172.21.15.150:3300/0 conn(0x7fbf6809c5e0 0x7fbf68137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): get_auth_request con 0x7fbf6809c5e0 auth_method 0 2024-09-18T10:13:56.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient(hunting): handle_auth_done global_id 54210 payload 274 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 -- 172.21.15.142:0/1310292988 >> v2:172.21.15.150:3300/0 conn(0x7fbf6809c5e0 msgr2=0x7fbf68137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1310292988 >> v2:172.21.15.150:3300/0 conn(0x7fbf6809c5e0 0x7fbf68137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 -- 172.21.15.142:0/1310292988 >> v1:172.21.15.142:6789/0 conn(0x7fbf6809b740 legacy=0x7fbf6809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 monclient: found mon.noname-a 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.879+0000 7fbf6659c700 1 -- 172.21.15.142:0/1310292988 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6806f810 con 0x7fbf6809abd0 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6659c700 1 -- 172.21.15.142:0/1310292988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809abd0 msgr2=0x7fbf680960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1310292988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809abd0 0x7fbf680960e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf68076de0 tx=0x7fbf580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e7775864fc8d58 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 1 -- 172.21.15.142:0/1310292988 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58011840 con 0x7fbf6809abd0 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: dump: 2024-09-18T10:13:56.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 1 -- 172.21.15.142:0/1310292988 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580119c0 con 0x7fbf6809abd0 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 5 monclient: authenticate success, global_id 54210 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 1 -- 172.21.15.142:0/1310292988 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58011d80 con 0x7fbf6809abd0 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 4 set_mon_vals no callback set 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6559a700 10 monclient: dump: 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.897 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 10 monclient: shutdown 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 -- 172.21.15.142:0/1310292988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809abd0 msgr2=0x7fbf680960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1310292988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809abd0 0x7fbf680960e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf68076de0 tx=0x7fbf580091a0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 -- 172.21.15.142:0/1310292988 shutdown_connections 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1310292988 >> v2:172.21.15.150:3300/0 conn(0x7fbf6809c5e0 0x7fbf68137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1310292988 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809abd0 0x7fbf680960e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.883+0000 7fbf6759e700 1 -- 172.21.15.142:0/1310292988 >> 172.21.15.142:0/1310292988 conn(0x7fbf68070ef0 msgr2=0x7fbf680712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- 172.21.15.142:0/1310292988 shutdown_connections 2024-09-18T10:13:56.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- 172.21.15.142:0/1310292988 wait complete. 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient: monmap: 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:56.905 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 Processor -- start 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- start start 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient: init 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf6809abd0 0x7fbf68092670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-c con 0x7fbf6809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6809b740 0x7fbf68094220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-b con 0x7fbf6809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): picked mon.noname-a con 0x7fbf6809c5e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:56.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68090320 con 0x7fbf6809c5e0 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6809d240 con 0x7fbf6809b740 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680a0030 con 0x7fbf6809abd0 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6759e700 10 monclient(hunting): authenticate will time out at 1125.815026s 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6659c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf6809abd0 msgr2=0x7fbf68092670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf6659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf6809abd0 0x7fbf68092670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6809b740 0x7fbf68094220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47080/0 (socket says 172.21.15.142:47080) 2024-09-18T10:13:56.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 learned_addr learned my addr 172.21.15.142:0/1218020321 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): get_auth_request con 0x7fbf6809c5e0 auth_method 0 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): get_auth_request con 0x7fbf6809b740 auth_method 0 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf65d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf6809abd0 msgr2=0x7fbf68092670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf6809abd0 0x7fbf68092670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6809b740 msgr2=0x7fbf68094220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.887+0000 7fbf66d9d700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6809b740 0x7fbf68094220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 1 monclient: found mon.noname-a 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58009420 con 0x7fbf6809c5e0 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf65d9b700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6809b740 0x7fbf68094220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:13:56.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 msgr2=0x7fbf68098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf66d9d700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009e10 tx=0x7fbf6000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6000cd60 con 0x7fbf6809c5e0 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: dump: 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.910 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 5 monclient: authenticate success, global_id 54213 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6000f040 con 0x7fbf6809c5e0 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600146d0 con 0x7fbf6809c5e0 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 10 monclient: dump: 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.911 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 monclient: _renew_subs 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf64d99700 4 set_mon_vals no callback set 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6809a570 con 0x7fbf6809c5e0 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 monclient: _renew_subs 2024-09-18T10:13:56.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68094760 con 0x7fbf6809c5e0 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbf60014a80 con 0x7fbf6809c5e0 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 0x7fbf68099350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.891+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68091580 con 0x7fbf68098f80 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 0x7fbf68099350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient: get_auth_request con 0x7fbf68098f80 auth_method 0 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf48035a70 0x7fbf48037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(124..124 src has 58..124) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fbf60049450 con 0x7fbf6809c5e0 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 10 monclient(hunting): handle_auth_done global_id 54216 payload 824 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf680916f0 con 0x7fbf68098f80 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf65d9b700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf48035a70 0x7fbf48037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf65d9b700 10 monclient: get_auth_request con 0x7fbf48035a70 auth_method 0 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 msgr2=0x7fbf68099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 0x7fbf68099350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbf58016000 tx=0x7fbf58017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5801f070 con 0x7fbf68098f80 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 10 monclient: dump: 2024-09-18T10:13:56.914 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf65d9b700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf48035a70 0x7fbf48037f20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c006fd0 tx=0x7fbf5c007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbf5800b890 con 0x7fbf68098f80 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 msgr2=0x7fbf68099350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 0x7fbf68099350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbf58016000 tx=0x7fbf58017000 comp rx=0 tx=0).stop 2024-09-18T10:13:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 msgr2=0x7fbf68099350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:56.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.895+0000 7fbf567fc700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68098f80 0x7fbf68099350 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbf58016000 tx=0x7fbf58017000 comp rx=0 tx=0).stop 2024-09-18T10:13:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:56.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 0x7fbf68087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680648a0 con 0x7fbf68099890 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 0x7fbf68087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient: get_auth_request con 0x7fbf68099890 auth_method 0 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 10 monclient(hunting): handle_auth_done global_id 54219 payload 824 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 1 -- 172.21.15.142:0/1218020321 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbf68072a80 con 0x7fbf68099890 2024-09-18T10:13:56.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 msgr2=0x7fbf68087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf6659c700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 0x7fbf68087fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf68072200 tx=0x7fbf5801c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.907+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5801a840 con 0x7fbf68099890 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 10 monclient: dump: 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.919 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fbf5801a9a0 con 0x7fbf68099890 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 msgr2=0x7fbf68087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 0x7fbf68087fe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf68072200 tx=0x7fbf5801c050 comp rx=0 tx=0).stop 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 msgr2=0x7fbf68087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:56.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf567fc700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf68099890 0x7fbf68087fe0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbf68072200 tx=0x7fbf5801c050 comp rx=0 tx=0).stop 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 60, 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:13:56.921 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:56.922 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:56.923 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:56.924 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:56.925 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:56.926 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:56.927 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:56.928 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:56.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:56.930 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:56.931 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:56.932 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:56.933 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf48035a70 msgr2=0x7fbf48037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbf48035a70 0x7fbf48037f20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c006fd0 tx=0x7fbf5c007f90 comp rx=0 tx=0).stop 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 10 monclient: shutdown 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 msgr2=0x7fbf68098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009e10 tx=0x7fbf6000b040 comp rx=0 tx=0).stop 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf66d9d700 1 -- 172.21.15.142:0/1218020321 reap_dead start 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 shutdown_connections 2024-09-18T10:13:56.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 --2- 172.21.15.142:0/1218020321 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf6809c5e0 0x7fbf68098a40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 >> 172.21.15.142:0/1218020321 conn(0x7fbf68070ef0 msgr2=0x7fbf6809de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 shutdown_connections 2024-09-18T10:13:56.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:56.911+0000 7fbf6759e700 1 -- 172.21.15.142:0/1218020321 wait complete. 2024-09-18T10:13:56.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:13:56.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:13:56.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:13:56.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.959+0000 7ff3607d1700 10 monclient: get_monmap_and_config 2024-09-18T10:13:56.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.959+0000 7ff3607d1700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.959+0000 7ff3607d1700 10 monclient: monmap: 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:56.963103+0000 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:56.963103+0000 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:56.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 Processor -- start 2024-09-18T10:13:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 -- start start 2024-09-18T10:13:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient: init 2024-09-18T10:13:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:13:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff35806ec00 0x7ff3581300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-c con 0x7ff35806ec00 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff35806e570 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 0x7ff3581377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-b con 0x7ff3581353d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35809f500 con 0x7ff35806e570 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff358096a70 con 0x7ff35806ec00 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff358097090 con 0x7ff3581353d0 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff3607d1700 10 monclient(hunting): authenticate will time out at 1119.057880s 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35ed6e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 msgr2=0x7ff3581377b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff35806ec00 0x7ff3581300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35ed6e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 0x7ff3581377b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff35806ec00 0x7ff3581300a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53138/0 (socket says 172.21.15.150:53138) 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 -- 172.21.15.150:0/2053363404 learned_addr learned my addr 172.21.15.150:0/2053363404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): get_auth_request con 0x7ff35806ec00 auth_method 0 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient(hunting): handle_auth_done global_id 15400 payload 274 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 -- 172.21.15.150:0/2053363404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 msgr2=0x7ff3581377b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2053363404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 0x7ff3581377b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 -- 172.21.15.150:0/2053363404 >> v1:172.21.15.150:6789/0 conn(0x7ff35806e570 legacy=0x7ff3581305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 monclient: found mon.noname-c 2024-09-18T10:13:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.963+0000 7ff35e56d700 1 -- 172.21.15.150:0/2053363404 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35806efd0 con 0x7ff35806ec00 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35e56d700 1 -- 172.21.15.150:0/2053363404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806ec00 msgr2=0x7ff3581300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2053363404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806ec00 0x7ff3581300a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0099b0 tx=0x7ff34c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb92fed7d0e9b491 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 1 -- 172.21.15.150:0/2053363404 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00baf0 con 0x7ff35806ec00 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: dump: 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 5 monclient: authenticate success, global_id 15400 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 1 -- 172.21.15.150:0/2053363404 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00bc50 con 0x7ff35806ec00 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 4 set_mon_vals no callback set 2024-09-18T10:13:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 1 -- 172.21.15.150:0/2053363404 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c011420 con 0x7ff35806ec00 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff35d56b700 10 monclient: dump: 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 10 monclient: shutdown 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- 172.21.15.150:0/2053363404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806ec00 msgr2=0x7ff3581300a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2053363404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806ec00 0x7ff3581300a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0099b0 tx=0x7ff34c009250 comp rx=0 tx=0).stop 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- 172.21.15.150:0/2053363404 shutdown_connections 2024-09-18T10:13:56.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2053363404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3581353d0 0x7ff3581377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2053363404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806ec00 0x7ff3581300a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- 172.21.15.150:0/2053363404 >> 172.21.15.150:0/2053363404 conn(0x7ff3580a40a0 msgr2=0x7ff3580a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- 172.21.15.150:0/2053363404 shutdown_connections 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- 172.21.15.150:0/2053363404 wait complete. 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 10 monclient: build_initial_monmap 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 10 monclient: monmap: 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 Processor -- start 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 1 -- start start 2024-09-18T10:13:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.967+0000 7ff3607d1700 10 monclient: init 2024-09-18T10:13:56.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-b con 0x7ff35806e570 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 0x7ff35809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-c con 0x7ff35806ec00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 0x7ff35809bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): picked mon.noname-a con 0x7ff3581353d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35809dda0 con 0x7ff3581353d0 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580a27b0 con 0x7ff35806e570 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:56.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35811a290 con 0x7ff35806ec00 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff3607d1700 10 monclient(hunting): authenticate will time out at 1119.065050s 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53142/0 (socket says 172.21.15.150:53142) 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 -- 172.21.15.150:0/2424144712 learned_addr learned my addr 172.21.15.150:0/2424144712 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): get_auth_request con 0x7ff35806e570 auth_method 0 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35dd6c700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 msgr2=0x7ff35809ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:56.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35dd6c700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 0x7ff35809ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35ed6e700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 0x7ff35809bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient(hunting): handle_auth_done global_id 15403 payload 824 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 msgr2=0x7ff35809ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 0x7ff35809ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 msgr2=0x7ff35809bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 0x7ff35809bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient: _finish_hunting 0 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 monclient: found mon.noname-b 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0094f0 con 0x7ff35806e570 2024-09-18T10:13:56.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35ed6e700 10 monclient: get_auth_request con 0x7ff3581353d0 auth_method 0 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35ed6e700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 0x7ff35809bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 msgr2=0x7ff35813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c000ca0 con 0x7ff35806e570 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 10 monclient: dump: 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.985 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.971+0000 7ff34a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c020070 con 0x7ff35806e570 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 5 monclient: authenticate success, global_id 15403 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _renew_subs 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff358096d00 con 0x7ff35806e570 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c01b8b0 con 0x7ff35806e570 2024-09-18T10:13:56.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff35cd6a700 4 set_mon_vals no callback set 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 10 monclient: dump: 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:56.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _renew_subs 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35809d5a0 con 0x7ff35806e570 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff34c022030 con 0x7ff35806e570 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35811a4d0 con 0x7ff35806e570 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 0x7ff33c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(124..124 src has 58..124) v4 ==== 6642+0+0 (secure 0 0 0) 0x7ff34c04ab50 con 0x7ff35806e570 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff35dd6c700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 0x7ff33c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.975+0000 7ff35dd6c700 10 monclient: get_auth_request con 0x7ff33c035a70 auth_method 0 2024-09-18T10:13:56.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.979+0000 7ff35dd6c700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 0x7ff33c037f20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff354007fd0 tx=0x7ff354007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:56.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:56.983+0000 7ff35e56d700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c016000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7ff34c01bc60 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:13:57.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: get_monmap_and_config 2024-09-18T10:13:57.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: build_initial_monmap 2024-09-18T10:13:57.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: monmap: 2024-09-18T10:13:57.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:57.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:13:57.040567+0000 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:13:57.040567+0000 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 Processor -- start 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 -- start start 2024-09-18T10:13:57.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: init 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 0x7fa758096c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-b con 0x7fa758070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa75809cd80 0x7fa75809d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-a con 0x7fa75809cd80 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa75809d690 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa758068f50 con 0x7fa75809d690 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75809e6b0 con 0x7fa75809cd80 2024-09-18T10:13:57.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580967c0 con 0x7fa758070ef0 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75dbc9700 10 monclient(hunting): authenticate will time out at 1125.966019s 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75cbc7700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 msgr2=0x7fa758096c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa75cbc7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 0x7fa758096c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa75809cd80 0x7fa75809d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa75809cd80 0x7fa75809d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47116/0 (socket says 172.21.15.142:47116) 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 -- 172.21.15.142:0/1775789548 learned_addr learned my addr 172.21.15.142:0/1775789548 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): get_auth_request con 0x7fa75809cd80 auth_method 0 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:57.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient(hunting): handle_auth_done global_id 54222 payload 274 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 -- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 msgr2=0x7fa758096c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 --2- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 0x7fa758096c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 -- 172.21.15.142:0/1775789548 >> v1:172.21.15.150:6789/0 conn(0x7fa75809d690 legacy=0x7fa758132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 monclient: found mon.noname-a 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:57.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.039+0000 7fa757fff700 1 -- 172.21.15.142:0/1775789548 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580712c0 con 0x7fa75809cd80 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa757fff700 1 -- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 msgr2=0x7fa75809d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa757fff700 1 --2- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75809d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7400099b0 tx=0x7fa740009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba38f3bef93dbd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 1 -- 172.21.15.142:0/1775789548 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74000bb50 con 0x7fa75809cd80 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: dump: 2024-09-18T10:13:57.056 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.057 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 1 -- 172.21.15.142:0/1775789548 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74000bcb0 con 0x7fa75809cd80 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 5 monclient: authenticate success, global_id 54222 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 1 -- 172.21.15.142:0/1775789548 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740011670 con 0x7fa75809cd80 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 4 set_mon_vals no callback set 2024-09-18T10:13:57.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa7577fe700 10 monclient: dump: 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.059 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: shutdown 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 msgr2=0x7fa75809d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75809d150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7400099b0 tx=0x7fa740009250 comp rx=0 tx=0).stop 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- 172.21.15.142:0/1775789548 shutdown_connections 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75809d150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/1775789548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa758070ef0 0x7fa758096c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- 172.21.15.142:0/1775789548 >> 172.21.15.142:0/1775789548 conn(0x7fa7580a2390 msgr2=0x7fa7580a2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- 172.21.15.142:0/1775789548 shutdown_connections 2024-09-18T10:13:57.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- 172.21.15.142:0/1775789548 wait complete. 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: build_initial_monmap 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: monmap: 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.061 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 Processor -- start 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- start start 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: init 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758070ef0 0x7fa75808c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-b con 0x7fa758070ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-a con 0x7fa75809cd80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa75809d690 0x7fa758094970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient(hunting): picked mon.noname-c con 0x7fa75809d690 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.043+0000 7fa75dbc9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75808e110 con 0x7fa75809cd80 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75808cbf0 con 0x7fa758070ef0 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75806ef90 con 0x7fa75809d690 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient(hunting): authenticate will time out at 1125.972024s 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75d3c8700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa75809d690 msgr2=0x7fa758094970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75d3c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa75809d690 0x7fa758094970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47120/0 (socket says 172.21.15.142:47120) 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 -- 172.21.15.142:0/548583037 learned_addr learned my addr 172.21.15.142:0/548583037 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): get_auth_request con 0x7fa75809cd80 auth_method 0 2024-09-18T10:13:57.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758070ef0 0x7fa75808c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 10 monclient(hunting): get_auth_request con 0x7fa758070ef0 auth_method 0 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:57.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient(hunting): handle_auth_done global_id 54225 payload 824 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa75809d690 msgr2=0x7fa758094970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa75809d690 0x7fa758094970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758070ef0 msgr2=0x7fa75808c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758070ef0 0x7fa75808c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient: _finish_hunting 0 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 monclient: found mon.noname-a 2024-09-18T10:13:57.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400094f0 con 0x7fa75809cd80 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758070ef0 0x7fa75808c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 msgr2=0x7fa75806e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa75806bad0 tx=0x7fa740016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa757fff700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa75806bad0 tx=0x7fa740016000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fa740006c00 mon_map magic: 0 v1 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740006c00 con 0x7fa75809cd80 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:13:57.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 10 monclient: dump: 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:57.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa740020070 con 0x7fa75809cd80 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 5 monclient: authenticate success, global_id 54225 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient: _renew_subs 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75806fa00 con 0x7fa75809cd80 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa754ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient: _renew_subs 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa7567fc700 4 set_mon_vals no callback set 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.047+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa758095050 con 0x7fa75809cd80 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75dbc9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 0x7fa758141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580648c0 con 0x7fa758141b00 2024-09-18T10:13:57.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74001b6a0 con 0x7fa75809cd80 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 0x7fa758141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 10 monclient: dump: 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.070 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa740022030 con 0x7fa75809cd80 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient: get_auth_request con 0x7fa758141b00 auth_method 0 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa754ff9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa738035a70 0x7fa738037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_done global_id 15409 payload 824 2024-09-18T10:13:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75cbc7700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa758072a80 con 0x7fa758141b00 2024-09-18T10:13:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75d3c8700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa738035a70 0x7fa738037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.051+0000 7fa75d3c8700 10 monclient: get_auth_request con 0x7fa738035a70 auth_method 0 2024-09-18T10:13:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa75cbc7700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 msgr2=0x7fa758141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:57.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 0x7fa758141ed0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa7480039d0 tx=0x7fa748004d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74800b9a0 con 0x7fa758141b00 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 10 monclient: dump: 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.075 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa75d3c8700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa738035a70 0x7fa738037f20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa750007fd0 tx=0x7fa750007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa74800bb00 con 0x7fa758141b00 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 msgr2=0x7fa758141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 0x7fa758141ed0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa7480039d0 tx=0x7fa748004d90 comp rx=0 tx=0).stop 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 msgr2=0x7fa758141ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.055+0000 7fa754ff9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758141b00 0x7fa758141ed0 secure :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa7480039d0 tx=0x7fa748004d90 comp rx=0 tx=0).stop 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75dbc9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 0x7fa75809a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:57.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75dbc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7580a1200 con 0x7fa758142410 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 0x7fa75809a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient: get_auth_request con 0x7fa758142410 auth_method 0 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 10 monclient(hunting): handle_auth_done global_id 15412 payload 824 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.063+0000 7fa75cbc7700 1 -- 172.21.15.142:0/548583037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7580a1a70 con 0x7fa758142410 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75cbc7700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 msgr2=0x7fa75809a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:57.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75cbc7700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 0x7fa75809a780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa748016000 tx=0x7fa748017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74801b610 con 0x7fa758142410 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 10 monclient: dump: 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:57.078 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fa74801b770 con 0x7fa758142410 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 msgr2=0x7fa75809a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 0x7fa75809a780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa748016000 tx=0x7fa748017000 comp rx=0 tx=0).stop 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 msgr2=0x7fa75809a780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa758142410 0x7fa75809a780 secure :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa748016000 tx=0x7fa748017000 comp rx=0 tx=0).stop 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:57.079 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 60, 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:57.080 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:57.081 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:13:57.082 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:13:57.083 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:13:57.084 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:13:57.085 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:13:57.086 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.087 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.088 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:13:57.089 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:13:57.090 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.091 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:57.092 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:13:57.093 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:13:57.094 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:13:57.094 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:13:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa738035a70 msgr2=0x7fa738037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa738035a70 0x7fa738037f20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa750007fd0 tx=0x7fa750007df0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 10 monclient: shutdown 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 msgr2=0x7fa75806e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa75806bad0 tx=0x7fa740016000 comp rx=0 tx=0).stop 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75d3c8700 1 -- 172.21.15.142:0/548583037 reap_dead start 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa754ff9700 1 -- 172.21.15.142:0/548583037 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(124..124 src has 58..124) v4 ==== 6642+0+0 (unknown 0 0 0) 0x7fa74004aa20 con 0x7fa75809cd80 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 shutdown_connections 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 --2- 172.21.15.142:0/548583037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa75809cd80 0x7fa75806e8d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:57.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 >> 172.21.15.142:0/548583037 conn(0x7fa7580a2390 msgr2=0x7fa758135740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.067+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 shutdown_connections 2024-09-18T10:13:57.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:13:57.071+0000 7fa75dbc9700 1 -- 172.21.15.142:0/548583037 wait complete. 2024-09-18T10:13:57.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:13:57.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:13:57.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.810+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff34c01bc60 con 0x7ff35806e570 2024-09-18T10:13:57.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.810+0000 7ff34a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:57.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.810+0000 7ff34a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:57.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.978+0000 7ff3607d1700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:57.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.978+0000 7ff3607d1700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:57.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.978+0000 7ff3607d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:57.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:57.978+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7ff35811a4d0 con 0x7ff35806e570 2024-09-18T10:13:58.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.438+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' created v125) v1 ==== 115+0+0 (secure 0 0 0) 0x7ff34c019e40 con 0x7ff35806e570 2024-09-18T10:13:58.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.438+0000 7ff34a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:58.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.438+0000 7ff34a7fc700 10 monclient: _finish_command 2 = system:0 pool 'fooset' created 2024-09-18T10:13:58.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.538+0000 7ff3607d1700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:58.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.538+0000 7ff3607d1700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:58.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.538+0000 7ff3607d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.538+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "fooset", "pg_num": 8} v 0) v1 -- 0x7ff35811a4d0 con 0x7ff35806e570 2024-09-18T10:13:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff34a7fc700 1 -- 172.21.15.150:0/2424144712 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}]=0 pool 'fooset' already exists v125) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff34c024b80 con 0x7ff35806e570 2024-09-18T10:13:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff34a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "fooset", "pg_num": 8}] 2024-09-18T10:13:58.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff34a7fc700 10 monclient: _finish_command 3 = system:0 pool 'fooset' already exists 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 msgr2=0x7ff33c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 0x7ff33c037f20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff354007fd0 tx=0x7ff354007b00 comp rx=0 tx=0).stop 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 10 monclient: shutdown 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 msgr2=0x7ff35813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c016000 comp rx=0 tx=0).stop 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 shutdown_connections 2024-09-18T10:13:58.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff33c035a70 0x7ff33c037f20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3581353d0 0x7ff35809bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff35806ec00 0x7ff35809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 --2- 172.21.15.150:0/2424144712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff35806e570 0x7ff35813fe30 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 >> 172.21.15.150:0/2424144712 conn(0x7ff3580a40a0 msgr2=0x7ff3580a01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 shutdown_connections 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.542+0000 7ff3607d1700 1 -- 172.21.15.150:0/2424144712 wait complete. 2024-09-18T10:13:58.549 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'fooset' already exists 2024-09-18T10:13:58.557 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create barset 8 2024-09-18T10:13:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.650+0000 7f291adea700 10 monclient: get_monmap_and_config 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.650+0000 7f291adea700 10 monclient: build_initial_monmap 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient: monmap: 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:58.656044+0000 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:58.656044+0000 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:58.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 Processor -- start 2024-09-18T10:13:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 -- start start 2024-09-18T10:13:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient: init 2024-09-18T10:13:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f29140a0120 0x7f291409e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): picked mon.noname-c con 0x7f29140a0120 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 0x7f29140a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): picked mon.noname-a con 0x7f29140a04f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 0x7f2914138d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): picked mon.noname-b con 0x7f29141369a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29140a21d0 con 0x7f29140a04f0 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291409ddc0 con 0x7f29140a0120 2024-09-18T10:13:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291409dae0 con 0x7f29141369a0 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f291adea700 10 monclient(hunting): authenticate will time out at 1120.749558s 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f29140a0120 0x7f291409e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2919387700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 msgr2=0x7f2914138d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f29140a0120 0x7f291409e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53156/0 (socket says 172.21.15.150:53156) 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2919387700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 0x7f2914138d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 -- 172.21.15.150:0/476443823 learned_addr learned my addr 172.21.15.150:0/476443823 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): get_auth_request con 0x7f29140a0120 auth_method 0 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:58.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 1 --2- 172.21.15.150:0/476443823 >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 0x7f29140a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 10 monclient(hunting): get_auth_request con 0x7f29140a04f0 auth_method 0 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient(hunting): handle_auth_done global_id 15415 payload 274 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 -- 172.21.15.150:0/476443823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 msgr2=0x7f2914138d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 --2- 172.21.15.150:0/476443823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 0x7f2914138d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 -- 172.21.15.150:0/476443823 >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 msgr2=0x7f29140a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 --2- 172.21.15.150:0/476443823 >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 0x7f29140a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient: _finish_hunting 0 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 monclient: found mon.noname-c 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2918b86700 1 -- 172.21.15.150:0/476443823 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140a1590 con 0x7f29140a0120 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.654+0000 7f2913fff700 1 --2- 172.21.15.150:0/476443823 >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 0x7f29140a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f2918b86700 1 -- 172.21.15.150:0/476443823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 msgr2=0x7f291409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f2918b86700 1 --2- 172.21.15.150:0/476443823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291409e7a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2914068ec0 tx=0x7f2908009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b1a24e03672859 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 1 -- 172.21.15.150:0/476443823 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290800ba70 con 0x7f29140a0120 2024-09-18T10:13:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: dump: 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: _finish_auth 0 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 5 monclient: authenticate success, global_id 15415 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 1 -- 172.21.15.150:0/476443823 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290800bbd0 con 0x7f29140a0120 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 4 set_mon_vals no callback set 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 1 -- 172.21.15.150:0/476443823 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908011420 con 0x7f29140a0120 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f29137fe700 10 monclient: dump: 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: shutdown 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- 172.21.15.150:0/476443823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 msgr2=0x7f291409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- 172.21.15.150:0/476443823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291409e7a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2914068ec0 tx=0x7f2908009250 comp rx=0 tx=0).stop 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- 172.21.15.150:0/476443823 shutdown_connections 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- 172.21.15.150:0/476443823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29141369a0 0x7f2914138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- 172.21.15.150:0/476443823 >> v2:172.21.15.142:3300/0 conn(0x7f29140a04f0 0x7f29140a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- 172.21.15.150:0/476443823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291409e7a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- 172.21.15.150:0/476443823 >> 172.21.15.150:0/476443823 conn(0x7f29140a5830 msgr2=0x7f29140a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- 172.21.15.150:0/476443823 shutdown_connections 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- 172.21.15.150:0/476443823 wait complete. 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: build_initial_monmap 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: monmap: 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.672 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 Processor -- start 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- start start 2024-09-18T10:13:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: init 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): picked mon.noname-b con 0x7f29140a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 0x7f291412cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): picked mon.noname-c con 0x7f29140a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 0x7f291412d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): picked mon.noname-a con 0x7f29141369a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2914130b20 con 0x7f29141369a0 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291409d720 con 0x7f29140a0120 2024-09-18T10:13:58.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291409bf60 con 0x7f29140a04f0 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f291adea700 10 monclient(hunting): authenticate will time out at 1120.755780s 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f2918b86700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.658+0000 7f2913fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 msgr2=0x7f291412cbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53162/0 (socket says 172.21.15.150:53162) 2024-09-18T10:13:58.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 -- 172.21.15.150:0/3452556228 learned_addr learned my addr 172.21.15.150:0/3452556228 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): get_auth_request con 0x7f29140a0120 auth_method 0 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2913fff700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 0x7f291412cbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:58.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 0x7f291412d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 10 monclient(hunting): get_auth_request con 0x7f29141369a0 auth_method 0 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:58.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient(hunting): handle_auth_done global_id 15418 payload 824 2024-09-18T10:13:58.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 -- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 msgr2=0x7f291412cbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:58.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 0x7f291412cbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 -- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 msgr2=0x7f291412d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:58.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 0x7f291412d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:58.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient: _finish_hunting 0 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 monclient: found mon.noname-b 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29080094f0 con 0x7f29140a0120 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2919387700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 0x7f291412d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:13:58.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 -- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 msgr2=0x7f291412c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2918b86700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f291409df00 tx=0x7f2908017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.662+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908006a90 con 0x7f29140a0120 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:58.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: dump: 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.687 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: _finish_auth 0 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290801a630 con 0x7f29140a0120 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:58.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290801a9f0 con 0x7f29140a0120 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 5 monclient: authenticate success, global_id 15418 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _renew_subs 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291412ff30 con 0x7f29140a0120 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2912ffd700 4 set_mon_vals no callback set 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 10 monclient: dump: 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:58.689 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _renew_subs 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291412f560 con 0x7f29140a0120 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29140a1c90 con 0x7f29140a0120 2024-09-18T10:13:58.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f290802f3f0 con 0x7f29140a0120 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 0x7f2900037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(125..125 src has 58..125) v4 ==== 6987+0+0 (secure 0 0 0) 0x7f290804a740 con 0x7f29140a0120 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2913fff700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 0x7f2900037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.666+0000 7f2913fff700 10 monclient: get_auth_request con 0x7f2900035ac0 auth_method 0 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.670+0000 7f2913fff700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 0x7f2900037f70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2904006f00 tx=0x7f2904008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.674+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2908023900 con 0x7f29140a0120 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.674+0000 7f2910ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:58.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.674+0000 7f2910ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:58.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.842+0000 7f291adea700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:58.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.842+0000 7f291adea700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:58.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.842+0000 7f291adea700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:58.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:58.842+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f2914063ed0 con 0x7f29140a0120 2024-09-18T10:13:59.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.458+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' created v126) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2908015070 con 0x7f29140a0120 2024-09-18T10:13:59.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.458+0000 7f2910ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:59.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.458+0000 7f2910ff9700 10 monclient: _finish_command 2 = system:0 pool 'barset' created 2024-09-18T10:13:59.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f291adea700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:59.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f291adea700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:59.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f291adea700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "barset", "pg_num": 8} v 0) v1 -- 0x7f291412fcc0 con 0x7f29140a0120 2024-09-18T10:13:59.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f2910ff9700 1 -- 172.21.15.150:0/3452556228 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}]=0 pool 'barset' already exists v126) v1 ==== 122+0+0 (secure 0 0 0) 0x7f29080253b0 con 0x7f29140a0120 2024-09-18T10:13:59.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f2910ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "barset", "pg_num": 8}] 2024-09-18T10:13:59.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.566+0000 7f2910ff9700 10 monclient: _finish_command 3 = system:0 pool 'barset' already exists 2024-09-18T10:13:59.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 msgr2=0x7f2900037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:59.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 0x7f2900037f70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2904006f00 tx=0x7f2904008040 comp rx=0 tx=0).stop 2024-09-18T10:13:59.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 10 monclient: shutdown 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 msgr2=0x7f291412c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f291409df00 tx=0x7f2908017000 comp rx=0 tx=0).stop 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 shutdown_connections 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2900035ac0 0x7f2900037f70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29141369a0 0x7f291412d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f29140a04f0 0x7f291412cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 --2- 172.21.15.150:0/3452556228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29140a0120 0x7f291412c670 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 >> 172.21.15.150:0/3452556228 conn(0x7f29140a5830 msgr2=0x7f291409ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:59.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 shutdown_connections 2024-09-18T10:13:59.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.570+0000 7f291adea700 1 -- 172.21.15.150:0/3452556228 wait complete. 2024-09-18T10:13:59.575 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'barset' already exists 2024-09-18T10:13:59.583 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set barset size 3 2024-09-18T10:13:59.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: get_monmap_and_config 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: build_initial_monmap 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: monmap: 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:13:59.686336+0000 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:13:59.686336+0000 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:13:59.687 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:13:59.688 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 Processor -- start 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 -- start start 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: init 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdec0a40d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 0x7fcdec1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-b con 0x7fcdec0a1be0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdec137a10 0x7fcdec139df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-c con 0x7fcdec137a10 addr v2:172.21.15.150:3300/0 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec06ac90 con 0x7fcdec0a40d0 2024-09-18T10:13:59.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec069cd0 con 0x7fcdec137a10 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.682+0000 7fcdf250a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0a44a0 con 0x7fcdec0a1be0 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 10 monclient(hunting): authenticate will time out at 1121.780024s 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeb7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 msgr2=0x7fcdec1374d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdec137a10 0x7fcdec139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 0x7fcdec1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdebfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcdec0a40d0 0x7fcdec06e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57564/0 (socket says 172.21.15.150:57564) 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdec137a10 0x7fcdec139df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35706/0 (socket says 172.21.15.150:35706) 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdebfff700 1 -- 172.21.15.150:0/3715736560 learned_addr learned my addr 172.21.15.150:0/3715736560 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): get_auth_request con 0x7fcdec137a10 auth_method 0 2024-09-18T10:13:59.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient(hunting): handle_auth_done global_id 15421 payload 274 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 -- 172.21.15.150:0/3715736560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 msgr2=0x7fcdec1374d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 --2- 172.21.15.150:0/3715736560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 0x7fcdec1374d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 -- 172.21.15.150:0/3715736560 >> v1:172.21.15.150:6789/0 conn(0x7fcdec0a40d0 legacy=0x7fcdec06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient: _finish_hunting 0 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 monclient: found mon.noname-c 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:59.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.693 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 -- 172.21.15.150:0/3715736560 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0a20b0 con 0x7fcdec137a10 2024-09-18T10:13:59.694 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 -- 172.21.15.150:0/3715736560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec137a10 msgr2=0x7fcdec139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf0aa7700 1 --2- 172.21.15.150:0/3715736560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec137a10 0x7fcdec139df0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e60 tx=0x7fcde4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41dcac65f6030ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 1 -- 172.21.15.150:0/3715736560 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde400e070 con 0x7fcdec137a10 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: dump: 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:59.695 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: _finish_auth 0 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 1 -- 172.21.15.150:0/3715736560 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde4012430 con 0x7fcdec137a10 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 5 monclient: authenticate success, global_id 15421 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 10 monclient: get_monmap_and_config success 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 1 -- 172.21.15.150:0/3715736560 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde40127d0 con 0x7fcdec137a10 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdf250a700 4 set_mon_vals no callback set 2024-09-18T10:13:59.696 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.686+0000 7fcdeaffd700 10 monclient: dump: 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: shutdown 2024-09-18T10:13:59.697 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- 172.21.15.150:0/3715736560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec137a10 msgr2=0x7fcdec139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3715736560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec137a10 0x7fcdec139df0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e60 tx=0x7fcde4009780 comp rx=0 tx=0).stop 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- 172.21.15.150:0/3715736560 shutdown_connections 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3715736560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0a1be0 0x7fcdec1374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3715736560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec137a10 0x7fcdec139df0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- 172.21.15.150:0/3715736560 >> 172.21.15.150:0/3715736560 conn(0x7fcdec0716e0 msgr2=0x7fcdec0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- 172.21.15.150:0/3715736560 shutdown_connections 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- 172.21.15.150:0/3715736560 wait complete. 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: build_initial_monmap 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: monmap: 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:13:59.698 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 Processor -- start 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- start start 2024-09-18T10:13:59.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: init 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: _reopen_session rank -1 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec0a1be0 0x7fcdec12dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-a con 0x7fcdec0a1be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-b con 0x7fcdec0a40d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 0x7fcdec133740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): picked mon.noname-c con 0x7fcdec137a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec137790 con 0x7fcdec0a1be0 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec1362c0 con 0x7fcdec0a40d0 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec06f0a0 con 0x7fcdec137a10 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): _renew_subs 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf250a700 10 monclient(hunting): authenticate will time out at 1121.787696s 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf0aa7700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 msgr2=0x7fcdec133740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:13:59.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35720/0 (socket says 172.21.15.150:35720) 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 1 -- 172.21.15.150:0/3410726138 learned_addr learned my addr 172.21.15.150:0/3410726138 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 10 monclient(hunting): get_auth_request con 0x7fcdec0a40d0 auth_method 0 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.690+0000 7fcdf0aa7700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 0x7fcdec133740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec0a1be0 0x7fcdec12dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): get_auth_request con 0x7fcdec0a1be0 auth_method 0 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:13:59.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient(hunting): handle_auth_done global_id 15424 payload 824 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 -- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 msgr2=0x7fcdec133740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 0x7fcdec133740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 -- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec0a1be0 msgr2=0x7fcdec12dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec0a1be0 0x7fcdec12dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.706 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4009a20 con 0x7fcdec0a40d0 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 -- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 msgr2=0x7fcdec133200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdeb7fe700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00b7e0 tx=0x7fcddc009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc003e20 con 0x7fcdec0a40d0 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: dump: 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.707 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: _finish_auth 0 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 5 monclient: authenticate success, global_id 15424 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc006150 con 0x7fcdec0a40d0 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 10 monclient: _renew_subs 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec135fe0 con 0x7fcdec0a40d0 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc006510 con 0x7fcdec0a40d0 2024-09-18T10:13:59.708 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcde9ffb700 4 set_mon_vals no callback set 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdd3fff700 10 monclient: dump: 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:13:59.709 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 10 monclient: _renew_subs 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.694+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec136f20 con 0x7fcdec0a40d0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdf250a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdf250a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdf250a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec06a9b0 con 0x7fcdec0a40d0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcddc0103c0 con 0x7fcdec0a40d0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdd3fff700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 0x7fcdd8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(126..126 src has 58..126) v4 ==== 7332+0+0 (secure 0 0 0) 0x7fcddc0485c0 con 0x7fcdec0a40d0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdebfff700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 0x7fcdd8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:13:59.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.698+0000 7fcdebfff700 10 monclient: get_auth_request con 0x7fcdd8035a70 auth_method 0 2024-09-18T10:13:59.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.702+0000 7fcdebfff700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 0x7fcdd8037f20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007880 tx=0x7fcde0007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:13:59.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.706+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcddc00ed30 con 0x7fcdec0a40d0 2024-09-18T10:13:59.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.706+0000 7fcdd3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:13:59.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.706+0000 7fcdd3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:13:59.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.874+0000 7fcdf250a700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:13:59.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.874+0000 7fcdf250a700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:13:59.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.874+0000 7fcdf250a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:13:59.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:13:59.874+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fcdec063ed0 con 0x7fcdec0a40d0 2024-09-18T10:14:00.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.466+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v127) v1 ==== 126+0+0 (secure 0 0 0) 0x7fcddc017030 con 0x7fcdec0a40d0 2024-09-18T10:14:00.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.466+0000 7fcdd3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:14:00.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.466+0000 7fcdd3fff700 10 monclient: _finish_command 2 = system:0 set pool 10 size to 3 2024-09-18T10:14:00.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.566+0000 7fcdf250a700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:14:00.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.566+0000 7fcdf250a700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:14:00.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.566+0000 7fcdf250a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:00.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:00.566+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"} v 0) v1 -- 0x7fcdec067b10 con 0x7fcdec0a40d0 2024-09-18T10:14:01.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.858+0000 7fcdd3fff700 1 -- 172.21.15.150:0/3410726138 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}]=0 set pool 10 size to 3 v128) v1 ==== 126+0+0 (secure 0 0 0) 0x7fcddc017020 con 0x7fcdec0a40d0 2024-09-18T10:14:01.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.858+0000 7fcdd3fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "barset", "var": "size", "val": "3"}] 2024-09-18T10:14:01.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.858+0000 7fcdd3fff700 10 monclient: _finish_command 3 = system:0 set pool 10 size to 3 2024-09-18T10:14:01.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 msgr2=0x7fcdd8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 0x7fcdd8037f20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007880 tx=0x7fcde0007f90 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 10 monclient: shutdown 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 msgr2=0x7fcdec133200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00b7e0 tx=0x7fcddc009220 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 shutdown_connections 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcdd8035a70 0x7fcdd8037f20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec137a10 0x7fcdec133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdec0a40d0 0x7fcdec133200 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 --2- 172.21.15.150:0/3410726138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec0a1be0 0x7fcdec12dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:01.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 >> 172.21.15.150:0/3410726138 conn(0x7fcdec0716e0 msgr2=0x7fcdec06cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:01.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 shutdown_connections 2024-09-18T10:14:01.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.866+0000 7fcdf250a700 1 -- 172.21.15.150:0/3410726138 wait complete. 2024-09-18T10:14:01.872 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 10 size to 3 2024-09-18T10:14:01.886 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T10:14:01.886 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:14:01.886 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T10:14:01.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: get_monmap_and_config 2024-09-18T10:14:01.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: build_initial_monmap 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: monmap: 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:01.997184+0000 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:01.997184+0000 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:01.998 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:01.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 1 Processor -- start 2024-09-18T10:14:01.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 1 -- start start 2024-09-18T10:14:01.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: init 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.994+0000 7f06b6171700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f06b00710a0 0x7f06b0096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-a con 0x7f06b00710a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06b0096fb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:02.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06b01345d0 0x7f06b01369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-c con 0x7f06b01345d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b0071470 con 0x7f06b0096fb0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b0099320 con 0x7f06b00710a0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b0098290 con 0x7f06b01345d0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient(hunting): authenticate will time out at 1124.092185s 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06b01345d0 0x7f06b01369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06aeffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f06b0096fb0 0x7f06b0097410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57580/0 (socket says 172.21.15.150:57580) 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06aeffd700 1 -- 172.21.15.150:0/972594921 learned_addr learned my addr 172.21.15.150:0/972594921 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): get_auth_request con 0x7f06b01345d0 auth_method 0 2024-09-18T10:14:02.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 1 --2- 172.21.15.150:0/972594921 >> v2:172.21.15.142:3300/0 conn(0x7f06b00710a0 0x7f06b0096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): get_auth_request con 0x7f06b00710a0 auth_method 0 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient(hunting): handle_auth_done global_id 15427 payload 274 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 -- 172.21.15.150:0/972594921 >> v2:172.21.15.142:3300/0 conn(0x7f06b00710a0 msgr2=0x7f06b0096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 --2- 172.21.15.150:0/972594921 >> v2:172.21.15.142:3300/0 conn(0x7f06b00710a0 0x7f06b0096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 -- 172.21.15.150:0/972594921 >> v1:172.21.15.150:6789/0 conn(0x7f06b0096fb0 legacy=0x7f06b0097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 monclient: found mon.noname-c 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 -- 172.21.15.150:0/972594921 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0098570 con 0x7f06b01345d0 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 -- 172.21.15.150:0/972594921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b01345d0 msgr2=0x7f06b01369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06affff700 1 --2- 172.21.15.150:0/972594921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b01345d0 0x7f06b01369b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f06b009dce0 tx=0x7f06a80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea149dcc36b7fe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 1 -- 172.21.15.150:0/972594921 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8011840 con 0x7f06b01345d0 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: dump: 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.008 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 5 monclient: authenticate success, global_id 15427 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 1 -- 172.21.15.150:0/972594921 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80119a0 con 0x7f06b01345d0 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:02.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06b6171700 4 set_mon_vals no callback set 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 1 -- 172.21.15.150:0/972594921 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8011d40 con 0x7f06b01345d0 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:01.998+0000 7f06ae7fc700 10 monclient: dump: 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.010 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 10 monclient: shutdown 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- 172.21.15.150:0/972594921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b01345d0 msgr2=0x7f06b01369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 --2- 172.21.15.150:0/972594921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b01345d0 0x7f06b01369b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f06b009dce0 tx=0x7f06a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- 172.21.15.150:0/972594921 shutdown_connections 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 --2- 172.21.15.150:0/972594921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b01345d0 0x7f06b01369b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 --2- 172.21.15.150:0/972594921 >> v2:172.21.15.142:3300/0 conn(0x7f06b00710a0 0x7f06b0096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- 172.21.15.150:0/972594921 >> 172.21.15.150:0/972594921 conn(0x7f06b0068590 msgr2=0x7f06b00a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- 172.21.15.150:0/972594921 shutdown_connections 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- 172.21.15.150:0/972594921 wait complete. 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 10 monclient: build_initial_monmap 2024-09-18T10:14:02.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 10 monclient: monmap: 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 Processor -- start 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 1 -- start start 2024-09-18T10:14:02.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.002+0000 7f06b6171700 10 monclient: init 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06b00710a0 0x7f06b006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-c con 0x7f06b00710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b0096fb0 0x7f06b006da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-b con 0x7f06b0096fb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b01345d0 0x7f06b006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): picked mon.noname-a con 0x7f06b01345d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b011a2e0 con 0x7f06b01345d0 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b0097e40 con 0x7f06b0096fb0 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00a1020 con 0x7f06b00710a0 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06b6171700 10 monclient(hunting): authenticate will time out at 1124.100528s 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06af7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06b00710a0 msgr2=0x7f06b006d530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06af7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06b00710a0 0x7f06b006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b0096fb0 0x7f06b006da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b0096fb0 0x7f06b006da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35740/0 (socket says 172.21.15.150:35740) 2024-09-18T10:14:02.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 -- 172.21.15.150:0/2391923038 learned_addr learned my addr 172.21.15.150:0/2391923038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): get_auth_request con 0x7f06b0096fb0 auth_method 0 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b01345d0 0x7f06b006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): get_auth_request con 0x7f06b01345d0 auth_method 0 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient(hunting): handle_auth_done global_id 15430 payload 824 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 -- 172.21.15.150:0/2391923038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06b00710a0 msgr2=0x7f06b006d530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06b00710a0 0x7f06b006d530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 -- 172.21.15.150:0/2391923038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b01345d0 msgr2=0x7f06b006dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06b01345d0 0x7f06b006dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 monclient: found mon.noname-b 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 -- 172.21.15.150:0/2391923038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8009420 con 0x7f06b0096fb0 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 -- 172.21.15.150:0/2391923038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b0096fb0 msgr2=0x7f06b006da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:02.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.006+0000 7f06aeffd700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06b0096fb0 0x7f06b006da70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f069800ea00 tx=0x7f069800c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800c440 con 0x7f06b0096fb0 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: dump: 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.021 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 5 monclient: authenticate success, global_id 15430 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0698005d40 con 0x7f06b0096fb0 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _renew_subs 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 1 -- 172.21.15.150:0/2391923038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b01184d0 con 0x7f06b0096fb0 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698007340 con 0x7f06b0096fb0 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06adffb700 4 set_mon_vals no callback set 2024-09-18T10:14:02.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 10 monclient: dump: 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _renew_subs 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 1 -- 172.21.15.150:0/2391923038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b011a0b0 con 0x7f06b0096fb0 2024-09-18T10:14:02.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06b6171700 1 -- 172.21.15.150:0/2391923038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b0063ed0 con 0x7f06b0096fb0 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0698004f60 con 0x7f06b0096fb0 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0690035a70 0x7f0690037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(128..128 src has 58..128) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f069804b190 con 0x7f06b0096fb0 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06af7fe700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0690035a70 0x7f0690037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.010+0000 7f06af7fe700 10 monclient: get_auth_request con 0x7f0690035a70 auth_method 0 2024-09-18T10:14:02.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.014+0000 7f06af7fe700 1 --2- 172.21.15.150:0/2391923038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0690035a70 0x7f0690037f20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009a20 tx=0x7f06a000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.018+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0698005210 con 0x7f06b0096fb0 2024-09-18T10:14:02.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.018+0000 7f06a77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.018+0000 7f06a77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:02.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.190+0000 7f06b6171700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:02.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.190+0000 7f06b6171700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:02.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.190+0000 7f06b6171700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.190+0000 7f06b6171700 1 -- 172.21.15.150:0/2391923038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f06b0071930 con 0x7f06b0096fb0 2024-09-18T10:14:02.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.194+0000 7f06a77fe700 1 -- 172.21.15.150:0/2391923038 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=-2 no weight-set for pool 'fooset' v128) v1 ==== 160+0+0 (secure 0 0 0) 0x7f0698022350 con 0x7f06b0096fb0 2024-09-18T10:14:02.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.194+0000 7f06a77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:02.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.194+0000 7f06a77fe700 10 monclient: _finish_command 2 = mon:2 no weight-set for pool 'fooset' 2024-09-18T10:14:02.200 INFO:tasks.workunit.client.0.smithi150.stderr:Error ENOENT: no weight-set for pool 'fooset' 2024-09-18T10:14:02.203 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:14:02.204 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create fooset flat 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.294+0000 7f8bc8f1e700 10 monclient: get_monmap_and_config 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.294+0000 7f8bc8f1e700 10 monclient: build_initial_monmap 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.294+0000 7f8bc8f1e700 10 monclient: monmap: 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:02.298460+0000 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:02.298460+0000 2024-09-18T10:14:02.299 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:02.300 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.294+0000 7f8bc8f1e700 1 Processor -- start 2024-09-18T10:14:02.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 -- start start 2024-09-18T10:14:02.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient: init 2024-09-18T10:14:02.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:02.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bc40989b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8bc41316b0 0x7f8bc4133a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-a con 0x7f8bc41316b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 0x7f8bc41363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-b con 0x7f8bc4133fd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc4096e80 con 0x7f8bc40989b0 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc4098220 con 0x7f8bc41316b0 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4098d80 con 0x7f8bc4133fd0 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc8f1e700 10 monclient(hunting): authenticate will time out at 1124.393625s 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc2d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 msgr2=0x7f8bc41363b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 0x7f8bc41363b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8bc41316b0 0x7f8bc4133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc259c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8bc40989b0 0x7f8bc409bab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:43550/0 (socket says 172.21.15.150:43550) 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8bc41316b0 0x7f8bc4133a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:43864/0 (socket says 172.21.15.150:43864) 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bc259c700 1 -- 172.21.15.150:0/2843982554 learned_addr learned my addr 172.21.15.150:0/2843982554 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): get_auth_request con 0x7f8bc41316b0 auth_method 0 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient(hunting): handle_auth_done global_id 54243 payload 274 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 -- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 msgr2=0x7f8bc41363b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 --2- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 0x7f8bc41363b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 -- 172.21.15.150:0/2843982554 >> v1:172.21.15.142:6789/0 conn(0x7f8bc40989b0 legacy=0x7f8bc409bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 monclient: found mon.noname-a 2024-09-18T10:14:02.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:02.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:02.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.298+0000 7f8bbbfff700 1 -- 172.21.15.150:0/2843982554 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc4097970 con 0x7f8bc41316b0 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bbbfff700 1 -- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc41316b0 msgr2=0x7f8bc4133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bbbfff700 1 --2- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc41316b0 0x7f8bc4133a90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00aa90 tx=0x7f8bac00a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46498837f231cec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 1 -- 172.21.15.150:0/2843982554 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac00cd30 con 0x7f8bc41316b0 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: dump: 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.309 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: _finish_auth 0 2024-09-18T10:14:02.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 1 -- 172.21.15.150:0/2843982554 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac00ce90 con 0x7f8bc41316b0 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 5 monclient: authenticate success, global_id 54243 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 1 -- 172.21.15.150:0/2843982554 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac018640 con 0x7f8bc41316b0 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 4 set_mon_vals no callback set 2024-09-18T10:14:02.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc1d9b700 10 monclient: dump: 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.312 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 10 monclient: shutdown 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc41316b0 msgr2=0x7f8bc4133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc41316b0 0x7f8bc4133a90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00aa90 tx=0x7f8bac00a330 comp rx=0 tx=0).stop 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/2843982554 shutdown_connections 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc4133fd0 0x7f8bc41363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/2843982554 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc41316b0 0x7f8bc4133a90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/2843982554 >> 172.21.15.150:0/2843982554 conn(0x7f8bc4068730 msgr2=0x7f8bc40a1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/2843982554 shutdown_connections 2024-09-18T10:14:02.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/2843982554 wait complete. 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.302+0000 7f8bc8f1e700 10 monclient: build_initial_monmap 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient: monmap: 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:02.314 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 Processor -- start 2024-09-18T10:14:02.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 -- start start 2024-09-18T10:14:02.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient: init 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 0x7f8bc413d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-a con 0x7f8bc40989b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 0x7f8bc4130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-c con 0x7f8bc41316b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): picked mon.noname-b con 0x7f8bc4133fd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:02.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4097760 con 0x7f8bc40989b0 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40a0430 con 0x7f8bc4133fd0 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc411a4f0 con 0x7f8bc41316b0 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc8f1e700 10 monclient(hunting): authenticate will time out at 1124.402531s 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bbbfff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 msgr2=0x7f8bc4130a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bbbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 0x7f8bc4130a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:02.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35744/0 (socket says 172.21.15.150:35744) 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 -- 172.21.15.150:0/3646735722 learned_addr learned my addr 172.21.15.150:0/3646735722 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): get_auth_request con 0x7f8bc4133fd0 auth_method 0 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 0x7f8bc413d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): get_auth_request con 0x7f8bc40989b0 auth_method 0 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:02.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient(hunting): handle_auth_done global_id 15433 payload 824 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 -- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 msgr2=0x7f8bc4130a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 0x7f8bc4130a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 -- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 msgr2=0x7f8bc413d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 0x7f8bc413d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 monclient: found mon.noname-b 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc2d9d700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac00a5d0 con 0x7f8bc4133fd0 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.306+0000 7f8bc259c700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 0x7f8bc413d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc2d9d700 1 -- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 msgr2=0x7f8bc4118530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc2d9d700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00bb40 tx=0x7f8bbc0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00ca50 con 0x7f8bc4133fd0 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: dump: 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.325 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 5 monclient: authenticate success, global_id 15433 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc00cbd0 con 0x7f8bc4133fd0 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00cd80 con 0x7f8bc4133fd0 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 10 monclient: dump: 2024-09-18T10:14:02.326 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc159a700 4 set_mon_vals no callback set 2024-09-18T10:14:02.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _renew_subs 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc4096a70 con 0x7f8bc4133fd0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _renew_subs 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc411a000 con 0x7f8bc4133fd0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc40973c0 con 0x7f8bc4133fd0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.310+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8bbc012020 con 0x7f8bc4133fd0 2024-09-18T10:14:02.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.314+0000 7f8bba7fc700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 0x7f8ba4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:02.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.314+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(128..128 src has 58..128) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f8bbc048b00 con 0x7f8bc4133fd0 2024-09-18T10:14:02.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.314+0000 7f8bc259c700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 0x7f8ba4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:02.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.314+0000 7f8bc259c700 10 monclient: get_auth_request con 0x7f8ba4035a70 auth_method 0 2024-09-18T10:14:02.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.314+0000 7f8bc259c700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 0x7f8ba4037f20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009ed0 tx=0x7f8bb4009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:02.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.318+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8bbc01dd50 con 0x7f8bc4133fd0 2024-09-18T10:14:02.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.318+0000 7f8bba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:02.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.318+0000 7f8bba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:02.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.486+0000 7f8bc8f1e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.486+0000 7f8bc8f1e700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.486+0000 7f8bc8f1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.486+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f8bc4063ed0 con 0x7f8bc4133fd0 2024-09-18T10:14:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.878+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 v129) v1 ==== 109+0+0 (secure 0 0 0) 0x7f8bbc01d770 con 0x7f8bc4133fd0 2024-09-18T10:14:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.878+0000 7f8bba7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.878+0000 7f8bba7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:02.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.990+0000 7f8bc8f1e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.990+0000 7f8bc8f1e700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.990+0000 7f8bc8f1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:02.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.990+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"} v 0) v1 -- 0x7f8bc413db50 con 0x7f8bc4133fd0 2024-09-18T10:14:02.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bba7fc700 1 -- 172.21.15.150:0/3646735722 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}]=0 weight-set for pool 'fooset' already created v129) v1 ==== 153+0+0 (secure 0 0 0) 0x7f8bbc0240b0 con 0x7f8bc4133fd0 2024-09-18T10:14:02.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bba7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "fooset", "mode": "flat"}] 2024-09-18T10:14:02.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bba7fc700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'fooset' already created 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 msgr2=0x7f8ba4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 0x7f8ba4037f20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009ed0 tx=0x7f8bb4009060 comp rx=0 tx=0).stop 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 10 monclient: shutdown 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 msgr2=0x7f8bc4118530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00bb40 tx=0x7f8bbc0097d0 comp rx=0 tx=0).stop 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 shutdown_connections 2024-09-18T10:14:02.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8ba4035a70 0x7f8ba4037f20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bc4133fd0 0x7f8bc4118530 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8bc41316b0 0x7f8bc4130a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 --2- 172.21.15.150:0/3646735722 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8bc40989b0 0x7f8bc413d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 >> 172.21.15.150:0/3646735722 conn(0x7f8bc4068730 msgr2=0x7f8bc40a14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 shutdown_connections 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:02.994+0000 7f8bc8f1e700 1 -- 172.21.15.150:0/3646735722 wait complete. 2024-09-18T10:14:03.000 INFO:tasks.workunit.client.0.smithi150.stderr:weight-set for pool 'fooset' already created 2024-09-18T10:14:03.010 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create barset positional 2024-09-18T10:14:03.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: get_monmap_and_config 2024-09-18T10:14:03.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: build_initial_monmap 2024-09-18T10:14:03.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: monmap: 2024-09-18T10:14:03.117 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:03.117 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:03.117071+0000 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:03.117071+0000 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:03.118 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 Processor -- start 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 -- start start 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: init 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f781809a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): picked mon.noname-b con 0x7f781809bca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f781809e140 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:03.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7818137650 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781809edd0 con 0x7f781809e140 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7818064bf0 con 0x7f7818137650 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7818099b90 con 0x7f781809bca0 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f782033f700 10 monclient(hunting): authenticate will time out at 1125.210607s 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f781e0db700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 msgr2=0x7f781809a410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:03.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f781e8dc700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7818137650 0x7f7818139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57584/0 (socket says 172.21.15.150:57584) 2024-09-18T10:14:03.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f781e8dc700 1 -- 172.21.15.150:0/2918910039 learned_addr learned my addr 172.21.15.150:0/2918910039 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:03.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.114+0000 7f781e0db700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f781809a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:03.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 1 -- 172.21.15.150:0/2918910039 >> v1:172.21.15.150:6789/0 conn(0x7f7818137650 legacy=0x7f7818139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:03.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 -- 172.21.15.150:0/2918910039 >> v1:172.21.15.142:6789/0 conn(0x7f781809e140 legacy=0x7f781809c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:03.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722253895 0 0) 0x7f7818064bf0 con 0x7f7818137650 2024-09-18T10:14:03.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): my global_id is 15436 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7800003570 con 0x7f7818137650 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670100053 0 0) 0x7f781809edd0 con 0x7f781809e140 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): my global_id is 54249 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818064bf0 con 0x7f781809e140 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263925901 0 0) 0x7f7800003570 con 0x7f7818137650 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 msgr2=0x7f781809a410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f781809a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 >> v1:172.21.15.142:6789/0 conn(0x7f781809e140 legacy=0x7f781809c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: _finish_hunting 0 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781809e810 con 0x7f7818137650 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: _finish_auth 0 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f782033f700 5 monclient: authenticate success, global_id 15436 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f7810001860 con 0x7f7818137650 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f782033f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:03.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: dump: 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:14:03.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 >> v1:172.21.15.150:6789/0 conn(0x7f7818137650 legacy=0x7f7818139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f7800003c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): picked mon.c con 0x7f781809bca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 0x7f7800004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): picked mon.b con 0x7f7800004210 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7800004f50 0x7f780000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): picked mon.a con 0x7f7800004f50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7818099b90 con 0x7f7800004f50 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800005640 con 0x7f7800004210 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780000bfa0 con 0x7f781809bca0 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d0d9700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f782033f700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e0db700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 msgr2=0x7f7800003c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e0db700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f7800003c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 0x7f7800004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:03.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): get_auth_request con 0x7f7800004210 auth_method 0 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7800004f50 0x7f780000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): get_auth_request con 0x7f7800004f50 auth_method 0 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:03.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient(hunting): handle_auth_done global_id 15436 payload 293 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 msgr2=0x7f7800003c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f781809bca0 0x7f7800003c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7800004f50 msgr2=0x7f780000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7800004f50 0x7f780000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient: _finish_hunting 0 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 monclient: found mon.b 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 1 -- 172.21.15.150:0/2918910039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7800003570 con 0x7f7800004210 2024-09-18T10:14:03.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 10 monclient: _finish_auth 0 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.118+0000 7f781d8da700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d8da700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 msgr2=0x7f7800004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d8da700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 0x7f7800004810 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7808001d10 tx=0x7f7808007cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61996fff3aab8911 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780800c8b0 con 0x7f7800004210 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: dump: 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:03.143 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780800ca10 con 0x7f7800004210 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:03.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 4 set_mon_vals no callback set 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 1 -- 172.21.15.150:0/2918910039 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780800cdd0 con 0x7f7800004210 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781d0d9700 10 monclient: dump: 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.145 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 10 monclient: shutdown 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:03.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 -- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 msgr2=0x7f7800004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 --2- 172.21.15.150:0/2918910039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7800004210 0x7f7800004810 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7808001d10 tx=0x7f7808007cf0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f781e8dc700 1 -- 172.21.15.150:0/2918910039 reap_dead start 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 -- 172.21.15.150:0/2918910039 shutdown_connections 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 -- 172.21.15.150:0/2918910039 >> 172.21.15.150:0/2918910039 conn(0x7f78180a24a0 msgr2=0x7f78180a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 -- 172.21.15.150:0/2918910039 shutdown_connections 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 1 -- 172.21.15.150:0/2918910039 wait complete. 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 10 monclient: build_initial_monmap 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.122+0000 7f782033f700 10 monclient: monmap: 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:03.147 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 Processor -- start 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 -- start start 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient: init 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 0x7f7818136660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): picked mon.noname-c con 0x7f7800004210 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): picked mon.noname-b con 0x7f7818136ba0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f781808cd50 0x7f781808d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): picked mon.noname-a con 0x7f781808cd50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78180907f0 con 0x7f781808cd50 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78181374b0 con 0x7f7818136ba0 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7818090510 con 0x7f7800004210 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:03.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f782033f700 10 monclient(hunting): authenticate will time out at 1125.221481s 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e0db700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 msgr2=0x7f7818136660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e0db700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 0x7f7818136660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35752/0 (socket says 172.21.15.150:35752) 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 -- 172.21.15.150:0/2876490507 learned_addr learned my addr 172.21.15.150:0/2876490507 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): get_auth_request con 0x7f7818136ba0 auth_method 0 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:03.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f781808cd50 0x7f781808d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): get_auth_request con 0x7f781808cd50 auth_method 0 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781e8dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient(hunting): handle_auth_done global_id 15439 payload 824 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 -- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 msgr2=0x7f7818136660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 0x7f7818136660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 -- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f781808cd50 msgr2=0x7f781808d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f781808cd50 0x7f781808d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:03.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient: _finish_hunting 0 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 monclient: found mon.noname-b 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.126+0000 7f781d8da700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818090230 con 0x7f7818136ba0 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f781d8da700 1 -- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 msgr2=0x7f7818136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f781d8da700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f78000057d0 tx=0x7f7808015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78080016b0 con 0x7f7818136ba0 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: dump: 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.152 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 5 monclient: authenticate success, global_id 15439 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _renew_subs 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7808005c30 con 0x7f7818136ba0 2024-09-18T10:14:03.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781808e780 con 0x7f7818136ba0 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780801e400 con 0x7f7818136ba0 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f781c8d8700 4 set_mon_vals no callback set 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 10 monclient: dump: 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:03.154 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _renew_subs 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78180951a0 con 0x7f7818136ba0 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7808019030 con 0x7f7818136ba0 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 0x7f7804037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:03.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(129..129 src has 58..129) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f7808049f30 con 0x7f7818136ba0 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f781e0db700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 0x7f7804037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:03.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.130+0000 7f781e0db700 10 monclient: get_auth_request con 0x7f7804035aa0 auth_method 0 2024-09-18T10:14:03.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.134+0000 7f781e0db700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 0x7f7804037f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f780c003d00 tx=0x7f780c009350 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:03.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.138+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:03.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.138+0000 7f78167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:03.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.138+0000 7f78167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.310+0000 7f782033f700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.310+0000 7f782033f700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.310+0000 7f782033f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.310+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.882+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 v130) v1 ==== 115+0+0 (secure 0 0 0) 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.882+0000 7f78167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.882+0000 7f78167fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:03.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.994+0000 7f782033f700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:03.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.994+0000 7f782033f700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:03.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.994+0000 7f782033f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:03.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.994+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"} v 0) v1 -- 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:04.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.994+0000 7f78167fc700 1 -- 172.21.15.150:0/2876490507 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}]=0 weight-set for pool 'barset' already created v130) v1 ==== 159+0+0 (secure 0 0 0) 0x7f7818090930 con 0x7f7818136ba0 2024-09-18T10:14:04.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f78167fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "barset", "mode": "positional"}] 2024-09-18T10:14:04.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f78167fc700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'barset' already created 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 msgr2=0x7f7804037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 0x7f7804037f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f780c003d00 tx=0x7f780c009350 comp rx=0 tx=0).stop 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 10 monclient: shutdown 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 msgr2=0x7f7818136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f78000057d0 tx=0x7f7808015000 comp rx=0 tx=0).stop 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 shutdown_connections 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035aa0 0x7f7804037f50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f781808cd50 0x7f781808d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7818136ba0 0x7f7818136f70 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 --2- 172.21.15.150:0/2876490507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7800004210 0x7f7818136660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 >> 172.21.15.150:0/2876490507 conn(0x7f78180a24a0 msgr2=0x7f78180a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 shutdown_connections 2024-09-18T10:14:04.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:03.998+0000 7f782033f700 1 -- 172.21.15.150:0/2876490507 wait complete. 2024-09-18T10:14:04.004 INFO:tasks.workunit.client.0.smithi150.stderr:weight-set for pool 'barset' already created 2024-09-18T10:14:04.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:04.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep fooset 2024-09-18T10:14:04.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: get_monmap_and_config 2024-09-18T10:14:04.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: monmap: 2024-09-18T10:14:04.293 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.293 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:04.112729+0000 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:04.112729+0000 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:04.294 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 Processor -- start 2024-09-18T10:14:04.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 -- start start 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: init 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1a5c09d970 0x7f1a5c0955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c09d970 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a5c098ac0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a5c0995a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c09b0a0 con 0x7f1a5c0995a0 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c09ab50 con 0x7f1a5c098ac0 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c099a00 con 0x7f1a5c09d970 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a62621700 10 monclient(hunting): authenticate will time out at 1126.206270s 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5b7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1a5c098ac0 0x7f1a5c098e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57598/0 (socket says 172.21.15.150:57598) 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5b7fe700 1 -- 172.21.15.150:0/338562344 learned_addr learned my addr 172.21.15.150:0/338562344 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/338562344 >> v2:172.21.15.142:3300/0 conn(0x7f1a5c09d970 0x7f1a5c0955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): get_auth_request con 0x7f1a5c09d970 auth_method 0 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient(hunting): handle_auth_done global_id 54255 payload 274 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 1 -- 172.21.15.150:0/338562344 >> v1:172.21.15.150:6789/0 conn(0x7f1a5c098ac0 legacy=0x7f1a5c098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 1 -- 172.21.15.150:0/338562344 >> v1:172.21.15.142:6789/0 conn(0x7f1a5c0995a0 legacy=0x7f1a5c136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 1 monclient: found mon.noname-a 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.110+0000 7f1a5bfff700 1 -- 172.21.15.150:0/338562344 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c09ae30 con 0x7f1a5c09d970 2024-09-18T10:14:04.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5bfff700 1 -- 172.21.15.150:0/338562344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c09d970 msgr2=0x7f1a5c0955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/338562344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c09d970 0x7f1a5c0955b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a500098e0 tx=0x7f1a500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c41a896447d67a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 1 -- 172.21.15.150:0/338562344 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5000ba70 con 0x7f1a5c09d970 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: dump: 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.306 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 5 monclient: authenticate success, global_id 54255 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 1 -- 172.21.15.150:0/338562344 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5000bbd0 con 0x7f1a5c09d970 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 4 set_mon_vals no callback set 2024-09-18T10:14:04.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 1 -- 172.21.15.150:0/338562344 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50011420 con 0x7f1a5c09d970 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a5affd700 10 monclient: dump: 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.308 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 10 monclient: shutdown 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 -- 172.21.15.150:0/338562344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c09d970 msgr2=0x7f1a5c0955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 --2- 172.21.15.150:0/338562344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c09d970 0x7f1a5c0955b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a500098e0 tx=0x7f1a500091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 -- 172.21.15.150:0/338562344 shutdown_connections 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 --2- 172.21.15.150:0/338562344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c09d970 0x7f1a5c0955b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 -- 172.21.15.150:0/338562344 >> 172.21.15.150:0/338562344 conn(0x7f1a5c0a15e0 msgr2=0x7f1a5c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 -- 172.21.15.150:0/338562344 shutdown_connections 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.114+0000 7f1a62621700 1 -- 172.21.15.150:0/338562344 wait complete. 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient: monmap: 2024-09-18T10:14:04.309 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 Processor -- start 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 -- start start 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient: init 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 0x7f1a5c09c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-a con 0x7f1a5c098ac0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 0x7f1a5c094440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-c con 0x7f1a5c0995a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): picked mon.noname-b con 0x7f1a5c09d970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c09d580 con 0x7f1a5c098ac0 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c098450 con 0x7f1a5c09d970 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c097c30 con 0x7f1a5c0995a0 2024-09-18T10:14:04.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a62621700 10 monclient(hunting): authenticate will time out at 1126.214234s 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35766/0 (socket says 172.21.15.150:35766) 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 -- 172.21.15.150:0/2573450964 learned_addr learned my addr 172.21.15.150:0/2573450964 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): get_auth_request con 0x7f1a5c09d970 auth_method 0 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5b7fe700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 msgr2=0x7f1a5c094440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5b7fe700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 0x7f1a5c094440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 0x7f1a5c09c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 10 monclient(hunting): get_auth_request con 0x7f1a5c098ac0 auth_method 0 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient(hunting): handle_auth_done global_id 15442 payload 824 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 msgr2=0x7f1a5c094440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 0x7f1a5c094440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 msgr2=0x7f1a5c09c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 0x7f1a5c09c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 monclient: found mon.noname-b 2024-09-18T10:14:04.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a60bbe700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009420 con 0x7f1a5c09d970 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.118+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 0x7f1a5c09c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a60bbe700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 msgr2=0x7f1a5c08cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a60bbe700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c960 tx=0x7f1a540026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400cea0 con 0x7f1a5c09d970 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: dump: 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 5 monclient: authenticate success, global_id 15442 2024-09-18T10:14:04.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5400a9b0 con 0x7f1a5c09d970 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _renew_subs 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c140830 con 0x7f1a5c09d970 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400ad50 con 0x7f1a5c09d970 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 10 monclient: dump: 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _renew_subs 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c13fe80 con 0x7f1a5c09d970 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a5a7fc700 4 set_mon_vals no callback set 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1a54012630 con 0x7f1a5c09d970 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c08fa40 con 0x7f1a5c09d970 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 0x7f1a48037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 58..130) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f1a54049580 con 0x7f1a5c09d970 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.122+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 0x7f1a48037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.126+0000 7f1a5bfff700 10 monclient: get_auth_request con 0x7f1a48035a70 auth_method 0 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.126+0000 7f1a5bfff700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 0x7f1a48037f20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a500093f0 tx=0x7f1a50016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.134+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1a54022d70 con 0x7f1a5c09d970 2024-09-18T10:14:04.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.134+0000 7f1a43fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.134+0000 7f1a43fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:04.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a62621700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a62621700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a62621700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f1a5c063ed0 con 0x7f1a5c09d970 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7f1a54016070 con 0x7f1a5c09d970 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a43fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.298+0000 7f1a43fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:04.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f1a5c0008d0 con 0x7f1a5c09d970 2024-09-18T10:14:04.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a43fff700 1 -- 172.21.15.150:0/2573450964 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7f1a54022770 con 0x7f1a5c09d970 2024-09-18T10:14:04.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a43fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a43fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 msgr2=0x7f1a48037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 0x7f1a48037f20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a500093f0 tx=0x7f1a50016000 comp rx=0 tx=0).stop 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 10 monclient: shutdown 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 msgr2=0x7f1a5c08cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c960 tx=0x7f1a540026e0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 shutdown_connections 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1a48035a70 0x7f1a48037f20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a5c09d970 0x7f1a5c08cff0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a5c0995a0 0x7f1a5c094440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 --2- 172.21.15.150:0/2573450964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a5c098ac0 0x7f1a5c09c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 >> 172.21.15.150:0/2573450964 conn(0x7f1a5c0a15e0 msgr2=0x7f1a5c0a0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 shutdown_connections 2024-09-18T10:14:04.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.402+0000 7f1a62621700 1 -- 172.21.15.150:0/2573450964 wait complete. 2024-09-18T10:14:04.417 INFO:tasks.workunit.client.0.smithi150.stdout:fooset 2024-09-18T10:14:04.418 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep barset 2024-09-18T10:14:04.418 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:04.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 10 monclient: get_monmap_and_config 2024-09-18T10:14:04.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 10 monclient: monmap: 2024-09-18T10:14:04.519 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.519 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:04.518856+0000 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:04.518856+0000 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:04.520 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 1 Processor -- start 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 1 -- start start 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.514+0000 7fca43ff5700 10 monclient: init 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T10:14:04.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca3c09d970 0x7fca3c0955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-c con 0x7fca3c09d970 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca3c098ac0 0x7fca3c098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-a con 0x7fca3c098ac0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca3c0993d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c09b0a0 con 0x7fca3c0993d0 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c09ab50 con 0x7fca3c098ac0 2024-09-18T10:14:04.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c09ae30 con 0x7fca3c09d970 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient(hunting): authenticate will time out at 1126.612184s 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca3c09d970 0x7fca3c0955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca3c09d970 0x7fca3c0955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35768/0 (socket says 172.21.15.150:35768) 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 -- 172.21.15.150:0/3887803200 learned_addr learned my addr 172.21.15.150:0/3887803200 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): get_auth_request con 0x7fca3c09d970 auth_method 0 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 1 --2- 172.21.15.150:0/3887803200 >> v2:172.21.15.142:3300/0 conn(0x7fca3c098ac0 0x7fca3c098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): get_auth_request con 0x7fca3c098ac0 auth_method 0 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient(hunting): handle_auth_done global_id 15445 payload 274 2024-09-18T10:14:04.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 -- 172.21.15.150:0/3887803200 >> v2:172.21.15.142:3300/0 conn(0x7fca3c098ac0 msgr2=0x7fca3c098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 --2- 172.21.15.150:0/3887803200 >> v2:172.21.15.142:3300/0 conn(0x7fca3c098ac0 0x7fca3c098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 -- 172.21.15.150:0/3887803200 >> v1:172.21.15.142:6789/0 conn(0x7fca3c0993d0 legacy=0x7fca3c135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 monclient: found mon.noname-c 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 -- 172.21.15.150:0/3887803200 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c09dd40 con 0x7fca3c09d970 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 -- 172.21.15.150:0/3887803200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c09d970 msgr2=0x7fca3c0955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca41d91700 1 --2- 172.21.15.150:0/3887803200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c09d970 0x7fca3c0955b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fca3c067420 tx=0x7fca30009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b6d597ca2689bfc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 1 -- 172.21.15.150:0/3887803200 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30009e70 con 0x7fca3c09d970 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:04.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: dump: 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 5 monclient: authenticate success, global_id 15445 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 1 -- 172.21.15.150:0/3887803200 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3000ba70 con 0x7fca3c09d970 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca43ff5700 4 set_mon_vals no callback set 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 1 -- 172.21.15.150:0/3887803200 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3000be30 con 0x7fca3c09d970 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.518+0000 7fca40d8f700 10 monclient: dump: 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.532 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: shutdown 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- 172.21.15.150:0/3887803200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c09d970 msgr2=0x7fca3c0955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- 172.21.15.150:0/3887803200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c09d970 0x7fca3c0955b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fca3c067420 tx=0x7fca30009250 comp rx=0 tx=0).stop 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- 172.21.15.150:0/3887803200 shutdown_connections 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- 172.21.15.150:0/3887803200 >> v2:172.21.15.142:3300/0 conn(0x7fca3c098ac0 0x7fca3c098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- 172.21.15.150:0/3887803200 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c09d970 0x7fca3c0955b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- 172.21.15.150:0/3887803200 >> 172.21.15.150:0/3887803200 conn(0x7fca3c0a15e0 msgr2=0x7fca3c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- 172.21.15.150:0/3887803200 shutdown_connections 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- 172.21.15.150:0/3887803200 wait complete. 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: monmap: 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:04.534 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 Processor -- start 2024-09-18T10:14:04.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- start start 2024-09-18T10:14:04.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: init 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-b con 0x7fca3c098ac0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca3c0993d0 0x7fca3c13dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-a con 0x7fca3c0993d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 0x7fca3c089e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): picked mon.noname-c con 0x7fca3c13e2a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:04.541 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c13e6d0 con 0x7fca3c0993d0 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c08be30 con 0x7fca3c098ac0 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0a0a60 con 0x7fca3c13e2a0 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca43ff5700 10 monclient(hunting): authenticate will time out at 1126.618585s 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca42592700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 msgr2=0x7fca3c089e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca42592700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 0x7fca3c089e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35784/0 (socket says 172.21.15.150:35784) 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 -- 172.21.15.150:0/1024770726 learned_addr learned my addr 172.21.15.150:0/1024770726 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:04.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): get_auth_request con 0x7fca3c098ac0 auth_method 0 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca3c0993d0 0x7fca3c13dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): get_auth_request con 0x7fca3c0993d0 auth_method 0 2024-09-18T10:14:04.543 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41590700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient(hunting): handle_auth_done global_id 15448 payload 824 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 -- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 msgr2=0x7fca3c089e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 0x7fca3c089e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 -- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca3c0993d0 msgr2=0x7fca3c13dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca3c0993d0 0x7fca3c13dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 monclient: found mon.noname-b 2024-09-18T10:14:04.544 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.522+0000 7fca41d91700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300094f0 con 0x7fca3c098ac0 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca41d91700 1 -- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 msgr2=0x7fca3c13d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca41d91700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fca3c069a40 tx=0x7fca30016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30011920 con 0x7fca3c098ac0 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: dump: 2024-09-18T10:14:04.545 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca30020070 con 0x7fca3c098ac0 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 5 monclient: authenticate success, global_id 15448 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _renew_subs 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c098450 con 0x7fca3c098ac0 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30011cf0 con 0x7fca3c098ac0 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca39ffb700 10 monclient: dump: 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _renew_subs 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca3bfff700 4 set_mon_vals no callback set 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c08e710 con 0x7fca3c098ac0 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.526+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c063ed0 con 0x7fca3c098ac0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fca30022030 con 0x7fca3c098ac0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca39ffb700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 0x7fca28037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 58..130) v4 ==== 8276+0+0 (secure 0 0 0) 0x7fca3004adb0 con 0x7fca3c098ac0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca41590700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 0x7fca28037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca41590700 10 monclient: get_auth_request con 0x7fca28035aa0 auth_method 0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.530+0000 7fca41590700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 0x7fca28037f50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009db0 tx=0x7fca2c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.538+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fca3002cdb0 con 0x7fca3c098ac0 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.538+0000 7fca39ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.538+0000 7fca39ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:04.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca43ff5700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca43ff5700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca43ff5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fca3c063ed0 con 0x7fca3c098ac0 2024-09-18T10:14:04.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7fca3002c7f0 con 0x7fca3c098ac0 2024-09-18T10:14:04.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca39ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.720 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.714+0000 7fca39ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:04.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca43ff5700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca43ff5700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca43ff5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fca3c063ed0 con 0x7fca3c098ac0 2024-09-18T10:14:04.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca39ffb700 1 -- 172.21.15.150:0/1024770726 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v130) v1 ==== 71+0+14 (secure 0 0 0) 0x7fca3002cdb0 con 0x7fca3c098ac0 2024-09-18T10:14:04.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca39ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:04.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.814+0000 7fca39ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:04.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 msgr2=0x7fca28037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 0x7fca28037f50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009db0 tx=0x7fca2c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 10 monclient: shutdown 2024-09-18T10:14:04.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 msgr2=0x7fca3c13d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fca3c069a40 tx=0x7fca30016000 comp rx=0 tx=0).stop 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 shutdown_connections 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca28035aa0 0x7fca28037f50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca3c13e2a0 0x7fca3c089e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca3c0993d0 0x7fca3c13dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 --2- 172.21.15.150:0/1024770726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3c098ac0 0x7fca3c13d820 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 >> 172.21.15.150:0/1024770726 conn(0x7fca3c0a15e0 msgr2=0x7fca3c0963c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 shutdown_connections 2024-09-18T10:14:04.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.818+0000 7fca43ff5700 1 -- 172.21.15.150:0/1024770726 wait complete. 2024-09-18T10:14:04.830 INFO:tasks.workunit.client.0.smithi150.stdout:barset 2024-09-18T10:14:04.830 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set dump 2024-09-18T10:14:04.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: get_monmap_and_config 2024-09-18T10:14:04.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: monmap: 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:04.930057+0000 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:04.930057+0000 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:04.931 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:04.932 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:04.932 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:04.932 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 Processor -- start 2024-09-18T10:14:04.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 -- start start 2024-09-18T10:14:04.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: init 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f983c09cf90 0x7f983c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-c con 0x7f983c09cf90 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f983c09e480 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 0x7f983c139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-b con 0x7f983c0990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c09e060 con 0x7f983c09e480 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f983c0a66c0 con 0x7f983c09cf90 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c09d750 con 0x7f983c0990a0 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f9843c25700 10 monclient(hunting): authenticate will time out at 1127.023532s 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98411c0700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f983c09e480 0x7f983c0a07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57612/0 (socket says 172.21.15.150:57612) 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98411c0700 1 -- 172.21.15.150:0/3971046335 learned_addr learned my addr 172.21.15.150:0/3971046335 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98421c2700 1 -- 172.21.15.150:0/3971046335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 msgr2=0x7f983c139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98421c2700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 0x7f983c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98409bf700 1 -- 172.21.15.150:0/3971046335 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98419c1700 1 --2- 172.21.15.150:0/3971046335 >> v2:172.21.15.150:3300/0 conn(0x7f983c09cf90 0x7f983c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98419c1700 10 monclient(hunting): get_auth_request con 0x7f983c09cf90 auth_method 0 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98419c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98419c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.926+0000 7f98419c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient(hunting): handle_auth_done global_id 15451 payload 274 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 -- 172.21.15.150:0/3971046335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 msgr2=0x7f983c139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 0x7f983c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 -- 172.21.15.150:0/3971046335 >> v1:172.21.15.150:6789/0 conn(0x7f983c09e480 legacy=0x7f983c0a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 monclient: found mon.noname-c 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 -- 172.21.15.150:0/3971046335 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c09d360 con 0x7f983c09cf90 2024-09-18T10:14:04.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 -- 172.21.15.150:0/3971046335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 msgr2=0x7f983c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98419c1700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c0a0290 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f98380098e0 tx=0x7f98380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db144b033643eab9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 1 -- 172.21.15.150:0/3971046335 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983800ba70 con 0x7f983c09cf90 2024-09-18T10:14:04.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: dump: 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 1 -- 172.21.15.150:0/3971046335 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983800bbf0 con 0x7f983c09cf90 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 5 monclient: authenticate success, global_id 15451 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 1 -- 172.21.15.150:0/3971046335 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838011420 con 0x7f983c09cf90 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 4 set_mon_vals no callback set 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f98409bf700 10 monclient: dump: 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.943 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 10 monclient: shutdown 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 -- 172.21.15.150:0/3971046335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 msgr2=0x7f983c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c0a0290 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f98380098e0 tx=0x7f98380091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 -- 172.21.15.150:0/3971046335 shutdown_connections 2024-09-18T10:14:04.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c0990a0 0x7f983c139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 --2- 172.21.15.150:0/3971046335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c0a0290 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.930+0000 7f9843c25700 1 -- 172.21.15.150:0/3971046335 >> 172.21.15.150:0/3971046335 conn(0x7f983c0a58a0 msgr2=0x7f983c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- 172.21.15.150:0/3971046335 shutdown_connections 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- 172.21.15.150:0/3971046335 wait complete. 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient: build_initial_monmap 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient: monmap: 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.945 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 Processor -- start 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- start start 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient: init 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c0990a0 0x7f983c089690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-a con 0x7f983c0990a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-b con 0x7f983c09cf90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 0x7f983c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): picked mon.noname-c con 0x7f983c09e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0a4220 con 0x7f983c0990a0 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c088f90 con 0x7f983c09cf90 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c088cb0 con 0x7f983c09e480 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f9843c25700 10 monclient(hunting): authenticate will time out at 1127.031415s 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35802/0 (socket says 172.21.15.150:35802) 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98421c2700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 msgr2=0x7f983c1431d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 -- 172.21.15.150:0/301641187 learned_addr learned my addr 172.21.15.150:0/301641187 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:04.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): get_auth_request con 0x7f983c09cf90 auth_method 0 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98421c2700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 0x7f983c1431d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98419c1700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c0990a0 0x7f983c089690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98419c1700 10 monclient(hunting): get_auth_request con 0x7f983c0990a0 auth_method 0 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98419c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98419c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98419c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 10 monclient(hunting): handle_auth_done global_id 15454 payload 824 2024-09-18T10:14:04.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 -- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 msgr2=0x7f983c1431d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 0x7f983c1431d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 -- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c0990a0 msgr2=0x7f983c089690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.934+0000 7f98411c0700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c0990a0 0x7f983c089690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 10 monclient: _finish_hunting 0 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 1 monclient: found mon.noname-b 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838009420 con 0x7f983c09cf90 2024-09-18T10:14:04.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 1 -- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 msgr2=0x7f983c142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:04.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f98411c0700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c610 tx=0x7f982c004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c005d40 con 0x7f983c09cf90 2024-09-18T10:14:04.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: dump: 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 5 monclient: authenticate success, global_id 15454 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c0070d0 con 0x7f983c09cf90 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 10 monclient: _renew_subs 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c0889d0 con 0x7f983c09cf90 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c007490 con 0x7f983c09cf90 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9833fff700 4 set_mon_vals no callback set 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9831ffb700 10 monclient: dump: 2024-09-18T10:14:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 10 monclient: _renew_subs 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.938+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c143710 con 0x7f983c09cf90 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f982c01b3b0 con 0x7f983c09cf90 2024-09-18T10:14:04.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9843c25700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9843c25700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9843c25700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c0a0ee0 con 0x7f983c09cf90 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9831ffb700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 0x7f9820037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 58..130) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f982c048840 con 0x7f983c09cf90 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f98419c1700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 0x7f9820037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f98419c1700 10 monclient: get_auth_request con 0x7f9820035a70 auth_method 0 2024-09-18T10:14:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.942+0000 7f98419c1700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 0x7f9820037f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f98380098e0 tx=0x7f9838016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.950+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f982c00f860 con 0x7f983c09cf90 2024-09-18T10:14:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.950+0000 7f9831ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:04.950+0000 7f9831ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:05.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9843c25700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9843c25700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9843c25700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f983c0892c0 con 0x7f983c09cf90 2024-09-18T10:14:05.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v130) v1 ==== 73+0+6748 (secure 0 0 0) 0x7f982c017070 con 0x7f983c09cf90 2024-09-18T10:14:05.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9831ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.114+0000 7f9831ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:05.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9843c25700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9843c25700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9843c25700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set dump"} v 0) v1 -- 0x7f983c063ed0 con 0x7f983c09cf90 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9831ffb700 1 -- 172.21.15.150:0/301641187 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set dump"}]=0 v130) v1 ==== 73+0+6748 (secure 0 0 0) 0x7f982c00f260 con 0x7f983c09cf90 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9831ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set dump"}] 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.214+0000 7f9831ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stdout:{ 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stdout: "9": [ 2024-09-18T10:14:05.220 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -1, 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.221 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -2, 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.222 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -3, 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.223 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -4, 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -5, 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.224 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -6, 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.225 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -7, 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.226 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.227 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.227 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.227 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.227 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -8, 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -9, 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.228 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -10, 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.229 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.230 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.230 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.230 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -11, 2024-09-18T10:14:05.230 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.230 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.231 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.231 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.231 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.231 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -12, 2024-09-18T10:14:05.231 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.232 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.232 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.232 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.232 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.232 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -13, 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -14, 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.233 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -15, 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: "10": [ 2024-09-18T10:14:05.234 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -1, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.235 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: 0.263671875, 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.236 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -2, 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.237 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -3, 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.238 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -4, 2024-09-18T10:14:05.239 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -5, 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.240 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.241 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -6, 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.242 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.243 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -7, 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.244 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -8, 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.245 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -9, 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.246 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: ], 2024-09-18T10:14:05.247 INFO:tasks.workunit.client.0.smithi150.stdout: [ 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: 0.087890625, 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: 0, 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: 0 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -10, 2024-09-18T10:14:05.248 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -11, 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.249 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -12, 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.250 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -13, 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -14, 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.251 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: }, 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: { 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: "bucket_id": -15, 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: "weight_set": [ 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: [], 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: [] 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: } 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout: ] 2024-09-18T10:14:05.252 INFO:tasks.workunit.client.0.smithi150.stdout:} 2024-09-18T10:14:05.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 msgr2=0x7f9820037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:05.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 0x7f9820037f20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f98380098e0 tx=0x7f9838016000 comp rx=0 tx=0).stop 2024-09-18T10:14:05.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 10 monclient: shutdown 2024-09-18T10:14:05.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:05.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 msgr2=0x7f983c142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c610 tx=0x7f982c004a80 comp rx=0 tx=0).stop 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 shutdown_connections 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9820035a70 0x7f9820037f20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09e480 0x7f983c1431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09cf90 0x7f983c142c90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 --2- 172.21.15.150:0/301641187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c0990a0 0x7f983c089690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 >> 172.21.15.150:0/301641187 conn(0x7f983c0a58a0 msgr2=0x7f983c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 shutdown_connections 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.218+0000 7f9843c25700 1 -- 172.21.15.150:0/301641187 wait complete. 2024-09-18T10:14:05.255 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 2024-09-18T10:14:05.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: get_monmap_and_config 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: build_initial_monmap 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: monmap: 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:05.333050+0000 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:05.333050+0000 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:05.334 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:05.335 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:05.335 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:05.335 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 Processor -- start 2024-09-18T10:14:05.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 -- start start 2024-09-18T10:14:05.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: init 2024-09-18T10:14:05.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 0x7f9940096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-b con 0x7f9940098a40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f994009bba0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f994009bf70 0x7f994009ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-c con 0x7f994009bf70 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994009e4f0 con 0x7f994009bba0 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9940097680 con 0x7f994009bf70 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:05.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9940098340 con 0x7f9940098a40 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.330+0000 7f9947c27700 10 monclient(hunting): authenticate will time out at 1127.427998s 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99459c3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 msgr2=0x7f9940096fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99459c3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 0x7f9940096fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99451c2700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f994009bba0 0x7f994009c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57618/0 (socket says 172.21.15.150:57618) 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99451c2700 1 -- 172.21.15.150:0/1858494417 learned_addr learned my addr 172.21.15.150:0/1858494417 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 --2- 172.21.15.150:0/1858494417 >> v2:172.21.15.150:3300/0 conn(0x7f994009bf70 0x7f994009ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): get_auth_request con 0x7f994009bf70 auth_method 0 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:05.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient(hunting): handle_auth_done global_id 15457 payload 274 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 -- 172.21.15.150:0/1858494417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 msgr2=0x7f9940096fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 --2- 172.21.15.150:0/1858494417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 0x7f9940096fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 -- 172.21.15.150:0/1858494417 >> v1:172.21.15.150:6789/0 conn(0x7f994009bba0 legacy=0x7f994009c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient: _finish_hunting 0 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 monclient: found mon.noname-c 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:05.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 -- 172.21.15.150:0/1858494417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9940097f60 con 0x7f994009bf70 2024-09-18T10:14:05.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 -- 172.21.15.150:0/1858494417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994009bf70 msgr2=0x7f994009ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99461c4700 1 --2- 172.21.15.150:0/1858494417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994009bf70 0x7f994009ce30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f993800aac0 tx=0x7f993800a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266487218a35dedd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 1 -- 172.21.15.150:0/1858494417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800ccf0 con 0x7f994009bf70 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: dump: 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:05.347 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: _finish_auth 0 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f9947c27700 5 monclient: authenticate success, global_id 15457 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f9947c27700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 1 -- 172.21.15.150:0/1858494417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800ce50 con 0x7f994009bf70 2024-09-18T10:14:05.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 1 -- 172.21.15.150:0/1858494417 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938018640 con 0x7f994009bf70 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f9947c27700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f9947c27700 4 set_mon_vals no callback set 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.334+0000 7f99449c1700 10 monclient: dump: 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:05.349 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 10 monclient: shutdown 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- 172.21.15.150:0/1858494417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994009bf70 msgr2=0x7f994009ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 --2- 172.21.15.150:0/1858494417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994009bf70 0x7f994009ce30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f993800aac0 tx=0x7f993800a330 comp rx=0 tx=0).stop 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- 172.21.15.150:0/1858494417 shutdown_connections 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 --2- 172.21.15.150:0/1858494417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994009bf70 0x7f994009ce30 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 --2- 172.21.15.150:0/1858494417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9940098a40 0x7f9940096fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- 172.21.15.150:0/1858494417 >> 172.21.15.150:0/1858494417 conn(0x7f9940068730 msgr2=0x7f99400a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- 172.21.15.150:0/1858494417 shutdown_connections 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- 172.21.15.150:0/1858494417 wait complete. 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 10 monclient: build_initial_monmap 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 10 monclient: monmap: 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:05.351 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 Processor -- start 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 1 -- start start 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.338+0000 7f9947c27700 10 monclient: init 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:05.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-b con 0x7f9940098a40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 0x7f9940139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-c con 0x7f994009bba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f994009bf70 0x7f994013d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): picked mon.noname-a con 0x7f994009bf70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994011eba0 con 0x7f994009bf70 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994011e5e0 con 0x7f9940098a40 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:05.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994011c0c0 con 0x7f994009bba0 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient(hunting): authenticate will time out at 1127.436341s 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99451c2700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 msgr2=0x7f9940139260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99451c2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 0x7f9940139260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35828/0 (socket says 172.21.15.150:35828) 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 -- 172.21.15.150:0/427139261 learned_addr learned my addr 172.21.15.150:0/427139261 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): get_auth_request con 0x7f9940098a40 auth_method 0 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:05.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f994009bf70 0x7f994013d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): get_auth_request con 0x7f994009bf70 auth_method 0 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:05.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99461c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient(hunting): handle_auth_done global_id 15460 payload 824 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 -- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 msgr2=0x7f9940139260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 0x7f9940139260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 -- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f994009bf70 msgr2=0x7f994013d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f994009bf70 0x7f994013d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient: _finish_hunting 0 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 monclient: found mon.noname-b 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993800a5d0 con 0x7f9940098a40 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 -- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 msgr2=0x7f9940139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f99459c3700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f993000bb80 tx=0x7f99300097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000cad0 con 0x7f9940098a40 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: dump: 2024-09-18T10:14:05.359 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:05.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 5 monclient: authenticate success, global_id 15460 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993000cc30 con 0x7f9940098a40 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _renew_subs 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994011e300 con 0x7f9940098a40 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000cde0 con 0x7f9940098a40 2024-09-18T10:14:05.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993dffb700 10 monclient: dump: 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:05.362 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f993ffff700 4 set_mon_vals no callback set 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _renew_subs 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.342+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9940139940 con 0x7f9940098a40 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f9947c27700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f9947c27700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f9947c27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9930012020 con 0x7f9940098a40 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9940063eb0 con 0x7f9940098a40 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f993dffb700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 0x7f992c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(130..130 src has 58..130) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f9930048f40 con 0x7f9940098a40 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f99451c2700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 0x7f992c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.346+0000 7f99451c2700 10 monclient: get_auth_request con 0x7f992c035a70 auth_method 0 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.350+0000 7f99451c2700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 0x7f992c037f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f99400693e0 tx=0x7f9934007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.354+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f993001d900 con 0x7f9940098a40 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.354+0000 7f993dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:05.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.354+0000 7f993dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:05.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.526+0000 7f9947c27700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:05.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.526+0000 7f9947c27700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:05.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.526+0000 7f9947c27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:05.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.526+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f9940068d20 con 0x7f9940098a40 2024-09-18T10:14:05.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.898+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v131) v1 ==== 129+0+0 (secure 0 0 0) 0x7f9930014340 con 0x7f9940098a40 2024-09-18T10:14:05.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.902+0000 7f993dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:05.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:05.902+0000 7f993dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:06.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.006+0000 7f9947c27700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:06.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.006+0000 7f9947c27700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:06.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.006+0000 7f9947c27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:06.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.006+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f994013de30 con 0x7f9940098a40 2024-09-18T10:14:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.918+0000 7f993dffb700 1 -- 172.21.15.150:0/427139261 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}]=0 v132) v1 ==== 129+0+0 (secure 0 0 0) 0x7f99300250c0 con 0x7f9940098a40 2024-09-18T10:14:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.918+0000 7f993dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.918+0000 7f993dffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 msgr2=0x7f992c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 0x7f992c037f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f99400693e0 tx=0x7f9934007c10 comp rx=0 tx=0).stop 2024-09-18T10:14:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 10 monclient: shutdown 2024-09-18T10:14:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 msgr2=0x7f9940139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f993000bb80 tx=0x7f99300097f0 comp rx=0 tx=0).stop 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 shutdown_connections 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f992c035a70 0x7f992c037f20 secure :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f99400693e0 tx=0x7f9934007c10 comp rx=0 tx=0).stop 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f994009bf70 0x7f994013d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f994009bba0 0x7f9940139260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 --2- 172.21.15.150:0/427139261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9940098a40 0x7f9940139bb0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 >> 172.21.15.150:0/427139261 conn(0x7f9940068730 msgr2=0x7f994009ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 shutdown_connections 2024-09-18T10:14:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:06.922+0000 7f9947c27700 1 -- 172.21.15.150:0/427139261 wait complete. 2024-09-18T10:14:06.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-18T10:14:06.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:14:06.942 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight fooset osd.0 .9 .9 2024-09-18T10:14:07.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: get_monmap_and_config 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: monmap: 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:07.052470+0000 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:07.052470+0000 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:07.054 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 Processor -- start 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 -- start start 2024-09-18T10:14:07.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: init 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58e40a41a0 0x7f58e4132590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-c con 0x7f58e40a41a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 0x7f58e406eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-b con 0x7f58e406e710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 0x7f58e409e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-a con 0x7f58e409e260 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e4133070 con 0x7f58e409e260 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e4132ad0 con 0x7f58e40a41a0 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40a0020 con 0x7f58e406e710 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58eadc9700 10 monclient(hunting): authenticate will time out at 1129.145865s 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58e40a41a0 0x7f58e4132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58e40a41a0 0x7f58e4132590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35852/0 (socket says 172.21.15.150:35852) 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 -- 172.21.15.150:0/3204690674 learned_addr learned my addr 172.21.15.150:0/3204690674 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): get_auth_request con 0x7f58e40a41a0 auth_method 0 2024-09-18T10:14:07.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e3fff700 1 -- 172.21.15.150:0/3204690674 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 msgr2=0x7f58e406eae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e3fff700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 0x7f58e406eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e9366700 1 --2- 172.21.15.150:0/3204690674 >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 0x7f58e409e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient(hunting): handle_auth_done global_id 15463 payload 274 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 -- 172.21.15.150:0/3204690674 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 msgr2=0x7f58e406eae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 0x7f58e406eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 -- 172.21.15.150:0/3204690674 >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 msgr2=0x7f58e409e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 --2- 172.21.15.150:0/3204690674 >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 0x7f58e409e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 monclient: found mon.noname-c 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e8b65700 1 -- 172.21.15.150:0/3204690674 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e406ad00 con 0x7f58e40a41a0 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e9366700 10 monclient: get_auth_request con 0x7f58e409e260 auth_method 0 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.050+0000 7f58e9366700 1 --2- 172.21.15.150:0/3204690674 >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 0x7f58e409e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:07.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e8b65700 1 -- 172.21.15.150:0/3204690674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e40a41a0 msgr2=0x7f58e4132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e8b65700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e40a41a0 0x7f58e4132590 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f58e406fb80 tx=0x7f58d8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da8827e8d7d7842 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 1 -- 172.21.15.150:0/3204690674 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d800bb10 con 0x7f58e40a41a0 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: dump: 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.063 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.064 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 1 -- 172.21.15.150:0/3204690674 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d800bc70 con 0x7f58e40a41a0 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 5 monclient: authenticate success, global_id 15463 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 1 -- 172.21.15.150:0/3204690674 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8011420 con 0x7f58e40a41a0 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 4 set_mon_vals no callback set 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58e37fe700 10 monclient: dump: 2024-09-18T10:14:07.065 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.066 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 10 monclient: shutdown 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- 172.21.15.150:0/3204690674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e40a41a0 msgr2=0x7f58e4132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e40a41a0 0x7f58e4132590 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f58e406fb80 tx=0x7f58d8009250 comp rx=0 tx=0).stop 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- 172.21.15.150:0/3204690674 shutdown_connections 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 --2- 172.21.15.150:0/3204690674 >> v2:172.21.15.142:3300/0 conn(0x7f58e409e260 0x7f58e409e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e406e710 0x7f58e406eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 --2- 172.21.15.150:0/3204690674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e40a41a0 0x7f58e4132590 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- 172.21.15.150:0/3204690674 >> 172.21.15.150:0/3204690674 conn(0x7f58e40718f0 msgr2=0x7f58e40a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- 172.21.15.150:0/3204690674 shutdown_connections 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- 172.21.15.150:0/3204690674 wait complete. 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 10 monclient: monmap: 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:07.073 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 Processor -- start 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 1 -- start start 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.054+0000 7f58eadc9700 10 monclient: init 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e406e710 0x7f58e4141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-b con 0x7f58e406e710 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58e409e260 0x7f58e4131370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-a con 0x7f58e409e260 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e40a41a0 0x7f58e411aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): picked mon.noname-c con 0x7f58e40a41a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e411cf50 con 0x7f58e409e260 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e406f080 con 0x7f58e406e710 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e406d740 con 0x7f58e40a41a0 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient(hunting): authenticate will time out at 1129.152538s 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e406e710 0x7f58e4141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e406e710 0x7f58e4141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35856/0 (socket says 172.21.15.150:35856) 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 -- 172.21.15.150:0/4190878010 learned_addr learned my addr 172.21.15.150:0/4190878010 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): get_auth_request con 0x7f58e406e710 auth_method 0 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e9366700 1 -- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e40a41a0 msgr2=0x7f58e411aea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e9366700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e40a41a0 0x7f58e411aea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient(hunting): handle_auth_done global_id 15466 payload 824 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 -- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e40a41a0 msgr2=0x7f58e411aea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58e40a41a0 0x7f58e411aea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 -- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58e409e260 msgr2=0x7f58e4131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e3fff700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58e409e260 0x7f58e4131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58e409e260 0x7f58e4131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 monclient: found mon.noname-b 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 -- 172.21.15.150:0/4190878010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d80094f0 con 0x7f58e406e710 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e3fff700 10 monclient: get_auth_request con 0x7f58e409e260 auth_method 0 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e3fff700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58e409e260 0x7f58e4131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 -- 172.21.15.150:0/4190878010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e406e710 msgr2=0x7f58e4141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e8b65700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58e406e710 0x7f58e4141700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f58d8006fd0 tx=0x7f58d8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d801f070 con 0x7f58e406e710 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:07.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: dump: 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 5 monclient: authenticate success, global_id 15466 2024-09-18T10:14:07.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d8000bc0 con 0x7f58e406e710 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8025650 con 0x7f58e406e710 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e0ff9700 10 monclient: dump: 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.080 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.081 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.081 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.081 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.081 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.081 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58e2ffd700 4 set_mon_vals no callback set 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _renew_subs 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 -- 172.21.15.150:0/4190878010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e4141fc0 con 0x7f58e406e710 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _renew_subs 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.058+0000 7f58eadc9700 1 -- 172.21.15.150:0/4190878010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e411c9d0 con 0x7f58e406e710 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f58d8023070 con 0x7f58e406e710 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58e0ff9700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58c8035a70 0x7f58c8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(132..132 src has 58..132) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f58d804a710 con 0x7f58e406e710 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58e3fff700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58c8035a70 0x7f58c8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58e3fff700 10 monclient: get_auth_request con 0x7f58c8035a70 auth_method 0 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58eadc9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58eadc9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58eadc9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.062+0000 7f58eadc9700 1 -- 172.21.15.150:0/4190878010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e411e100 con 0x7f58e406e710 2024-09-18T10:14:07.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.066+0000 7f58e3fff700 1 --2- 172.21.15.150:0/4190878010 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58c8035a70 0x7f58c8037f20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f58d0003980 tx=0x7f58d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.070+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f58d802a070 con 0x7f58e406e710 2024-09-18T10:14:07.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.070+0000 7f58e0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.070+0000 7f58e0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:07.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58eadc9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T10:14:07.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58eadc9700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T10:14:07.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58eadc9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58eadc9700 1 -- 172.21.15.150:0/4190878010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]} v 0) v1 -- 0x7f58e409df40 con 0x7f58e406e710 2024-09-18T10:14:07.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58e0ff9700 1 -- 172.21.15.150:0/4190878010 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}]=-22 must specify exact 1 weight values v132) v1 ==== 168+0+0 (secure 0 0 0) 0x7f58d8015030 con 0x7f58e406e710 2024-09-18T10:14:07.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58e0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "fooset", "item": "osd.0", "weight": [0.9, 0.9]}] 2024-09-18T10:14:07.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.246+0000 7f58e0ff9700 10 monclient: _finish_command 2 = mon:22 must specify exact 1 weight values 2024-09-18T10:14:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: must specify exact 1 weight values 2024-09-18T10:14:07.256 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:14:07.256 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect_false ceph osd crush weight-set reweight barset osd.0 .9 2024-09-18T10:14:07.256 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -x 2024-09-18T10:14:07.256 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 2024-09-18T10:14:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.346+0000 7f9351594700 10 monclient: get_monmap_and_config 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.346+0000 7f9351594700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.346+0000 7f9351594700 10 monclient: monmap: 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:07.351215+0000 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:07.351215+0000 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:07.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 Processor -- start 2024-09-18T10:14:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 -- start start 2024-09-18T10:14:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient: init 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f934c0710a0 0x7f934c096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): picked mon.noname-a con 0x7f934c0710a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f934c096fb0 0x7f934c0973d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): picked mon.noname-c con 0x7f934c096fb0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f934c097910 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934c071470 con 0x7f934c097910 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c099320 con 0x7f934c0710a0 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f934c097d70 con 0x7f934c096fb0 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9351594700 10 monclient(hunting): authenticate will time out at 1129.446027s 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f934c096fb0 0x7f934c0973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f934c096fb0 0x7f934c0973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35858/0 (socket says 172.21.15.150:35858) 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 -- 172.21.15.150:0/2657846095 learned_addr learned my addr 172.21.15.150:0/2657846095 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): get_auth_request con 0x7f934c096fb0 auth_method 0 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f934affd700 1 --2- 172.21.15.150:0/2657846095 >> v2:172.21.15.142:3300/0 conn(0x7f934c0710a0 0x7f934c096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f934affd700 10 monclient(hunting): get_auth_request con 0x7f934c0710a0 auth_method 0 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f934affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f934affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f934affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient(hunting): handle_auth_done global_id 15469 payload 274 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 -- 172.21.15.150:0/2657846095 >> v2:172.21.15.142:3300/0 conn(0x7f934c0710a0 msgr2=0x7f934c096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 --2- 172.21.15.150:0/2657846095 >> v2:172.21.15.142:3300/0 conn(0x7f934c0710a0 0x7f934c096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 -- 172.21.15.150:0/2657846095 >> v1:172.21.15.142:6789/0 conn(0x7f934c097910 legacy=0x7f934c1355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:07.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:07.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.350+0000 7f9343fff700 1 -- 172.21.15.150:0/2657846095 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c09dad0 con 0x7f934c096fb0 2024-09-18T10:14:07.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9343fff700 1 -- 172.21.15.150:0/2657846095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c096fb0 msgr2=0x7f934c0973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9343fff700 1 --2- 172.21.15.150:0/2657846095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c096fb0 0x7f934c0973d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f934c0a2d70 tx=0x7f9334009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b33a454a204faa6c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 1 -- 172.21.15.150:0/2657846095 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400ba70 con 0x7f934c096fb0 2024-09-18T10:14:07.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: dump: 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 5 monclient: authenticate success, global_id 15469 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 1 -- 172.21.15.150:0/2657846095 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400bbd0 con 0x7f934c096fb0 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 4 set_mon_vals no callback set 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 1 -- 172.21.15.150:0/2657846095 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334011420 con 0x7f934c096fb0 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f934a7fc700 10 monclient: dump: 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.363 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 10 monclient: shutdown 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 -- 172.21.15.150:0/2657846095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c096fb0 msgr2=0x7f934c0973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 --2- 172.21.15.150:0/2657846095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c096fb0 0x7f934c0973d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f934c0a2d70 tx=0x7f9334009250 comp rx=0 tx=0).stop 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 -- 172.21.15.150:0/2657846095 shutdown_connections 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 --2- 172.21.15.150:0/2657846095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c096fb0 0x7f934c0973d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 --2- 172.21.15.150:0/2657846095 >> v2:172.21.15.142:3300/0 conn(0x7f934c0710a0 0x7f934c096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 -- 172.21.15.150:0/2657846095 >> 172.21.15.150:0/2657846095 conn(0x7f934c068590 msgr2=0x7f934c0a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 -- 172.21.15.150:0/2657846095 shutdown_connections 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 1 -- 172.21.15.150:0/2657846095 wait complete. 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.354+0000 7f9351594700 10 monclient: monmap: 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:07.365 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:07.366 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 Processor -- start 2024-09-18T10:14:07.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 -- start start 2024-09-18T10:14:07.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient: init 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c0710a0 0x7f934c09c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): picked mon.noname-b con 0x7f934c0710a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f934c096fb0 0x7f934c13f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): picked mon.noname-c con 0x7f934c096fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f934c097910 0x7f934c13b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): picked mon.noname-a con 0x7f934c097910 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c121080 con 0x7f934c097910 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934c09ad70 con 0x7f934c0710a0 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934c11fa90 con 0x7f934c096fb0 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9351594700 10 monclient(hunting): authenticate will time out at 1129.454185s 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9343fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f934c096fb0 msgr2=0x7f934c13f940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c0710a0 0x7f934c09c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f9343fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f934c096fb0 0x7f934c13f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c0710a0 0x7f934c09c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35862/0 (socket says 172.21.15.150:35862) 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 -- 172.21.15.150:0/1911307753 learned_addr learned my addr 172.21.15.150:0/1911307753 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): get_auth_request con 0x7f934c0710a0 auth_method 0 2024-09-18T10:14:07.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f934c097910 0x7f934c13b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): get_auth_request con 0x7f934c097910 auth_method 0 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient(hunting): handle_auth_done global_id 15472 payload 824 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 -- 172.21.15.150:0/1911307753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f934c096fb0 msgr2=0x7f934c13f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f934c096fb0 0x7f934c13f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 -- 172.21.15.150:0/1911307753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f934c097910 msgr2=0x7f934c13b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f934c097910 0x7f934c13b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 monclient: found mon.noname-b 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.358+0000 7f934affd700 1 -- 172.21.15.150:0/1911307753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340094f0 con 0x7f934c0710a0 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f934affd700 1 -- 172.21.15.150:0/1911307753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c0710a0 msgr2=0x7f934c09c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f934affd700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f934c0710a0 0x7f934c09c850 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f933c00d8c0 tx=0x7f933c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c012800 con 0x7f934c0710a0 2024-09-18T10:14:07.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: dump: 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933c012960 con 0x7f934c0710a0 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 5 monclient: authenticate success, global_id 15472 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c012d00 con 0x7f934c0710a0 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 10 monclient: dump: 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.375 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _renew_subs 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 1 -- 172.21.15.150:0/1911307753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934c09d410 con 0x7f934c0710a0 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f93497fa700 4 set_mon_vals no callback set 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _renew_subs 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 1 -- 172.21.15.150:0/1911307753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934c140360 con 0x7f934c0710a0 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f933c014640 con 0x7f934c0710a0 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9351594700 1 -- 172.21.15.150:0/1911307753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f934c098370 con 0x7f934c0710a0 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f932c035ac0 0x7f932c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(132..132 src has 58..132) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f933c0496f0 con 0x7f934c0710a0 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9343fff700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f932c035ac0 0x7f932c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.362+0000 7f9343fff700 10 monclient: get_auth_request con 0x7f932c035ac0 auth_method 0 2024-09-18T10:14:07.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.366+0000 7f9343fff700 1 --2- 172.21.15.150:0/1911307753 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f932c035ac0 0x7f932c037f70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f934c0a2d70 tx=0x7f9334016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.374+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f933c012e60 con 0x7f934c0710a0 2024-09-18T10:14:07.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.374+0000 7f9342ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.374+0000 7f9342ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:07.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9351594700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:07.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9351594700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:07.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9351594700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9351594700 1 -- 172.21.15.150:0/1911307753 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]} v 0) v1 -- 0x7f934c063ed0 con 0x7f934c0710a0 2024-09-18T10:14:07.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9342ffd700 1 -- 172.21.15.150:0/1911307753 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}]=-22 must specify exact 3 weight values v132) v1 ==== 163+0+0 (secure 0 0 0) 0x7f933c01c360 con 0x7f934c0710a0 2024-09-18T10:14:07.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9342ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9]}] 2024-09-18T10:14:07.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.542+0000 7f9342ffd700 10 monclient: _finish_command 2 = mon:22 must specify exact 3 weight values 2024-09-18T10:14:07.549 INFO:tasks.workunit.client.0.smithi150.stderr:Error EINVAL: must specify exact 3 weight values 2024-09-18T10:14:07.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:14:07.553 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight barset osd.0 .9 .9 .9 2024-09-18T10:14:07.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: get_monmap_and_config 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: monmap: 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:07.656358+0000 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:07.656358+0000 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:07.657 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:07.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 Processor -- start 2024-09-18T10:14:07.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 -- start start 2024-09-18T10:14:07.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: init 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66ec09cf70 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66ec09da50 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66ec09b4b0 0x7f66ec137e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-a con 0x7f66ec09b4b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec09aad0 con 0x7f66ec09da50 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec09ea70 con 0x7f66ec09cf70 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec09b190 con 0x7f66ec09b4b0 2024-09-18T10:14:07.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66f1c95700 10 monclient(hunting): authenticate will time out at 1129.751247s 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66eb7fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f66ec09cf70 0x7f66ec09d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57632/0 (socket says 172.21.15.150:57632) 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66eb7fe700 1 -- 172.21.15.150:0/897220073 learned_addr learned my addr 172.21.15.150:0/897220073 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 1 --2- 172.21.15.150:0/897220073 >> v2:172.21.15.142:3300/0 conn(0x7f66ec09b4b0 0x7f66ec137e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): get_auth_request con 0x7f66ec09b4b0 auth_method 0 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 10 monclient(hunting): handle_auth_done global_id 54279 payload 274 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.654+0000 7f66ebfff700 1 -- 172.21.15.150:0/897220073 >> v1:172.21.15.150:6789/0 conn(0x7f66ec09cf70 legacy=0x7f66ec09d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 1 -- 172.21.15.150:0/897220073 >> v1:172.21.15.142:6789/0 conn(0x7f66ec09da50 legacy=0x7f66ec134700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 1 monclient: found mon.noname-a 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:07.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 1 -- 172.21.15.150:0/897220073 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec09a7f0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 1 -- 172.21.15.150:0/897220073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09b4b0 msgr2=0x7f66ec137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ebfff700 1 --2- 172.21.15.150:0/897220073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09b4b0 0x7f66ec137e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009e60 tx=0x7f66e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e484b357f79724 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 1 -- 172.21.15.150:0/897220073 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e400e070 con 0x7f66ec09b4b0 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: dump: 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.667 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66f1c95700 5 monclient: authenticate success, global_id 54279 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66f1c95700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 1 -- 172.21.15.150:0/897220073 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e4012430 con 0x7f66ec09b4b0 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66f1c95700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:07.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66f1c95700 4 set_mon_vals no callback set 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 1 -- 172.21.15.150:0/897220073 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40127d0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.658+0000 7f66ea7fc700 10 monclient: dump: 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.669 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: shutdown 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- 172.21.15.150:0/897220073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09b4b0 msgr2=0x7f66ec137e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 --2- 172.21.15.150:0/897220073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09b4b0 0x7f66ec137e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009e60 tx=0x7f66e4009780 comp rx=0 tx=0).stop 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- 172.21.15.150:0/897220073 shutdown_connections 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 --2- 172.21.15.150:0/897220073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09b4b0 0x7f66ec137e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- 172.21.15.150:0/897220073 >> 172.21.15.150:0/897220073 conn(0x7f66ec0a2580 msgr2=0x7f66ec0a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- 172.21.15.150:0/897220073 shutdown_connections 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- 172.21.15.150:0/897220073 wait complete. 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: build_initial_monmap 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: monmap: 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:07.670 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 Processor -- start 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- start start 2024-09-18T10:14:07.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: init 2024-09-18T10:14:07.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:07.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-b con 0x7f66ec09b4b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 0x7f66ec08d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-c con 0x7f66ec09cf70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09da50 0x7f66ec08d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): picked mon.noname-a con 0x7f66ec09da50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec098bf0 con 0x7f66ec09da50 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec08f850 con 0x7f66ec09b4b0 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:07.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec08e330 con 0x7f66ec09cf70 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66f1c95700 10 monclient(hunting): authenticate will time out at 1129.759753s 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66eaffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 msgr2=0x7f66ec08d330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66eaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 0x7f66ec08d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.662+0000 7f66eb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35874/0 (socket says 172.21.15.150:35874) 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 -- 172.21.15.150:0/3828883914 learned_addr learned my addr 172.21.15.150:0/3828883914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): get_auth_request con 0x7f66ec09b4b0 auth_method 0 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09da50 0x7f66ec08d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): get_auth_request con 0x7f66ec09da50 auth_method 0 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient(hunting): handle_auth_done global_id 15475 payload 824 2024-09-18T10:14:07.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 -- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 msgr2=0x7f66ec08d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 0x7f66ec08d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 -- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09da50 msgr2=0x7f66ec08d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09da50 0x7f66ec08d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e4009a20 con 0x7f66ec09b4b0 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 -- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 msgr2=0x7f66ec08cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66eb7fe700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00c5e0 tx=0x7f66dc004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc005d40 con 0x7f66ec09b4b0 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: dump: 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:07.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 5 monclient: authenticate success, global_id 15475 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 10 monclient: _renew_subs 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec08deb0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 10 monclient: _renew_subs 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec0904f0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc0070d0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc007490 con 0x7f66ec09b4b0 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66d37fe700 10 monclient: dump: 2024-09-18T10:14:07.682 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.666+0000 7f66e9ffb700 4 set_mon_vals no callback set 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f66dc01c440 con 0x7f66ec09b4b0 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66d37fe700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 0x7f66d8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:07.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(132..132 src has 58..132) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f66dc048670 con 0x7f66ec09b4b0 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66f1c95700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66f1c95700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66f1c95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66eaffd700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 0x7f66d8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc0050b0 con 0x7f66ec09b4b0 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66eaffd700 10 monclient: get_auth_request con 0x7f66d8035a70 auth_method 0 2024-09-18T10:14:07.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.670+0000 7f66eaffd700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 0x7f66d8037f20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f66e0007fd0 tx=0x7f66e0007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:07.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.678+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f66dc017070 con 0x7f66ec09b4b0 2024-09-18T10:14:07.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.678+0000 7f66d37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:07.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.678+0000 7f66d37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:07.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.850+0000 7f66f1c95700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:07.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.850+0000 7f66f1c95700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:07.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.850+0000 7f66f1c95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:07.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:07.850+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f66cc005360 con 0x7f66ec09b4b0 2024-09-18T10:14:08.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.034+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v133) v1 ==== 139+0+0 (secure 0 0 0) 0x7f66dc01b3b0 con 0x7f66ec09b4b0 2024-09-18T10:14:08.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.034+0000 7f66d37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:08.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.034+0000 7f66d37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:08.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.142+0000 7f66f1c95700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:08.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.142+0000 7f66f1c95700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:08.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.142+0000 7f66f1c95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:08.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:08.142+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]} v 0) v1 -- 0x7f66cc002250 con 0x7f66ec09b4b0 2024-09-18T10:14:09.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.046+0000 7f66d37fe700 1 -- 172.21.15.150:0/3828883914 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}]=0 v134) v1 ==== 139+0+0 (secure 0 0 0) 0x7f66dc010300 con 0x7f66ec09b4b0 2024-09-18T10:14:09.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.046+0000 7f66d37fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "barset", "item": "osd.0", "weight": [0.9, 0.9, 0.9]}] 2024-09-18T10:14:09.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.046+0000 7f66d37fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 msgr2=0x7f66d8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 0x7f66d8037f20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f66e0007fd0 tx=0x7f66e0007b00 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 10 monclient: shutdown 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 msgr2=0x7f66ec08cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00c5e0 tx=0x7f66dc004a80 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 shutdown_connections 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66d8035a70 0x7f66d8037f20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66ec09da50 0x7f66ec08d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66ec09cf70 0x7f66ec08d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 --2- 172.21.15.150:0/3828883914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66ec09b4b0 0x7f66ec08cdf0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 >> 172.21.15.150:0/3828883914 conn(0x7f66ec0a2580 msgr2=0x7f66ec098390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:09.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 shutdown_connections 2024-09-18T10:14:09.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.050+0000 7f66f1c95700 1 -- 172.21.15.150:0/3828883914 wait complete. 2024-09-18T10:14:09.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:09.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c fooset 2024-09-18T10:14:09.064 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 1 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: get_monmap_and_config 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: build_initial_monmap 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: monmap: 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:09.165732+0000 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:09.165732+0000 2024-09-18T10:14:09.166 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:09.167 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 Processor -- start 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 -- start start 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: init 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb24809d970 0x7fb2480955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-c con 0x7fb24809d970 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb248098ac0 0x7fb248098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-a con 0x7fb248098ac0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2480993d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24809b0a0 con 0x7fb2480993d0 2024-09-18T10:14:09.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24809ab50 con 0x7fb248098ac0 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24809ae30 con 0x7fb24809d970 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24fe8e700 10 monclient(hunting): authenticate will time out at 1131.259168s 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb24809d970 0x7fb2480955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb24809d970 0x7fb2480955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60116/0 (socket says 172.21.15.150:60116) 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 1 -- 172.21.15.150:0/101861981 learned_addr learned my addr 172.21.15.150:0/101861981 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): get_auth_request con 0x7fb24809d970 auth_method 0 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:09.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24d429700 1 --2- 172.21.15.150:0/101861981 >> v2:172.21.15.142:3300/0 conn(0x7fb248098ac0 0x7fb248098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24d429700 10 monclient(hunting): get_auth_request con 0x7fb248098ac0 auth_method 0 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24d429700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24d429700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24d429700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 10 monclient(hunting): handle_auth_done global_id 15478 payload 274 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 1 -- 172.21.15.150:0/101861981 >> v2:172.21.15.142:3300/0 conn(0x7fb248098ac0 msgr2=0x7fb248098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.162+0000 7fb24dc2a700 1 --2- 172.21.15.150:0/101861981 >> v2:172.21.15.142:3300/0 conn(0x7fb248098ac0 0x7fb248098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 1 -- 172.21.15.150:0/101861981 >> v1:172.21.15.142:6789/0 conn(0x7fb2480993d0 legacy=0x7fb248135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 10 monclient: _finish_hunting 0 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 1 monclient: found mon.noname-c 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:09.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 1 -- 172.21.15.150:0/101861981 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24809dd40 con 0x7fb24809d970 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 1 -- 172.21.15.150:0/101861981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24809d970 msgr2=0x7fb2480955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24dc2a700 1 --2- 172.21.15.150:0/101861981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24809d970 0x7fb2480955b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb248067420 tx=0x7fb240009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dba3509c8f78d485 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 1 -- 172.21.15.150:0/101861981 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24000e070 con 0x7fb24809d970 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: dump: 2024-09-18T10:14:09.178 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: _finish_auth 0 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 5 monclient: authenticate success, global_id 15478 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 1 -- 172.21.15.150:0/101861981 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24000bc60 con 0x7fb24809d970 2024-09-18T10:14:09.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 1 -- 172.21.15.150:0/101861981 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2400092e0 con 0x7fb24809d970 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 4 set_mon_vals no callback set 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24cc28700 10 monclient: dump: 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.180 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 10 monclient: shutdown 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 -- 172.21.15.150:0/101861981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24809d970 msgr2=0x7fb2480955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/101861981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24809d970 0x7fb2480955b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb248067420 tx=0x7fb240009840 comp rx=0 tx=0).stop 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 -- 172.21.15.150:0/101861981 shutdown_connections 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/101861981 >> v2:172.21.15.142:3300/0 conn(0x7fb248098ac0 0x7fb248098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/101861981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb24809d970 0x7fb2480955b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 -- 172.21.15.150:0/101861981 >> 172.21.15.150:0/101861981 conn(0x7fb2480a15e0 msgr2=0x7fb2480a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 -- 172.21.15.150:0/101861981 shutdown_connections 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.166+0000 7fb24fe8e700 1 -- 172.21.15.150:0/101861981 wait complete. 2024-09-18T10:14:09.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient: build_initial_monmap 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient: monmap: 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 Processor -- start 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 -- start start 2024-09-18T10:14:09.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient: init 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 0x7fb248141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-c con 0x7fb248098ac0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-b con 0x7fb2480993d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb24809d970 0x7fb24808e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): picked mon.noname-a con 0x7fb24809d970 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248142940 con 0x7fb24809d970 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24808fcb0 con 0x7fb2480993d0 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb248098450 con 0x7fb248098ac0 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24fe8e700 10 monclient(hunting): authenticate will time out at 1131.266171s 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24dc2a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 msgr2=0x7fb248141a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24dc2a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 0x7fb248141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60126/0 (socket says 172.21.15.150:60126) 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 -- 172.21.15.150:0/1182972910 learned_addr learned my addr 172.21.15.150:0/1182972910 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): get_auth_request con 0x7fb2480993d0 auth_method 0 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:09.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb24809d970 0x7fb24808e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): get_auth_request con 0x7fb24809d970 auth_method 0 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24e42b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient(hunting): handle_auth_done global_id 15481 payload 824 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 -- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 msgr2=0x7fb248141a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 0x7fb248141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 -- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb24809d970 msgr2=0x7fb24808e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb24809d970 0x7fb24808e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient: _finish_hunting 0 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 monclient: found mon.noname-b 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.170+0000 7fb24d429700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240009ae0 con 0x7fb2480993d0 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24d429700 1 -- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 msgr2=0x7fb248094530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:09.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24d429700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00d8c0 tx=0x7fb23c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c012800 con 0x7fb2480993d0 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: dump: 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.187 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 5 monclient: authenticate success, global_id 15481 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c012960 con 0x7fb2480993d0 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _renew_subs 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248091750 con 0x7fb2480993d0 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c012d20 con 0x7fb2480993d0 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 10 monclient: dump: 2024-09-18T10:14:09.188 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _renew_subs 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2480907f0 con 0x7fb2480993d0 2024-09-18T10:14:09.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb247fff700 4 set_mon_vals no callback set 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb248063ed0 con 0x7fb2480993d0 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.174+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb23c014640 con 0x7fb2480993d0 2024-09-18T10:14:09.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.178+0000 7fb245ffb700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 0x7fb234037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.178+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(134..134 src has 58..134) v4 ==== 8276+0+0 (secure 0 0 0) 0x7fb23c049710 con 0x7fb2480993d0 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.178+0000 7fb24dc2a700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 0x7fb234037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.178+0000 7fb24dc2a700 10 monclient: get_auth_request con 0x7fb234035a70 auth_method 0 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.178+0000 7fb24dc2a700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 0x7fb234037f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb24808f6e0 tx=0x7fb240017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.186+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb23c011570 con 0x7fb2480993d0 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.186+0000 7fb245ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.186+0000 7fb245ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb24fe8e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb24fe8e700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb24fe8e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb248067b10 con 0x7fb2480993d0 2024-09-18T10:14:09.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb23c01dbb0 con 0x7fb2480993d0 2024-09-18T10:14:09.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb245ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.358+0000 7fb245ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:09.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb24fe8e700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb24fe8e700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb24fe8e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb248068c60 con 0x7fb2480993d0 2024-09-18T10:14:09.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb245ffb700 1 -- 172.21.15.150:0/1182972910 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v134) v1 ==== 71+0+14 (secure 0 0 0) 0x7fb23c01dd90 con 0x7fb2480993d0 2024-09-18T10:14:09.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb245ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:09.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.462+0000 7fb245ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:09.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 msgr2=0x7fb234037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 0x7fb234037f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb24808f6e0 tx=0x7fb240017000 comp rx=0 tx=0).stop 2024-09-18T10:14:09.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 10 monclient: shutdown 2024-09-18T10:14:09.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 msgr2=0x7fb248094530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00d8c0 tx=0x7fb23c008040 comp rx=0 tx=0).stop 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 shutdown_connections 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb234035a70 0x7fb234037f20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb24809d970 0x7fb24808e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2480993d0 0x7fb248094530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 --2- 172.21.15.150:0/1182972910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb248098ac0 0x7fb248141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 >> 172.21.15.150:0/1182972910 conn(0x7fb2480a15e0 msgr2=0x7fb24806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 shutdown_connections 2024-09-18T10:14:09.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.466+0000 7fb24fe8e700 1 -- 172.21.15.150:0/1182972910 wait complete. 2024-09-18T10:14:09.479 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm fooset 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: get_monmap_and_config 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: build_initial_monmap 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: monmap: 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:09.577449+0000 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:09.577449+0000 2024-09-18T10:14:09.578 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:09.579 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 Processor -- start 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 -- start start 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: init 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59280a4270 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f592806ed20 0x7f59280a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): picked mon.noname-a con 0x7f592806ed20 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5928137590 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:09.582 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59280a2220 con 0x7f5928137590 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59280a2830 con 0x7f59280a4270 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59280a4640 con 0x7f592806ed20 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592f349700 10 monclient(hunting): authenticate will time out at 1131.671148s 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592d0e5700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f59280a4270 0x7f592806e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37646/0 (socket says 172.21.15.150:37646) 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592d0e5700 1 -- 172.21.15.150:0/3714489824 learned_addr learned my addr 172.21.15.150:0/3714489824 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 1 --2- 172.21.15.150:0/3714489824 >> v2:172.21.15.142:3300/0 conn(0x7f592806ed20 0x7f59280a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): get_auth_request con 0x7f592806ed20 auth_method 0 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.583 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.574+0000 7f592c8e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 10 monclient(hunting): handle_auth_done global_id 54291 payload 274 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 -- 172.21.15.150:0/3714489824 >> v1:172.21.15.150:6789/0 conn(0x7f59280a4270 legacy=0x7f592806e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 -- 172.21.15.150:0/3714489824 >> v1:172.21.15.142:6789/0 conn(0x7f5928137590 legacy=0x7f5928139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 10 monclient: _finish_hunting 0 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 monclient: found mon.noname-a 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:09.584 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 -- 172.21.15.150:0/3714489824 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592806b990 con 0x7f592806ed20 2024-09-18T10:14:09.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 -- 172.21.15.150:0/3714489824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f592806ed20 msgr2=0x7f59280a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:09.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592c8e4700 1 --2- 172.21.15.150:0/3714489824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f592806ed20 0x7f59280a1ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59240098e0 tx=0x7f59240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0802002756e40b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 1 -- 172.21.15.150:0/3714489824 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592400ba70 con 0x7f592806ed20 2024-09-18T10:14:09.586 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: dump: 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.587 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: _finish_auth 0 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 5 monclient: authenticate success, global_id 54291 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 1 -- 172.21.15.150:0/3714489824 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592400bbd0 con 0x7f592806ed20 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 1 -- 172.21.15.150:0/3714489824 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924011420 con 0x7f592806ed20 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 4 set_mon_vals no callback set 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:09.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f591ffff700 10 monclient: dump: 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 10 monclient: shutdown 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:09.589 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 -- 172.21.15.150:0/3714489824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f592806ed20 msgr2=0x7f59280a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 --2- 172.21.15.150:0/3714489824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f592806ed20 0x7f59280a1ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59240098e0 tx=0x7f59240091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 -- 172.21.15.150:0/3714489824 shutdown_connections 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 --2- 172.21.15.150:0/3714489824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f592806ed20 0x7f59280a1ce0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 -- 172.21.15.150:0/3714489824 >> 172.21.15.150:0/3714489824 conn(0x7f59280716e0 msgr2=0x7f59280a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 -- 172.21.15.150:0/3714489824 shutdown_connections 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.578+0000 7f592f349700 1 -- 172.21.15.150:0/3714489824 wait complete. 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient: build_initial_monmap 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient: monmap: 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:09.590 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 Processor -- start 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 -- start start 2024-09-18T10:14:09.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient: init 2024-09-18T10:14:09.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:09.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): picked mon.noname-b con 0x7f592806ed20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 0x7f592811b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): picked mon.noname-a con 0x7f59280a4270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 0x7f592811be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): picked mon.noname-c con 0x7f5928137590 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592806f160 con 0x7f59280a4270 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592806c9d0 con 0x7f592806ed20 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:09.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592811f1c0 con 0x7f5928137590 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592f349700 10 monclient(hunting): authenticate will time out at 1131.678306s 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60128/0 (socket says 172.21.15.150:60128) 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 -- 172.21.15.150:0/4218635022 learned_addr learned my addr 172.21.15.150:0/4218635022 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): get_auth_request con 0x7f592806ed20 auth_method 0 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d8e6700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 msgr2=0x7f592811be50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:09.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d8e6700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 0x7f592811be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 0x7f592811b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 10 monclient(hunting): get_auth_request con 0x7f59280a4270 auth_method 0 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient(hunting): handle_auth_done global_id 15484 payload 824 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 msgr2=0x7f592811be50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:09.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 0x7f592811be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 msgr2=0x7f592811b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 0x7f592811b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient: _finish_hunting 0 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 monclient: found mon.noname-b 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592d0e5700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924009420 con 0x7f592806ed20 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.582+0000 7f592c8e4700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 0x7f592811b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592d0e5700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 msgr2=0x7f5928136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592d0e5700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f591800c5e0 tx=0x7f5918004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918005d40 con 0x7f592806ed20 2024-09-18T10:14:09.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: dump: 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.600 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 5 monclient: authenticate success, global_id 15484 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59180070d0 con 0x7f592806ed20 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _renew_subs 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5928136590 con 0x7f592806ed20 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918007490 con 0x7f592806ed20 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:09.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 10 monclient: dump: 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591effd700 4 set_mon_vals no callback set 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _renew_subs 2024-09-18T10:14:09.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592811eee0 con 0x7f592806ed20 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f591801b3b0 con 0x7f592806ed20 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 0x7f590c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(134..134 src has 58..134) v4 ==== 8276+0+0 (secure 0 0 0) 0x7f5918048810 con 0x7f592806ed20 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592811fce0 con 0x7f592806ed20 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592c8e4700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 0x7f590c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.586+0000 7f592c8e4700 10 monclient: get_auth_request con 0x7f590c035a70 auth_method 0 2024-09-18T10:14:09.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.590+0000 7f592c8e4700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 0x7f590c037f20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f59240030c0 tx=0x7f5924016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:09.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.594+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5918017070 con 0x7f592806ed20 2024-09-18T10:14:09.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.594+0000 7f591d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:09.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.594+0000 7f591d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:09.778 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.774+0000 7f592f349700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:09.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.774+0000 7f592f349700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:09.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.774+0000 7f592f349700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:09.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:09.774+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f592811f300 con 0x7f592806ed20 2024-09-18T10:14:10.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.046+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v135) v1 ==== 89+0+0 (secure 0 0 0) 0x7f591801a360 con 0x7f592806ed20 2024-09-18T10:14:10.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.046+0000 7f591d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:10.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.046+0000 7f591d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:10.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.158+0000 7f592f349700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:10.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.158+0000 7f592f349700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:10.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.158+0000 7f592f349700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:10.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:10.158+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "fooset"} v 0) v1 -- 0x7f5928063ed0 con 0x7f592806ed20 2024-09-18T10:14:11.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.054+0000 7f591d7fa700 1 -- 172.21.15.150:0/4218635022 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "fooset"}]=0 v136) v1 ==== 89+0+0 (secure 0 0 0) 0x7f591800e290 con 0x7f592806ed20 2024-09-18T10:14:11.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.054+0000 7f591d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "fooset"}] 2024-09-18T10:14:11.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.054+0000 7f591d7fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 msgr2=0x7f590c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 0x7f590c037f20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f59240030c0 tx=0x7f5924016000 comp rx=0 tx=0).stop 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 10 monclient: shutdown 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 msgr2=0x7f5928136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f591800c5e0 tx=0x7f5918004a80 comp rx=0 tx=0).stop 2024-09-18T10:14:11.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 shutdown_connections 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f590c035a70 0x7f590c037f20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5928137590 0x7f592811be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59280a4270 0x7f592811b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 --2- 172.21.15.150:0/4218635022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f592806ed20 0x7f5928136f80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 >> 172.21.15.150:0/4218635022 conn(0x7f59280716e0 msgr2=0x7f592809e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 shutdown_connections 2024-09-18T10:14:11.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.062+0000 7f592f349700 1 -- 172.21.15.150:0/4218635022 wait complete. 2024-09-18T10:14:11.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:11.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 0 2024-09-18T10:14:11.079 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c fooset 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: get_monmap_and_config 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: build_initial_monmap 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: monmap: 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:11.177481+0000 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:11.177481+0000 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.178 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 1 Processor -- start 2024-09-18T10:14:11.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 1 -- start start 2024-09-18T10:14:11.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: init 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.174+0000 7f4fadce8700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fa809e6d0 0x7f4fa80a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-c con 0x7f4fa809e6d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 0x7f4fa80a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-b con 0x7f4fa80a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fa8136a10 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa80a1110 con 0x7f4fa8136a10 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa809ee30 con 0x7f4fa809e6d0 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa809e220 con 0x7f4fa80a07d0 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fadce8700 10 monclient(hunting): authenticate will time out at 1133.272233s 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fa809e6d0 0x7f4fa80a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa6ffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 msgr2=0x7f4fa80a0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 0x7f4fa80a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fa809e6d0 0x7f4fa80a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60142/0 (socket says 172.21.15.150:60142) 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 -- 172.21.15.150:0/192657081 learned_addr learned my addr 172.21.15.150:0/192657081 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): get_auth_request con 0x7f4fa809e6d0 auth_method 0 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa7fff700 1 --1- 172.21.15.150:0/192657081 >> v1:172.21.15.150:6789/0 conn(0x7f4fa8136a10 0x7f4fa8138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37650/0 (socket says 172.21.15.150:37650) 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa67fc700 1 -- 172.21.15.150:0/192657081 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:14:11.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa7fff700 1 -- 172.21.15.150:0/192657081 >> v1:172.21.15.150:6789/0 conn(0x7f4fa8136a10 legacy=0x7f4fa8138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_done global_id 15487 payload 274 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 -- 172.21.15.150:0/192657081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 msgr2=0x7f4fa80a0bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 --2- 172.21.15.150:0/192657081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 0x7f4fa80a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 -- 172.21.15.150:0/192657081 >> v1:172.21.15.150:6789/0 conn(0x7f4fa8136a10 legacy=0x7f4fa8138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 monclient: found mon.noname-c 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.178+0000 7f4fa77fe700 1 -- 172.21.15.150:0/192657081 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa809eaa0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa77fe700 1 -- 172.21.15.150:0/192657081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 msgr2=0x7f4fa80a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa77fe700 1 --2- 172.21.15.150:0/192657081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa80a0290 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009a60 tx=0x7f4f98009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a509e06d42b55cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 1 -- 172.21.15.150:0/192657081 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9800bb60 con 0x7f4fa809e6d0 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:11.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: dump: 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:11.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 5 monclient: authenticate success, global_id 15487 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 1 -- 172.21.15.150:0/192657081 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9800bcc0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 4 set_mon_vals no callback set 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 1 -- 172.21.15.150:0/192657081 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98011670 con 0x7f4fa809e6d0 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fa67fc700 10 monclient: dump: 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 10 monclient: shutdown 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 -- 172.21.15.150:0/192657081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 msgr2=0x7f4fa80a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 --2- 172.21.15.150:0/192657081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa80a0290 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009a60 tx=0x7f4f98009250 comp rx=0 tx=0).stop 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 -- 172.21.15.150:0/192657081 shutdown_connections 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 --2- 172.21.15.150:0/192657081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa80a07d0 0x7f4fa80a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 --2- 172.21.15.150:0/192657081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa80a0290 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 -- 172.21.15.150:0/192657081 >> 172.21.15.150:0/192657081 conn(0x7f4fa80a58a0 msgr2=0x7f4fa80a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 -- 172.21.15.150:0/192657081 shutdown_connections 2024-09-18T10:14:11.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.182+0000 7f4fadce8700 1 -- 172.21.15.150:0/192657081 wait complete. 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient: build_initial_monmap 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient: monmap: 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:11.195 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 Processor -- start 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 -- start start 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient: init 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-b con 0x7f4fa809e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4fa80a07d0 0x7f4fa812b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-a con 0x7f4fa80a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 0x7f4fa812bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): picked mon.noname-c con 0x7f4fa8136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa812e9c0 con 0x7f4fa80a07d0 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa812cc90 con 0x7f4fa809e6d0 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa812d970 con 0x7f4fa8136a10 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fadce8700 10 monclient(hunting): authenticate will time out at 1133.282192s 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa7fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 msgr2=0x7f4fa812bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 0x7f4fa812bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60144/0 (socket says 172.21.15.150:60144) 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 -- 172.21.15.150:0/394460769 learned_addr learned my addr 172.21.15.150:0/394460769 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): get_auth_request con 0x7f4fa809e6d0 auth_method 0 2024-09-18T10:14:11.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4fa80a07d0 0x7f4fa812b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): get_auth_request con 0x7f4fa80a07d0 auth_method 0 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient(hunting): handle_auth_done global_id 15493 payload 824 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 -- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 msgr2=0x7f4fa812bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 0x7f4fa812bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 -- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4fa80a07d0 msgr2=0x7f4fa812b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4fa80a07d0 0x7f4fa812b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 monclient: found mon.noname-b 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.186+0000 7f4fa77fe700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f980094f0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fa77fe700 1 -- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 msgr2=0x7f4fa812b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fa77fe700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009930 tx=0x7f4f98017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9801f070 con 0x7f4fa809e6d0 2024-09-18T10:14:11.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: dump: 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f98011630 con 0x7f4fa809e6d0 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 5 monclient: authenticate success, global_id 15493 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _renew_subs 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa812d3b0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f98025670 con 0x7f4fa809e6d0 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 10 monclient: dump: 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.201 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fa5ffb700 4 set_mon_vals no callback set 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _renew_subs 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa812e6e0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4f98023070 con 0x7f4fa809e6d0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa8063ed0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 0x7f4f94037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(136..136 src has 58..136) v4 ==== 7944+0+0 (secure 0 0 0) 0x7f4f9804a4d0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fa6ffd700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 0x7f4f94037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.190+0000 7f4fa6ffd700 10 monclient: get_auth_request con 0x7f4f94035aa0 auth_method 0 2024-09-18T10:14:11.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.194+0000 7f4fa6ffd700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 0x7f4f94037f50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c003980 tx=0x7f4f9c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.202+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4f98000cc0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.202+0000 7f4f8f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.202+0000 7f4f8f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4fadce8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4fadce8700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4fadce8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4fa8067b10 con 0x7f4fa809e6d0 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+7 (secure 0 0 0) 0x7f4f9802a030 con 0x7f4fa809e6d0 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4f8f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.374+0000 7f4f8f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:11.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4fadce8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4fadce8700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.478 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4fadce8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f4fa81356e0 con 0x7f4fa809e6d0 2024-09-18T10:14:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4f8f7fe700 1 -- 172.21.15.150:0/394460769 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+7 (secure 0 0 0) 0x7f4f9802a020 con 0x7f4fa809e6d0 2024-09-18T10:14:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4f8f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.479 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.474+0000 7f4f8f7fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 msgr2=0x7f4f94037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 0x7f4f94037f50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c003980 tx=0x7f4f9c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 10 monclient: shutdown 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 msgr2=0x7f4fa812b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009930 tx=0x7f4f98017000 comp rx=0 tx=0).stop 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 shutdown_connections 2024-09-18T10:14:11.481 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f94035aa0 0x7f4f94037f50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4fa8136a10 0x7f4fa812bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4fa80a07d0 0x7f4fa812b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 --2- 172.21.15.150:0/394460769 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fa809e6d0 0x7f4fa812b040 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 >> 172.21.15.150:0/394460769 conn(0x7f4fa80a58a0 msgr2=0x7f4fa8136230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 shutdown_connections 2024-09-18T10:14:11.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.478+0000 7f4fadce8700 1 -- 172.21.15.150:0/394460769 wait complete. 2024-09-18T10:14:11.491 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:11.491 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep barset 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 10 monclient: get_monmap_and_config 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 10 monclient: build_initial_monmap 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 10 monclient: monmap: 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:11.586699+0000 2024-09-18T10:14:11.587 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:11.586699+0000 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:11.588 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.590 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 1 Processor -- start 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 1 -- start start 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.582+0000 7f24a02f4700 10 monclient: init 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f249809ad30 0x7f249809b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-a con 0x7f249809ad30 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2498099180 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24980995e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249809a5b0 con 0x7f24980995e0 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249809cdf0 con 0x7f2498099180 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249809a080 con 0x7f249809ad30 2024-09-18T10:14:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f24a02f4700 10 monclient(hunting): authenticate will time out at 1133.680307s 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249d88f700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2498099180 0x7f249809bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37656/0 (socket says 172.21.15.150:37656) 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249d88f700 1 -- 172.21.15.150:0/3738396183 learned_addr learned my addr 172.21.15.150:0/3738396183 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 --2- 172.21.15.150:0/3738396183 >> v2:172.21.15.142:3300/0 conn(0x7f249809ad30 0x7f249809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): get_auth_request con 0x7f249809ad30 auth_method 0 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient(hunting): handle_auth_done global_id 54300 payload 274 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 -- 172.21.15.150:0/3738396183 >> v1:172.21.15.150:6789/0 conn(0x7f2498099180 legacy=0x7f249809bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 -- 172.21.15.150:0/3738396183 >> v1:172.21.15.142:6789/0 conn(0x7f24980995e0 legacy=0x7f249809c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient: _finish_hunting 0 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 monclient: found mon.noname-a 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 -- 172.21.15.150:0/3738396183 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2498099ca0 con 0x7f249809ad30 2024-09-18T10:14:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 -- 172.21.15.150:0/3738396183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f249809ad30 msgr2=0x7f249809b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.586+0000 7f249e090700 1 --2- 172.21.15.150:0/3738396183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f249809ad30 0x7f249809b100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f248c0098e0 tx=0x7f248c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae0a8bbfb4bc739 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 1 -- 172.21.15.150:0/3738396183 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c00ba70 con 0x7f249809ad30 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: dump: 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.596 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: _finish_auth 0 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 5 monclient: authenticate success, global_id 54300 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 1 -- 172.21.15.150:0/3738396183 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c00bbd0 con 0x7f249809ad30 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 1 -- 172.21.15.150:0/3738396183 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c011420 con 0x7f249809ad30 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 4 set_mon_vals no callback set 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f249d08e700 10 monclient: dump: 2024-09-18T10:14:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: shutdown 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- 172.21.15.150:0/3738396183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f249809ad30 msgr2=0x7f249809b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 --2- 172.21.15.150:0/3738396183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f249809ad30 0x7f249809b100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f248c0098e0 tx=0x7f248c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- 172.21.15.150:0/3738396183 shutdown_connections 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 --2- 172.21.15.150:0/3738396183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f249809ad30 0x7f249809b100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- 172.21.15.150:0/3738396183 >> 172.21.15.150:0/3738396183 conn(0x7f24980a15e0 msgr2=0x7f24980a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- 172.21.15.150:0/3738396183 shutdown_connections 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- 172.21.15.150:0/3738396183 wait complete. 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: build_initial_monmap 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: monmap: 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 Processor -- start 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- start start 2024-09-18T10:14:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: init 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2498099180 0x7f2498145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-a con 0x7f2498099180 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-b con 0x7f24980995e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 0x7f249806b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): picked mon.noname-c con 0x7f249806d660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249806f200 con 0x7f2498099180 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981397b0 con 0x7f24980995e0 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249806ef20 con 0x7f249806d660 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.590+0000 7f24a02f4700 10 monclient(hunting): authenticate will time out at 1133.687930s 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e891700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 msgr2=0x7f249806b770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e891700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 0x7f249806b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60152/0 (socket says 172.21.15.150:60152) 2024-09-18T10:14:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 -- 172.21.15.150:0/2703063970 learned_addr learned my addr 172.21.15.150:0/2703063970 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): get_auth_request con 0x7f24980995e0 auth_method 0 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2498099180 0x7f2498145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): get_auth_request con 0x7f2498099180 auth_method 0 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249e090700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient(hunting): handle_auth_done global_id 15496 payload 824 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 -- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 msgr2=0x7f249806b770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 0x7f249806b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 -- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2498099180 msgr2=0x7f2498145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2498099180 0x7f2498145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient: _finish_hunting 0 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 monclient: found mon.noname-b 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c009420 con 0x7f24980995e0 2024-09-18T10:14:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 -- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 msgr2=0x7f2498138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f249d88f700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f249000cca0 tx=0x7f2490005480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000abd0 con 0x7f24980995e0 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: dump: 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.609 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249000ad50 con 0x7f24980995e0 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 5 monclient: authenticate success, global_id 15496 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 10 monclient: _renew_subs 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2498146d40 con 0x7f24980995e0 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490007340 con 0x7f24980995e0 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24967fc700 10 monclient: dump: 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f2497fff700 4 set_mon_vals no callback set 2024-09-18T10:14:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 10 monclient: _renew_subs 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.594+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2498146480 con 0x7f24980995e0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2490007730 con 0x7f24980995e0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24967fc700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 0x7f2484037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(136..136 src has 58..136) v4 ==== 7944+0+0 (secure 0 0 0) 0x7f24900494f0 con 0x7f24980995e0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24a02f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24a02f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24a02f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f249e090700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 0x7f2484037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249809dd00 con 0x7f24980995e0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.598+0000 7f249e090700 10 monclient: get_auth_request con 0x7f2484035a70 auth_method 0 2024-09-18T10:14:11.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.602+0000 7f249e090700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 0x7f2484037f20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2498072390 tx=0x7f248c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.606+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2490012400 con 0x7f24980995e0 2024-09-18T10:14:11.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.606+0000 7f24967fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:11.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.606+0000 7f24967fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:11.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.774+0000 7f24a02f4700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.774+0000 7f24a02f4700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.774+0000 7f24a02f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.780 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.774+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f2498063ed0 con 0x7f24980995e0 2024-09-18T10:14:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.778+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+7 (secure 0 0 0) 0x7f2490028020 con 0x7f24980995e0 2024-09-18T10:14:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.778+0000 7f24967fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.778+0000 7f24967fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:11.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24a02f4700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24a02f4700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24a02f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:11.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f24980008d0 con 0x7f24980995e0 2024-09-18T10:14:11.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24967fc700 1 -- 172.21.15.150:0/2703063970 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v136) v1 ==== 71+0+7 (secure 0 0 0) 0x7f2490028020 con 0x7f24980995e0 2024-09-18T10:14:11.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24967fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:11.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.878+0000 7f24967fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 msgr2=0x7f2484037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 0x7f2484037f20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2498072390 tx=0x7f248c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 10 monclient: shutdown 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 msgr2=0x7f2498138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f249000cca0 tx=0x7f2490005480 comp rx=0 tx=0).stop 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 shutdown_connections 2024-09-18T10:14:11.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2484035a70 0x7f2484037f20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f249806d660 0x7f249806b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24980995e0 0x7f2498138bb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 --2- 172.21.15.150:0/2703063970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2498099180 0x7f2498145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 >> 172.21.15.150:0/2703063970 conn(0x7f24980a15e0 msgr2=0x7f24981399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 shutdown_connections 2024-09-18T10:14:11.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.882+0000 7f24a02f4700 1 -- 172.21.15.150:0/2703063970 wait complete. 2024-09-18T10:14:11.894 INFO:tasks.workunit.client.0.smithi150.stdout:barset 2024-09-18T10:14:11.894 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm barset 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: get_monmap_and_config 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: build_initial_monmap 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: monmap: 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:11.985657+0000 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:11.985657+0000 2024-09-18T10:14:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:11.987 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 Processor -- start 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 -- start start 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: init 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f687c09bc90 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 0x7f687c099d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-b con 0x7f687c09c770 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f687c133aa0 0x7f687c135e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-a con 0x7f687c133aa0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c09a250 con 0x7f687c09bc90 2024-09-18T10:14:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687c09cb40 con 0x7f687c133aa0 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c099040 con 0x7f687c09c770 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6882d7c700 10 monclient(hunting): authenticate will time out at 1134.079319s 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6881579700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 msgr2=0x7f687c099d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6881579700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 0x7f687c099d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6881d7a700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f687c09bc90 0x7f687c09c060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37668/0 (socket says 172.21.15.150:37668) 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f6881d7a700 1 -- 172.21.15.150:0/138294183 learned_addr learned my addr 172.21.15.150:0/138294183 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 1 --2- 172.21.15.150:0/138294183 >> v2:172.21.15.142:3300/0 conn(0x7f687c133aa0 0x7f687c135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): get_auth_request con 0x7f687c133aa0 auth_method 0 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:11.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.982+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 10 monclient(hunting): handle_auth_done global_id 54306 payload 274 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 -- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 msgr2=0x7f687c099d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 --2- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 0x7f687c099d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 -- 172.21.15.150:0/138294183 >> v1:172.21.15.150:6789/0 conn(0x7f687c09bc90 legacy=0x7f687c09c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 10 monclient: _finish_hunting 0 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 monclient: found mon.noname-a 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:11.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 -- 172.21.15.150:0/138294183 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c098af0 con 0x7f687c133aa0 2024-09-18T10:14:11.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 -- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c133aa0 msgr2=0x7f687c135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:11.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f688257b700 1 --2- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c133aa0 0x7f687c135e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6874009e60 tx=0x7f6874009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484ff67b523f8153 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:11.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 1 -- 172.21.15.150:0/138294183 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400e070 con 0x7f687c133aa0 2024-09-18T10:14:11.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:11.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: dump: 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: _finish_auth 0 2024-09-18T10:14:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 5 monclient: authenticate success, global_id 54306 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 1 -- 172.21.15.150:0/138294183 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6874012430 con 0x7f687c133aa0 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 4 set_mon_vals no callback set 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 1 -- 172.21.15.150:0/138294183 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740127d0 con 0x7f687c133aa0 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6880d78700 10 monclient: dump: 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.001 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 10 monclient: shutdown 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 -- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c133aa0 msgr2=0x7f687c135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 --2- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c133aa0 0x7f687c135e80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6874009e60 tx=0x7f6874009780 comp rx=0 tx=0).stop 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 -- 172.21.15.150:0/138294183 shutdown_connections 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 --2- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c133aa0 0x7f687c135e80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:12.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 --2- 172.21.15.150:0/138294183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09c770 0x7f687c099d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 -- 172.21.15.150:0/138294183 >> 172.21.15.150:0/138294183 conn(0x7f687c0a12a0 msgr2=0x7f687c0a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 -- 172.21.15.150:0/138294183 shutdown_connections 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.986+0000 7f6882d7c700 1 -- 172.21.15.150:0/138294183 wait complete. 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient: build_initial_monmap 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient: monmap: 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:12.003 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 Processor -- start 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 -- start start 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient: init 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 0x7f687c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-c con 0x7f687c09bc90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c09c770 0x7f687c117900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-a con 0x7f687c09c770 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): picked mon.noname-b con 0x7f687c133aa0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c118cd0 con 0x7f687c09c770 2024-09-18T10:14:12.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c09f130 con 0x7f687c133aa0 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c118aa0 con 0x7f687c09bc90 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6882d7c700 10 monclient(hunting): authenticate will time out at 1134.086159s 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881d7a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 msgr2=0x7f687c06dde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881d7a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 0x7f687c06dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60166/0 (socket says 172.21.15.150:60166) 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 -- 172.21.15.150:0/350569812 learned_addr learned my addr 172.21.15.150:0/350569812 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): get_auth_request con 0x7f687c133aa0 auth_method 0 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:12.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c09c770 0x7f687c117900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): get_auth_request con 0x7f687c09c770 auth_method 0 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f6881579700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient(hunting): handle_auth_done global_id 15499 payload 824 2024-09-18T10:14:12.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 -- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 msgr2=0x7f687c06dde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 0x7f687c06dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 -- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c09c770 msgr2=0x7f687c117900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c09c770 0x7f687c117900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient: _finish_hunting 0 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 monclient: found mon.noname-b 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.990+0000 7f688257b700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6874009a20 con 0x7f687c133aa0 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f688257b700 1 -- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 msgr2=0x7f687c117e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f688257b700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f687c064c60 tx=0x7f6874018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f688257b700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f687c064c60 tx=0x7f6874018000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f6874026430 mon_map magic: 0 v1 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6874026430 con 0x7f687c133aa0 2024-09-18T10:14:12.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: dump: 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:12.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6874021070 con 0x7f687c133aa0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 5 monclient: authenticate success, global_id 15499 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _renew_subs 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c119c70 con 0x7f687c133aa0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6873fff700 4 set_mon_vals no callback set 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _renew_subs 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c119f50 con 0x7f687c133aa0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:12.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c04bbf0 con 0x7f687c133aa0 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740267e0 con 0x7f687c133aa0 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 10 monclient: dump: 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:12.010 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:12.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6874023030 con 0x7f687c133aa0 2024-09-18T10:14:12.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6871ffb700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 0x7f6860037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:12.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6881d7a700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 0x7f6860037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:12.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.994+0000 7f6881d7a700 10 monclient: get_auth_request con 0x7f6860035af0 auth_method 0 2024-09-18T10:14:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:11.998+0000 7f6881d7a700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 0x7f6860037fa0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f686c004860 tx=0x7f686c007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:12.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.002+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(136..136 src has 58..136) v4 ==== 7944+0+0 (secure 0 0 0) 0x7f687404b360 con 0x7f687c133aa0 2024-09-18T10:14:12.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.618+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f687402f440 con 0x7f687c133aa0 2024-09-18T10:14:12.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.618+0000 7f6871ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:12.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.618+0000 7f6871ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:12.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.794+0000 7f6882d7c700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:12.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.794+0000 7f6882d7c700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:12.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.794+0000 7f6882d7c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:12.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:12.794+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f687c063ed0 con 0x7f687c133aa0 2024-09-18T10:14:13.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.066+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v137) v1 ==== 89+0+0 (secure 0 0 0) 0x7f687401b3a0 con 0x7f687c133aa0 2024-09-18T10:14:13.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.066+0000 7f6871ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:13.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.066+0000 7f6871ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:13.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.178+0000 7f6882d7c700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:13.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.178+0000 7f6882d7c700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:13.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.178+0000 7f6882d7c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:13.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:13.178+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm", "pool": "barset"} v 0) v1 -- 0x7f687c063ed0 con 0x7f687c133aa0 2024-09-18T10:14:14.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.070+0000 7f6871ffb700 1 -- 172.21.15.150:0/350569812 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm", "pool": "barset"}]=0 v138) v1 ==== 89+0+0 (secure 0 0 0) 0x7f687401b580 con 0x7f687c133aa0 2024-09-18T10:14:14.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.070+0000 7f6871ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm", "pool": "barset"}] 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.070+0000 7f6871ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 msgr2=0x7f6860037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 0x7f6860037fa0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f686c004860 tx=0x7f686c007c10 comp rx=0 tx=0).stop 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 10 monclient: shutdown 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 msgr2=0x7f687c117e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f687c064c60 tx=0x7f6874018000 comp rx=0 tx=0).stop 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 shutdown_connections 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6860035af0 0x7f6860037fa0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f687c133aa0 0x7f687c117e40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f687c09c770 0x7f687c117900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 --2- 172.21.15.150:0/350569812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f687c09bc90 0x7f687c06dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 >> 172.21.15.150:0/350569812 conn(0x7f687c0a12a0 msgr2=0x7f687c09dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:14.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 shutdown_connections 2024-09-18T10:14:14.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.074+0000 7f6882d7c700 1 -- 172.21.15.150:0/350569812 wait complete. 2024-09-18T10:14:14.095 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:14.095 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -c barset 2024-09-18T10:14:14.095 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep -q 0 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: get_monmap_and_config 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: build_initial_monmap 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: monmap: 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:14.184801+0000 2024-09-18T10:14:14.185 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:14.184801+0000 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:14.186 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 Processor -- start 2024-09-18T10:14:14.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 -- start start 2024-09-18T10:14:14.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: init 2024-09-18T10:14:14.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:14.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T10:14:14.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb68c09e6d0 0x7fb68c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-c con 0x7fb68c09e6d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb68c0a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 0x7fb68c13a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-b con 0x7fb68c0a0bd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c09e220 con 0x7fb68c0a07d0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c09ee30 con 0x7fb68c09e6d0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09eaa0 con 0x7fb68c0a0bd0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb6943cd700 10 monclient(hunting): authenticate will time out at 1136.279497s 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb69296a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 msgr2=0x7fb68c13a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:14.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb69296a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 0x7fb68c13a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb68c09e6d0 0x7fb68c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb68c09e6d0 0x7fb68c0a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60180/0 (socket says 172.21.15.150:60180) 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 1 -- 172.21.15.150:0/2365390100 learned_addr learned my addr 172.21.15.150:0/2365390100 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 10 monclient(hunting): get_auth_request con 0x7fb68c09e6d0 auth_method 0 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.182+0000 7fb692169700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient(hunting): handle_auth_done global_id 15502 payload 274 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 -- 172.21.15.150:0/2365390100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 msgr2=0x7fb68c13a140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 --2- 172.21.15.150:0/2365390100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 0x7fb68c13a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 -- 172.21.15.150:0/2365390100 >> v1:172.21.15.142:6789/0 conn(0x7fb68c0a07d0 legacy=0x7fb68c136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient: _finish_hunting 0 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 monclient: found mon.noname-c 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 -- 172.21.15.150:0/2365390100 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c09d930 con 0x7fb68c09e6d0 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 -- 172.21.15.150:0/2365390100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c09e6d0 msgr2=0x7fb68c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb692169700 1 --2- 172.21.15.150:0/2365390100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c09e6d0 0x7fb68c0a0290 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb6880099b0 tx=0x7fb688009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad1f0ff070256bca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 1 -- 172.21.15.150:0/2365390100 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800bb50 con 0x7fb68c09e6d0 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: dump: 2024-09-18T10:14:14.195 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: _finish_auth 0 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb6943cd700 5 monclient: authenticate success, global_id 15502 2024-09-18T10:14:14.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb6943cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 1 -- 172.21.15.150:0/2365390100 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68800bcb0 con 0x7fb68c09e6d0 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 1 -- 172.21.15.150:0/2365390100 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688011670 con 0x7fb68c09e6d0 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb6943cd700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb6943cd700 4 set_mon_vals no callback set 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.186+0000 7fb691167700 10 monclient: dump: 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:14.197 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 10 monclient: shutdown 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- 172.21.15.150:0/2365390100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c09e6d0 msgr2=0x7fb68c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2365390100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c09e6d0 0x7fb68c0a0290 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb6880099b0 tx=0x7fb688009250 comp rx=0 tx=0).stop 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- 172.21.15.150:0/2365390100 shutdown_connections 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2365390100 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c0a0bd0 0x7fb68c13a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2365390100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c09e6d0 0x7fb68c0a0290 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- 172.21.15.150:0/2365390100 >> 172.21.15.150:0/2365390100 conn(0x7fb68c0a58a0 msgr2=0x7fb68c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:14.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- 172.21.15.150:0/2365390100 shutdown_connections 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- 172.21.15.150:0/2365390100 wait complete. 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 10 monclient: build_initial_monmap 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 10 monclient: monmap: 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:14.199 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 Processor -- start 2024-09-18T10:14:14.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 1 -- start start 2024-09-18T10:14:14.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.190+0000 7fb6943cd700 10 monclient: init 2024-09-18T10:14:14.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:14.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:14.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 0x7fb68c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-c con 0x7fb68c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:14.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-b con 0x7fb68c0a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c0a0bd0 0x7fb68c08df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): picked mon.noname-a con 0x7fb68c0a0bd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c0994e0 con 0x7fb68c0a0bd0 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09c2b0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c091830 con 0x7fb68c09e6d0 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient(hunting): authenticate will time out at 1136.288460s 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb692169700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 msgr2=0x7fb68c141d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:14.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb692169700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 0x7fb68c141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60182/0 (socket says 172.21.15.150:60182) 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 -- 172.21.15.150:0/2397088295 learned_addr learned my addr 172.21.15.150:0/2397088295 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): get_auth_request con 0x7fb68c0a07d0 auth_method 0 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c0a0bd0 0x7fb68c08df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:14.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): get_auth_request con 0x7fb68c0a0bd0 auth_method 0 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb69296a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient(hunting): handle_auth_done global_id 15505 payload 824 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 -- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 msgr2=0x7fb68c141d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 0x7fb68c141d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 -- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c0a0bd0 msgr2=0x7fb68c08df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c0a0bd0 0x7fb68c08df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient: _finish_hunting 0 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 monclient: found mon.noname-b 2024-09-18T10:14:14.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6880094f0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 -- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 msgr2=0x7fb68c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb68000c600 tx=0x7fb680007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb691968700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb68000c600 tx=0x7fb680007f60 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fb6800117a0 mon_map magic: 0 v1 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6800117a0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 10 monclient: dump: 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:14.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 5 monclient: authenticate success, global_id 15505 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _renew_subs 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb680011900 con 0x7fb68c0a07d0 2024-09-18T10:14:14.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c091260 con 0x7fb68c0a07d0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _renew_subs 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c090660 con 0x7fb68c0a07d0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.194+0000 7fb67ffff700 4 set_mon_vals no callback set 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb6943cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb6943cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb6943cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c0919a0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680011cc0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 10 monclient: dump: 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb680012630 con 0x7fb68c0a07d0 2024-09-18T10:14:14.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb67e7fc700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 0x7fb670037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:14.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb692169700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 0x7fb670037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:14.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.198+0000 7fb692169700 10 monclient: get_auth_request con 0x7fb670035a70 auth_method 0 2024-09-18T10:14:14.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.202+0000 7fb692169700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 0x7fb670037f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb6880099b0 tx=0x7fb688016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:14.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.202+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(138..138 src has 58..138) v4 ==== 7332+0+0 (secure 0 0 0) 0x7fb680048c00 con 0x7fb68c0a07d0 2024-09-18T10:14:14.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.818+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb680011e60 con 0x7fb68c0a07d0 2024-09-18T10:14:14.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.818+0000 7fb67e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:14.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.818+0000 7fb67e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:14.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.990+0000 7fb6943cd700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:14.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.990+0000 7fb6943cd700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:14.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.990+0000 7fb6943cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:14.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.990+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb68c063ed0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.994+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v138) v1 ==== 71+0+0 (secure 0 0 0) 0x7fb6800102e0 con 0x7fb68c0a07d0 2024-09-18T10:14:14.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.994+0000 7fb67e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:14.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:14.994+0000 7fb67e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:15.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:15.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:15.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7fb68c0008d0 con 0x7fb68c0a07d0 2024-09-18T10:14:15.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb67e7fc700 1 -- 172.21.15.150:0/2397088295 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v138) v1 ==== 71+0+0 (secure 0 0 0) 0x7fb6800104c0 con 0x7fb68c0a07d0 2024-09-18T10:14:15.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb67e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:15.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb67e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 msgr2=0x7fb670037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 0x7fb670037f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb6880099b0 tx=0x7fb688016000 comp rx=0 tx=0).stop 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 10 monclient: shutdown 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 msgr2=0x7fb68c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb68000c600 tx=0x7fb680007f60 comp rx=0 tx=0).stop 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 shutdown_connections 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb670035a70 0x7fb670037f20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb68c0a0bd0 0x7fb68c08df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb68c0a07d0 0x7fb68c142250 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 --2- 172.21.15.150:0/2397088295 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb68c09e6d0 0x7fb68c141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.094+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 >> 172.21.15.150:0/2397088295 conn(0x7fb68c0a58a0 msgr2=0x7fb68c09ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:15.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.098+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 shutdown_connections 2024-09-18T10:14:15.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.098+0000 7fb6943cd700 1 -- 172.21.15.150:0/2397088295 wait complete. 2024-09-18T10:14:15.110 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T10:14:15.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: get_monmap_and_config 2024-09-18T10:14:15.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: build_initial_monmap 2024-09-18T10:14:15.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: monmap: 2024-09-18T10:14:15.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:15.208825+0000 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:15.208825+0000 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:15.210 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:15.211 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:15.211 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 Processor -- start 2024-09-18T10:14:15.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 -- start start 2024-09-18T10:14:15.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: init 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 0x7fea180977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-b con 0x7fea1809c930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea1809cd00 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fea18137770 0x7fea18139b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-c con 0x7fea18137770 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea18098f00 con 0x7fea1809cd00 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1809d7e0 con 0x7fea18137770 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea18097d00 con 0x7fea1809c930 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:15.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea1d4c0700 10 monclient(hunting): authenticate will time out at 1137.302329s 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fea18137770 0x7fea18139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea16ffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 msgr2=0x7fea180977c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea16ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 0x7fea180977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fea18137770 0x7fea18139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60188/0 (socket says 172.21.15.150:60188) 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 -- 172.21.15.150:0/586547916 learned_addr learned my addr 172.21.15.150:0/586547916 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): get_auth_request con 0x7fea18137770 auth_method 0 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:15.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient(hunting): handle_auth_done global_id 15508 payload 274 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 -- 172.21.15.150:0/586547916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 msgr2=0x7fea180977c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 --2- 172.21.15.150:0/586547916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 0x7fea180977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 -- 172.21.15.150:0/586547916 >> v1:172.21.15.142:6789/0 conn(0x7fea1809cd00 legacy=0x7fea1809d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 monclient: found mon.noname-c 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:15.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.206+0000 7fea177fe700 1 -- 172.21.15.150:0/586547916 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea18097030 con 0x7fea18137770 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea177fe700 1 -- 172.21.15.150:0/586547916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 msgr2=0x7fea18139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea177fe700 1 --2- 172.21.15.150:0/586547916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea18139b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fea10009f20 tx=0x7fea10009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cc288fdbc77bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 1 -- 172.21.15.150:0/586547916 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1000e070 con 0x7fea18137770 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: dump: 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.224 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 1 -- 172.21.15.150:0/586547916 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1000bc90 con 0x7fea18137770 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 5 monclient: authenticate success, global_id 15508 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:15.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 1 -- 172.21.15.150:0/586547916 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10012660 con 0x7fea18137770 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 4 set_mon_vals no callback set 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea15ffb700 10 monclient: dump: 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:15.226 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 10 monclient: shutdown 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 -- 172.21.15.150:0/586547916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 msgr2=0x7fea18139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/586547916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea18139b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fea10009f20 tx=0x7fea10009840 comp rx=0 tx=0).stop 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 -- 172.21.15.150:0/586547916 shutdown_connections 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/586547916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea18139b50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/586547916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809c930 0x7fea180977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 -- 172.21.15.150:0/586547916 >> 172.21.15.150:0/586547916 conn(0x7fea18068360 msgr2=0x7fea180a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 -- 172.21.15.150:0/586547916 shutdown_connections 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 1 -- 172.21.15.150:0/586547916 wait complete. 2024-09-18T10:14:15.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 10 monclient: build_initial_monmap 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.210+0000 7fea1d4c0700 10 monclient: monmap: 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 Processor -- start 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 -- start start 2024-09-18T10:14:15.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient: init 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 0x7fea18119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-a con 0x7fea1809c930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 0x7fea1811a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-c con 0x7fea1809cd00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): picked mon.noname-b con 0x7fea18137770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1809e3f0 con 0x7fea1809c930 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1809b980 con 0x7fea18137770 2024-09-18T10:14:15.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1811d630 con 0x7fea1809cd00 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea1d4c0700 10 monclient(hunting): authenticate will time out at 1137.309441s 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60190/0 (socket says 172.21.15.150:60190) 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 -- 172.21.15.150:0/1065048954 learned_addr learned my addr 172.21.15.150:0/1065048954 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): get_auth_request con 0x7fea18137770 auth_method 0 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:15.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 0x7fea18119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 10 monclient(hunting): get_auth_request con 0x7fea1809c930 auth_method 0 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient(hunting): handle_auth_done global_id 15511 payload 824 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 msgr2=0x7fea1811a420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea167fc700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 msgr2=0x7fea1811a420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea167fc700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 0x7fea1811a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 0x7fea1811a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 msgr2=0x7fea18119ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:15.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 0x7fea18119ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 monclient: found mon.noname-b 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea177fe700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea10009ae0 con 0x7fea18137770 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.214+0000 7fea16ffd700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 0x7fea18119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea177fe700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 msgr2=0x7fea1811a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea177fe700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fea1000b6c0 tx=0x7fea10017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10012e70 con 0x7fea18137770 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: dump: 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:15.232 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea100070e0 con 0x7fea18137770 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 5 monclient: authenticate success, global_id 15511 2024-09-18T10:14:15.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _renew_subs 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1811d350 con 0x7fea18137770 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1001e650 con 0x7fea18137770 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 10 monclient: dump: 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:15.234 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _renew_subs 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1811c460 con 0x7fea18137770 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea14ff9700 4 set_mon_vals no callback set 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fea10019030 con 0x7fea18137770 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:15.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1811d7c0 con 0x7fea18137770 2024-09-18T10:14:15.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 0x7fea04037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:15.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(138..138 src has 58..138) v4 ==== 7332+0+0 (secure 0 0 0) 0x7fea1004ab20 con 0x7fea18137770 2024-09-18T10:14:15.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea16ffd700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 0x7fea04037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:15.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.218+0000 7fea16ffd700 10 monclient: get_auth_request con 0x7fea04035a70 auth_method 0 2024-09-18T10:14:15.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.222+0000 7fea16ffd700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 0x7fea04037f20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea0c004fd0 tx=0x7fea0c004aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:15.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.226+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fea1001ea00 con 0x7fea18137770 2024-09-18T10:14:15.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.226+0000 7fe9feffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:15.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.226+0000 7fe9feffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:15.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.394+0000 7fea1d4c0700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:15.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.394+0000 7fea1d4c0700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:15.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.394+0000 7fea1d4c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:15.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:15.394+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fea18063ed0 con 0x7fea18137770 2024-09-18T10:14:16.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.090+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v139) v1 ==== 82+0+0 (secure 0 0 0) 0x7fea1001a7f0 con 0x7fea18137770 2024-09-18T10:14:16.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.090+0000 7fe9feffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:16.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.090+0000 7fe9feffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:16.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.202+0000 7fea1d4c0700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:16.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.202+0000 7fea1d4c0700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:16.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.202+0000 7fea1d4c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.202+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fea180008d0 con 0x7fea18137770 2024-09-18T10:14:16.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fe9feffd700 1 -- 172.21.15.150:0/1065048954 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v139) v1 ==== 115+0+0 (secure 0 0 0) 0x7fea1001a9d0 con 0x7fea18137770 2024-09-18T10:14:16.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fe9feffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:16.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fe9feffd700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T10:14:16.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 msgr2=0x7fea04037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 0x7fea04037f20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fea0c004fd0 tx=0x7fea0c004aa0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 10 monclient: shutdown 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 msgr2=0x7fea1811a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fea1000b6c0 tx=0x7fea10017000 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 shutdown_connections 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fea04035a70 0x7fea04037f20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fea18137770 0x7fea1811a960 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea1809cd00 0x7fea1811a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 --2- 172.21.15.150:0/1065048954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea1809c930 0x7fea18119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 >> 172.21.15.150:0/1065048954 conn(0x7fea18068360 msgr2=0x7fea180a0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:16.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 shutdown_connections 2024-09-18T10:14:16.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.206+0000 7fea1d4c0700 1 -- 172.21.15.150:0/1065048954 wait complete. 2024-09-18T10:14:16.213 INFO:tasks.workunit.client.0.smithi150.stderr:compat weight-set already created 2024-09-18T10:14:16.222 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set ls 2024-09-18T10:14:16.222 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '(compat)' 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: get_monmap_and_config 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: build_initial_monmap 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: monmap: 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:16.326549+0000 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:16.326549+0000 2024-09-18T10:14:16.327 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:16.328 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 Processor -- start 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 -- start start 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: init 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f47ec096a70 0x7f47ec06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-a con 0x7f47ec096a70 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47ec09bc90 0x7f47ec09c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-c con 0x7f47ec09bc90 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 0x7f47ec135ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-b con 0x7f47ec09c5d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec0995c0 con 0x7f47ec096a70 2024-09-18T10:14:16.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47ec06f8e0 con 0x7f47ec09bc90 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec09d500 con 0x7f47ec09c5d0 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.322+0000 7f47f32e7700 10 monclient(hunting): authenticate will time out at 1138.419854s 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f2ae6700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 msgr2=0x7f47ec135ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47ec09bc90 0x7f47ec09c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f2ae6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 0x7f47ec135ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47ec09bc90 0x7f47ec09c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60198/0 (socket says 172.21.15.150:60198) 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 -- 172.21.15.150:0/3337757385 learned_addr learned my addr 172.21.15.150:0/3337757385 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): get_auth_request con 0x7f47ec09bc90 auth_method 0 2024-09-18T10:14:16.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 1 --2- 172.21.15.150:0/3337757385 >> v2:172.21.15.142:3300/0 conn(0x7f47ec096a70 0x7f47ec06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): get_auth_request con 0x7f47ec096a70 auth_method 0 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f22e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_done global_id 15514 payload 274 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 -- 172.21.15.150:0/3337757385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 msgr2=0x7f47ec135ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/3337757385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 0x7f47ec135ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 -- 172.21.15.150:0/3337757385 >> v2:172.21.15.142:3300/0 conn(0x7f47ec096a70 msgr2=0x7f47ec06f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/3337757385 >> v2:172.21.15.142:3300/0 conn(0x7f47ec096a70 0x7f47ec06f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient: _finish_hunting 0 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 monclient: found mon.noname-c 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 -- 172.21.15.150:0/3337757385 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec096e40 con 0x7f47ec09bc90 2024-09-18T10:14:16.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 -- 172.21.15.150:0/3337757385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09bc90 msgr2=0x7f47ec09c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:16.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/3337757385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09bc90 0x7f47ec09c090 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f47ec064c00 tx=0x7f47e800b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63645a891557f4cc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 1 -- 172.21.15.150:0/3337757385 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e800ddb0 con 0x7f47ec09bc90 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: dump: 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.342 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: _finish_auth 0 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 1 -- 172.21.15.150:0/3337757385 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e8019480 con 0x7f47ec09bc90 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f32e7700 5 monclient: authenticate success, global_id 15514 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f32e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 1 -- 172.21.15.150:0/3337757385 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8019820 con 0x7f47ec09bc90 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f32e7700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:16.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f32e7700 4 set_mon_vals no callback set 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.326+0000 7f47f12e3700 10 monclient: dump: 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.344 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 10 monclient: shutdown 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- 172.21.15.150:0/3337757385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09bc90 msgr2=0x7f47ec09c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 --2- 172.21.15.150:0/3337757385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09bc90 0x7f47ec09c090 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f47ec064c00 tx=0x7f47e800b4f0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- 172.21.15.150:0/3337757385 shutdown_connections 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 --2- 172.21.15.150:0/3337757385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec09c5d0 0x7f47ec135ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 --2- 172.21.15.150:0/3337757385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09bc90 0x7f47ec09c090 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 --2- 172.21.15.150:0/3337757385 >> v2:172.21.15.142:3300/0 conn(0x7f47ec096a70 0x7f47ec06f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- 172.21.15.150:0/3337757385 >> 172.21.15.150:0/3337757385 conn(0x7f47ec0a12a0 msgr2=0x7f47ec0a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- 172.21.15.150:0/3337757385 shutdown_connections 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- 172.21.15.150:0/3337757385 wait complete. 2024-09-18T10:14:16.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 10 monclient: build_initial_monmap 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 10 monclient: monmap: 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 Processor -- start 2024-09-18T10:14:16.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 1 -- start start 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 10 monclient: init 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.330+0000 7f47f32e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 0x7f47ec13d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-c con 0x7f47ec096a70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-a con 0x7f47ec09bc90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 0x7f47ec117900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): picked mon.noname-b con 0x7f47ec09c5d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec09b190 con 0x7f47ec09bc90 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec119420 con 0x7f47ec09c5d0 2024-09-18T10:14:16.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec119910 con 0x7f47ec096a70 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f22e5700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 msgr2=0x7f47ec13d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient(hunting): authenticate will time out at 1138.428240s 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f22e5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 0x7f47ec13d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 0x7f47ec117900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:45528/0 (socket says 172.21.15.150:45528) 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 -- 172.21.15.150:0/640114405 learned_addr learned my addr 172.21.15.150:0/640114405 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): get_auth_request con 0x7f47ec09bc90 auth_method 0 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 0x7f47ec117900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60212/0 (socket says 172.21.15.150:60212) 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): get_auth_request con 0x7f47ec09c5d0 auth_method 0 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f2ae6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient(hunting): handle_auth_done global_id 54321 payload 824 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 -- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 msgr2=0x7f47ec13d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:16.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 0x7f47ec13d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 -- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 msgr2=0x7f47ec117900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 0x7f47ec117900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient: _finish_hunting 0 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 monclient: found mon.noname-a 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e800b790 con 0x7f47ec09bc90 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 -- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 msgr2=0x7f47ec12dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f1ae4700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06fa20 tx=0x7f47e801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8019800 con 0x7f47ec09bc90 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:16.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: dump: 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:16.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e8010680 con 0x7f47ec09bc90 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 5 monclient: authenticate success, global_id 54321 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient: _renew_subs 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec06edb0 con 0x7f47ec09bc90 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e8010a40 con 0x7f47ec09bc90 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47e27fc700 10 monclient: dump: 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.334+0000 7f47f0ae2700 4 set_mon_vals no callback set 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 10 monclient: _renew_subs 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec11aec0 con 0x7f47ec09bc90 2024-09-18T10:14:16.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec11ab30 con 0x7f47ec09bc90 2024-09-18T10:14:16.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f47e80183e0 con 0x7f47ec09bc90 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47e27fc700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 0x7f47d0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(139..139 src has 58..139) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f47e8014070 con 0x7f47ec09bc90 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f22e5700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 0x7f47d0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.338+0000 7f47f22e5700 10 monclient: get_auth_request con 0x7f47d0035a70 auth_method 0 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.342+0000 7f47f22e5700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 0x7f47d0037f20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009420 tx=0x7f47dc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.350+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f47e8018e40 con 0x7f47ec09bc90 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.350+0000 7f47e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.350+0000 7f47e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:16.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47f32e7700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47f32e7700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47f32e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f47ec04bbf0 con 0x7f47ec09bc90 2024-09-18T10:14:16.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+9 (secure 0 0 0) 0x7f47e800e360 con 0x7f47ec09bc90 2024-09-18T10:14:16.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.526+0000 7f47e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:16.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set ls"} v 0) v1 -- 0x7f47ec063ed0 con 0x7f47ec09bc90 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47e27fc700 1 -- 172.21.15.150:0/640114405 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set ls"}]=0 v139) v1 ==== 71+0+9 (secure 0 0 0) 0x7f47e800e540 con 0x7f47ec09bc90 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47e27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set ls"}] 2024-09-18T10:14:16.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47e27fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:16.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 msgr2=0x7f47d0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 0x7f47d0037f20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009420 tx=0x7f47dc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 10 monclient: shutdown 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 msgr2=0x7f47ec12dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f47ec06fa20 tx=0x7f47e801b000 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.626+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 shutdown_connections 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f47d0035a70 0x7f47d0037f20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47ec09c5d0 0x7f47ec117900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47ec09bc90 0x7f47ec12dda0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 --2- 172.21.15.150:0/640114405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47ec096a70 0x7f47ec13d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 >> 172.21.15.150:0/640114405 conn(0x7f47ec0a12a0 msgr2=0x7f47ec130f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 shutdown_connections 2024-09-18T10:14:16.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.630+0000 7f47f32e7700 1 -- 172.21.15.150:0/640114405 wait complete. 2024-09-18T10:14:16.641 INFO:tasks.workunit.client.0.smithi150.stdout:(compat) 2024-09-18T10:14:16.642 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T10:14:16.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: get_monmap_and_config 2024-09-18T10:14:16.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: build_initial_monmap 2024-09-18T10:14:16.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: monmap: 2024-09-18T10:14:16.741 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:16.741 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:16.740212+0000 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:16.740212+0000 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 Processor -- start 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 -- start start 2024-09-18T10:14:16.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: init 2024-09-18T10:14:16.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:16.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:14:16.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-b con 0x7f450c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:16.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f450c0a07d0 0x7f450c0a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-c con 0x7f450c0a07d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f450c136a10 0x7f450c138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-a con 0x7f450c136a10 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c09ee30 con 0x7f450c136a10 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450c0a10e0 con 0x7f450c0a07d0 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c09e220 con 0x7f450c09e6d0 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f45140e8700 10 monclient(hunting): authenticate will time out at 1138.833713s 2024-09-18T10:14:16.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511e84700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 msgr2=0x7f450c0a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f450c0a07d0 0x7f450c0a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511e84700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f450c0a07d0 0x7f450c0a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60218/0 (socket says 172.21.15.150:60218) 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 -- 172.21.15.150:0/973557189 learned_addr learned my addr 172.21.15.150:0/973557189 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): get_auth_request con 0x7f450c0a07d0 auth_method 0 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 1 --2- 172.21.15.150:0/973557189 >> v2:172.21.15.142:3300/0 conn(0x7f450c136a10 0x7f450c138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): get_auth_request con 0x7f450c136a10 auth_method 0 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4512685700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient(hunting): handle_auth_done global_id 15520 payload 274 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 -- 172.21.15.150:0/973557189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 msgr2=0x7f450c0a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 --2- 172.21.15.150:0/973557189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 -- 172.21.15.150:0/973557189 >> v2:172.21.15.142:3300/0 conn(0x7f450c136a10 msgr2=0x7f450c138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:16.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 --2- 172.21.15.150:0/973557189 >> v2:172.21.15.142:3300/0 conn(0x7f450c136a10 0x7f450c138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient: _finish_hunting 0 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 monclient: found mon.noname-c 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.738+0000 7f4511683700 1 -- 172.21.15.150:0/973557189 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c09eaa0 con 0x7f450c0a07d0 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4511683700 1 -- 172.21.15.150:0/973557189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 msgr2=0x7f450c0a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4511683700 1 --2- 172.21.15.150:0/973557189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c0a0ba0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f450c06a610 tx=0x7f4508009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6462a9b1834e0f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 1 -- 172.21.15.150:0/973557189 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508009e70 con 0x7f450c0a07d0 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: dump: 2024-09-18T10:14:16.750 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: _finish_auth 0 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 1 -- 172.21.15.150:0/973557189 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450800ba70 con 0x7f450c0a07d0 2024-09-18T10:14:16.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 5 monclient: authenticate success, global_id 15520 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 1 -- 172.21.15.150:0/973557189 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450800be30 con 0x7f450c0a07d0 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 4 set_mon_vals no callback set 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:16.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f4510e82700 10 monclient: dump: 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 10 monclient: shutdown 2024-09-18T10:14:16.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- 172.21.15.150:0/973557189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 msgr2=0x7f450c0a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 --2- 172.21.15.150:0/973557189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c0a0ba0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f450c06a610 tx=0x7f4508009250 comp rx=0 tx=0).stop 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- 172.21.15.150:0/973557189 shutdown_connections 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 --2- 172.21.15.150:0/973557189 >> v2:172.21.15.142:3300/0 conn(0x7f450c136a10 0x7f450c138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 --2- 172.21.15.150:0/973557189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c0a0ba0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 --2- 172.21.15.150:0/973557189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c0a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- 172.21.15.150:0/973557189 >> 172.21.15.150:0/973557189 conn(0x7f450c0a58a0 msgr2=0x7f450c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- 172.21.15.150:0/973557189 shutdown_connections 2024-09-18T10:14:16.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- 172.21.15.150:0/973557189 wait complete. 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 10 monclient: build_initial_monmap 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 10 monclient: monmap: 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:16.759 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 Processor -- start 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 1 -- start start 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.742+0000 7f45140e8700 10 monclient: init 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c12b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-c con 0x7f450c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-b con 0x7f450c0a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 0x7f450c12bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): picked mon.noname-a con 0x7f450c136a10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c09bd90 con 0x7f450c136a10 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f450c12ed60 con 0x7f450c0a07d0 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450c12ea80 con 0x7f450c09e6d0 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:16.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient(hunting): authenticate will time out at 1138.840234s 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511e84700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 msgr2=0x7f450c12b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511e84700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c12b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60234/0 (socket says 172.21.15.150:60234) 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 -- 172.21.15.150:0/1533368907 learned_addr learned my addr 172.21.15.150:0/1533368907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): get_auth_request con 0x7f450c0a07d0 auth_method 0 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 0x7f450c12bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 10 monclient(hunting): get_auth_request con 0x7f450c136a10 auth_method 0 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient(hunting): handle_auth_done global_id 15523 payload 824 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 -- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 msgr2=0x7f450c12b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c12b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 -- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 msgr2=0x7f450c12bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 0x7f450c12bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient: _finish_hunting 0 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 monclient: found mon.noname-b 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45080094f0 con 0x7f450c0a07d0 2024-09-18T10:14:16.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4512685700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 0x7f450c12bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 -- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 msgr2=0x7f450c12b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4511683700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f450c0a1220 tx=0x7f4508017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450801f070 con 0x7f450c0a07d0 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: dump: 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 5 monclient: authenticate success, global_id 15523 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4508000c90 con 0x7f450c0a07d0 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _renew_subs 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c12e7a0 con 0x7f450c0a07d0 2024-09-18T10:14:16.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450800be10 con 0x7f450c0a07d0 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4501ffb700 10 monclient: dump: 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f4503fff700 4 set_mon_vals no callback set 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _renew_subs 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.746+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c135730 con 0x7f450c0a07d0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4508023070 con 0x7f450c0a07d0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f45140e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f45140e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f45140e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c12eef0 con 0x7f450c0a07d0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f4501ffb700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 0x7f44f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(139..139 src has 58..139) v4 ==== 7664+0+0 (secure 0 0 0) 0x7f450804a160 con 0x7f450c0a07d0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f4511e84700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 0x7f44f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.750+0000 7f4511e84700 10 monclient: get_auth_request con 0x7f44f0035a70 auth_method 0 2024-09-18T10:14:16.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.754+0000 7f4511e84700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 0x7f44f0037f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f44fc006fd0 tx=0x7f44fc006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:16.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.758+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4508023550 con 0x7f450c0a07d0 2024-09-18T10:14:16.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.758+0000 7f4501ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:16.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.758+0000 7f4501ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:16.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.926+0000 7f45140e8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:16.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.926+0000 7f45140e8700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:16.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.926+0000 7f45140e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:16.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:16.926+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f450c063ed0 con 0x7f450c0a07d0 2024-09-18T10:14:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.030+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v140) v1 ==== 78+0+0 (secure 0 0 0) 0x7f450802a090 con 0x7f450c0a07d0 2024-09-18T10:14:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.030+0000 7f4501ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:17.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.030+0000 7f4501ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:17.098 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:14:17.098 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:14:17.099 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:14:17.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:14:17.101 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:14:17.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:14:17.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:17.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.138+0000 7f45140e8700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:17.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.138+0000 7f45140e8700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:17.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.138+0000 7f45140e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:17.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:17.138+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f450c0008d0 con 0x7f450c0a07d0 2024-09-18T10:14:17.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:14:17.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: build_initial_monmap 2024-09-18T10:14:17.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: monmap: 2024-09-18T10:14:17.202 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:17.202 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:17.201303+0000 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:17.201303+0000 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:17.203 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 Processor -- start 2024-09-18T10:14:17.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 -- start start 2024-09-18T10:14:17.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: init 2024-09-18T10:14:17.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f226c09ffe0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f226c09cce0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f226c0a11d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c09f930 con 0x7f226c09cce0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c09dd40 con 0x7f226c09ffe0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c069130 con 0x7f226c0a11d0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226bfff700 10 monclient(hunting): authenticate will time out at 1146.126975s 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60872/0 (socket says 172.21.15.142:60872) 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 learned_addr learned my addr 172.21.15.142:0/1870230411 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 10 monclient(hunting): get_auth_request con 0x7f226c0a11d0 auth_method 0 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 msgr2=0x7f226c13a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 msgr2=0x7f226c13a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:17.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.198+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 1 -- 172.21.15.142:0/1870230411 >> v1:172.21.15.142:6789/0 conn(0x7f226c09cce0 legacy=0x7f226c0a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226affd700 1 -- 172.21.15.142:0/1870230411 >> v1:172.21.15.150:6789/0 conn(0x7f226c09ffe0 legacy=0x7f226c0a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742638024 0 0) 0x7f226c09f930 con 0x7f226c09cce0 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): my global_id is 54330 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2254003570 con 0x7f226c09cce0 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846018750 0 0) 0x7f226c09dd40 con 0x7f226c09ffe0 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): my global_id is 15526 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f226c09f930 con 0x7f226c09ffe0 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771687827 0 0) 0x7f2254003570 con 0x7f226c09cce0 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 msgr2=0x7f226c13a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f226c13a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 >> v1:172.21.15.150:6789/0 conn(0x7f226c09ffe0 legacy=0x7f226c0a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: _finish_hunting 0 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c09d430 con 0x7f226c09cce0 2024-09-18T10:14:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f22600017f0 con 0x7f226c09cce0 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: dump: 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 >> v1:172.21.15.142:6789/0 conn(0x7f226c09cce0 legacy=0x7f226c0a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f2254003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): picked mon.c con 0x7f226c0a11d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2254004340 0x7f2254004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): picked mon.b con 0x7f2254004340 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 0x7f225400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): picked mon.a con 0x7f2254004f30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c069130 con 0x7f2254004f30 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225400bfa0 con 0x7f2254004340 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225400c0e0 con 0x7f226c0a11d0 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f2269ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 0x7f225400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): get_auth_request con 0x7f2254004f30 auth_method 0 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2254004340 0x7f2254004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): get_auth_request con 0x7f2254004340 auth_method 0 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 10 monclient(hunting): handle_auth_done global_id 54330 payload 293 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 msgr2=0x7f2254003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.202+0000 7f226affd700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f2254003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f2254003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2254004340 msgr2=0x7f2254004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2254004340 0x7f2254004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 monclient: found mon.a 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254003570 con 0x7f2254004f30 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 5 monclient: authenticate success, global_id 54330 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226affd700 10 monclient: get_auth_request con 0x7f226c0a11d0 auth_method 0 2024-09-18T10:14:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226affd700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f226c0a11d0 0x7f2254003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 msgr2=0x7f225400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 0x7f225400ba60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2264009650 tx=0x7f2264009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d2503837d872e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264017070 con 0x7f2254004f30 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: dump: 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400f040 con 0x7f2254004f30 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 1 -- 172.21.15.142:0/1870230411 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264007d80 con 0x7f2254004f30 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f2269ffb700 10 monclient: dump: 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 4 set_mon_vals no callback set 2024-09-18T10:14:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 10 monclient: shutdown 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 1 -- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 msgr2=0x7f225400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 1 --2- 172.21.15.142:0/1870230411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2254004f30 0x7f225400ba60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2264009650 tx=0x7f2264009110 comp rx=0 tx=0).stop 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226b7fe700 1 -- 172.21.15.142:0/1870230411 reap_dead start 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 1 -- 172.21.15.142:0/1870230411 shutdown_connections 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 1 -- 172.21.15.142:0/1870230411 >> 172.21.15.142:0/1870230411 conn(0x7f226c0a55f0 msgr2=0x7f226c0a59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.206+0000 7f226bfff700 1 -- 172.21.15.142:0/1870230411 shutdown_connections 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 -- 172.21.15.142:0/1870230411 wait complete. 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient: build_initial_monmap 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient: monmap: 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 Processor -- start 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 -- start start 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient: init 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f2254004f30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f226c08f410 0x7f226c137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f226c08f410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f226c137cd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0905e0 con 0x7f226c137cd0 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c08e8b0 con 0x7f226c08f410 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c091160 con 0x7f2254004f30 2024-09-18T10:14:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226bfff700 10 monclient(hunting): authenticate will time out at 1146.138852s 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58490/0 (socket says 172.21.15.142:58490) 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 -- 172.21.15.142:0/4232574034 learned_addr learned my addr 172.21.15.142:0/4232574034 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): get_auth_request con 0x7f226c137cd0 auth_method 0 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 10 monclient(hunting): get_auth_request con 0x7f2254004f30 auth_method 0 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f226c08f410 0x7f226c137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): get_auth_request con 0x7f226c08f410 auth_method 0 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 msgr2=0x7f226c098310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 msgr2=0x7f226c098310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient(hunting): handle_auth_done global_id 54333 payload 824 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 msgr2=0x7f226c098310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226affd700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f226c08f410 msgr2=0x7f226c137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f226c08f410 0x7f226c137790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.210+0000 7f226b7fe700 1 -- 172.21.15.142:0/4232574034 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c08efd0 con 0x7f226c137cd0 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226b7fe700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 msgr2=0x7f226c138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226b7fe700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2254005710 tx=0x7f2264017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264007d60 con 0x7f226c137cd0 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: dump: 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:17.231 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 5 monclient: authenticate success, global_id 54333 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _renew_subs 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226401f650 con 0x7f226c137cd0 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0a3a90 con 0x7f226c137cd0 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226401f7e0 con 0x7f226c137cd0 2024-09-18T10:14:17.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f2252ffd700 10 monclient: dump: 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _renew_subs 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f22697fa700 4 set_mon_vals no callback set 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c08cd30 con 0x7f226c137cd0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.214+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c072a80 con 0x7f226c137cd0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2264025030 con 0x7f226c137cd0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f2252ffd700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 0x7f2258037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(140..140 src has 58..140) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f226404aaf0 con 0x7f226c137cd0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f226affd700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 0x7f2258037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f226affd700 10 monclient: get_auth_request con 0x7f2258035ac0 auth_method 0 2024-09-18T10:14:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.218+0000 7f226affd700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 0x7f2258037f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f225c001520 tx=0x7f225c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.226+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f226401fbd0 con 0x7f226c137cd0 2024-09-18T10:14:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.226+0000 7f2252ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.226+0000 7f2252ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:17.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:17.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:17.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:17.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f226c0676a0 con 0x7f226c137cd0 2024-09-18T10:14:17.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f2252ffd700 1 -- 172.21.15.142:0/4232574034 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f226401d3c0 con 0x7f226c137cd0 2024-09-18T10:14:17.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f2252ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:17.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f2252ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:17.454 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:17.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 msgr2=0x7f2258037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 0x7f2258037f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f225c001520 tx=0x7f225c009040 comp rx=0 tx=0).stop 2024-09-18T10:14:17.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 10 monclient: shutdown 2024-09-18T10:14:17.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:17.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 msgr2=0x7f226c138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2254005710 tx=0x7f2264017000 comp rx=0 tx=0).stop 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 shutdown_connections 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2258035ac0 0x7f2258037f70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f226c137cd0 0x7f226c138100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f226c08f410 0x7f226c137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 --2- 172.21.15.142:0/4232574034 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2254004f30 0x7f226c098310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 >> 172.21.15.142:0/4232574034 conn(0x7f226c0a55f0 msgr2=0x7f226c09b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 shutdown_connections 2024-09-18T10:14:17.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:17.450+0000 7f226bfff700 1 -- 172.21.15.142:0/4232574034 wait complete. 2024-09-18T10:14:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.106+0000 7f4501ffb700 1 -- 172.21.15.150:0/1533368907 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v141) v1 ==== 78+0+0 (secure 0 0 0) 0x7f4508015020 con 0x7f450c0a07d0 2024-09-18T10:14:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.106+0000 7f4501ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:14:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.106+0000 7f4501ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 msgr2=0x7f44f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 0x7f44f0037f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f44fc006fd0 tx=0x7f44fc006e10 comp rx=0 tx=0).stop 2024-09-18T10:14:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 10 monclient: shutdown 2024-09-18T10:14:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 msgr2=0x7f450c12b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f450c0a1220 tx=0x7f4508017000 comp rx=0 tx=0).stop 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 shutdown_connections 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f44f0035a70 0x7f44f0037f20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f450c136a10 0x7f450c12bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f450c0a07d0 0x7f450c12b710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 --2- 172.21.15.150:0/1533368907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f450c09e6d0 0x7f450c12b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 >> 172.21.15.150:0/1533368907 conn(0x7f450c0a58a0 msgr2=0x7f450c136230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 shutdown_connections 2024-09-18T10:14:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.110+0000 7f45140e8700 1 -- 172.21.15.150:0/1533368907 wait complete. 2024-09-18T10:14:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create cool 2 2024-09-18T10:14:18.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: get_monmap_and_config 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: build_initial_monmap 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: monmap: 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:18.234221+0000 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:18.234221+0000 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:18.235 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:18.236 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:18.236 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:18.236 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 Processor -- start 2024-09-18T10:14:18.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 -- start start 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: init 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a1409c750 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a1409cb20 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f0a1413ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-b con 0x7f0a141387b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a14096e80 con 0x7f0a1409c750 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1409d600 con 0x7f0a1409cb20 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14097f70 con 0x7f0a141387b0 2024-09-18T10:14:18.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.230+0000 7f0a18ca9700 10 monclient(hunting): authenticate will time out at 1140.327792s 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0a1409cb20 0x7f0a1409cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37682/0 (socket says 172.21.15.150:37682) 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 1 -- 172.21.15.150:0/296249796 learned_addr learned my addr 172.21.15.150:0/296249796 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f0a1413ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): get_auth_request con 0x7f0a141387b0 auth_method 0 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 msgr2=0x7f0a1413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:18.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 msgr2=0x7f0a1413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:18.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f0a1413ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:18.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f0a1413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a137fe700 1 -- 172.21.15.150:0/296249796 >> v1:172.21.15.142:6789/0 conn(0x7f0a1409c750 legacy=0x7f0a14097860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 1 -- 172.21.15.150:0/296249796 >> v1:172.21.15.150:6789/0 conn(0x7f0a1409cb20 legacy=0x7f0a1409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544407966 0 0) 0x7f0a1409d600 con 0x7f0a1409cb20 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): my global_id is 15532 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09fc003570 con 0x7f0a1409cb20 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087739781 0 0) 0x7f0a14096e80 con 0x7f0a1409c750 2024-09-18T10:14:18.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): my global_id is 54336 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a1409d600 con 0x7f0a1409c750 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618897194 0 0) 0x7f09fc003570 con 0x7f0a1409cb20 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 msgr2=0x7f0a1413ab90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f0a1413ab90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 >> v1:172.21.15.142:6789/0 conn(0x7f0a1409c750 legacy=0x7f0a14097860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: _finish_hunting 0 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:18.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14096ba0 con 0x7f0a1409cb20 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a18ca9700 5 monclient: authenticate success, global_id 15532 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f0a080017a0 con 0x7f0a1409cb20 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a18ca9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: dump: 2024-09-18T10:14:18.257 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:14:18.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 >> v1:172.21.15.150:6789/0 conn(0x7f0a1409cb20 legacy=0x7f0a1409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f09fc004e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): picked mon.c con 0x7f0a141387b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc0053f0 0x7f09fc0037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): picked mon.a con 0x7f09fc0053f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 0x7f09fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): picked mon.b con 0x7f09fc003f10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14097f70 con 0x7f09fc0053f0 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00bfa0 con 0x7f09fc003f10 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc00c0e0 con 0x7f0a141387b0 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a127fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a18ca9700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 0x7f09fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): get_auth_request con 0x7f09fc003f10 auth_method 0 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc0053f0 0x7f09fc0037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient(hunting): handle_auth_done global_id 15532 payload 293 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 msgr2=0x7f09fc004e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a141387b0 0x7f09fc004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc0053f0 msgr2=0x7f09fc0037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc0053f0 0x7f09fc0037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 monclient: found mon.b 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc003570 con 0x7f09fc003f10 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 10 monclient: _finish_auth 0 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a13fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 10 monclient: get_auth_request con 0x7f09fc0053f0 auth_method 0 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.234+0000 7f0a12ffd700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc0053f0 0x7f09fc0037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 msgr2=0x7f09fc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:18.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a13fff700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 0x7f09fc00ba60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0073c0 tx=0x7f0a0c0096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e2aef87d0fc87f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c007e80 con 0x7f09fc003f10 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: dump: 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:18.263 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c009920 con 0x7f09fc003f10 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 1 -- 172.21.15.150:0/296249796 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c014650 con 0x7f09fc003f10 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 4 set_mon_vals no callback set 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a127fc700 10 monclient: dump: 2024-09-18T10:14:18.264 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 10 monclient: shutdown 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 msgr2=0x7f09fc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/296249796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09fc003f10 0x7f09fc00ba60 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0073c0 tx=0x7f0a0c0096f0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a13fff700 1 -- 172.21.15.150:0/296249796 reap_dead start 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/296249796 shutdown_connections 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/296249796 >> 172.21.15.150:0/296249796 conn(0x7f0a140a1d60 msgr2=0x7f0a140a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/296249796 shutdown_connections 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/296249796 wait complete. 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient: build_initial_monmap 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient: monmap: 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:18.266 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 Processor -- start 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 -- start start 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient: init 2024-09-18T10:14:18.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc003f10 0x7f0a14141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-a con 0x7f09fc003f10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-c con 0x7f0a14142150 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): picked mon.noname-b con 0x7f0a14119db0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1411be70 con 0x7f09fc003f10 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14142a80 con 0x7f0a14119db0 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1409bc50 con 0x7f0a14142150 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a18ca9700 10 monclient(hunting): authenticate will time out at 1140.337986s 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60250/0 (socket says 172.21.15.150:60250) 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 -- 172.21.15.150:0/49737730 learned_addr learned my addr 172.21.15.150:0/49737730 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): get_auth_request con 0x7f0a14119db0 auth_method 0 2024-09-18T10:14:18.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc003f10 0x7f0a14141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): get_auth_request con 0x7f09fc003f10 auth_method 0 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 10 monclient(hunting): get_auth_request con 0x7f0a14142150 auth_method 0 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:18.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient(hunting): handle_auth_done global_id 15535 payload 824 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 msgr2=0x7f0a14142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 msgr2=0x7f0a14142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 msgr2=0x7f0a14142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a12ffd700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc003f10 msgr2=0x7f0a14141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:18.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc003f10 0x7f0a14141c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 monclient: found mon.noname-b 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.242+0000 7f0a13fff700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1411e6b0 con 0x7f0a14119db0 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a13fff700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 msgr2=0x7f0a1411a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a13fff700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f09fc004770 tx=0x7f0a0c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c007e80 con 0x7f0a14119db0 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: dump: 2024-09-18T10:14:18.272 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 5 monclient: authenticate success, global_id 15535 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c01f820 con 0x7f0a14119db0 2024-09-18T10:14:18.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _renew_subs 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1411d420 con 0x7f0a14119db0 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c01fbc0 con 0x7f0a14119db0 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 10 monclient: dump: 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:18.274 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _renew_subs 2024-09-18T10:14:18.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a11ffb700 4 set_mon_vals no callback set 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1411d700 con 0x7f0a14119db0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0a0c01a030 con 0x7f0a14119db0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 0x7f0a00037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(141..141 src has 58..141) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f0a0c04abd0 con 0x7f0a14119db0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a137fe700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 0x7f0a00037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.246+0000 7f0a137fe700 10 monclient: get_auth_request con 0x7f0a00035ac0 auth_method 0 2024-09-18T10:14:18.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.250+0000 7f0a137fe700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 0x7f0a00037f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a04001490 tx=0x7f0a04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:18.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.258+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:18.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.258+0000 7f09fb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:18.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.258+0000 7f09fb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:18.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.422+0000 7f0a18ca9700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:18.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.422+0000 7f0a18ca9700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:18.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.422+0000 7f0a18ca9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:18.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:18.422+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.126+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' created v142) v1 ==== 111+0+0 (secure 0 0 0) 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.126+0000 7f09fb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:19.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.126+0000 7f09fb7fe700 10 monclient: _finish_command 2 = system:0 pool 'cool' created 2024-09-18T10:14:19.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.230+0000 7f0a18ca9700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:19.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.230+0000 7f0a18ca9700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:19.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.230+0000 7f0a18ca9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.230+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cool", "pg_num": 2} v 0) v1 -- 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:19.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.234+0000 7f09fb7fe700 1 -- 172.21.15.150:0/49737730 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}]=0 pool 'cool' already exists v142) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0a1411da40 con 0x7f0a14119db0 2024-09-18T10:14:19.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.234+0000 7f09fb7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cool", "pg_num": 2}] 2024-09-18T10:14:19.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.234+0000 7f09fb7fe700 10 monclient: _finish_command 3 = system:0 pool 'cool' already exists 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 msgr2=0x7f0a00037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 0x7f0a00037f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a04001490 tx=0x7f0a04009040 comp rx=0 tx=0).stop 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 10 monclient: shutdown 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 msgr2=0x7f0a1411a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f09fc004770 tx=0x7f0a0c017000 comp rx=0 tx=0).stop 2024-09-18T10:14:19.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 shutdown_connections 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0a00035ac0 0x7f0a00037f70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a14119db0 0x7f0a1411a250 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0a14142150 0x7f0a14142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 --2- 172.21.15.150:0/49737730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09fc003f10 0x7f0a14141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 >> 172.21.15.150:0/49737730 conn(0x7f0a140a1d60 msgr2=0x7f0a1409a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 shutdown_connections 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.238+0000 7f0a18ca9700 1 -- 172.21.15.150:0/49737730 wait complete. 2024-09-18T10:14:19.242 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'cool' already exists 2024-09-18T10:14:19.251 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool create cold 2 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: get_monmap_and_config 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: build_initial_monmap 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: monmap: 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:19.356 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:19.353300+0000 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:19.353300+0000 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 Processor -- start 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 -- start start 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: init 2024-09-18T10:14:19.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d400a0120 0x7f4d4009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-c con 0x7f4d400a0120 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 0x7f4d400a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-a con 0x7f4d400a04f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 0x7f4d40138d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-b con 0x7f4d401369a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d400a21d0 con 0x7f4d400a04f0 2024-09-18T10:14:19.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4009ddc0 con 0x7f4d400a0120 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4009dae0 con 0x7f4d401369a0 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d46b76700 10 monclient(hunting): authenticate will time out at 1141.446758s 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d400a0120 0x7f4d4009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d400a0120 0x7f4d4009e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44758/0 (socket says 172.21.15.150:44758) 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 -- 172.21.15.150:0/1110727560 learned_addr learned my addr 172.21.15.150:0/1110727560 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): get_auth_request con 0x7f4d400a0120 auth_method 0 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:19.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d45113700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 0x7f4d40138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d3ffff700 1 --2- 172.21.15.150:0/1110727560 >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 0x7f4d400a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient(hunting): handle_auth_done global_id 15538 payload 274 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 -- 172.21.15.150:0/1110727560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 msgr2=0x7f4d40138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 0x7f4d40138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 -- 172.21.15.150:0/1110727560 >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 msgr2=0x7f4d400a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 --2- 172.21.15.150:0/1110727560 >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 0x7f4d400a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient: _finish_hunting 0 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 monclient: found mon.noname-c 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d44912700 1 -- 172.21.15.150:0/1110727560 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400a1590 con 0x7f4d400a0120 2024-09-18T10:14:19.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d45113700 10 monclient: get_auth_request con 0x7f4d401369a0 auth_method 0 2024-09-18T10:14:19.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d3ffff700 10 monclient: get_auth_request con 0x7f4d400a04f0 auth_method 0 2024-09-18T10:14:19.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d45113700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 0x7f4d40138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:19.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.350+0000 7f4d3ffff700 1 --2- 172.21.15.150:0/1110727560 >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 0x7f4d400a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:19.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d44912700 1 -- 172.21.15.150:0/1110727560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a0120 msgr2=0x7f4d4009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:19.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d44912700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a0120 0x7f4d4009e7a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4d40068ec0 tx=0x7f4d28009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec232c88c1f87d4f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:19.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 1 -- 172.21.15.150:0/1110727560 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800ba70 con 0x7f4d400a0120 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: dump: 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:19.366 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 5 monclient: authenticate success, global_id 15538 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 1 -- 172.21.15.150:0/1110727560 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2800bbd0 con 0x7f4d400a0120 2024-09-18T10:14:19.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 1 -- 172.21.15.150:0/1110727560 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28011420 con 0x7f4d400a0120 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d3f7fe700 10 monclient: dump: 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:19.368 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 4 set_mon_vals no callback set 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 10 monclient: shutdown 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 -- 172.21.15.150:0/1110727560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a0120 msgr2=0x7f4d4009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a0120 0x7f4d4009e7a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4d40068ec0 tx=0x7f4d28009250 comp rx=0 tx=0).stop 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 -- 172.21.15.150:0/1110727560 shutdown_connections 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d401369a0 0x7f4d40138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 --2- 172.21.15.150:0/1110727560 >> v2:172.21.15.142:3300/0 conn(0x7f4d400a04f0 0x7f4d400a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 --2- 172.21.15.150:0/1110727560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a0120 0x7f4d4009e7a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 -- 172.21.15.150:0/1110727560 >> 172.21.15.150:0/1110727560 conn(0x7f4d400a5830 msgr2=0x7f4d400a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:19.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 -- 172.21.15.150:0/1110727560 shutdown_connections 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.354+0000 7f4d46b76700 1 -- 172.21.15.150:0/1110727560 wait complete. 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient: build_initial_monmap 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient: monmap: 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 Processor -- start 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 -- start start 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient: init 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d400a0120 0x7f4d4012c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-c con 0x7f4d400a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-b con 0x7f4d400a04f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d401369a0 0x7f4d4012cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): picked mon.noname-a con 0x7f4d401369a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d40135cc0 con 0x7f4d401369a0 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4012e3f0 con 0x7f4d400a04f0 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4009d720 con 0x7f4d400a0120 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d46b76700 10 monclient(hunting): authenticate will time out at 1141.455313s 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44774/0 (socket says 172.21.15.150:44774) 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 -- 172.21.15.150:0/716169598 learned_addr learned my addr 172.21.15.150:0/716169598 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): get_auth_request con 0x7f4d400a04f0 auth_method 0 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:19.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d44912700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d400a0120 0x7f4d4012c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d45113700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d401369a0 0x7f4d4012cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d44912700 10 monclient(hunting): get_auth_request con 0x7f4d400a0120 auth_method 0 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d44912700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d44912700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d44912700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d45113700 10 monclient(hunting): get_auth_request con 0x7f4d401369a0 auth_method 0 2024-09-18T10:14:19.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d45113700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d45113700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d45113700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient(hunting): handle_auth_done global_id 15541 payload 824 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 -- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d400a0120 msgr2=0x7f4d4012c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d400a0120 0x7f4d4012c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 -- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d401369a0 msgr2=0x7f4d4012cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:19.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d401369a0 0x7f4d4012cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:19.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:19.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 monclient: found mon.noname-b 2024-09-18T10:14:19.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:19.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.358+0000 7f4d3ffff700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280094f0 con 0x7f4d400a04f0 2024-09-18T10:14:19.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3ffff700 1 -- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 msgr2=0x7f4d4012c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:19.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3ffff700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c8b0 tx=0x7f4d30008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:19.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30011870 con 0x7f4d400a04f0 2024-09-18T10:14:19.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: dump: 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:19.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d300119d0 con 0x7f4d400a04f0 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 5 monclient: authenticate success, global_id 15541 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 10 monclient: _renew_subs 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4009bfe0 con 0x7f4d400a04f0 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30011d50 con 0x7f4d400a04f0 2024-09-18T10:14:19.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3cff9700 10 monclient: dump: 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.380 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d3effd700 4 set_mon_vals no callback set 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 10 monclient: _renew_subs 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.362+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d40136510 con 0x7f4d400a04f0 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4d30010560 con 0x7f4d400a04f0 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d46b76700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d46b76700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:19.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d46b76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4006e3e0 con 0x7f4d400a04f0 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d3cff9700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 0x7f4d20037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(142..142 src has 58..142) v4 ==== 7675+0+0 (secure 0 0 0) 0x7f4d30049360 con 0x7f4d400a04f0 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d44912700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 0x7f4d20037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d44912700 10 monclient: get_auth_request con 0x7f4d20035ac0 auth_method 0 2024-09-18T10:14:19.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.366+0000 7f4d44912700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 0x7f4d20037f70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4d40068ec0 tx=0x7f4d28017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:19.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.370+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4d30018900 con 0x7f4d400a04f0 2024-09-18T10:14:19.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.370+0000 7f4d3cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:19.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.370+0000 7f4d3cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:19.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.538+0000 7f4d46b76700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:19.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.538+0000 7f4d46b76700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:19.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.538+0000 7f4d46b76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:19.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:19.538+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f4d400a1c90 con 0x7f4d400a04f0 2024-09-18T10:14:20.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.130+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' created v143) v1 ==== 111+0+0 (secure 0 0 0) 0x7f4d3001e070 con 0x7f4d400a04f0 2024-09-18T10:14:20.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.130+0000 7f4d3cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:20.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.130+0000 7f4d3cff9700 10 monclient: _finish_command 2 = system:0 pool 'cold' created 2024-09-18T10:14:20.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d46b76700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:20.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d46b76700 10 monclient: _send_command 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:20.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d46b76700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cold", "pg_num": 2} v 0) v1 -- 0x7f4d40063ed0 con 0x7f4d400a04f0 2024-09-18T10:14:20.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d3cff9700 1 -- 172.21.15.150:0/716169598 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}]=0 pool 'cold' already exists v143) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4d30018350 con 0x7f4d400a04f0 2024-09-18T10:14:20.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d3cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool create", "pool": "cold", "pg_num": 2}] 2024-09-18T10:14:20.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.238+0000 7f4d3cff9700 10 monclient: _finish_command 3 = system:0 pool 'cold' already exists 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 msgr2=0x7f4d20037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 0x7f4d20037f70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4d40068ec0 tx=0x7f4d28017000 comp rx=0 tx=0).stop 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 10 monclient: shutdown 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 msgr2=0x7f4d4012c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000c8b0 tx=0x7f4d30008040 comp rx=0 tx=0).stop 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 shutdown_connections 2024-09-18T10:14:20.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d20035ac0 0x7f4d20037f70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d401369a0 0x7f4d4012cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d400a04f0 0x7f4d4012c610 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 --2- 172.21.15.150:0/716169598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d400a0120 0x7f4d4012c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 >> 172.21.15.150:0/716169598 conn(0x7f4d400a5830 msgr2=0x7f4d4009a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 shutdown_connections 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.242+0000 7f4d46b76700 1 -- 172.21.15.150:0/716169598 wait complete. 2024-09-18T10:14:20.247 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'cold' already exists 2024-09-18T10:14:20.255 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool set cold size 2 2024-09-18T10:14:20.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: get_monmap_and_config 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: build_initial_monmap 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: monmap: 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:20.361208+0000 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:20.361208+0000 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:20.362 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:20.363 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:20.363 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 Processor -- start 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 -- start start 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: init 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 0x7fed4409c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-b con 0x7fed4406f160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed4409cc90 0x7fed4409d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-c con 0x7fed4409cc90 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed44132560 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4409d5d0 con 0x7fed44132560 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4409e170 con 0x7fed4409cc90 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed44096fa0 con 0x7fed4406f160 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed4c0fe700 10 monclient(hunting): authenticate will time out at 1142.454883s 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed4409cc90 0x7fed4409d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fed4409cc90 0x7fed4409d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44782/0 (socket says 172.21.15.150:44782) 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 -- 172.21.15.150:0/2782316311 learned_addr learned my addr 172.21.15.150:0/2782316311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): get_auth_request con 0x7fed4409cc90 auth_method 0 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:20.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49e9a700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 0x7fed4409c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient(hunting): handle_auth_done global_id 15544 payload 274 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 -- 172.21.15.150:0/2782316311 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 msgr2=0x7fed4409c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 0x7fed4409c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 -- 172.21.15.150:0/2782316311 >> v1:172.21.15.142:6789/0 conn(0x7fed44132560 legacy=0x7fed44132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient: _finish_hunting 0 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 monclient: found mon.noname-c 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49699700 1 -- 172.21.15.150:0/2782316311 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4406f530 con 0x7fed4409cc90 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49e9a700 10 monclient: get_auth_request con 0x7fed4406f160 auth_method 0 2024-09-18T10:14:20.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.358+0000 7fed49e9a700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 0x7fed4409c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed49699700 1 -- 172.21.15.150:0/2782316311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 msgr2=0x7fed4409d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed49699700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed4409d090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fed400099b0 tx=0x7fed40009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8387309db2f6f87c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 1 -- 172.21.15.150:0/2782316311 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4000bb50 con 0x7fed4409cc90 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: dump: 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:20.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: _finish_auth 0 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 1 -- 172.21.15.150:0/2782316311 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4000bcb0 con 0x7fed4409cc90 2024-09-18T10:14:20.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 5 monclient: authenticate success, global_id 15544 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 1 -- 172.21.15.150:0/2782316311 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40011650 con 0x7fed4409cc90 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 4 set_mon_vals no callback set 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed48e98700 10 monclient: dump: 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.377 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 10 monclient: shutdown 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 -- 172.21.15.150:0/2782316311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 msgr2=0x7fed4409d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed4409d090 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fed400099b0 tx=0x7fed40009250 comp rx=0 tx=0).stop 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 -- 172.21.15.150:0/2782316311 shutdown_connections 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed4409d090 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/2782316311 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed4406f160 0x7fed4409c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 -- 172.21.15.150:0/2782316311 >> 172.21.15.150:0/2782316311 conn(0x7fed440681c0 msgr2=0x7fed440a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 -- 172.21.15.150:0/2782316311 shutdown_connections 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.362+0000 7fed4c0fe700 1 -- 172.21.15.150:0/2782316311 wait complete. 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient: build_initial_monmap 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient: monmap: 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.379 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 Processor -- start 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 -- start start 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient: init 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed4406f160 0x7fed44141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-a con 0x7fed4406f160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-b con 0x7fed4409cc90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): picked mon.noname-c con 0x7fed44132560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4409bd10 con 0x7fed4406f160 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4409df60 con 0x7fed4409cc90 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed44121740 con 0x7fed44132560 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4c0fe700 10 monclient(hunting): authenticate will time out at 1142.462082s 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44794/0 (socket says 172.21.15.150:44794) 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 -- 172.21.15.150:0/617554971 learned_addr learned my addr 172.21.15.150:0/617554971 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:20.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): get_auth_request con 0x7fed4409cc90 auth_method 0 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed4406f160 0x7fed44141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): get_auth_request con 0x7fed4406f160 auth_method 0 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 10 monclient(hunting): get_auth_request con 0x7fed44132560 auth_method 0 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 msgr2=0x7fed441424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 msgr2=0x7fed441424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49e9a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed4a69b700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient(hunting): handle_auth_done global_id 15547 payload 824 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 msgr2=0x7fed441424d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed4406f160 msgr2=0x7fed44141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed4406f160 0x7fed44141a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient: _finish_hunting 0 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 monclient: found mon.noname-b 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.366+0000 7fed49699700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed400094f0 con 0x7fed4409cc90 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed49699700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 msgr2=0x7fed44141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed49699700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fed40009290 tx=0x7fed40016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4001b680 con 0x7fed4409cc90 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: dump: 2024-09-18T10:14:20.384 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4001b7e0 con 0x7fed4409cc90 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 5 monclient: authenticate success, global_id 15547 2024-09-18T10:14:20.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _renew_subs 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed441214c0 con 0x7fed4409cc90 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4001bb80 con 0x7fed4409cc90 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 10 monclient: dump: 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.386 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed3bfff700 4 set_mon_vals no callback set 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _renew_subs 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed44123950 con 0x7fed4409cc90 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fed4001c450 con 0x7fed4409cc90 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed44099c60 con 0x7fed4409cc90 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 0x7fed28037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.370+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(143..143 src has 58..143) v4 ==== 8018+0+0 (secure 0 0 0) 0x7fed4004ada0 con 0x7fed4409cc90 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.374+0000 7fed49e9a700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 0x7fed28037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.374+0000 7fed49e9a700 10 monclient: get_auth_request con 0x7fed28035ac0 auth_method 0 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.374+0000 7fed49e9a700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 0x7fed28037f70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed34006fd0 tx=0x7fed34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.382+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fed400188d0 con 0x7fed4409cc90 2024-09-18T10:14:20.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.382+0000 7fed39ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.382+0000 7fed39ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:20.469 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:20.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.542+0000 7fed4c0fe700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:20.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.542+0000 7fed4c0fe700 10 monclient: _send_command 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:20.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.542+0000 7fed4c0fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:20.549 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:20.542+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fed44063ed0 con 0x7fed4409cc90 2024-09-18T10:14:20.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: get_monmap_and_config 2024-09-18T10:14:20.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: build_initial_monmap 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: monmap: 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:20.580638+0000 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:20.580638+0000 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:20.582 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:20.583 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:20.583 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:20.583 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:20.583 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 Processor -- start 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 -- start start 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: init 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f130009d870 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:20.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13000954b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1300135680 0x7f1300137a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-a con 0x7f1300135680 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1300069c70 con 0x7f130009d870 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000958b0 con 0x7f13000954b0 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130006f9d0 con 0x7f1300135680 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f13053dc700 10 monclient(hunting): authenticate will time out at 1149.506110s 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12feffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f130009d870 0x7f13000989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33572/0 (socket says 172.21.15.142:33572) 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12feffd700 1 -- 172.21.15.142:0/744559885 learned_addr learned my addr 172.21.15.142:0/744559885 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:20.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 1 --2- 172.21.15.142:0/744559885 >> v2:172.21.15.142:3300/0 conn(0x7f1300135680 0x7f1300137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): get_auth_request con 0x7f1300135680 auth_method 0 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient(hunting): handle_auth_done global_id 54348 payload 274 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 1 -- 172.21.15.142:0/744559885 >> v1:172.21.15.150:6789/0 conn(0x7f13000954b0 legacy=0x7f13000990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:20.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 1 -- 172.21.15.142:0/744559885 >> v1:172.21.15.142:6789/0 conn(0x7f130009d870 legacy=0x7f13000989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:20.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:20.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 1 monclient: found mon.noname-a 2024-09-18T10:14:20.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:20.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.578+0000 7f12ff7fe700 1 -- 172.21.15.142:0/744559885 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130009dc40 con 0x7f1300135680 2024-09-18T10:14:20.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12ff7fe700 1 -- 172.21.15.142:0/744559885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1300135680 msgr2=0x7f1300137a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:20.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12ff7fe700 1 --2- 172.21.15.142:0/744559885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1300135680 0x7f1300137a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009e60 tx=0x7f12f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d4b3d0dd82519f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 1 -- 172.21.15.142:0/744559885 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800e070 con 0x7f1300135680 2024-09-18T10:14:20.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: dump: 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.592 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 1 -- 172.21.15.142:0/744559885 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f8012430 con 0x7f1300135680 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 5 monclient: authenticate success, global_id 54348 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 1 -- 172.21.15.142:0/744559885 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f80127d0 con 0x7f1300135680 2024-09-18T10:14:20.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 4 set_mon_vals no callback set 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f12fe7fc700 10 monclient: dump: 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 10 monclient: shutdown 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 -- 172.21.15.142:0/744559885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1300135680 msgr2=0x7f1300137a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 --2- 172.21.15.142:0/744559885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1300135680 0x7f1300137a60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009e60 tx=0x7f12f8009780 comp rx=0 tx=0).stop 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 -- 172.21.15.142:0/744559885 shutdown_connections 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 --2- 172.21.15.142:0/744559885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1300135680 0x7f1300137a60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 -- 172.21.15.142:0/744559885 >> 172.21.15.142:0/744559885 conn(0x7f13000a14e0 msgr2=0x7f13000a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 -- 172.21.15.142:0/744559885 shutdown_connections 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.582+0000 7f13053dc700 1 -- 172.21.15.142:0/744559885 wait complete. 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient: build_initial_monmap 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient: monmap: 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:20.597 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:20.598 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:20.598 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 Processor -- start 2024-09-18T10:14:20.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 -- start start 2024-09-18T10:14:20.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient: init 2024-09-18T10:14:20.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:20.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-a con 0x7f13000954b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f130009d870 0x7f130006df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-b con 0x7f130009d870 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): picked mon.noname-c con 0x7f1300135680 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1300097ee0 con 0x7f13000954b0 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13000971f0 con 0x7f130009d870 2024-09-18T10:14:20.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001272b0 con 0x7f1300135680 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f13053dc700 10 monclient(hunting): authenticate will time out at 1149.515624s 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58536/0 (socket says 172.21.15.142:58536) 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 1 -- 172.21.15.142:0/2096057220 learned_addr learned my addr 172.21.15.142:0/2096057220 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): get_auth_request con 0x7f13000954b0 auth_method 0 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:20.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 10 monclient(hunting): get_auth_request con 0x7f1300135680 auth_method 0 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 msgr2=0x7f130006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 msgr2=0x7f130006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12ff7fe700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.586+0000 7f12f7fff700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f130009d870 0x7f130006df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12ff7fe700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f7fff700 10 monclient(hunting): get_auth_request con 0x7f130009d870 auth_method 0 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:20.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 10 monclient(hunting): handle_auth_done global_id 54351 payload 824 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 msgr2=0x7f130006e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f130009d870 msgr2=0x7f130006df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f130009d870 0x7f130006df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 monclient: found mon.noname-a 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 -- 172.21.15.142:0/2096057220 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8009a20 con 0x7f13000954b0 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 msgr2=0x7f130009c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:20.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12feffd700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c5e0 tx=0x7f12e8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8005d40 con 0x7f13000954b0 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: dump: 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 5 monclient: authenticate success, global_id 54351 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 10 monclient: _renew_subs 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e80070d0 con 0x7f13000954b0 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1300127b10 con 0x7f13000954b0 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8007490 con 0x7f13000954b0 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12fdffb700 4 set_mon_vals no callback set 2024-09-18T10:14:20.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f12f6ffd700 10 monclient: dump: 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 10 monclient: _renew_subs 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.590+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130006e9d0 con 0x7f13000954b0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f12e801b3b0 con 0x7f13000954b0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f6ffd700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 0x7f12e4037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(143..143 src has 58..143) v4 ==== 8018+0+0 (secure 0 0 0) 0x7f12e80486e0 con 0x7f13000954b0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f13053dc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f13053dc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f13053dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f7fff700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 0x7f12e4037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13000696a0 con 0x7f13000954b0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f7fff700 10 monclient: get_auth_request con 0x7f12e4035af0 auth_method 0 2024-09-18T10:14:20.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.594+0000 7f12f7fff700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 0x7f12e4037fa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f12f0006fd0 tx=0x7f12f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:20.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.602+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f12e8017020 con 0x7f13000954b0 2024-09-18T10:14:20.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.602+0000 7f12f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:20.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.602+0000 7f12f6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:20.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.818+0000 7f13053dc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:20.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.818+0000 7f13053dc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:20.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.818+0000 7f13053dc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:20.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.818+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1300096e10 con 0x7f13000954b0 2024-09-18T10:14:20.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f12f6ffd700 1 -- 172.21.15.142:0/2096057220 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f12e800f800 con 0x7f13000954b0 2024-09-18T10:14:20.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f12f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:20.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f12f6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:20.825 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:20.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 msgr2=0x7f12e4037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 0x7f12e4037fa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f12f0006fd0 tx=0x7f12f0008040 comp rx=0 tx=0).stop 2024-09-18T10:14:20.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 10 monclient: shutdown 2024-09-18T10:14:20.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:20.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 msgr2=0x7f130009c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:20.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f12e800c5e0 tx=0x7f12e8004a80 comp rx=0 tx=0).stop 2024-09-18T10:14:20.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 shutdown_connections 2024-09-18T10:14:20.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12e4035af0 0x7f12e4037fa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1300135680 0x7f130006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f130009d870 0x7f130006df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 --2- 172.21.15.142:0/2096057220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13000954b0 0x7f130009c990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:20.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 >> 172.21.15.142:0/2096057220 conn(0x7f13000a14e0 msgr2=0x7f130009f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:20.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 shutdown_connections 2024-09-18T10:14:20.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:20.822+0000 7f13053dc700 1 -- 172.21.15.142:0/2096057220 wait complete. 2024-09-18T10:14:21.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.134+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v144) v1 ==== 124+0+0 (secure 0 0 0) 0x7fed40018350 con 0x7fed4409cc90 2024-09-18T10:14:21.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.134+0000 7fed39ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:21.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.134+0000 7fed39ffb700 10 monclient: _finish_command 2 = system:0 set pool 12 size to 2 2024-09-18T10:14:21.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.234+0000 7fed4c0fe700 10 start_mon_command cmd=[{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:21.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.238+0000 7fed4c0fe700 10 monclient: _send_command 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:21.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.238+0000 7fed4c0fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:21.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:21.238+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"} v 0) v1 -- 0x7fed44063ed0 con 0x7fed4409cc90 2024-09-18T10:14:22.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.146+0000 7fed39ffb700 1 -- 172.21.15.150:0/617554971 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}]=0 set pool 12 size to 2 v145) v1 ==== 124+0+0 (secure 0 0 0) 0x7fed400188d0 con 0x7fed4409cc90 2024-09-18T10:14:22.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.146+0000 7fed39ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool set", "pool": "cold", "var": "size", "val": "2"}] 2024-09-18T10:14:22.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.146+0000 7fed39ffb700 10 monclient: _finish_command 3 = system:0 set pool 12 size to 2 2024-09-18T10:14:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 msgr2=0x7fed28037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 0x7fed28037f70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed34006fd0 tx=0x7fed34008040 comp rx=0 tx=0).stop 2024-09-18T10:14:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 10 monclient: shutdown 2024-09-18T10:14:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:22.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 msgr2=0x7fed44141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fed40009290 tx=0x7fed40016000 comp rx=0 tx=0).stop 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 shutdown_connections 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fed28035ac0 0x7fed28037f70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed44132560 0x7fed441424d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed4409cc90 0x7fed44141f90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 --2- 172.21.15.150:0/617554971 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed4406f160 0x7fed44141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 >> 172.21.15.150:0/617554971 conn(0x7fed440681c0 msgr2=0x7fed4409a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 shutdown_connections 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.150+0000 7fed4c0fe700 1 -- 172.21.15.150:0/617554971 wait complete. 2024-09-18T10:14:22.155 INFO:tasks.workunit.client.0.smithi150.stderr:set pool 12 size to 2 2024-09-18T10:14:22.167 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T10:14:22.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 10 monclient: get_monmap_and_config 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 10 monclient: build_initial_monmap 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 10 monclient: monmap: 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:22.274184+0000 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:22.274184+0000 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:22.275 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 1 Processor -- start 2024-09-18T10:14:22.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 1 -- start start 2024-09-18T10:14:22.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.270+0000 7fd2d7ba9700 10 monclient: init 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2d0130240 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2d0130610 0x7fd2d006d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2d0130610 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2d0135570 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:22.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0097800 con 0x7fd2d0135570 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0097520 con 0x7fd2d0130240 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d006e1d0 con 0x7fd2d0130610 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d7ba9700 10 monclient(hunting): authenticate will time out at 1144.369116s 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5945700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd2d0130240 0x7fd2d009f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:54644/0 (socket says 172.21.15.150:54644) 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5945700 1 -- 172.21.15.150:0/3853556239 learned_addr learned my addr 172.21.15.150:0/3853556239 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 1 --2- 172.21.15.150:0/3853556239 >> v2:172.21.15.150:3300/0 conn(0x7fd2d0130610 0x7fd2d006d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): get_auth_request con 0x7fd2d0130610 auth_method 0 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:22.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:22.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5945700 1 -- 172.21.15.150:0/3853556239 >> v1:172.21.15.150:6789/0 conn(0x7fd2d0130240 legacy=0x7fd2d009f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d6146700 1 -- 172.21.15.150:0/3853556239 >> v1:172.21.15.142:6789/0 conn(0x7fd2d0135570 legacy=0x7fd2d0137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 10 monclient(hunting): handle_auth_done global_id 15553 payload 274 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.274+0000 7fd2d4943700 1 -- 172.21.15.150:0/3853556239 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330839303 0 0) 0x7fd2d0097800 con 0x7fd2d0135570 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 -- 172.21.15.150:0/3853556239 >> v1:172.21.15.150:6789/0 conn(0x7fd2d0130240 legacy=0x7fd2d009f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 -- 172.21.15.150:0/3853556239 >> v1:172.21.15.142:6789/0 conn(0x7fd2d0135570 legacy=0x7fd2d0137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 10 monclient: _finish_hunting 0 2024-09-18T10:14:22.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 monclient: found mon.noname-c 2024-09-18T10:14:22.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:22.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 -- 172.21.15.150:0/3853556239 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0097240 con 0x7fd2d0130610 2024-09-18T10:14:22.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 -- 172.21.15.150:0/3853556239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 msgr2=0x7fd2d006d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d5144700 1 --2- 172.21.15.150:0/3853556239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d006d690 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00099e0 tx=0x7fd2c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd4bdded2a9b757 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 1 -- 172.21.15.150:0/3853556239 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c000bba0 con 0x7fd2d0130610 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: dump: 2024-09-18T10:14:22.286 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:22.287 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: _finish_auth 0 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 1 -- 172.21.15.150:0/3853556239 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c000bd20 con 0x7fd2d0130610 2024-09-18T10:14:22.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d7ba9700 5 monclient: authenticate success, global_id 15553 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d7ba9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 1 -- 172.21.15.150:0/3853556239 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0011670 con 0x7fd2d0130610 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d7ba9700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d7ba9700 4 set_mon_vals no callback set 2024-09-18T10:14:22.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.278+0000 7fd2d4943700 10 monclient: dump: 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:22.290 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: shutdown 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/3853556239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 msgr2=0x7fd2d006d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/3853556239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d006d690 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00099e0 tx=0x7fd2c00091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/3853556239 shutdown_connections 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/3853556239 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d006d690 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/3853556239 >> 172.21.15.150:0/3853556239 conn(0x7fd2d00a3eb0 msgr2=0x7fd2d00710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:22.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/3853556239 shutdown_connections 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/3853556239 wait complete. 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: build_initial_monmap 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: monmap: 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:22.292 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:22.293 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 Processor -- start 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- start start 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: init 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 0x7fd2d012d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2d0130240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-b con 0x7fd2d0130610 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:22.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2d0135570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d009b340 con 0x7fd2d0130240 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d009c190 con 0x7fd2d0130610 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d012d9b0 con 0x7fd2d0135570 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d7ba9700 10 monclient(hunting): authenticate will time out at 1144.379428s 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44802/0 (socket says 172.21.15.150:44802) 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 1 -- 172.21.15.150:0/2694269330 learned_addr learned my addr 172.21.15.150:0/2694269330 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): get_auth_request con 0x7fd2d0130610 auth_method 0 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:22.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5144700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5945700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 0x7fd2d012d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:22.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d6146700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5945700 10 monclient(hunting): get_auth_request con 0x7fd2d0130240 auth_method 0 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5945700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5945700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d5945700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d6146700 10 monclient(hunting): get_auth_request con 0x7fd2d0135570 auth_method 0 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d6146700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d6146700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:22.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.282+0000 7fd2d6146700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 10 monclient(hunting): handle_auth_done global_id 15559 payload 824 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d6146700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 msgr2=0x7fd2d00699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d6146700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 msgr2=0x7fd2d00699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 msgr2=0x7fd2d00699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d6146700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d6146700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 msgr2=0x7fd2d012d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 0x7fd2d012d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 10 monclient: _finish_hunting 0 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 monclient: found mon.noname-b 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:22.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0009420 con 0x7fd2d0130610 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5945700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 0x7fd2d012d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 msgr2=0x7fd2d009df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d5144700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009fa0 tx=0x7fd2c0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c001f070 con 0x7fd2d0130610 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: dump: 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.303 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c0011630 con 0x7fd2d0130610 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 5 monclient: authenticate success, global_id 15559 2024-09-18T10:14:22.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 10 monclient: _renew_subs 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d009caa0 con 0x7fd2d0130610 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c001a670 con 0x7fd2d0130610 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c7fff700 4 set_mon_vals no callback set 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2c5ffb700 10 monclient: dump: 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:22.305 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 10 monclient: _renew_subs 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.286+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d00a3020 con 0x7fd2d0130610 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d7ba9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d7ba9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d7ba9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d0131010 con 0x7fd2d0130610 2024-09-18T10:14:22.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd2c0023070 con 0x7fd2d0130610 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2c5ffb700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 0x7fd2b4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(145..145 src has 58..145) v4 ==== 8018+0+0 (secure 0 0 0) 0x7fd2c004a600 con 0x7fd2d0130610 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d5945700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 0x7fd2b4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.290+0000 7fd2d5945700 10 monclient: get_auth_request con 0x7fd2b4035ac0 auth_method 0 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.294+0000 7fd2d5945700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 0x7fd2b4037f70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0058d0 tx=0x7fd2cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.298+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd2c0006c90 con 0x7fd2d0130610 2024-09-18T10:14:22.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.298+0000 7fd2c5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:22.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.298+0000 7fd2c5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:22.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.474+0000 7fd2d7ba9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:22.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.474+0000 7fd2d7ba9700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:22.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.474+0000 7fd2d7ba9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:22.477 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:22.474+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fd2d00650e0 con 0x7fd2d0130610 2024-09-18T10:14:23.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.154+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v146) v1 ==== 82+0+0 (secure 0 0 0) 0x7fd2c0015020 con 0x7fd2d0130610 2024-09-18T10:14:23.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.154+0000 7fd2c5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:23.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.154+0000 7fd2c5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:23.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.266+0000 7fd2d7ba9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:23.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.266+0000 7fd2d7ba9700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:23.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.266+0000 7fd2d7ba9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.272 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.266+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7fd2d0063ed0 con 0x7fd2d0130610 2024-09-18T10:14:23.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.522+0000 7fd2c5ffb700 1 -- 172.21.15.150:0/2694269330 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v146) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd2c0010380 con 0x7fd2d0130610 2024-09-18T10:14:23.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.522+0000 7fd2c5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:14:23.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.522+0000 7fd2c5ffb700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T10:14:23.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 msgr2=0x7fd2b4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 0x7fd2b4037f70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0058d0 tx=0x7fd2cc009040 comp rx=0 tx=0).stop 2024-09-18T10:14:23.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 10 monclient: shutdown 2024-09-18T10:14:23.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:23.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 msgr2=0x7fd2d009df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009fa0 tx=0x7fd2c0017000 comp rx=0 tx=0).stop 2024-09-18T10:14:23.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 shutdown_connections 2024-09-18T10:14:23.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b4035ac0 0x7fd2b4037f70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2d0135570 0x7fd2d00699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2d0130610 0x7fd2d009df60 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 --2- 172.21.15.150:0/2694269330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2d0130240 0x7fd2d012d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 >> 172.21.15.150:0/2694269330 conn(0x7fd2d00a3eb0 msgr2=0x7fd2d00a16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:23.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 shutdown_connections 2024-09-18T10:14:23.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.526+0000 7fd2d7ba9700 1 -- 172.21.15.150:0/2694269330 wait complete. 2024-09-18T10:14:23.532 INFO:tasks.workunit.client.0.smithi150.stderr:compat weight-set already created 2024-09-18T10:14:23.544 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create cool flat 2024-09-18T10:14:23.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.650+0000 7f035568f700 10 monclient: get_monmap_and_config 2024-09-18T10:14:23.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.650+0000 7f035568f700 10 monclient: build_initial_monmap 2024-09-18T10:14:23.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.650+0000 7f035568f700 10 monclient: monmap: 2024-09-18T10:14:23.656 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:23.656 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:23.655872+0000 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:23.655872+0000 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:23.657 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 Processor -- start 2024-09-18T10:14:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 -- start start 2024-09-18T10:14:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient: init 2024-09-18T10:14:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:23.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03500a0290 0x7f035009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): picked mon.noname-c con 0x7f03500a0290 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03500a0660 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0350137a20 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035009e360 con 0x7f0350137a20 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035009da50 con 0x7f03500a0660 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f035009ef80 con 0x7f03500a0290 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f035568f700 10 monclient(hunting): authenticate will time out at 1145.749716s 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03500a0290 0x7f035009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03500a0290 0x7f035009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44810/0 (socket says 172.21.15.150:44810) 2024-09-18T10:14:23.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 1 -- 172.21.15.150:0/4084403681 learned_addr learned my addr 172.21.15.150:0/4084403681 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): get_auth_request con 0x7f03500a0290 auth_method 0 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.654+0000 7f034effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034e7fc700 1 -- 172.21.15.150:0/4084403681 >> v1:172.21.15.150:6789/0 conn(0x7f03500a0660 legacy=0x7f03500a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034f7fe700 1 -- 172.21.15.150:0/4084403681 >> v1:172.21.15.142:6789/0 conn(0x7f0350137a20 legacy=0x7f0350139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 10 monclient(hunting): handle_auth_done global_id 15562 payload 274 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 -- 172.21.15.150:0/4084403681 >> v1:172.21.15.150:6789/0 conn(0x7f03500a0660 legacy=0x7f03500a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 -- 172.21.15.150:0/4084403681 >> v1:172.21.15.142:6789/0 conn(0x7f0350137a20 legacy=0x7f0350139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 monclient: found mon.noname-c 2024-09-18T10:14:23.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:23.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 -- 172.21.15.150:0/4084403681 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035009dd30 con 0x7f03500a0290 2024-09-18T10:14:23.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 -- 172.21.15.150:0/4084403681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03500a0290 msgr2=0x7f035009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034effd700 1 --2- 172.21.15.150:0/4084403681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03500a0290 0x7f035009ea40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0344009ab0 tx=0x7f0344009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25304e95782de129 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 1 -- 172.21.15.150:0/4084403681 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034400bc70 con 0x7f03500a0290 2024-09-18T10:14:23.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:23.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: dump: 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f035568f700 5 monclient: authenticate success, global_id 15562 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 1 -- 172.21.15.150:0/4084403681 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034400bdd0 con 0x7f03500a0290 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f035568f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f035568f700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 1 -- 172.21.15.150:0/4084403681 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344011670 con 0x7f03500a0290 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f035568f700 4 set_mon_vals no callback set 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.658+0000 7f034dffb700 10 monclient: dump: 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:23.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.667 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: shutdown 2024-09-18T10:14:23.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:23.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- 172.21.15.150:0/4084403681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03500a0290 msgr2=0x7f035009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 --2- 172.21.15.150:0/4084403681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03500a0290 0x7f035009ea40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0344009ab0 tx=0x7f0344009250 comp rx=0 tx=0).stop 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- 172.21.15.150:0/4084403681 shutdown_connections 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 --2- 172.21.15.150:0/4084403681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03500a0290 0x7f035009ea40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- 172.21.15.150:0/4084403681 >> 172.21.15.150:0/4084403681 conn(0x7f03500a58a0 msgr2=0x7f03500a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- 172.21.15.150:0/4084403681 shutdown_connections 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- 172.21.15.150:0/4084403681 wait complete. 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: build_initial_monmap 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: monmap: 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:23.671 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 Processor -- start 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- start start 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: init 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:23.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): picked mon.noname-c con 0x7f03500a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 0x7f03501423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): picked mon.noname-a con 0x7f03500a0660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): picked mon.noname-b con 0x7f0350137a20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035012ec90 con 0x7f03500a0660 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035009d3f0 con 0x7f0350137a20 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f035009c0f0 con 0x7f03500a0290 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f035568f700 10 monclient(hunting): authenticate will time out at 1145.759584s 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 0x7f03501423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44816/0 (socket says 172.21.15.150:44816) 2024-09-18T10:14:23.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 1 -- 172.21.15.150:0/1719128289 learned_addr learned my addr 172.21.15.150:0/1719128289 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 10 monclient(hunting): get_auth_request con 0x7f0350137a20 auth_method 0 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034effd700 10 monclient(hunting): get_auth_request con 0x7f03500a0290 auth_method 0 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.662+0000 7f034effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): get_auth_request con 0x7f03500a0660 auth_method 0 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034effd700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 msgr2=0x7f0350141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034effd700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 msgr2=0x7f0350141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034effd700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034effd700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient(hunting): handle_auth_done global_id 15565 payload 824 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 msgr2=0x7f0350141e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 msgr2=0x7f03501423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 0x7f03501423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03440094f0 con 0x7f0350137a20 2024-09-18T10:14:23.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034e7fc700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 0x7f03501423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 msgr2=0x7f0350142910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034f7fe700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f034800c910 tx=0x7f0348001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03480070f0 con 0x7f0350137a20 2024-09-18T10:14:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: dump: 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:23.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0348007250 con 0x7f0350137a20 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 5 monclient: authenticate success, global_id 15565 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 10 monclient: _renew_subs 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0350130140 con 0x7f0350137a20 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0348007610 con 0x7f0350137a20 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:23.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f0336ffd700 10 monclient: dump: 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 10 monclient: _renew_subs 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035012e0c0 con 0x7f0350137a20 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.666+0000 7f034d7fa700 4 set_mon_vals no callback set 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f035568f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f035568f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f035568f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035009d6c0 con 0x7f0350137a20 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f034800a3f0 con 0x7f0350137a20 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f0336ffd700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 0x7f033c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(146..146 src has 58..146) v4 ==== 8350+0+0 (secure 0 0 0) 0x7f03480488e0 con 0x7f0350137a20 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f034effd700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 0x7f033c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.670+0000 7f034effd700 10 monclient: get_auth_request con 0x7f033c035ac0 auth_method 0 2024-09-18T10:14:23.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.674+0000 7f034effd700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 0x7f033c037f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0344009ab0 tx=0x7f0344017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.678+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f034801b3b0 con 0x7f0350137a20 2024-09-18T10:14:23.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.678+0000 7f0336ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.678+0000 7f0336ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:23.841 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:23.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.850+0000 7f035568f700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:23.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.850+0000 7f035568f700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.850+0000 7f035568f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:23.850+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f0350063ed0 con 0x7f0350137a20 2024-09-18T10:14:23.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: get_monmap_and_config 2024-09-18T10:14:23.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: build_initial_monmap 2024-09-18T10:14:23.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: monmap: 2024-09-18T10:14:23.953 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:23.952313+0000 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:23.952313+0000 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:23.954 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 Processor -- start 2024-09-18T10:14:23.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 -- start start 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: init 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dc40a5f90 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dc40990b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4dc409e360 0x7f4dc413aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-c con 0x7f4dc409e360 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc409d980 con 0x7f4dc40990b0 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc40a6360 con 0x7f4dc40a5f90 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc409d780 con 0x7f4dc409e360 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:23.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc9b64700 10 monclient(hunting): authenticate will time out at 1152.878669s 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc2ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4dc40990b0 0x7f4dc40a0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49860/0 (socket says 172.21.15.142:49860) 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc2ffd700 1 -- 172.21.15.142:0/1403688689 learned_addr learned my addr 172.21.15.142:0/1403688689 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 1 --2- 172.21.15.142:0/1403688689 >> v2:172.21.15.150:3300/0 conn(0x7f4dc409e360 0x7f4dc413aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): get_auth_request con 0x7f4dc409e360 auth_method 0 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_done global_id 15568 payload 274 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 1 -- 172.21.15.142:0/1403688689 >> v1:172.21.15.150:6789/0 conn(0x7f4dc40a5f90 legacy=0x7f4dc40a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:23.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 1 -- 172.21.15.142:0/1403688689 >> v1:172.21.15.142:6789/0 conn(0x7f4dc40990b0 legacy=0x7f4dc40a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:23.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:23.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:23.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:23.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:23.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.950+0000 7f4dc3fff700 1 -- 172.21.15.142:0/1403688689 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc40a1090 con 0x7f4dc409e360 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc3fff700 1 -- 172.21.15.142:0/1403688689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 msgr2=0x7f4dc413aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc3fff700 1 --2- 172.21.15.142:0/1403688689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc413aa20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4099720 tx=0x7f4dbc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73a96adf20e6807 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 1 -- 172.21.15.142:0/1403688689 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00e070 con 0x7f4dc409e360 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: dump: 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:23.960 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 5 monclient: authenticate success, global_id 15568 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 1 -- 172.21.15.142:0/1403688689 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc012430 con 0x7f4dc409e360 2024-09-18T10:14:23.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 4 set_mon_vals no callback set 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 1 -- 172.21.15.142:0/1403688689 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0127f0 con 0x7f4dc409e360 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc27fc700 10 monclient: dump: 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 10 monclient: shutdown 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 -- 172.21.15.142:0/1403688689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 msgr2=0x7f4dc413aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/1403688689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc413aa20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4099720 tx=0x7f4dbc009780 comp rx=0 tx=0).stop 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 -- 172.21.15.142:0/1403688689 shutdown_connections 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/1403688689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc413aa20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 -- 172.21.15.142:0/1403688689 >> 172.21.15.142:0/1403688689 conn(0x7f4dc4071680 msgr2=0x7f4dc40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 -- 172.21.15.142:0/1403688689 shutdown_connections 2024-09-18T10:14:23.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.954+0000 7f4dc9b64700 1 -- 172.21.15.142:0/1403688689 wait complete. 2024-09-18T10:14:23.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient: build_initial_monmap 2024-09-18T10:14:23.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient: monmap: 2024-09-18T10:14:23.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 Processor -- start 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 -- start start 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient: init 2024-09-18T10:14:23.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-c con 0x7f4dc40990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc408cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-b con 0x7f4dc409e360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): picked mon.noname-a con 0x7f4dc40a5f90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc408e120 con 0x7f4dc40a5f90 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc408e750 con 0x7f4dc409e360 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:23.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4090280 con 0x7f4dc40990b0 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc9b64700 10 monclient(hunting): authenticate will time out at 1152.885876s 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43856/0 (socket says 172.21.15.142:43856) 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 -- 172.21.15.142:0/3364029370 learned_addr learned my addr 172.21.15.142:0/3364029370 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): get_auth_request con 0x7f4dc40a5f90 auth_method 0 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc408cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): get_auth_request con 0x7f4dc409e360 auth_method 0 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 10 monclient(hunting): get_auth_request con 0x7f4dc40990b0 auth_method 0 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:23.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 msgr2=0x7f4dc408fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 msgr2=0x7f4dc408fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient(hunting): handle_auth_done global_id 54366 payload 824 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 msgr2=0x7f4dc408fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc37fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 msgr2=0x7f4dc408cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc408cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 monclient: found mon.noname-a 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.958+0000 7f4dc3fff700 1 -- 172.21.15.142:0/3364029370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009a20 con 0x7f4dc40a5f90 2024-09-18T10:14:23.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc3fff700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 msgr2=0x7f4dc408d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc3fff700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0097c0 tx=0x7f4dbc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc01d640 con 0x7f4dc40a5f90 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: dump: 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.972 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 5 monclient: authenticate success, global_id 54366 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc01d7a0 con 0x7f4dc40a5f90 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _renew_subs 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4090560 con 0x7f4dc40a5f90 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc01db40 con 0x7f4dc40a5f90 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:23.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 10 monclient: dump: 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc17fa700 4 set_mon_vals no callback set 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _renew_subs 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:23.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc408f320 con 0x7f4dc40a5f90 2024-09-18T10:14:23.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc40696a0 con 0x7f4dc40a5f90 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4dbc024070 con 0x7f4dc40a5f90 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 0x7f4db0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.962+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(146..146 src has 58..146) v4 ==== 8350+0+0 (secure 0 0 0) 0x7f4dbc04bb50 con 0x7f4dc40a5f90 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.966+0000 7f4dc37fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 0x7f4db0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.966+0000 7f4dc37fe700 10 monclient: get_auth_request con 0x7f4db0035ac0 auth_method 0 2024-09-18T10:14:23.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.966+0000 7f4dc37fe700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 0x7f4db0037f70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4db8007cb0 tx=0x7f4db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:23.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.974+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4dbc04de60 con 0x7f4dc40a5f90 2024-09-18T10:14:23.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.974+0000 7f4dab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:23.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:23.974+0000 7f4dab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:24.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.166+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 v147) v1 ==== 107+0+0 (secure 0 0 0) 0x7f0348017030 con 0x7f0350137a20 2024-09-18T10:14:24.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.166+0000 7f0336ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:24.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.166+0000 7f0336ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:24.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:24.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:24.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4dc4067950 con 0x7f4dc40a5f90 2024-09-18T10:14:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dab7fe700 1 -- 172.21.15.142:0/3364029370 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4dbc011b70 con 0x7f4dc40a5f90 2024-09-18T10:14:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:24.206 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:24.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 msgr2=0x7f4db0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 0x7f4db0037f70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4db8007cb0 tx=0x7f4db8009040 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 10 monclient: shutdown 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 msgr2=0x7f4dc408d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0097c0 tx=0x7f4dbc018000 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 shutdown_connections 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4db0035ac0 0x7f4db0037f70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4dc40a5f90 0x7f4dc408d1a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dc409e360 0x7f4dc408cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 --2- 172.21.15.142:0/3364029370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4dc40990b0 0x7f4dc408fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.202+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 >> 172.21.15.142:0/3364029370 conn(0x7f4dc4071680 msgr2=0x7f4dc409b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.206+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 shutdown_connections 2024-09-18T10:14:24.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.206+0000 7f4dc9b64700 1 -- 172.21.15.142:0/3364029370 wait complete. 2024-09-18T10:14:24.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:14:24.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:14:24.220 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:14:24.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f035568f700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:24.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f035568f700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:24.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f035568f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:24.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"} v 0) v1 -- 0x7f0350067b10 con 0x7f0350137a20 2024-09-18T10:14:24.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f0336ffd700 1 -- 172.21.15.150:0/1719128289 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}]=0 weight-set for pool 'cool' already created v147) v1 ==== 149+0+0 (secure 0 0 0) 0x7f0348017020 con 0x7f0350137a20 2024-09-18T10:14:24.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f0336ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cool", "mode": "flat"}] 2024-09-18T10:14:24.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.278+0000 7f0336ffd700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cool' already created 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 msgr2=0x7f033c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 0x7f033c037f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0344009ab0 tx=0x7f0344017000 comp rx=0 tx=0).stop 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 10 monclient: shutdown 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 msgr2=0x7f0350142910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f034800c910 tx=0x7f0348001970 comp rx=0 tx=0).stop 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 shutdown_connections 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f033c035ac0 0x7f033c037f70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0350137a20 0x7f0350142910 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03500a0660 0x7f03501423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 --2- 172.21.15.150:0/1719128289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03500a0290 0x7f0350141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 >> 172.21.15.150:0/1719128289 conn(0x7f03500a58a0 msgr2=0x7f03500a4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 shutdown_connections 2024-09-18T10:14:24.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.282+0000 7f035568f700 1 -- 172.21.15.150:0/1719128289 wait complete. 2024-09-18T10:14:24.287 INFO:tasks.workunit.client.0.smithi150.stderr:weight-set for pool 'cool' already created 2024-09-18T10:14:24.296 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create cold positional 2024-09-18T10:14:24.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.314+0000 7f95f359e700 10 monclient: get_monmap_and_config 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient: monmap: 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:24.319765+0000 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:24.319765+0000 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:24.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:24.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:24.322 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:24.322 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:24.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 Processor -- start 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 -- start start 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient: init 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95f409abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:24.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f95f409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f95f4134bb0 0x7f95f4136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-a con 0x7f95f4134bb0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f409dc90 con 0x7f95f409abd0 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95f409b500 con 0x7f95f4134bb0 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40967f0 con 0x7f95f409b740 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f359e700 10 monclient(hunting): authenticate will time out at 1153.245204s 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f95f4134bb0 0x7f95f4136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f259c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95f409abd0 0x7f95f40960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:42132/0 (socket says 172.21.15.142:42132) 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f259c700 1 -- 172.21.15.142:0/624996822 learned_addr learned my addr 172.21.15.142:0/624996822 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): get_auth_request con 0x7f95f4134bb0 auth_method 0 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 10 monclient(hunting): get_auth_request con 0x7f95f409b740 auth_method 0 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 1 -- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 1 -- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_done global_id 54369 payload 274 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 -- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4134670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:24.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 -- 172.21.15.142:0/624996822 >> v1:172.21.15.150:6789/0 conn(0x7f95f409abd0 legacy=0x7f95f40960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 monclient: found mon.noname-a 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.318+0000 7f95f2d9d700 1 -- 172.21.15.142:0/624996822 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f409afa0 con 0x7f95f4134bb0 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f2d9d700 1 -- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 msgr2=0x7f95f4136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f2d9d700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f4136f90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95f406fd20 tx=0x7f95e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458ffeaf4af47989 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 1 -- 172.21.15.142:0/624996822 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400b8c0 con 0x7f95f4134bb0 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: dump: 2024-09-18T10:14:24.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 1 -- 172.21.15.142:0/624996822 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e400ba20 con 0x7f95f4134bb0 2024-09-18T10:14:24.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 5 monclient: authenticate success, global_id 54369 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 1 -- 172.21.15.142:0/624996822 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e400bdc0 con 0x7f95f4134bb0 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 4 set_mon_vals no callback set 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f159a700 10 monclient: dump: 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.332 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.333 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.333 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.333 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.333 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.333 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 10 monclient: shutdown 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 -- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 msgr2=0x7f95f4136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f4136f90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f95f406fd20 tx=0x7f95e40091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 -- 172.21.15.142:0/624996822 shutdown_connections 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f4136f90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 --2- 172.21.15.142:0/624996822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 -- 172.21.15.142:0/624996822 >> 172.21.15.142:0/624996822 conn(0x7f95f4070ef0 msgr2=0x7f95f40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 -- 172.21.15.142:0/624996822 shutdown_connections 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 1 -- 172.21.15.142:0/624996822 wait complete. 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.322+0000 7f95f359e700 10 monclient: monmap: 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 Processor -- start 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 -- start start 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient: init 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f409abd0 0x7f95f408b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f95f409abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-c con 0x7f95f409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f40894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): picked mon.noname-a con 0x7f95f4134bb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f409d240 con 0x7f95f4134bb0 2024-09-18T10:14:24.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f409f480 con 0x7f95f409abd0 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f408a630 con 0x7f95f409b740 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f359e700 10 monclient(hunting): authenticate will time out at 1153.253951s 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f40894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f40894f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43884/0 (socket says 172.21.15.142:43884) 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 learned_addr learned my addr 172.21.15.142:0/2839727550 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): get_auth_request con 0x7f95f4134bb0 auth_method 0 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f409abd0 0x7f95f408b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): get_auth_request con 0x7f95f409abd0 auth_method 0 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 10 monclient(hunting): get_auth_request con 0x7f95f409b740 auth_method 0 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4088fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4088fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient(hunting): handle_auth_done global_id 54372 payload 824 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 msgr2=0x7f95f4088fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95f409b740 0x7f95f4088fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f409abd0 msgr2=0x7f95f408b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f409abd0 0x7f95f408b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 monclient: found mon.noname-a 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.326+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4009420 con 0x7f95f4134bb0 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 msgr2=0x7f95f40894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f2d9d700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f40894f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f95e4016000 tx=0x7f95e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4006af0 con 0x7f95f4134bb0 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: dump: 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.347 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 5 monclient: authenticate success, global_id 54372 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e4026480 con 0x7f95f4134bb0 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e4026840 con 0x7f95f4134bb0 2024-09-18T10:14:24.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 monclient: _renew_subs 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f0d99700 4 set_mon_vals no callback set 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f408b190 con 0x7f95f4134bb0 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 10 monclient: dump: 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.349 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 monclient: _renew_subs 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f40945e0 con 0x7f95f4134bb0 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 0x7f95f406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f40648a0 con 0x7f95f4089a30 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 0x7f95f406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient: get_auth_request con 0x7f95f4089a30 auth_method 0 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 10 monclient(hunting): handle_auth_done global_id 54375 payload 824 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f259c700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95f408c770 con 0x7f95f4089a30 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f95e4026c70 con 0x7f95f4134bb0 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95d4035af0 0x7f95d4037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(147..147 src has 58..147) v4 ==== 8682+0+0 (secure 0 0 0) 0x7f95e404adb0 con 0x7f95f4134bb0 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.330+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95d4035af0 0x7f95d4037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95f1d9b700 10 monclient: get_auth_request con 0x7f95d4035af0 auth_method 0 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95f259c700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 msgr2=0x7f95f406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95f259c700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 0x7f95f406c370 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0079a0 tx=0x7f95ec0053c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec007b50 con 0x7f95f4089a30 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 10 monclient: dump: 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.352 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f95ec006020 con 0x7f95f4089a30 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 msgr2=0x7f95f406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 0x7f95f406c370 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0079a0 tx=0x7f95ec0053c0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 msgr2=0x7f95f406c370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95e27fc700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4089a30 0x7f95f406c370 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.334+0000 7f95f1d9b700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95d4035af0 0x7f95d4037fa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006f30 tx=0x7f95e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:24.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f359e700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 0x7f95f406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f408b130 con 0x7f95f406c8b0 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 0x7f95f406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient: get_auth_request con 0x7f95f406c8b0 auth_method 0 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 10 monclient(hunting): handle_auth_done global_id 54378 payload 824 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.346+0000 7f95f259c700 1 -- 172.21.15.142:0/2839727550 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f95f4072a80 con 0x7f95f406c8b0 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f259c700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 msgr2=0x7f95f406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f259c700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 0x7f95f406ccb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f95ec017000 tx=0x7f95ec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec01cae0 con 0x7f95f406c8b0 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 10 monclient: dump: 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f95ec00fcf0 con 0x7f95f406c8b0 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:24.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 msgr2=0x7f95f406ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 0x7f95f406ccb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f95ec017000 tx=0x7f95ec018000 comp rx=0 tx=0).stop 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 msgr2=0x7f95f406ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95e27fc700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f406c8b0 0x7f95f406ccb0 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f95ec017000 tx=0x7f95ec018000 comp rx=0 tx=0).stop 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:24.359 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:24.360 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:24.361 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:24.362 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:24.363 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:24.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:24.365 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:24.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:24.369 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:24.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:24.372 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:24.372 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:24.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95d4035af0 msgr2=0x7f95d4037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95d4035af0 0x7f95d4037fa0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006f30 tx=0x7f95e8008040 comp rx=0 tx=0).stop 2024-09-18T10:14:24.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 10 monclient: shutdown 2024-09-18T10:14:24.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 msgr2=0x7f95f40894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 --2- 172.21.15.142:0/2839727550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95f4134bb0 0x7f95f40894f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f95e4016000 tx=0x7f95e4017000 comp rx=0 tx=0).stop 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 reap_dead start 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f2d9d700 1 -- 172.21.15.142:0/2839727550 reap_dead start 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 shutdown_connections 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 >> 172.21.15.142:0/2839727550 conn(0x7f95f4070ef0 msgr2=0x7f95f409de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 shutdown_connections 2024-09-18T10:14:24.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.350+0000 7f95f359e700 1 -- 172.21.15.142:0/2839727550 wait complete. 2024-09-18T10:14:24.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:24.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:14:24.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:14:24.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: get_monmap_and_config 2024-09-18T10:14:24.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: monmap: 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:24.398139+0000 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:24.398139+0000 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:24.399 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 Processor -- start 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 -- start start 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: init 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4a806b5a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:24.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4a806e710 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff4a806eb10 0x7ff4a813ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-a con 0x7ff4a806eb10 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a806b970 con 0x7ff4a806e710 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a806ef90 con 0x7ff4a806b5a0 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a809fb10 con 0x7ff4a806eb10 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.394+0000 7ff4b01b6700 10 monclient(hunting): authenticate will time out at 1146.491894s 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4adf52700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff4a806b5a0 0x7ff4a809e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:54672/0 (socket says 172.21.15.150:54672) 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4adf52700 1 -- 172.21.15.150:0/2780098485 learned_addr learned my addr 172.21.15.150:0/2780098485 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 --2- 172.21.15.150:0/2780098485 >> v2:172.21.15.142:3300/0 conn(0x7ff4a806eb10 0x7ff4a813ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): get_auth_request con 0x7ff4a806eb10 auth_method 0 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient(hunting): handle_auth_done global_id 54381 payload 274 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 -- 172.21.15.150:0/2780098485 >> v1:172.21.15.150:6789/0 conn(0x7ff4a806b5a0 legacy=0x7ff4a809e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 -- 172.21.15.150:0/2780098485 >> v1:172.21.15.142:6789/0 conn(0x7ff4a806e710 legacy=0x7ff4a81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 monclient: found mon.noname-a 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 -- 172.21.15.150:0/2780098485 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a809ee40 con 0x7ff4a806eb10 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 -- 172.21.15.150:0/2780098485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806eb10 msgr2=0x7ff4a813ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4ae753700 1 --2- 172.21.15.150:0/2780098485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806eb10 0x7ff4a813ac20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009e60 tx=0x7ff4a0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500c52d571f48aec server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 1 -- 172.21.15.150:0/2780098485 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a000e070 con 0x7ff4a806eb10 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: dump: 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:24.406 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4b01b6700 5 monclient: authenticate success, global_id 54381 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4b01b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:24.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 1 -- 172.21.15.150:0/2780098485 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a0012430 con 0x7ff4a806eb10 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4b01b6700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4b01b6700 4 set_mon_vals no callback set 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 1 -- 172.21.15.150:0/2780098485 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00127d0 con 0x7ff4a806eb10 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.398+0000 7ff4acf50700 10 monclient: dump: 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.408 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: shutdown 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2780098485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806eb10 msgr2=0x7ff4a813ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2780098485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806eb10 0x7ff4a813ac20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0009e60 tx=0x7ff4a0009780 comp rx=0 tx=0).stop 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2780098485 shutdown_connections 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2780098485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806eb10 0x7ff4a813ac20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2780098485 >> 172.21.15.150:0/2780098485 conn(0x7ff4a8071b60 msgr2=0x7ff4a80a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2780098485 shutdown_connections 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2780098485 wait complete. 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: monmap: 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:24.410 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 Processor -- start 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- start start 2024-09-18T10:14:24.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: init 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 0x7ff4a81414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-b con 0x7ff4a806b5a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 0x7ff4a8134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-a con 0x7ff4a806e710 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): picked mon.noname-c con 0x7ff4a806eb10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a81209e0 con 0x7ff4a806e710 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a8120ed0 con 0x7ff4a806b5a0 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a8122760 con 0x7ff4a806eb10 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4b01b6700 10 monclient(hunting): authenticate will time out at 1146.498700s 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 0x7ff4a81414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 0x7ff4a81414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44830/0 (socket says 172.21.15.150:44830) 2024-09-18T10:14:24.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 1 -- 172.21.15.150:0/2106978038 learned_addr learned my addr 172.21.15.150:0/2106978038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): get_auth_request con 0x7ff4a806b5a0 auth_method 0 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 0x7ff4a8134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): get_auth_request con 0x7ff4a806e710 auth_method 0 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4adf52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 10 monclient(hunting): get_auth_request con 0x7ff4a806eb10 auth_method 0 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 msgr2=0x7ff4a811ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 msgr2=0x7ff4a811ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ae753700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient(hunting): handle_auth_done global_id 54384 payload 824 2024-09-18T10:14:24.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 msgr2=0x7ff4a811ee50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 msgr2=0x7ff4a81414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 0x7ff4a81414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 monclient: found mon.noname-a 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.402+0000 7ff4ad751700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0009a20 con 0x7ff4a806e710 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4ad751700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 msgr2=0x7ff4a8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4ad751700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 0x7ff4a8134350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c510 tx=0x7ff4a4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40117a0 con 0x7ff4a806e710 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:24.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: dump: 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 5 monclient: authenticate success, global_id 54384 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a4011920 con 0x7ff4a806e710 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 10 monclient: _renew_subs 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a8142300 con 0x7ff4a806e710 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4011ce0 con 0x7ff4a806e710 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49dffb700 10 monclient: dump: 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:24.421 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff49f7fe700 4 set_mon_vals no callback set 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 10 monclient: _renew_subs 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.406+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a8121520 con 0x7ff4a806e710 2024-09-18T10:14:24.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff4a4012630 con 0x7ff4a806e710 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4b01b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4b01b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4b01b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a806a7e0 con 0x7ff4a806e710 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff49dffb700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 0x7ff48c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(147..147 src has 58..147) v4 ==== 8682+0+0 (secure 0 0 0) 0x7ff4a4048f70 con 0x7ff4a806e710 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4adf52700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 0x7ff48c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4adf52700 10 monclient: get_auth_request con 0x7ff48c035b10 auth_method 0 2024-09-18T10:14:24.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.410+0000 7ff4adf52700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 0x7ff48c037fc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff498007880 tx=0x7ff498004a80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.422+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff4a4011e40 con 0x7ff4a806e710 2024-09-18T10:14:24.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.422+0000 7ff49dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.422+0000 7ff49dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:24.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: get_monmap_and_config 2024-09-18T10:14:24.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: monmap: 2024-09-18T10:14:24.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:24.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:24.478447+0000 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:24.478447+0000 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:24.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 1 Processor -- start 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 1 -- start start 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: init 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe07c09d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe07c0990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.474+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe07c0954b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe07c134670 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c069c70 con 0x7fe07c134670 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0958b0 con 0x7fe07c09d870 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06f9d0 con 0x7fe07c0954b0 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 10 monclient(hunting): authenticate will time out at 1153.404100s 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe081447700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe07c134670 0x7fe07c136a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49864/0 (socket says 172.21.15.142:49864) 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe081447700 1 -- 172.21.15.142:0/813747651 learned_addr learned my addr 172.21.15.142:0/813747651 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe07c0990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 10 monclient(hunting): get_auth_request con 0x7fe07c0954b0 auth_method 0 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 msgr2=0x7fe07c0990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 msgr2=0x7fe07c0990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe07c0990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe07c0990d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe081447700 1 -- 172.21.15.142:0/813747651 >> v1:172.21.15.142:6789/0 conn(0x7fe07c134670 legacy=0x7fe07c136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297917858 0 0) 0x7fe07c069c70 con 0x7fe07c134670 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): my global_id is 54387 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 1 -- 172.21.15.142:0/813747651 >> v1:172.21.15.150:6789/0 conn(0x7fe07c09d870 legacy=0x7fe07c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe068003570 con 0x7fe07c134670 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614100839 0 0) 0x7fe07c0958b0 con 0x7fe07c09d870 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): my global_id is 15580 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c069c70 con 0x7fe07c09d870 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3646455661 0 0) 0x7fe068003570 con 0x7fe07c134670 2024-09-18T10:14:24.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 msgr2=0x7fe07c0990d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe07c0990d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 >> v1:172.21.15.150:6789/0 conn(0x7fe07c09d870 legacy=0x7fe07c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c09dc40 con 0x7fe07c134670 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fe06c0017a0 con 0x7fe07c134670 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 5 monclient: authenticate success, global_id 54387 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: dump: 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:14:24.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 >> v1:172.21.15.142:6789/0 conn(0x7fe07c134670 legacy=0x7fe07c136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe068003d90 0x7fe068004180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): picked mon.a con 0x7fe068003d90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe0680048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): picked mon.c con 0x7fe07c0954b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe068004f20 0x7fe06800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): picked mon.b con 0x7fe068004f20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680056d0 con 0x7fe068003d90 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06800bfd0 con 0x7fe068004f20 2024-09-18T10:14:24.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe068003d90 0x7fe068004180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06800c190 con 0x7fe07c0954b0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe082eaa700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 10 monclient(hunting): get_auth_request con 0x7fe068003d90 auth_method 0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe080c46700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe0680048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.478+0000 7fe07bfff700 10 monclient(hunting): get_auth_request con 0x7fe07c0954b0 auth_method 0 2024-09-18T10:14:24.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:24.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe068004f20 0x7fe06800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 10 monclient(hunting): get_auth_request con 0x7fe068004f20 auth_method 0 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient(hunting): handle_auth_done global_id 54387 payload 293 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe068004f20 msgr2=0x7fe06800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe068004f20 0x7fe06800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe068003d90 msgr2=0x7fe068004180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe068003d90 0x7fe068004180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 monclient: found mon.c 2024-09-18T10:14:24.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068003570 con 0x7fe07c0954b0 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe080c46700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe068003d90 0x7fe068004180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 msgr2=0x7fe0680048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07bfff700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe0680048a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe074007440 tx=0x7fe074009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55de38b80943ebdd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074009a30 con 0x7fe07c0954b0 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: dump: 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074014400 con 0x7fe07c0954b0 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 4 set_mon_vals no callback set 2024-09-18T10:14:24.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 1 -- 172.21.15.142:0/813747651 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0740147c0 con 0x7fe07c0954b0 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe07b7fe700 10 monclient: dump: 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 10 monclient: shutdown 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 -- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 msgr2=0x7fe0680048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 --2- 172.21.15.142:0/813747651 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c0954b0 0x7fe0680048a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe074007440 tx=0x7fe074009720 comp rx=0 tx=0).stop 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe081447700 1 -- 172.21.15.142:0/813747651 reap_dead start 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 -- 172.21.15.142:0/813747651 shutdown_connections 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 -- 172.21.15.142:0/813747651 >> 172.21.15.142:0/813747651 conn(0x7fe07c0a14e0 msgr2=0x7fe07c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 -- 172.21.15.142:0/813747651 shutdown_connections 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.482+0000 7fe082eaa700 1 -- 172.21.15.142:0/813747651 wait complete. 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient: build_initial_monmap 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient: monmap: 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:24.504 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 Processor -- start 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 -- start start 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient: init 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:24.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe07c0954b0 0x7fe07c133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-b con 0x7fe07c0954b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-c con 0x7fe07c133bc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 0x7fe07c08ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): picked mon.noname-a con 0x7fe07c08d9a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c097b80 con 0x7fe07c08d9a0 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c096d10 con 0x7fe07c0954b0 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c1344d0 con 0x7fe07c133bc0 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe082eaa700 10 monclient(hunting): authenticate will time out at 1153.413151s 2024-09-18T10:14:24.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59452/0 (socket says 172.21.15.142:59452) 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 -- 172.21.15.142:0/2569329470 learned_addr learned my addr 172.21.15.142:0/2569329470 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 0x7fe07c08ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 10 monclient(hunting): get_auth_request con 0x7fe07c133bc0 auth_method 0 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): get_auth_request con 0x7fe07c08d9a0 auth_method 0 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe07c0954b0 0x7fe07c133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 msgr2=0x7fe07c133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 msgr2=0x7fe07c133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): get_auth_request con 0x7fe07c0954b0 auth_method 0 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe07bfff700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:24.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient(hunting): handle_auth_done global_id 54390 payload 824 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 msgr2=0x7fe07c133f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c133bc0 0x7fe07c133f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe07c0954b0 msgr2=0x7fe07c133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe07c0954b0 0x7fe07c133680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient: _finish_hunting 0 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 monclient: found mon.noname-a 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.486+0000 7fe081447700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0900e0 con 0x7fe07c08d9a0 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe081447700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 msgr2=0x7fe07c08ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe081447700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 0x7fe07c08ddb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe07c095170 tx=0x7fe06c0010c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c011700 con 0x7fe07c08d9a0 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: dump: 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 5 monclient: authenticate success, global_id 54390 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 monclient: _renew_subs 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c011880 con 0x7fe07c08d9a0 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c094e90 con 0x7fe07c08d9a0 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c011c40 con 0x7fe07c08d9a0 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:24.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe07affd700 4 set_mon_vals no callback set 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 10 monclient: dump: 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 monclient: _renew_subs 2024-09-18T10:14:24.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c090fe0 con 0x7fe07c08d9a0 2024-09-18T10:14:24.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:24.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c064950 con 0x7fe07c08e350 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c08e350 auth_method 0 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe06c0133e0 con 0x7fe07c08d9a0 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe05c035ac0 0x7fe05c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(147..147 src has 58..147) v4 ==== 8682+0+0 (secure 0 0 0) 0x7fe06c0489f0 con 0x7fe07c08d9a0 2024-09-18T10:14:24.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe07bfff700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe05c035ac0 0x7fe05c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.490+0000 7fe07bfff700 10 monclient: get_auth_request con 0x7fe05c035ac0 auth_method 0 2024-09-18T10:14:24.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.494+0000 7fe07bfff700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe05c035ac0 0x7fe05c037f70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe07400afd0 tx=0x7fe074017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:24.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.594+0000 7ff4b01b6700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:24.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.594+0000 7ff4b01b6700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:24.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.594+0000 7ff4b01b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:24.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:24.594+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7ff4a8063ed0 con 0x7ff4a806e710 2024-09-18T10:14:24.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:24.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c08e350 auth_method 0 2024-09-18T10:14:24.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:24.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:24.694+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:14:25.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c08e350 auth_method 0 2024-09-18T10:14:25.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:25.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:25.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:25.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:25.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.098+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:14:25.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.182+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 v148) v1 ==== 113+0+0 (secure 0 0 0) 0x7ff4a40102e0 con 0x7ff4a806e710 2024-09-18T10:14:25.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.182+0000 7ff49dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:25.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.182+0000 7ff49dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:25.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff4b01b6700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:25.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff4b01b6700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:25.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff4b01b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:25.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"} v 0) v1 -- 0x7ff4a8122ad0 con 0x7ff4a806e710 2024-09-18T10:14:25.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff49dffb700 1 -- 172.21.15.150:0/2106978038 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}]=0 weight-set for pool 'cold' already created v148) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff4a40104c0 con 0x7ff4a806e710 2024-09-18T10:14:25.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff49dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create", "pool": "cold", "mode": "positional"}] 2024-09-18T10:14:25.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.294+0000 7ff49dffb700 10 monclient: _finish_command 3 = system:0 weight-set for pool 'cold' already created 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 msgr2=0x7ff48c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 0x7ff48c037fc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff498007880 tx=0x7ff498004a80 comp rx=0 tx=0).stop 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 10 monclient: shutdown 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 msgr2=0x7ff4a8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 0x7ff4a8134350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400c510 tx=0x7ff4a4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 shutdown_connections 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff48c035b10 0x7ff48c037fc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4a806eb10 0x7ff4a811ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4a806e710 0x7ff4a8134350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 --2- 172.21.15.150:0/2106978038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a806b5a0 0x7ff4a81414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 >> 172.21.15.150:0/2106978038 conn(0x7ff4a8071b60 msgr2=0x7ff4a809cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 shutdown_connections 2024-09-18T10:14:25.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.298+0000 7ff4b01b6700 1 -- 172.21.15.150:0/2106978038 wait complete. 2024-09-18T10:14:25.304 INFO:tasks.workunit.client.0.smithi150.stderr:weight-set for pool 'cold' already created 2024-09-18T10:14:25.313 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.398+0000 7fc4b9f3f700 10 monclient: get_monmap_and_config 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.398+0000 7fc4b9f3f700 10 monclient: build_initial_monmap 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.398+0000 7fc4b9f3f700 10 monclient: monmap: 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:25.403101+0000 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:25.403101+0000 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:25.411 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 Processor -- start 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 -- start start 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient: init 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4b409e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:25.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4b40a09a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc4b413b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-b con 0x7fc4b40a0da0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b409e220 con 0x7fc4b40a09a0 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b409eaa0 con 0x7fc4b409e6d0 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b409ee30 con 0x7fc4b40a0da0 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b9f3f700 10 monclient(hunting): authenticate will time out at 1147.496947s 2024-09-18T10:14:25.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b37fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc4b409e6d0 0x7fc4b40a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:54674/0 (socket says 172.21.15.150:54674) 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b37fe700 1 -- 172.21.15.150:0/1720363001 learned_addr learned my addr 172.21.15.150:0/1720363001 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc4b413b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request con 0x7fc4b40a0da0 auth_method 0 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 msgr2=0x7fc4b413b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 msgr2=0x7fc4b413b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc4b413b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc4b413b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:25.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b37fe700 1 -- 172.21.15.150:0/1720363001 >> v1:172.21.15.150:6789/0 conn(0x7fc4b409e6d0 legacy=0x7fc4b40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b2ffd700 1 -- 172.21.15.150:0/1720363001 >> v1:172.21.15.142:6789/0 conn(0x7fc4b40a09a0 legacy=0x7fc4b4137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969078234 0 0) 0x7fc4b409eaa0 con 0x7fc4b409e6d0 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b27fc700 10 monclient(hunting): my global_id is 15586 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.402+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49c003570 con 0x7fc4b409e6d0 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200672639 0 0) 0x7fc4b409e220 con 0x7fc4b40a09a0 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): my global_id is 54393 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b409eaa0 con 0x7fc4b40a09a0 2024-09-18T10:14:25.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981865847 0 0) 0x7fc49c003570 con 0x7fc4b409e6d0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 msgr2=0x7fc4b413b150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc4b413b150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 >> v1:172.21.15.142:6789/0 conn(0x7fc4b40a09a0 legacy=0x7fc4b4137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b409d930 con 0x7fc4b409e6d0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fc4a80017a0 con 0x7fc4b409e6d0 2024-09-18T10:14:25.416 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: dump: 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:25.417 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 >> v1:172.21.15.150:6789/0 conn(0x7fc4b409e6d0 legacy=0x7fc4b40a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:25.418 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc49c003db0 0x7fc49c0041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): picked mon.a con 0x7fc49c003db0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:25.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 0x7fc49c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): picked mon.b con 0x7fc49c004860 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:25.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc49c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): picked mon.c con 0x7fc4b40a0da0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b409ee30 con 0x7fc49c003db0 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c00bfa0 con 0x7fc49c004860 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.420 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c00c0e0 con 0x7fc4b40a0da0 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 0x7fc49c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): get_auth_request con 0x7fc49c004860 auth_method 0 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:25.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc49c003db0 0x7fc49c0041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): get_auth_request con 0x7fc49c003db0 auth_method 0 2024-09-18T10:14:25.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.422 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc49c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request con 0x7fc4b40a0da0 auth_method 0 2024-09-18T10:14:25.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:25.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_done global_id 15586 payload 293 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 msgr2=0x7fc49c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc49c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc49c003db0 msgr2=0x7fc49c0041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc49c003db0 0x7fc49c0041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:25.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 monclient: found mon.b 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 -- 172.21.15.150:0/1720363001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c003570 con 0x7fc49c004860 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b9f3f700 5 monclient: authenticate success, global_id 15586 2024-09-18T10:14:25.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b9f3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:25.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4b40a0da0 0x7fc49c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:25.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 msgr2=0x7fc49c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:25.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b2ffd700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 0x7fc49c004cd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4001d30 tx=0x7fc4a4007cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2652777046f580fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:25.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400c8b0 con 0x7fc49c004860 2024-09-18T10:14:25.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: dump: 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.438 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b9f3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:25.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a400ca10 con 0x7fc49c004860 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b9f3f700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b9f3f700 4 set_mon_vals no callback set 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 1 -- 172.21.15.150:0/1720363001 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400cdd0 con 0x7fc49c004860 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:25.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.406+0000 7fc4b27fc700 10 monclient: dump: 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.441 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: shutdown 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 msgr2=0x7fc49c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/1720363001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc49c004860 0x7fc49c004cd0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4001d30 tx=0x7fc4a4007cb0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 1 -- 172.21.15.150:0/1720363001 reap_dead start 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/1720363001 shutdown_connections 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/1720363001 >> 172.21.15.150:0/1720363001 conn(0x7fc4b40a58a0 msgr2=0x7fc4b40a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/1720363001 shutdown_connections 2024-09-18T10:14:25.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/1720363001 wait complete. 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: build_initial_monmap 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: monmap: 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:25.444 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 Processor -- start 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- start start 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: init 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-c con 0x7fc49c004860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4b4142020 0x7fc4b41423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-a con 0x7fc4b4142020 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): picked mon.noname-b con 0x7fc4b408df60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b409c120 con 0x7fc4b4142020 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4091aa0 con 0x7fc4b408df60 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4142930 con 0x7fc49c004860 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b9f3f700 10 monclient(hunting): authenticate will time out at 1147.507299s 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44840/0 (socket says 172.21.15.150:44840) 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 1 -- 172.21.15.150:0/181558489 learned_addr learned my addr 172.21.15.150:0/181558489 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request con 0x7fc4b408df60 auth_method 0 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4b4142020 0x7fc4b41423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 10 monclient(hunting): get_auth_request con 0x7fc49c004860 auth_method 0 2024-09-18T10:14:25.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): get_auth_request con 0x7fc4b4142020 auth_method 0 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 msgr2=0x7fc4b4141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 msgr2=0x7fc4b4141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:25.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 10 monclient(hunting): handle_auth_done global_id 15589 payload 824 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.410+0000 7fc4b3fff700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 msgr2=0x7fc4b4141ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4b4142020 msgr2=0x7fc4b41423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4b4142020 0x7fc4b41423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 monclient: found mon.noname-b 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b40917c0 con 0x7fc4b408df60 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 msgr2=0x7fc4b408e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b3fff700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4065260 tx=0x7fc4ac009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:25.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac014410 con 0x7fc4b408df60 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: dump: 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:25.450 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 5 monclient: authenticate success, global_id 15589 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 10 monclient: _renew_subs 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4099490 con 0x7fc4b408df60 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 10 monclient: _renew_subs 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b40927a0 con 0x7fc4b408df60 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac00f050 con 0x7fc4b408df60 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0147e0 con 0x7fc4b408df60 2024-09-18T10:14:25.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc4b1ffb700 4 set_mon_vals no callback set 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.414+0000 7fc49b7fe700 10 monclient: dump: 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:25.452 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc4ac003040 con 0x7fc4b408df60 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b9f3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b9f3f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b9f3f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc49b7fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 0x7fc4a0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4940050b0 con 0x7fc4b408df60 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(148..148 src has 58..148) v4 ==== 9154+0+0 (secure 0 0 0) 0x7fc4ac049640 con 0x7fc4b408df60 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 0x7fc4a0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b37fe700 10 monclient: get_auth_request con 0x7fc4a0035ac0 auth_method 0 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.418+0000 7fc4b37fe700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 0x7fc4a0037f70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8001980 tx=0x7fc4a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.426+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc4ac011230 con 0x7fc4b408df60 2024-09-18T10:14:25.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.426+0000 7fc49b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:25.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.426+0000 7fc49b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:25.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.590+0000 7fc4b9f3f700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:25.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.590+0000 7fc4b9f3f700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:25.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.590+0000 7fc4b9f3f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:25.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:25.590+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fc4940058e0 con 0x7fc4b408df60 2024-09-18T10:14:25.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:25.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c08e350 auth_method 0 2024-09-18T10:14:25.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:25.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:25.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:25.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:14:25.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:25.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:25.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:25.902+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:14:26.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.190+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v149) v1 ==== 123+0+0 (secure 0 0 0) 0x7fc4ac0122d0 con 0x7fc4b408df60 2024-09-18T10:14:26.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.190+0000 7fc49b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:26.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.190+0000 7fc49b7fe700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T10:14:26.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.302+0000 7fc4b9f3f700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:26.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.302+0000 7fc4b9f3f700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:26.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.302+0000 7fc4b9f3f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:26.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:26.302+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fc494005360 con 0x7fc4b408df60 2024-09-18T10:14:27.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.034+0000 7fc49b7fe700 1 -- 172.21.15.150:0/181558489 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v150) v1 ==== 149+0+0 (secure 0 0 0) 0x7fc4ac012920 con 0x7fc4b408df60 2024-09-18T10:14:27.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.034+0000 7fc49b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:27.038 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.034+0000 7fc49b7fe700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:14:27.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 msgr2=0x7fc4a0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 0x7fc4a0037f70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8001980 tx=0x7fc4a8009040 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 10 monclient: shutdown 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 msgr2=0x7fc4b408e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4065260 tx=0x7fc4ac009880 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 shutdown_connections 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4a0035ac0 0x7fc4a0037f70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4b408df60 0x7fc4b408e3b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4b4142020 0x7fc4b41423f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 --2- 172.21.15.150:0/181558489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc49c004860 0x7fc4b4141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 >> 172.21.15.150:0/181558489 conn(0x7fc4b40a58a0 msgr2=0x7fc4b409ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 shutdown_connections 2024-09-18T10:14:27.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.038+0000 7fc4b9f3f700 1 -- 172.21.15.150:0/181558489 wait complete. 2024-09-18T10:14:27.044 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:14:27.057 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight-compat osd.0 10.5 2024-09-18T10:14:27.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 10 monclient: get_monmap_and_config 2024-09-18T10:14:27.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 10 monclient: monmap: 2024-09-18T10:14:27.163 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:27.162894+0000 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:27.162894+0000 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:27.164 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 1 Processor -- start 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 1 -- start start 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.158+0000 7f6782df9700 10 monclient: init 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f677c09b2b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 0x7f677c09b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-c con 0x7f677c0955b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f677c098180 0x7f677c135980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-a con 0x7f677c098180 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677c0985a0 con 0x7f677c09b2b0 2024-09-18T10:14:27.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c09aab0 con 0x7f677c098180 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c09a7d0 con 0x7f677c0955b0 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6782df9700 10 monclient(hunting): authenticate will time out at 1149.256430s 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 0x7f677c09b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 0x7f677c09b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44856/0 (socket says 172.21.15.150:44856) 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 1 -- 172.21.15.150:0/1430292456 learned_addr learned my addr 172.21.15.150:0/1430292456 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): get_auth_request con 0x7f677c0955b0 auth_method 0 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f677bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 --2- 172.21.15.150:0/1430292456 >> v2:172.21.15.142:3300/0 conn(0x7f677c098180 0x7f677c135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): get_auth_request con 0x7f677c098180 auth_method 0 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient(hunting): handle_auth_done global_id 54399 payload 274 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 -- 172.21.15.150:0/1430292456 >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 msgr2=0x7f677c09b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 --2- 172.21.15.150:0/1430292456 >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 0x7f677c09b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 -- 172.21.15.150:0/1430292456 >> v1:172.21.15.150:6789/0 conn(0x7f677c09b2b0 legacy=0x7f677c097a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 monclient: found mon.noname-a 2024-09-18T10:14:27.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:27.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.162+0000 7f6781396700 1 -- 172.21.15.150:0/1430292456 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c0992f0 con 0x7f677c098180 2024-09-18T10:14:27.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6781396700 1 -- 172.21.15.150:0/1430292456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c098180 msgr2=0x7f677c135980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6781396700 1 --2- 172.21.15.150:0/1430292456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c098180 0x7f677c135980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67740098e0 tx=0x7f67740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caead23cd17e7e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 1 -- 172.21.15.150:0/1430292456 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011840 con 0x7f677c098180 2024-09-18T10:14:27.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: dump: 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 5 monclient: authenticate success, global_id 54399 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 1 -- 172.21.15.150:0/1430292456 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67740119a0 con 0x7f677c098180 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 1 -- 172.21.15.150:0/1430292456 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011d40 con 0x7f677c098180 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 4 set_mon_vals no callback set 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f677b7fe700 10 monclient: dump: 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.176 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: shutdown 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- 172.21.15.150:0/1430292456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c098180 msgr2=0x7f677c135980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- 172.21.15.150:0/1430292456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c098180 0x7f677c135980 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67740098e0 tx=0x7f67740091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- 172.21.15.150:0/1430292456 shutdown_connections 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- 172.21.15.150:0/1430292456 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c098180 0x7f677c135980 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- 172.21.15.150:0/1430292456 >> v2:172.21.15.150:3300/0 conn(0x7f677c0955b0 0x7f677c09b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- 172.21.15.150:0/1430292456 >> 172.21.15.150:0/1430292456 conn(0x7f677c0a15e0 msgr2=0x7f677c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- 172.21.15.150:0/1430292456 shutdown_connections 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- 172.21.15.150:0/1430292456 wait complete. 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: monmap: 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 Processor -- start 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- start start 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: init 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c0955b0 0x7f677c13da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-a con 0x7f677c0955b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-c con 0x7f677c098180 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): picked mon.noname-b con 0x7f677c09b2b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c06fba0 con 0x7f677c0955b0 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677c097730 con 0x7f677c09b2b0 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c08d4b0 con 0x7f677c098180 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.166+0000 7f6782df9700 10 monclient(hunting): authenticate will time out at 1149.263836s 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44870/0 (socket says 172.21.15.150:44870) 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 -- 172.21.15.150:0/3474911042 learned_addr learned my addr 172.21.15.150:0/3474911042 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): get_auth_request con 0x7f677c09b2b0 auth_method 0 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c0955b0 0x7f677c13da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): get_auth_request con 0x7f677c0955b0 auth_method 0 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 10 monclient(hunting): get_auth_request con 0x7f677c098180 auth_method 0 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6780b95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 msgr2=0x7f677c13df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 msgr2=0x7f677c13df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677bfff700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient(hunting): handle_auth_done global_id 15595 payload 824 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 msgr2=0x7f677c13df90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c0955b0 msgr2=0x7f677c13da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c0955b0 0x7f677c13da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 monclient: found mon.noname-b 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774009420 con 0x7f677c09b2b0 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 msgr2=0x7f677c089c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6781396700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f67740030c0 tx=0x7f6774016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677400bd40 con 0x7f677c09b2b0 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: dump: 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.185 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 5 monclient: authenticate success, global_id 15595 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774020070 con 0x7f677c09b2b0 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 10 monclient: _renew_subs 2024-09-18T10:14:27.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c08d140 con 0x7f677c09b2b0 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677401b6a0 con 0x7f677c09b2b0 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6778ff9700 10 monclient: dump: 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f677affd700 4 set_mon_vals no callback set 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 10 monclient: _renew_subs 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.170+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c08cdc0 con 0x7f677c09b2b0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6774022030 con 0x7f677c09b2b0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6782df9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6782df9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6782df9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c0990d0 con 0x7f677c09b2b0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6778ff9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 0x7f675c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(150..150 src has 58..150) v4 ==== 9122+0+0 (secure 0 0 0) 0x7f677404b350 con 0x7f677c09b2b0 2024-09-18T10:14:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6780b95700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 0x7f675c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.174+0000 7f6780b95700 10 monclient: get_auth_request con 0x7f675c035ac0 auth_method 0 2024-09-18T10:14:27.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.178+0000 7f6780b95700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 0x7f675c037f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f676c007fd0 tx=0x7f676c0026a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.182+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f677400bea0 con 0x7f677c09b2b0 2024-09-18T10:14:27.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.182+0000 7f6778ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.182+0000 7f6778ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:27.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.354+0000 7f6782df9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:27.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.354+0000 7f6782df9700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:27.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.354+0000 7f6782df9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:27.354+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f677c0990d0 con 0x7f677c09b2b0 2024-09-18T10:14:27.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.486+0000 7fe0737fe700 10 monclient: tick 2024-09-18T10:14:27.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.486+0000 7fe0737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.486+0000 7fe0737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c08e350 auth_method 0 2024-09-18T10:14:27.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 10 monclient(hunting): handle_auth_done global_id 64100 payload 824 2024-09-18T10:14:27.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe07c0913d0 con 0x7fe07c08e350 2024-09-18T10:14:27.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.506+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe064007b40 tx=0x7fe064009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06400c070 con 0x7fe07c08e350 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 10 monclient: dump: 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.513 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe064012c70 con 0x7fe07c08e350 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe064007b40 tx=0x7fe064009040 comp rx=0 tx=0).stop 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 msgr2=0x7fe07c08e720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:27.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.510+0000 7fe078ff9700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c08e350 0x7fe07c08e720 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe064007b40 tx=0x7fe064009040 comp rx=0 tx=0).stop 2024-09-18T10:14:27.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe082eaa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe082eaa700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 0x7fe07c08a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe082eaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0696a0 con 0x7fe07c089ff0 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 0x7fe07c08a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient: get_auth_request con 0x7fe07c089ff0 auth_method 0 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-18T10:14:27.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.522+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe07c0915a0 con 0x7fe07c089ff0 2024-09-18T10:14:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe080c46700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 msgr2=0x7fe07c08a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe080c46700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 0x7fe07c08a3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe06401c000 tx=0x7fe06401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06402c070 con 0x7fe07c089ff0 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 10 monclient: dump: 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.532 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fe06401bb50 con 0x7fe07c089ff0 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 msgr2=0x7fe07c08a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 0x7fe07c08a3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe06401c000 tx=0x7fe06401d000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 msgr2=0x7fe07c08a3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.526+0000 7fe078ff9700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe07c089ff0 0x7fe07c08a3f0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe06401c000 tx=0x7fe06401d000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:27.534 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.535 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.536 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:27.537 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:27.538 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:27.538 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:27.538 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:27.538 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:27.538 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.539 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.540 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.541 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.545 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:27.545 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:27.545 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.549 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:27.550 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:27.551 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe05c035ac0 msgr2=0x7fe05c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe05c035ac0 0x7fe05c037f70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe07400afd0 tx=0x7fe074017000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 10 monclient: shutdown 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 msgr2=0x7fe07c08ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe081447700 1 -- 172.21.15.142:0/2569329470 reap_dead start 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 0x7fe07c08ddb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe07c095170 tx=0x7fe06c0010c0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 shutdown_connections 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 --2- 172.21.15.142:0/2569329470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe07c08d9a0 0x7fe07c08ddb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 >> 172.21.15.142:0/2569329470 conn(0x7fe07c0a14e0 msgr2=0x7fe07c09c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 shutdown_connections 2024-09-18T10:14:27.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.530+0000 7fe082eaa700 1 -- 172.21.15.142:0/2569329470 wait complete. 2024-09-18T10:14:27.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:27.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:14:27.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:14:27.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 10 monclient: get_monmap_and_config 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 10 monclient: monmap: 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:27.666760+0000 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:27.666760+0000 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:27.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:27.669 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:27.669 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:27.669 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:27.669 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 1 Processor -- start 2024-09-18T10:14:27.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 1 -- start start 2024-09-18T10:14:27.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.662+0000 7f2be568e700 10 monclient: init 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2be009b0b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2be009b6f0 0x7f2be012edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-c con 0x7f2be009b6f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2be01339d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be009b480 con 0x7f2be01339d0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2be0098cd0 con 0x7f2be009b0b0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2be006f940 con 0x7f2be009b6f0 2024-09-18T10:14:27.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2be568e700 10 monclient(hunting): authenticate will time out at 1156.592711s 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bdf7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2be01339d0 0x7f2be0135db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49872/0 (socket says 172.21.15.142:49872) 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bdf7fe700 1 -- 172.21.15.142:0/1381685844 learned_addr learned my addr 172.21.15.142:0/1381685844 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 1 --2- 172.21.15.142:0/1381685844 >> v2:172.21.15.150:3300/0 conn(0x7f2be009b6f0 0x7f2be012edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): get_auth_request con 0x7f2be009b6f0 auth_method 0 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_done global_id 15598 payload 274 2024-09-18T10:14:27.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1381685844 >> v1:172.21.15.150:6789/0 conn(0x7f2be009b0b0 legacy=0x7f2be012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1381685844 >> v1:172.21.15.142:6789/0 conn(0x7f2be01339d0 legacy=0x7f2be0135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.666+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1381685844 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be0099110 con 0x7f2be009b6f0 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1381685844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be009b6f0 msgr2=0x7f2be012edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bd7fff700 1 --2- 172.21.15.142:0/1381685844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be009b6f0 0x7f2be012edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00098e0 tx=0x7f2bd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40773b2725bd56ff server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 1 -- 172.21.15.142:0/1381685844 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd000ba70 con 0x7f2be009b6f0 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:27.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: dump: 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 5 monclient: authenticate success, global_id 15598 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 1 -- 172.21.15.142:0/1381685844 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd000bbd0 con 0x7f2be009b6f0 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 4 set_mon_vals no callback set 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 1 -- 172.21.15.142:0/1381685844 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd00183f0 con 0x7f2be009b6f0 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2bde7fc700 10 monclient: dump: 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: shutdown 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- 172.21.15.142:0/1381685844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be009b6f0 msgr2=0x7f2be012edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 --2- 172.21.15.142:0/1381685844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be009b6f0 0x7f2be012edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00098e0 tx=0x7f2bd00091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- 172.21.15.142:0/1381685844 shutdown_connections 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 --2- 172.21.15.142:0/1381685844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be009b6f0 0x7f2be012edb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- 172.21.15.142:0/1381685844 >> 172.21.15.142:0/1381685844 conn(0x7f2be00a14e0 msgr2=0x7f2be00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- 172.21.15.142:0/1381685844 shutdown_connections 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- 172.21.15.142:0/1381685844 wait complete. 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: monmap: 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:27.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 Processor -- start 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- start start 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: init 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2be009b0b0 0x7f2be0141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-a con 0x7f2be009b0b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 0x7f2be0142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-c con 0x7f2be009b6f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01339d0 0x7f2be01425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): picked mon.noname-b con 0x7f2be01339d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:27.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be006fa90 con 0x7f2be009b0b0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be0136c50 con 0x7f2be01339d0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be0136470 con 0x7f2be009b6f0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.670+0000 7f2be568e700 10 monclient(hunting): authenticate will time out at 1156.599984s 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 0x7f2be0142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2be009b0b0 0x7f2be0141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 0x7f2be0142070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59520/0 (socket says 172.21.15.142:59520) 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01339d0 0x7f2be01425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1657606541 learned_addr learned my addr 172.21.15.142:0/1657606541 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:27.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): get_auth_request con 0x7f2be009b6f0 auth_method 0 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): get_auth_request con 0x7f2be009b0b0 auth_method 0 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): get_auth_request con 0x7f2be01339d0 auth_method 0 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient(hunting): handle_auth_done global_id 64106 payload 824 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01339d0 msgr2=0x7f2be01425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01339d0 0x7f2be01425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2be009b0b0 msgr2=0x7f2be0141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2be009b0b0 0x7f2be0141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0009420 con 0x7f2be009b6f0 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdf7fe700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01339d0 0x7f2be01425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bdeffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2be009b0b0 0x7f2be0141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 msgr2=0x7f2be0142070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd7fff700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 0x7f2be0142070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00030c0 tx=0x7f2bd0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd001d070 con 0x7f2be009b6f0 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:27.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: dump: 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd0022470 con 0x7f2be009b6f0 2024-09-18T10:14:27.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 5 monclient: authenticate success, global_id 64106 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 10 monclient: _renew_subs 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be0138280 con 0x7f2be009b6f0 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0022810 con 0x7f2be009b6f0 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bd6ffd700 10 monclient: dump: 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.692 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2bddffb700 4 set_mon_vals no callback set 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 10 monclient: _renew_subs 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.674+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be0137040 con 0x7f2be009b6f0 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2be568e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2be568e700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 0x7f2be0131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be0138140 con 0x7f2be01317b0 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 0x7f2be0131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient: get_auth_request con 0x7f2be01317b0 auth_method 0 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2bd00103f0 con 0x7f2be009b6f0 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_done global_id 15604 payload 824 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdeffd700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2be00696a0 con 0x7f2be01317b0 2024-09-18T10:14:27.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bd6ffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bc0035ac0 0x7f2bc0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(150..150 src has 58..150) v4 ==== 9122+0+0 (secure 0 0 0) 0x7f2bd004aea0 con 0x7f2be009b6f0 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdf7fe700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bc0035ac0 0x7f2bc0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.678+0000 7f2bdf7fe700 10 monclient: get_auth_request con 0x7f2bc0035ac0 auth_method 0 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bdeffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 msgr2=0x7f2be0131b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bdeffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 0x7f2be0131b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800fb30 tx=0x7f2bc8007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc800dbc0 con 0x7f2be01317b0 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 10 monclient: dump: 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.697 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bdf7fe700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bc0035ac0 0x7f2bc0037f70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800d6a0 tx=0x7f2bd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2bc800dd20 con 0x7f2be01317b0 2024-09-18T10:14:27.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:27.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 msgr2=0x7f2be0131b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 0x7f2be0131b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800fb30 tx=0x7f2bc8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 msgr2=0x7f2be0131b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:27.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.682+0000 7f2bd6ffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01317b0 0x7f2be0131b80 secure :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800fb30 tx=0x7f2bc8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2be568e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2be568e700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 0x7f2be0134d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2be568e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be006aa30 con 0x7f2be01320c0 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 0x7f2be0134d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient: get_auth_request con 0x7f2be01320c0 auth_method 0 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 10 monclient(hunting): handle_auth_done global_id 15607 payload 824 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.694+0000 7f2bdeffd700 1 -- 172.21.15.142:0/1657606541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2be0067950 con 0x7f2be01320c0 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bdeffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 msgr2=0x7f2be0134d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bdeffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 0x7f2be0134d70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8019000 tx=0x7f2bc801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc801eba0 con 0x7f2be01320c0 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 10 monclient: dump: 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.702 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f2bc801ed00 con 0x7f2be01320c0 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 msgr2=0x7f2be0134d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 0x7f2be0134d70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8019000 tx=0x7f2bc801a000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 msgr2=0x7f2be0134d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:27.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bd6ffd700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2be01320c0 0x7f2be0134d70 secure :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8019000 tx=0x7f2bc801a000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:27.704 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.705 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.706 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:27.707 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:27.708 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.709 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:27.710 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:27.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:27.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:27.715 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:27.716 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:27.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bc0035ac0 msgr2=0x7f2bc0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bc0035ac0 0x7f2bc0037f70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800d6a0 tx=0x7f2bd8009040 comp rx=0 tx=0).stop 2024-09-18T10:14:27.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 10 monclient: shutdown 2024-09-18T10:14:27.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 msgr2=0x7f2be0142070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 --2- 172.21.15.142:0/1657606541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2be009b6f0 0x7f2be0142070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00030c0 tx=0x7f2bd0017000 comp rx=0 tx=0).stop 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bdf7fe700 1 -- 172.21.15.142:0/1657606541 reap_dead start 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2bdf7fe700 1 -- 172.21.15.142:0/1657606541 reap_dead start 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 shutdown_connections 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 >> 172.21.15.142:0/1657606541 conn(0x7f2be00a14e0 msgr2=0x7f2be009f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 shutdown_connections 2024-09-18T10:14:27.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.698+0000 7f2be568e700 1 -- 172.21.15.142:0/1657606541 wait complete. 2024-09-18T10:14:27.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:27.718 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:14:27.719 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.818+0000 7f0ecffff700 10 monclient: get_monmap_and_config 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.818+0000 7f0ecffff700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.818+0000 7f0ecffff700 10 monclient: monmap: 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:27.823226+0000 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:27.823226+0000 2024-09-18T10:14:27.824 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:27.825 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 Processor -- start 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 -- start start 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient: init 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ed0132030 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ed0132400 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0ed0138a80 0x7f0ed013ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0ed0138a80 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00a17b0 con 0x7f0ed0132030 2024-09-18T10:14:27.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed0132b90 con 0x7f0ed0132400 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed00a1a90 con 0x7f0ed0138a80 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecffff700 10 monclient(hunting): authenticate will time out at 1156.748853s 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0eceffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0ed0132030 0x7f0ed009ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49880/0 (socket says 172.21.15.142:49880) 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0eceffd700 1 -- 172.21.15.142:0/25465817 learned_addr learned my addr 172.21.15.142:0/25465817 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 1 --2- 172.21.15.142:0/25465817 >> v2:172.21.15.150:3300/0 conn(0x7f0ed0138a80 0x7f0ed013ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): get_auth_request con 0x7f0ed0138a80 auth_method 0 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_done global_id 15610 payload 274 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 1 -- 172.21.15.142:0/25465817 >> v1:172.21.15.150:6789/0 conn(0x7f0ed0132400 legacy=0x7f0ed0137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 1 -- 172.21.15.142:0/25465817 >> v1:172.21.15.142:6789/0 conn(0x7f0ed0132030 legacy=0x7f0ed009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 1 monclient: found mon.noname-c 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:27.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.822+0000 7f0ecf7fe700 1 -- 172.21.15.142:0/25465817 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00a00f0 con 0x7f0ed0138a80 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecf7fe700 1 -- 172.21.15.142:0/25465817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0138a80 msgr2=0x7f0ed013ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecf7fe700 1 --2- 172.21.15.142:0/25465817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0138a80 0x7f0ed013ae60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009e60 tx=0x7f0ec8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e9fb1901206a656 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 1 -- 172.21.15.142:0/25465817 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800e070 con 0x7f0ed0138a80 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: dump: 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.832 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 1 -- 172.21.15.142:0/25465817 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec8012430 con 0x7f0ed0138a80 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 5 monclient: authenticate success, global_id 15610 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:27.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 4 set_mon_vals no callback set 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 1 -- 172.21.15.142:0/25465817 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80127d0 con 0x7f0ed0138a80 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ece7fc700 10 monclient: dump: 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.834 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 10 monclient: shutdown 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- 172.21.15.142:0/25465817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0138a80 msgr2=0x7f0ed013ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 --2- 172.21.15.142:0/25465817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0138a80 0x7f0ed013ae60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009e60 tx=0x7f0ec8009780 comp rx=0 tx=0).stop 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- 172.21.15.142:0/25465817 shutdown_connections 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 --2- 172.21.15.142:0/25465817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0138a80 0x7f0ed013ae60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- 172.21.15.142:0/25465817 >> 172.21.15.142:0/25465817 conn(0x7f0ed00a5700 msgr2=0x7f0ed00a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- 172.21.15.142:0/25465817 shutdown_connections 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- 172.21.15.142:0/25465817 wait complete. 2024-09-18T10:14:27.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 10 monclient: build_initial_monmap 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 10 monclient: monmap: 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:27.836 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:27.837 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 Processor -- start 2024-09-18T10:14:27.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 1 -- start start 2024-09-18T10:14:27.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.826+0000 7f0ecffff700 10 monclient: init 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0132030 0x7f0ed006c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-b con 0x7f0ed0132030 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-c con 0x7f0ed0132400 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 0x7f0ed006cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): picked mon.noname-a con 0x7f0ed0138a80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed009f6e0 con 0x7f0ed0138a80 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00a49e0 con 0x7f0ed0132030 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed011c0e0 con 0x7f0ed0132400 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 10 monclient(hunting): authenticate will time out at 1156.756319s 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 0x7f0ed006cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59530/0 (socket says 172.21.15.142:59530) 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): get_auth_request con 0x7f0ed0138a80 auth_method 0 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 -- 172.21.15.142:0/126265290 learned_addr learned my addr 172.21.15.142:0/126265290 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): get_auth_request con 0x7f0ed0132400 auth_method 0 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0132030 0x7f0ed006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): get_auth_request con 0x7f0ed0132030 auth_method 0 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0eceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient(hunting): handle_auth_done global_id 64109 payload 824 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 -- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0132030 msgr2=0x7f0ed006c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0132030 0x7f0ed006c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 -- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 msgr2=0x7f0ed006cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 0x7f0ed006cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 monclient: found mon.noname-c 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 -- 172.21.15.142:0/126265290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec8009a20 con 0x7f0ed0132400 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecf7fe700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 0x7f0ed006cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 -- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 msgr2=0x7f0ed006c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec67fc700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800b640 tx=0x7f0eb8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800f720 con 0x7f0ed0132400 2024-09-18T10:14:27.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 10 monclient: dump: 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.846 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ecffff700 5 monclient: authenticate success, global_id 64109 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.830+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb800f880 con 0x7f0ed0132400 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800fc40 con 0x7f0ed0132400 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _renew_subs 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed011be00 con 0x7f0ed0132400 2024-09-18T10:14:27.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecdffb700 4 set_mon_vals no callback set 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 10 monclient: dump: 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:27.848 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _renew_subs 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed0131ba0 con 0x7f0ed0132400 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed0072a80 con 0x7f0ed0132400 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0eb8011610 con 0x7f0ed0132400 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 0x7f0eb0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(150..150 src has 58..150) v4 ==== 9122+0+0 (secure 0 0 0) 0x7f0eb8048f20 con 0x7f0ed0132400 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0eceffd700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 0x7f0eb0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:27.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.834+0000 7f0eceffd700 10 monclient: get_auth_request con 0x7f0eb0035ac0 auth_method 0 2024-09-18T10:14:27.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.838+0000 7f0eceffd700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 0x7f0eb0037f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009ad0 tx=0x7f0ec0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:27.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.842+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0eb8017a90 con 0x7f0ed0132400 2024-09-18T10:14:27.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.842+0000 7f0ec77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:27.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:27.842+0000 7f0ec77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:28.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.062+0000 7f0ecffff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:14:28.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.062+0000 7f0ecffff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:14:28.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.062+0000 7f0ecffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:28.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.062+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0ed00a0310 con 0x7f0ed0132400 2024-09-18T10:14:28.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.066+0000 7f0ec77fe700 1 -- 172.21.15.142:0/126265290 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0eb80172b0 con 0x7f0ed0132400 2024-09-18T10:14:28.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.066+0000 7f0ec77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:14:28.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.066+0000 7f0ec77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=1646282061,config=3140514009,health=1163051649,kv=4261215878,logm=3696314753}) 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=7,logm=68} crc {auth=960869166,config=3140514009,health=1974477175,kv=264744547,logm=3545587880}) 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=1646282061,config=3140514009,health=1163051649,kv=4261215878,logm=3696314753}) 2024-09-18T10:14:28.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=1646282061,config=3140514009,health=1974477175,kv=4261215878,logm=3169237716}) 2024-09-18T10:14:28.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.072 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100521145}) 2024-09-18T10:14:28.072 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455790755}) 2024-09-18T10:14:28.072 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.072 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100521145}) 2024-09-18T10:14:28.072 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.066+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=721747733}) 2024-09-18T10:14:28.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 msgr2=0x7f0eb0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:28.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 0x7f0eb0037f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009ad0 tx=0x7f0ec0009060 comp rx=0 tx=0).stop 2024-09-18T10:14:28.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 10 monclient: shutdown 2024-09-18T10:14:28.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:28.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 msgr2=0x7f0ed006c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800b640 tx=0x7f0eb8009120 comp rx=0 tx=0).stop 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 shutdown_connections 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0eb0035ac0 0x7f0eb0037f70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0ed0138a80 0x7f0ed006cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0ed0132400 0x7f0ed006c6c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:28.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 --2- 172.21.15.142:0/126265290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0ed0132030 0x7f0ed006c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:28.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 >> 172.21.15.142:0/126265290 conn(0x7f0ed00a5700 msgr2=0x7f0ed009ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:28.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 shutdown_connections 2024-09-18T10:14:28.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:28.070+0000 7f0ecffff700 1 -- 172.21.15.142:0/126265290 wait complete. 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167852877}) 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302718844}) 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167852877}) 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591727704}) 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768209888}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4033424623}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768209888}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.070+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376759503}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606717772}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075409102}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606717772}) 2024-09-18T10:14:28.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466870686}) 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128144417}) 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170204798}) 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128144417}) 2024-09-18T10:14:28.080 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.074+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913773430}) 2024-09-18T10:14:28.081 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.081 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549527060}) 2024-09-18T10:14:28.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528913487}) 2024-09-18T10:14:28.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549527060}) 2024-09-18T10:14:28.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115761962}) 2024-09-18T10:14:28.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3423139796}) 2024-09-18T10:14:28.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643375880}) 2024-09-18T10:14:28.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3423139796}) 2024-09-18T10:14:28.084 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520405827}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194594319}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155974380}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194594319}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.078+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327944747}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859465298}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16433111}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859465298}) 2024-09-18T10:14:28.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912993260}) 2024-09-18T10:14:28.087 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.087 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848615894}) 2024-09-18T10:14:28.087 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982635200}) 2024-09-18T10:14:28.088 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.088 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848615894}) 2024-09-18T10:14:28.088 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6100379}) 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456252744}) 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2498330629}) 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456252744}) 2024-09-18T10:14:28.089 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.082+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=121290289}) 2024-09-18T10:14:28.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:14:28.092 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.092 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400959196}) 2024-09-18T10:14:28.092 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2684223920}) 2024-09-18T10:14:28.092 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400959196}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725757710}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833291323}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402400524}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833291323}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597508838}) 2024-09-18T10:14:28.093 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147251929}) 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346798986}) 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147251929}) 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259042386}) 2024-09-18T10:14:28.094 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.095 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666481655}) 2024-09-18T10:14:28.095 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478795637}) 2024-09-18T10:14:28.095 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.095 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666481655}) 2024-09-18T10:14:28.095 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.086+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955096352}) 2024-09-18T10:14:28.097 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.097 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426912785}) 2024-09-18T10:14:28.097 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2606044027}) 2024-09-18T10:14:28.097 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.097 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426912785}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400547425}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145439277}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540100528}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145439277}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786122710}) 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.098 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315624217}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3778192350}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315624217}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3077445628}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006730690}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386868655}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006730690}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3664602890}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609791786}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=416780170}) 2024-09-18T10:14:28.099 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609791786}) 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.090+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2877666002}) 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557398457}) 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880909804}) 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557398457}) 2024-09-18T10:14:28.100 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495810308}) 2024-09-18T10:14:28.101 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.101 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1162738683}) 2024-09-18T10:14:28.101 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227513951}) 2024-09-18T10:14:28.101 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.101 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1162738683}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3196192070}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=3947306039,mdsmap=1531151339,mgr=1666935248,mgr_command_descs=47663135,mgr_metadata=2738435100}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=89,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1} crc {logm=3014618403,mdsmap=1531151339,mgr=288718270,mgr_command_descs=47663135,mgr_metadata=2738435100}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1} crc {logm=3947306039,mdsmap=1531151339,mgr=1666935248,mgr_command_descs=47663135,mgr_metadata=2738435100}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.094+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=3} crc {logm=306411155,mdsmap=1531151339,mgr=3316658068,mgr_metadata=2738435100,mgrstat=3756839451}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=735298944,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=4102354296,osd_snap=454132730,osdmap=1844049438}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=688831411,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=4102354296,osd_snap=454132730,osdmap=1625482421}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=73} crc {mgrstat=735298944,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=4102354296,osd_snap=454132730,osdmap=1844049438}) 2024-09-18T10:14:28.102 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.098+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=11,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=74} crc {mgrstat=2701755046,mon_config_key=2207406588,monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=4102354296,osd_snap=454132730,osdmap=2031162349}) 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=3593297963}) 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=75} crc {osdmap=4149659937}) 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=3593297963}) 2024-09-18T10:14:28.106 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:14:28.102+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=79} crc {osdmap=2084456195}) 2024-09-18T10:14:28.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.218+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v151) v1 ==== 119+0+0 (secure 0 0 0) 0x7f677401ad20 con 0x7f677c09b2b0 2024-09-18T10:14:28.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.218+0000 7f6778ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:28.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.218+0000 7f6778ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.330+0000 7f6782df9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.330+0000 7f6782df9700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.330+0000 7f6782df9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:28.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:28.330+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]} v 0) v1 -- 0x7f677c0990d0 con 0x7f677c09b2b0 2024-09-18T10:14:29.092 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:14:29.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:29.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.198+0000 7f00b95c5700 10 monclient: get_monmap_and_config 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.198+0000 7f00b95c5700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.198+0000 7f00b95c5700 10 monclient: monmap: 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:29.202712+0000 2024-09-18T10:14:29.204 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:29.202712+0000 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:29.205 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:29.206 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:29.206 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 Processor -- start 2024-09-18T10:14:29.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 -- start start 2024-09-18T10:14:29.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient: init 2024-09-18T10:14:29.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00b4132030 0x7f00b4132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-c con 0x7f00b4132030 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-a con 0x7f00b409ece0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:29.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-b con 0x7f00b409ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b409f960 con 0x7f00b409ece0 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b40a1a90 con 0x7f00b4132030 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4132940 con 0x7f00b409ae60 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b95c5700 10 monclient(hunting): authenticate will time out at 1158.128791s 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59536/0 (socket says 172.21.15.142:59536) 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43954/0 (socket says 172.21.15.142:43954) 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 -- 172.21.15.142:0/2662189168 learned_addr learned my addr 172.21.15.142:0/2662189168 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): get_auth_request con 0x7f00b409ae60 auth_method 0 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.150:3300/0 conn(0x7f00b4132030 0x7f00b4132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): get_auth_request con 0x7f00b409ece0 auth_method 0 2024-09-18T10:14:29.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): get_auth_request con 0x7f00b4132030 auth_method 0 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_done global_id 64112 payload 274 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 -- 172.21.15.142:0/2662189168 >> v2:172.21.15.150:3300/0 conn(0x7f00b4132030 msgr2=0x7f00b4132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.150:3300/0 conn(0x7f00b4132030 0x7f00b4132400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 -- 172.21.15.142:0/2662189168 >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 msgr2=0x7f00b409f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 monclient: found mon.noname-b 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00b8dc4700 1 -- 172.21.15.142:0/2662189168 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40a17b0 con 0x7f00b409ae60 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.202+0000 7f00ab7fe700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:29.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b8dc4700 1 -- 172.21.15.142:0/2662189168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 msgr2=0x7f00b409b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b8dc4700 1 --2- 172.21.15.142:0/2662189168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f00b406a4d0 tx=0x7f00ac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fda9c3fd326af4c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 1 -- 172.21.15.142:0/2662189168 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac00e070 con 0x7f00b409ae60 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: dump: 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.216 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 1 -- 172.21.15.142:0/2662189168 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ac00bb80 con 0x7f00b409ae60 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 5 monclient: authenticate success, global_id 64112 2024-09-18T10:14:29.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 1 -- 172.21.15.142:0/2662189168 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0092e0 con 0x7f00b409ae60 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 4 set_mon_vals no callback set 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b37fe700 10 monclient: dump: 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.219 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 10 monclient: shutdown 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- 172.21.15.142:0/2662189168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 msgr2=0x7f00b409b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 --2- 172.21.15.142:0/2662189168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f00b406a4d0 tx=0x7f00ac009840 comp rx=0 tx=0).stop 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- 172.21.15.142:0/2662189168 shutdown_connections 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 --2- 172.21.15.142:0/2662189168 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b409b2d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.142:3300/0 conn(0x7f00b409ece0 0x7f00b409f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 --2- 172.21.15.142:0/2662189168 >> v2:172.21.15.150:3300/0 conn(0x7f00b4132030 0x7f00b4132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- 172.21.15.142:0/2662189168 >> 172.21.15.142:0/2662189168 conn(0x7f00b40a5700 msgr2=0x7f00b40a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- 172.21.15.142:0/2662189168 shutdown_connections 2024-09-18T10:14:29.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- 172.21.15.142:0/2662189168 wait complete. 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 10 monclient: monmap: 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.227 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 Processor -- start 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 1 -- start start 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.206+0000 7f00b95c5700 10 monclient: init 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-c con 0x7f00b409ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b409ece0 0x7f00b4141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-b con 0x7f00b409ece0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 0x7f00b4141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): picked mon.noname-a con 0x7f00b4132030 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b411f530 con 0x7f00b4132030 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b4141f80 con 0x7f00b409ece0 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b411f250 con 0x7f00b409ae60 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b95c5700 10 monclient(hunting): authenticate will time out at 1158.136425s 2024-09-18T10:14:29.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 0x7f00b4141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59550/0 (socket says 172.21.15.142:59550) 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 0x7f00b4141a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43960/0 (socket says 172.21.15.142:43960) 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 -- 172.21.15.142:0/1115739729 learned_addr learned my addr 172.21.15.142:0/1115739729 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): get_auth_request con 0x7f00b409ae60 auth_method 0 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b409ece0 0x7f00b4141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): get_auth_request con 0x7f00b4132030 auth_method 0 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): get_auth_request con 0x7f00b409ece0 auth_method 0 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b8dc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 -- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b409ece0 msgr2=0x7f00b4141500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b409ece0 0x7f00b4141500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 -- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 msgr2=0x7f00b4141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 0x7f00b4141a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 -- 172.21.15.142:0/1115739729 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c009420 con 0x7f00b409ae60 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 -- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 msgr2=0x7f00b4131120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.210+0000 7f00b3fff700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00a400ec80 tx=0x7f00a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a40096b0 con 0x7f00b409ae60 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 10 monclient: dump: 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.233 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 5 monclient: authenticate success, global_id 64115 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 10 monclient: _renew_subs 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac009ae0 con 0x7f00b409ae60 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 10 monclient: _renew_subs 2024-09-18T10:14:29.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b411d790 con 0x7f00b409ae60 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a400c9e0 con 0x7f00b409ae60 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.214+0000 7f00b2ffd700 4 set_mon_vals no callback set 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400cda0 con 0x7f00b409ae60 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 10 monclient: dump: 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.235 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f00a4013680 con 0x7f00b409ae60 2024-09-18T10:14:29.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 0x7f0094037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f00a40497a0 con 0x7f00b409ae60 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b95c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b95c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b95c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.218+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00980050b0 con 0x7f00b409ae60 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.222+0000 7f00ab7fe700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 0x7f0094037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.222+0000 7f00ab7fe700 10 monclient: get_auth_request con 0x7f0094035b10 auth_method 0 2024-09-18T10:14:29.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.226+0000 7f00ab7fe700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 0x7f0094037fc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f009c009f40 tx=0x7f009c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.230+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f00a4023ad0 con 0x7f00b409ae60 2024-09-18T10:14:29.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.230+0000 7f00b0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.230+0000 7f00b0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.218+0000 7f6778ff9700 1 -- 172.21.15.150:0/3474911042 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}]=0 v152) v1 ==== 119+0+0 (secure 0 0 0) 0x7f6774010380 con 0x7f677c09b2b0 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.218+0000 7f6778ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [10.5]}] 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.218+0000 7f6778ff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 msgr2=0x7f675c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 0x7f675c037f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f676c007fd0 tx=0x7f676c0026a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 10 monclient: shutdown 2024-09-18T10:14:29.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 msgr2=0x7f677c089c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f67740030c0 tx=0x7f6774016000 comp rx=0 tx=0).stop 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 shutdown_connections 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f675c035ac0 0x7f675c037f70 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f676c007fd0 tx=0x7f676c0026a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f677c09b2b0 0x7f677c089c40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f677c098180 0x7f677c13df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 --2- 172.21.15.150:0/3474911042 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f677c0955b0 0x7f677c13da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.226+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 >> 172.21.15.150:0/3474911042 conn(0x7f677c0a15e0 msgr2=0x7f677c09fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.230+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 shutdown_connections 2024-09-18T10:14:29.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.230+0000 7f6782df9700 1 -- 172.21.15.150:0/3474911042 wait complete. 2024-09-18T10:14:29.249 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight cool osd.0 11.5 2024-09-18T10:14:29.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.346+0000 7fcf683a2700 10 monclient: get_monmap_and_config 2024-09-18T10:14:29.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.346+0000 7fcf683a2700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.346+0000 7fcf683a2700 10 monclient: monmap: 2024-09-18T10:14:29.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:29.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:29.350666+0000 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:29.350666+0000 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:29.352 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:29.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.346+0000 7fcf683a2700 1 Processor -- start 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 -- start start 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient: init 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-b con 0x7fcf6009acd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf6009bfd0 0x7fcf6009c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-c con 0x7fcf6009bfd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf6009c8e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf60095e70 con 0x7fcf6009c8e0 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf60096250 con 0x7fcf6009bfd0 2024-09-18T10:14:29.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60095920 con 0x7fcf6009acd0 2024-09-18T10:14:29.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf683a2700 10 monclient(hunting): authenticate will time out at 1151.444940s 2024-09-18T10:14:29.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf6009bfd0 0x7fcf6009c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6693f700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcf6009c8e0 0x7fcf60136780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:43230/0 (socket says 172.21.15.150:43230) 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:52958/0 (socket says 172.21.15.150:52958) 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6693f700 1 -- 172.21.15.150:0/4076501309 learned_addr learned my addr 172.21.15.150:0/4076501309 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): get_auth_request con 0x7fcf6009acd0 auth_method 0 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): get_auth_request con 0x7fcf6009bfd0 auth_method 0 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 -- 172.21.15.150:0/4076501309 >> v2:172.21.15.150:3300/0 conn(0x7fcf6009bfd0 msgr2=0x7fcf6009c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 --2- 172.21.15.150:0/4076501309 >> v2:172.21.15.150:3300/0 conn(0x7fcf6009bfd0 0x7fcf6009c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 -- 172.21.15.150:0/4076501309 >> v1:172.21.15.142:6789/0 conn(0x7fcf6009c8e0 legacy=0x7fcf60136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 monclient: found mon.noname-b 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.350+0000 7fcf6613e700 1 -- 172.21.15.150:0/4076501309 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6009b0a0 con 0x7fcf6009acd0 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 -- 172.21.15.150:0/4076501309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60096b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 --2- 172.21.15.150:0/4076501309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0098e0 tx=0x7fcf5c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9a22420815ed4ae server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 1 -- 172.21.15.150:0/4076501309 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00b850 con 0x7fcf6009acd0 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: dump: 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.364 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 5 monclient: authenticate success, global_id 64118 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 1 -- 172.21.15.150:0/4076501309 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c00b9b0 con 0x7fcf6009acd0 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 4 set_mon_vals no callback set 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 1 -- 172.21.15.150:0/4076501309 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00bd50 con 0x7fcf6009acd0 2024-09-18T10:14:29.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6513c700 10 monclient: dump: 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.366 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: shutdown 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- 172.21.15.150:0/4076501309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60096b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- 172.21.15.150:0/4076501309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0098e0 tx=0x7fcf5c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- 172.21.15.150:0/4076501309 shutdown_connections 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- 172.21.15.150:0/4076501309 >> v2:172.21.15.150:3300/0 conn(0x7fcf6009bfd0 0x7fcf6009c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- 172.21.15.150:0/4076501309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60096b40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- 172.21.15.150:0/4076501309 >> 172.21.15.150:0/4076501309 conn(0x7fcf600a15e0 msgr2=0x7fcf600a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- 172.21.15.150:0/4076501309 shutdown_connections 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- 172.21.15.150:0/4076501309 wait complete. 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: monmap: 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:29.368 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 Processor -- start 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- start start 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: init 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-c con 0x7fcf6009acd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf60141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-a con 0x7fcf6009bfd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf60145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): picked mon.noname-b con 0x7fcf6009c8e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:29.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6009e9c0 con 0x7fcf6009bfd0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60093850 con 0x7fcf6009c8e0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60126f80 con 0x7fcf6009acd0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf683a2700 10 monclient(hunting): authenticate will time out at 1151.450888s 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf60145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf60141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf60141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56794/0 (socket says 172.21.15.150:56794) 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:52960/0 (socket says 172.21.15.150:52960) 2024-09-18T10:14:29.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 1 -- 172.21.15.150:0/1511315584 learned_addr learned my addr 172.21.15.150:0/1511315584 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): get_auth_request con 0x7fcf6009bfd0 auth_method 0 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): get_auth_request con 0x7fcf6009acd0 auth_method 0 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): get_auth_request con 0x7fcf6009c8e0 auth_method 0 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 msgr2=0x7fcf60145d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf60145d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 msgr2=0x7fcf60141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf60141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 monclient: found mon.noname-c 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5c009420 con 0x7fcf6009acd0 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.354+0000 7fcf6593d700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf60141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c015000 tx=0x7fcf5c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c011b00 con 0x7fcf6009acd0 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: dump: 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.373 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 5 monclient: authenticate success, global_id 64121 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 10 monclient: _renew_subs 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c020070 con 0x7fcf6009acd0 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf60127260 con 0x7fcf6009acd0 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c01b450 con 0x7fcf6009acd0 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf57fff700 4 set_mon_vals no callback set 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 10 monclient: dump: 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.375 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 10 monclient: _renew_subs 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf601425c0 con 0x7fcf6009acd0 2024-09-18T10:14:29.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.358+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcf5c022030 con 0x7fcf6009acd0 2024-09-18T10:14:29.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf567fc700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 0x7fcf44037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7fcf5c04b1a0 con 0x7fcf6009acd0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf683a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf683a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf683a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf6593d700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 0x7fcf44037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6006f5a0 con 0x7fcf6009acd0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf6593d700 10 monclient: get_auth_request con 0x7fcf44035ac0 auth_method 0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.362+0000 7fcf6593d700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 0x7fcf44037f70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009fd0 tx=0x7fcf500091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.374+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcf6006f5a0 con 0x7fcf6009acd0 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.374+0000 7fcf567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.374+0000 7fcf567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:29.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b95c5700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:29.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b95c5700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:29.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b95c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f00980058e0 con 0x7f00b409ae60 2024-09-18T10:14:29.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b0ff9700 1 -- 172.21.15.142:0/1115739729 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f00a40232f0 con 0x7f00b409ae60 2024-09-18T10:14:29.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:29.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.454+0000 7f00b0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:29.460 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:29.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 msgr2=0x7f0094037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 0x7f0094037fc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f009c009f40 tx=0x7f009c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 10 monclient: shutdown 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 msgr2=0x7f00b4131120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00a400ec80 tx=0x7f00a40091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 shutdown_connections 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0094035b10 0x7f0094037fc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00b4132030 0x7f00b4141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b409ece0 0x7f00b4141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 --2- 172.21.15.142:0/1115739729 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00b409ae60 0x7f00b4131120 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 >> 172.21.15.142:0/1115739729 conn(0x7f00b40a5700 msgr2=0x7f00b409d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 shutdown_connections 2024-09-18T10:14:29.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.458+0000 7f00b95c5700 1 -- 172.21.15.142:0/1115739729 wait complete. 2024-09-18T10:14:29.474 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:14:29.474 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:14:29.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:14:29.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:14:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.546+0000 7fcf683a2700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.546+0000 7fcf683a2700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.546+0000 7fcf683a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:29.546+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7fcf6006f5a0 con 0x7fcf6009acd0 2024-09-18T10:14:29.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: get_monmap_and_config 2024-09-18T10:14:29.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: monmap: 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:29.576990+0000 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:29.576990+0000 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:29.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 Processor -- start 2024-09-18T10:14:29.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 -- start start 2024-09-18T10:14:29.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: init 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ab006f3b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ab00990d0 0x7f9ab0134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-a con 0x7f9ab00990d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9ab00994f0 0x7f9ab0136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-c con 0x7f9ab00994f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab009c330 con 0x7f9ab006f3b0 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab009b380 con 0x7f9ab00990d0 2024-09-18T10:14:29.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab009b030 con 0x7f9ab00994f0 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab507c700 10 monclient(hunting): authenticate will time out at 1158.502991s 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9ab006f3b0 0x7f9ab00989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49890/0 (socket says 172.21.15.142:49890) 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaffff700 1 -- 172.21.15.142:0/273229146 learned_addr learned my addr 172.21.15.142:0/273229146 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/273229146 >> v2:172.21.15.142:3300/0 conn(0x7f9ab00990d0 0x7f9ab0134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request con 0x7f9ab00990d0 auth_method 0 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 1 --2- 172.21.15.142:0/273229146 >> v2:172.21.15.150:3300/0 conn(0x7f9ab00994f0 0x7f9ab0136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): get_auth_request con 0x7f9ab00994f0 auth_method 0 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_done global_id 54420 payload 274 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/273229146 >> v2:172.21.15.150:3300/0 conn(0x7f9ab00994f0 msgr2=0x7f9ab0136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/273229146 >> v2:172.21.15.150:3300/0 conn(0x7f9ab00994f0 0x7f9ab0136bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/273229146 >> v1:172.21.15.142:6789/0 conn(0x7f9ab006f3b0 legacy=0x7f9ab00989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 monclient: found mon.noname-a 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.574+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/273229146 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab009c530 con 0x7f9ab00990d0 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/273229146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00990d0 msgr2=0x7f9ab0134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/273229146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00990d0 0x7f9ab0134670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a7f0 tx=0x7f9a980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1465ee540eaa877 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 1 -- 172.21.15.142:0/273229146 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98011840 con 0x7f9ab00990d0 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: dump: 2024-09-18T10:14:29.601 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 1 -- 172.21.15.142:0/273229146 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a980119a0 con 0x7f9ab00990d0 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 5 monclient: authenticate success, global_id 54420 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 4 set_mon_vals no callback set 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 1 -- 172.21.15.142:0/273229146 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98011d40 con 0x7f9ab00990d0 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9aaeffd700 10 monclient: dump: 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 10 monclient: shutdown 2024-09-18T10:14:29.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 -- 172.21.15.142:0/273229146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00990d0 msgr2=0x7f9ab0134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 --2- 172.21.15.142:0/273229146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00990d0 0x7f9ab0134670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a7f0 tx=0x7f9a980091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 -- 172.21.15.142:0/273229146 shutdown_connections 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 --2- 172.21.15.142:0/273229146 >> v2:172.21.15.150:3300/0 conn(0x7f9ab00994f0 0x7f9ab0136bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 --2- 172.21.15.142:0/273229146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00990d0 0x7f9ab0134670 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 -- 172.21.15.142:0/273229146 >> 172.21.15.142:0/273229146 conn(0x7f9ab0070ef0 msgr2=0x7f9ab00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 -- 172.21.15.142:0/273229146 shutdown_connections 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 1 -- 172.21.15.142:0/273229146 wait complete. 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.578+0000 7f9ab507c700 10 monclient: monmap: 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.605 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 Processor -- start 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 -- start start 2024-09-18T10:14:29.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient: init 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 0x7f9ab013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-c con 0x7f9ab006f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab00990d0 0x7f9ab013fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-b con 0x7f9ab00990d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:29.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00994f0 0x7f9ab01401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): picked mon.noname-a con 0x7f9ab00994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab008f150 con 0x7f9ab00994f0 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab008f7a0 con 0x7f9ab00990d0 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab008d690 con 0x7f9ab006f3b0 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab507c700 10 monclient(hunting): authenticate will time out at 1158.509250s 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 0x7f9ab013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 0x7f9ab013f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59558/0 (socket says 172.21.15.142:59558) 2024-09-18T10:14:29.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00994f0 0x7f9ab01401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 -- 172.21.15.142:0/730897254 learned_addr learned my addr 172.21.15.142:0/730897254 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): get_auth_request con 0x7f9ab006f3b0 auth_method 0 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab00990d0 0x7f9ab013fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): get_auth_request con 0x7f9ab00994f0 auth_method 0 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request con 0x7f9ab00990d0 auth_method 0 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9ab487b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient(hunting): handle_auth_done global_id 64124 payload 824 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab00990d0 msgr2=0x7f9ab013fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab00990d0 0x7f9ab013fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00994f0 msgr2=0x7f9ab01401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00994f0 0x7f9ab01401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 monclient: found mon.noname-c 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.582+0000 7f9aaffff700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98009420 con 0x7f9ab006f3b0 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaffff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 msgr2=0x7f9ab013f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaffff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 0x7f9ab013f740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000cca0 tx=0x7f9aa0005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa000ab20 con 0x7f9ab006f3b0 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: dump: 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.612 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 5 monclient: authenticate success, global_id 64124 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 monclient: _renew_subs 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa000aca0 con 0x7f9ab006f3b0 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab006eda0 con 0x7f9ab006f3b0 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0007340 con 0x7f9ab006f3b0 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 10 monclient: dump: 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.614 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aadffb700 4 set_mon_vals no callback set 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 monclient: _renew_subs 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab006fa20 con 0x7f9ab006f3b0 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 0x7f9ab00883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00648a0 con 0x7f9ab0087fe0 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 0x7f9ab00883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient: get_auth_request con 0x7f9ab0087fe0 auth_method 0 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9aa00076f0 con 0x7f9ab006f3b0 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a90035ac0 0x7f9a90037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f9aa00499e0 con 0x7f9ab006f3b0 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_done global_id 54426 payload 824 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ab008e380 con 0x7f9ab0087fe0 2024-09-18T10:14:29.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab487b700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a90035ac0 0x7f9a90037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.586+0000 7f9ab487b700 10 monclient: get_auth_request con 0x7f9a90035ac0 auth_method 0 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 msgr2=0x7f9ab00883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 0x7f9ab00883b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a7f0 tx=0x7f9a98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98021070 con 0x7f9ab0087fe0 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 10 monclient: dump: 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.618 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9a9800be00 con 0x7f9ab0087fe0 2024-09-18T10:14:29.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9ab487b700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a90035ac0 0x7f9a90037f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009db0 tx=0x7f9aa80091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 msgr2=0x7f9ab00883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 0x7f9ab00883b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a7f0 tx=0x7f9a98016000 comp rx=0 tx=0).stop 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 msgr2=0x7f9ab00883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:29.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.590+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab0087fe0 0x7f9ab00883b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9ab507c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:29.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9ab507c700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 0x7f9ab008bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9ab507c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0072a80 con 0x7f9ab00888f0 2024-09-18T10:14:29.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 0x7f9ab008bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient: get_auth_request con 0x7f9ab00888f0 auth_method 0 2024-09-18T10:14:29.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 10 monclient(hunting): handle_auth_done global_id 54429 payload 824 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.598+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/730897254 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9ab00678a0 con 0x7f9ab00888f0 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aaf7fe700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 msgr2=0x7f9ab008bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aaf7fe700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 0x7f9ab008bb00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009860 tx=0x7f9a9801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98034470 con 0x7f9ab00888f0 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 10 monclient: dump: 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.628 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.629 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.629 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.629 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.629 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.629 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f9a9800bbf0 con 0x7f9ab00888f0 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:29.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 msgr2=0x7f9ab008bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 0x7f9ab008bb00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009860 tx=0x7f9a9801e070 comp rx=0 tx=0).stop 2024-09-18T10:14:29.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 msgr2=0x7f9ab008bb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:29.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ab00888f0 0x7f9ab008bb00 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009860 tx=0x7f9a9801e070 comp rx=0 tx=0).stop 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:29.632 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:29.633 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.634 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.635 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:29.636 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.637 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:29.638 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.640 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:29.641 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.644 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.645 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:29.646 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a90035ac0 msgr2=0x7f9a90037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a90035ac0 0x7f9a90037f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009db0 tx=0x7f9aa80091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 10 monclient: shutdown 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 msgr2=0x7f9ab013f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab507c700 1 --2- 172.21.15.142:0/730897254 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ab006f3b0 0x7f9ab013f740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000cca0 tx=0x7f9aa0005480 comp rx=0 tx=0).stop 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab487b700 1 -- 172.21.15.142:0/730897254 reap_dead start 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.602+0000 7f9ab487b700 1 -- 172.21.15.142:0/730897254 reap_dead start 2024-09-18T10:14:29.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.606+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 shutdown_connections 2024-09-18T10:14:29.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.606+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 >> 172.21.15.142:0/730897254 conn(0x7f9ab0070ef0 msgr2=0x7f9ab006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.606+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 shutdown_connections 2024-09-18T10:14:29.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.606+0000 7f9ab507c700 1 -- 172.21.15.142:0/730897254 wait complete. 2024-09-18T10:14:29.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:29.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:14:29.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: get_monmap_and_config 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: monmap: 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:29.764242+0000 2024-09-18T10:14:29.765 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:29.764242+0000 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:29.766 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 Processor -- start 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 -- start start 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: init 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:14:29.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e5c09a6a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c09b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c06f7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e5c09b6c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c09dae0 con 0x7f5e5c09a6a0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c099bc0 con 0x7f5e5c09b6c0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06fbc0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e60929700 10 monclient(hunting): authenticate will time out at 1158.689771s 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c09b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e5b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f5e5c09a6a0 0x7f5e5c09aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49902/0 (socket says 172.21.15.142:49902) 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e5b7fe700 1 -- 172.21.15.142:0/306098757 learned_addr learned my addr 172.21.15.142:0/306098757 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request con 0x7f5e5c06f7f0 auth_method 0 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_done global_id 64127 payload 274 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 1 -- 172.21.15.142:0/306098757 >> v1:172.21.15.150:6789/0 conn(0x7f5e5c09b6c0 legacy=0x7f5e5c13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 1 -- 172.21.15.142:0/306098757 >> v1:172.21.15.142:6789/0 conn(0x7f5e5c09a6a0 legacy=0x7f5e5c09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 1 monclient: found mon.noname-b 2024-09-18T10:14:29.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.762+0000 7f5e52ffd700 1 -- 172.21.15.142:0/306098757 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0998e0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e52ffd700 1 -- 172.21.15.142:0/306098757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 msgr2=0x7f5e5c09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/306098757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c09b180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6869d32a55e171b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 1 -- 172.21.15.142:0/306098757 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c00ba70 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: dump: 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.775 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 5 monclient: authenticate success, global_id 64127 2024-09-18T10:14:29.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 1 -- 172.21.15.142:0/306098757 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c00bbd0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 4 set_mon_vals no callback set 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 1 -- 172.21.15.142:0/306098757 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c011420 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e5affd700 10 monclient: dump: 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.777 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.778 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 10 monclient: shutdown 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- 172.21.15.142:0/306098757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 msgr2=0x7f5e5c09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 --2- 172.21.15.142:0/306098757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c09b180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- 172.21.15.142:0/306098757 shutdown_connections 2024-09-18T10:14:29.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 --2- 172.21.15.142:0/306098757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c09b180 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- 172.21.15.142:0/306098757 >> 172.21.15.142:0/306098757 conn(0x7f5e5c070ef0 msgr2=0x7f5e5c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- 172.21.15.142:0/306098757 shutdown_connections 2024-09-18T10:14:29.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- 172.21.15.142:0/306098757 wait complete. 2024-09-18T10:14:29.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 10 monclient: monmap: 2024-09-18T10:14:29.781 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.781 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:29.781 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:29.781 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:29.781 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 Processor -- start 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 1 -- start start 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.766+0000 7f5e60929700 10 monclient: init 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-c con 0x7f5e5c06f7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e5c09a6a0 0x7f5e5c06c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-b con 0x7f5e5c09a6a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:29.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 0x7f5e5c06cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): picked mon.noname-a con 0x7f5e5c09b6c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:29.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06eac0 con 0x7f5e5c09b6c0 2024-09-18T10:14:29.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06e7e0 con 0x7f5e5c09a6a0 2024-09-18T10:14:29.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c094830 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e60929700 10 monclient(hunting): authenticate will time out at 1158.696239s 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 0x7f5e5c06cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 0x7f5e5c06cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43984/0 (socket says 172.21.15.142:43984) 2024-09-18T10:14:29.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59574/0 (socket says 172.21.15.142:59574) 2024-09-18T10:14:29.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 1 -- 172.21.15.142:0/2851879064 learned_addr learned my addr 172.21.15.142:0/2851879064 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:29.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e5c09a6a0 0x7f5e5c06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): get_auth_request con 0x7f5e5c09b6c0 auth_method 0 2024-09-18T10:14:29.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): get_auth_request con 0x7f5e5c06f7f0 auth_method 0 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request con 0x7f5e5c09a6a0 auth_method 0 2024-09-18T10:14:29.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient(hunting): handle_auth_done global_id 64130 payload 824 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e5c09a6a0 msgr2=0x7f5e5c06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e5c09a6a0 0x7f5e5c06c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 msgr2=0x7f5e5c06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 0x7f5e5c06cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:14:29.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c009420 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e5c09a6a0 0x7f5e5c06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:29.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5bfff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c09b6c0 0x7f5e5c06cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:29.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 msgr2=0x7f5e5c06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.770+0000 7f5e5b7fe700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c560 tx=0x7f5e44007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44005d40 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: dump: 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.803 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.804 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.804 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.804 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.804 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e440070d0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 5 monclient: authenticate success, global_id 64130 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 monclient: _renew_subs 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e5a7fc700 4 set_mon_vals no callback set 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44007490 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c0a0580 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 10 monclient: dump: 2024-09-18T10:14:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.809 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.809 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.809 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 monclient: _renew_subs 2024-09-18T10:14:29.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c126fc0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 0x7f5e5c092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0648a0 con 0x7f5e5c06d2b0 2024-09-18T10:14:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 0x7f5e5c092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient: get_auth_request con 0x7f5e5c06d2b0 auth_method 0 2024-09-18T10:14:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e52ffd700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e5c126a60 con 0x7f5e5c06d2b0 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5e4401b3b0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e3c035ac0 0x7f5e3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f5e440489e0 con 0x7f5e5c06f7f0 2024-09-18T10:14:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e5bfff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e3c035ac0 0x7f5e3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.774+0000 7f5e5bfff700 10 monclient: get_auth_request con 0x7f5e3c035ac0 auth_method 0 2024-09-18T10:14:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e52ffd700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 msgr2=0x7f5e5c092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 0x7f5e5c092800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c021070 con 0x7f5e5c06d2b0 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 10 monclient: dump: 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5e4c01c950 con 0x7f5e5c06d2b0 2024-09-18T10:14:29.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 msgr2=0x7f5e5c092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 0x7f5e5c092800 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009910 tx=0x7f5e4c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 msgr2=0x7f5e5c092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e53fff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06d2b0 0x7f5e5c092800 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.778+0000 7f5e5bfff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e3c035ac0 0x7f5e3c037f70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c06e740 tx=0x7f5e540026a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.786+0000 7f5e60929700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.786+0000 7f5e60929700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 0x7f5e5c093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e60929700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c072a80 con 0x7f5e5c092d40 2024-09-18T10:14:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 0x7f5e5c093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient: get_auth_request con 0x7f5e5c092d40 auth_method 0 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 10 monclient(hunting): handle_auth_done global_id 64136 payload 824 2024-09-18T10:14:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 1 -- 172.21.15.142:0/2851879064 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5e5c06d680 con 0x7f5e5c092d40 2024-09-18T10:14:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 msgr2=0x7f5e5c093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e52ffd700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 0x7f5e5c093140 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009860 tx=0x7f5e4c01e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c038070 con 0x7f5e5c092d40 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 10 monclient: dump: 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f5e4c01c740 con 0x7f5e5c092d40 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 msgr2=0x7f5e5c093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 0x7f5e5c093140 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009860 tx=0x7f5e4c01e070 comp rx=0 tx=0).stop 2024-09-18T10:14:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 msgr2=0x7f5e5c093140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.790+0000 7f5e53fff700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c092d40 0x7f5e5c093140 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009860 tx=0x7f5e4c01e070 comp rx=0 tx=0).stop 2024-09-18T10:14:29.825 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:29.825 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:14:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:29.826 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:29.827 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.828 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.829 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.830 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:29.832 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:29.833 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:29.834 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:29.835 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:29.836 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:29.837 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.839 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.841 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:29.842 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.843 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:29.844 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:14:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:29.845 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:29.845 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:29.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e3c035ac0 msgr2=0x7f5e3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e3c035ac0 0x7f5e3c037f70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c06e740 tx=0x7f5e540026a0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 10 monclient: shutdown 2024-09-18T10:14:29.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:29.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 msgr2=0x7f5e5c06c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c560 tx=0x7f5e44007f90 comp rx=0 tx=0).stop 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e5bfff700 1 -- 172.21.15.142:0/2851879064 reap_dead start 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 shutdown_connections 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 --2- 172.21.15.142:0/2851879064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e5c06f7f0 0x7f5e5c06c2f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 >> 172.21.15.142:0/2851879064 conn(0x7f5e5c070ef0 msgr2=0x7f5e5c09c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 shutdown_connections 2024-09-18T10:14:29.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.794+0000 7f5e60929700 1 -- 172.21.15.142:0/2851879064 wait complete. 2024-09-18T10:14:29.847 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:29.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:14:29.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:14:29.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: get_monmap_and_config 2024-09-18T10:14:29.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: build_initial_monmap 2024-09-18T10:14:29.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: monmap: 2024-09-18T10:14:29.982 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:29.982 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.982 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:29.961422+0000 2024-09-18T10:14:29.982 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:29.961422+0000 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:29.983 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 Processor -- start 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 -- start start 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: init 2024-09-18T10:14:29.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5a2009abd0 0x7f5a200960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-a con 0x7f5a2009abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a2009b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:29.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-b con 0x7f5a2009c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2009dc90 con 0x7f5a2009b740 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2009b500 con 0x7f5a2009abd0 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2009afa0 con 0x7f5a2009c5e0 2024-09-18T10:14:29.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a251ec700 10 monclient(hunting): authenticate will time out at 1158.886910s 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5a2009abd0 0x7f5a200960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59600/0 (socket says 172.21.15.142:59600) 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 -- 172.21.15.142:0/3317125373 learned_addr learned my addr 172.21.15.142:0/3317125373 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:29.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): get_auth_request con 0x7f5a2009c5e0 auth_method 0 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): get_auth_request con 0x7f5a2009abd0 auth_method 0 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:29.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:29.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:29.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:29.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient(hunting): handle_auth_done global_id 64139 payload 274 2024-09-18T10:14:29.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 -- 172.21.15.142:0/3317125373 >> v2:172.21.15.142:3300/0 conn(0x7f5a2009abd0 msgr2=0x7f5a200960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 --2- 172.21.15.142:0/3317125373 >> v2:172.21.15.142:3300/0 conn(0x7f5a2009abd0 0x7f5a200960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:29.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 -- 172.21.15.142:0/3317125373 >> v1:172.21.15.150:6789/0 conn(0x7f5a2009b740 legacy=0x7f5a2009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient: _finish_hunting 0 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 monclient: found mon.noname-b 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.958+0000 7f5a249eb700 1 -- 172.21.15.142:0/3317125373 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2006f810 con 0x7f5a2009c5e0 2024-09-18T10:14:29.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a249eb700 1 -- 172.21.15.142:0/3317125373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 msgr2=0x7f5a20137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a249eb700 1 --2- 172.21.15.142:0/3317125373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009f20 tx=0x7f5a18009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b56df41fb5d056d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 1 -- 172.21.15.142:0/3317125373 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1800e070 con 0x7f5a2009c5e0 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: dump: 2024-09-18T10:14:29.992 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.993 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.993 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.993 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:29.993 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:29.993 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:29.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 5 monclient: authenticate success, global_id 64139 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 1 -- 172.21.15.142:0/3317125373 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1800bbe0 con 0x7f5a2009c5e0 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:29.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 1 -- 172.21.15.142:0/3317125373 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180092e0 con 0x7f5a2009c5e0 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 4 set_mon_vals no callback set 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a1f7fe700 10 monclient: dump: 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:29.996 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:29.997 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.001 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.001 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.001 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.001 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 10 monclient: shutdown 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:30.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 -- 172.21.15.142:0/3317125373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 msgr2=0x7f5a20137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 --2- 172.21.15.142:0/3317125373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009f20 tx=0x7f5a18009840 comp rx=0 tx=0).stop 2024-09-18T10:14:30.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 -- 172.21.15.142:0/3317125373 shutdown_connections 2024-09-18T10:14:30.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 --2- 172.21.15.142:0/3317125373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009c5e0 0x7f5a20137690 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 --2- 172.21.15.142:0/3317125373 >> v2:172.21.15.142:3300/0 conn(0x7f5a2009abd0 0x7f5a200960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 -- 172.21.15.142:0/3317125373 >> 172.21.15.142:0/3317125373 conn(0x7f5a20070ef0 msgr2=0x7f5a200712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 -- 172.21.15.142:0/3317125373 shutdown_connections 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.962+0000 7f5a251ec700 1 -- 172.21.15.142:0/3317125373 wait complete. 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient: build_initial_monmap 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient: monmap: 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:30.004 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 Processor -- start 2024-09-18T10:14:30.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 -- start start 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient: init 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-c con 0x7f5a2009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2009b740 0x7f5a20141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-a con 0x7f5a2009b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2009c5e0 0x7f5a200989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): picked mon.noname-b con 0x7f5a2009c5e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:30.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2009d240 con 0x7f5a2009b740 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2006eb10 con 0x7f5a2009c5e0 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2009a4d0 con 0x7f5a2009abd0 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a251ec700 10 monclient(hunting): authenticate will time out at 1158.893457s 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2009b740 0x7f5a20141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59614/0 (socket says 172.21.15.142:59614) 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): get_auth_request con 0x7f5a2009b740 auth_method 0 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 -- 172.21.15.142:0/1965648121 learned_addr learned my addr 172.21.15.142:0/1965648121 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): get_auth_request con 0x7f5a2009abd0 auth_method 0 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2009c5e0 0x7f5a200989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): get_auth_request con 0x7f5a2009c5e0 auth_method 0 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a249eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient(hunting): handle_auth_done global_id 64142 payload 824 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2009c5e0 msgr2=0x7f5a200989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a2009c5e0 0x7f5a200989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2009b740 msgr2=0x7f5a20141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2009b740 0x7f5a20141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 monclient: found mon.noname-c 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a1ffff700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a18009ae0 con 0x7f5a2009abd0 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.966+0000 7f5a177fe700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a2009b740 0x7f5a20141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1ffff700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 msgr2=0x7f5a20141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1ffff700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800ec80 tx=0x7f5a080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08009690 con 0x7f5a2009abd0 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: dump: 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.012 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 5 monclient: authenticate success, global_id 64142 2024-09-18T10:14:30.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0800ca60 con 0x7f5a2009abd0 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 monclient: _renew_subs 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0800ce20 con 0x7f5a2009abd0 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a20091210 con 0x7f5a2009abd0 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 10 monclient: dump: 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.014 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.015 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1e7fc700 4 set_mon_vals no callback set 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 monclient: _renew_subs 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a20142410 con 0x7f5a2009abd0 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:30.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 0x7f5a20098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a200648a0 con 0x7f5a20092430 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 0x7f5a20098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient: get_auth_request con 0x7f5a20092430 auth_method 0 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5a08013680 con 0x7f5a2009abd0 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a00035b10 0x7f5a00037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f5a080497e0 con 0x7f5a2009abd0 2024-09-18T10:14:30.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 10 monclient(hunting): handle_auth_done global_id 15640 payload 824 2024-09-18T10:14:30.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.970+0000 7f5a177fe700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a20072a80 con 0x7f5a20092430 2024-09-18T10:14:30.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a249eb700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a00035b10 0x7f5a00037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a249eb700 10 monclient: get_auth_request con 0x7f5a00035b10 auth_method 0 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a177fe700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 msgr2=0x7f5a20098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a177fe700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 0x7f5a20098f00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000fb30 tx=0x7f5a10007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1000dbc0 con 0x7f5a20092430 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:30.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 10 monclient: dump: 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.021 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a249eb700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a00035b10 0x7f5a00037fc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009ab0 tx=0x7f5a18017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5a1000dd20 con 0x7f5a20092430 2024-09-18T10:14:30.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:30.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:30.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 msgr2=0x7f5a20098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 0x7f5a20098f00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000fb30 tx=0x7f5a10007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 msgr2=0x7f5a20098f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:30.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.974+0000 7f5a1cff9700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20092430 0x7f5a20098f00 secure :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5a1000fb30 tx=0x7f5a10007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a251ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a251ec700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 0x7f5a20099840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a251ec700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a20099d80 con 0x7f5a20099440 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 0x7f5a20099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient: get_auth_request con 0x7f5a20099440 auth_method 0 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 10 monclient(hunting): handle_auth_done global_id 15643 payload 824 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.986+0000 7f5a177fe700 1 -- 172.21.15.142:0/1965648121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a200a0c80 con 0x7f5a20099440 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a177fe700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 msgr2=0x7f5a20099840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a177fe700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 0x7f5a20099840 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5a10019000 tx=0x7f5a1001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1001eb70 con 0x7f5a20099440 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 10 monclient: dump: 2024-09-18T10:14:30.029 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.030 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f5a1001ecd0 con 0x7f5a20099440 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 msgr2=0x7f5a20099840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 0x7f5a20099840 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5a10019000 tx=0x7f5a1001a000 comp rx=0 tx=0).stop 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 msgr2=0x7f5a20099840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:30.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.990+0000 7f5a1cff9700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a20099440 0x7f5a20099840 secure :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5a10019000 tx=0x7f5a1001a000 comp rx=0 tx=0).stop 2024-09-18T10:14:30.032 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:30.032 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:30.032 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:30.032 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:14:30.032 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 62, 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:30.033 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:30.034 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:30.035 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:30.036 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:30.037 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:30.038 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.039 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:30.040 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.041 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:30.042 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.043 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:30.044 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:30.045 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.046 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.047 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.048 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:30.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:30.050 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:30.051 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a00035b10 msgr2=0x7f5a00037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5a00035b10 0x7f5a00037fc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009ab0 tx=0x7f5a18017000 comp rx=0 tx=0).stop 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 10 monclient: shutdown 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 msgr2=0x7f5a20141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a249eb700 1 -- 172.21.15.142:0/1965648121 reap_dead start 2024-09-18T10:14:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800ec80 tx=0x7f5a080091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 shutdown_connections 2024-09-18T10:14:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 --2- 172.21.15.142:0/1965648121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a2009abd0 0x7f5a20141990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 >> 172.21.15.142:0/1965648121 conn(0x7f5a20070ef0 msgr2=0x7f5a2009de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 shutdown_connections 2024-09-18T10:14:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:29.994+0000 7f5a251ec700 1 -- 172.21.15.142:0/1965648121 wait complete. 2024-09-18T10:14:30.053 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:14:30.054 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:14:30.054 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:14:30.054 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:14:30.054 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:14:30.054 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:14:30.054 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:14:30.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:30.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c015000 tx=0x7fcf5c016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c015000 tx=0x7fcf5c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 msgr2=0x7fcf60142830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf6009acd0 0x7fcf60142830 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 0x7fcf44042f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 10 monclient(hunting): picked mon.c con 0x7fcf44040b00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf44045800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 10 monclient(hunting): picked mon.b con 0x7fcf6009c8e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 msgr2=0x7fcf44042f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:30.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 0x7fcf44042f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.054+0000 7fcf567fc700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf440481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 10 monclient(hunting): picked mon.a con 0x7fcf6009bfd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44048b30 con 0x7fcf6009bfd0 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44048c70 con 0x7fcf6009c8e0 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44048db0 con 0x7fcf44040b00 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf567fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf44045800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): get_auth_request con 0x7fcf6009c8e0 auth_method 0 2024-09-18T10:14:30.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf440481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): get_auth_request con 0x7fcf6009bfd0 auth_method 0 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6613e700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient(hunting): handle_auth_done global_id 64121 payload 293 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 msgr2=0x7fcf44042f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 0x7fcf44042f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 msgr2=0x7fcf440481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf440481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient: _finish_hunting 0 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 monclient: found mon.b 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fcf44048fa0 con 0x7fcf6009c8e0 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:30.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7fcf58006e10 con 0x7fcf6009c8e0 2024-09-18T10:14:30.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 10 monclient: _finish_auth 0 2024-09-18T10:14:30.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:30.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 msgr2=0x7fcf44045800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.058+0000 7fcf6693f700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf44045800 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007350 tx=0x7fcf580072e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58007ac0 con 0x7fcf6009c8e0 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 10 monclient: dump: 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:30.068 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf58006260 con 0x7fcf6009c8e0 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf57fff700 4 set_mon_vals no callback set 2024-09-18T10:14:30.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:30.062+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcf580063d0 con 0x7fcf6009c8e0 2024-09-18T10:14:30.072 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:14:30.072 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:14:30.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: get_monmap_and_config 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: build_initial_monmap 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: monmap: 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:30.220863+0000 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:30.220863+0000 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:30.222 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 Processor -- start 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 -- start start 2024-09-18T10:14:30.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: init 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d40070ef0 0x7f1d400712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-c con 0x7f1d40070ef0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 0x7f1d40071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-b con 0x7f1d400a3210 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d40136890 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d40071d40 con 0x7f1d40136890 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4006aa60 con 0x7f1d40070ef0 2024-09-18T10:14:30.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4009dc00 con 0x7f1d400a3210 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d45152700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 msgr2=0x7f1d40071800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.218+0000 7f1d47bb7700 10 monclient(hunting): authenticate will time out at 1159.147998s 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45152700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 0x7f1d40071800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d40070ef0 0x7f1d400712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d40070ef0 0x7f1d400712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:35044/0 (socket says 172.21.15.142:35044) 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 -- 172.21.15.142:0/46897772 learned_addr learned my addr 172.21.15.142:0/46897772 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): get_auth_request con 0x7f1d40070ef0 auth_method 0 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient(hunting): handle_auth_done global_id 15646 payload 274 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 -- 172.21.15.142:0/46897772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 msgr2=0x7f1d40071800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 --2- 172.21.15.142:0/46897772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 0x7f1d40071800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 -- 172.21.15.142:0/46897772 >> v1:172.21.15.150:6789/0 conn(0x7f1d40136890 legacy=0x7f1d40136c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:30.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient: _finish_hunting 0 2024-09-18T10:14:30.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 monclient: found mon.noname-c 2024-09-18T10:14:30.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:30.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:30.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 -- 172.21.15.142:0/46897772 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4009d0c0 con 0x7f1d40070ef0 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 -- 172.21.15.142:0/46897772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d40070ef0 msgr2=0x7f1d400712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d45953700 1 --2- 172.21.15.142:0/46897772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d40070ef0 0x7f1d400712c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0099b0 tx=0x7f1d3c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2094b2e78288d223 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 1 -- 172.21.15.142:0/46897772 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00baf0 con 0x7f1d40070ef0 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: dump: 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.231 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: _finish_auth 0 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d47bb7700 5 monclient: authenticate success, global_id 15646 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 1 -- 172.21.15.142:0/46897772 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c00bc50 con 0x7f1d40070ef0 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d47bb7700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d47bb7700 4 set_mon_vals no callback set 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 1 -- 172.21.15.142:0/46897772 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c011420 con 0x7f1d40070ef0 2024-09-18T10:14:30.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.222+0000 7f1d44951700 10 monclient: dump: 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: shutdown 2024-09-18T10:14:30.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- 172.21.15.142:0/46897772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d40070ef0 msgr2=0x7f1d400712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/46897772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d40070ef0 0x7f1d400712c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0099b0 tx=0x7f1d3c009250 comp rx=0 tx=0).stop 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- 172.21.15.142:0/46897772 shutdown_connections 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/46897772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d400a3210 0x7f1d40071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/46897772 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d40070ef0 0x7f1d400712c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- 172.21.15.142:0/46897772 >> 172.21.15.142:0/46897772 conn(0x7f1d4006a2e0 msgr2=0x7f1d400a26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- 172.21.15.142:0/46897772 shutdown_connections 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- 172.21.15.142:0/46897772 wait complete. 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: build_initial_monmap 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: monmap: 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:30.234 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 Processor -- start 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- start start 2024-09-18T10:14:30.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: init 2024-09-18T10:14:30.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:30.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:30.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-a con 0x7f1d40070ef0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d400a3210 0x7f1d401464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-b con 0x7f1d400a3210 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 0x7f1d4006bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): picked mon.noname-c con 0x7f1d40136890 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4006f500 con 0x7f1d40070ef0 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4009d2f0 con 0x7f1d400a3210 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d400965a0 con 0x7f1d40136890 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:30.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d47bb7700 10 monclient(hunting): authenticate will time out at 1159.155247s 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d46154700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 msgr2=0x7f1d4006bd00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d46154700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 0x7f1d4006bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44018/0 (socket says 172.21.15.142:44018) 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 1 -- 172.21.15.142:0/2934931310 learned_addr learned my addr 172.21.15.142:0/2934931310 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): get_auth_request con 0x7f1d40070ef0 auth_method 0 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45953700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45152700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d400a3210 0x7f1d401464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45152700 10 monclient(hunting): get_auth_request con 0x7f1d400a3210 auth_method 0 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45152700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45152700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.226+0000 7f1d45152700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45152700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45152700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45152700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 10 monclient(hunting): handle_auth_done global_id 54441 payload 824 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 -- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 msgr2=0x7f1d4006bd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 0x7f1d4006bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 -- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d400a3210 msgr2=0x7f1d401464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d400a3210 0x7f1d401464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 10 monclient: _finish_hunting 0 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 monclient: found mon.noname-a 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 -- 172.21.15.142:0/2934931310 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0094f0 con 0x7f1d40070ef0 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 -- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 msgr2=0x7f1d40145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d45953700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1d4006b690 tx=0x7f1d3c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c011420 con 0x7f1d40070ef0 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: dump: 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.243 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 5 monclient: authenticate success, global_id 54441 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c020070 con 0x7f1d40070ef0 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 10 monclient: _renew_subs 2024-09-18T10:14:30.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c01b6a0 con 0x7f1d40070ef0 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d40127b70 con 0x7f1d40070ef0 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d377fe700 4 set_mon_vals no callback set 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d35ffb700 10 monclient: dump: 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:30.245 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 10 monclient: _renew_subs 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.230+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4006dfe0 con 0x7f1d40070ef0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1d3c022030 con 0x7f1d40070ef0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d47bb7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d47bb7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d47bb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4006f670 con 0x7f1d40070ef0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d35ffb700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 0x7f1d24037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f1d3c04b3b0 con 0x7f1d40070ef0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d45152700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 0x7f1d24037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:30.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.234+0000 7f1d45152700 10 monclient: get_auth_request con 0x7f1d24035a70 auth_method 0 2024-09-18T10:14:30.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.238+0000 7f1d45152700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 0x7f1d24037f20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1d300039d0 tx=0x7f1d300091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:30.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.242+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1d3c010580 con 0x7f1d40070ef0 2024-09-18T10:14:30.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.242+0000 7f1d35ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:30.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.242+0000 7f1d35ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:30.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d47bb7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:30.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d47bb7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:30.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d47bb7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:30.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1d4006f640 con 0x7f1d40070ef0 2024-09-18T10:14:30.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d35ffb700 1 -- 172.21.15.142:0/2934931310 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1d3c01abc0 con 0x7f1d40070ef0 2024-09-18T10:14:30.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d35ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:30.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.470+0000 7f1d35ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 msgr2=0x7f1d24037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 0x7f1d24037f20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1d300039d0 tx=0x7f1d300091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 10 monclient: shutdown 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 msgr2=0x7f1d40145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1d4006b690 tx=0x7f1d3c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:30.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 shutdown_connections 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d24035a70 0x7f1d24037f20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d40136890 0x7f1d4006bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d400a3210 0x7f1d401464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 --2- 172.21.15.142:0/2934931310 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d40070ef0 0x7f1d40145fb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 >> 172.21.15.142:0/2934931310 conn(0x7f1d4006a2e0 msgr2=0x7f1d400a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 shutdown_connections 2024-09-18T10:14:30.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:30.474+0000 7f1d47bb7700 1 -- 172.21.15.142:0/2934931310 wait complete. 2024-09-18T10:14:32.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:32.354+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:32.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:32.354+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:32.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:32.354+0000 7fcf557fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:33.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:33.354+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:33.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:33.354+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:33.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:33.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.602+0000 7f177e340700 10 monclient: get_monmap_and_config 2024-09-18T10:14:33.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.602+0000 7f177e340700 10 monclient: build_initial_monmap 2024-09-18T10:14:33.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.602+0000 7f177e340700 10 monclient: monmap: 2024-09-18T10:14:33.608 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:33.607307+0000 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:33.607307+0000 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:33.609 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 Processor -- start 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 -- start start 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient: init 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f177809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): picked mon.noname-b con 0x7f177806e860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:33.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f177806f460 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:33.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f177809ca30 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:33.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177806f220 con 0x7f177809ca30 2024-09-18T10:14:33.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1778099780 con 0x7f177806f460 2024-09-18T10:14:33.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778099910 con 0x7f177806e860 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177e340700 10 monclient(hunting): authenticate will time out at 1162.532869s 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177d33e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 msgr2=0x7f177809bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177d33e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f177809bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177db3f700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f177809ca30 0x7f17781395c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48182/0 (socket says 172.21.15.142:48182) 2024-09-18T10:14:33.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 learned_addr learned my addr 172.21.15.142:0/4064425807 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:14:33.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 >> v1:172.21.15.142:6789/0 conn(0x7f177809ca30 legacy=0x7f17781395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:33.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f177cb3d700 1 -- 172.21.15.142:0/4064425807 >> v1:172.21.15.150:6789/0 conn(0x7f177806f460 legacy=0x7f177809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:33.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638717288 0 0) 0x7f177806f220 con 0x7f177809ca30 2024-09-18T10:14:33.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f176ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f176ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.606+0000 7f176ffff700 10 monclient(hunting): my global_id is 54444 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764003570 con 0x7f177809ca30 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1115535016 0 0) 0x7f1764003570 con 0x7f177809ca30 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 msgr2=0x7f177809bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f177809bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 >> v1:172.21.15.150:6789/0 conn(0x7f177806f460 legacy=0x7f177809c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:14:33.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177809b770 con 0x7f177809ca30 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: _finish_auth 0 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177e340700 5 monclient: authenticate success, global_id 54444 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f1770001b20 con 0x7f177809ca30 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177e340700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: dump: 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.630 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 >> v1:172.21.15.142:6789/0 conn(0x7f177809ca30 legacy=0x7f17781395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1764003bb0 0x7f1764003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): picked mon.b con 0x7f1764003bb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f1764004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): picked mon.c con 0x7f177806e860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 0x7f176400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): picked mon.a con 0x7f1764004ff0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177cb3d700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 msgr2=0x7f1764004890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778099910 con 0x7f1764004ff0 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177cb3d700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f1764004890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176400bfa0 con 0x7f1764003bb0 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176400c0e0 con 0x7f177806e860 2024-09-18T10:14:33.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f176ffff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177e340700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 0x7f176400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1764003bb0 0x7f1764003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): get_auth_request con 0x7f1764004ff0 auth_method 0 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): get_auth_request con 0x7f1764003bb0 auth_method 0 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:33.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient(hunting): handle_auth_done global_id 54444 payload 293 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 msgr2=0x7f1764004890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f177806e860 0x7f1764004890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1764003bb0 msgr2=0x7f1764003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1764003bb0 0x7f1764003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient: _finish_hunting 0 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 monclient: found mon.a 2024-09-18T10:14:33.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1764003570 con 0x7f1764004ff0 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 10 monclient: _finish_auth 0 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.610+0000 7f177db3f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 msgr2=0x7f176400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177db3f700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 0x7f176400ba60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1770001850 tx=0x7f177000a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9544413b131e70b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000c5d0 con 0x7f1764004ff0 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: dump: 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.635 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177000c730 con 0x7f1764004ff0 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 4 set_mon_vals no callback set 2024-09-18T10:14:33.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 1 -- 172.21.15.142:0/4064425807 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000caf0 con 0x7f1764004ff0 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f176ffff700 10 monclient: dump: 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:33.637 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: shutdown 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 msgr2=0x7f176400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 --2- 172.21.15.142:0/4064425807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1764004ff0 0x7f176400ba60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1770001850 tx=0x7f177000a220 comp rx=0 tx=0).stop 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177db3f700 1 -- 172.21.15.142:0/4064425807 reap_dead start 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- 172.21.15.142:0/4064425807 shutdown_connections 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- 172.21.15.142:0/4064425807 >> 172.21.15.142:0/4064425807 conn(0x7f17780a1fe0 msgr2=0x7f1778070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- 172.21.15.142:0/4064425807 shutdown_connections 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- 172.21.15.142:0/4064425807 wait complete. 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: build_initial_monmap 2024-09-18T10:14:33.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: monmap: 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 Processor -- start 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- start start 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: init 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 0x7f1778141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): picked mon.noname-b con 0x7f1778141810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 0x7f1778090180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): picked mon.noname-c con 0x7f1778142120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:33.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): picked mon.noname-a con 0x7f17780906c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177809d230 con 0x7f17780906c0 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780a0350 con 0x7f1778141810 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17781424f0 con 0x7f1778142120 2024-09-18T10:14:33.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.614+0000 7f177e340700 10 monclient(hunting): authenticate will time out at 1162.543966s 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177cb3d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 msgr2=0x7f1778090180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177cb3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 0x7f1778090180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 0x7f1778141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:33.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37878/0 (socket says 172.21.15.142:37878) 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 -- 172.21.15.142:0/218964098 learned_addr learned my addr 172.21.15.142:0/218964098 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): get_auth_request con 0x7f17780906c0 auth_method 0 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:33.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): get_auth_request con 0x7f1778141810 auth_method 0 2024-09-18T10:14:33.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:33.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:33.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:33.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:33.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:33.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:33.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:33.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:33.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:33.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient(hunting): handle_auth_done global_id 54447 payload 824 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 -- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 msgr2=0x7f1778090180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 0x7f1778090180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 -- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 msgr2=0x7f1778141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 0x7f1778141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient: _finish_hunting 0 2024-09-18T10:14:33.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 monclient: found mon.noname-a 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 -- 172.21.15.142:0/218964098 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778091be0 con 0x7f17780906c0 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177d33e700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 0x7f1778141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 -- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 msgr2=0x7f1778090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177db3f700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1770000f80 tx=0x7f1770016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:33.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000cab0 con 0x7f17780906c0 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: dump: 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.648 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 5 monclient: authenticate success, global_id 54447 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770021070 con 0x7f17780906c0 2024-09-18T10:14:33.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 10 monclient: _renew_subs 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1778093710 con 0x7f17780906c0 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000ce60 con 0x7f17780906c0 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176effd700 4 set_mon_vals no callback set 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f176d7fa700 10 monclient: dump: 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:33.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 10 monclient: _renew_subs 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.618+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17780927b0 con 0x7f17780906c0 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177e340700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177e340700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177e340700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177808ca00 con 0x7f17780906c0 2024-09-18T10:14:33.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1770023030 con 0x7f17780906c0 2024-09-18T10:14:33.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f176d7fa700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 0x7f175c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:33.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f177004b1c0 con 0x7f17780906c0 2024-09-18T10:14:33.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177d33e700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 0x7f175c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:33.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.622+0000 7f177d33e700 10 monclient: get_auth_request con 0x7f175c035a70 auth_method 0 2024-09-18T10:14:33.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.626+0000 7f177d33e700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 0x7f175c037f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1774003fd0 tx=0x7f17740090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:33.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.630+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1770011b80 con 0x7f17780906c0 2024-09-18T10:14:33.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.630+0000 7f176d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:33.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.630+0000 7f176d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:33.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:33.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:33.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:33.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1778072a80 con 0x7f17780906c0 2024-09-18T10:14:33.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f176d7fa700 1 -- 172.21.15.142:0/218964098 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f17700113a0 con 0x7f17780906c0 2024-09-18T10:14:33.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f176d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:33.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f176d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:33.858 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:33.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 msgr2=0x7f175c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 0x7f175c037f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1774003fd0 tx=0x7f17740090f0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 10 monclient: shutdown 2024-09-18T10:14:33.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 msgr2=0x7f1778090b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1770000f80 tx=0x7f1770016000 comp rx=0 tx=0).stop 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 shutdown_connections 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f175c035a70 0x7f175c037f20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17780906c0 0x7f1778090b10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1778142120 0x7f1778090180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 --2- 172.21.15.142:0/218964098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1778141810 0x7f1778141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 >> 172.21.15.142:0/218964098 conn(0x7f17780a1fe0 msgr2=0x7f17780a0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 shutdown_connections 2024-09-18T10:14:33.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:33.854+0000 7f177e340700 1 -- 172.21.15.142:0/218964098 wait complete. 2024-09-18T10:14:34.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:34.354+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:34.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:34.354+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:35.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:35.354+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:35.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:35.354+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:36.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:36.354+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:36.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:36.354+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:36.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.974+0000 7f374eb59700 10 monclient: get_monmap_and_config 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.974+0000 7f374eb59700 10 monclient: build_initial_monmap 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.974+0000 7f374eb59700 10 monclient: monmap: 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:36.979620+0000 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:36.979620+0000 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:36.982 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 Processor -- start 2024-09-18T10:14:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 -- start start 2024-09-18T10:14:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient: init 2024-09-18T10:14:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 0x7f374809e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-c con 0x7f374809cdd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 0x7f37480a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-b con 0x7f37480a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f37480a0bf0 0x7f3748138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-a con 0x7f37480a0bf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374809d1a0 con 0x7f37480a0bf0 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374809e980 con 0x7f374809cdd0 2024-09-18T10:14:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374809d780 con 0x7f37480a02e0 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374eb59700 10 monclient(hunting): authenticate will time out at 1165.905506s 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f3747fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 msgr2=0x7f37480a06b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f3747fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 0x7f37480a06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f37480a0bf0 0x7f3748138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f37480a0bf0 0x7f3748138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37882/0 (socket says 172.21.15.142:37882) 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 -- 172.21.15.142:0/646790627 learned_addr learned my addr 172.21.15.142:0/646790627 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): get_auth_request con 0x7f37480a0bf0 auth_method 0 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 1 --2- 172.21.15.142:0/646790627 >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 0x7f374809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): get_auth_request con 0x7f374809cdd0 auth_method 0 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient(hunting): handle_auth_done global_id 54450 payload 274 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 -- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 msgr2=0x7f37480a06b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 --2- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 0x7f37480a06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 -- 172.21.15.142:0/646790627 >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 msgr2=0x7f374809e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 --2- 172.21.15.142:0/646790627 >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 0x7f374809e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient: _finish_hunting 0 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 monclient: found mon.noname-a 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374d0f6700 1 -- 172.21.15.142:0/646790627 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374809dd40 con 0x7f37480a0bf0 2024-09-18T10:14:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.978+0000 7f374c8f5700 1 --2- 172.21.15.142:0/646790627 >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 0x7f374809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374d0f6700 1 -- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37480a0bf0 msgr2=0x7f3748138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374d0f6700 1 --2- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37480a0bf0 0x7f3748138a70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3748071e50 tx=0x7f3738009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6140d5c04a54c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 1 -- 172.21.15.142:0/646790627 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3738009e70 con 0x7f37480a0bf0 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: dump: 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 1 -- 172.21.15.142:0/646790627 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373800ba70 con 0x7f37480a0bf0 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 5 monclient: authenticate success, global_id 54450 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 4 set_mon_vals no callback set 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 1 -- 172.21.15.142:0/646790627 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373800be30 con 0x7f37480a0bf0 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f37477fe700 10 monclient: dump: 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:36.995 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 10 monclient: shutdown 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 -- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37480a0bf0 msgr2=0x7f3748138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 --2- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37480a0bf0 0x7f3748138a70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3748071e50 tx=0x7f3738009250 comp rx=0 tx=0).stop 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 -- 172.21.15.142:0/646790627 shutdown_connections 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 --2- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37480a0bf0 0x7f3748138a70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 --2- 172.21.15.142:0/646790627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a02e0 0x7f37480a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 --2- 172.21.15.142:0/646790627 >> v2:172.21.15.150:3300/0 conn(0x7f374809cdd0 0x7f374809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 -- 172.21.15.142:0/646790627 >> 172.21.15.142:0/646790627 conn(0x7f3748071680 msgr2=0x7f37480a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 -- 172.21.15.142:0/646790627 shutdown_connections 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 1 -- 172.21.15.142:0/646790627 wait complete. 2024-09-18T10:14:36.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.982+0000 7f374eb59700 10 monclient: build_initial_monmap 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient: monmap: 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 Processor -- start 2024-09-18T10:14:36.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 -- start start 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient: init 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-a con 0x7f374809cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37480a02e0 0x7f374808c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-b con 0x7f37480a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 0x7f374808cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): picked mon.noname-c con 0x7f37480a0bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374808e640 con 0x7f374809cdd0 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374809bdd0 con 0x7f37480a02e0 2024-09-18T10:14:36.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374808eac0 con 0x7f37480a0bf0 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374eb59700 10 monclient(hunting): authenticate will time out at 1165.914302s 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374d0f6700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 msgr2=0x7f374808cf10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374d0f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 0x7f374808cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37884/0 (socket says 172.21.15.142:37884) 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 -- 172.21.15.142:0/3282617839 learned_addr learned my addr 172.21.15.142:0/3282617839 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): get_auth_request con 0x7f374809cdd0 auth_method 0 2024-09-18T10:14:36.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37480a02e0 0x7f374808c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): get_auth_request con 0x7f37480a02e0 auth_method 0 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:37.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f3747fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient(hunting): handle_auth_done global_id 54453 payload 824 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 -- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 msgr2=0x7f374808cf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 0x7f374808cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 -- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37480a02e0 msgr2=0x7f374808c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37480a02e0 0x7f374808c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient: _finish_hunting 0 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 monclient: found mon.noname-a 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:37.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:37.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.986+0000 7f374c8f5700 1 -- 172.21.15.142:0/3282617839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009420 con 0x7f374809cdd0 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374c8f5700 1 -- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 msgr2=0x7f3748098080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374c8f5700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f373c00ec80 tx=0x7f373c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c00cb00 con 0x7f374809cdd0 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: dump: 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:37.004 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c00cc60 con 0x7f374809cdd0 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 5 monclient: authenticate success, global_id 54453 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _renew_subs 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37380094f0 con 0x7f374809cdd0 2024-09-18T10:14:37.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c013470 con 0x7f374809cdd0 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3744ff9700 10 monclient: dump: 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:37.006 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f3746ffd700 4 set_mon_vals no callback set 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _renew_subs 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3748090070 con 0x7f374809cdd0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.990+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3748090750 con 0x7f374809cdd0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f373c0137e0 con 0x7f374809cdd0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3744ff9700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 0x7f3734037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(152..152 src has 58..152) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f373c049840 con 0x7f374809cdd0 2024-09-18T10:14:37.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3747fff700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 0x7f3734037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:37.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3747fff700 10 monclient: get_auth_request con 0x7f3734035af0 auth_method 0 2024-09-18T10:14:37.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:36.994+0000 7f3747fff700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 0x7f3734037fa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3740009fd0 tx=0x7f374000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:37.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.002+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f373c0236b0 con 0x7f374809cdd0 2024-09-18T10:14:37.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.002+0000 7f3744ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:37.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.002+0000 7f3744ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:37.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f374eb59700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:37.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f374eb59700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:37.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f374eb59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:37.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f37480696a0 con 0x7f374809cdd0 2024-09-18T10:14:37.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f3744ff9700 1 -- 172.21.15.142:0/3282617839 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f373c0232f0 con 0x7f374809cdd0 2024-09-18T10:14:37.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f3744ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:37.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.222+0000 7f3744ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:37.227 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 msgr2=0x7f3734037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 0x7f3734037fa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3740009fd0 tx=0x7f374000d040 comp rx=0 tx=0).stop 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 10 monclient: shutdown 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 msgr2=0x7f3748098080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f373c00ec80 tx=0x7f373c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 shutdown_connections 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3734035af0 0x7f3734037fa0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37480a0bf0 0x7f374808cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37480a02e0 0x7f374808c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 --2- 172.21.15.142:0/3282617839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f374809cdd0 0x7f3748098080 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 >> 172.21.15.142:0/3282617839 conn(0x7f3748071680 msgr2=0x7f374809b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 shutdown_connections 2024-09-18T10:14:37.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:37.226+0000 7f374eb59700 1 -- 172.21.15.142:0/3282617839 wait complete. 2024-09-18T10:14:37.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:37.358+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:37.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:37.358+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:38.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:38.358+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:38.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:38.358+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:39.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:39.353+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:39.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:39.353+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:40.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: get_monmap_and_config 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: build_initial_monmap 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: monmap: 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:40.355979+0000 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:40.355979+0000 2024-09-18T10:14:40.357 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:40.358 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 Processor -- start 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 -- start start 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: init 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-b con 0x7f71ac09a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71ac099220 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:40.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-a con 0x7f71ac09c4f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac09abc0 con 0x7f71ac099220 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac09d950 con 0x7f71ac09c4f0 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac09db90 con 0x7f71ac09a7f0 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b3534700 10 monclient(hunting): authenticate will time out at 1169.281470s 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2532700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2532700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37892/0 (socket says 172.21.15.142:37892) 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 learned_addr learned my addr 172.21.15.142:0/2230539129 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:40.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 10 monclient(hunting): get_auth_request con 0x7f71ac09c4f0 auth_method 0 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b1d31700 1 --1- 172.21.15.142:0/2230539129 >> v1:172.21.15.142:6789/0 conn(0x7f71ac099220 0x7f71ac09bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48192/0 (socket says 172.21.15.142:48192) 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:40.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.354+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:40.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:40.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:40.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:40.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:40.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.358+0000 7f71b1d31700 1 -- 172.21.15.142:0/2230539129 >> v1:172.21.15.142:6789/0 conn(0x7f71ac099220 legacy=0x7f71ac09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 10 monclient(hunting): get_auth_request con 0x7f71ac09c4f0 auth_method 0 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.558+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:14:40.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:40.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:14:40.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:40.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 10 monclient(hunting): get_auth_request con 0x7f71ac09c4f0 auth_method 0 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:40.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:40.962+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:14:41.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:41.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:41.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:41.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:41.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 10 monclient(hunting): get_auth_request con 0x7f71ac09c4f0 auth_method 0 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:41.766+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:14:42.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:42.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:42.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:42.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:42.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:42.357+0000 7fcf557fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): tick 2024-09-18T10:14:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:14:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:14:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac098ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f71ac098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 msgr2=0x7f71ac13e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.142:3300/0 conn(0x7f71ac09c4f0 0x7f71ac13e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 -- 172.21.15.142:0/2230539129 >> v1:172.21.15.142:6789/0 conn(0x7f71ac099220 legacy=0x7f71ac09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient: _add_conns ranks=[4,1] 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f7190003790 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71900062f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 -- 172.21.15.142:0/2230539129 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac099a30 con 0x7f71900062f0 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 1 -- 172.21.15.142:0/2230539129 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac09d950 con 0x7f7190003790 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 10 monclient(hunting): get_auth_request con 0x7f7190003790 auth_method 0 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.354+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:43.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:43.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:43.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:43.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.358+0000 7f71b1d31700 1 -- 172.21.15.142:0/2230539129 >> v1:172.21.15.150:6789/0 conn(0x7f71900062f0 legacy=0x7f7190008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 10 monclient(hunting): get_auth_request con 0x7f7190003790 auth_method 0 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:43.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:14:43.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:43.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:43.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.558+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:14:43.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:43.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 10 monclient(hunting): get_auth_request con 0x7f7190003790 auth_method 0 2024-09-18T10:14:43.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:43.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:43.962+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.298+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155471672 0 0) 0x7f71ac099a30 con 0x7f71900062f0 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.298+0000 7f71b1530700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.298+0000 7f71b1530700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.298+0000 7f71b1530700 10 monclient(hunting): my global_id is 15661 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.298+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7198003570 con 0x7f71900062f0 2024-09-18T10:14:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011915687 0 0) 0x7f7198003570 con 0x7f71900062f0 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 msgr2=0x7f7190005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 --2- 172.21.15.142:0/2230539129 >> v2:172.21.15.150:3300/0 conn(0x7f7190003790 0x7f7190005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719000b090 con 0x7f71900062f0 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f719c001760 con 0x7f71900062f0 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:14:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: dump: 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 >> v1:172.21.15.150:6789/0 conn(0x7f71900062f0 legacy=0x7f7190008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f7198003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): picked mon.b con 0x7f7198003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7198004810 0x7f7198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): picked mon.a con 0x7f7198004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f7198005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): picked mon.c con 0x7f71ac09a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac09d950 con 0x7f7198004810 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719800bbf0 con 0x7f7198003a60 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719800bdb0 con 0x7f71ac09a7f0 2024-09-18T10:14:44.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b1530700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 reap_dead start 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2d33700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f7198005290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f7198005290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f7198003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): get_auth_request con 0x7f7198003a60 auth_method 0 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient(hunting): handle_auth_done global_id 15661 payload 293 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 msgr2=0x7f7198005290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f7198005290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7198004810 msgr2=0x7f7198004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7198004810 0x7f7198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 monclient: found mon.b 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719000b090 con 0x7f7198003a60 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b2532700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b3534700 5 monclient: authenticate success, global_id 15661 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.302+0000 7f71b3534700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b2532700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 msgr2=0x7f7198003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b2532700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f7198003e50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f71a4011630 tx=0x7f71a4007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8fe6ed62c4fd3d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4016070 con 0x7f7198003a60 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: dump: 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.317 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a400e600 con 0x7f7198003a60 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 1 -- 172.21.15.142:0/2230539129 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a400e9c0 con 0x7f7198003a60 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b1530700 10 monclient: dump: 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.318 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b3534700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.306+0000 7f71b3534700 4 set_mon_vals no callback set 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: shutdown 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 msgr2=0x7f7198003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f7198003e50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f71a4011630 tx=0x7f71a4007900 comp rx=0 tx=0).stop 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- 172.21.15.142:0/2230539129 shutdown_connections 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71ac09a7f0 0x7f7198005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f7198003e50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- 172.21.15.142:0/2230539129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7198004810 0x7f7198004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- 172.21.15.142:0/2230539129 >> 172.21.15.142:0/2230539129 conn(0x7f71ac0a1fe0 msgr2=0x7f71ac070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- 172.21.15.142:0/2230539129 shutdown_connections 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- 172.21.15.142:0/2230539129 wait complete. 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: build_initial_monmap 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: monmap: 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:44.320 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:44.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 Processor -- start 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- start start 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: init 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-b con 0x7f7198003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 0x7f71ac141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-c con 0x7f7198004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): picked mon.noname-a con 0x7f71ac09a7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac06f220 con 0x7f71ac09a7f0 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac126e80 con 0x7f7198003a60 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac06d2f0 con 0x7f7198004810 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b3534700 10 monclient(hunting): authenticate will time out at 1173.239710s 2024-09-18T10:14:44.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b1d31700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 msgr2=0x7f71ac141540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b1d31700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 0x7f71ac141540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35150/0 (socket says 172.21.15.142:35150) 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 1 -- 172.21.15.142:0/2082753810 learned_addr learned my addr 172.21.15.142:0/2082753810 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:58008/0 (socket says 172.21.15.142:58008) 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 10 monclient(hunting): get_auth_request con 0x7f71ac09a7f0 auth_method 0 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2d33700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 10 monclient(hunting): get_auth_request con 0x7f7198003a60 auth_method 0 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.310+0000 7f71b2532700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2d33700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2d33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2d33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient(hunting): handle_auth_done global_id 15664 payload 824 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 -- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 msgr2=0x7f71ac141540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 0x7f71ac141540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 -- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 msgr2=0x7f71ac145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 monclient: found mon.noname-b 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 -- 172.21.15.142:0/2082753810 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac1268c0 con 0x7f7198003a60 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2d33700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 -- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 msgr2=0x7f71ac141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b2532700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f71a4003d50 tx=0x7f71a401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4007040 con 0x7f7198003a60 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: dump: 2024-09-18T10:14:44.329 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a400ee30 con 0x7f7198003a60 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 5 monclient: authenticate success, global_id 15664 2024-09-18T10:14:44.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 10 monclient: _renew_subs 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac126ba0 con 0x7f7198003a60 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a401cea0 con 0x7f7198003a60 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a27fc700 10 monclient: dump: 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.331 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71a3fff700 4 set_mon_vals no callback set 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 10 monclient: _renew_subs 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.314+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac141c20 con 0x7f7198003a60 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b3534700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b3534700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b3534700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac072a80 con 0x7f7198003a60 2024-09-18T10:14:44.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f71a401a380 con 0x7f7198003a60 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71a27fc700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 0x7f7190038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(153..153 src has 58..153) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f71a404bc80 con 0x7f7198003a60 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b1d31700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 0x7f7190038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.318+0000 7f71b1d31700 10 monclient: get_auth_request con 0x7f71900366b0 auth_method 0 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.322+0000 7f71b1d31700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 0x7f7190038b60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06e040 tx=0x7f719c0018a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.326+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f71a401ae60 con 0x7f7198003a60 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.326+0000 7f71a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.326+0000 7f71a27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:44.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:44.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:44.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:44.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71b3534700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:14:44.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71b3534700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:44.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71b3534700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:44.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f71ac09d730 con 0x7f7198003a60 2024-09-18T10:14:44.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71a27fc700 1 -- 172.21.15.142:0/2082753810 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f71a402b030 con 0x7f7198003a60 2024-09-18T10:14:44.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71a27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:14:44.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.546+0000 7f71a27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:44.551 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 msgr2=0x7f7190038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 0x7f7190038b60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06e040 tx=0x7f719c0018a0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 10 monclient: shutdown 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 msgr2=0x7f71ac141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f71a4003d50 tx=0x7f71a401d000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 shutdown_connections 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f71900366b0 0x7f7190038b60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71ac09a7f0 0x7f71ac145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7198004810 0x7f71ac141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 --2- 172.21.15.142:0/2082753810 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7198003a60 0x7f71ac141e90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 >> 172.21.15.142:0/2082753810 conn(0x7f71ac0a1fe0 msgr2=0x7f71ac09fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 shutdown_connections 2024-09-18T10:14:44.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.550+0000 7f71b3534700 1 -- 172.21.15.142:0/2082753810 wait complete. 2024-09-18T10:14:44.566 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:14:44.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:14:44.566 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:14:44.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: get_monmap_and_config 2024-09-18T10:14:44.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: build_initial_monmap 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: monmap: 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:44.669637+0000 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:44.669637+0000 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:44.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 Processor -- start 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 -- start start 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: init 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 0x7f33d406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-b con 0x7f33d409a6a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33d409aa70 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33d409b550 0x7f33d413d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-a con 0x7f33d409b550 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d409dae0 con 0x7f33d409aa70 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d4099bc0 con 0x7f33d409b550 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40998e0 con 0x7f33d409a6a0 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33da39d700 10 monclient(hunting): authenticate will time out at 1173.595315s 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d939b700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 msgr2=0x7f33d406f7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d939b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 0x7f33d406f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33d409b550 0x7f33d413d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33d409b550 0x7f33d413d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35152/0 (socket says 172.21.15.142:35152) 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3307006826 learned_addr learned my addr 172.21.15.142:0/3307006826 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): get_auth_request con 0x7f33d409b550 auth_method 0 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_done global_id 54459 payload 274 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 msgr2=0x7f33d406f7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 --2- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 0x7f33d406f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3307006826 >> v1:172.21.15.150:6789/0 conn(0x7f33d409aa70 legacy=0x7f33d409ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:44.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.666+0000 7f33d9b9c700 1 monclient: found mon.noname-a 2024-09-18T10:14:44.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33d9b9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33d9b9c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3307006826 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d409bed0 con 0x7f33d409b550 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 msgr2=0x7f33d413d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33d9b9c700 1 --2- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d413d6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0099b0 tx=0x7f33cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=507cdd1fa94062c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 1 -- 172.21.15.142:0/3307006826 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00bb50 con 0x7f33d409b550 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: dump: 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.681 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 5 monclient: authenticate success, global_id 54459 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:44.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 1 -- 172.21.15.142:0/3307006826 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc00bcb0 con 0x7f33d409b550 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 4 set_mon_vals no callback set 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 1 -- 172.21.15.142:0/3307006826 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc011670 con 0x7f33d409b550 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33cbfff700 10 monclient: dump: 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.683 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 10 monclient: shutdown 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 -- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 msgr2=0x7f33d413d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 --2- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d413d6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0099b0 tx=0x7f33cc009250 comp rx=0 tx=0).stop 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 -- 172.21.15.142:0/3307006826 shutdown_connections 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 --2- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d413d6f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 --2- 172.21.15.142:0/3307006826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409a6a0 0x7f33d406f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 -- 172.21.15.142:0/3307006826 >> 172.21.15.142:0/3307006826 conn(0x7f33d4070ef0 msgr2=0x7f33d40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.670+0000 7f33da39d700 1 -- 172.21.15.142:0/3307006826 shutdown_connections 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 -- 172.21.15.142:0/3307006826 wait complete. 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient: build_initial_monmap 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient: monmap: 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:44.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 Processor -- start 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 -- start start 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient: init 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33d409a6a0 0x7f33d406bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-b con 0x7f33d409a6a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409aa70 0x7f33d406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-c con 0x7f33d409aa70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): picked mon.noname-a con 0x7f33d409b550 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4126d50 con 0x7f33d409b550 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4097e80 con 0x7f33d409a6a0 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d413d7b0 con 0x7f33d409aa70 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33da39d700 10 monclient(hunting): authenticate will time out at 1173.603137s 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d8b9a700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409aa70 msgr2=0x7f33d406c470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d8b9a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409aa70 0x7f33d406c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35160/0 (socket says 172.21.15.142:35160) 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 learned_addr learned my addr 172.21.15.142:0/3757438274 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): get_auth_request con 0x7f33d409b550 auth_method 0 2024-09-18T10:14:44.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33d409a6a0 0x7f33d406bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): get_auth_request con 0x7f33d409a6a0 auth_method 0 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient(hunting): handle_auth_done global_id 54462 payload 824 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409aa70 msgr2=0x7f33d406c470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33d409aa70 0x7f33d406c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33d409a6a0 msgr2=0x7f33d406bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33d409a6a0 0x7f33d406bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 monclient: found mon.noname-a 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.674+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0094f0 con 0x7f33d409b550 2024-09-18T10:14:44.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 msgr2=0x7f33d406c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33d9b9c700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc006b30 con 0x7f33d409b550 2024-09-18T10:14:44.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:44.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: dump: 2024-09-18T10:14:44.693 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.693 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 5 monclient: authenticate success, global_id 54462 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc020070 con 0x7f33d409b550 2024-09-18T10:14:44.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 monclient: _renew_subs 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d4126790 con 0x7f33d409b550 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc01c680 con 0x7f33d409b550 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33c97fa700 10 monclient: dump: 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.695 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33cb7fe700 4 set_mon_vals no callback set 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 monclient: _renew_subs 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d4126b20 con 0x7f33d409b550 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 0x7f33d406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.678+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d40648a0 con 0x7f33d402ae80 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 0x7f33d406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f33cc022030 con 0x7f33d409b550 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient: get_auth_request con 0x7f33d402ae80 auth_method 0 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33b8035aa0 0x7f33b8037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 10 monclient(hunting): handle_auth_done global_id 54465 payload 824 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33d4126f90 con 0x7f33d402ae80 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(153..153 src has 58..153) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f33cc04b6a0 con 0x7f33d409b550 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d8b9a700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33b8035aa0 0x7f33b8037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d8b9a700 10 monclient: get_auth_request con 0x7f33b8035aa0 auth_method 0 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 msgr2=0x7f33d406cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d939b700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 0x7f33d406cef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33c40039d0 tx=0x7f33c4009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4012940 con 0x7f33d402ae80 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 10 monclient: dump: 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.698 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.699 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.699 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.699 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f33c4012aa0 con 0x7f33d402ae80 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 msgr2=0x7f33d406cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 0x7f33d406cef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33c40039d0 tx=0x7f33c4009120 comp rx=0 tx=0).stop 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 msgr2=0x7f33d406cef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33c97fa700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d402ae80 0x7f33d406cef0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f33c40039d0 tx=0x7f33c4009120 comp rx=0 tx=0).stop 2024-09-18T10:14:44.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.682+0000 7f33d8b9a700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33b8035aa0 0x7f33b8037f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f33d409a050 tx=0x7f33d0007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33da39d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33da39d700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 0x7f33d405ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33da39d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d4093670 con 0x7f33d405e890 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 0x7f33d405ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient: get_auth_request con 0x7f33d405e890 auth_method 0 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 10 monclient(hunting): handle_auth_done global_id 54468 payload 824 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.694+0000 7f33d939b700 1 -- 172.21.15.142:0/3757438274 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f33d4072a80 con 0x7f33d405e890 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33d939b700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 msgr2=0x7f33d405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33d939b700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 0x7f33d405ec90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33c4017000 tx=0x7f33c4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c401b990 con 0x7f33d405e890 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 10 monclient: dump: 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.704 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f33c401baf0 con 0x7f33d405e890 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 msgr2=0x7f33d405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 0x7f33d405ec90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33c4017000 tx=0x7f33c4018000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 msgr2=0x7f33d405ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33c97fa700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d405e890 0x7f33d405ec90 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f33c4017000 tx=0x7f33c4018000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 66, 2024-09-18T10:14:44.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.707 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.708 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:44.709 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.710 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.711 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:44.712 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.713 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.714 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.715 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:44.716 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.717 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.718 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.719 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:44.720 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:44.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33b8035aa0 msgr2=0x7f33b8037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33b8035aa0 0x7f33b8037f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f33d409a050 tx=0x7f33d0007e80 comp rx=0 tx=0).stop 2024-09-18T10:14:44.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 10 monclient: shutdown 2024-09-18T10:14:44.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 msgr2=0x7f33d406c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33d9b9c700 1 -- 172.21.15.142:0/3757438274 reap_dead start 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009fd0 tx=0x7f33cc016000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 shutdown_connections 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 --2- 172.21.15.142:0/3757438274 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33d409b550 0x7f33d406c9b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 >> 172.21.15.142:0/3757438274 conn(0x7f33d4070ef0 msgr2=0x7f33d40907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 shutdown_connections 2024-09-18T10:14:44.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.698+0000 7f33da39d700 1 -- 172.21.15.142:0/3757438274 wait complete. 2024-09-18T10:14:44.723 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:14:44.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:14:44.724 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.830+0000 7f92d2be5700 10 monclient: get_monmap_and_config 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.830+0000 7f92d2be5700 10 monclient: build_initial_monmap 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient: monmap: 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:14:44.835710+0000 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:14:44.835710+0000 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:44.837 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:44.838 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 Processor -- start 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 -- start start 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient: init 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f92cc09cf70 0x7f92cc09e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-a con 0x7f92cc09cf70 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:44.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92cc0a0270 0x7f92cc0a0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-c con 0x7f92cc0a0270 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 0x7f92cc138a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-b con 0x7f92cc0a0bd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc0a1df0 con 0x7f92cc09cf70 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc09dfb0 con 0x7f92cc0a0270 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc0a1650 con 0x7f92cc0a0bd0 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d2be5700 10 monclient(hunting): authenticate will time out at 1173.761882s 2024-09-18T10:14:44.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f92cc09cf70 0x7f92cc09e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d1182700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 msgr2=0x7f92cc138a00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d1182700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 0x7f92cc138a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f92cc09cf70 0x7f92cc09e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35198/0 (socket says 172.21.15.142:35198) 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 -- 172.21.15.142:0/284846794 learned_addr learned my addr 172.21.15.142:0/284846794 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): get_auth_request con 0x7f92cc09cf70 auth_method 0 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92cbfff700 1 --2- 172.21.15.142:0/284846794 >> v2:172.21.15.150:3300/0 conn(0x7f92cc0a0270 0x7f92cc0a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92cbfff700 10 monclient(hunting): get_auth_request con 0x7f92cc0a0270 auth_method 0 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient(hunting): handle_auth_done global_id 54471 payload 274 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 -- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 msgr2=0x7f92cc138a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 --2- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 0x7f92cc138a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 -- 172.21.15.142:0/284846794 >> v2:172.21.15.150:3300/0 conn(0x7f92cc0a0270 msgr2=0x7f92cc0a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 --2- 172.21.15.142:0/284846794 >> v2:172.21.15.150:3300/0 conn(0x7f92cc0a0270 0x7f92cc0a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 monclient: found mon.noname-a 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.834+0000 7f92d0981700 1 -- 172.21.15.142:0/284846794 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc09d340 con 0x7f92cc09cf70 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d0981700 1 -- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc09cf70 msgr2=0x7f92cc09e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d0981700 1 --2- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc09cf70 0x7f92cc09e3d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0a1c90 tx=0x7f92c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98df0df4d4a0d02c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 1 -- 172.21.15.142:0/284846794 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400e070 con 0x7f92cc09cf70 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: dump: 2024-09-18T10:14:44.846 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.847 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.847 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.847 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.847 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.847 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 1 -- 172.21.15.142:0/284846794 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c400bc60 con 0x7f92cc09cf70 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 5 monclient: authenticate success, global_id 54471 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:44.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 1 -- 172.21.15.142:0/284846794 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c40092e0 con 0x7f92cc09cf70 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 4 set_mon_vals no callback set 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92cb7fe700 10 monclient: dump: 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.849 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.850 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.850 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 10 monclient: shutdown 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 -- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc09cf70 msgr2=0x7f92cc09e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 --2- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc09cf70 0x7f92cc09e3d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0a1c90 tx=0x7f92c4009840 comp rx=0 tx=0).stop 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 -- 172.21.15.142:0/284846794 shutdown_connections 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 --2- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc0a0bd0 0x7f92cc138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 --2- 172.21.15.142:0/284846794 >> v2:172.21.15.150:3300/0 conn(0x7f92cc0a0270 0x7f92cc0a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 --2- 172.21.15.142:0/284846794 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc09cf70 0x7f92cc09e3d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 -- 172.21.15.142:0/284846794 >> 172.21.15.142:0/284846794 conn(0x7f92cc0a5880 msgr2=0x7f92cc0a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 -- 172.21.15.142:0/284846794 shutdown_connections 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 1 -- 172.21.15.142:0/284846794 wait complete. 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 10 monclient: build_initial_monmap 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.838+0000 7f92d2be5700 10 monclient: monmap: 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:44.852 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:44.853 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 Processor -- start 2024-09-18T10:14:44.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 -- start start 2024-09-18T10:14:44.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient: init 2024-09-18T10:14:44.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc09cf70 0x7f92cc0910e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-c con 0x7f92cc09cf70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-a con 0x7f92cc0a0270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc0a0bd0 0x7f92cc08d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): picked mon.noname-b con 0x7f92cc0a0bd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d0981700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc09cf70 msgr2=0x7f92cc0910e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc090cc0 con 0x7f92cc0a0270 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc091620 con 0x7f92cc0a0bd0 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc08f460 con 0x7f92cc09cf70 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d0981700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc09cf70 0x7f92cc0910e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d2be5700 10 monclient(hunting): authenticate will time out at 1173.770245s 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35200/0 (socket says 172.21.15.142:35200) 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 -- 172.21.15.142:0/3434082279 learned_addr learned my addr 172.21.15.142:0/3434082279 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): get_auth_request con 0x7f92cc0a0270 auth_method 0 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc0a0bd0 0x7f92cc08d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): get_auth_request con 0x7f92cc0a0bd0 auth_method 0 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92d1182700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient(hunting): handle_auth_done global_id 54474 payload 824 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc09cf70 msgr2=0x7f92cc0910e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92cc09cf70 0x7f92cc0910e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc0a0bd0 msgr2=0x7f92cc08d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc0a0bd0 0x7f92cc08d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 monclient: found mon.noname-a 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.842+0000 7f92cbfff700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c4009ae0 con 0x7f92cc0a0270 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92cbfff700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 msgr2=0x7f92cc08d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92cbfff700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f92b400d890 tx=0x7f92b4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012800 con 0x7f92cc0a0270 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: dump: 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 5 monclient: authenticate success, global_id 54474 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 monclient: _renew_subs 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc08eb70 con 0x7f92cc0a0270 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b4012960 con 0x7f92cc0a0270 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012ce0 con 0x7f92cc0a0270 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 10 monclient: dump: 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.866 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92caffd700 4 set_mon_vals no callback set 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 monclient: _renew_subs 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc08f260 con 0x7f92cc0a0270 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 0x7f92cc02ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc064850 con 0x7f92cc08de40 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 0x7f92cc02ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient: get_auth_request con 0x7f92cc08de40 auth_method 0 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f92b4014640 con 0x7f92cc0a0270 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92ac035a70 0x7f92ac037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.846+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(153..153 src has 58..153) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f92b4049b90 con 0x7f92cc0a0270 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d1182700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92ac035a70 0x7f92ac037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d0981700 10 monclient(hunting): handle_auth_done global_id 15676 payload 824 2024-09-18T10:14:44.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d0981700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92cc08e210 con 0x7f92cc08de40 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d1182700 10 monclient: get_auth_request con 0x7f92ac035a70 auth_method 0 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d0981700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 msgr2=0x7f92cc02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d0981700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 0x7f92cc02ae80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f92c4015000 tx=0x7f92c4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92d1182700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92ac035a70 0x7f92ac037f20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0039d0 tx=0x7f92bc009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c4007a30 con 0x7f92cc08de40 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 10 monclient: dump: 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.869 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f92c4007b90 con 0x7f92cc08de40 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 msgr2=0x7f92cc02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 0x7f92cc02ae80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f92c4015000 tx=0x7f92c4018000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 msgr2=0x7f92cc02ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:44.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.850+0000 7f92c8ff9700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc08de40 0x7f92cc02ae80 secure :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f92c4015000 tx=0x7f92c4018000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d2be5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:14:44.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d2be5700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 0x7f92cc05ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:44.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d2be5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc06d9e0 con 0x7f92cc05e890 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 0x7f92cc05ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient: get_auth_request con 0x7f92cc05e890 auth_method 0 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 10 monclient(hunting): handle_auth_done global_id 15679 payload 824 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.862+0000 7f92d0981700 1 -- 172.21.15.142:0/3434082279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f92cc0696a0 con 0x7f92cc05e890 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d0981700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 msgr2=0x7f92cc05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:44.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d0981700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 0x7f92cc05ec90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f92c40098c0 tx=0x7f92c401e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c401d990 con 0x7f92cc05e890 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 10 monclient: dump: 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:44.874 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f92c401daf0 con 0x7f92cc05e890 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 msgr2=0x7f92cc05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 0x7f92cc05ec90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f92c40098c0 tx=0x7f92c401e000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 msgr2=0x7f92cc05ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92c8ff9700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92cc05e890 0x7f92cc05ec90 secure :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f92c40098c0 tx=0x7f92c401e000 comp rx=0 tx=0).stop 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:14:44.875 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 66, 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.876 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.877 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:14:44.878 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:14:44.879 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.880 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.881 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.882 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:14:44.884 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:14:44.885 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:14:44.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:14:44.887 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:14:44.888 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92ac035a70 msgr2=0x7f92ac037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92ac035a70 0x7f92ac037f20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92bc0039d0 tx=0x7f92bc009060 comp rx=0 tx=0).stop 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 10 monclient: shutdown 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 msgr2=0x7f92cc08d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d1182700 1 -- 172.21.15.142:0/3434082279 reap_dead start 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f92b400d890 tx=0x7f92b4008040 comp rx=0 tx=0).stop 2024-09-18T10:14:44.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 shutdown_connections 2024-09-18T10:14:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 --2- 172.21.15.142:0/3434082279 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92cc0a0270 0x7f92cc08d3c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 >> 172.21.15.142:0/3434082279 conn(0x7f92cc0a5880 msgr2=0x7f92cc06ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 shutdown_connections 2024-09-18T10:14:44.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:14:44.866+0000 7f92d2be5700 1 -- 172.21.15.142:0/3434082279 wait complete. 2024-09-18T10:14:44.890 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:14:44.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:14:45.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.309+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v154) v1 ==== 128+0+0 (secure 0 0 0) 0x7fcf5801a320 con 0x7fcf6009c8e0 2024-09-18T10:14:45.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.309+0000 7fcf567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:45.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.309+0000 7fcf567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.357+0000 7fcf557fa700 10 monclient: tick 2024-09-18T10:14:45.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.357+0000 7fcf557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:45.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.421+0000 7fcf683a2700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:45.428 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.421+0000 7fcf683a2700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:45.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.421+0000 7fcf683a2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:45.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:45.421+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]} v 0) v1 -- 0x7fcf6009cd40 con 0x7fcf6009c8e0 2024-09-18T10:14:46.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.309+0000 7fcf567fc700 1 -- 172.21.15.150:0/1511315584 <== mon.1 v2:172.21.15.150:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}]=0 v155) v1 ==== 128+0+0 (secure 0 0 0) 0x7fcf5801a550 con 0x7fcf6009c8e0 2024-09-18T10:14:46.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.309+0000 7fcf567fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [11.5]}] 2024-09-18T10:14:46.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.309+0000 7fcf567fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 msgr2=0x7fcf44037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 0x7fcf44037f70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009fd0 tx=0x7fcf500091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 10 monclient: shutdown 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 msgr2=0x7fcf44045800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf44045800 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007350 tx=0x7fcf580072e0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 shutdown_connections 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf6009bfd0 0x7fcf440481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf6009c8e0 0x7fcf44045800 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf44040b00 0x7fcf44042f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 --2- 172.21.15.150:0/1511315584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf44035ac0 0x7fcf44037f70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 >> 172.21.15.150:0/1511315584 conn(0x7fcf600a15e0 msgr2=0x7fcf6009fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:46.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 shutdown_connections 2024-09-18T10:14:46.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.313+0000 7fcf683a2700 1 -- 172.21.15.150:0/1511315584 wait complete. 2024-09-18T10:14:46.330 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight cold osd.0 12.5 12.4 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.421+0000 7f974130b700 10 monclient: get_monmap_and_config 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.421+0000 7f974130b700 10 monclient: build_initial_monmap 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient: monmap: 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:46.428085+0000 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:46.428085+0000 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:46.429 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:46.430 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:46.430 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:46.430 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:46.430 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 Processor -- start 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 -- start start 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient: init 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 0x7f973c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): picked mon.noname-b con 0x7f973c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:46.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f973c0a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:46.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f973c0a0ba0 0x7f973c13a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): picked mon.noname-a con 0x7f973c0a0ba0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f973c09e220 con 0x7f973c0a07d0 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973c09ee30 con 0x7f973c0a0ba0 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c09eaa0 con 0x7f973c09e6d0 2024-09-18T10:14:46.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:46.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f974130b700 10 monclient(hunting): authenticate will time out at 1168.521930s 2024-09-18T10:14:46.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973affd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 msgr2=0x7f973c0a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:46.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 0x7f973c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:46.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f973c0a0ba0 0x7f973c13a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:46.436 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f973c0a0ba0 0x7f973c13a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:43730/0 (socket says 172.21.15.150:43730) 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 -- 172.21.15.150:0/2187445003 learned_addr learned my addr 172.21.15.150:0/2187445003 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): get_auth_request con 0x7f973c0a0ba0 auth_method 0 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient(hunting): handle_auth_done global_id 54477 payload 274 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 -- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 msgr2=0x7f973c0a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 --2- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 0x7f973c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 -- 172.21.15.150:0/2187445003 >> v1:172.21.15.142:6789/0 conn(0x7f973c0a07d0 legacy=0x7f973c136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:46.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:46.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:14:46.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:46.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:46.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.425+0000 7f973b7fe700 1 -- 172.21.15.150:0/2187445003 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c09d930 con 0x7f973c0a0ba0 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f973b7fe700 1 -- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a0ba0 msgr2=0x7f973c13a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f973b7fe700 1 --2- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a0ba0 0x7f973c13a140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f20 tx=0x7f9734009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8389926c6bb3e9af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 1 -- 172.21.15.150:0/2187445003 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400e070 con 0x7f973c0a0ba0 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: dump: 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:46.444 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 5 monclient: authenticate success, global_id 54477 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 1 -- 172.21.15.150:0/2187445003 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973400bc90 con 0x7f973c0a0ba0 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:46.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 4 set_mon_vals no callback set 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 1 -- 172.21.15.150:0/2187445003 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734012660 con 0x7f973c0a0ba0 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f9739ffb700 10 monclient: dump: 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:46.446 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 10 monclient: shutdown 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 -- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a0ba0 msgr2=0x7f973c13a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 --2- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a0ba0 0x7f973c13a140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9734009f20 tx=0x7f9734009840 comp rx=0 tx=0).stop 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 -- 172.21.15.150:0/2187445003 shutdown_connections 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 --2- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a0ba0 0x7f973c13a140 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 --2- 172.21.15.150:0/2187445003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c09e6d0 0x7f973c0a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 -- 172.21.15.150:0/2187445003 >> 172.21.15.150:0/2187445003 conn(0x7f973c0a58a0 msgr2=0x7f973c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 -- 172.21.15.150:0/2187445003 shutdown_connections 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 1 -- 172.21.15.150:0/2187445003 wait complete. 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 10 monclient: build_initial_monmap 2024-09-18T10:14:46.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.429+0000 7f974130b700 10 monclient: monmap: 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 Processor -- start 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 -- start start 2024-09-18T10:14:46.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient: init 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): picked mon.noname-b con 0x7f973c09e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a07d0 0x7f973c08e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): picked mon.noname-a con 0x7f973c0a07d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 0x7f973c08e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): picked mon.noname-c con 0x7f973c0a0ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c08f8c0 con 0x7f973c0a07d0 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c08fdb0 con 0x7f973c09e6d0 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:46.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c0915d0 con 0x7f973c0a0ba0 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f974130b700 10 monclient(hunting): authenticate will time out at 1168.529513s 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46670/0 (socket says 172.21.15.150:46670) 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 -- 172.21.15.150:0/4217812252 learned_addr learned my addr 172.21.15.150:0/4217812252 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973b7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 msgr2=0x7f973c08e9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): get_auth_request con 0x7f973c09e6d0 auth_method 0 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973b7fe700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 0x7f973c08e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:46.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a07d0 0x7f973c08e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): get_auth_request con 0x7f973c0a07d0 auth_method 0 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient(hunting): handle_auth_done global_id 15682 payload 824 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 -- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 msgr2=0x7f973c08e9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:46.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 0x7f973c08e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 -- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a07d0 msgr2=0x7f973c08e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a07d0 0x7f973c08e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient: _finish_hunting 0 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 monclient: found mon.noname-b 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.433+0000 7f973affd700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734009ae0 con 0x7f973c09e6d0 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f973affd700 1 -- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 msgr2=0x7f973c08df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f973affd700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b810 tx=0x7f972c009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c003e20 con 0x7f973c09e6d0 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: dump: 2024-09-18T10:14:46.452 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c006150 con 0x7f973c09e6d0 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 5 monclient: authenticate success, global_id 15682 2024-09-18T10:14:46.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _renew_subs 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973c09bff0 con 0x7f973c09e6d0 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c006510 con 0x7f973c09e6d0 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 10 monclient: dump: 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:46.454 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f97397fa700 4 set_mon_vals no callback set 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _renew_subs 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973c08ef20 con 0x7f973c09e6d0 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f972c01b3b0 con 0x7f973c09e6d0 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 0x7f9728037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:46.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f973c0919d0 con 0x7f973c09e6d0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(155..155 src has 58..155) v4 ==== 9279+0+0 (secure 0 0 0) 0x7f972c048b10 con 0x7f973c09e6d0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f973a7fc700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 0x7f9728037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.437+0000 7f973a7fc700 10 monclient: get_auth_request con 0x7f9728035ac0 auth_method 0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.441+0000 7f973a7fc700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 0x7f9728037f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9730003a30 tx=0x7f9730007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.445+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f972c00e820 con 0x7f973c09e6d0 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.445+0000 7f9722ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:46.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.445+0000 7f9722ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:46.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.617+0000 7f974130b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:46.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.617+0000 7f974130b700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:46.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.617+0000 7f974130b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:46.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:46.617+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f973c091710 con 0x7f973c09e6d0 2024-09-18T10:14:47.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.037+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v156) v1 ==== 134+0+0 (secure 0 0 0) 0x7f973c091710 con 0x7f973c09e6d0 2024-09-18T10:14:47.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.037+0000 7f9722ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:47.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.037+0000 7f9722ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:47.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.149+0000 7f974130b700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:47.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.149+0000 7f974130b700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:47.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.149+0000 7f974130b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:47.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:47.149+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]} v 0) v1 -- 0x7f973c091710 con 0x7f973c09e6d0 2024-09-18T10:14:48.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.049+0000 7f9722ffd700 1 -- 172.21.15.150:0/4217812252 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}]=0 v157) v1 ==== 134+0+0 (secure 0 0 0) 0x7f973c091710 con 0x7f973c09e6d0 2024-09-18T10:14:48.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.049+0000 7f9722ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [12.5, 12.4]}] 2024-09-18T10:14:48.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.049+0000 7f9722ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:48.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 msgr2=0x7f9728037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:48.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 0x7f9728037f70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9730003a30 tx=0x7f9730007ea0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 10 monclient: shutdown 2024-09-18T10:14:48.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:48.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 msgr2=0x7f973c08df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b810 tx=0x7f972c009220 comp rx=0 tx=0).stop 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 shutdown_connections 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9728035ac0 0x7f9728037f70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f973c0a0ba0 0x7f973c08e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f973c0a07d0 0x7f973c08e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 --2- 172.21.15.150:0/4217812252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f973c09e6d0 0x7f973c08df60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 >> 172.21.15.150:0/4217812252 conn(0x7f973c0a58a0 msgr2=0x7f973c09a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 shutdown_connections 2024-09-18T10:14:48.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.053+0000 7f974130b700 1 -- 172.21.15.150:0/4217812252 wait complete. 2024-09-18T10:14:48.075 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush set-device-class fish osd.0 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: get_monmap_and_config 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: build_initial_monmap 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: monmap: 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:48.182 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:48.181489+0000 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:48.181489+0000 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:48.183 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 Processor -- start 2024-09-18T10:14:48.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 -- start start 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: init 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe580699a0 0x7fbe5809bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-c con 0x7fbe580699a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 0x7fbe5809c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-a con 0x7fbe58098c60 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe58134760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-b con 0x7fbe5809ca50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5809b920 con 0x7fbe58098c60 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe5809ce70 con 0x7fbe580699a0 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5809a380 con 0x7fbe5809ca50 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe600f4700 10 monclient(hunting): authenticate will time out at 1170.274863s 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe580699a0 0x7fbe5809bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5e691700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 msgr2=0x7fbe58134760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:48.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe580699a0 0x7fbe5809bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46676/0 (socket says 172.21.15.150:46676) 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 -- 172.21.15.150:0/2912978075 learned_addr learned my addr 172.21.15.150:0/2912978075 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): get_auth_request con 0x7fbe580699a0 auth_method 0 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5e691700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe58134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:48.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 1 --2- 172.21.15.150:0/2912978075 >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 0x7fbe5809c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 10 monclient(hunting): get_auth_request con 0x7fbe58098c60 auth_method 0 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient(hunting): handle_auth_done global_id 15685 payload 274 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 -- 172.21.15.150:0/2912978075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 msgr2=0x7fbe58134760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe58134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 -- 172.21.15.150:0/2912978075 >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 msgr2=0x7fbe5809c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 --2- 172.21.15.150:0/2912978075 >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 0x7fbe5809c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient: _finish_hunting 0 2024-09-18T10:14:48.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 monclient: found mon.noname-c 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5de90700 1 -- 172.21.15.150:0/2912978075 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5809a0a0 con 0x7fbe580699a0 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.177+0000 7fbe5d68f700 1 --2- 172.21.15.150:0/2912978075 >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 0x7fbe5809c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5de90700 1 -- 172.21.15.150:0/2912978075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 msgr2=0x7fbe5809bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5de90700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5809bfd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbe5804bb70 tx=0x7fbe54009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f6dc1839fc19295 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 1 -- 172.21.15.150:0/2912978075 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5400ba70 con 0x7fbe580699a0 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: dump: 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:48.193 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: _finish_auth 0 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 1 -- 172.21.15.150:0/2912978075 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5400bbd0 con 0x7fbe580699a0 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 5 monclient: authenticate success, global_id 15685 2024-09-18T10:14:48.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 4 set_mon_vals no callback set 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 1 -- 172.21.15.150:0/2912978075 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54011420 con 0x7fbe580699a0 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe5ce8e700 10 monclient: dump: 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 10 monclient: shutdown 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 -- 172.21.15.150:0/2912978075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 msgr2=0x7fbe5809bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:48.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5809bfd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fbe5804bb70 tx=0x7fbe54009250 comp rx=0 tx=0).stop 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 -- 172.21.15.150:0/2912978075 shutdown_connections 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe58134760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 --2- 172.21.15.150:0/2912978075 >> v2:172.21.15.142:3300/0 conn(0x7fbe58098c60 0x7fbe5809c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 --2- 172.21.15.150:0/2912978075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5809bfd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 -- 172.21.15.150:0/2912978075 >> 172.21.15.150:0/2912978075 conn(0x7fbe580a15e0 msgr2=0x7fbe580a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 -- 172.21.15.150:0/2912978075 shutdown_connections 2024-09-18T10:14:48.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.181+0000 7fbe600f4700 1 -- 172.21.15.150:0/2912978075 wait complete. 2024-09-18T10:14:48.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient: build_initial_monmap 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient: monmap: 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 Processor -- start 2024-09-18T10:14:48.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 -- start start 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient: init 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-b con 0x7fbe580699a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe58098c60 0x7fbe58089e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-a con 0x7fbe58098c60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe5808a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): picked mon.noname-c con 0x7fbe5809ca50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5806f760 con 0x7fbe58098c60 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58097d80 con 0x7fbe580699a0 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe58094940 con 0x7fbe5809ca50 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe600f4700 10 monclient(hunting): authenticate will time out at 1170.283207s 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46682/0 (socket says 172.21.15.150:46682) 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 -- 172.21.15.150:0/995069307 learned_addr learned my addr 172.21.15.150:0/995069307 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): get_auth_request con 0x7fbe580699a0 auth_method 0 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5e691700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 msgr2=0x7fbe5808a3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:48.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5e691700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe5808a3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe58098c60 0x7fbe58089e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): get_auth_request con 0x7fbe58098c60 auth_method 0 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5d68f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient(hunting): handle_auth_done global_id 15688 payload 824 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 msgr2=0x7fbe5808a3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:48.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe5808a3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe58098c60 msgr2=0x7fbe58089e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe58098c60 0x7fbe58089e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient: _finish_hunting 0 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 monclient: found mon.noname-b 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.185+0000 7fbe5de90700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540094f0 con 0x7fbe580699a0 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.189+0000 7fbe5de90700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 msgr2=0x7fbe5808cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.189+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:14:48.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.189+0000 7fbe5de90700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009290 tx=0x7fbe54017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5401a660 con 0x7fbe580699a0 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: dump: 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:48.208 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5401a7e0 con 0x7fbe580699a0 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5401aba0 con 0x7fbe580699a0 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 10 monclient: dump: 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.209 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 5 monclient: authenticate success, global_id 15688 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _renew_subs 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4ffff700 4 set_mon_vals no callback set 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5808b7d0 con 0x7fbe580699a0 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _renew_subs 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5808d2a0 con 0x7fbe580699a0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbe54010380 con 0x7fbe580699a0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe58099d90 con 0x7fbe580699a0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 0x7fbe3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(157..157 src has 58..157) v4 ==== 9279+0+0 (secure 0 0 0) 0x7fbe5404aa30 con 0x7fbe580699a0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe5d68f700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 0x7fbe3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:48.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.193+0000 7fbe5d68f700 10 monclient: get_auth_request con 0x7fbe3c035ac0 auth_method 0 2024-09-18T10:14:48.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.197+0000 7fbe5d68f700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 0x7fbe3c037f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbe5808b460 tx=0x7fbe48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:48.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.201+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbe54018540 con 0x7fbe580699a0 2024-09-18T10:14:48.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.201+0000 7fbe4dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:48.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.201+0000 7fbe4dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.373+0000 7fbe600f4700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.373+0000 7fbe600f4700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.373+0000 7fbe600f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:48.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:48.373+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fbe58067b10 con 0x7fbe580699a0 2024-09-18T10:14:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.061+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'fish' v158) v1 ==== 137+0+0 (secure 0 0 0) 0x7fbe54020020 con 0x7fbe580699a0 2024-09-18T10:14:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.061+0000 7fbe4dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.061+0000 7fbe4dffb700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'fish' 2024-09-18T10:14:49.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.169+0000 7fbe600f4700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:49.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.169+0000 7fbe600f4700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:49.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.169+0000 7fbe600f4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:49.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:49.169+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]} v 0) v1 -- 0x7fbe58063ed0 con 0x7fbe580699a0 2024-09-18T10:14:50.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.061+0000 7fbe4dffb700 1 -- 172.21.15.150:0/995069307 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}]=0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' v159) v1 ==== 241+0+0 (secure 0 0 0) 0x7fbe54020070 con 0x7fbe580699a0 2024-09-18T10:14:50.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.061+0000 7fbe4dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "fish", "ids": ["osd.0"]}] 2024-09-18T10:14:50.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.061+0000 7fbe4dffb700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-18T10:14:50.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 msgr2=0x7fbe3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 0x7fbe3c037f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbe5808b460 tx=0x7fbe48008040 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 10 monclient: shutdown 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 msgr2=0x7fbe5808cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009290 tx=0x7fbe54017000 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 shutdown_connections 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe3c035ac0 0x7fbe3c037f70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe5809ca50 0x7fbe5808a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe58098c60 0x7fbe58089e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 --2- 172.21.15.150:0/995069307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe580699a0 0x7fbe5808cbd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 >> 172.21.15.150:0/995069307 conn(0x7fbe580a15e0 msgr2=0x7fbe5809f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:50.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 shutdown_connections 2024-09-18T10:14:50.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.065+0000 7fbe600f4700 1 -- 172.21.15.150:0/995069307 wait complete. 2024-09-18T10:14:50.073 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 already set to class fish. set-device-class item id 0 name 'osd.0' device_class 'fish': no change. set osd(s) to class 'fish' 2024-09-18T10:14:50.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:50.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:50.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep fish 2024-09-18T10:14:50.088 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '10\.' 2024-09-18T10:14:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: get_monmap_and_config 2024-09-18T10:14:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: build_initial_monmap 2024-09-18T10:14:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: monmap: 2024-09-18T10:14:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:50.186590+0000 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:50.186590+0000 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:50.188 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:50.189 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:50.189 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:50.189 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 Processor -- start 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 -- start start 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: init 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 0x7f959009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): picked mon.noname-b con 0x7f95900a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f95900a04f0 0x7f95900a0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): picked mon.noname-a con 0x7f95900a04f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95901369a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959009dae0 con 0x7f95901369a0 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900a21d0 con 0x7f95900a04f0 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959009ddc0 con 0x7f95900a0120 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:50.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.181+0000 7f9595099700 10 monclient(hunting): authenticate will time out at 1172.279997s 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958f59e700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95901369a0 0x7f9590138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55868/0 (socket says 172.21.15.150:55868) 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958f59e700 1 -- 172.21.15.150:0/1364304993 learned_addr learned my addr 172.21.15.150:0/1364304993 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958ed9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 msgr2=0x7f959009e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958ed9d700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 0x7f959009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 --2- 172.21.15.150:0/1364304993 >> v2:172.21.15.142:3300/0 conn(0x7f95900a04f0 0x7f95900a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): get_auth_request con 0x7f95900a04f0 auth_method 0 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient(hunting): handle_auth_done global_id 54489 payload 274 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 -- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 msgr2=0x7f959009e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 0x7f959009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 -- 172.21.15.150:0/1364304993 >> v1:172.21.15.150:6789/0 conn(0x7f95901369a0 legacy=0x7f9590138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 monclient: found mon.noname-a 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:50.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 -- 172.21.15.150:0/1364304993 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900a1590 con 0x7f95900a04f0 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 -- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95900a04f0 msgr2=0x7f95900a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958e59c700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95900a04f0 0x7f95900a0920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95800098e0 tx=0x7f95800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44459e215c46e05 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 1 -- 172.21.15.150:0/1364304993 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000ba70 con 0x7f95900a04f0 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: dump: 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.198 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f9595099700 5 monclient: authenticate success, global_id 54489 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 1 -- 172.21.15.150:0/1364304993 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000bbf0 con 0x7f95900a04f0 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:50.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f9595099700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f9595099700 4 set_mon_vals no callback set 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 1 -- 172.21.15.150:0/1364304993 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580011420 con 0x7f95900a04f0 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.185+0000 7f958dd9b700 10 monclient: dump: 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: shutdown 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95900a04f0 msgr2=0x7f95900a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95900a04f0 0x7f95900a0920 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95800098e0 tx=0x7f95800091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- 172.21.15.150:0/1364304993 shutdown_connections 2024-09-18T10:14:50.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95900a04f0 0x7f95900a0920 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- 172.21.15.150:0/1364304993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a0120 0x7f959009e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- 172.21.15.150:0/1364304993 >> 172.21.15.150:0/1364304993 conn(0x7f95900a5830 msgr2=0x7f95900a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- 172.21.15.150:0/1364304993 shutdown_connections 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- 172.21.15.150:0/1364304993 wait complete. 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: build_initial_monmap 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: monmap: 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:50.202 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 Processor -- start 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- start start 2024-09-18T10:14:50.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: init 2024-09-18T10:14:50.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:50.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): picked mon.noname-b con 0x7f95900a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 0x7f9590142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): picked mon.noname-c con 0x7f95900a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95901369a0 0x7f95901428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): picked mon.noname-a con 0x7f95901369a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959012eba0 con 0x7f95901369a0 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959009d720 con 0x7f95900a0120 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959009c010 con 0x7f95900a04f0 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f9595099700 10 monclient(hunting): authenticate will time out at 1172.286232s 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958e59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 msgr2=0x7f9590142360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 0x7f9590142360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52892/0 (socket says 172.21.15.150:52892) 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 -- 172.21.15.150:0/2115675909 learned_addr learned my addr 172.21.15.150:0/2115675909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95901369a0 0x7f95901428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): get_auth_request con 0x7f95900a0120 auth_method 0 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): get_auth_request con 0x7f95901369a0 auth_method 0 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:50.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient(hunting): handle_auth_done global_id 15691 payload 824 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 -- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 msgr2=0x7f9590142360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 0x7f9590142360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 -- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95901369a0 msgr2=0x7f95901428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95901369a0 0x7f95901428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 monclient: found mon.noname-b 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:50.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.189+0000 7f958ed9d700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009420 con 0x7f95900a0120 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f958ed9d700 1 -- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 msgr2=0x7f9590141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f958ed9d700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f958400b7e0 tx=0x7f9584009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584003e20 con 0x7f95900a0120 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: dump: 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.212 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 5 monclient: authenticate success, global_id 15691 2024-09-18T10:14:50.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584006150 con 0x7f95900a0120 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584006510 con 0x7f95900a0120 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 10 monclient: dump: 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.214 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f958d59a700 4 set_mon_vals no callback set 2024-09-18T10:14:50.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _renew_subs 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9590130db0 con 0x7f95900a0120 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _renew_subs 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959012e6d0 con 0x7f95900a0120 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f95840103c0 con 0x7f95900a0120 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 0x7f957c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(159..159 src has 58..159) v4 ==== 9944+0+0 (secure 0 0 0) 0x7f9584049030 con 0x7f95900a0120 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959009d8b0 con 0x7f95900a0120 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.193+0000 7f958e59c700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 0x7f957c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.197+0000 7f958e59c700 10 monclient: get_auth_request con 0x7f957c035a70 auth_method 0 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.197+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.197+0000 7f958e59c700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 0x7f957c037f20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f95800098e0 tx=0x7f9580016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.201+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9584017030 con 0x7f95900a0120 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.201+0000 7f9576ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.201+0000 7f9576ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:50.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9595099700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9595099700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9595099700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f95900a1c90 con 0x7f95900a0120 2024-09-18T10:14:50.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f9584017020 con 0x7f95900a0120 2024-09-18T10:14:50.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9576ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.377+0000 7f9576ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:50.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9595099700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9595099700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9595099700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9590063ed0 con 0x7f95900a0120 2024-09-18T10:14:50.492 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9576ffd700 1 -- 172.21.15.150:0/2115675909 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f958400ed30 con 0x7f95900a0120 2024-09-18T10:14:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9576ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.485+0000 7f9576ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:50.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 msgr2=0x7f957c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 0x7f957c037f20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f95800098e0 tx=0x7f9580016000 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 10 monclient: shutdown 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 msgr2=0x7f9590141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f958400b7e0 tx=0x7f9584009220 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 shutdown_connections 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f957c035a70 0x7f957c037f20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95901369a0 0x7f95901428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95900a04f0 0x7f9590142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 --2- 172.21.15.150:0/2115675909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95900a0120 0x7f9590141e20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 >> 172.21.15.150:0/2115675909 conn(0x7f95900a5830 msgr2=0x7f959009abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 shutdown_connections 2024-09-18T10:14:50.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.489+0000 7f9595099700 1 -- 172.21.15.150:0/2115675909 wait complete. 2024-09-18T10:14:50.509 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:50.509 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:50.509 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:50.509 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:50.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep fish 2024-09-18T10:14:50.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '11\.' 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: get_monmap_and_config 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: build_initial_monmap 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: monmap: 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:50.608546+0000 2024-09-18T10:14:50.609 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:50.608546+0000 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:50.610 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 Processor -- start 2024-09-18T10:14:50.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 -- start start 2024-09-18T10:14:50.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: init 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06340710a0 0x7f0634096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): picked mon.noname-c con 0x7f06340710a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06340999c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:50.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 0x7f0634136540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): picked mon.noname-b con 0x7f06340976c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f063409e040 con 0x7f06340999c0 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f063406f730 con 0x7f06340710a0 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0634071470 con 0x7f06340976c0 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0638966700 10 monclient(hunting): authenticate will time out at 1172.703464s 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06340710a0 0x7f0634096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0632d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 msgr2=0x7f0634136540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:50.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06340710a0 0x7f0634096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52904/0 (socket says 172.21.15.150:52904) 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 1 -- 172.21.15.150:0/1555760087 learned_addr learned my addr 172.21.15.150:0/1555760087 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0629d9b700 1 --1- 172.21.15.150:0/1555760087 >> v1:172.21.15.150:6789/0 conn(0x7f06340999c0 0x7f0634096fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55882/0 (socket says 172.21.15.150:55882) 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 10 monclient(hunting): get_auth_request con 0x7f06340710a0 auth_method 0 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f063259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.605+0000 7f0632d9d700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 0x7f0634136540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:50.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f0629d9b700 1 -- 172.21.15.150:0/1555760087 >> v1:172.21.15.150:6789/0 conn(0x7f06340999c0 legacy=0x7f0634096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient(hunting): handle_auth_done global_id 15694 payload 274 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 1 -- 172.21.15.150:0/1555760087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 msgr2=0x7f0634136540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 0x7f0634136540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 1 -- 172.21.15.150:0/1555760087 >> v1:172.21.15.150:6789/0 conn(0x7f06340999c0 legacy=0x7f0634096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient: _finish_hunting 0 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 1 monclient: found mon.noname-c 2024-09-18T10:14:50.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.609+0000 7f063259c700 1 -- 172.21.15.150:0/1555760087 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0634098c10 con 0x7f06340710a0 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f063259c700 1 -- 172.21.15.150:0/1555760087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340710a0 msgr2=0x7f0634096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f063259c700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340710a0 0x7f0634096a70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0624009a30 tx=0x7f0624009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0f218c1a8c5f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 1 -- 172.21.15.150:0/1555760087 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062400ba50 con 0x7f06340710a0 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: dump: 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.620 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: _finish_auth 0 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 5 monclient: authenticate success, global_id 15694 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 1 -- 172.21.15.150:0/1555760087 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062400bbb0 con 0x7f06340710a0 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:50.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 4 set_mon_vals no callback set 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 1 -- 172.21.15.150:0/1555760087 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624011420 con 0x7f06340710a0 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0631d9b700 10 monclient: dump: 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.622 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 10 monclient: shutdown 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- 172.21.15.150:0/1555760087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340710a0 msgr2=0x7f0634096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340710a0 0x7f0634096a70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0624009a30 tx=0x7f0624009250 comp rx=0 tx=0).stop 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- 172.21.15.150:0/1555760087 shutdown_connections 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340976c0 0x7f0634136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.623 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 --2- 172.21.15.150:0/1555760087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340710a0 0x7f0634096a70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- 172.21.15.150:0/1555760087 >> 172.21.15.150:0/1555760087 conn(0x7f0634068700 msgr2=0x7f06340a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:50.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- 172.21.15.150:0/1555760087 shutdown_connections 2024-09-18T10:14:50.624 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- 172.21.15.150:0/1555760087 wait complete. 2024-09-18T10:14:50.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 10 monclient: build_initial_monmap 2024-09-18T10:14:50.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 10 monclient: monmap: 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 Processor -- start 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 1 -- start start 2024-09-18T10:14:50.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.613+0000 7f0638966700 10 monclient: init 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 0x7f0634139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): picked mon.noname-c con 0x7f06340710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): picked mon.noname-b con 0x7f06340976c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 0x7f063413d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): picked mon.noname-a con 0x7f063411d670 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063411ede0 con 0x7f063411d670 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063411e820 con 0x7f06340976c0 2024-09-18T10:14:50.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f063259c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 msgr2=0x7f0634139df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f063259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 0x7f0634139df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063411c2b0 con 0x7f06340710a0 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0638966700 10 monclient(hunting): authenticate will time out at 1172.713297s 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52918/0 (socket says 172.21.15.150:52918) 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 -- 172.21.15.150:0/2710502308 learned_addr learned my addr 172.21.15.150:0/2710502308 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): get_auth_request con 0x7f06340976c0 auth_method 0 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 0x7f063413d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 10 monclient(hunting): get_auth_request con 0x7f063411d670 auth_method 0 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient(hunting): handle_auth_done global_id 15700 payload 824 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 msgr2=0x7f0634139df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 0x7f0634139df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 msgr2=0x7f063413d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 0x7f063413d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 monclient: found mon.noname-b 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0629d9b700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240094f0 con 0x7f06340976c0 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.617+0000 7f0632d9d700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 0x7f063413d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0629d9b700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 msgr2=0x7f06341394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:50.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0629d9b700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c7c0 tx=0x7f061c007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c007510 con 0x7f06340976c0 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: dump: 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.635 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: _finish_auth 0 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 5 monclient: authenticate success, global_id 15700 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061c007690 con 0x7f06340976c0 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _renew_subs 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063411e540 con 0x7f06340976c0 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061c004a80 con 0x7f06340976c0 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:50.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 10 monclient: dump: 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:50.637 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063159a700 4 set_mon_vals no callback set 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _renew_subs 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341399e0 con 0x7f06340976c0 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f061c01c3b0 con 0x7f06340976c0 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0634063ed0 con 0x7f06340976c0 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:50.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(159..159 src has 58..159) v4 ==== 9944+0+0 (secure 0 0 0) 0x7f061c048d30 con 0x7f06340976c0 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 10 monclient: get_auth_request con 0x7f0614035aa0 auth_method 0 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 msgr2=0x7f0614037f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 msgr2=0x7f0614037f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.621+0000 7f063259c700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.629+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f061c004e30 con 0x7f06340976c0 2024-09-18T10:14:50.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.629+0000 7f062affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:50.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.629+0000 7f062affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:50.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f0638966700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f0638966700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f0638966700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f0634068d20 con 0x7f06340976c0 2024-09-18T10:14:50.806 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f061c018020 con 0x7f06340976c0 2024-09-18T10:14:50.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f062affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.801+0000 7f062affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:50.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.825+0000 7f063259c700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:50.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.825+0000 7f063259c700 10 monclient: get_auth_request con 0x7f0614035aa0 auth_method 0 2024-09-18T10:14:50.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.825+0000 7f063259c700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0624009a30 tx=0x7f0624016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:50.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.901+0000 7f0638966700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.901+0000 7f0638966700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.901+0000 7f0638966700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:50.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.901+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f06340999c0 con 0x7f06340976c0 2024-09-18T10:14:50.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f062affd700 1 -- 172.21.15.150:0/2710502308 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7f061c00e290 con 0x7f06340976c0 2024-09-18T10:14:50.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f062affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:50.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f062affd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 msgr2=0x7f0614037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0624009a30 tx=0x7f0624016000 comp rx=0 tx=0).stop 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 10 monclient: shutdown 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 msgr2=0x7f06341394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f061c00c7c0 tx=0x7f061c007d20 comp rx=0 tx=0).stop 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 shutdown_connections 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0614035aa0 0x7f0614037f50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f063411d670 0x7f063413d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06340976c0 0x7f06341394a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 --2- 172.21.15.150:0/2710502308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06340710a0 0x7f0634139df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:50.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 >> 172.21.15.150:0/2710502308 conn(0x7f0634068700 msgr2=0x7f063409fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:50.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 shutdown_connections 2024-09-18T10:14:50.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:50.905+0000 7f0638966700 1 -- 172.21.15.150:0/2710502308 wait complete. 2024-09-18T10:14:50.925 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:50.925 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:50.926 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:50.926 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:50.926 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep fish 2024-09-18T10:14:50.926 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '12\.' 2024-09-18T10:14:51.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.021+0000 7fd6b11e3700 10 monclient: get_monmap_and_config 2024-09-18T10:14:51.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.021+0000 7fd6b11e3700 10 monclient: build_initial_monmap 2024-09-18T10:14:51.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.021+0000 7fd6b11e3700 10 monclient: monmap: 2024-09-18T10:14:51.028 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:51.028 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:51.026927+0000 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:51.026927+0000 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:51.029 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:51.030 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:51.030 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:51.030 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:51.030 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 Processor -- start 2024-09-18T10:14:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 -- start start 2024-09-18T10:14:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient: init 2024-09-18T10:14:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:51.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6ac071a40 0x7fd6ac09ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-c con 0x7fd6ac071a40 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6ac06e7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6ac09f2a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac071e10 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac096c70 con 0x7fd6ac06e7f0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac06d4a0 con 0x7fd6ac071a40 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6b11e3700 10 monclient(hunting): authenticate will time out at 1173.121758s 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6ac071a40 0x7fd6ac09ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6ac071a40 0x7fd6ac09ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52934/0 (socket says 172.21.15.150:52934) 2024-09-18T10:14:51.035 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 1 -- 172.21.15.150:0/281575538 learned_addr learned my addr 172.21.15.150:0/281575538 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aa59c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd6ac06e7f0 0x7fd6ac06c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55898/0 (socket says 172.21.15.150:55898) 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 10 monclient(hunting): get_auth_request con 0x7fd6ac071a40 auth_method 0 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.025+0000 7fd6aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aa59c700 1 -- 172.21.15.150:0/281575538 >> v1:172.21.15.150:6789/0 conn(0x7fd6ac06e7f0 legacy=0x7fd6ac06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_done global_id 15703 payload 274 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 -- 172.21.15.150:0/281575538 >> v1:172.21.15.150:6789/0 conn(0x7fd6ac06e7f0 legacy=0x7fd6ac06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6ab59e700 1 -- 172.21.15.150:0/281575538 >> v1:172.21.15.142:6789/0 conn(0x7fd6ac09f2a0 legacy=0x7fd6ac06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 -- 172.21.15.150:0/281575538 >> v1:172.21.15.142:6789/0 conn(0x7fd6ac09f2a0 legacy=0x7fd6ac06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 monclient: found mon.noname-c 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.037 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 -- 172.21.15.150:0/281575538 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac130470 con 0x7fd6ac071a40 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 -- 172.21.15.150:0/281575538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac071a40 msgr2=0x7fd6ac09ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6aad9d700 1 --2- 172.21.15.150:0/281575538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac071a40 0x7fd6ac09ed60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac130660 tx=0x7fd69c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cd855cb6a72ceb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 1 -- 172.21.15.150:0/281575538 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00bc20 con 0x7fd6ac071a40 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: dump: 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.042 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6b11e3700 5 monclient: authenticate success, global_id 15703 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 1 -- 172.21.15.150:0/281575538 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c00bd80 con 0x7fd6ac071a40 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:51.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6b11e3700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6b11e3700 4 set_mon_vals no callback set 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 1 -- 172.21.15.150:0/281575538 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c011670 con 0x7fd6ac071a40 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.029+0000 7fd6a9d9b700 10 monclient: dump: 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.044 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: shutdown 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- 172.21.15.150:0/281575538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac071a40 msgr2=0x7fd6ac09ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/281575538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac071a40 0x7fd6ac09ed60 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac130660 tx=0x7fd69c009250 comp rx=0 tx=0).stop 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- 172.21.15.150:0/281575538 shutdown_connections 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/281575538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac071a40 0x7fd6ac09ed60 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- 172.21.15.150:0/281575538 >> 172.21.15.150:0/281575538 conn(0x7fd6ac0a41f0 msgr2=0x7fd6ac0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- 172.21.15.150:0/281575538 shutdown_connections 2024-09-18T10:14:51.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- 172.21.15.150:0/281575538 wait complete. 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: build_initial_monmap 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: monmap: 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 Processor -- start 2024-09-18T10:14:51.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- start start 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: init 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6ac06e7f0 0x7fd6ac13f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-a con 0x7fd6ac06e7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 0x7fd6ac12f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-c con 0x7fd6ac071a40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): picked mon.noname-b con 0x7fd6ac09f2a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac140670 con 0x7fd6ac06e7f0 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac11baf0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac06ed10 con 0x7fd6ac071a40 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6b11e3700 10 monclient(hunting): authenticate will time out at 1173.130913s 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aa59c700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 msgr2=0x7fd6ac12f460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aa59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 0x7fd6ac12f460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52950/0 (socket says 172.21.15.150:52950) 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 -- 172.21.15.150:0/1291752607 learned_addr learned my addr 172.21.15.150:0/1291752607 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): get_auth_request con 0x7fd6ac09f2a0 auth_method 0 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6ac06e7f0 0x7fd6ac13f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): get_auth_request con 0x7fd6ac06e7f0 auth_method 0 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:51.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient(hunting): handle_auth_done global_id 15709 payload 824 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 -- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 msgr2=0x7fd6ac12f460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 0x7fd6ac12f460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 -- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6ac06e7f0 msgr2=0x7fd6ac13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6ac06e7f0 0x7fd6ac13f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient: _finish_hunting 0 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 monclient: found mon.noname-b 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.033+0000 7fd6ab59e700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c0094f0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6ab59e700 1 -- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 msgr2=0x7fd6ac12f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6ab59e700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c970 tx=0x7fd6a4001970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400ce60 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.052 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: dump: 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:51.053 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400a550 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 5 monclient: authenticate success, global_id 15709 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 10 monclient: _renew_subs 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac11d590 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a400a8f0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd692ffd700 10 monclient: dump: 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.054 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6a959a700 4 set_mon_vals no callback set 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 10 monclient: _renew_subs 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.037+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac11c630 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6b11e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6b11e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6b11e3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac11dc70 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd6a400aca0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd692ffd700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 0x7fd698037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(159..159 src has 58..159) v4 ==== 9944+0+0 (secure 0 0 0) 0x7fd6a4048eb0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6aad9d700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 0x7fd698037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6aad9d700 10 monclient: get_auth_request con 0x7fd698035ac0 auth_method 0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.041+0000 7fd6aad9d700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 0x7fd698037f70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac130660 tx=0x7fd69c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.049+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd6a400f840 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.049+0000 7fd692ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.049+0000 7fd692ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:51.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd6b11e3700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd6b11e3700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd6b11e3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd6ac063ed0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fd6a4017070 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd692ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.221+0000 7fd692ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd6b11e3700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd6b11e3700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd6b11e3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd6ac0008d0 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd692ffd700 1 -- 172.21.15.150:0/1291752607 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v159) v1 ==== 83+0+2175 (secure 0 0 0) 0x7fd6a400f260 con 0x7fd6ac09f2a0 2024-09-18T10:14:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd692ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.321+0000 7fd692ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 msgr2=0x7fd698037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 0x7fd698037f70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac130660 tx=0x7fd69c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 10 monclient: shutdown 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 msgr2=0x7fd6ac12f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c970 tx=0x7fd6a4001970 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 shutdown_connections 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd698035ac0 0x7fd698037f70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6ac09f2a0 0x7fd6ac12f9a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6ac071a40 0x7fd6ac12f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 --2- 172.21.15.150:0/1291752607 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6ac06e7f0 0x7fd6ac13f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 >> 172.21.15.150:0/1291752607 conn(0x7fd6ac0a41f0 msgr2=0x7fd6ac130d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 shutdown_connections 2024-09-18T10:14:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.325+0000 7fd6b11e3700 1 -- 172.21.15.150:0/1291752607 wait complete. 2024-09-18T10:14:51.340 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:51.340 INFO:tasks.workunit.client.0.smithi150.stdout: 0 fish 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:51.340 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T10:14:51.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.425+0000 7fa2ed414700 10 monclient: get_monmap_and_config 2024-09-18T10:14:51.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.425+0000 7fa2ed414700 10 monclient: build_initial_monmap 2024-09-18T10:14:51.432 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.425+0000 7fa2ed414700 10 monclient: monmap: 2024-09-18T10:14:51.432 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:51.432 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:51.431694+0000 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:51.431694+0000 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:51.433 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 Processor -- start 2024-09-18T10:14:51.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 -- start start 2024-09-18T10:14:51.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient: init 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2e81328b0 0x7fa2e8132c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-c con 0x7fa2e81328b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa2e8098070 0x7fa2e809c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-a con 0x7fa2e8098070 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2e809db20 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e80970c0 con 0x7fa2e809db20 2024-09-18T10:14:51.437 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e8096de0 con 0x7fa2e8098070 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e81331c0 con 0x7fa2e81328b0 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2ed414700 10 monclient(hunting): authenticate will time out at 1173.526548s 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e6ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2e81328b0 0x7fa2e8132c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e77fe700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa2e809db20 0x7fa2e809cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55914/0 (socket says 172.21.15.150:55914) 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e77fe700 1 -- 172.21.15.150:0/3421179180 learned_addr learned my addr 172.21.15.150:0/3421179180 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e6ffd700 10 monclient(hunting): get_auth_request con 0x7fa2e81328b0 auth_method 0 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/3421179180 >> v2:172.21.15.142:3300/0 conn(0x7fa2e8098070 0x7fa2e809c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): get_auth_request con 0x7fa2e8098070 auth_method 0 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_done global_id 54501 payload 274 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 -- 172.21.15.150:0/3421179180 >> v2:172.21.15.150:3300/0 conn(0x7fa2e81328b0 msgr2=0x7fa2e8132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/3421179180 >> v2:172.21.15.150:3300/0 conn(0x7fa2e81328b0 0x7fa2e8132c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 -- 172.21.15.150:0/3421179180 >> v1:172.21.15.150:6789/0 conn(0x7fa2e809db20 legacy=0x7fa2e809cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:14:51.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 monclient: found mon.noname-a 2024-09-18T10:14:51.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:51.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:51.440 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.429+0000 7fa2e67fc700 1 -- 172.21.15.150:0/3421179180 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8133330 con 0x7fa2e8098070 2024-09-18T10:14:51.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e67fc700 1 -- 172.21.15.150:0/3421179180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 msgr2=0x7fa2e809c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/3421179180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e809c6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8068bd0 tx=0x7fa2dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a496461cf953d166 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 1 -- 172.21.15.150:0/3421179180 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00b890 con 0x7fa2e8098070 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: dump: 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.442 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 5 monclient: authenticate success, global_id 54501 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 1 -- 172.21.15.150:0/3421179180 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc00b9f0 con 0x7fa2e8098070 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:51.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 4 set_mon_vals no callback set 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 1 -- 172.21.15.150:0/3421179180 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00bd90 con 0x7fa2e8098070 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2e5ffb700 10 monclient: dump: 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.444 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 10 monclient: shutdown 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 -- 172.21.15.150:0/3421179180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 msgr2=0x7fa2e809c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 --2- 172.21.15.150:0/3421179180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e809c6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8068bd0 tx=0x7fa2dc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 -- 172.21.15.150:0/3421179180 shutdown_connections 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 --2- 172.21.15.150:0/3421179180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e809c6b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 --2- 172.21.15.150:0/3421179180 >> v2:172.21.15.150:3300/0 conn(0x7fa2e81328b0 0x7fa2e8132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.433+0000 7fa2ed414700 1 -- 172.21.15.150:0/3421179180 >> 172.21.15.150:0/3421179180 conn(0x7fa2e80a1cc0 msgr2=0x7fa2e80a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- 172.21.15.150:0/3421179180 shutdown_connections 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- 172.21.15.150:0/3421179180 wait complete. 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient: build_initial_monmap 2024-09-18T10:14:51.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient: monmap: 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:51.446 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 Processor -- start 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- start start 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient: init 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e81419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-a con 0x7fa2e8098070 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-b con 0x7fa2e809db20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 0x7fa2e811c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): picked mon.noname-c con 0x7fa2e81328b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81428e0 con 0x7fa2e8098070 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e809e000 con 0x7fa2e809db20 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e811d6e0 con 0x7fa2e81328b0 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2ed414700 10 monclient(hunting): authenticate will time out at 1173.534827s 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e77fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 msgr2=0x7fa2e811c270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:51.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 0x7fa2e811c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52968/0 (socket says 172.21.15.150:52968) 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 -- 172.21.15.150:0/2795169714 learned_addr learned my addr 172.21.15.150:0/2795169714 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): get_auth_request con 0x7fa2e809db20 auth_method 0 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e6ffd700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e81419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e6ffd700 10 monclient(hunting): get_auth_request con 0x7fa2e8098070 auth_method 0 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:51.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient(hunting): handle_auth_done global_id 15715 payload 824 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 -- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 msgr2=0x7fa2e811c270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 0x7fa2e811c270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 -- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 msgr2=0x7fa2e81419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e81419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 monclient: found mon.noname-b 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.437+0000 7fa2e67fc700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc009420 con 0x7fa2e809db20 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2e67fc700 1 -- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 msgr2=0x7fa2e81317e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:51.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2e67fc700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa2e806f8a0 tx=0x7fa2dc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc006af0 con 0x7fa2e809db20 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: dump: 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.455 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc011db0 con 0x7fa2e809db20 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 5 monclient: authenticate success, global_id 15715 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _renew_subs 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e811e3c0 con 0x7fa2e809db20 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc01a6a0 con 0x7fa2e809db20 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:51.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 10 monclient: dump: 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:51.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _renew_subs 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2e57fa700 4 set_mon_vals no callback set 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e811fbf0 con 0x7fa2e809db20 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa2dc01aa50 con 0x7fa2e809db20 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e809d8c0 con 0x7fa2e809db20 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 0x7fa2d4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(159..159 src has 58..159) v4 ==== 9944+0+0 (secure 0 0 0) 0x7fa2dc04ac60 con 0x7fa2e809db20 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.441+0000 7fa2e6ffd700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 0x7fa2d4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.445+0000 7fa2e6ffd700 10 monclient: get_auth_request con 0x7fa2d4035a70 auth_method 0 2024-09-18T10:14:51.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.445+0000 7fa2e6ffd700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 0x7fa2d4037f20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.453+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa2dc01b430 con 0x7fa2e809db20 2024-09-18T10:14:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.453+0000 7fa2ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:51.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.453+0000 7fa2ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:51.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.613+0000 7fa2ed414700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:51.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.613+0000 7fa2ed414700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:51.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.613+0000 7fa2ed414700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:51.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:51.613+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fa2e8133330 con 0x7fa2e809db20 2024-09-18T10:14:52.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.041+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v160) v1 ==== 123+0+0 (secure 0 0 0) 0x7fa2e8133330 con 0x7fa2e809db20 2024-09-18T10:14:52.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.041+0000 7fa2ceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:52.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.041+0000 7fa2ceffd700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T10:14:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.141+0000 7fa2ed414700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.141+0000 7fa2ed414700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.141+0000 7fa2ed414700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:52.141+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7fa2e8133330 con 0x7fa2e809db20 2024-09-18T10:14:53.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.089+0000 7fa2ceffd700 1 -- 172.21.15.150:0/2795169714 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v161) v1 ==== 149+0+0 (secure 0 0 0) 0x7fa2e8133330 con 0x7fa2e809db20 2024-09-18T10:14:53.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.089+0000 7fa2ceffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:14:53.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.089+0000 7fa2ceffd700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:14:53.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 msgr2=0x7fa2d4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 0x7fa2d4037f20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8006fd0 tx=0x7fa2d8008040 comp rx=0 tx=0).stop 2024-09-18T10:14:53.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 10 monclient: shutdown 2024-09-18T10:14:53.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:53.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 msgr2=0x7fa2e81317e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa2e806f8a0 tx=0x7fa2dc017000 comp rx=0 tx=0).stop 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 shutdown_connections 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa2d4035a70 0x7fa2d4037f20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2e81328b0 0x7fa2e811c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2e809db20 0x7fa2e81317e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 --2- 172.21.15.150:0/2795169714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2e8098070 0x7fa2e81419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.093+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 >> 172.21.15.150:0/2795169714 conn(0x7fa2e80a1cc0 msgr2=0x7fa2e80998a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.097+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 shutdown_connections 2024-09-18T10:14:53.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.097+0000 7fa2ed414700 1 -- 172.21.15.150:0/2795169714 wait complete. 2024-09-18T10:14:53.102 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:14:53.114 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush set-device-class globster osd.0 2024-09-18T10:14:53.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.217+0000 7f136ac37700 10 monclient: get_monmap_and_config 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.217+0000 7f136ac37700 10 monclient: build_initial_monmap 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.217+0000 7f136ac37700 10 monclient: monmap: 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:53.223992+0000 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:53.223992+0000 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:53.225 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:53.226 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 Processor -- start 2024-09-18T10:14:53.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 -- start start 2024-09-18T10:14:53.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient: init 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f136409cf90 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f136409e480 0x7f13640a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-c con 0x7f136409e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13640990a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136409e060 con 0x7f13640990a0 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136409d750 con 0x7f136409cf90 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13640a66c0 con 0x7f136409e480 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:53.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f136ac37700 10 monclient(hunting): authenticate will time out at 1175.317543s 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f136409e480 0x7f13640a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f136409e480 0x7f13640a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52970/0 (socket says 172.21.15.150:52970) 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 1 -- 172.21.15.150:0/2189091558 learned_addr learned my addr 172.21.15.150:0/2189091558 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): get_auth_request con 0x7f136409e480 auth_method 0 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:53.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.221+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:53.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13689d3700 1 -- 172.21.15.150:0/2189091558 >> v1:172.21.15.150:6789/0 conn(0x7f136409cf90 legacy=0x7f13640a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:53.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13691d4700 1 -- 172.21.15.150:0/2189091558 >> v1:172.21.15.142:6789/0 conn(0x7f13640990a0 legacy=0x7f1364139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:53.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 10 monclient(hunting): handle_auth_done global_id 15718 payload 274 2024-09-18T10:14:53.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 -- 172.21.15.150:0/2189091558 >> v1:172.21.15.150:6789/0 conn(0x7f136409cf90 legacy=0x7f13640a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 -- 172.21.15.150:0/2189091558 >> v1:172.21.15.142:6789/0 conn(0x7f13640990a0 legacy=0x7f1364139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 monclient: found mon.noname-c 2024-09-18T10:14:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 -- 172.21.15.150:0/2189091558 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136409d360 con 0x7f136409e480 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 -- 172.21.15.150:0/2189091558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409e480 msgr2=0x7f13640a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f1363fff700 1 --2- 172.21.15.150:0/2189091558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409e480 0x7f13640a09a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1354009ab0 tx=0x7f1354009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97b7ad06d8a50031 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 1 -- 172.21.15.150:0/2189091558 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135400bc70 con 0x7f136409e480 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: dump: 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:53.233 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f136ac37700 5 monclient: authenticate success, global_id 15718 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f136ac37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 1 -- 172.21.15.150:0/2189091558 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135400bdd0 con 0x7f136409e480 2024-09-18T10:14:53.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f136ac37700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 1 -- 172.21.15.150:0/2189091558 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354011670 con 0x7f136409e480 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f136ac37700 4 set_mon_vals no callback set 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.225+0000 7f13637fe700 10 monclient: dump: 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:53.235 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: shutdown 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- 172.21.15.150:0/2189091558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409e480 msgr2=0x7f13640a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 --2- 172.21.15.150:0/2189091558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409e480 0x7f13640a09a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1354009ab0 tx=0x7f1354009250 comp rx=0 tx=0).stop 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- 172.21.15.150:0/2189091558 shutdown_connections 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 --2- 172.21.15.150:0/2189091558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409e480 0x7f13640a09a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- 172.21.15.150:0/2189091558 >> 172.21.15.150:0/2189091558 conn(0x7f13640a58a0 msgr2=0x7f13640a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- 172.21.15.150:0/2189091558 shutdown_connections 2024-09-18T10:14:53.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- 172.21.15.150:0/2189091558 wait complete. 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: build_initial_monmap 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: monmap: 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:53.239 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 Processor -- start 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- start start 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: init 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 0x7f136408fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-c con 0x7f13640990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:53.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-b con 0x7f136409cf90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f136408d480 0x7f136408d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): picked mon.noname-a con 0x7f136408d480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136408ead0 con 0x7f136408d480 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640a43b0 con 0x7f136409cf90 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136408ef50 con 0x7f13640990a0 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f136ac37700 10 monclient(hunting): authenticate will time out at 1175.327688s 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f13689d3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 msgr2=0x7f136408fd70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:53.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f13689d3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 0x7f136408fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52974/0 (socket says 172.21.15.150:52974) 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f13691d4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f136408d480 0x7f136408d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 1 -- 172.21.15.150:0/2875953232 learned_addr learned my addr 172.21.15.150:0/2875953232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 10 monclient(hunting): get_auth_request con 0x7f136409cf90 auth_method 0 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.229+0000 7f1363fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): get_auth_request con 0x7f136408d480 auth_method 0 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:53.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f13691d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient(hunting): handle_auth_done global_id 15721 payload 824 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 -- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 msgr2=0x7f136408fd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 0x7f136408fd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 -- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f136408d480 msgr2=0x7f136408d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f136408d480 0x7f136408d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 monclient: found mon.noname-b 2024-09-18T10:14:53.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:53.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540094f0 con 0x7f136409cf90 2024-09-18T10:14:53.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 -- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 msgr2=0x7f136408cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:53.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.233+0000 7f1363fff700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f135400b3c0 tx=0x7f1354017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135401f070 con 0x7f136409cf90 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: dump: 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.246 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 5 monclient: authenticate success, global_id 15721 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1354011630 con 0x7f136409cf90 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _renew_subs 2024-09-18T10:14:53.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135401a860 con 0x7f136409cf90 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364090500 con 0x7f136409cf90 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1362ffd700 4 set_mon_vals no callback set 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 10 monclient: dump: 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:53.248 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _renew_subs 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136408f610 con 0x7f136409cf90 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f135401ac50 con 0x7f136409cf90 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13640a0ee0 con 0x7f136409cf90 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 0x7f1350037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:53.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.237+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(161..161 src has 58..161) v4 ==== 9337+0+0 (secure 0 0 0) 0x7f135404aca0 con 0x7f136409cf90 2024-09-18T10:14:53.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.241+0000 7f13689d3700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 0x7f1350037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:53.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.241+0000 7f13689d3700 10 monclient: get_auth_request con 0x7f1350035aa0 auth_method 0 2024-09-18T10:14:53.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.241+0000 7f13689d3700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 0x7f1350037f50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1358007fd0 tx=0x7f1358007a70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:53.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.249+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f135402e3f0 con 0x7f136409cf90 2024-09-18T10:14:53.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.249+0000 7f1360ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:53.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.249+0000 7f1360ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:53.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.417+0000 7f136ac37700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:53.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.417+0000 7f136ac37700 10 monclient: _send_command 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:53.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.417+0000 7f136ac37700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:53.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:53.417+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f1364063ed0 con 0x7f136409cf90 2024-09-18T10:14:54.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.093+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 set osd(s) 0 to class 'globster' v162) v1 ==== 145+0+0 (secure 0 0 0) 0x7f1354029020 con 0x7f136409cf90 2024-09-18T10:14:54.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.093+0000 7f1360ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:54.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.093+0000 7f1360ff9700 10 monclient: _finish_command 2 = system:0 set osd(s) 0 to class 'globster' 2024-09-18T10:14:54.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.197+0000 7f136ac37700 10 start_mon_command cmd=[{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:54.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.197+0000 7f136ac37700 10 monclient: _send_command 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:54.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.197+0000 7f136ac37700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:54.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:54.197+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]} v 0) v1 -- 0x7f1364097b20 con 0x7f136409cf90 2024-09-18T10:14:55.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.109+0000 7f1360ff9700 1 -- 172.21.15.150:0/2875953232 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}]=0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' v163) v1 ==== 257+0+0 (secure 0 0 0) 0x7f1354015070 con 0x7f136409cf90 2024-09-18T10:14:55.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.109+0000 7f1360ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush set-device-class", "class": "globster", "ids": ["osd.0"]}] 2024-09-18T10:14:55.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.109+0000 7f1360ff9700 10 monclient: _finish_command 3 = system:0 osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-18T10:14:55.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 msgr2=0x7f1350037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:55.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 0x7f1350037f50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1358007fd0 tx=0x7f1358007a70 comp rx=0 tx=0).stop 2024-09-18T10:14:55.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 10 monclient: shutdown 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 msgr2=0x7f136408cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f135400b3c0 tx=0x7f1354017000 comp rx=0 tx=0).stop 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 shutdown_connections 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1350035aa0 0x7f1350037f50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f136408d480 0x7f136408d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f136409cf90 0x7f136408cf40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 --2- 172.21.15.150:0/2875953232 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13640990a0 0x7f136408fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 >> 172.21.15.150:0/2875953232 conn(0x7f13640a58a0 msgr2=0x7f13640a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 shutdown_connections 2024-09-18T10:14:55.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.113+0000 7f136ac37700 1 -- 172.21.15.150:0/2875953232 wait complete. 2024-09-18T10:14:55.121 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 already set to class globster. set-device-class item id 0 name 'osd.0' device_class 'globster': no change. set osd(s) to class 'globster' 2024-09-18T10:14:55.134 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:14:55.135 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '10\.' 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.233+0000 7fa117239700 10 monclient: get_monmap_and_config 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.233+0000 7fa117239700 10 monclient: build_initial_monmap 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.233+0000 7fa117239700 10 monclient: monmap: 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:55.239869+0000 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:55.239869+0000 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 Processor -- start 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 -- start start 2024-09-18T10:14:55.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient: init 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 0x7fa11009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): picked mon.noname-b con 0x7fa1100a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa1100a04f0 0x7fa1100a0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): picked mon.noname-a con 0x7fa1100a04f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1101369a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11009dae0 con 0x7fa1101369a0 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1100a21d0 con 0x7fa1100a04f0 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:55.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11009ddc0 con 0x7fa1100a0120 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa117239700 10 monclient(hunting): authenticate will time out at 1177.333579s 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa114fd5700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 msgr2=0x7fa11009e7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa114fd5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 0x7fa11009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa1157d6700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa1101369a0 0x7fa110138d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55936/0 (socket says 172.21.15.150:55936) 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa1157d6700 1 -- 172.21.15.150:0/2717566697 learned_addr learned my addr 172.21.15.150:0/2717566697 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 --2- 172.21.15.150:0/2717566697 >> v2:172.21.15.142:3300/0 conn(0x7fa1100a04f0 0x7fa1100a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): get_auth_request con 0x7fa1100a04f0 auth_method 0 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:55.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient(hunting): handle_auth_done global_id 54513 payload 274 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 -- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 msgr2=0x7fa11009e7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 --2- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 0x7fa11009e7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 -- 172.21.15.150:0/2717566697 >> v1:172.21.15.150:6789/0 conn(0x7fa1101369a0 legacy=0x7fa110138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 monclient: found mon.noname-a 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:55.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.237+0000 7fa10ffff700 1 -- 172.21.15.150:0/2717566697 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100a1590 con 0x7fa1100a04f0 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10ffff700 1 -- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1100a04f0 msgr2=0x7fa1100a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10ffff700 1 --2- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1100a04f0 0x7fa1100a0920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e60 tx=0x7fa100009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a251a6d0b41d42a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 1 -- 172.21.15.150:0/2717566697 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10000e070 con 0x7fa1100a04f0 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:55.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: dump: 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:55.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 1 -- 172.21.15.150:0/2717566697 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa100012430 con 0x7fa1100a04f0 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 5 monclient: authenticate success, global_id 54513 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 1 -- 172.21.15.150:0/2717566697 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1000127d0 con 0x7fa1100a04f0 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa10f7fe700 10 monclient: dump: 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:55.251 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 4 set_mon_vals no callback set 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 10 monclient: shutdown 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 -- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1100a04f0 msgr2=0x7fa1100a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 --2- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1100a04f0 0x7fa1100a0920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e60 tx=0x7fa100009780 comp rx=0 tx=0).stop 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 -- 172.21.15.150:0/2717566697 shutdown_connections 2024-09-18T10:14:55.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 --2- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1100a04f0 0x7fa1100a0920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 --2- 172.21.15.150:0/2717566697 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a0120 0x7fa11009e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 -- 172.21.15.150:0/2717566697 >> 172.21.15.150:0/2717566697 conn(0x7fa1100a5830 msgr2=0x7fa1100a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 -- 172.21.15.150:0/2717566697 shutdown_connections 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.241+0000 7fa117239700 1 -- 172.21.15.150:0/2717566697 wait complete. 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient: build_initial_monmap 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient: monmap: 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:55.253 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:55.254 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:55.254 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 Processor -- start 2024-09-18T10:14:55.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 -- start start 2024-09-18T10:14:55.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient: init 2024-09-18T10:14:55.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): picked mon.noname-b con 0x7fa1100a0120 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 0x7fa11012cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): picked mon.noname-c con 0x7fa1100a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1101369a0 0x7fa11012d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): picked mon.noname-a con 0x7fa1101369a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11009d790 con 0x7fa1101369a0 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11009bff0 con 0x7fa1100a0120 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:55.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11012fcf0 con 0x7fa1100a04f0 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa117239700 10 monclient(hunting): authenticate will time out at 1177.342893s 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa10ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 msgr2=0x7fa11012cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 0x7fa11012cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52984/0 (socket says 172.21.15.150:52984) 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 -- 172.21.15.150:0/2495304263 learned_addr learned my addr 172.21.15.150:0/2495304263 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): get_auth_request con 0x7fa1100a0120 auth_method 0 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:55.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient(hunting): handle_auth_done global_id 15724 payload 824 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 -- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 msgr2=0x7fa11012cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 0x7fa11012cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 -- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1101369a0 msgr2=0x7fa11012d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1101369a0 0x7fa11012d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient: _finish_hunting 0 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 monclient: found mon.noname-b 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.245+0000 7fa114fd5700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100009a20 con 0x7fa1100a0120 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa114fd5700 1 -- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 msgr2=0x7fa11012c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:55.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa114fd5700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b7e0 tx=0x7fa104009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104015070 con 0x7fa1100a0120 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: dump: 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:55.261 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: _finish_auth 0 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 5 monclient: authenticate success, global_id 15724 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104003e20 con 0x7fa1100a0120 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104006360 con 0x7fa1100a0120 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10cff9700 10 monclient: dump: 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:55.262 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 10 monclient: _renew_subs 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa10e7fc700 4 set_mon_vals no callback set 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11012fa10 con 0x7fa1100a0120 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 10 monclient: _renew_subs 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:55.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.249+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa110130bc0 con 0x7fa1100a0120 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa117239700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa104006750 con 0x7fa1100a0120 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa117239700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa117239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1100a1c90 con 0x7fa1100a0120 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10cff9700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 0x7fa0fc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(163..163 src has 58..163) v4 ==== 10026+0+0 (secure 0 0 0) 0x7fa104049440 con 0x7fa1100a0120 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10ffff700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 0x7fa0fc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:55.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10ffff700 10 monclient: get_auth_request con 0x7fa0fc035a70 auth_method 0 2024-09-18T10:14:55.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.253+0000 7fa10ffff700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 0x7fa0fc037f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e60 tx=0x7fa100018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:55.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.257+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa104018af0 con 0x7fa1100a0120 2024-09-18T10:14:55.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.261+0000 7fa10cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:55.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.261+0000 7fa10cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:55.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.429+0000 7fa117239700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:55.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.429+0000 7fa117239700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:55.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.429+0000 7fa117239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:55.433 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.429+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa110063ed0 con 0x7fa1100a0120 2024-09-18T10:14:55.434 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:55.429+0000 7fa114fd5700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b7e0 tx=0x7fa104009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fa104018310 mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 2024-09-18T10:14:56.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.257+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fa104018310 con 0x7fa1100a0120 2024-09-18T10:14:56.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.257+0000 7fa10cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.257+0000 7fa10cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa117239700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa117239700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa117239700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fa1100008d0 con 0x7fa1100a0120 2024-09-18T10:14:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa10cff9700 1 -- 172.21.15.150:0/2495304263 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fa1040184f0 con 0x7fa1100a0120 2024-09-18T10:14:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa10cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.369+0000 7fa10cff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 msgr2=0x7fa0fc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 0x7fa0fc037f20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa100009e60 tx=0x7fa100018000 comp rx=0 tx=0).stop 2024-09-18T10:14:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 10 monclient: shutdown 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 msgr2=0x7fa11012c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa10400b7e0 tx=0x7fa104009220 comp rx=0 tx=0).stop 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 shutdown_connections 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0fc035a70 0x7fa0fc037f20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa1101369a0 0x7fa11012d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1100a04f0 0x7fa11012cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 --2- 172.21.15.150:0/2495304263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1100a0120 0x7fa11012c7f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 >> 172.21.15.150:0/2495304263 conn(0x7fa1100a5830 msgr2=0x7fa11009ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 shutdown_connections 2024-09-18T10:14:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.373+0000 7fa117239700 1 -- 172.21.15.150:0/2495304263 wait complete. 2024-09-18T10:14:56.388 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:56.388 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:14:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '11\.' 2024-09-18T10:14:56.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: get_monmap_and_config 2024-09-18T10:14:56.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: build_initial_monmap 2024-09-18T10:14:56.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: monmap: 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:56.488720+0000 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:56.488720+0000 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:56.490 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 Processor -- start 2024-09-18T10:14:56.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 -- start start 2024-09-18T10:14:56.491 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: init 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc140071240 0x7fc140096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): picked mon.noname-c con 0x7fc140071240 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc14009e1e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:56.494 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc1400994f0 0x7fc1401364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): picked mon.noname-a con 0x7fc1400994f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc140098cf0 con 0x7fc14009e1e0 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14009e640 con 0x7fc1400994f0 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14009df80 con 0x7fc140071240 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc148756700 10 monclient(hunting): authenticate will time out at 1178.583611s 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc1464f2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc140071240 0x7fc140096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.495 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc145cf1700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc14009e1e0 0x7fc140096fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55948/0 (socket says 172.21.15.150:55948) 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc145cf1700 1 -- 172.21.15.150:0/2044721084 learned_addr learned my addr 172.21.15.150:0/2044721084 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc1464f2700 10 monclient(hunting): get_auth_request con 0x7fc140071240 auth_method 0 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc1464f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc1464f2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc1464f2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.485+0000 7fc146cf3700 1 --2- 172.21.15.150:0/2044721084 >> v2:172.21.15.142:3300/0 conn(0x7fc1400994f0 0x7fc1401364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.496 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): get_auth_request con 0x7fc1400994f0 auth_method 0 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient(hunting): handle_auth_done global_id 54516 payload 274 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 -- 172.21.15.150:0/2044721084 >> v2:172.21.15.150:3300/0 conn(0x7fc140071240 msgr2=0x7fc140096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 --2- 172.21.15.150:0/2044721084 >> v2:172.21.15.150:3300/0 conn(0x7fc140071240 0x7fc140096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 -- 172.21.15.150:0/2044721084 >> v1:172.21.15.150:6789/0 conn(0x7fc14009e1e0 legacy=0x7fc140096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:56.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient: _finish_hunting 0 2024-09-18T10:14:56.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 monclient: found mon.noname-a 2024-09-18T10:14:56.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:56.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:56.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 -- 172.21.15.150:0/2044721084 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140098a10 con 0x7fc1400994f0 2024-09-18T10:14:56.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 -- 172.21.15.150:0/2044721084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 msgr2=0x7fc1401364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:56.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc146cf3700 1 --2- 172.21.15.150:0/2044721084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc1401364d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc14004cf90 tx=0x7fc138009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fae9d9aa33973f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 1 -- 172.21.15.150:0/2044721084 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13800e070 con 0x7fc1400994f0 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: dump: 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.502 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: _finish_auth 0 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc148756700 5 monclient: authenticate success, global_id 54516 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc148756700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 1 -- 172.21.15.150:0/2044721084 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13800be40 con 0x7fc1400994f0 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc148756700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc148756700 4 set_mon_vals no callback set 2024-09-18T10:14:56.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 1 -- 172.21.15.150:0/2044721084 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138012680 con 0x7fc1400994f0 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.489+0000 7fc1454f0700 10 monclient: dump: 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.504 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: shutdown 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- 172.21.15.150:0/2044721084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 msgr2=0x7fc1401364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- 172.21.15.150:0/2044721084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc1401364d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc14004cf90 tx=0x7fc138009780 comp rx=0 tx=0).stop 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- 172.21.15.150:0/2044721084 shutdown_connections 2024-09-18T10:14:56.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- 172.21.15.150:0/2044721084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc1401364d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- 172.21.15.150:0/2044721084 >> v2:172.21.15.150:3300/0 conn(0x7fc140071240 0x7fc140096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- 172.21.15.150:0/2044721084 >> 172.21.15.150:0/2044721084 conn(0x7fc1400a1cc0 msgr2=0x7fc1400a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- 172.21.15.150:0/2044721084 shutdown_connections 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- 172.21.15.150:0/2044721084 wait complete. 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: build_initial_monmap 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: monmap: 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:56.507 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 Processor -- start 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- start start 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: init 2024-09-18T10:14:56.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 0x7fc1401363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): picked mon.noname-c con 0x7fc140071240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc14011bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): picked mon.noname-a con 0x7fc1400994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): picked mon.noname-b con 0x7fc14009e1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14011f930 con 0x7fc1400994f0 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1400980f0 con 0x7fc14009e1e0 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14009b340 con 0x7fc140071240 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc148756700 10 monclient(hunting): authenticate will time out at 1178.591128s 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc1464f2700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 msgr2=0x7fc1401363e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc1464f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 0x7fc1401363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53004/0 (socket says 172.21.15.150:53004) 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 -- 172.21.15.150:0/731554037 learned_addr learned my addr 172.21.15.150:0/731554037 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): get_auth_request con 0x7fc14009e1e0 auth_method 0 2024-09-18T10:14:56.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc14011bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 10 monclient(hunting): get_auth_request con 0x7fc1400994f0 auth_method 0 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient(hunting): handle_auth_done global_id 15730 payload 824 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 -- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 msgr2=0x7fc1401363e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 0x7fc1401363e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 -- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 msgr2=0x7fc14011bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc14011bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient: _finish_hunting 0 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 monclient: found mon.noname-b 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc146cf3700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc138009a20 con 0x7fc14009e1e0 2024-09-18T10:14:56.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.493+0000 7fc145cf1700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc14011bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:56.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc146cf3700 1 -- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 msgr2=0x7fc14011c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc146cf3700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc1400980c0 tx=0x7fc138018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc138012e10 con 0x7fc14009e1e0 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: dump: 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.515 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 5 monclient: authenticate success, global_id 15730 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc13801e850 con 0x7fc14009e1e0 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 10 monclient: _renew_subs 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14011ffa0 con 0x7fc14009e1e0 2024-09-18T10:14:56.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13801ec10 con 0x7fc14009e1e0 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc1367fc700 10 monclient: dump: 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.517 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc137fff700 4 set_mon_vals no callback set 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 10 monclient: _renew_subs 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.497+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14011f2e0 con 0x7fc14009e1e0 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc148756700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc148756700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc148756700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc138024070 con 0x7fc14009e1e0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1367fc700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 0x7fc124037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(163..163 src has 58..163) v4 ==== 10026+0+0 (secure 0 0 0) 0x7fc13804c1c0 con 0x7fc14009e1e0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1464f2700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 0x7fc124037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1464f2700 10 monclient: get_auth_request con 0x7fc124035a70 auth_method 0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.501+0000 7fc1464f2700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 0x7fc124037f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c009530 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.513+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.513+0000 7fc1367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.513+0000 7fc1367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:56.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc148756700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc148756700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc148756700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc1367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.692 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.685+0000 7fc1367fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:56.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc148756700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc148756700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc148756700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc1367fc700 1 -- 172.21.15.150:0/731554037 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v163) v1 ==== 83+0+2378 (secure 0 0 0) 0x7fc14011f500 con 0x7fc14009e1e0 2024-09-18T10:14:56.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc1367fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:56.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.789+0000 7fc1367fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:56.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 msgr2=0x7fc124037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 0x7fc124037f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc13c006fd0 tx=0x7fc13c009530 comp rx=0 tx=0).stop 2024-09-18T10:14:56.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 10 monclient: shutdown 2024-09-18T10:14:56.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 msgr2=0x7fc14011c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc1400980c0 tx=0x7fc138018000 comp rx=0 tx=0).stop 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 shutdown_connections 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc124035a70 0x7fc124037f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc14009e1e0 0x7fc14011c260 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1400994f0 0x7fc14011bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 --2- 172.21.15.150:0/731554037 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc140071240 0x7fc1401363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 >> 172.21.15.150:0/731554037 conn(0x7fc1400a1cc0 msgr2=0x7fc14009fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 shutdown_connections 2024-09-18T10:14:56.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.793+0000 7fc148756700 1 -- 172.21.15.150:0/731554037 wait complete. 2024-09-18T10:14:56.807 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:56.807 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:56.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:14:56.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:14:56.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:14:56.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '12\.' 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.897+0000 7f96587de700 10 monclient: get_monmap_and_config 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.897+0000 7f96587de700 10 monclient: build_initial_monmap 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.897+0000 7f96587de700 10 monclient: monmap: 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.904 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:56.903654+0000 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:56.903654+0000 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:56.905 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 Processor -- start 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 -- start start 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient: init 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f965009d970 addr v1:172.21.15.150:6789/0 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9650131750 0x7f9650133b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): picked mon.noname-c con 0x7f9650131750 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f9650136450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): picked mon.noname-b con 0x7f9650134070 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965009b0a0 con 0x7f965009d970 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f965009ab50 con 0x7f9650131750 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965009ae30 con 0x7f9650134070 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f96587de700 10 monclient(hunting): authenticate will time out at 1178.997176s 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f965657a700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f965009d970 0x7f96500955b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55958/0 (socket says 172.21.15.150:55958) 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f965657a700 1 -- 172.21.15.150:0/4112875315 learned_addr learned my addr 172.21.15.150:0/4112875315 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9656d7b700 1 -- 172.21.15.150:0/4112875315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 msgr2=0x7f9650136450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9656d7b700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f9650136450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:56.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 1 --2- 172.21.15.150:0/4112875315 >> v2:172.21.15.150:3300/0 conn(0x7f9650131750 0x7f9650133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): get_auth_request con 0x7f9650131750 auth_method 0 2024-09-18T10:14:56.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f965657a700 1 -- 172.21.15.150:0/4112875315 >> v1:172.21.15.150:6789/0 conn(0x7f965009d970 legacy=0x7f96500955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.901+0000 7f9655d79700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 10 monclient(hunting): handle_auth_done global_id 15733 payload 274 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 -- 172.21.15.150:0/4112875315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 msgr2=0x7f9650136450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f9650136450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 -- 172.21.15.150:0/4112875315 >> v1:172.21.15.150:6789/0 conn(0x7f965009d970 legacy=0x7f96500955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 10 monclient: _finish_hunting 0 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 monclient: found mon.noname-c 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:56.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 -- 172.21.15.150:0/4112875315 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650095cc0 con 0x7f9650131750 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 -- 172.21.15.150:0/4112875315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9650131750 msgr2=0x7f9650133b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655d79700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9650131750 0x7f9650133b30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9640009990 tx=0x7f96400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8083d7529d8528d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 1 -- 172.21.15.150:0/4112875315 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964000ba30 con 0x7f9650131750 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: dump: 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.913 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: _finish_auth 0 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 1 -- 172.21.15.150:0/4112875315 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964000bb90 con 0x7f9650131750 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f96587de700 5 monclient: authenticate success, global_id 15733 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f96587de700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f96587de700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:56.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f96587de700 4 set_mon_vals no callback set 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 1 -- 172.21.15.150:0/4112875315 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640011420 con 0x7f9650131750 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.905+0000 7f9655578700 10 monclient: dump: 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.915 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: shutdown 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- 172.21.15.150:0/4112875315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9650131750 msgr2=0x7f9650133b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9650131750 0x7f9650133b30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9640009990 tx=0x7f96400091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- 172.21.15.150:0/4112875315 shutdown_connections 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f9650136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- 172.21.15.150:0/4112875315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9650131750 0x7f9650133b30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- 172.21.15.150:0/4112875315 >> 172.21.15.150:0/4112875315 conn(0x7f96500a15e0 msgr2=0x7f96500a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- 172.21.15.150:0/4112875315 shutdown_connections 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- 172.21.15.150:0/4112875315 wait complete. 2024-09-18T10:14:56.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: build_initial_monmap 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: monmap: 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 Processor -- start 2024-09-18T10:14:56.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- start start 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: init 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): picked mon.noname-b con 0x7f965009d970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 0x7f965006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): picked mon.noname-a con 0x7f9650131750 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f965006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): picked mon.noname-c con 0x7f9650134070 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96500984c0 con 0x7f9650131750 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965009d120 con 0x7f965009d970 2024-09-18T10:14:56.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f965006f360 con 0x7f9650134070 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f96587de700 10 monclient(hunting): authenticate will time out at 1179.006817s 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53022/0 (socket says 172.21.15.150:53022) 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 -- 172.21.15.150:0/2634780404 learned_addr learned my addr 172.21.15.150:0/2634780404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): get_auth_request con 0x7f965009d970 auth_method 0 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9656d7b700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 msgr2=0x7f965006d9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:56.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9656d7b700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f965006d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 0x7f965006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 10 monclient(hunting): get_auth_request con 0x7f9650131750 auth_method 0 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient(hunting): handle_auth_done global_id 15739 payload 824 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 msgr2=0x7f965006d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f965006d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 msgr2=0x7f965006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:56.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 0x7f965006d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient: _finish_hunting 0 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 monclient: found mon.noname-b 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f965657a700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640009420 con 0x7f965009d970 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.909+0000 7f9655d79700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 0x7f965006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f965657a700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 msgr2=0x7f965009c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f965657a700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c5c0 tx=0x7f964c007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c005d40 con 0x7f965009d970 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: dump: 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c0070d0 con 0x7f965009d970 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 5 monclient: authenticate success, global_id 15739 2024-09-18T10:14:56.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 10 monclient: _renew_subs 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9650129c10 con 0x7f965009d970 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c005ea0 con 0x7f965009d970 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96467fc700 10 monclient: dump: 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:56.928 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f9647fff700 4 set_mon_vals no callback set 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 10 monclient: _renew_subs 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.913+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965006e780 con 0x7f965009d970 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96587de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96587de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96587de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9650128b00 con 0x7f965009d970 2024-09-18T10:14:56.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f964c01b3b0 con 0x7f965009d970 2024-09-18T10:14:56.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96467fc700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 0x7f9634037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:56.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(163..163 src has 58..163) v4 ==== 10026+0+0 (secure 0 0 0) 0x7f964c048c90 con 0x7f965009d970 2024-09-18T10:14:56.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f9655d79700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 0x7f9634037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:56.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.917+0000 7f9655d79700 10 monclient: get_auth_request con 0x7f9634035a70 auth_method 0 2024-09-18T10:14:56.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.921+0000 7f9655d79700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 0x7f9634037f20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9640003080 tx=0x7f9640017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:56.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.929+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9650128b00 con 0x7f965009d970 2024-09-18T10:14:56.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.929+0000 7f96467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:56.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:56.929+0000 7f96467fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:57.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96587de700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96587de700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96587de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f9650128b00 con 0x7f965009d970 2024-09-18T10:14:57.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f9650128b00 con 0x7f965009d970 2024-09-18T10:14:57.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.105+0000 7f96467fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:57.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96587de700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96587de700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96587de700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f96501477f0 con 0x7f965009d970 2024-09-18T10:14:57.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96467fc700 1 -- 172.21.15.150:0/2634780404 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v164) v1 ==== 83+0+2378 (secure 0 0 0) 0x7f964c017070 con 0x7f965009d970 2024-09-18T10:14:57.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96467fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:14:57.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.205+0000 7f96467fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:57.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 msgr2=0x7f9634037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:57.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 0x7f9634037f20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9640003080 tx=0x7f9640017000 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 10 monclient: shutdown 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 msgr2=0x7f965009c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f964c00c5c0 tx=0x7f964c007e80 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 shutdown_connections 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9634035a70 0x7f9634037f20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9650134070 0x7f965006d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650131750 0x7f965006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 --2- 172.21.15.150:0/2634780404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f965009d970 0x7f965009c860 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 >> 172.21.15.150:0/2634780404 conn(0x7f96500a15e0 msgr2=0x7f9650097c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 shutdown_connections 2024-09-18T10:14:57.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.209+0000 7f96587de700 1 -- 172.21.15.150:0/2634780404 wait complete. 2024-09-18T10:14:57.223 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:57.223 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 10.50000 11.50000 12.50000 osd.0 2024-09-18T10:14:57.223 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight-compat osd.0 7.5 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: get_monmap_and_config 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: build_initial_monmap 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: monmap: 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:57.326208+0000 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:57.326208+0000 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:57.327 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:57.328 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 1 Processor -- start 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 1 -- start start 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: init 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.321+0000 7f238db43700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f238809c750 0x7f23880972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): picked mon.noname-a con 0x7f238809c750 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 0x7f238809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): picked mon.noname-b con 0x7f238809cb20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2388132560 addr v1:172.21.15.142:6789/0 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238809e680 con 0x7f2388132560 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:57.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23880a2610 con 0x7f238809c750 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23880a28b0 con 0x7f238809cb20 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f238db43700 10 monclient(hunting): authenticate will time out at 1179.420211s 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f2386ffd700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 msgr2=0x7f238809cef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f2386ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 0x7f238809cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f2387fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2388132560 0x7f2388132980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:36412/0 (socket says 172.21.15.150:36412) 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f238809c750 0x7f23880972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f2387fff700 1 -- 172.21.15.150:0/368640637 learned_addr learned my addr 172.21.15.150:0/368640637 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): get_auth_request con 0x7f238809c750 auth_method 0 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:57.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient(hunting): handle_auth_done global_id 54525 payload 274 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 -- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 msgr2=0x7f238809cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 --2- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 0x7f238809cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 -- 172.21.15.150:0/368640637 >> v1:172.21.15.142:6789/0 conn(0x7f2388132560 legacy=0x7f2388132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient: _finish_hunting 0 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 monclient: found mon.noname-a 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:57.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:14:57.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 -- 172.21.15.150:0/368640637 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238809e3a0 con 0x7f238809c750 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 -- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809c750 msgr2=0x7f23880972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.325+0000 7f23877fe700 1 --2- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809c750 0x7f23880972f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098e0 tx=0x7f237c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698b51194cc6f0bb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 1 -- 172.21.15.150:0/368640637 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c00ba70 con 0x7f238809c750 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: dump: 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:57.340 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 5 monclient: authenticate success, global_id 54525 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 1 -- 172.21.15.150:0/368640637 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c00bbf0 con 0x7f238809c750 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 1 -- 172.21.15.150:0/368640637 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c011420 con 0x7f238809c750 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 4 set_mon_vals no callback set 2024-09-18T10:14:57.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f23867fc700 10 monclient: dump: 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: shutdown 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809c750 msgr2=0x7f23880972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809c750 0x7f23880972f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098e0 tx=0x7f237c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- 172.21.15.150:0/368640637 shutdown_connections 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809cb20 0x7f238809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- 172.21.15.150:0/368640637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809c750 0x7f23880972f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- 172.21.15.150:0/368640637 >> 172.21.15.150:0/368640637 conn(0x7f23880681c0 msgr2=0x7f23880a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- 172.21.15.150:0/368640637 shutdown_connections 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- 172.21.15.150:0/368640637 wait complete. 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: build_initial_monmap 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: monmap: 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:57.343 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 Processor -- start 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- start start 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: init 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:57.344 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 0x7f2388141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient(hunting): picked mon.noname-c con 0x7f238809c750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 0x7f2388141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient(hunting): picked mon.noname-a con 0x7f238809cb20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient(hunting): picked mon.noname-b con 0x7f2388132560 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238809ded0 con 0x7f238809cb20 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.329+0000 7f238db43700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23880a1d50 con 0x7f2388132560 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2388123840 con 0x7f238809c750 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:57.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 10 monclient(hunting): authenticate will time out at 1179.428028s 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f23877fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 msgr2=0x7f2388141a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f23877fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 0x7f2388141a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 0x7f2388141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53024/0 (socket says 172.21.15.150:53024) 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 -- 172.21.15.150:0/1157412019 learned_addr learned my addr 172.21.15.150:0/1157412019 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): get_auth_request con 0x7f2388132560 auth_method 0 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): get_auth_request con 0x7f238809cb20 auth_method 0 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:57.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient(hunting): handle_auth_done global_id 15742 payload 824 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 -- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 msgr2=0x7f2388141a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 0x7f2388141a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 -- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 msgr2=0x7f2388141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 0x7f2388141f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient: _finish_hunting 0 2024-09-18T10:14:57.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 monclient: found mon.noname-b 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c009420 con 0x7f2388132560 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2386ffd700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 0x7f2388141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 -- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 msgr2=0x7f23881424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:57.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f2387fff700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f238000ccf0 tx=0x7f2380005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238000ab30 con 0x7f2388132560 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 10 monclient: dump: 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:57.350 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 5 monclient: authenticate success, global_id 15742 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238000acb0 con 0x7f2388132560 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 10 monclient: _renew_subs 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.333+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23881216c0 con 0x7f2388132560 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2380007340 con 0x7f2388132560 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 10 monclient: dump: 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:57.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 10 monclient: _renew_subs 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2388122890 con 0x7f2388132560 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f23857fa700 4 set_mon_vals no callback set 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2388122ab0 con 0x7f2388132560 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f23800076b0 con 0x7f2388132560 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 0x7f2374037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:57.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(164..164 src has 58..164) v4 ==== 10026+0+0 (secure 0 0 0) 0x7f2380049ce0 con 0x7f2388132560 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f23877fe700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 0x7f2374037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.337+0000 7f23877fe700 10 monclient: get_auth_request con 0x7f2374035a70 auth_method 0 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.341+0000 7f23877fe700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 0x7f2374037f20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098e0 tx=0x7f237c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.349+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f238001a590 con 0x7f2388132560 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.349+0000 7f236f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:57.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.349+0000 7f236f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:14:57.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.521+0000 7f238db43700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:57.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.521+0000 7f238db43700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:57.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.521+0000 7f238db43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:57.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:57.521+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f2388120b50 con 0x7f2388132560 2024-09-18T10:14:58.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.121+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v165) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2380010af0 con 0x7f2388132560 2024-09-18T10:14:58.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.121+0000 7f236f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:58.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.121+0000 7f236f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:14:58.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.233+0000 7f238db43700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:58.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.233+0000 7f238db43700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:58.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.233+0000 7f238db43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:58.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:58.233+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]} v 0) v1 -- 0x7f238809d430 con 0x7f2388132560 2024-09-18T10:14:59.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.141+0000 7f236f7fe700 1 -- 172.21.15.150:0/1157412019 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}]=0 v166) v1 ==== 118+0+0 (secure 0 0 0) 0x7f238002a0b0 con 0x7f2388132560 2024-09-18T10:14:59.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.141+0000 7f236f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight-compat", "item": "osd.0", "weight": [7.5]}] 2024-09-18T10:14:59.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.141+0000 7f236f7fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 msgr2=0x7f2374037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 0x7f2374037f20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f237c0098e0 tx=0x7f237c016000 comp rx=0 tx=0).stop 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 10 monclient: shutdown 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 msgr2=0x7f23881424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f238000ccf0 tx=0x7f2380005400 comp rx=0 tx=0).stop 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 shutdown_connections 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2374035a70 0x7f2374037f20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2388132560 0x7f23881424d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f238809cb20 0x7f2388141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 --2- 172.21.15.150:0/1157412019 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f238809c750 0x7f2388141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 >> 172.21.15.150:0/1157412019 conn(0x7f23880681c0 msgr2=0x7f238809a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:59.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 shutdown_connections 2024-09-18T10:14:59.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.145+0000 7f238db43700 1 -- 172.21.15.150:0/1157412019 wait complete. 2024-09-18T10:14:59.164 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight cool osd.0 8.5 2024-09-18T10:14:59.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 10 monclient: get_monmap_and_config 2024-09-18T10:14:59.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 10 monclient: build_initial_monmap 2024-09-18T10:14:59.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 10 monclient: monmap: 2024-09-18T10:14:59.271 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:59.271 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:14:59.270678+0000 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:14:59.270678+0000 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:14:59.272 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 1 Processor -- start 2024-09-18T10:14:59.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 1 -- start start 2024-09-18T10:14:59.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.265+0000 7f2c60299700 10 monclient: init 2024-09-18T10:14:59.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c580a0290 0x7f2c5809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-c con 0x7f2c580a0290 addr v2:172.21.15.150:3300/0 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2c580a0660 0x7f2c580a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-a con 0x7f2c580a0660 addr v2:172.21.15.142:3300/0 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 0x7f2c58138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-b con 0x7f2c58136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5809ef80 con 0x7f2c580a0660 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c580a1fb0 con 0x7f2c580a0290 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5809e360 con 0x7f2c58136a10 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient(hunting): authenticate will time out at 1181.364377s 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c580a0290 0x7f2c5809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e836700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 msgr2=0x7f2c58138df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c580a0290 0x7f2c5809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39518/0 (socket says 172.21.15.150:39518) 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e836700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 0x7f2c58138df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 -- 172.21.15.150:0/2423327169 learned_addr learned my addr 172.21.15.150:0/2423327169 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): get_auth_request con 0x7f2c580a0290 auth_method 0 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:59.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d834700 1 --2- 172.21.15.150:0/2423327169 >> v2:172.21.15.142:3300/0 conn(0x7f2c580a0660 0x7f2c580a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d834700 10 monclient(hunting): get_auth_request con 0x7f2c580a0660 auth_method 0 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d834700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d834700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d834700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient(hunting): handle_auth_done global_id 15745 payload 274 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 -- 172.21.15.150:0/2423327169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 msgr2=0x7f2c58138df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 --2- 172.21.15.150:0/2423327169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 0x7f2c58138df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 -- 172.21.15.150:0/2423327169 >> v2:172.21.15.142:3300/0 conn(0x7f2c580a0660 msgr2=0x7f2c580a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 --2- 172.21.15.150:0/2423327169 >> v2:172.21.15.142:3300/0 conn(0x7f2c580a0660 0x7f2c580a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient: _finish_hunting 0 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 monclient: found mon.noname-c 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 -- 172.21.15.150:0/2423327169 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5809da50 con 0x7f2c580a0290 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 -- 172.21.15.150:0/2423327169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0290 msgr2=0x7f2c5809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5e035700 1 --2- 172.21.15.150:0/2423327169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0290 0x7f2c5809ea40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2c580691e0 tx=0x7f2c48009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14f3ecd1e0b241d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 1 -- 172.21.15.150:0/2423327169 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4800ba70 con 0x7f2c580a0290 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: dump: 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.280 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: _finish_auth 0 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 5 monclient: authenticate success, global_id 15745 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 1 -- 172.21.15.150:0/2423327169 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4800bbd0 con 0x7f2c580a0290 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 10 monclient: get_monmap_and_config success 2024-09-18T10:14:59.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c60299700 4 set_mon_vals no callback set 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 1 -- 172.21.15.150:0/2423327169 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48011420 con 0x7f2c580a0290 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.269+0000 7f2c5d033700 10 monclient: dump: 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:59.282 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: shutdown 2024-09-18T10:14:59.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:14:59.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- 172.21.15.150:0/2423327169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0290 msgr2=0x7f2c5809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- 172.21.15.150:0/2423327169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0290 0x7f2c5809ea40 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2c580691e0 tx=0x7f2c48009250 comp rx=0 tx=0).stop 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- 172.21.15.150:0/2423327169 shutdown_connections 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- 172.21.15.150:0/2423327169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c58136a10 0x7f2c58138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- 172.21.15.150:0/2423327169 >> v2:172.21.15.142:3300/0 conn(0x7f2c580a0660 0x7f2c580a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- 172.21.15.150:0/2423327169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0290 0x7f2c5809ea40 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- 172.21.15.150:0/2423327169 >> 172.21.15.150:0/2423327169 conn(0x7f2c580a58a0 msgr2=0x7f2c580a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- 172.21.15.150:0/2423327169 shutdown_connections 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- 172.21.15.150:0/2423327169 wait complete. 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: build_initial_monmap 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: monmap: 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:14:59.288 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 Processor -- start 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- start start 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: init 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: _reopen_session rank -1 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:14:59.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 0x7f2c581420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-c con 0x7f2c580a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-b con 0x7f2c580a0660 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 0x7f2c5812c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): picked mon.noname-a con 0x7f2c58136a10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5809d3f0 con 0x7f2c58136a10 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5809c0a0 con 0x7f2c580a0660 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): start opening mon connection 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5812ff40 con 0x7f2c580a0290 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): _renew_subs 2024-09-18T10:14:59.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c60299700 10 monclient(hunting): authenticate will time out at 1181.370976s 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e035700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 msgr2=0x7f2c581420c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e035700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 0x7f2c581420c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39534/0 (socket says 172.21.15.150:39534) 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 -- 172.21.15.150:0/3383749649 learned_addr learned my addr 172.21.15.150:0/3383749649 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): get_auth_request con 0x7f2c580a0660 auth_method 0 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 0x7f2c5812c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:14:59.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 10 monclient(hunting): get_auth_request con 0x7f2c58136a10 auth_method 0 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient(hunting): handle_auth_done global_id 15748 payload 824 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 -- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 msgr2=0x7f2c581420c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 0x7f2c581420c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 -- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 msgr2=0x7f2c5812c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 0x7f2c5812c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient: _finish_hunting 0 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 monclient: found mon.noname-b 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:14:59.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5d834700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c480094f0 con 0x7f2c580a0660 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.273+0000 7f2c5e836700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 0x7f2c5812c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c5d834700 1 -- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 msgr2=0x7f2c58142600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c5d834700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d890 tx=0x7f2c4c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c5d834700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d890 tx=0x7f2c4c008040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f2c4c00b810 mon_map magic: 0 v1 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00b810 con 0x7f2c580a0660 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 10 monclient: dump: 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.293 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 10 monclient: _finish_auth 0 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 5 monclient: authenticate success, global_id 15748 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c00b970 con 0x7f2c580a0660 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 10 monclient: _renew_subs 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5812e130 con 0x7f2c580a0660 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:14:59.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 10 monclient: _renew_subs 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c5c832700 4 set_mon_vals no callback set 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c58142b40 con 0x7f2c580a0660 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.277+0000 7f2c60299700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c60299700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c60299700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c5809d6c0 con 0x7f2c580a0660 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00bcf0 con 0x7f2c580a0660 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 10 monclient: dump: 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:14:59.295 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2c4c014640 con 0x7f2c580a0660 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c567fc700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 0x7f2c44037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c5e035700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 0x7f2c44037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:14:59.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.281+0000 7f2c5e035700 10 monclient: get_auth_request con 0x7f2c44035ac0 auth_method 0 2024-09-18T10:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.285+0000 7f2c5e035700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 0x7f2c44037f70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2c580691e0 tx=0x7f2c48017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:14:59.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.289+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(166..166 src has 58..166) v4 ==== 9964+0+0 (secure 0 0 0) 0x7f2c4c049fb0 con 0x7f2c580a0660 2024-09-18T10:14:59.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.901+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2c4c011db0 con 0x7f2c580a0660 2024-09-18T10:14:59.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.901+0000 7f2c567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:14:59.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:14:59.901+0000 7f2c567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:00.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.077+0000 7f2c60299700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:00.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.077+0000 7f2c60299700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:00.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.077+0000 7f2c60299700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:00.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.077+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f2c58130100 con 0x7f2c580a0660 2024-09-18T10:15:00.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.141+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v167) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2c4c0117b0 con 0x7f2c580a0660 2024-09-18T10:15:00.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.141+0000 7f2c567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:00.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.141+0000 7f2c567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:00.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.253+0000 7f2c60299700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:00.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.253+0000 7f2c60299700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:00.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.253+0000 7f2c60299700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:00.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:00.253+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]} v 0) v1 -- 0x7f2c58063ed0 con 0x7f2c580a0660 2024-09-18T10:15:01.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.161+0000 7f2c567fc700 1 -- 172.21.15.150:0/3383749649 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}]=0 v168) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2c4c02c070 con 0x7f2c580a0660 2024-09-18T10:15:01.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.161+0000 7f2c567fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cool", "item": "osd.0", "weight": [8.5]}] 2024-09-18T10:15:01.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.161+0000 7f2c567fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 msgr2=0x7f2c44037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 0x7f2c44037f70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2c580691e0 tx=0x7f2c48017000 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 10 monclient: shutdown 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 msgr2=0x7f2c58142600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d890 tx=0x7f2c4c008040 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 shutdown_connections 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c44035ac0 0x7f2c44037f70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c58136a10 0x7f2c5812c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c580a0660 0x7f2c58142600 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 --2- 172.21.15.150:0/3383749649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c580a0290 0x7f2c581420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 >> 172.21.15.150:0/3383749649 conn(0x7f2c580a58a0 msgr2=0x7f2c5809ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:01.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 shutdown_connections 2024-09-18T10:15:01.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.165+0000 7f2c60299700 1 -- 172.21.15.150:0/3383749649 wait complete. 2024-09-18T10:15:01.184 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set reweight cold osd.0 6.5 6.6 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: get_monmap_and_config 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: build_initial_monmap 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: monmap: 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:01.292726+0000 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:01.292726+0000 2024-09-18T10:15:01.293 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 Processor -- start 2024-09-18T10:15:01.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 -- start start 2024-09-18T10:15:01.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: init 2024-09-18T10:15:01.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd59409fb60 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 0x7fd594135340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-b con 0x7fd594068220 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd594068620 0x7fd594137890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-a con 0x7fd594068620 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59409ff30 con 0x7fd59409fb60 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59406ea50 con 0x7fd594068620 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd594097020 con 0x7fd594068220 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:01.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59c096700 10 monclient(hunting): authenticate will time out at 1183.386987s 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd599631700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 msgr2=0x7fd594135340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd599631700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 0x7fd594135340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd599e32700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd59409fb60 0x7fd594067b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40212/0 (socket says 172.21.15.150:40212) 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd599e32700 1 -- 172.21.15.150:0/2371775730 learned_addr learned my addr 172.21.15.150:0/2371775730 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 --2- 172.21.15.150:0/2371775730 >> v2:172.21.15.142:3300/0 conn(0x7fd594068620 0x7fd594137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): get_auth_request con 0x7fd594068620 auth_method 0 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient(hunting): handle_auth_done global_id 54537 payload 274 2024-09-18T10:15:01.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 -- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 msgr2=0x7fd594135340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 --2- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 0x7fd594135340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 -- 172.21.15.150:0/2371775730 >> v1:172.21.15.150:6789/0 conn(0x7fd59409fb60 legacy=0x7fd594067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient: _finish_hunting 0 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 monclient: found mon.noname-a 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:01.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.289+0000 7fd59a633700 1 -- 172.21.15.150:0/2371775730 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594097300 con 0x7fd594068620 2024-09-18T10:15:01.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59a633700 1 -- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 msgr2=0x7fd594137890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:01.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59a633700 1 --2- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594137890 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009e60 tx=0x7fd58c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4322876413b363 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:01.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 1 -- 172.21.15.150:0/2371775730 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00e070 con 0x7fd594068620 2024-09-18T10:15:01.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: dump: 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: _finish_auth 0 2024-09-18T10:15:01.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 5 monclient: authenticate success, global_id 54537 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 1 -- 172.21.15.150:0/2371775730 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c012430 con 0x7fd594068620 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 4 set_mon_vals no callback set 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 1 -- 172.21.15.150:0/2371775730 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c0127f0 con 0x7fd594068620 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd598e30700 10 monclient: dump: 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.303 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 10 monclient: shutdown 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:01.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 1 -- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 msgr2=0x7fd594137890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.293+0000 7fd59c096700 1 --2- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594137890 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009e60 tx=0x7fd58c009780 comp rx=0 tx=0).stop 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- 172.21.15.150:0/2371775730 shutdown_connections 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 --2- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594137890 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 --2- 172.21.15.150:0/2371775730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd594068220 0x7fd594135340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- 172.21.15.150:0/2371775730 >> 172.21.15.150:0/2371775730 conn(0x7fd5940a31e0 msgr2=0x7fd59409f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- 172.21.15.150:0/2371775730 shutdown_connections 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- 172.21.15.150:0/2371775730 wait complete. 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient: build_initial_monmap 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient: monmap: 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:01.305 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:01.306 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 Processor -- start 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- start start 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient: init 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-b con 0x7fd594068220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-a con 0x7fd594068620 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 0x7fd59411df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): picked mon.noname-c con 0x7fd59409fb60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:01.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941427b0 con 0x7fd594068620 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59411fa80 con 0x7fd594068220 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941350e0 con 0x7fd59409fb60 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59c096700 10 monclient(hunting): authenticate will time out at 1183.394995s 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59a633700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 msgr2=0x7fd59411df40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd59a633700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 0x7fd59411df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39546/0 (socket says 172.21.15.150:39546) 2024-09-18T10:15:01.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 -- 172.21.15.150:0/230592937 learned_addr learned my addr 172.21.15.150:0/230592937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): get_auth_request con 0x7fd594068220 auth_method 0 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): get_auth_request con 0x7fd594068620 auth_method 0 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:01.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599631700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient(hunting): handle_auth_done global_id 15751 payload 824 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 -- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 msgr2=0x7fd59411df40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 0x7fd59411df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 -- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 msgr2=0x7fd594134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient: _finish_hunting 0 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 monclient: found mon.noname-b 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.297+0000 7fd599e32700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c009a20 con 0x7fd594068220 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd599e32700 1 -- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 msgr2=0x7fd5941418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd599e32700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c610 tx=0x7fd590004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:01.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590005d40 con 0x7fd594068220 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: dump: 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:01.316 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5900070d0 con 0x7fd594068220 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 5 monclient: authenticate success, global_id 15751 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 10 monclient: _renew_subs 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd594121520 con 0x7fd594068220 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590007490 con 0x7fd594068220 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:01.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd58bfff700 4 set_mon_vals no callback set 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd589ffb700 10 monclient: dump: 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 10 monclient: _renew_subs 2024-09-18T10:15:01.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.301+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5941205c0 con 0x7fd594068220 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd59c096700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd59c096700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd59c096700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59406de50 con 0x7fd594068220 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd59001b3b0 con 0x7fd594068220 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd589ffb700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 0x7fd578037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(168..168 src has 58..168) v4 ==== 9964+0+0 (secure 0 0 0) 0x7fd590048d60 con 0x7fd594068220 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd599631700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 0x7fd578037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:01.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.305+0000 7fd599631700 10 monclient: get_auth_request con 0x7fd578035a70 auth_method 0 2024-09-18T10:15:01.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.309+0000 7fd599631700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 0x7fd578037f20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a2520 tx=0x7fd5840091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:01.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.313+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd59000f840 con 0x7fd594068220 2024-09-18T10:15:01.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.313+0000 7fd589ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:01.320 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.313+0000 7fd589ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:01.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.481+0000 7fd59c096700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:01.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.481+0000 7fd59c096700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:01.488 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.481+0000 7fd59c096700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:01.489 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:01.481+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7fd594063ed0 con 0x7fd594068220 2024-09-18T10:15:02.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.165+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v169) v1 ==== 132+0+0 (secure 0 0 0) 0x7fd590017070 con 0x7fd594068220 2024-09-18T10:15:02.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.165+0000 7fd589ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:02.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.165+0000 7fd589ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.277+0000 7fd59c096700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.277+0000 7fd59c096700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.277+0000 7fd59c096700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:02.277+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]} v 0) v1 -- 0x7fd594121b70 con 0x7fd594068220 2024-09-18T10:15:03.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.185+0000 7fd589ffb700 1 -- 172.21.15.150:0/230592937 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}]=0 v170) v1 ==== 132+0+0 (secure 0 0 0) 0x7fd59000f260 con 0x7fd594068220 2024-09-18T10:15:03.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.185+0000 7fd589ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set reweight", "pool": "cold", "item": "osd.0", "weight": [6.5, 6.6]}] 2024-09-18T10:15:03.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.185+0000 7fd589ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:03.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 msgr2=0x7fd578037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 0x7fd578037f20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd5940a2520 tx=0x7fd5840091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 10 monclient: shutdown 2024-09-18T10:15:03.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:03.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 msgr2=0x7fd5941418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c610 tx=0x7fd590004a80 comp rx=0 tx=0).stop 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 shutdown_connections 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd578035a70 0x7fd578037f20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd59409fb60 0x7fd59411df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd594068620 0x7fd594134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 --2- 172.21.15.150:0/230592937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd594068220 0x7fd5941418b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 >> 172.21.15.150:0/230592937 conn(0x7fd5940a31e0 msgr2=0x7fd5940a14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 shutdown_connections 2024-09-18T10:15:03.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.189+0000 7fd59c096700 1 -- 172.21.15.150:0/230592937 wait complete. 2024-09-18T10:15:03.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:15:03.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:15:03.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:15:03.209 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '7\.' 2024-09-18T10:15:03.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: get_monmap_and_config 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: build_initial_monmap 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: monmap: 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:03.310004+0000 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:03.310004+0000 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:03.311 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:03.312 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 Processor -- start 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 -- start start 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: init 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 0x7f05800a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-b con 0x7f058009cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f058009e480 0x7f05800a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-c con 0x7f058009e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:03.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05800990a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058009e060 con 0x7f05800990a0 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800a66c0 con 0x7f058009e480 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058009d750 con 0x7f058009cf90 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0587d13700 10 monclient(hunting): authenticate will time out at 1185.403650s 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f058009e480 0x7f05800a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0585aaf700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 msgr2=0x7f05800a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:03.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f0585aaf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 0x7f05800a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f058009e480 0x7f05800a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39556/0 (socket says 172.21.15.150:39556) 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 1 -- 172.21.15.150:0/3814364451 learned_addr learned my addr 172.21.15.150:0/3814364451 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 10 monclient(hunting): get_auth_request con 0x7f058009e480 auth_method 0 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.305+0000 7f05852ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient(hunting): handle_auth_done global_id 15754 payload 274 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 -- 172.21.15.150:0/3814364451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 msgr2=0x7f05800a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 --2- 172.21.15.150:0/3814364451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 0x7f05800a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 -- 172.21.15.150:0/3814364451 >> v1:172.21.15.142:6789/0 conn(0x7f05800990a0 legacy=0x7f0580138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient: _finish_hunting 0 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 monclient: found mon.noname-c 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:03.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 -- 172.21.15.150:0/3814364451 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058009d360 con 0x7f058009e480 2024-09-18T10:15:03.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 -- 172.21.15.150:0/3814364451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009e480 msgr2=0x7f05800a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:03.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f05852ae700 1 --2- 172.21.15.150:0/3814364451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009e480 0x7f05800a07d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f05740099b0 tx=0x7f0574009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644c60b813969e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 1 -- 172.21.15.150:0/3814364451 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400bb50 con 0x7f058009e480 2024-09-18T10:15:03.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: dump: 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.327 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: _finish_auth 0 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 5 monclient: authenticate success, global_id 15754 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 1 -- 172.21.15.150:0/3814364451 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400bcb0 con 0x7f058009e480 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 4 set_mon_vals no callback set 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 1 -- 172.21.15.150:0/3814364451 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574011670 con 0x7f058009e480 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0584aad700 10 monclient: dump: 2024-09-18T10:15:03.328 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 10 monclient: shutdown 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 1 -- 172.21.15.150:0/3814364451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009e480 msgr2=0x7f05800a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.309+0000 7f0587d13700 1 --2- 172.21.15.150:0/3814364451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009e480 0x7f05800a07d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f05740099b0 tx=0x7f0574009250 comp rx=0 tx=0).stop 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- 172.21.15.150:0/3814364451 shutdown_connections 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 --2- 172.21.15.150:0/3814364451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009e480 0x7f05800a07d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 --2- 172.21.15.150:0/3814364451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009cf90 0x7f05800a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- 172.21.15.150:0/3814364451 >> 172.21.15.150:0/3814364451 conn(0x7f05800a58a0 msgr2=0x7f05800a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- 172.21.15.150:0/3814364451 shutdown_connections 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- 172.21.15.150:0/3814364451 wait complete. 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient: build_initial_monmap 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient: monmap: 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:03.330 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 Processor -- start 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- start start 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient: init 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05800990a0 0x7f0580141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-a con 0x7f05800990a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-b con 0x7f058009cf90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 0x7f058008cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): picked mon.noname-c con 0x7f058009e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800984d0 con 0x7f05800990a0 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058009c050 con 0x7f058009cf90 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058008eef0 con 0x7f058009e480 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0587d13700 10 monclient(hunting): authenticate will time out at 1185.410087s 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05862b0700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 msgr2=0x7f058008cee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05862b0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 0x7f058008cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39560/0 (socket says 172.21.15.150:39560) 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 -- 172.21.15.150:0/1980610122 learned_addr learned my addr 172.21.15.150:0/1980610122 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): get_auth_request con 0x7f058009cf90 auth_method 0 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05800990a0 0x7f0580141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): get_auth_request con 0x7f05800990a0 auth_method 0 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f0585aaf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient(hunting): handle_auth_done global_id 15757 payload 824 2024-09-18T10:15:03.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 -- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 msgr2=0x7f058008cee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 0x7f058008cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 -- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05800990a0 msgr2=0x7f0580141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05800990a0 0x7f0580141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient: _finish_hunting 0 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 monclient: found mon.noname-b 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.313+0000 7f05852ae700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740094f0 con 0x7f058009cf90 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f05852ae700 1 -- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 msgr2=0x7f05801422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f05852ae700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0574006fd0 tx=0x7f0574016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574006c00 con 0x7f058009cf90 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: dump: 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.336 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0574020070 con 0x7f058009cf90 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 5 monclient: authenticate success, global_id 15757 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _renew_subs 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800904a0 con 0x7f058009cf90 2024-09-18T10:15:03.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057401c450 con 0x7f058009cf90 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 10 monclient: dump: 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.339 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _renew_subs 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057f7fe700 4 set_mon_vals no callback set 2024-09-18T10:15:03.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05801427e0 con 0x7f058009cf90 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0574022030 con 0x7f058009cf90 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 0x7f056c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(170..170 src has 58..170) v4 ==== 9964+0+0 (secure 0 0 0) 0x7f057404b5f0 con 0x7f058009cf90 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0585aaf700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 0x7f056c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0585aaf700 10 monclient: get_auth_request con 0x7f056c035a70 auth_method 0 2024-09-18T10:15:03.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.317+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.325+0000 7f0585aaf700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 0x7f056c037f20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0578003980 tx=0x7f05780091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.325+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.325+0000 7f057dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.343 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.325+0000 7f057dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:03.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f0587d13700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f0587d13700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f0587d13700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f057dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.501+0000 7f057dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:03.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.601+0000 7f0587d13700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.601+0000 7f0587d13700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.601+0000 7f0587d13700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:03.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.601+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f057dffb700 1 -- 172.21.15.150:0/1980610122 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f0580090f80 con 0x7f058009cf90 2024-09-18T10:15:03.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f057dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f057dffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 msgr2=0x7f056c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 0x7f056c037f20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0578003980 tx=0x7f05780091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 10 monclient: shutdown 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 msgr2=0x7f05801422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0574006fd0 tx=0x7f0574016000 comp rx=0 tx=0).stop 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 shutdown_connections 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f056c035a70 0x7f056c037f20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f058009e480 0x7f058008cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f058009cf90 0x7f05801422a0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 --2- 172.21.15.150:0/1980610122 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05800990a0 0x7f0580141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 >> 172.21.15.150:0/1980610122 conn(0x7f05800a58a0 msgr2=0x7f05800a2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:03.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 shutdown_connections 2024-09-18T10:15:03.615 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.605+0000 7f0587d13700 1 -- 172.21.15.150:0/1980610122 wait complete. 2024-09-18T10:15:03.622 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:03.622 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:03.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:15:03.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:15:03.623 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:15:03.624 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '8\.' 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: get_monmap_and_config 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: build_initial_monmap 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: monmap: 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.722 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:03.721608+0000 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:03.721608+0000 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:03.723 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 Processor -- start 2024-09-18T10:15:03.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 -- start start 2024-09-18T10:15:03.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: init 2024-09-18T10:15:03.725 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f269409acd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2694096a90 0x7f269409c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): picked mon.noname-a con 0x7f2694096a90 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 0x7f2694137790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): picked mon.noname-b con 0x7f269409cc20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940958e0 con 0x7f269409acd0 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f269409d080 con 0x7f2694096a90 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269409b0a0 con 0x7f269409cc20 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:03.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f269949d700 10 monclient(hunting): authenticate will time out at 1185.815026s 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f2692ffd700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f269409acd0 0x7f269409bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40220/0 (socket says 172.21.15.150:40220) 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f2692ffd700 1 -- 172.21.15.150:0/33620154 learned_addr learned my addr 172.21.15.150:0/33620154 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26937fe700 1 -- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 msgr2=0x7f2694137790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26937fe700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 0x7f2694137790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 1 --2- 172.21.15.150:0/33620154 >> v2:172.21.15.142:3300/0 conn(0x7f2694096a90 0x7f269409c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): get_auth_request con 0x7f2694096a90 auth_method 0 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.717+0000 7f26927fc700 10 monclient(hunting): handle_auth_done global_id 54546 payload 274 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 -- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 msgr2=0x7f2694137790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 0x7f2694137790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 -- 172.21.15.150:0/33620154 >> v1:172.21.15.150:6789/0 conn(0x7f269409acd0 legacy=0x7f269409bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 10 monclient: _finish_hunting 0 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 monclient: found mon.noname-a 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 -- 172.21.15.150:0/33620154 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694095b50 con 0x7f2694096a90 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 -- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 msgr2=0x7f269409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f26927fc700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269409c6e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f26880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c31409f5a6435b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 1 -- 172.21.15.150:0/33620154 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268800ba30 con 0x7f2694096a90 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:03.736 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: dump: 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:03.737 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 5 monclient: authenticate success, global_id 54546 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 1 -- 172.21.15.150:0/33620154 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268800bbb0 con 0x7f2694096a90 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 4 set_mon_vals no callback set 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 1 -- 172.21.15.150:0/33620154 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2688011420 con 0x7f2694096a90 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f2691ffb700 10 monclient: dump: 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.738 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 10 monclient: shutdown 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 -- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 msgr2=0x7f269409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269409c6e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f26880091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 -- 172.21.15.150:0/33620154 shutdown_connections 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409cc20 0x7f2694137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 --2- 172.21.15.150:0/33620154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269409c6e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 -- 172.21.15.150:0/33620154 >> 172.21.15.150:0/33620154 conn(0x7f26940a15e0 msgr2=0x7f26940a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 -- 172.21.15.150:0/33620154 shutdown_connections 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.721+0000 7f269949d700 1 -- 172.21.15.150:0/33620154 wait complete. 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient: build_initial_monmap 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient: monmap: 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:03.740 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 Processor -- start 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 -- start start 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient: init 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): picked mon.noname-a con 0x7f2694096a90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 0x7f26940946d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): picked mon.noname-c con 0x7f269409acd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 0x7f2694094c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): picked mon.noname-b con 0x7f269409cc20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:03.741 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694098410 con 0x7f2694096a90 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26940a01a0 con 0x7f269409cc20 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2694140aa0 con 0x7f269409acd0 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f269949d700 10 monclient(hunting): authenticate will time out at 1185.821683s 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26927fc700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 msgr2=0x7f26940946d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26927fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 0x7f26940946d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 0x7f2694094c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:46324/0 (socket says 172.21.15.150:46324) 2024-09-18T10:15:03.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 -- 172.21.15.150:0/1111086617 learned_addr learned my addr 172.21.15.150:0/1111086617 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): get_auth_request con 0x7f2694096a90 auth_method 0 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): get_auth_request con 0x7f269409cc20 auth_method 0 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:03.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient(hunting): handle_auth_done global_id 54549 payload 824 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 -- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 msgr2=0x7f26940946d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 0x7f26940946d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 -- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 msgr2=0x7f2694094c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 0x7f2694094c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 monclient: found mon.noname-a 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f2692ffd700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688009420 con 0x7f2694096a90 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.725+0000 7f26937fe700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 0x7f2694094c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:03.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f2692ffd700 1 -- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 msgr2=0x7f269413fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f2692ffd700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f268c00cca0 tx=0x7f268c005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c00abd0 con 0x7f2694096a90 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: dump: 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.745 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: _finish_auth 0 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c00ad50 con 0x7f2694096a90 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 5 monclient: authenticate success, global_id 54549 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _renew_subs 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269408de70 con 0x7f2694096a90 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c007340 con 0x7f2694096a90 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:03.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 10 monclient: dump: 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _renew_subs 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f26917fa700 4 set_mon_vals no callback set 2024-09-18T10:15:03.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269408e9e0 con 0x7f2694096a90 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269406f4a0 con 0x7f2694096a90 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f268c0076f0 con 0x7f2694096a90 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 0x7f2680037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.729+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(170..170 src has 58..170) v4 ==== 9964+0+0 (secure 0 0 0) 0x7f268c049d10 con 0x7f2694096a90 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.733+0000 7f26927fc700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 0x7f2680037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:03.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.733+0000 7f26927fc700 10 monclient: get_auth_request con 0x7f2680035a70 auth_method 0 2024-09-18T10:15:03.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.733+0000 7f26927fc700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 0x7f2680037f20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f2688016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:03.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.741+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f268c050080 con 0x7f2694096a90 2024-09-18T10:15:03.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.741+0000 7f267affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:03.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.741+0000 7f267affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f269949d700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f269949d700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f269949d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f2694063ed0 con 0x7f2694096a90 2024-09-18T10:15:03.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f268c01a960 con 0x7f2694096a90 2024-09-18T10:15:03.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f267affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:03.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:03.909+0000 7f267affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:04.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f269949d700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f269949d700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f269949d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7f269408f930 con 0x7f2694096a90 2024-09-18T10:15:04.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f267affd700 1 -- 172.21.15.150:0/1111086617 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7f268c029090 con 0x7f2694096a90 2024-09-18T10:15:04.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f267affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.009+0000 7f267affd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 msgr2=0x7f2680037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 0x7f2680037f20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f2688016000 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 10 monclient: shutdown 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 msgr2=0x7f269413fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f268c00cca0 tx=0x7f268c005480 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 shutdown_connections 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2680035a70 0x7f2680037f20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269409cc20 0x7f2694094c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269409acd0 0x7f26940946d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 --2- 172.21.15.150:0/1111086617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2694096a90 0x7f269413fc30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 >> 172.21.15.150:0/1111086617 conn(0x7f26940a15e0 msgr2=0x7f2694096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:04.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 shutdown_connections 2024-09-18T10:15:04.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.013+0000 7f269949d700 1 -- 172.21.15.150:0/1111086617 wait complete. 2024-09-18T10:15:04.027 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:04.027 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:04.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush tree --show-shadow 2024-09-18T10:15:04.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep 'osd\.0' 2024-09-18T10:15:04.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep '6\.' 2024-09-18T10:15:04.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep globster 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: get_monmap_and_config 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: build_initial_monmap 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: monmap: 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:04.130335+0000 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:04.130335+0000 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:04.131 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:04.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 Processor -- start 2024-09-18T10:15:04.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 -- start start 2024-09-18T10:15:04.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: init 2024-09-18T10:15:04.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6580a1810 0x7fd65806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-c con 0x7fd6580a1810 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd6580a1be0 0x7fd6580a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-a con 0x7fd6580a1be0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6581301d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6580a33e0 con 0x7fd6581301d0 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65806c620 con 0x7fd6580a1be0 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd65806d750 con 0x7fd6580a1810 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:04.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.125+0000 7fd65fd64700 10 monclient(hunting): authenticate will time out at 1186.223930s 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6580a1810 0x7fd65806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd6580a1810 0x7fd65806e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39572/0 (socket says 172.21.15.150:39572) 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 -- 172.21.15.150:0/2603129133 learned_addr learned my addr 172.21.15.150:0/2603129133 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): get_auth_request con 0x7fd6580a1810 auth_method 0 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 1 --2- 172.21.15.150:0/2603129133 >> v2:172.21.15.142:3300/0 conn(0x7fd6580a1be0 0x7fd6580a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): get_auth_request con 0x7fd6580a1be0 auth_method 0 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient(hunting): handle_auth_done global_id 15763 payload 274 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 -- 172.21.15.150:0/2603129133 >> v2:172.21.15.142:3300/0 conn(0x7fd6580a1be0 msgr2=0x7fd6580a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 --2- 172.21.15.150:0/2603129133 >> v2:172.21.15.142:3300/0 conn(0x7fd6580a1be0 0x7fd6580a1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 -- 172.21.15.150:0/2603129133 >> v1:172.21.15.142:6789/0 conn(0x7fd6581301d0 legacy=0x7fd658130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient: _finish_hunting 0 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 monclient: found mon.noname-c 2024-09-18T10:15:04.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:04.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 -- 172.21.15.150:0/2603129133 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65806da30 con 0x7fd6580a1810 2024-09-18T10:15:04.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 -- 172.21.15.150:0/2603129133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6580a1810 msgr2=0x7fd65806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65db00700 1 --2- 172.21.15.150:0/2603129133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6580a1810 0x7fd65806e9e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd65806ca00 tx=0x7fd648009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fe7a33d217ebf31 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 1 -- 172.21.15.150:0/2603129133 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd648009e70 con 0x7fd6580a1810 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: dump: 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.140 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: _finish_auth 0 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 1 -- 172.21.15.150:0/2603129133 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64800ba70 con 0x7fd6580a1810 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65fd64700 5 monclient: authenticate success, global_id 15763 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65fd64700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65fd64700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 1 -- 172.21.15.150:0/2603129133 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64800be30 con 0x7fd6580a1810 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65fd64700 4 set_mon_vals no callback set 2024-09-18T10:15:04.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.129+0000 7fd65cafe700 10 monclient: dump: 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 10 monclient: shutdown 2024-09-18T10:15:04.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- 172.21.15.150:0/2603129133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6580a1810 msgr2=0x7fd65806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 --2- 172.21.15.150:0/2603129133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6580a1810 0x7fd65806e9e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd65806ca00 tx=0x7fd648009250 comp rx=0 tx=0).stop 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- 172.21.15.150:0/2603129133 shutdown_connections 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 --2- 172.21.15.150:0/2603129133 >> v2:172.21.15.142:3300/0 conn(0x7fd6580a1be0 0x7fd6580a1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 --2- 172.21.15.150:0/2603129133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6580a1810 0x7fd65806e9e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- 172.21.15.150:0/2603129133 >> 172.21.15.150:0/2603129133 conn(0x7fd6580716e0 msgr2=0x7fd6580a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- 172.21.15.150:0/2603129133 shutdown_connections 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- 172.21.15.150:0/2603129133 wait complete. 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 10 monclient: build_initial_monmap 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 10 monclient: monmap: 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:04.143 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 Processor -- start 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 1 -- start start 2024-09-18T10:15:04.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.133+0000 7fd65fd64700 10 monclient: init 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 0x7fd65813f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-c con 0x7fd6580a1810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6580a1be0 0x7fd65813fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-a con 0x7fd6580a1be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): picked mon.noname-b con 0x7fd6581301d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65806ef80 con 0x7fd6580a1be0 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65809dad0 con 0x7fd6581301d0 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd658120480 con 0x7fd6580a1810 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65db00700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 msgr2=0x7fd65813f930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65db00700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 0x7fd65813f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient(hunting): authenticate will time out at 1186.232322s 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39588/0 (socket says 172.21.15.150:39588) 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 -- 172.21.15.150:0/1454421945 learned_addr learned my addr 172.21.15.150:0/1454421945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): get_auth_request con 0x7fd6581301d0 auth_method 0 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6580a1be0 0x7fd65813fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): get_auth_request con 0x7fd6580a1be0 auth_method 0 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65d2ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient(hunting): handle_auth_done global_id 15766 payload 824 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 -- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 msgr2=0x7fd65813f930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 0x7fd65813f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 -- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6580a1be0 msgr2=0x7fd65813fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6580a1be0 0x7fd65813fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient: _finish_hunting 0 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 monclient: found mon.noname-b 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480094f0 con 0x7fd6581301d0 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 -- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 msgr2=0x7fd65811cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65e301700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c930 tx=0x7fd6500026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650012840 con 0x7fd6581301d0 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: dump: 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.153 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 5 monclient: authenticate success, global_id 15766 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6500129c0 con 0x7fd6581301d0 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient: _renew_subs 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65811e670 con 0x7fd6581301d0 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650012d80 con 0x7fd6581301d0 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:04.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64ffff700 4 set_mon_vals no callback set 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.137+0000 7fd64dffb700 10 monclient: dump: 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 10 monclient: _renew_subs 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65811f240 con 0x7fd6581301d0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd65000a5c0 con 0x7fd6581301d0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6580a25e0 con 0x7fd6581301d0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd64dffb700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 0x7fd63c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(170..170 src has 58..170) v4 ==== 9964+0+0 (secure 0 0 0) 0x7fd6500497f0 con 0x7fd6581301d0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65db00700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 0x7fd63c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.141+0000 7fd65db00700 10 monclient: get_auth_request con 0x7fd63c035ac0 auth_method 0 2024-09-18T10:15:04.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.145+0000 7fd65db00700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 0x7fd63c037f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd65806ca00 tx=0x7fd648016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.149+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd650010d50 con 0x7fd6581301d0 2024-09-18T10:15:04.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.153+0000 7fd64dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.153+0000 7fd64dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:04.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd65fd64700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd65fd64700 10 monclient: _send_command 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd65fd64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd658063ed0 con 0x7fd6581301d0 2024-09-18T10:15:04.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fd650010750 con 0x7fd6581301d0 2024-09-18T10:15:04.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd64dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.325+0000 7fd64dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:04.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd65fd64700 10 start_mon_command cmd=[{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd65fd64700 10 monclient: _send_command 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd65fd64700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:04.438 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tree", "show_shadow": true} v 0) v1 -- 0x7fd65811fe40 con 0x7fd6581301d0 2024-09-18T10:15:04.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd64dffb700 1 -- 172.21.15.150:0/1454421945 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush tree", "show_shadow": true}]=0 v170) v1 ==== 83+0+2320 (secure 0 0 0) 0x7fd650010930 con 0x7fd6581301d0 2024-09-18T10:15:04.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd64dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush tree", "show_shadow": true}] 2024-09-18T10:15:04.439 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.433+0000 7fd64dffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 msgr2=0x7fd63c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 0x7fd63c037f70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd65806ca00 tx=0x7fd648016000 comp rx=0 tx=0).stop 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 10 monclient: shutdown 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 msgr2=0x7fd65811cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c930 tx=0x7fd6500026e0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 shutdown_connections 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd63c035ac0 0x7fd63c037f70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6581301d0 0x7fd65811cbc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd6580a1be0 0x7fd65813fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 --2- 172.21.15.150:0/1454421945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd6580a1810 0x7fd65813f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 >> 172.21.15.150:0/1454421945 conn(0x7fd6580716e0 msgr2=0x7fd658099ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:04.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 shutdown_connections 2024-09-18T10:15:04.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.437+0000 7fd65fd64700 1 -- 172.21.15.150:0/1454421945 wait complete. 2024-09-18T10:15:04.452 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:04.452 INFO:tasks.workunit.client.0.smithi150.stdout: 0 globster 0.08789 7.50000 8.50000 6.50000 osd.0 2024-09-18T10:15:04.452 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm-device-class osd.0 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.541+0000 7f2e7c621700 10 monclient: get_monmap_and_config 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.541+0000 7f2e7c621700 10 monclient: build_initial_monmap 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.541+0000 7f2e7c621700 10 monclient: monmap: 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:04.547468+0000 2024-09-18T10:15:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:04.547468+0000 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:04.549 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 Processor -- start 2024-09-18T10:15:04.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 -- start start 2024-09-18T10:15:04.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient: init 2024-09-18T10:15:04.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:04.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 0x7f2e7409e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-c con 0x7f2e7409e370 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2e7406cbc0 0x7f2e7409ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-a con 0x7f2e7406cbc0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e740699a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7406af30 con 0x7f2e740699a0 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7409f1c0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7406b630 con 0x7f2e7409e370 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7c621700 10 monclient(hunting): authenticate will time out at 1186.640949s 2024-09-18T10:15:04.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 0x7f2e7409e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 0x7f2e7409e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39594/0 (socket says 172.21.15.150:39594) 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 1 -- 172.21.15.150:0/755569902 learned_addr learned my addr 172.21.15.150:0/755569902 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): get_auth_request con 0x7f2e7409e370 auth_method 0 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/755569902 >> v2:172.21.15.142:3300/0 conn(0x7f2e7406cbc0 0x7f2e7409ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): get_auth_request con 0x7f2e7406cbc0 auth_method 0 2024-09-18T10:15:04.554 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_done global_id 54558 payload 274 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 -- 172.21.15.150:0/755569902 >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 msgr2=0x7f2e7409e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/755569902 >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 0x7f2e7409e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 -- 172.21.15.150:0/755569902 >> v1:172.21.15.150:6789/0 conn(0x7f2e740699a0 legacy=0x7f2e74132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient: _finish_hunting 0 2024-09-18T10:15:04.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 monclient: found mon.noname-a 2024-09-18T10:15:04.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:04.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.545+0000 7f2e79bbc700 1 -- 172.21.15.150:0/755569902 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7406ac50 con 0x7f2e7406cbc0 2024-09-18T10:15:04.562 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 1 -- 172.21.15.150:0/755569902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 msgr2=0x7f2e7409ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/755569902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e7409ec80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e7406f0e0 tx=0x7f2e640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6fdeae2d4a07bd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 1 -- 172.21.15.150:0/755569902 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64011840 con 0x7f2e7406cbc0 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: dump: 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.563 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: _finish_auth 0 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 5 monclient: authenticate success, global_id 54558 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 1 -- 172.21.15.150:0/755569902 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e640119c0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:04.564 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 4 set_mon_vals no callback set 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 1 -- 172.21.15.150:0/755569902 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64011d80 con 0x7f2e7406cbc0 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e793bb700 10 monclient: dump: 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.565 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: shutdown 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- 172.21.15.150:0/755569902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 msgr2=0x7f2e7409ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- 172.21.15.150:0/755569902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e7409ec80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e7406f0e0 tx=0x7f2e640091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- 172.21.15.150:0/755569902 shutdown_connections 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- 172.21.15.150:0/755569902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e7409ec80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- 172.21.15.150:0/755569902 >> v2:172.21.15.150:3300/0 conn(0x7f2e7409e370 0x7f2e7409e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- 172.21.15.150:0/755569902 >> 172.21.15.150:0/755569902 conn(0x7f2e74071850 msgr2=0x7f2e740a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- 172.21.15.150:0/755569902 shutdown_connections 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- 172.21.15.150:0/755569902 wait complete. 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: build_initial_monmap 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: monmap: 2024-09-18T10:15:04.566 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 Processor -- start 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- start start 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: init 2024-09-18T10:15:04.567 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-b con 0x7f2e740699a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e74131580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-a con 0x7f2e7406cbc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 0x7f2e7411e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): picked mon.noname-c con 0x7f2e7409e370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e741216d0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e740a2c80 con 0x7f2e740699a0 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74120330 con 0x7f2e7409e370 2024-09-18T10:15:04.568 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7c621700 10 monclient(hunting): authenticate will time out at 1186.647395s 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7abbe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 msgr2=0x7f2e7411e0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7abbe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 0x7f2e7411e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39596/0 (socket says 172.21.15.150:39596) 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 1 -- 172.21.15.150:0/3715939211 learned_addr learned my addr 172.21.15.150:0/3715939211 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): get_auth_request con 0x7f2e740699a0 auth_method 0 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e74131580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): get_auth_request con 0x7f2e7406cbc0 auth_method 0 2024-09-18T10:15:04.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e7a3bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.549+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 10 monclient(hunting): handle_auth_done global_id 54561 payload 824 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 -- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 msgr2=0x7f2e7411e0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 0x7f2e7411e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 -- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 msgr2=0x7f2e741417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:04.570 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 10 monclient: _finish_hunting 0 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 monclient: found mon.noname-a 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e64009420 con 0x7f2e7406cbc0 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7a3bd700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 -- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 msgr2=0x7f2e74131580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e79bbc700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e74131580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e64016000 tx=0x7f2e64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6401f070 con 0x7f2e7406cbc0 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: dump: 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.571 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 5 monclient: authenticate success, global_id 54561 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64006af0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 10 monclient: _renew_subs 2024-09-18T10:15:04.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7406f110 con 0x7f2e7406cbc0 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400bbc0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e78bba700 4 set_mon_vals no callback set 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e6a7fc700 10 monclient: dump: 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:04.573 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 10 monclient: _renew_subs 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.553+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e74120770 con 0x7f2e7406cbc0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7c621700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7c621700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7c621700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e740a4540 con 0x7f2e7406cbc0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2e64023070 con 0x7f2e7406cbc0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e6a7fc700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 0x7f2e58037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(170..170 src has 58..170) v4 ==== 9964+0+0 (secure 0 0 0) 0x7f2e6404a680 con 0x7f2e7406cbc0 2024-09-18T10:15:04.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7a3bd700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 0x7f2e58037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:04.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7a3bd700 10 monclient: get_auth_request con 0x7f2e58035a70 auth_method 0 2024-09-18T10:15:04.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.557+0000 7f2e7a3bd700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 0x7f2e58037f20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2e70011540 tx=0x7f2e70009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:04.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.565+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2e64023320 con 0x7f2e7406cbc0 2024-09-18T10:15:04.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.565+0000 7f2e6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:04.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.565+0000 7f2e6a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:04.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.717+0000 7f2e7c621700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:04.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.717+0000 7f2e7c621700 10 monclient: _send_command 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:04.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.717+0000 7f2e7c621700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:04.724 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:04.717+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f2e7406dfc0 con 0x7f2e7406cbc0 2024-09-18T10:15:04.890 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:15:04.891 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:15:04.891 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:15:04.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:15:04.893 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:15:04.893 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:15:04.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:15:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 10 monclient: build_initial_monmap 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 10 monclient: monmap: 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:04.994842+0000 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:04.994842+0000 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:04.998 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 1 Processor -- start 2024-09-18T10:15:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 1 -- start start 2024-09-18T10:15:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.989+0000 7f49d455e700 10 monclient: init 2024-09-18T10:15:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T10:15:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-b con 0x7f49cc09c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49cc09cb30 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f49cc133b00 0x7f49cc135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-a con 0x7f49cc133b00 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc0989c0 con 0x7f49cc09cb30 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc09b6d0 con 0x7f49cc133b00 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc09d610 con 0x7f49cc09c760 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient(hunting): authenticate will time out at 1193.920624s 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d1af9700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f49cc09cb30 0x7f49cc09cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44636/0 (socket says 172.21.15.142:44636) 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d1af9700 1 -- 172.21.15.142:0/3413567451 learned_addr learned my addr 172.21.15.142:0/3413567451 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 --2- 172.21.15.142:0/3413567451 >> v2:172.21.15.142:3300/0 conn(0x7f49cc133b00 0x7f49cc135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): get_auth_request con 0x7f49cc133b00 auth_method 0 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 10 monclient(hunting): get_auth_request con 0x7f49cc09c760 auth_method 0 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 1 -- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 msgr2=0x7f49cc09adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 1 -- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 msgr2=0x7f49cc09adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d22fa700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient(hunting): handle_auth_done global_id 54564 payload 274 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 -- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 msgr2=0x7f49cc09adf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 -- 172.21.15.142:0/3413567451 >> v1:172.21.15.142:6789/0 conn(0x7f49cc09cb30 legacy=0x7f49cc09cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient: _finish_hunting 0 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 monclient: found mon.noname-a 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 -- 172.21.15.142:0/3413567451 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc09dcd0 con 0x7f49cc133b00 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 -- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc133b00 msgr2=0x7f49cc135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d2afb700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc133b00 0x7f49cc135ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009e60 tx=0x7f49c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e8a128edb6e3d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 1 -- 172.21.15.142:0/3413567451 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400e070 con 0x7f49cc133b00 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: dump: 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: _finish_auth 0 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 1 -- 172.21.15.142:0/3413567451 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c400be40 con 0x7f49cc133b00 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 5 monclient: authenticate success, global_id 54564 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 1 -- 172.21.15.142:0/3413567451 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4012660 con 0x7f49cc133b00 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d455e700 4 set_mon_vals no callback set 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.993+0000 7f49d12f8700 10 monclient: dump: 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:05.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: shutdown 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc133b00 msgr2=0x7f49cc135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc133b00 0x7f49cc135ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009e60 tx=0x7f49c4009780 comp rx=0 tx=0).stop 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- 172.21.15.142:0/3413567451 shutdown_connections 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc133b00 0x7f49cc135ee0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- 172.21.15.142:0/3413567451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc09c760 0x7f49cc09adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- 172.21.15.142:0/3413567451 >> 172.21.15.142:0/3413567451 conn(0x7f49cc071820 msgr2=0x7f49cc0a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:05.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- 172.21.15.142:0/3413567451 shutdown_connections 2024-09-18T10:15:05.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- 172.21.15.142:0/3413567451 wait complete. 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: build_initial_monmap 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: monmap: 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:05.014 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 Processor -- start 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- start start 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: init 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-a con 0x7f49cc09c760 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:05.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49cc132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-b con 0x7f49cc09cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49cc1330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): picked mon.noname-c con 0x7f49cc133b00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc08da00 con 0x7f49cc09c760 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc0a0aa0 con 0x7f49cc09cb30 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc08e9f0 con 0x7f49cc133b00 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d455e700 10 monclient(hunting): authenticate will time out at 1193.927217s 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42444/0 (socket says 172.21.15.142:42444) 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 learned_addr learned my addr 172.21.15.142:0/1245689622 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): get_auth_request con 0x7f49cc09c760 auth_method 0 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49cc1330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 10 monclient(hunting): get_auth_request con 0x7f49cc133b00 auth_method 0 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:05.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 msgr2=0x7f49cc1330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 msgr2=0x7f49cc1330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49cc1330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d1af9700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49cc132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d2afb700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49cc1330f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d1af9700 10 monclient(hunting): get_auth_request con 0x7f49cc09cb30 auth_method 0 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d1af9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d1af9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d1af9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient(hunting): handle_auth_done global_id 54567 payload 824 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 msgr2=0x7f49cc1330f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:05.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49cc1330f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 msgr2=0x7f49cc132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49cc132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient: _finish_hunting 0 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 monclient: found mon.noname-a 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:04.997+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c4009a20 con 0x7f49cc09c760 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 msgr2=0x7f49cc132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d860 tx=0x7f49c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8012800 con 0x7f49cc09c760 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:05.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: dump: 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:05.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 5 monclient: authenticate success, global_id 54567 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8012960 con 0x7f49cc09c760 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _renew_subs 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc08c7c0 con 0x7f49cc09c760 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8012d20 con 0x7f49cc09c760 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: dump: 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.021 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d0af7700 4 set_mon_vals no callback set 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _renew_subs 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc06f540 con 0x7f49cc09c760 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 msgr2=0x7f49cc132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 msgr2=0x7f49cc132670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:05.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d860 tx=0x7f49c8008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f49c800d860 tx=0x7f49c8008040 comp rx=0 tx=0).stop 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 msgr2=0x7f49cc132670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49cc09c760 0x7f49cc132670 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 0x7f49b0005c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): picked mon.a con 0x7f49b0003820 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49b00086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): picked mon.c con 0x7f49cc133b00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49b000b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): picked mon.b con 0x7f49cc09cb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:05.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b000ba30 con 0x7f49b0003820 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b000bb70 con 0x7f49cc09cb30 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b000bcb0 con 0x7f49cc133b00 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49be7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 0x7f49b0005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49b00086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): get_auth_request con 0x7f49b0003820 auth_method 0 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): get_auth_request con 0x7f49cc133b00 auth_method 0 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49b000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d1af9700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 10 monclient(hunting): get_auth_request con 0x7f49cc09cb30 auth_method 0 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:05.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient(hunting): handle_auth_done global_id 54567 payload 293 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 msgr2=0x7f49b00086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49b00086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 msgr2=0x7f49b000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49b000b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient: _finish_hunting 0 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 monclient: found mon.a 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f49b000bf90 con 0x7f49b0003820 2024-09-18T10:15:05.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 10 monclient: _finish_auth 0 2024-09-18T10:15:05.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d22fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d2afb700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49b000b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.001+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc08dc40 con 0x7f49b0003820 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49d22fa700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 msgr2=0x7f49b0005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49d22fa700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 0x7f49b0005c00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06fa70 tx=0x7f49c801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8014a50 con 0x7f49b0003820 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:05.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 10 monclient: dump: 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8014bb0 con 0x7f49b0003820 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:05.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49bffff700 4 set_mon_vals no callback set 2024-09-18T10:15:05.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.005+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f49c8014d20 con 0x7f49b0003820 2024-09-18T10:15:05.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.009+0000 7f49be7fc700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 0x7f49b0040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:05.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.009+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 4 ==== osd_map(170..170 src has 58..170) v4 ==== 9964+0+0 (secure 0 0 0) 0x7f49c804ca10 con 0x7f49b0003820 2024-09-18T10:15:05.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.009+0000 7f49d1af9700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 0x7f49b0040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:05.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.009+0000 7f49d1af9700 10 monclient: get_auth_request con 0x7f49b003e190 auth_method 0 2024-09-18T10:15:05.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.009+0000 7f49d1af9700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 0x7f49b0040640 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f49c000ca60 tx=0x7f49c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:05.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.013+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f49c800b990 con 0x7f49b0003820 2024-09-18T10:15:05.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.013+0000 7f49be7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:05.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.013+0000 7f49be7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:05.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.193+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 done removing class of osd(s): 0 v171) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2e6402a070 con 0x7f2e7406cbc0 2024-09-18T10:15:05.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.193+0000 7f2e6a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:05.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.193+0000 7f2e6a7fc700 10 monclient: _finish_command 2 = system:0 done removing class of osd(s): 0 2024-09-18T10:15:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:15:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f49cc0696a0 con 0x7f49b0003820 2024-09-18T10:15:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49be7fc700 1 -- 172.21.15.142:0/1245689622 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f49c8000e60 con 0x7f49b0003820 2024-09-18T10:15:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49be7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:05.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49be7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:05.246 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:15:05.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 msgr2=0x7f49b0040640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 0x7f49b0040640 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f49c000ca60 tx=0x7f49c0009040 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 10 monclient: shutdown 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 msgr2=0x7f49b0005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 0x7f49b0005c00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49cc06fa70 tx=0x7f49c801b000 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 shutdown_connections 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f49b003e190 0x7f49b0040640 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49cc09cb30 0x7f49b000b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49cc133b00 0x7f49b00086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 --2- 172.21.15.142:0/1245689622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b0003820 0x7f49b0005c00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 >> 172.21.15.142:0/1245689622 conn(0x7f49cc071820 msgr2=0x7f49cc133ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 shutdown_connections 2024-09-18T10:15:05.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:05.241+0000 7f49d455e700 1 -- 172.21.15.142:0/1245689622 wait complete. 2024-09-18T10:15:05.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.301+0000 7f2e7c621700 10 start_mon_command cmd=[{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:05.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.301+0000 7f2e7c621700 10 monclient: _send_command 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:05.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.301+0000 7f2e7c621700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:05.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:05.301+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm-device-class", "ids": ["osd.0"]} v 0) v1 -- 0x7f2e74063e80 con 0x7f2e7406cbc0 2024-09-18T10:15:06.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.205+0000 7f2e6a7fc700 1 -- 172.21.15.150:0/3715939211 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}]=0 osd.0 belongs to no class, done removing class of osd(s): v172) v1 ==== 149+0+0 (secure 0 0 0) 0x7f2e64015030 con 0x7f2e7406cbc0 2024-09-18T10:15:06.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.205+0000 7f2e6a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm-device-class", "ids": ["osd.0"]}] 2024-09-18T10:15:06.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.205+0000 7f2e6a7fc700 10 monclient: _finish_command 3 = system:0 osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:15:06.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 msgr2=0x7f2e58037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:06.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 0x7f2e58037f20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2e70011540 tx=0x7f2e70009250 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 10 monclient: shutdown 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 msgr2=0x7f2e74131580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e74131580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e64016000 tx=0x7f2e64017000 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 shutdown_connections 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2e58035a70 0x7f2e58037f20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e7409e370 0x7f2e7411e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e7406cbc0 0x7f2e74131580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 --2- 172.21.15.150:0/3715939211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e740699a0 0x7f2e741417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 >> 172.21.15.150:0/3715939211 conn(0x7f2e74071850 msgr2=0x7f2e7406e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 shutdown_connections 2024-09-18T10:15:06.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.209+0000 7f2e7c621700 1 -- 172.21.15.150:0/3715939211 wait complete. 2024-09-18T10:15:06.217 INFO:tasks.workunit.client.0.smithi150.stderr:osd.0 belongs to no class, done removing class of osd(s): 2024-09-18T10:15:06.230 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool rm cool cool --yes-i-really-really-mean-it 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: get_monmap_and_config 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: build_initial_monmap 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: monmap: 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:06.333492+0000 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:06.333492+0000 2024-09-18T10:15:06.334 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:06.335 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 Processor -- start 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 -- start start 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: init 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 0x7f1da80a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-c con 0x7f1da80a2810 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1da80a1550 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:06.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1da80a1950 0x7f1da8132e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-a con 0x7f1da80a1950 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da80697a0 con 0x7f1da80a1550 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da806a240 con 0x7f1da80a1950 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da809d170 con 0x7f1da80a2810 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dada7f700 10 monclient(hunting): authenticate will time out at 1188.427097s 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 0x7f1da80a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 0x7f1da80a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39620/0 (socket says 172.21.15.150:39620) 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 1 -- 172.21.15.150:0/2519726633 learned_addr learned my addr 172.21.15.150:0/2519726633 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:06.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 10 monclient(hunting): get_auth_request con 0x7f1da80a2810 auth_method 0 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1daca7d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 1 --2- 172.21.15.150:0/2519726633 >> v2:172.21.15.142:3300/0 conn(0x7f1da80a1950 0x7f1da8132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): get_auth_request con 0x7f1da80a1950 auth_method 0 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.329+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 10 monclient(hunting): handle_auth_done global_id 54570 payload 274 2024-09-18T10:15:06.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 -- 172.21.15.150:0/2519726633 >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 msgr2=0x7f1da80a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 --2- 172.21.15.150:0/2519726633 >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 0x7f1da80a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 -- 172.21.15.150:0/2519726633 >> v1:172.21.15.150:6789/0 conn(0x7f1da80a1550 legacy=0x7f1da8132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 10 monclient: _finish_hunting 0 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 monclient: found mon.noname-a 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:06.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 -- 172.21.15.150:0/2519726633 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da809cec0 con 0x7f1da80a1950 2024-09-18T10:15:06.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 -- 172.21.15.150:0/2519726633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 msgr2=0x7f1da8132e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:06.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dad27e700 1 --2- 172.21.15.150:0/2519726633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8132e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1da8069e90 tx=0x7f1da0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f237d8c21813b8ab server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:06.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 1 -- 172.21.15.150:0/2519726633 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da000e070 con 0x7f1da80a1950 2024-09-18T10:15:06.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: dump: 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dada7f700 5 monclient: authenticate success, global_id 54570 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dada7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 1 -- 172.21.15.150:0/2519726633 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da000be40 con 0x7f1da80a1950 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dada7f700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1dada7f700 4 set_mon_vals no callback set 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 1 -- 172.21.15.150:0/2519726633 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0012660 con 0x7f1da80a1950 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:06.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.333+0000 7f1da77fe700 10 monclient: dump: 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: shutdown 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:06.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- 172.21.15.150:0/2519726633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 msgr2=0x7f1da8132e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- 172.21.15.150:0/2519726633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8132e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1da8069e90 tx=0x7f1da0009780 comp rx=0 tx=0).stop 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- 172.21.15.150:0/2519726633 shutdown_connections 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- 172.21.15.150:0/2519726633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8132e70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- 172.21.15.150:0/2519726633 >> v2:172.21.15.150:3300/0 conn(0x7f1da80a2810 0x7f1da80a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- 172.21.15.150:0/2519726633 >> 172.21.15.150:0/2519726633 conn(0x7f1da80a60b0 msgr2=0x7f1da80a2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- 172.21.15.150:0/2519726633 shutdown_connections 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- 172.21.15.150:0/2519726633 wait complete. 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: build_initial_monmap 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: monmap: 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:06.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 Processor -- start 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- start start 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: init 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-c con 0x7f1da80a1550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-a con 0x7f1da80a1950 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): picked mon.noname-b con 0x7f1da80a2810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81324a0 con 0x7f1da80a1950 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da806efc0 con 0x7f1da80a2810 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da8131730 con 0x7f1da80a1550 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dada7f700 10 monclient(hunting): authenticate will time out at 1188.435225s 2024-09-18T10:15:06.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39632/0 (socket says 172.21.15.150:39632) 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 -- 172.21.15.150:0/656632267 learned_addr learned my addr 172.21.15.150:0/656632267 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): get_auth_request con 0x7f1da80a2810 auth_method 0 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 10 monclient(hunting): get_auth_request con 0x7f1da80a1550 auth_method 0 2024-09-18T10:15:06.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): get_auth_request con 0x7f1da80a1950 auth_method 0 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 msgr2=0x7f1da8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 msgr2=0x7f1da8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1daca7d700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:06.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1da7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient(hunting): handle_auth_done global_id 15781 payload 824 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 msgr2=0x7f1da8141850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 msgr2=0x7f1da8141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient: _finish_hunting 0 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 monclient: found mon.noname-b 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.337+0000 7f1dad27e700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0009a20 con 0x7f1da80a2810 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dad27e700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 msgr2=0x7f1da81422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:06.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dad27e700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1da8067c60 tx=0x7f1da0018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0026430 con 0x7f1da80a2810 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: dump: 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:06.357 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 5 monclient: authenticate success, global_id 15781 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 10 monclient: _renew_subs 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0021070 con 0x7f1da80a2810 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da8131150 con 0x7f1da80a2810 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0026800 con 0x7f1da80a2810 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 10 monclient: dump: 2024-09-18T10:15:06.358 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da6ffd700 4 set_mon_vals no callback set 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 10 monclient: _renew_subs 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:06.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da809c1f0 con 0x7f1da80a2810 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.341+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1da0023030 con 0x7f1da80a2810 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1dada7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1da4ff9700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 0x7f1d8c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1dada7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1dada7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da804bbf0 con 0x7f1da80a2810 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(172..172 src has 58..172) v4 ==== 9275+0+0 (secure 0 0 0) 0x7f1da004b760 con 0x7f1da80a2810 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1daca7d700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 0x7f1d8c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1daca7d700 10 monclient: get_auth_request con 0x7f1d8c035ac0 auth_method 0 2024-09-18T10:15:06.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.345+0000 7f1daca7d700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 0x7f1d8c037f70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1d94006fd0 tx=0x7f1d94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:06.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.349+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1da0011b70 con 0x7f1da80a2810 2024-09-18T10:15:06.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.349+0000 7f1da4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:06.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.349+0000 7f1da4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:06.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.517+0000 7f1dada7f700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:06.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.517+0000 7f1dada7f700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:06.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.517+0000 7f1dada7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:06.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:06.517+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1da8063ed0 con 0x7f1da80a2810 2024-09-18T10:15:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.217+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' removed v173) v1 ==== 148+0+0 (secure 0 0 0) 0x7f1da0011390 con 0x7f1da80a2810 2024-09-18T10:15:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.217+0000 7f1da4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.217+0000 7f1da4ff9700 10 monclient: _finish_command 2 = system:0 pool 'cool' removed 2024-09-18T10:15:07.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1dada7f700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1dada7f700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1dada7f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1da8063ed0 con 0x7f1da80a2810 2024-09-18T10:15:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1da4ff9700 1 -- 172.21.15.150:0/656632267 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}]=0 pool 'cool' does not exist v173) v1 ==== 155+0+0 (secure 0 0 0) 0x7f1da003d0c0 con 0x7f1da80a2810 2024-09-18T10:15:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1da4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cool", "pool2": "cool", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.325+0000 7f1da4ff9700 10 monclient: _finish_command 3 = system:0 pool 'cool' does not exist 2024-09-18T10:15:07.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 msgr2=0x7f1d8c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:07.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 0x7f1d8c037f70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1d94006fd0 tx=0x7f1d94008040 comp rx=0 tx=0).stop 2024-09-18T10:15:07.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 10 monclient: shutdown 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 msgr2=0x7f1da81422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1da8067c60 tx=0x7f1da0018000 comp rx=0 tx=0).stop 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 shutdown_connections 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d8c035ac0 0x7f1d8c037f70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da80a2810 0x7f1da81422d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1da80a1950 0x7f1da8141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 --2- 172.21.15.150:0/656632267 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1da80a1550 0x7f1da8141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 >> 172.21.15.150:0/656632267 conn(0x7f1da80a60b0 msgr2=0x7f1da806d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 shutdown_connections 2024-09-18T10:15:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.329+0000 7f1dada7f700 1 -- 172.21.15.150:0/656632267 wait complete. 2024-09-18T10:15:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'cool' does not exist 2024-09-18T10:15:07.343 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd pool rm cold cold --yes-i-really-really-mean-it 2024-09-18T10:15:07.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: get_monmap_and_config 2024-09-18T10:15:07.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: build_initial_monmap 2024-09-18T10:15:07.445 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: monmap: 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:07.444735+0000 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:07.444735+0000 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:07.446 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 Processor -- start 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 -- start start 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: init 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f06140a40d0 0x7f061406e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-a con 0x7f06140a40d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:07.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06140a1be0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-b con 0x7f061406ec50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061406ac90 con 0x7f06140a1be0 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0614069cd0 con 0x7f06140a40d0 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06140a44a0 con 0x7f061406ec50 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06197e2700 10 monclient(hunting): authenticate will time out at 1189.538484s 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f06140a40d0 0x7f061406e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f060bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f06140a1be0 0x7f0614132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:59864/0 (socket says 172.21.15.150:59864) 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f060bfff700 1 -- 172.21.15.150:0/3061496297 learned_addr learned my addr 172.21.15.150:0/3061496297 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): get_auth_request con 0x7f06140a40d0 auth_method 0 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 10 monclient(hunting): get_auth_request con 0x7f061406ec50 auth_method 0 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:07.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 1 -- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 msgr2=0x7f0614132870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 1 -- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 msgr2=0x7f0614132870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f06137fe700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient(hunting): handle_auth_done global_id 54576 payload 274 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 -- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 msgr2=0x7f0614132870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 -- 172.21.15.150:0/3061496297 >> v1:172.21.15.142:6789/0 conn(0x7f06140a1be0 legacy=0x7f0614132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 monclient: found mon.noname-a 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:07.452 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:07.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.441+0000 7f0612ffd700 1 -- 172.21.15.150:0/3061496297 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140a20b0 con 0x7f06140a40d0 2024-09-18T10:15:07.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f0612ffd700 1 -- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 msgr2=0x7f061406e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:07.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f0612ffd700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061406e710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f06040098e0 tx=0x7f06040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e407d8a3ee390d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:07.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 1 -- 172.21.15.150:0/3061496297 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604009e80 con 0x7f06140a40d0 2024-09-18T10:15:07.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: dump: 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.459 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 5 monclient: authenticate success, global_id 54576 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 1 -- 172.21.15.150:0/3061496297 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060400ba70 con 0x7f06140a40d0 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 4 set_mon_vals no callback set 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 1 -- 172.21.15.150:0/3061496297 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060400bdf0 con 0x7f06140a40d0 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06127fc700 10 monclient: dump: 2024-09-18T10:15:07.460 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 10 monclient: shutdown 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 -- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 msgr2=0x7f061406e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:07.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061406e710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f06040098e0 tx=0x7f06040091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 -- 172.21.15.150:0/3061496297 shutdown_connections 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f061406ec50 0x7f0614132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 --2- 172.21.15.150:0/3061496297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061406e710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 -- 172.21.15.150:0/3061496297 >> 172.21.15.150:0/3061496297 conn(0x7f06140716e0 msgr2=0x7f06140a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 -- 172.21.15.150:0/3061496297 shutdown_connections 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.445+0000 7f06197e2700 1 -- 172.21.15.150:0/3061496297 wait complete. 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient: build_initial_monmap 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient: monmap: 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:07.462 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 Processor -- start 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 -- start start 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient: init 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-b con 0x7f061406ec50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-c con 0x7f06140a1be0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): picked mon.noname-a con 0x7f06140a40d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06141352a0 con 0x7f06140a40d0 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061411d8b0 con 0x7f061406ec50 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061411f370 con 0x7f06140a1be0 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06197e2700 10 monclient(hunting): authenticate will time out at 1189.545417s 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39636/0 (socket says 172.21.15.150:39636) 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 -- 172.21.15.150:0/4018295927 learned_addr learned my addr 172.21.15.150:0/4018295927 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): get_auth_request con 0x7f061406ec50 auth_method 0 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): get_auth_request con 0x7f06140a40d0 auth_method 0 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 10 monclient(hunting): get_auth_request con 0x7f06140a1be0 auth_method 0 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:07.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 msgr2=0x7f061411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 msgr2=0x7f061411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f060bfff700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:07.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient(hunting): handle_auth_done global_id 15784 payload 824 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 msgr2=0x7f061411bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 msgr2=0x7f061411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061411c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f0612ffd700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0604009420 con 0x7f061406ec50 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.449+0000 7f06137fe700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061411c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f0612ffd700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 msgr2=0x7f0614132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f0612ffd700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f061406ae20 tx=0x7f0604016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:07.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604006af0 con 0x7f061406ec50 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: dump: 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: _finish_auth 0 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 5 monclient: authenticate success, global_id 15784 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _renew_subs 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061411f650 con 0x7f061406ec50 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0604020070 con 0x7f061406ec50 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604000d10 con 0x7f061406ec50 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _renew_subs 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f0611ffb700 4 set_mon_vals no callback set 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0614136be0 con 0x7f061406ec50 2024-09-18T10:15:07.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 10 monclient: dump: 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:07.471 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0614063ed0 con 0x7f061406ec50 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0604022030 con 0x7f061406ec50 2024-09-18T10:15:07.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 0x7f05f4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(173..173 src has 58..173) v4 ==== 8604+0+0 (secure 0 0 0) 0x7f060404af90 con 0x7f061406ec50 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060bfff700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 0x7f05f4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.453+0000 7f060bfff700 10 monclient: get_auth_request con 0x7f05f4035ac0 auth_method 0 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.457+0000 7f060bfff700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 0x7f05f4037f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f05fc007fd0 tx=0x7f05fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.461+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f060401b660 con 0x7f061406ec50 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.461+0000 7f060affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:07.473 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.461+0000 7f060affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:07.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.621+0000 7f06197e2700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.621+0000 7f06197e2700 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:07.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.621+0000 7f06197e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:07.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:07.621+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0614063ed0 con 0x7f061406ec50 2024-09-18T10:15:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.221+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' removed v174) v1 ==== 148+0+0 (secure 0 0 0) 0x7f0604023e60 con 0x7f061406ec50 2024-09-18T10:15:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.221+0000 7f060affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.221+0000 7f060affd700 10 monclient: _finish_command 2 = system:0 pool 'cold' removed 2024-09-18T10:15:08.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:15:08.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f06197e2700 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:08.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f06197e2700 10 monclient: _send_command 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:08.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f06197e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0614063ed0 con 0x7f061406ec50 2024-09-18T10:15:08.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f060affd700 1 -- 172.21.15.150:0/4018295927 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}]=0 pool 'cold' does not exist v174) v1 ==== 155+0+0 (secure 0 0 0) 0x7f060401a3d0 con 0x7f061406ec50 2024-09-18T10:15:08.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f060affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd pool rm", "pool": "cold", "pool2": "cold", "yes_i_really_really_mean_it": true}] 2024-09-18T10:15:08.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.325+0000 7f060affd700 10 monclient: _finish_command 3 = system:0 pool 'cold' does not exist 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 msgr2=0x7f05f4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 0x7f05f4037f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f05fc007fd0 tx=0x7f05fc009040 comp rx=0 tx=0).stop 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 10 monclient: shutdown 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 msgr2=0x7f0614132250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f061406ae20 tx=0x7f0604016000 comp rx=0 tx=0).stop 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 shutdown_connections 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05f4035ac0 0x7f05f4037f70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06140a40d0 0x7f061411c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06140a1be0 0x7f061411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 --2- 172.21.15.150:0/4018295927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061406ec50 0x7f0614132250 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 >> 172.21.15.150:0/4018295927 conn(0x7f06140716e0 msgr2=0x7f06141354b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 shutdown_connections 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.329+0000 7f06197e2700 1 -- 172.21.15.150:0/4018295927 wait complete. 2024-09-18T10:15:08.335 INFO:tasks.workunit.client.0.smithi150.stderr:pool 'cold' does not exist 2024-09-18T10:15:08.344 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T10:15:08.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: get_monmap_and_config 2024-09-18T10:15:08.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: build_initial_monmap 2024-09-18T10:15:08.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: monmap: 2024-09-18T10:15:08.371 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:08.370373+0000 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:08.370373+0000 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:08.372 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:08.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 Processor -- start 2024-09-18T10:15:08.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 -- start start 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: init 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01e009d920 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01e006f660 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01e01375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-b con 0x7f01e00973c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e009b0b0 con 0x7f01e006f660 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e0095ad0 con 0x7f01e009d920 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009a7e0 con 0x7f01e00973c0 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:08.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.365+0000 7f01e530c700 10 monclient(hunting): authenticate will time out at 1197.295844s 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01df7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f01e006f660 0x7f01e0096cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44644/0 (socket says 172.21.15.142:44644) 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01df7fe700 1 -- 172.21.15.142:0/232675937 learned_addr learned my addr 172.21.15.142:0/232675937 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01e01375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request con 0x7f01e00973c0 auth_method 0 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 msgr2=0x7f01e01375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 msgr2=0x7f01e01375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01e01375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:08.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01e01375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01dffff700 1 -- 172.21.15.142:0/232675937 >> v1:172.21.15.150:6789/0 conn(0x7f01e009d920 legacy=0x7f01e00953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01df7fe700 1 -- 172.21.15.142:0/232675937 >> v1:172.21.15.142:6789/0 conn(0x7f01e006f660 legacy=0x7f01e0096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879016748 0 0) 0x7f01e0095ad0 con 0x7f01e009d920 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): my global_id is 15787 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c8003570 con 0x7f01e009d920 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813384293 0 0) 0x7f01e009b0b0 con 0x7f01e006f660 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): my global_id is 54582 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e0095ad0 con 0x7f01e006f660 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1263458180 0 0) 0x7f01e0095ad0 con 0x7f01e006f660 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 msgr2=0x7f01e01375a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01e01375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 >> v1:172.21.15.150:6789/0 conn(0x7f01e009d920 legacy=0x7f01e00953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e009aaf0 con 0x7f01e006f660 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:08.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e530c700 5 monclient: authenticate success, global_id 54582 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e530c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f01d4001780 con 0x7f01e006f660 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: dump: 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.383 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 >> v1:172.21.15.142:6789/0 conn(0x7f01e006f660 legacy=0x7f01e0096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 0x7f01c8004140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): picked mon.b con 0x7f01c8003d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01c8004810 0x7f01c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): picked mon.a con 0x7f01c8004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01c800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): picked mon.c con 0x7f01e00973c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0001630 con 0x7f01c8004810 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e009a7e0 con 0x7f01c8003d70 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c800bfa0 con 0x7f01e00973c0 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01deffd700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e530c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01df7fe700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01c8004810 0x7f01c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01dffff700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 0x7f01c8004140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.369+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01c800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): get_auth_request con 0x7f01c8003d70 auth_method 0 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request con 0x7f01e00973c0 auth_method 0 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): get_auth_request con 0x7f01c8004810 auth_method 0 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient(hunting): handle_auth_done global_id 54582 payload 293 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 msgr2=0x7f01c800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01c800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01c8004810 msgr2=0x7f01c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01c8004810 0x7f01c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 monclient: found mon.b 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 -- 172.21.15.142:0/232675937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0095ad0 con 0x7f01c8003d70 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 10 monclient: _finish_auth 0 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e00973c0 0x7f01c800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 msgr2=0x7f01c8004140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01dffff700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 0x7f01c8004140 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f01d00019d0 tx=0x7f01d0007a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790537f8a72aac4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d000cd20 con 0x7f01c8003d70 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:08.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: dump: 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:08.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d00070d0 con 0x7f01c8003d70 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 4 set_mon_vals no callback set 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 1 -- 172.21.15.142:0/232675937 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d000cea0 con 0x7f01c8003d70 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01deffd700 10 monclient: dump: 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.390 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 10 monclient: shutdown 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 1 -- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 msgr2=0x7f01c8004140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e530c700 1 --2- 172.21.15.142:0/232675937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c8003d70 0x7f01c8004140 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f01d00019d0 tx=0x7f01d0007a50 comp rx=0 tx=0).stop 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.373+0000 7f01e4b0b700 1 -- 172.21.15.142:0/232675937 reap_dead start 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- 172.21.15.142:0/232675937 shutdown_connections 2024-09-18T10:15:08.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- 172.21.15.142:0/232675937 >> 172.21.15.142:0/232675937 conn(0x7f01e00a1fe0 msgr2=0x7f01e0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:08.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- 172.21.15.142:0/232675937 shutdown_connections 2024-09-18T10:15:08.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- 172.21.15.142:0/232675937 wait complete. 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient: build_initial_monmap 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient: monmap: 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.397 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 Processor -- start 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- start start 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient: init 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-c con 0x7f01e013d630 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 0x7f01e0089bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-a con 0x7f01e013df40 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:08.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 0x7f01e008a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): picked mon.noname-b con 0x7f01e008a0f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00a0ac0 con 0x7f01e013df40 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e008e4b0 con 0x7f01e008a0f0 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e008bc50 con 0x7f01e013d630 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01e530c700 10 monclient(hunting): authenticate will time out at 1197.307655s 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01df7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 0x7f01e0089bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46700/0 (socket says 172.21.15.142:46700) 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 1 -- 172.21.15.142:0/4233065021 learned_addr learned my addr 172.21.15.142:0/4233065021 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 10 monclient(hunting): get_auth_request con 0x7f01e013d630 auth_method 0 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01df7fe700 10 monclient(hunting): get_auth_request con 0x7f01e013df40 auth_method 0 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01df7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.377+0000 7f01df7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 0x7f01e008a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request con 0x7f01e008a0f0 auth_method 0 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01dffff700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 msgr2=0x7f01e013da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01dffff700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 msgr2=0x7f01e013da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01dffff700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01dffff700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:08.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient(hunting): handle_auth_done global_id 54585 payload 824 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 msgr2=0x7f01e013da00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 msgr2=0x7f01e008a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 0x7f01e008a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 monclient: found mon.noname-a 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 -- 172.21.15.142:0/4233065021 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e008b7b0 con 0x7f01e013df40 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e4b0b700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 0x7f01e008a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 msgr2=0x7f01e0089bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01df7fe700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 0x7f01e0089bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f01d4001960 tx=0x7f01d4007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d40117a0 con 0x7f01e013df40 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: dump: 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.403 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: _finish_auth 0 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 5 monclient: authenticate success, global_id 54585 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d4011900 con 0x7f01e013df40 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d4011cc0 con 0x7f01e013df40 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 10 monclient: _renew_subs 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e008d200 con 0x7f01e013df40 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01de7fc700 4 set_mon_vals no callback set 2024-09-18T10:15:08.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01c7fff700 10 monclient: dump: 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 10 monclient: _renew_subs 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.381+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e008c310 con 0x7f01e013df40 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01e530c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01e530c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01e530c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e006da20 con 0x7f01e013df40 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f01d4013600 con 0x7f01e013df40 2024-09-18T10:15:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01c7fff700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 0x7f01cc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(174..174 src has 58..174) v4 ==== 7797+0+0 (secure 0 0 0) 0x7f01d4049060 con 0x7f01e013df40 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01dffff700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 0x7f01cc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.385+0000 7f01dffff700 10 monclient: get_auth_request con 0x7f01cc035ac0 auth_method 0 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.389+0000 7f01dffff700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 0x7f01cc037f70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f01d000c3d0 tx=0x7f01d0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.393+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f01d401aaf0 con 0x7f01e013df40 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.393+0000 7f01c7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.393+0000 7f01c7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:08.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: get_monmap_and_config 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: build_initial_monmap 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: monmap: 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:08.441564+0000 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:08.441564+0000 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:08.443 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 Processor -- start 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 -- start start 2024-09-18T10:15:08.444 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: init 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f888009cf90 0x7f88800a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-c con 0x7f888009cf90 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f888009e480 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:08.447 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-b con 0x7f88800990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888009e060 con 0x7f888009e480 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88800a66c0 con 0x7f888009cf90 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888009d750 con 0x7f88800990a0 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f88867a1700 10 monclient(hunting): authenticate will time out at 1190.535016s 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f888009cf90 0x7f88800a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.448 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f888009cf90 0x7f88800a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39638/0 (socket says 172.21.15.150:39638) 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:55214/0 (socket says 172.21.15.150:55214) 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 1 -- 172.21.15.150:0/821884619 learned_addr learned my addr 172.21.15.150:0/821884619 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 10 monclient(hunting): get_auth_request con 0x7f888009cf90 auth_method 0 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f887ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 10 monclient(hunting): get_auth_request con 0x7f88800990a0 auth_method 0 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.449 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 -- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 -- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.437+0000 7f8884d3e700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887f7fe700 1 -- 172.21.15.150:0/821884619 >> v1:172.21.15.150:6789/0 conn(0x7f888009e480 legacy=0x7f88800a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient(hunting): handle_auth_done global_id 15793 payload 274 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 1 -- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 1 -- 172.21.15.150:0/821884619 >> v1:172.21.15.150:6789/0 conn(0x7f888009e480 legacy=0x7f88800a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:08.450 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 1 monclient: found mon.noname-c 2024-09-18T10:15:08.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:08.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887ffff700 1 -- 172.21.15.150:0/821884619 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888009d360 con 0x7f888009cf90 2024-09-18T10:15:08.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887effd700 1 -- 172.21.15.150:0/821884619 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643530289 0 0) 0x7f888009e060 con 0x7f888009e480 2024-09-18T10:15:08.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.441+0000 7f887effd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887ffff700 1 -- 172.21.15.150:0/821884619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009cf90 msgr2=0x7f88800a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887ffff700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009cf90 0x7f88800a0290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8874009a30 tx=0x7f8874009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324780129f3533b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 1 -- 172.21.15.150:0/821884619 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400ba70 con 0x7f888009cf90 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: dump: 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.453 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: _finish_auth 0 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 1 -- 172.21.15.150:0/821884619 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400bbd0 con 0x7f888009cf90 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 5 monclient: authenticate success, global_id 15793 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 1 -- 172.21.15.150:0/821884619 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874011420 con 0x7f888009cf90 2024-09-18T10:15:08.454 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:08.455 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f887effd700 10 monclient: dump: 2024-09-18T10:15:08.455 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:08.455 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.455 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 4 set_mon_vals no callback set 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 10 monclient: shutdown 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 -- 172.21.15.150:0/821884619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009cf90 msgr2=0x7f88800a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.456 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009cf90 0x7f88800a0290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8874009a30 tx=0x7f8874009250 comp rx=0 tx=0).stop 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 -- 172.21.15.150:0/821884619 shutdown_connections 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 --2- 172.21.15.150:0/821884619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009cf90 0x7f88800a0290 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 -- 172.21.15.150:0/821884619 >> 172.21.15.150:0/821884619 conn(0x7f88800a58a0 msgr2=0x7f88800a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 -- 172.21.15.150:0/821884619 shutdown_connections 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 1 -- 172.21.15.150:0/821884619 wait complete. 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 10 monclient: build_initial_monmap 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.445+0000 7f88867a1700 10 monclient: monmap: 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:08.457 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 Processor -- start 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 -- start start 2024-09-18T10:15:08.458 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient: init 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-c con 0x7f88800990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 0x7f8880146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-a con 0x7f888009cf90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): picked mon.noname-b con 0x7f888009e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880097640 con 0x7f888009cf90 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.463 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880088990 con 0x7f888009e480 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8880097130 con 0x7f88800990a0 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f88867a1700 10 monclient(hunting): authenticate will time out at 1190.545007s 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39646/0 (socket says 172.21.15.150:39646) 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 -- 172.21.15.150:0/1124783098 learned_addr learned my addr 172.21.15.150:0/1124783098 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): get_auth_request con 0x7f888009e480 auth_method 0 2024-09-18T10:15:08.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 0x7f8880146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 10 monclient(hunting): get_auth_request con 0x7f88800990a0 auth_method 0 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 10 monclient(hunting): get_auth_request con 0x7f888009cf90 auth_method 0 2024-09-18T10:15:08.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient(hunting): handle_auth_done global_id 15799 payload 824 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 msgr2=0x7f8880145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887ffff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 msgr2=0x7f8880146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 0x7f8880146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient: _finish_hunting 0 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 monclient: found mon.noname-b 2024-09-18T10:15:08.466 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88740094f0 con 0x7f888009e480 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f887f7fe700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 0x7f8880146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 msgr2=0x7f8880095630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.449+0000 7f8884d3e700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8878007ef0 tx=0x7f8878009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887800ee80 con 0x7f888009e480 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: dump: 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.467 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: _finish_auth 0 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 5 monclient: authenticate success, global_id 15799 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8878014420 con 0x7f888009e480 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878014820 con 0x7f888009e480 2024-09-18T10:15:08.468 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _renew_subs 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888009bf70 con 0x7f888009e480 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f887dffb700 4 set_mon_vals no callback set 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 10 monclient: dump: 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:08.469 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _renew_subs 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8880146740 con 0x7f888009e480 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.470 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8878014c50 con 0x7f888009e480 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8880146960 con 0x7f888009e480 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 0x7f886c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(174..174 src has 58..174) v4 ==== 7797+0+0 (secure 0 0 0) 0x7f8878049750 con 0x7f888009e480 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f887ffff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 0x7f886c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.453+0000 7f887ffff700 10 monclient: get_auth_request con 0x7f886c035ac0 auth_method 0 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.457+0000 7f887ffff700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 0x7f886c037f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8874009a30 tx=0x7f8874017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:08.471 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.461+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f887801ecb0 con 0x7f888009e480 2024-09-18T10:15:08.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.461+0000 7f8867fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:08.472 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.461+0000 7f8867fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.621+0000 7f01e530c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:15:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.621+0000 7f01e530c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.621+0000 7f01e530c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.621+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f01e0072a80 con 0x7f01e013df40 2024-09-18T10:15:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01c7fff700 1 -- 172.21.15.142:0/4233065021 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f01d401a310 con 0x7f01e013df40 2024-09-18T10:15:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01c7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01c7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:08.630 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:15:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 msgr2=0x7f01cc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 0x7f01cc037f70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f01d000c3d0 tx=0x7f01d0017000 comp rx=0 tx=0).stop 2024-09-18T10:15:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 10 monclient: shutdown 2024-09-18T10:15:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:08.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 msgr2=0x7f01e0089bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 0x7f01e0089bb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f01d4001960 tx=0x7f01d4007a80 comp rx=0 tx=0).stop 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 shutdown_connections 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f01cc035ac0 0x7f01cc037f70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e008a0f0 0x7f01e008a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01e013df40 0x7f01e0089bb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 --2- 172.21.15.142:0/4233065021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01e013d630 0x7f01e013da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 >> 172.21.15.142:0/4233065021 conn(0x7f01e00a1fe0 msgr2=0x7f01e00a0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 shutdown_connections 2024-09-18T10:15:08.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:08.625+0000 7f01e530c700 1 -- 172.21.15.142:0/4233065021 wait complete. 2024-09-18T10:15:08.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.633+0000 7f88867a1700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:08.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.633+0000 7f88867a1700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:08.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.633+0000 7f88867a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:08.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:08.633+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f88800a0ee0 con 0x7f888009e480 2024-09-18T10:15:09.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.241+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v175) v1 ==== 78+0+0 (secure 0 0 0) 0x7f887801b310 con 0x7f888009e480 2024-09-18T10:15:09.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.241+0000 7f8867fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:09.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.241+0000 7f8867fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:09.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.357+0000 7f88867a1700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.357+0000 7f88867a1700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.357+0000 7f88867a1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:09.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:09.357+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f8880063ed0 con 0x7f888009e480 2024-09-18T10:15:10.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.245+0000 7f8867fff700 1 -- 172.21.15.150:0/1124783098 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v176) v1 ==== 78+0+0 (secure 0 0 0) 0x7f88780122e0 con 0x7f888009e480 2024-09-18T10:15:10.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.245+0000 7f8867fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:10.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.245+0000 7f8867fff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 msgr2=0x7f886c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 0x7f886c037f70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8874009a30 tx=0x7f8874017000 comp rx=0 tx=0).stop 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 10 monclient: shutdown 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 msgr2=0x7f8880095630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:10.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.249+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8878007ef0 tx=0x7f8878009870 comp rx=0 tx=0).stop 2024-09-18T10:15:10.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 shutdown_connections 2024-09-18T10:15:10.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f886c035ac0 0x7f886c037f70 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8874009a30 tx=0x7f8874017000 comp rx=0 tx=0).stop 2024-09-18T10:15:10.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f888009e480 0x7f8880095630 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f888009cf90 0x7f8880146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 --2- 172.21.15.150:0/1124783098 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88800990a0 0x7f8880145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.253+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 >> 172.21.15.150:0/1124783098 conn(0x7f88800a58a0 msgr2=0x7f888009c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:10.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.257+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 shutdown_connections 2024-09-18T10:15:10.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.257+0000 7f88867a1700 1 -- 172.21.15.150:0/1124783098 wait complete. 2024-09-18T10:15:10.275 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: get_monmap_and_config 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: build_initial_monmap 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: monmap: 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:10.372125+0000 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:10.372125+0000 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:10.373 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:10.374 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 Processor -- start 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 -- start start 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: init 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d4c09e6d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d4c0a09a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d4c0a0da0 0x7f9d4c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-a con 0x7f9d4c0a0da0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c09e220 con 0x7f9d4c0a09a0 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c09eaa0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:10.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c09ee30 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d50e73700 10 monclient(hunting): authenticate will time out at 1192.465881s 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4a59c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9d4c09e6d0 0x7f9d4c0a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46028/0 (socket says 172.21.15.150:46028) 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4a59c700 1 -- 172.21.15.150:0/3554002117 learned_addr learned my addr 172.21.15.150:0/3554002117 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 1 --2- 172.21.15.150:0/3554002117 >> v2:172.21.15.142:3300/0 conn(0x7f9d4c0a0da0 0x7f9d4c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): get_auth_request con 0x7f9d4c0a0da0 auth_method 0 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:10.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_done global_id 54591 payload 274 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 1 -- 172.21.15.150:0/3554002117 >> v1:172.21.15.150:6789/0 conn(0x7f9d4c09e6d0 legacy=0x7f9d4c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 1 -- 172.21.15.150:0/3554002117 >> v1:172.21.15.142:6789/0 conn(0x7f9d4c0a09a0 legacy=0x7f9d4c137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 1 monclient: found mon.noname-a 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:10.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.369+0000 7f9d4ad9d700 1 -- 172.21.15.150:0/3554002117 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c09d930 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4ad9d700 1 -- 172.21.15.150:0/3554002117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 msgr2=0x7f9d4c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4ad9d700 1 --2- 172.21.15.150:0/3554002117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c13b150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009e60 tx=0x7f9d44009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e06d72ace38c63 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 1 -- 172.21.15.150:0/3554002117 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4400e070 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: dump: 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:10.385 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: _finish_auth 0 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 5 monclient: authenticate success, global_id 54591 2024-09-18T10:15:10.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 1 -- 172.21.15.150:0/3554002117 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d44012430 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 4 set_mon_vals no callback set 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 1 -- 172.21.15.150:0/3554002117 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440127d0 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d4959a700 10 monclient: dump: 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.387 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 10 monclient: shutdown 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 -- 172.21.15.150:0/3554002117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 msgr2=0x7f9d4c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 --2- 172.21.15.150:0/3554002117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c13b150 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009e60 tx=0x7f9d44009780 comp rx=0 tx=0).stop 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 -- 172.21.15.150:0/3554002117 shutdown_connections 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 --2- 172.21.15.150:0/3554002117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c13b150 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 -- 172.21.15.150:0/3554002117 >> 172.21.15.150:0/3554002117 conn(0x7f9d4c0a58a0 msgr2=0x7f9d4c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 -- 172.21.15.150:0/3554002117 shutdown_connections 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 1 -- 172.21.15.150:0/3554002117 wait complete. 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 10 monclient: build_initial_monmap 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.373+0000 7f9d50e73700 10 monclient: monmap: 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:10.389 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 Processor -- start 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 -- start start 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient: init 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-b con 0x7f9d4c09e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-c con 0x7f9d4c0a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c08eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): picked mon.noname-a con 0x7f9d4c0a0da0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:10.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0924e0 con 0x7f9d4c0a0da0 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c09bfa0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c091660 con 0x7f9d4c0a09a0 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d50e73700 10 monclient(hunting): authenticate will time out at 1192.473504s 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53174/0 (socket says 172.21.15.150:53174) 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 -- 172.21.15.150:0/4083913356 learned_addr learned my addr 172.21.15.150:0/4083913356 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): get_auth_request con 0x7f9d4c09e6d0 auth_method 0 2024-09-18T10:15:10.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 10 monclient(hunting): get_auth_request con 0x7f9d4c0a09a0 auth_method 0 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c08eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:10.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): get_auth_request con 0x7f9d4c0a0da0 auth_method 0 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 msgr2=0x7f9d4c08e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 msgr2=0x7f9d4c08e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d49d9b700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient(hunting): handle_auth_done global_id 15802 payload 824 2024-09-18T10:15:10.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 msgr2=0x7f9d4c08e5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 msgr2=0x7f9d4c08eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c08eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 monclient: found mon.noname-b 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:10.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.377+0000 7f9d4a59c700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44009a20 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d4a59c700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 msgr2=0x7f9d4c08e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d4a59c700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c5e0 tx=0x7f9d34004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34005d40 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: dump: 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.397 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 5 monclient: authenticate success, global_id 15802 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _renew_subs 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c099440 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340070d0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34007490 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 10 monclient: dump: 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:10.399 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:10.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d48d99700 4 set_mon_vals no callback set 2024-09-18T10:15:10.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _renew_subs 2024-09-18T10:15:10.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:10.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c0913b0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c09dbf0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9d3401b3b0 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 0x7f9d2c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.381+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(176..176 src has 58..176) v4 ==== 7345+0+0 (secure 0 0 0) 0x7f9d34048360 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.385+0000 7f9d49d9b700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 0x7f9d2c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.385+0000 7f9d49d9b700 10 monclient: get_auth_request con 0x7f9d2c035ac0 auth_method 0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.385+0000 7f9d49d9b700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 0x7f9d2c037f70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c007cb0 tx=0x7f9d3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.389+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9d3400f820 con 0x7f9d4c09e6d0 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.389+0000 7f9d427fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:10.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.389+0000 7f9d427fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.565+0000 7f9d50e73700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.565+0000 7f9d50e73700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.565+0000 7f9d50e73700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:10.569 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:10.565+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f9d4c063ed0 con 0x7f9d4c09e6d0 2024-09-18T10:15:11.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:15:11.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.661+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v177) v1 ==== 82+0+0 (secure 0 0 0) 0x7f9d34017070 con 0x7f9d4c09e6d0 2024-09-18T10:15:11.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.661+0000 7f9d427fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:11.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.661+0000 7f9d427fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:11.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: get_monmap_and_config 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: build_initial_monmap 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: monmap: 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:11.749661+0000 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:11.749661+0000 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:11.751 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:11.752 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:11.752 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 Processor -- start 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 -- start start 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: init 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f69700a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-b con 0x7f69700a3ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f697006e620 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:11.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f697006e9f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697006aa60 con 0x7f697006e620 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700a42c0 con 0x7f697006e9f0 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697006b020 con 0x7f69700a3ef0 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f69760ee700 10 monclient(hunting): authenticate will time out at 1200.675276s 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696effd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f697006e620 0x7f6970132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44656/0 (socket says 172.21.15.142:44656) 2024-09-18T10:15:11.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696effd700 1 -- 172.21.15.142:0/2481192238 learned_addr learned my addr 172.21.15.142:0/2481192238 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f69700a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 10 monclient(hunting): get_auth_request con 0x7f69700a3ef0 auth_method 0 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 msgr2=0x7f69700a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 msgr2=0x7f69700a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f69700a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.745+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f69700a15e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:11.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 1 -- 172.21.15.142:0/2481192238 >> v1:172.21.15.142:6789/0 conn(0x7f697006e620 legacy=0x7f6970132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084178340 0 0) 0x7f697006aa60 con 0x7f697006e620 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): my global_id is 54597 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958003570 con 0x7f697006e620 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 1 -- 172.21.15.142:0/2481192238 >> v1:172.21.15.150:6789/0 conn(0x7f697006e9f0 legacy=0x7f6970132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632044873 0 0) 0x7f6958003570 con 0x7f697006e620 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 msgr2=0x7f69700a15e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f69700a15e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 >> v1:172.21.15.150:6789/0 conn(0x7f697006e9f0 legacy=0x7f6970132740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:15:11.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697006ee50 con 0x7f697006e620 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f69760ee700 5 monclient: authenticate success, global_id 54597 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f6960001860 con 0x7f697006e620 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f69760ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: dump: 2024-09-18T10:15:11.765 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 >> v1:172.21.15.142:6789/0 conn(0x7f697006e620 legacy=0x7f6970132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f6958003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): picked mon.c con 0x7f69700a3ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580041e0 0x7f6958004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): picked mon.b con 0x7f69580041e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6958004ea0 0x7f695800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): picked mon.a con 0x7f6958004ea0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697006b020 con 0x7f6958004ea0 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6958005610 con 0x7f69580041e0 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695800bfa0 con 0x7f69700a3ef0 2024-09-18T10:15:11.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f69760ee700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6958004ea0 0x7f695800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f6958003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580041e0 0x7f6958004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): get_auth_request con 0x7f6958004ea0 auth_method 0 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): get_auth_request con 0x7f69580041e0 auth_method 0 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): get_auth_request con 0x7f69700a3ef0 auth_method 0 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient(hunting): handle_auth_done global_id 54597 payload 293 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580041e0 msgr2=0x7f6958004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580041e0 0x7f6958004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6958004ea0 msgr2=0x7f695800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6958004ea0 0x7f695800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 monclient: found mon.c 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958003570 con 0x7f69700a3ef0 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696ffff700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6958004ea0 0x7f695800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.749+0000 7f696effd700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69580041e0 0x7f6958004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696f7fe700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 msgr2=0x7f6958003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696f7fe700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f6958003b60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f696400bb40 tx=0x7f6964008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d2b97b4dd6eafc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964018070 con 0x7f69700a3ef0 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: dump: 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400d7e0 con 0x7f69700a3ef0 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 4 set_mon_vals no callback set 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 1 -- 172.21.15.142:0/2481192238 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400db80 con 0x7f69700a3ef0 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696e7fc700 10 monclient: dump: 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 10 monclient: shutdown 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 -- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 msgr2=0x7f6958003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 --2- 172.21.15.142:0/2481192238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69700a3ef0 0x7f6958003b60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f696400bb40 tx=0x7f6964008040 comp rx=0 tx=0).stop 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f696ffff700 1 -- 172.21.15.142:0/2481192238 reap_dead start 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 -- 172.21.15.142:0/2481192238 shutdown_connections 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 -- 172.21.15.142:0/2481192238 >> 172.21.15.142:0/2481192238 conn(0x7f6970071460 msgr2=0x7f69700a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 -- 172.21.15.142:0/2481192238 shutdown_connections 2024-09-18T10:15:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 1 -- 172.21.15.142:0/2481192238 wait complete. 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 10 monclient: build_initial_monmap 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.753+0000 7f69760ee700 10 monclient: monmap: 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 Processor -- start 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 -- start start 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient: init 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69700a3ef0 0x7f6970141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-b con 0x7f69700a3ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-a con 0x7f6970141aa0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): picked mon.noname-c con 0x7f69701319f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970131e60 con 0x7f6970141aa0 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970069ad0 con 0x7f69700a3ef0 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697009f660 con 0x7f69701319f0 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f69760ee700 10 monclient(hunting): authenticate will time out at 1200.685135s 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42524/0 (socket says 172.21.15.142:42524) 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46750/0 (socket says 172.21.15.142:46750) 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 -- 172.21.15.142:0/1907338353 learned_addr learned my addr 172.21.15.142:0/1907338353 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): get_auth_request con 0x7f6970141aa0 auth_method 0 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 10 monclient(hunting): get_auth_request con 0x7f69701319f0 auth_method 0 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69700a3ef0 0x7f6970141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 msgr2=0x7f697011e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 msgr2=0x7f697011e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): get_auth_request con 0x7f69700a3ef0 auth_method 0 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696ffff700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient(hunting): handle_auth_done global_id 54600 payload 824 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 msgr2=0x7f697011e0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69700a3ef0 msgr2=0x7f6970141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69700a3ef0 0x7f6970141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 monclient: found mon.noname-a 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.757+0000 7f696effd700 1 -- 172.21.15.142:0/1907338353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697011fc10 con 0x7f6970141aa0 2024-09-18T10:15:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d50e73700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d50e73700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:11.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d50e73700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:11.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f9d4c0008d0 con 0x7f9d4c09e6d0 2024-09-18T10:15:11.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d427fc700 1 -- 172.21.15.150:0/4083913356 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v177) v1 ==== 115+0+0 (secure 0 0 0) 0x7f9d3400f820 con 0x7f9d4c09e6d0 2024-09-18T10:15:11.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d427fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:11.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.773+0000 7f9d427fc700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T10:15:11.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f696effd700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 msgr2=0x7f69701314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f696effd700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f69600073c0 tx=0x7f6960007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000cc40 con 0x7f6970141aa0 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: dump: 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000cda0 con 0x7f6970141aa0 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 5 monclient: authenticate success, global_id 54600 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _renew_subs 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6970121600 con 0x7f6970141aa0 2024-09-18T10:15:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600135c0 con 0x7f6970141aa0 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 10 monclient: dump: 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _renew_subs 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f696dffb700 4 set_mon_vals no callback set 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697011e5f0 con 0x7f6970141aa0 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f69600139f0 con 0x7f6970141aa0 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69700696a0 con 0x7f6970141aa0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 0x7f695c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(177..177 src has 58..177) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f6960049000 con 0x7f6970141aa0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f696f7fe700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 0x7f695c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.761+0000 7f696f7fe700 10 monclient: get_auth_request con 0x7f695c035af0 auth_method 0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.765+0000 7f696f7fe700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 0x7f695c037fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f696400b5e0 tx=0x7f696401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.773+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f696001c4a0 con 0x7f6970141aa0 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.773+0000 7f69577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:11.773+0000 7f69577fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 msgr2=0x7f9d2c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 0x7f9d2c037f70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c007cb0 tx=0x7f9d3c009040 comp rx=0 tx=0).stop 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 10 monclient: shutdown 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 msgr2=0x7f9d4c08e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400c5e0 tx=0x7f9d34004a80 comp rx=0 tx=0).stop 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 shutdown_connections 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d2c035ac0 0x7f9d2c037f70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4c0a0da0 0x7f9d4c08eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d4c0a09a0 0x7f9d4c08e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 --2- 172.21.15.150:0/4083913356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d4c09e6d0 0x7f9d4c08e080 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 >> 172.21.15.150:0/4083913356 conn(0x7f9d4c0a58a0 msgr2=0x7f9d4c09a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 shutdown_connections 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.777+0000 7f9d50e73700 1 -- 172.21.15.150:0/4083913356 wait complete. 2024-09-18T10:15:11.791 INFO:tasks.workunit.client.0.smithi150.stderr:compat weight-set already created 2024-09-18T10:15:11.793 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket fooo host 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: get_monmap_and_config 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: build_initial_monmap 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: monmap: 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:11.900232+0000 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:11.900232+0000 2024-09-18T10:15:11.901 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:11.902 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:11.903 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 Processor -- start 2024-09-18T10:15:11.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 -- start start 2024-09-18T10:15:11.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: init 2024-09-18T10:15:11.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 0x7f546c09ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): picked mon.noname-c con 0x7f546c06ed20 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f546c06f0f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f546c09ce30 0x7f546c136540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): picked mon.noname-a con 0x7f546c09ce30 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546c098620 con 0x7f546c06f0f0 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c09e670 con 0x7f546c09ce30 2024-09-18T10:15:11.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546c09e430 con 0x7f546c06ed20 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5474500700 10 monclient(hunting): authenticate will time out at 1193.994943s 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 0x7f546c09ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 0x7f546c09ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53188/0 (socket says 172.21.15.150:53188) 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 1 -- 172.21.15.150:0/4091900012 learned_addr learned my addr 172.21.15.150:0/4091900012 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 10 monclient(hunting): get_auth_request con 0x7f546c06ed20 auth_method 0 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f547229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5471a9b700 1 --1- 172.21.15.150:0/4091900012 >> v1:172.21.15.150:6789/0 conn(0x7f546c06f0f0 0x7f546c09c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46032/0 (socket says 172.21.15.150:46032) 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 --2- 172.21.15.150:0/4091900012 >> v2:172.21.15.142:3300/0 conn(0x7f546c09ce30 0x7f546c136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): get_auth_request con 0x7f546c09ce30 auth_method 0 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient(hunting): handle_auth_done global_id 54603 payload 274 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 -- 172.21.15.150:0/4091900012 >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 msgr2=0x7f546c09ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 --2- 172.21.15.150:0/4091900012 >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 0x7f546c09ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 -- 172.21.15.150:0/4091900012 >> v1:172.21.15.150:6789/0 conn(0x7f546c06f0f0 legacy=0x7f546c09c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 monclient: found mon.noname-a 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:11.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.897+0000 7f5472a9d700 1 -- 172.21.15.150:0/4091900012 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c097d10 con 0x7f546c09ce30 2024-09-18T10:15:11.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5472a9d700 1 -- 172.21.15.150:0/4091900012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 msgr2=0x7f546c136540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5472a9d700 1 --2- 172.21.15.150:0/4091900012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c136540 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f546c096fc0 tx=0x7f5464009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e2c2646b06f81e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 1 -- 172.21.15.150:0/4091900012 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546400e070 con 0x7f546c09ce30 2024-09-18T10:15:11.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: dump: 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.917 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: _finish_auth 0 2024-09-18T10:15:11.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 1 -- 172.21.15.150:0/4091900012 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546400be40 con 0x7f546c09ce30 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 5 monclient: authenticate success, global_id 54603 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 4 set_mon_vals no callback set 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 1 -- 172.21.15.150:0/4091900012 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464012660 con 0x7f546c09ce30 2024-09-18T10:15:11.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f547129a700 10 monclient: dump: 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.920 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 10 monclient: shutdown 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 -- 172.21.15.150:0/4091900012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 msgr2=0x7f546c136540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 --2- 172.21.15.150:0/4091900012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c136540 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f546c096fc0 tx=0x7f5464009780 comp rx=0 tx=0).stop 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 -- 172.21.15.150:0/4091900012 shutdown_connections 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 --2- 172.21.15.150:0/4091900012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c136540 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 --2- 172.21.15.150:0/4091900012 >> v2:172.21.15.150:3300/0 conn(0x7f546c06ed20 0x7f546c09ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.901+0000 7f5474500700 1 -- 172.21.15.150:0/4091900012 >> 172.21.15.150:0/4091900012 conn(0x7f546c068430 msgr2=0x7f546c0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- 172.21.15.150:0/4091900012 shutdown_connections 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- 172.21.15.150:0/4091900012 wait complete. 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient: build_initial_monmap 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient: monmap: 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:11.922 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 Processor -- start 2024-09-18T10:15:11.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- start start 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient: init 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): picked mon.noname-c con 0x7f546c06ed20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): picked mon.noname-b con 0x7f546c06f0f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c1416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): picked mon.noname-a con 0x7f546c09ce30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c120e50 con 0x7f546c09ce30 2024-09-18T10:15:11.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546c097060 con 0x7f546c06f0f0 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546c09ade0 con 0x7f546c06ed20 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5474500700 10 monclient(hunting): authenticate will time out at 1194.002472s 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53196/0 (socket says 172.21.15.150:53196) 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c1416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 -- 172.21.15.150:0/1434267992 learned_addr learned my addr 172.21.15.150:0/1434267992 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): get_auth_request con 0x7f546c06f0f0 auth_method 0 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): get_auth_request con 0x7f546c09ce30 auth_method 0 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 10 monclient(hunting): get_auth_request con 0x7f546c06ed20 auth_method 0 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5472a9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:11.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 msgr2=0x7f546c13e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 msgr2=0x7f546c13e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f547229c700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient(hunting): handle_auth_done global_id 15814 payload 824 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 msgr2=0x7f546c13e160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 msgr2=0x7f546c1416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c1416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 monclient: found mon.noname-b 2024-09-18T10:15:11.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.905+0000 7f5471a9b700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464009a20 con 0x7f546c06f0f0 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5471a9b700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 msgr2=0x7f546c13d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5471a9b700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c610 tx=0x7f545c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c005c80 con 0x7f546c06f0f0 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: dump: 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.929 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 5 monclient: authenticate success, global_id 15814 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c005e00 con 0x7f546c06f0f0 2024-09-18T10:15:11.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _renew_subs 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c007320 con 0x7f546c06f0f0 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546c122b90 con 0x7f546c06f0f0 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5470a99700 4 set_mon_vals no callback set 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 10 monclient: dump: 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:11.931 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:11.932 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:11.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _renew_subs 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546c13dd50 con 0x7f546c06f0f0 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c06fc90 con 0x7f546c06f0f0 2024-09-18T10:15:11.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f545c01b3b0 con 0x7f546c06f0f0 2024-09-18T10:15:11.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.909+0000 7f54627fc700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 0x7f5450037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:11.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.913+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(177..177 src has 58..177) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f545c0484b0 con 0x7f546c06f0f0 2024-09-18T10:15:11.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.913+0000 7f547229c700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 0x7f5450037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:11.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.913+0000 7f547229c700 10 monclient: get_auth_request con 0x7f5450035ac0 auth_method 0 2024-09-18T10:15:11.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.913+0000 7f547229c700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 0x7f5450037f70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5468006fd0 tx=0x7f5468008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:11.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.917+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f545c00f800 con 0x7f546c06f0f0 2024-09-18T10:15:11.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.917+0000 7f54627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:11.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:11.917+0000 7f54627fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:12.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69760ee700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:15:12.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69760ee700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:12.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69760ee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6970067950 con 0x7f6970141aa0 2024-09-18T10:15:12.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69577fe700 1 -- 172.21.15.142:0/1907338353 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f696001a030 con 0x7f6970141aa0 2024-09-18T10:15:12.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:12.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.001+0000 7f69577fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:12.009 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:15:12.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 msgr2=0x7f695c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 0x7f695c037fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f696400b5e0 tx=0x7f696401a000 comp rx=0 tx=0).stop 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 10 monclient: shutdown 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 msgr2=0x7f69701314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f69600073c0 tx=0x7f6960007c30 comp rx=0 tx=0).stop 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 shutdown_connections 2024-09-18T10:15:12.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f695c035af0 0x7f695c037fa0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69701319f0 0x7f697011e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6970141aa0 0x7f69701314b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 --2- 172.21.15.142:0/1907338353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69700a3ef0 0x7f6970141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 >> 172.21.15.142:0/1907338353 conn(0x7f6970071460 msgr2=0x7f69700a06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 shutdown_connections 2024-09-18T10:15:12.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.005+0000 7f69760ee700 1 -- 172.21.15.142:0/1907338353 wait complete. 2024-09-18T10:15:12.022 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:15:12.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:15:12.022 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:15:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:12.085+0000 7f5474500700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:12.085+0000 7f5474500700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:12.085+0000 7f5474500700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:12.085+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f546c063ed0 con 0x7f546c06f0f0 2024-09-18T10:15:12.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: get_monmap_and_config 2024-09-18T10:15:12.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: build_initial_monmap 2024-09-18T10:15:12.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: monmap: 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:12.132380+0000 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:12.132380+0000 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:12.137 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 Processor -- start 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 -- start start 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: init 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 0x7facc40989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): picked mon.noname-c con 0x7facc406f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): picked mon.noname-b con 0x7facc4098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7facc4133660 0x7facc4135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): picked mon.noname-a con 0x7facc4133660 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc409b030 con 0x7facc4133660 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc4099830 con 0x7facc406f3b0 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc409c330 con 0x7facc4098f00 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facca51a700 10 monclient(hunting): authenticate will time out at 1201.058067s 2024-09-18T10:15:12.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7facc4133660 0x7facc4135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38850/0 (socket says 172.21.15.142:38850) 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 0x7facc40989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 -- 172.21.15.142:0/3210082250 learned_addr learned my addr 172.21.15.142:0/3210082250 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): get_auth_request con 0x7facc4133660 auth_method 0 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 10 monclient(hunting): get_auth_request con 0x7facc4098f00 auth_method 0 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): get_auth_request con 0x7facc406f3b0 auth_method 0 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 -- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc40992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 -- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc40992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc8d17700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient(hunting): handle_auth_done global_id 54609 payload 274 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 -- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc40992f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 -- 172.21.15.142:0/3210082250 >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 msgr2=0x7facc40989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 --2- 172.21.15.142:0/3210082250 >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 0x7facc40989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient: _finish_hunting 0 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 monclient: found mon.noname-a 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9d19700 1 -- 172.21.15.142:0/3210082250 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc409c530 con 0x7facc4133660 2024-09-18T10:15:12.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.129+0000 7facc9518700 1 --2- 172.21.15.142:0/3210082250 >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 0x7facc40989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facc9d19700 1 -- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 msgr2=0x7facc4135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facc9d19700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4135a40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7facc4076de0 tx=0x7facbc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9a61440468fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 1 -- 172.21.15.142:0/3210082250 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc00e070 con 0x7facc4133660 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: dump: 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.152 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: _finish_auth 0 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 5 monclient: authenticate success, global_id 54609 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 1 -- 172.21.15.142:0/3210082250 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facbc00bd70 con 0x7facc4133660 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:12.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 4 set_mon_vals no callback set 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 1 -- 172.21.15.142:0/3210082250 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc012660 con 0x7facc4133660 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facbbfff700 10 monclient: dump: 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.154 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 10 monclient: shutdown 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 msgr2=0x7facc4135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4135a40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7facc4076de0 tx=0x7facbc009780 comp rx=0 tx=0).stop 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- 172.21.15.142:0/3210082250 shutdown_connections 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4135a40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 --2- 172.21.15.142:0/3210082250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc40992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 --2- 172.21.15.142:0/3210082250 >> v2:172.21.15.150:3300/0 conn(0x7facc406f3b0 0x7facc40989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- 172.21.15.142:0/3210082250 >> 172.21.15.142:0/3210082250 conn(0x7facc4070ef0 msgr2=0x7facc40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- 172.21.15.142:0/3210082250 shutdown_connections 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- 172.21.15.142:0/3210082250 wait complete. 2024-09-18T10:15:12.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 10 monclient: build_initial_monmap 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 10 monclient: monmap: 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:12.156 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 Processor -- start 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 1 -- start start 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.133+0000 7facca51a700 10 monclient: init 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 0x7facc413eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): picked mon.noname-b con 0x7facc406f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): picked mon.noname-c con 0x7facc4098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:12.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4128790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): picked mon.noname-a con 0x7facc4133660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc4132800 con 0x7facc4133660 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc412c650 con 0x7facc406f3b0 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc413f530 con 0x7facc4098f00 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient(hunting): authenticate will time out at 1201.064271s 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38860/0 (socket says 172.21.15.142:38860) 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 learned_addr learned my addr 172.21.15.142:0/2684730958 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:12.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 10 monclient(hunting): get_auth_request con 0x7facc4098f00 auth_method 0 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4128790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 0x7facc413eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): get_auth_request con 0x7facc4133660 auth_method 0 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): get_auth_request con 0x7facc406f3b0 auth_method 0 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc4131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc4131bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9d19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient(hunting): handle_auth_done global_id 15820 payload 824 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 msgr2=0x7facc4131bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:12.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7facc4098f00 0x7facc4131bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 msgr2=0x7facc4128790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4133660 0x7facc4128790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient: _finish_hunting 0 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 monclient: found mon.noname-b 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0009420 con 0x7facc406f3b0 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 msgr2=0x7facc413eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facc9518700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 0x7facc413eff0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7facc000ec80 tx=0x7facc00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0009700 con 0x7facc406f3b0 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 10 monclient: dump: 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.162 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 10 monclient: _finish_auth 0 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 5 monclient: authenticate success, global_id 15820 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc000ca00 con 0x7facc406f3b0 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient: _renew_subs 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc009a20 con 0x7facc406f3b0 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.137+0000 7facb97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc000cda0 con 0x7facc406f3b0 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 10 monclient: dump: 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facbb7fe700 4 set_mon_vals no callback set 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 10 monclient: _renew_subs 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc412c090 con 0x7facc406f3b0 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 0x7facc4128cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc40648a0 con 0x7facc4132100 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7facc0013680 con 0x7facc406f3b0 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 0x7facc4128cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient: get_auth_request con 0x7facc4132100 auth_method 0 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 10 monclient(hunting): handle_auth_done global_id 54615 payload 824 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facc406fab0 con 0x7facc4132100 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faca8035ac0 0x7faca8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(178..178 src has 58..178) v4 ==== 7673+0+0 (secure 0 0 0) 0x7facc0049040 con 0x7facc406f3b0 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc9d19700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faca8035ac0 0x7faca8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.141+0000 7facc9d19700 10 monclient: get_auth_request con 0x7faca8035ac0 auth_method 0 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 msgr2=0x7facc4128cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 0x7facc4128cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7facb4009240 tx=0x7facb4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb400eab0 con 0x7facc4132100 2024-09-18T10:15:12.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 10 monclient: dump: 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7facb400c050 con 0x7facc4132100 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 msgr2=0x7facc4128cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 0x7facc4128cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7facb4009240 tx=0x7facb4008040 comp rx=0 tx=0).stop 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 msgr2=0x7facc4128cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facb97fa700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4132100 0x7facc4128cd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.145+0000 7facc9d19700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faca8035ac0 0x7faca8037f70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facc409a830 tx=0x7facbc019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facca51a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facca51a700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 0x7facc406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facca51a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc4129610 con 0x7facc4129210 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 0x7facc406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient: get_auth_request con 0x7facc4129210 auth_method 0 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 10 monclient(hunting): handle_auth_done global_id 54618 payload 824 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.153+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7facc4072a80 con 0x7facc4129210 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facc8d17700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 msgr2=0x7facc406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facc8d17700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 0x7facc406d380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7facb401a000 tx=0x7facb401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb4015bd0 con 0x7facc4129210 2024-09-18T10:15:12.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 10 monclient: dump: 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7facb400e8a0 con 0x7facc4129210 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 msgr2=0x7facc406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 0x7facc406d380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7facb401a000 tx=0x7facb401d000 comp rx=0 tx=0).stop 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 msgr2=0x7facc406d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facb97fa700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7facc4129210 0x7facc406d380 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7facb401a000 tx=0x7facb401d000 comp rx=0 tx=0).stop 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:12.173 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:12.174 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:12.175 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.176 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:12.177 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:12.178 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:12.181 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:12.182 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:12.183 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:12.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:15:12.185 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:12.186 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:12.186 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:12.186 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:12.186 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faca8035ac0 msgr2=0x7faca8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faca8035ac0 0x7faca8037f70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7facc409a830 tx=0x7facbc019000 comp rx=0 tx=0).stop 2024-09-18T10:15:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 10 monclient: shutdown 2024-09-18T10:15:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 msgr2=0x7facc413eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facc9d19700 1 -- 172.21.15.142:0/2684730958 reap_dead start 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 0x7facc413eff0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7facc000ec80 tx=0x7facc00091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 shutdown_connections 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 --2- 172.21.15.142:0/2684730958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc406f3b0 0x7facc413eff0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 >> 172.21.15.142:0/2684730958 conn(0x7facc4070ef0 msgr2=0x7facc409c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 shutdown_connections 2024-09-18T10:15:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.157+0000 7facca51a700 1 -- 172.21.15.142:0/2684730958 wait complete. 2024-09-18T10:15:12.188 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:12.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:15:12.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:15:12.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.293+0000 7fada359e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:12.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.293+0000 7fada359e700 10 monclient: build_initial_monmap 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.293+0000 7fada359e700 10 monclient: monmap: 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:12.299618+0000 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:12.299618+0000 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:12.303 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 Processor -- start 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 -- start start 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient: init 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fada40a0180 0x7fada409ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): picked mon.noname-a con 0x7fada40a0180 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fada40a0550 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fada413bb60 0x7fada413df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): picked mon.noname-c con 0x7fada413bb60 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:12.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada4099540 con 0x7fada40a0550 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada406a660 con 0x7fada40a0180 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada409b3d0 con 0x7fada413bb60 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada359e700 10 monclient(hunting): authenticate will time out at 1201.225140s 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fada40a0180 0x7fada409ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fada40a0180 0x7fada409ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36198/0 (socket says 172.21.15.142:36198) 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 -- 172.21.15.142:0/3115608576 learned_addr learned my addr 172.21.15.142:0/3115608576 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): get_auth_request con 0x7fada40a0180 auth_method 0 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada2d9d700 1 --2- 172.21.15.142:0/3115608576 >> v2:172.21.15.150:3300/0 conn(0x7fada413bb60 0x7fada413df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada2d9d700 10 monclient(hunting): get_auth_request con 0x7fada413bb60 auth_method 0 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient(hunting): handle_auth_done global_id 54621 payload 274 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 -- 172.21.15.142:0/3115608576 >> v2:172.21.15.150:3300/0 conn(0x7fada413bb60 msgr2=0x7fada413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 --2- 172.21.15.142:0/3115608576 >> v2:172.21.15.150:3300/0 conn(0x7fada413bb60 0x7fada413df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 -- 172.21.15.142:0/3115608576 >> v1:172.21.15.150:6789/0 conn(0x7fada40a0550 legacy=0x7fada40a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:12.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 monclient: found mon.noname-a 2024-09-18T10:15:12.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:12.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.297+0000 7fada259c700 1 -- 172.21.15.142:0/3115608576 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada408c230 con 0x7fada40a0180 2024-09-18T10:15:12.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 1 -- 172.21.15.142:0/3115608576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada40a0180 msgr2=0x7fada409ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 1 --2- 172.21.15.142:0/3115608576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada40a0180 0x7fada409ae90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fada406a240 tx=0x7fad98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b761c1398825106a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 1 -- 172.21.15.142:0/3115608576 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9800ba70 con 0x7fada40a0180 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: dump: 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.312 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: _finish_auth 0 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 5 monclient: authenticate success, global_id 54621 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 1 -- 172.21.15.142:0/3115608576 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9800bbd0 con 0x7fada40a0180 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 4 set_mon_vals no callback set 2024-09-18T10:15:12.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 1 -- 172.21.15.142:0/3115608576 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98011420 con 0x7fada40a0180 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada159a700 10 monclient: dump: 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.314 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: shutdown 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- 172.21.15.142:0/3115608576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada40a0180 msgr2=0x7fada409ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- 172.21.15.142:0/3115608576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada40a0180 0x7fada409ae90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fada406a240 tx=0x7fad98009250 comp rx=0 tx=0).stop 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- 172.21.15.142:0/3115608576 shutdown_connections 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- 172.21.15.142:0/3115608576 >> v2:172.21.15.150:3300/0 conn(0x7fada413bb60 0x7fada413df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- 172.21.15.142:0/3115608576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada40a0180 0x7fada409ae90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- 172.21.15.142:0/3115608576 >> 172.21.15.142:0/3115608576 conn(0x7fada40a5790 msgr2=0x7fada40a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- 172.21.15.142:0/3115608576 shutdown_connections 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- 172.21.15.142:0/3115608576 wait complete. 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: build_initial_monmap 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: monmap: 2024-09-18T10:15:12.315 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 Processor -- start 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- start start 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: init 2024-09-18T10:15:12.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada40a0180 0x7fada4146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): picked mon.noname-b con 0x7fada40a0180 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): picked mon.noname-c con 0x7fada40a0550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 0x7fada4096960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): picked mon.noname-a con 0x7fada413bb60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4099e40 con 0x7fada413bb60 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada406aab0 con 0x7fada40a0180 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada409f620 con 0x7fada40a0550 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada359e700 10 monclient(hunting): authenticate will time out at 1201.230894s 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 0x7fada4096960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38862/0 (socket says 172.21.15.142:38862) 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 -- 172.21.15.142:0/2938738470 learned_addr learned my addr 172.21.15.142:0/2938738470 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 10 monclient(hunting): get_auth_request con 0x7fada40a0550 auth_method 0 2024-09-18T10:15:12.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): get_auth_request con 0x7fada413bb60 auth_method 0 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 msgr2=0x7fada4146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 msgr2=0x7fada4146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada1d9b700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada40a0180 0x7fada4146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:12.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 10 monclient(hunting): get_auth_request con 0x7fada40a0180 auth_method 0 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient(hunting): handle_auth_done global_id 54624 payload 824 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 msgr2=0x7fada4146740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada40a0550 0x7fada4146740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada40a0180 msgr2=0x7fada4146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fada40a0180 0x7fada4146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 monclient: found mon.noname-a 2024-09-18T10:15:12.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.301+0000 7fada2d9d700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980094f0 con 0x7fada413bb60 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada2d9d700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 msgr2=0x7fada4096960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada2d9d700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 0x7fada4096960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fad9c007ea0 tx=0x7fad9c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00edd0 con 0x7fada413bb60 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: dump: 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 5 monclient: authenticate success, global_id 54624 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9c014420 con 0x7fada413bb60 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 monclient: _renew_subs 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada406bc50 con 0x7fada413bb60 2024-09-18T10:15:12.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c0147e0 con 0x7fada413bb60 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fad927fc700 10 monclient: dump: 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:12.323 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 monclient: _renew_subs 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:12.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada4098530 con 0x7fada413bb60 2024-09-18T10:15:12.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada0d99700 4 set_mon_vals no callback set 2024-09-18T10:15:12.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:12.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.305+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada40647f0 con 0x7fada4097670 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097670 auth_method 0 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fad9c014bd0 con 0x7fada413bb60 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fad927fc700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 0x7fad84037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(178..178 src has 58..178) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fad9c0496e0 con 0x7fada413bb60 2024-09-18T10:15:12.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada1d9b700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 0x7fad84037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada1d9b700 10 monclient: get_auth_request con 0x7fad84035ac0 auth_method 0 2024-09-18T10:15:12.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.309+0000 7fada1d9b700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 0x7fad84037f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad94007fd0 tx=0x7fad94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:12.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097670 auth_method 0 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.513+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:15:12.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097670 auth_method 0 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:12.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:12.917+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:15:13.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.049+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 added bucket fooo type host to crush map v179) v1 ==== 140+0+0 (secure 0 0 0) 0x7f545c017070 con 0x7f546c06f0f0 2024-09-18T10:15:13.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.049+0000 7f54627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:13.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.049+0000 7f54627fc700 10 monclient: _finish_command 2 = system:0 added bucket fooo type host to crush map 2024-09-18T10:15:13.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f5474500700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:13.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f5474500700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:13.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f5474500700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"} v 0) v1 -- 0x7f546c0008d0 con 0x7f546c06f0f0 2024-09-18T10:15:13.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f54627fc700 1 -- 172.21.15.150:0/1434267992 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}]=0 bucket 'fooo' already exists v179) v1 ==== 128+0+0 (secure 0 0 0) 0x7f545c00f800 con 0x7f546c06f0f0 2024-09-18T10:15:13.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f54627fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "fooo", "type": "host"}] 2024-09-18T10:15:13.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.161+0000 7f54627fc700 10 monclient: _finish_command 3 = system:0 bucket 'fooo' already exists 2024-09-18T10:15:13.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 msgr2=0x7f5450037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 0x7f5450037f70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5468006fd0 tx=0x7f5468008040 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 10 monclient: shutdown 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 msgr2=0x7f546c13d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f545c00c610 tx=0x7f545c007f90 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 shutdown_connections 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5450035ac0 0x7f5450037f70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f546c09ce30 0x7f546c1416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f546c06f0f0 0x7f546c13d810 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 --2- 172.21.15.150:0/1434267992 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f546c06ed20 0x7f546c13e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 >> 172.21.15.150:0/1434267992 conn(0x7f546c068430 msgr2=0x7f546c0a04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:13.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 shutdown_connections 2024-09-18T10:15:13.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.165+0000 7f5474500700 1 -- 172.21.15.150:0/1434267992 wait complete. 2024-09-18T10:15:13.172 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'fooo' already exists 2024-09-18T10:15:13.181 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush move fooo root=default 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: get_monmap_and_config 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: build_initial_monmap 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: monmap: 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:13.278114+0000 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:13.278114+0000 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:13.279 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:13.280 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 Processor -- start 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 -- start start 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: init 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc95409c930 0x7fc954097730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-a con 0x7fc95409c930 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc95409cd00 0x7fc95409d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-b con 0x7fc95409cd00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc954136760 0x7fc954138b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-c con 0x7fc954136760 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc954098f00 con 0x7fc95409c930 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc954096a70 con 0x7fc954136760 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc954097c70 con 0x7fc95409cd00 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:13.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc959af8700 10 monclient(hunting): authenticate will time out at 1195.371556s 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc954136760 0x7fc954138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc954136760 0x7fc954138b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53208/0 (socket says 172.21.15.150:53208) 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 1 -- 172.21.15.150:0/2072696638 learned_addr learned my addr 172.21.15.150:0/2072696638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): get_auth_request con 0x7fc954136760 auth_method 0 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc953fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc952ffd700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc95409cd00 0x7fc95409d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.273+0000 7fc9537fe700 1 --2- 172.21.15.150:0/2072696638 >> v2:172.21.15.142:3300/0 conn(0x7fc95409c930 0x7fc954097730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc952ffd700 10 monclient(hunting): get_auth_request con 0x7fc95409cd00 auth_method 0 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc952ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc952ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc952ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9537fe700 10 monclient(hunting): get_auth_request con 0x7fc95409c930 auth_method 0 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 10 monclient(hunting): handle_auth_done global_id 15829 payload 274 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 -- 172.21.15.150:0/2072696638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc95409cd00 msgr2=0x7fc95409d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc95409cd00 0x7fc95409d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 -- 172.21.15.150:0/2072696638 >> v2:172.21.15.142:3300/0 conn(0x7fc95409c930 msgr2=0x7fc954097730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 --2- 172.21.15.150:0/2072696638 >> v2:172.21.15.142:3300/0 conn(0x7fc95409c930 0x7fc954097730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:13.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 monclient: found mon.noname-c 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 -- 172.21.15.150:0/2072696638 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95406f730 con 0x7fc954136760 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 -- 172.21.15.150:0/2072696638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc954136760 msgr2=0x7fc954138b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc953fff700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc954136760 0x7fc954138b40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc95406f9e0 tx=0x7fc94c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5d386110cc3e66 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 1 -- 172.21.15.150:0/2072696638 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00e070 con 0x7fc954136760 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: dump: 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.288 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 5 monclient: authenticate success, global_id 15829 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 1 -- 172.21.15.150:0/2072696638 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c00bc60 con 0x7fc954136760 2024-09-18T10:15:13.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 4 set_mon_vals no callback set 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 1 -- 172.21.15.150:0/2072696638 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c0092e0 con 0x7fc954136760 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc9527fc700 10 monclient: dump: 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.290 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 10 monclient: shutdown 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 -- 172.21.15.150:0/2072696638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc954136760 msgr2=0x7fc954138b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc954136760 0x7fc954138b40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc95406f9e0 tx=0x7fc94c009840 comp rx=0 tx=0).stop 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 -- 172.21.15.150:0/2072696638 shutdown_connections 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc954136760 0x7fc954138b40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 --2- 172.21.15.150:0/2072696638 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc95409cd00 0x7fc95409d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 --2- 172.21.15.150:0/2072696638 >> v2:172.21.15.142:3300/0 conn(0x7fc95409c930 0x7fc954097730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 -- 172.21.15.150:0/2072696638 >> 172.21.15.150:0/2072696638 conn(0x7fc954068360 msgr2=0x7fc9540a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:13.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 -- 172.21.15.150:0/2072696638 shutdown_connections 2024-09-18T10:15:13.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.277+0000 7fc959af8700 1 -- 172.21.15.150:0/2072696638 wait complete. 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient: build_initial_monmap 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient: monmap: 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.296 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 Processor -- start 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 -- start start 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient: init 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 0x7fc954132170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-a con 0x7fc95409c930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-b con 0x7fc95409cd00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc954136760 0x7fc95411af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): picked mon.noname-c con 0x7fc954136760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95411e6f0 con 0x7fc95409c930 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95409e3f0 con 0x7fc95409cd00 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95409b810 con 0x7fc954136760 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc959af8700 10 monclient(hunting): authenticate will time out at 1195.377397s 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53210/0 (socket says 172.21.15.150:53210) 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 -- 172.21.15.150:0/3510002584 learned_addr learned my addr 172.21.15.150:0/3510002584 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): get_auth_request con 0x7fc95409cd00 auth_method 0 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc953fff700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc954136760 0x7fc95411af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc953fff700 10 monclient(hunting): get_auth_request con 0x7fc954136760 auth_method 0 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc953fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient(hunting): handle_auth_done global_id 15832 payload 824 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc9537fe700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 0x7fc954132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 -- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc954136760 msgr2=0x7fc95411af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc954136760 0x7fc95411af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 -- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 msgr2=0x7fc954132170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 0x7fc954132170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc952ffd700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c009ae0 con 0x7fc95409cd00 2024-09-18T10:15:13.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc9537fe700 10 monclient: get_auth_request con 0x7fc95409c930 auth_method 0 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.281+0000 7fc9537fe700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 0x7fc954132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc952ffd700 1 -- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 msgr2=0x7fc95411aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc952ffd700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc94400d890 tx=0x7fc944008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94400b810 con 0x7fc95409cd00 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: dump: 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.301 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 5 monclient: authenticate success, global_id 15832 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94400b970 con 0x7fc95409cd00 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94400bcf0 con 0x7fc95409cd00 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _renew_subs 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95411ed60 con 0x7fc95409cd00 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc951ffb700 4 set_mon_vals no callback set 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 10 monclient: dump: 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:13.303 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _renew_subs 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95411ca90 con 0x7fc95409cd00 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc944014640 con 0x7fc95409cd00 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95411e1c0 con 0x7fc95409cd00 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 0x7fc934037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(179..179 src has 58..179) v4 ==== 7701+0+0 (secure 0 0 0) 0x7fc944049750 con 0x7fc95409cd00 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc9537fe700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 0x7fc934037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.285+0000 7fc9537fe700 10 monclient: get_auth_request con 0x7fc934035ac0 auth_method 0 2024-09-18T10:15:13.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.289+0000 7fc9537fe700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 0x7fc934037f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc93c006fd0 tx=0x7fc93c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:13.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.297+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc94401de30 con 0x7fc95409cd00 2024-09-18T10:15:13.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.297+0000 7fc94b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:13.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.297+0000 7fc94b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:13.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.465+0000 7fc959af8700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:13.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.465+0000 7fc959af8700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:13.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.465+0000 7fc959af8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:13.469 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:13.465+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fc954097030 con 0x7fc95409cd00 2024-09-18T10:15:13.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:13.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097670 auth_method 0 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:13.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:13.721+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:15:14.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.061+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 moved item id -16 name 'fooo' to location {root=default} in crush map v180) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc94401d830 con 0x7fc95409cd00 2024-09-18T10:15:14.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.061+0000 7fc94b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:14.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.061+0000 7fc94b7fe700 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T10:15:14.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.169+0000 7fc959af8700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:14.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.169+0000 7fc959af8700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:14.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.169+0000 7fc959af8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.169+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]} v 0) v1 -- 0x7fc954063ed0 con 0x7fc95409cd00 2024-09-18T10:15:14.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc94b7fe700 1 -- 172.21.15.150:0/3510002584 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}]=0 no need to move item id -16 name 'fooo' to location {root=default} in crush map v180) v1 ==== 183+0+0 (secure 0 0 0) 0x7fc94401da10 con 0x7fc95409cd00 2024-09-18T10:15:14.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc94b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["root=default"]}] 2024-09-18T10:15:14.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc94b7fe700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T10:15:14.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 msgr2=0x7fc934037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:14.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 0x7fc934037f70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc93c006fd0 tx=0x7fc93c008040 comp rx=0 tx=0).stop 2024-09-18T10:15:14.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 10 monclient: shutdown 2024-09-18T10:15:14.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 msgr2=0x7fc95411aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.173+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc94400d890 tx=0x7fc944008040 comp rx=0 tx=0).stop 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 shutdown_connections 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc934035ac0 0x7fc934037f70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc954136760 0x7fc95411af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc95409cd00 0x7fc95411aa50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 --2- 172.21.15.150:0/3510002584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc95409c930 0x7fc954132170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 >> 172.21.15.150:0/3510002584 conn(0x7fc954068360 msgr2=0x7fc954135de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 shutdown_connections 2024-09-18T10:15:14.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.177+0000 7fc959af8700 1 -- 172.21.15.150:0/3510002584 wait complete. 2024-09-18T10:15:14.183 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -16 name 'fooo' to location {root=default} in crush map 2024-09-18T10:15:14.191 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket barr rack 2024-09-18T10:15:14.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 10 monclient: get_monmap_and_config 2024-09-18T10:15:14.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 10 monclient: build_initial_monmap 2024-09-18T10:15:14.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 10 monclient: monmap: 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:14.286750+0000 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:14.286750+0000 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:14.288 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 1 Processor -- start 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 1 -- start start 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.281+0000 7fd779238700 10 monclient: init 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7740a4270 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:14.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd77406ed20 0x7fd7740a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): picked mon.noname-c con 0x7fd77406ed20 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd774137590 0x7fd774139970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): picked mon.noname-a con 0x7fd774137590 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740a2220 con 0x7fd7740a4270 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7740a2830 con 0x7fd774137590 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7740a4640 con 0x7fd77406ed20 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd779238700 10 monclient(hunting): authenticate will time out at 1196.380586s 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd772d9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd7740a4270 0x7fd77406e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46036/0 (socket says 172.21.15.150:46036) 2024-09-18T10:15:14.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd772d9d700 1 -- 172.21.15.150:0/3810755207 learned_addr learned my addr 172.21.15.150:0/3810755207 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77259c700 1 --2- 172.21.15.150:0/3810755207 >> v2:172.21.15.150:3300/0 conn(0x7fd77406ed20 0x7fd7740a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77259c700 10 monclient(hunting): get_auth_request con 0x7fd77406ed20 auth_method 0 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 --2- 172.21.15.150:0/3810755207 >> v2:172.21.15.142:3300/0 conn(0x7fd774137590 0x7fd774139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): get_auth_request con 0x7fd774137590 auth_method 0 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:14.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient(hunting): handle_auth_done global_id 54630 payload 274 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 -- 172.21.15.150:0/3810755207 >> v2:172.21.15.150:3300/0 conn(0x7fd77406ed20 msgr2=0x7fd7740a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 --2- 172.21.15.150:0/3810755207 >> v2:172.21.15.150:3300/0 conn(0x7fd77406ed20 0x7fd7740a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 -- 172.21.15.150:0/3810755207 >> v1:172.21.15.150:6789/0 conn(0x7fd7740a4270 legacy=0x7fd77406e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient: _finish_hunting 0 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 monclient: found mon.noname-a 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:14.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.285+0000 7fd77359e700 1 -- 172.21.15.150:0/3810755207 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77406b990 con 0x7fd774137590 2024-09-18T10:15:14.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd77359e700 1 -- 172.21.15.150:0/3810755207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd774137590 msgr2=0x7fd774139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:14.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd77359e700 1 --2- 172.21.15.150:0/3810755207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd774137590 0x7fd774139970 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd77406b630 tx=0x7fd76c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a3b7dda06a34be server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:14.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 1 -- 172.21.15.150:0/3810755207 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00e070 con 0x7fd774137590 2024-09-18T10:15:14.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: dump: 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:14.299 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: _finish_auth 0 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 5 monclient: authenticate success, global_id 54630 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 1 -- 172.21.15.150:0/3810755207 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c00be40 con 0x7fd774137590 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 4 set_mon_vals no callback set 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 1 -- 172.21.15.150:0/3810755207 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c012680 con 0x7fd774137590 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd771d9b700 10 monclient: dump: 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:14.300 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: shutdown 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- 172.21.15.150:0/3810755207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd774137590 msgr2=0x7fd774139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:14.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- 172.21.15.150:0/3810755207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd774137590 0x7fd774139970 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd77406b630 tx=0x7fd76c009780 comp rx=0 tx=0).stop 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- 172.21.15.150:0/3810755207 shutdown_connections 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- 172.21.15.150:0/3810755207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd774137590 0x7fd774139970 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- 172.21.15.150:0/3810755207 >> v2:172.21.15.150:3300/0 conn(0x7fd77406ed20 0x7fd7740a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- 172.21.15.150:0/3810755207 >> 172.21.15.150:0/3810755207 conn(0x7fd7740716e0 msgr2=0x7fd7740a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- 172.21.15.150:0/3810755207 shutdown_connections 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- 172.21.15.150:0/3810755207 wait complete. 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: build_initial_monmap 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: monmap: 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:14.302 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 Processor -- start 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- start start 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: init 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): picked mon.noname-b con 0x7fd77406ed20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7740a4270 0x7fd77411f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): picked mon.noname-a con 0x7fd7740a4270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): picked mon.noname-c con 0x7fd774137590 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77406f160 con 0x7fd7740a4270 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77406ca50 con 0x7fd77406ed20 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd774121380 con 0x7fd774137590 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd779238700 10 monclient(hunting): authenticate will time out at 1196.387548s 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53242/0 (socket says 172.21.15.150:53242) 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 1 -- 172.21.15.150:0/2033222414 learned_addr learned my addr 172.21.15.150:0/2033222414 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 10 monclient(hunting): get_auth_request con 0x7fd77406ed20 auth_method 0 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd772d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd77259c700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7740a4270 0x7fd77411f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.289+0000 7fd77359e700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): get_auth_request con 0x7fd7740a4270 auth_method 0 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 10 monclient(hunting): get_auth_request con 0x7fd774137590 auth_method 0 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:14.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 msgr2=0x7fd77411f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 msgr2=0x7fd77411f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77359e700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient(hunting): handle_auth_done global_id 15838 payload 824 2024-09-18T10:15:14.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 msgr2=0x7fd77411f910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7740a4270 msgr2=0x7fd77411f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7740a4270 0x7fd77411f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 monclient: found mon.noname-b 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c009a20 con 0x7fd77406ed20 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 msgr2=0x7fd77411ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd772d9d700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c610 tx=0x7fd75c004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c005d40 con 0x7fd77406ed20 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: dump: 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:14.309 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: _finish_auth 0 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 5 monclient: authenticate success, global_id 15838 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c0070d0 con 0x7fd77406ed20 2024-09-18T10:15:14.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c007490 con 0x7fd77406ed20 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 10 monclient: _renew_subs 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd774122240 con 0x7fd77406ed20 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd77159a700 4 set_mon_vals no callback set 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd76affd700 10 monclient: dump: 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:14.311 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 10 monclient: _renew_subs 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.293+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7741372b0 con 0x7fd77406ed20 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd779238700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd779238700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd779238700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd75c01b3b0 con 0x7fd77406ed20 2024-09-18T10:15:14.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd774063ed0 con 0x7fd77406ed20 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd76affd700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 0x7fd754037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(180..180 src has 58..180) v4 ==== 7889+0+0 (secure 0 0 0) 0x7fd75c048580 con 0x7fd77406ed20 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd77259c700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 0x7fd754037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.297+0000 7fd77259c700 10 monclient: get_auth_request con 0x7fd754035ac0 auth_method 0 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.301+0000 7fd77259c700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 0x7fd754037f70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.305+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd75c00f860 con 0x7fd77406ed20 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.305+0000 7fd76affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:14.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.305+0000 7fd76affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:14.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.469+0000 7fd779238700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:14.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.469+0000 7fd779238700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:14.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.469+0000 7fd779238700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:14.475 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:14.469+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fd774122670 con 0x7fd77406ed20 2024-09-18T10:15:15.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.305+0000 7fad917fa700 10 monclient: tick 2024-09-18T10:15:15.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.305+0000 7fad917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:15.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.305+0000 7fad917fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:15.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097670 auth_method 0 2024-09-18T10:15:15.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:15.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:15:15.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:15.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:15.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:15.325+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T10:15:15.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.653+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 added bucket barr type rack to crush map v181) v1 ==== 140+0+0 (secure 0 0 0) 0x7fd75c017020 con 0x7fd77406ed20 2024-09-18T10:15:15.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.653+0000 7fd76affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:15.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.653+0000 7fd76affd700 10 monclient: _finish_command 2 = system:0 added bucket barr type rack to crush map 2024-09-18T10:15:15.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.761+0000 7fd779238700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:15.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd779238700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:15.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd779238700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"} v 0) v1 -- 0x7fd7740008d0 con 0x7fd77406ed20 2024-09-18T10:15:15.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd76affd700 1 -- 172.21.15.150:0/2033222414 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}]=0 bucket 'barr' already exists v181) v1 ==== 128+0+0 (secure 0 0 0) 0x7fd75c00f260 con 0x7fd77406ed20 2024-09-18T10:15:15.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd76affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "barr", "type": "rack"}] 2024-09-18T10:15:15.770 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.765+0000 7fd76affd700 10 monclient: _finish_command 3 = system:0 bucket 'barr' already exists 2024-09-18T10:15:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 msgr2=0x7fd754037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 0x7fd754037f70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764008040 comp rx=0 tx=0).stop 2024-09-18T10:15:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 10 monclient: shutdown 2024-09-18T10:15:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:15.774 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 msgr2=0x7fd77411ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00c610 tx=0x7fd75c004a80 comp rx=0 tx=0).stop 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 shutdown_connections 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd754035ac0 0x7fd754037f70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd774137590 0x7fd77411f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7740a4270 0x7fd77411f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.775 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 --2- 172.21.15.150:0/2033222414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd77406ed20 0x7fd77411ee90 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 >> 172.21.15.150:0/2033222414 conn(0x7fd7740716e0 msgr2=0x7fd77409e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 shutdown_connections 2024-09-18T10:15:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.769+0000 7fd779238700 1 -- 172.21.15.150:0/2033222414 wait complete. 2024-09-18T10:15:15.776 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'barr' already exists 2024-09-18T10:15:15.782 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush move barr root=default 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 10 monclient: get_monmap_and_config 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 10 monclient: build_initial_monmap 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 10 monclient: monmap: 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:15.885526+0000 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:15.885526+0000 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:15.919 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 1 Processor -- start 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 1 -- start start 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.881+0000 7f673a41c700 10 monclient: init 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 0x7f6734096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-a con 0x7f6734071240 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673409e1e0 0x7f6734096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-b con 0x7f673409e1e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67340994f0 0x7f67340974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-c con 0x7f67340994f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673409e640 con 0x7f6734071240 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6734098cf0 con 0x7f67340994f0 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673409df80 con 0x7f673409e1e0 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f673a41c700 10 monclient(hunting): authenticate will time out at 1197.980276s 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67340994f0 0x7f67340974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67340994f0 0x7f67340974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53244/0 (socket says 172.21.15.150:53244) 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 -- 172.21.15.150:0/3783371867 learned_addr learned my addr 172.21.15.150:0/3783371867 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): get_auth_request con 0x7f67340994f0 auth_method 0 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 1 --2- 172.21.15.150:0/3783371867 >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 0x7f6734096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673409e1e0 0x7f6734096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): get_auth_request con 0x7f6734071240 auth_method 0 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): get_auth_request con 0x7f673409e1e0 auth_method 0 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:15.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient(hunting): handle_auth_done global_id 15841 payload 274 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 -- 172.21.15.150:0/3783371867 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673409e1e0 msgr2=0x7f6734096fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673409e1e0 0x7f6734096fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 -- 172.21.15.150:0/3783371867 >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 msgr2=0x7f6734096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 --2- 172.21.15.150:0/3783371867 >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 0x7f6734096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient: _finish_hunting 0 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 monclient: found mon.noname-c 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 -- 172.21.15.150:0/3783371867 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734098a10 con 0x7f67340994f0 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6733fff700 1 --2- 172.21.15.150:0/3783371867 >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 0x7f6734096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:15.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 -- 172.21.15.150:0/3783371867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67340994f0 msgr2=0x7f67340974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f67389b9700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67340994f0 0x7f67340974f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f673409c2d0 tx=0x7f672c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13897299fb604435 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 1 -- 172.21.15.150:0/3783371867 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c00b850 con 0x7f67340994f0 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 10 monclient: dump: 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:15.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.885+0000 7f6732ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 5 monclient: authenticate success, global_id 15841 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 1 -- 172.21.15.150:0/3783371867 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c00b9b0 con 0x7f67340994f0 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 4 set_mon_vals no callback set 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 1 -- 172.21.15.150:0/3783371867 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c00bd50 con 0x7f67340994f0 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:15.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f6732ffd700 10 monclient: dump: 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:15.927 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 10 monclient: shutdown 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- 172.21.15.150:0/3783371867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67340994f0 msgr2=0x7f67340974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67340994f0 0x7f67340974f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f673409c2d0 tx=0x7f672c009250 comp rx=0 tx=0).stop 2024-09-18T10:15:15.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- 172.21.15.150:0/3783371867 shutdown_connections 2024-09-18T10:15:15.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67340994f0 0x7f67340974f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 --2- 172.21.15.150:0/3783371867 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f673409e1e0 0x7f6734096fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 --2- 172.21.15.150:0/3783371867 >> v2:172.21.15.142:3300/0 conn(0x7f6734071240 0x7f6734096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- 172.21.15.150:0/3783371867 >> 172.21.15.150:0/3783371867 conn(0x7f67340a1cc0 msgr2=0x7f67340a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:15.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- 172.21.15.150:0/3783371867 shutdown_connections 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- 172.21.15.150:0/3783371867 wait complete. 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 10 monclient: build_initial_monmap 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 10 monclient: monmap: 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:15.930 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:15.931 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 Processor -- start 2024-09-18T10:15:15.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 1 -- start start 2024-09-18T10:15:15.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.889+0000 7f673a41c700 10 monclient: init 2024-09-18T10:15:15.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:15.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6734071240 0x7f673411a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-c con 0x7f6734071240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67340994f0 0x7f673411af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-a con 0x7f67340994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): picked mon.noname-b con 0x7f673409e1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673411caf0 con 0x7f67340994f0 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673409b230 con 0x7f673409e1e0 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673411c570 con 0x7f6734071240 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient(hunting): authenticate will time out at 1197.988388s 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53254/0 (socket says 172.21.15.150:53254) 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 -- 172.21.15.150:0/1032715307 learned_addr learned my addr 172.21.15.150:0/1032715307 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:15.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): get_auth_request con 0x7f673409e1e0 auth_method 0 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:15.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67340994f0 0x7f673411af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f6733fff700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6734071240 0x7f673411a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): get_auth_request con 0x7f67340994f0 auth_method 0 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f6733fff700 10 monclient(hunting): get_auth_request con 0x7f6734071240 auth_method 0 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f6733fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f6733fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f6733fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:15.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient(hunting): handle_auth_done global_id 15844 payload 824 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 -- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6734071240 msgr2=0x7f673411a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6734071240 0x7f673411a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 -- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67340994f0 msgr2=0x7f673411af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67340994f0 0x7f673411af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient: _finish_hunting 0 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 monclient: found mon.noname-b 2024-09-18T10:15:15.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724009a20 con 0x7f673409e1e0 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 -- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 msgr2=0x7f673411b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67389b9700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c006a90 con 0x7f673409e1e0 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: dump: 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.938 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: _finish_auth 0 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 5 monclient: authenticate success, global_id 15844 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c024440 con 0x7f673409e1e0 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _renew_subs 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c0094f0 con 0x7f673409e1e0 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c024800 con 0x7f673409e1e0 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f672bfff700 10 monclient: dump: 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:15.940 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f67327fc700 4 set_mon_vals no callback set 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _renew_subs 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.893+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673411d260 con 0x7f673409e1e0 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f672c02d440 con 0x7f673409e1e0 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f673a41c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f673a41c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f673a41c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673411e0f0 con 0x7f673409e1e0 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f672bfff700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 0x7f6714037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:15.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(181..181 src has 58..181) v4 ==== 7917+0+0 (secure 0 0 0) 0x7f672c04a740 con 0x7f673409e1e0 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f6733fff700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 0x7f6714037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.897+0000 7f6733fff700 10 monclient: get_auth_request con 0x7f6714035ac0 auth_method 0 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.901+0000 7f6733fff700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 0x7f6714037f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f67240099f0 tx=0x7f672400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.905+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f672c01a420 con 0x7f673409e1e0 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.905+0000 7f672bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:15.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:15.905+0000 7f672bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:16.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.073+0000 7f673a41c700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.073+0000 7f673a41c700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.073+0000 7f673a41c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.073+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f673411d260 con 0x7f673409e1e0 2024-09-18T10:15:16.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 10 monclient: tick 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 0x7fad7c005a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 msgr2=0x7fada4029d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097670 0x7fada4029d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad917fa700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c0060c0 con 0x7fad7c003590 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 0x7fad7c005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient: get_auth_request con 0x7fad7c003590 auth_method 0 2024-09-18T10:15:16.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 10 monclient(hunting): handle_auth_done global_id 74105 payload 824 2024-09-18T10:15:16.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad7c006230 con 0x7fad7c003590 2024-09-18T10:15:16.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 msgr2=0x7fad7c005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 0x7fad7c005a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad98002000 tx=0x7fad98016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98022070 con 0x7fad7c003590 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.305+0000 7fad927fc700 10 monclient: dump: 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.312 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fad98000ca0 con 0x7fad7c003590 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 msgr2=0x7fad7c005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 0x7fad7c005a10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad98002000 tx=0x7fad98016000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 msgr2=0x7fad7c005a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:16.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.309+0000 7fad927fc700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad7c003590 0x7fad7c005a10 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fad98002000 tx=0x7fad98016000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 0x7fada402a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada4072a80 con 0x7fada4097a40 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 0x7fada402a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient: get_auth_request con 0x7fada4097a40 auth_method 0 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 10 monclient(hunting): handle_auth_done global_id 74108 payload 824 2024-09-18T10:15:16.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fada4067800 con 0x7fada4097a40 2024-09-18T10:15:16.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 msgr2=0x7fada402a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.325+0000 7fada259c700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 0x7fada402a240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad98009ef0 tx=0x7fad98022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad98027760 con 0x7fada4097a40 2024-09-18T10:15:16.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 10 monclient: dump: 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fad980278c0 con 0x7fada4097a40 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 msgr2=0x7fada402a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 0x7fada402a240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad98009ef0 tx=0x7fad98022000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 msgr2=0x7fada402a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fad927fc700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fada4097a40 0x7fada402a240 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad98009ef0 tx=0x7fad98022000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada2d9d700 1 -- 172.21.15.142:0/2938738470 reap_dead start 2024-09-18T10:15:16.335 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:15:16.336 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.337 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.338 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:16.339 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.340 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:16.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:16.344 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.346 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:16.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.347 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.347 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:16.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.348 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.348 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.348 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:16.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.350 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:16.350 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:16.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 msgr2=0x7fad84037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 0x7fad84037f70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad94007fd0 tx=0x7fad94009040 comp rx=0 tx=0).stop 2024-09-18T10:15:16.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 10 monclient: shutdown 2024-09-18T10:15:16.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 msgr2=0x7fada4096960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 0x7fada4096960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fad9c007ea0 tx=0x7fad9c009850 comp rx=0 tx=0).stop 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 shutdown_connections 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad84035ac0 0x7fad84037f70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 --2- 172.21.15.142:0/2938738470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fada413bb60 0x7fada4096960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 >> 172.21.15.142:0/2938738470 conn(0x7fada40a5790 msgr2=0x7fada4087fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 shutdown_connections 2024-09-18T10:15:16.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.329+0000 7fada359e700 1 -- 172.21.15.142:0/2938738470 wait complete. 2024-09-18T10:15:16.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:16.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:15:16.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: get_monmap_and_config 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: monmap: 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:16.464785+0000 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:16.464785+0000 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.466 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:16.467 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:16.467 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:16.467 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:16.467 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:16.467 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 Processor -- start 2024-09-18T10:15:16.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 -- start start 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: init 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd71809a830 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd71809ac00 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd71813ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-b con 0x7fd7180992e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd718099d50 con 0x7fd71809ac00 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71809dca0 con 0x7fd71809a830 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd718099a70 con 0x7fd7180992e0 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71fca2700 10 monclient(hunting): authenticate will time out at 1205.390402s 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd71813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71d23d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd71809ac00 0x7fd71813b6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41540/0 (socket says 172.21.15.142:41540) 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71d23d700 1 -- 172.21.15.142:0/3105384102 learned_addr learned my addr 172.21.15.142:0/3105384102 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): get_auth_request con 0x7fd7180992e0 auth_method 0 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient(hunting): handle_auth_done global_id 74111 payload 274 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 1 -- 172.21.15.142:0/3105384102 >> v1:172.21.15.150:6789/0 conn(0x7fd71809a830 legacy=0x7fd718098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 1 -- 172.21.15.142:0/3105384102 >> v1:172.21.15.142:6789/0 conn(0x7fd71809ac00 legacy=0x7fd71813b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 1 monclient: found mon.noname-b 2024-09-18T10:15:16.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.461+0000 7fd71e23f700 1 -- 172.21.15.142:0/3105384102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71809a030 con 0x7fd7180992e0 2024-09-18T10:15:16.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71e23f700 1 -- 172.21.15.142:0/3105384102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 msgr2=0x7fd71813ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71e23f700 1 --2- 172.21.15.142:0/3105384102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd71813ee10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd710009910 tx=0x7fd7100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76b1fef764ee1c0a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 1 -- 172.21.15.142:0/3105384102 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71000ba70 con 0x7fd7180992e0 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: dump: 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 1 -- 172.21.15.142:0/3105384102 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71000bbd0 con 0x7fd7180992e0 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 5 monclient: authenticate success, global_id 74111 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 1 -- 172.21.15.142:0/3105384102 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710011420 con 0x7fd7180992e0 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 4 set_mon_vals no callback set 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71ca3c700 10 monclient: dump: 2024-09-18T10:15:16.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 10 monclient: shutdown 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 -- 172.21.15.142:0/3105384102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 msgr2=0x7fd71813ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 --2- 172.21.15.142:0/3105384102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd71813ee10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd710009910 tx=0x7fd7100091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 -- 172.21.15.142:0/3105384102 shutdown_connections 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 --2- 172.21.15.142:0/3105384102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd71813ee10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 -- 172.21.15.142:0/3105384102 >> 172.21.15.142:0/3105384102 conn(0x7fd7180a14e0 msgr2=0x7fd7180a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 -- 172.21.15.142:0/3105384102 shutdown_connections 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 1 -- 172.21.15.142:0/3105384102 wait complete. 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.465+0000 7fd71fca2700 10 monclient: monmap: 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 Processor -- start 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 -- start start 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient: init 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd718145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-c con 0x7fd7180992e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71809a830 0x7fd71813e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-b con 0x7fd71809a830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:16.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 0x7fd71806c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): picked mon.noname-a con 0x7fd71809ac00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718126a30 con 0x7fd71809ac00 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718094830 con 0x7fd71809a830 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7180a0560 con 0x7fd7180992e0 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71fca2700 10 monclient(hunting): authenticate will time out at 1205.397432s 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 0x7fd71806c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd718145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 0x7fd71806c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36212/0 (socket says 172.21.15.142:36212) 2024-09-18T10:15:16.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 learned_addr learned my addr 172.21.15.142:0/2436752620 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): get_auth_request con 0x7fd71809ac00 auth_method 0 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): get_auth_request con 0x7fd7180992e0 auth_method 0 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71809a830 0x7fd71813e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): get_auth_request con 0x7fd71809a830 auth_method 0 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71d23d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient(hunting): handle_auth_done global_id 54642 payload 824 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 msgr2=0x7fd718145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7180992e0 0x7fd718145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71809a830 msgr2=0x7fd71813e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71809a830 0x7fd71813e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 monclient: found mon.noname-a 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:16.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.469+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd710009420 con 0x7fd71809ac00 2024-09-18T10:15:16.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 msgr2=0x7fd71806c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71e23f700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 0x7fd71806c320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd710009860 tx=0x7fd710016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71001b680 con 0x7fd71809ac00 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.489 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 5 monclient: authenticate success, global_id 54642 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71001b7e0 con 0x7fd71809ac00 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 monclient: _renew_subs 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd718146780 con 0x7fd71809ac00 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71001bb80 con 0x7fd71809ac00 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd717fff700 4 set_mon_vals no callback set 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 monclient: _renew_subs 2024-09-18T10:15:16.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71806e2d0 con 0x7fd71809ac00 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 0x7fd71806c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd718064950 con 0x7fd71813eea0 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd7100193b0 con 0x7fd71809ac00 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 0x7fd71806c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient: get_auth_request con 0x7fd71813eea0 auth_method 0 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035af0 0x7fd704037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(181..181 src has 58..181) v4 ==== 7917+0+0 (secure 0 0 0) 0x7fd71004a9f0 con 0x7fd71809ac00 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 10 monclient(hunting): handle_auth_done global_id 15850 payload 824 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.473+0000 7fd71da3e700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7180696a0 con 0x7fd71813eea0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd71d23d700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035af0 0x7fd704037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd71d23d700 10 monclient: get_auth_request con 0x7fd704035af0 auth_method 0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd71da3e700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 msgr2=0x7fd71806c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd71da3e700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 0x7fd71806c860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f7b0 tx=0x7fd70c007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd71d23d700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035af0 0x7fd704037fa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd7181268f0 tx=0x7fd7080053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c014890 con 0x7fd71813eea0 2024-09-18T10:15:16.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd70c0149f0 con 0x7fd71813eea0 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 msgr2=0x7fd71806c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 0x7fd71806c860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f7b0 tx=0x7fd70c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 msgr2=0x7fd71806c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:16.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.477+0000 7fd715ffb700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71813eea0 0x7fd71806c860 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71fca2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71fca2700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 0x7fd718092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71fca2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71806d1a0 con 0x7fd71806cda0 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 0x7fd718092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient: get_auth_request con 0x7fd71806cda0 auth_method 0 2024-09-18T10:15:16.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.489+0000 7fd71da3e700 10 monclient(hunting): handle_auth_done global_id 15853 payload 824 2024-09-18T10:15:16.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd71da3e700 1 -- 172.21.15.142:0/2436752620 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd7180008d0 con 0x7fd71806cda0 2024-09-18T10:15:16.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd71da3e700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 msgr2=0x7fd718092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd71da3e700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 0x7fd718092800 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f7b0 tx=0x7fd70c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c01cb10 con 0x7fd71806cda0 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fd70c024070 con 0x7fd71806cda0 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:16.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 msgr2=0x7fd718092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 0x7fd718092800 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00f7b0 tx=0x7fd70c019000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 msgr2=0x7fd718092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.493+0000 7fd715ffb700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd71806cda0 0x7fd718092800 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:15:16.502 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.503 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.504 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:16.505 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:16.506 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.507 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.510 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.512 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.513 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:16.514 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:16.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035af0 msgr2=0x7fd704037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035af0 0x7fd704037fa0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd7181268f0 tx=0x7fd7080053c0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 10 monclient: shutdown 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 msgr2=0x7fd71806c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 --2- 172.21.15.142:0/2436752620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd71809ac00 0x7fd71806c320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd710009860 tx=0x7fd710016000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 reap_dead start 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71e23f700 1 -- 172.21.15.142:0/2436752620 reap_dead start 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 shutdown_connections 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 >> 172.21.15.142:0/2436752620 conn(0x7fd7180a14e0 msgr2=0x7fd71809dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 shutdown_connections 2024-09-18T10:15:16.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.497+0000 7fd71fca2700 1 -- 172.21.15.142:0/2436752620 wait complete. 2024-09-18T10:15:16.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:16.520 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:15:16.521 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:15:16.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: monmap: 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:16.632582+0000 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:16.632582+0000 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:16.634 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:16.635 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:16.635 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 Processor -- start 2024-09-18T10:15:16.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 -- start start 2024-09-18T10:15:16.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: init 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5c1c09d920 0x7f5c1c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5c1c09d920 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c1c0953c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c1c095800 0x7f5c1c0973c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5c1c095800 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c09b0b0 con 0x7f5c1c0953c0 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c06fba0 con 0x7f5c1c09d920 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c09a7e0 con 0x7f5c1c095800 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1b59e700 10 monclient(hunting): authenticate will time out at 1205.558126s 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5c1c09d920 0x7f5c1c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5c1c09d920 0x7f5c1c06f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36228/0 (socket says 172.21.15.142:36228) 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 -- 172.21.15.142:0/1833619566 learned_addr learned my addr 172.21.15.142:0/1833619566 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): get_auth_request con 0x7f5c1c09d920 auth_method 0 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 1 --2- 172.21.15.142:0/1833619566 >> v2:172.21.15.150:3300/0 conn(0x7f5c1c095800 0x7f5c1c0973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): get_auth_request con 0x7f5c1c095800 auth_method 0 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_done global_id 54645 payload 274 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 -- 172.21.15.142:0/1833619566 >> v2:172.21.15.150:3300/0 conn(0x7f5c1c095800 msgr2=0x7f5c1c0973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 --2- 172.21.15.142:0/1833619566 >> v2:172.21.15.150:3300/0 conn(0x7f5c1c095800 0x7f5c1c0973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 -- 172.21.15.142:0/1833619566 >> v1:172.21.15.150:6789/0 conn(0x7f5c1c0953c0 legacy=0x7f5c1c096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 monclient: found mon.noname-a 2024-09-18T10:15:16.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:16.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.629+0000 7f5c1a59c700 1 -- 172.21.15.142:0/1833619566 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c097900 con 0x7f5c1c09d920 2024-09-18T10:15:16.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1a59c700 1 -- 172.21.15.142:0/1833619566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 msgr2=0x7f5c1c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1a59c700 1 --2- 172.21.15.142:0/1833619566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c06f660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c099a50 tx=0x7f5c0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2aba03df8b395f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 1 -- 172.21.15.142:0/1833619566 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c009e70 con 0x7f5c1c09d920 2024-09-18T10:15:16.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: dump: 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 5 monclient: authenticate success, global_id 54645 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 1 -- 172.21.15.142:0/1833619566 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c00ba70 con 0x7f5c1c09d920 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 4 set_mon_vals no callback set 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 1 -- 172.21.15.142:0/1833619566 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c00be30 con 0x7f5c1c09d920 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1959a700 10 monclient: dump: 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.645 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 10 monclient: shutdown 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1833619566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 msgr2=0x7f5c1c06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1833619566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c06f660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c099a50 tx=0x7f5c0c009250 comp rx=0 tx=0).stop 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1833619566 shutdown_connections 2024-09-18T10:15:16.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1833619566 >> v2:172.21.15.150:3300/0 conn(0x7f5c1c095800 0x7f5c1c0973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1833619566 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c06f660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1833619566 >> 172.21.15.142:0/1833619566 conn(0x7f5c1c0a1fe0 msgr2=0x7f5c1c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1833619566 shutdown_connections 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1833619566 wait complete. 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 10 monclient: monmap: 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.647 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 Processor -- start 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 1 -- start start 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.633+0000 7f5c1b59e700 10 monclient: init 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1c0953c0 0x7f5c1c1418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5c1c0953c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5c1c095800 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c08dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5c1c09d920 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0a0960 con 0x7f5c1c09d920 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0922d0 con 0x7f5c1c0953c0 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c091450 con 0x7f5c1c095800 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1b59e700 10 monclient(hunting): authenticate will time out at 1205.564736s 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38930/0 (socket says 172.21.15.142:38930) 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 -- 172.21.15.142:0/1323853160 learned_addr learned my addr 172.21.15.142:0/1323853160 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): get_auth_request con 0x7f5c1c095800 auth_method 0 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c08dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1c0953c0 0x7f5c1c1418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): get_auth_request con 0x7f5c1c0953c0 auth_method 0 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): get_auth_request con 0x7f5c1c09d920 auth_method 0 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient(hunting): handle_auth_done global_id 74117 payload 824 2024-09-18T10:15:16.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 -- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1c0953c0 msgr2=0x7f5c1c1418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1c0953c0 0x7f5c1c1418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 -- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 msgr2=0x7f5c1c08dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c08dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 monclient: found mon.noname-c 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 -- 172.21.15.142:0/1323853160 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c0094f0 con 0x7f5c1c095800 2024-09-18T10:15:16.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 -- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 msgr2=0x7f5c1c141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c19d9b700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400c950 tx=0x7f5c04002760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.637+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04012840 con 0x7f5c1c095800 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: dump: 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.655 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 5 monclient: authenticate success, global_id 74117 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c040129c0 con 0x7f5c1c095800 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c04012d80 con 0x7f5c1c095800 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _renew_subs 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c0901e0 con 0x7f5c1c095800 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c18d99700 4 set_mon_vals no callback set 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:16.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 10 monclient: dump: 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _renew_subs 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c091140 con 0x7f5c1c095800 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c0915d0 con 0x7f5c1c095800 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5c04011590 con 0x7f5c1c095800 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 0x7f5bfc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(181..181 src has 58..181) v4 ==== 7917+0+0 (secure 0 0 0) 0x7f5c04049410 con 0x7f5c1c095800 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1a59c700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 0x7f5bfc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.641+0000 7f5c1a59c700 10 monclient: get_auth_request con 0x7f5bfc035ac0 auth_method 0 2024-09-18T10:15:16.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.645+0000 7f5c1a59c700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 0x7f5bfc037f70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c099a50 tx=0x7f5c0c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.649+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 moved item id -19 name 'barr' to location {root=default} in crush map v182) v1 ==== 173+0+0 (secure 0 0 0) 0x7f673411d260 con 0x7f673409e1e0 2024-09-18T10:15:16.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.649+0000 7f672bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.649+0000 7f672bfff700 10 monclient: _finish_command 2 = system:0 moved item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T10:15:16.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.649+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5c04022d70 con 0x7f5c1c095800 2024-09-18T10:15:16.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.649+0000 7f5c127fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.649+0000 7f5c127fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f673a41c700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f673a41c700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f673a41c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "barr", "args": ["root=default"]} v 0) v1 -- 0x7f673411d260 con 0x7f673409e1e0 2024-09-18T10:15:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f672bfff700 1 -- 172.21.15.150:0/1032715307 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}]=0 no need to move item id -19 name 'barr' to location {root=default} in crush map v182) v1 ==== 183+0+0 (secure 0 0 0) 0x7f673411d260 con 0x7f673409e1e0 2024-09-18T10:15:16.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f672bfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "barr", "args": ["root=default"]}] 2024-09-18T10:15:16.768 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.761+0000 7f672bfff700 10 monclient: _finish_command 3 = system:0 no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 msgr2=0x7f6714037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 0x7f6714037f70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f67240099f0 tx=0x7f672400b040 comp rx=0 tx=0).stop 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 10 monclient: shutdown 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 msgr2=0x7f673411b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c017000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 shutdown_connections 2024-09-18T10:15:16.771 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6714035ac0 0x7f6714037f70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f673409e1e0 0x7f673411b460 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f67340994f0 0x7f673411af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 --2- 172.21.15.150:0/1032715307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6734071240 0x7f673411a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 >> 172.21.15.150:0/1032715307 conn(0x7f67340a1cc0 msgr2=0x7f673409f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 shutdown_connections 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.765+0000 7f673a41c700 1 -- 172.21.15.150:0/1032715307 wait complete. 2024-09-18T10:15:16.772 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -19 name 'barr' to location {root=default} in crush map 2024-09-18T10:15:16.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush move fooo rack=barr 2024-09-18T10:15:16.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.869+0000 7f5c1b59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:15:16.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.869+0000 7f5c1b59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:15:16.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.869+0000 7f5c1b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:16.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.869+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f5c1c08f700 con 0x7f5c1c095800 2024-09-18T10:15:16.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.873+0000 7f5c127fc700 1 -- 172.21.15.142:0/1323853160 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f5c1c08f700 con 0x7f5c1c095800 2024-09-18T10:15:16.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.873+0000 7f5c127fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:15:16.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.873+0000 7f5c127fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:16.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3521628468,config=3140514009,health=1186326501,kv=4261215878,logm=3330676585}) 2024-09-18T10:15:16.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=3521628468,config=3140514009,health=2296550598,kv=4261215878,logm=3444631705}) 2024-09-18T10:15:16.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3521628468,config=3140514009,health=1186326501,kv=4261215878,logm=3330676585}) 2024-09-18T10:15:16.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.873+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=8,logm=64} crc {auth=3521628468,config=3140514009,health=1186326500,kv=4261215878,logm=3991046764}) 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.873+0000 7fb76f369700 10 monclient: get_monmap_and_config 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.873+0000 7fb76f369700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.873+0000 7fb76f369700 10 monclient: monmap: 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:16.879655+0000 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:16.879655+0000 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.883 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 Processor -- start 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 -- start start 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient: init 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb76809c930 0x7fb7680977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-a con 0x7fb76809c930 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb76809cd00 0x7fb76809d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-c con 0x7fb76809cd00 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb768136760 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb768098f00 con 0x7fb768136760 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb768097d00 con 0x7fb76809c930 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb768097030 con 0x7fb76809cd00 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76f369700 10 monclient(hunting): authenticate will time out at 1198.973320s 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb76809cd00 0x7fb76809d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb76809cd00 0x7fb76809d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53262/0 (socket says 172.21.15.150:53262) 2024-09-18T10:15:16.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb76809c930 0x7fb7680977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 -- 172.21.15.150:0/943552757 learned_addr learned my addr 172.21.15.150:0/943552757 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): get_auth_request con 0x7fb76809cd00 auth_method 0 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): get_auth_request con 0x7fb76809c930 auth_method 0 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient(hunting): handle_auth_done global_id 15862 payload 274 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 -- 172.21.15.150:0/943552757 >> v2:172.21.15.142:3300/0 conn(0x7fb76809c930 msgr2=0x7fb7680977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 --2- 172.21.15.150:0/943552757 >> v2:172.21.15.142:3300/0 conn(0x7fb76809c930 0x7fb7680977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 -- 172.21.15.150:0/943552757 >> v1:172.21.15.142:6789/0 conn(0x7fb768136760 legacy=0x7fb768138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 monclient: found mon.noname-c 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.877+0000 7fb76c904700 1 -- 172.21.15.150:0/943552757 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76806f730 con 0x7fb76809cd00 2024-09-18T10:15:16.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 msgr2=0x7f5bfc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 0x7f5bfc037f70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c099a50 tx=0x7f5c0c016000 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 10 monclient: shutdown 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 msgr2=0x7f5c1c141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400c950 tx=0x7f5c04002760 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 shutdown_connections 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5bfc035ac0 0x7f5bfc037f70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c1c09d920 0x7f5c1c08dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c1c095800 0x7f5c1c141de0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 --2- 172.21.15.142:0/1323853160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c1c0953c0 0x7f5c1c1418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 >> 172.21.15.142:0/1323853160 conn(0x7f5c1c0a1fe0 msgr2=0x7f5c1c099160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 shutdown_connections 2024-09-18T10:15:16.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:16.877+0000 7f5c1b59e700 1 -- 172.21.15.142:0/1323853160 wait complete. 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238107132}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773234854}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238107132}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=710172487}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660035912}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622966683}) 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660035912}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.881+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955421772}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80433390}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3275370002}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80433390}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.885+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975635730}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330865962}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3197017488}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330865962}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335361665}) 2024-09-18T10:15:16.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719940588}) 2024-09-18T10:15:16.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460751860}) 2024-09-18T10:15:16.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719940588}) 2024-09-18T10:15:16.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736176388}) 2024-09-18T10:15:16.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606332159}) 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.889+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800108617}) 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.893+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.893+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606332159}) 2024-09-18T10:15:16.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.893+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598700419}) 2024-09-18T10:15:16.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76c904700 1 -- 172.21.15.150:0/943552757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809cd00 msgr2=0x7fb76809d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76c904700 1 --2- 172.21.15.150:0/943552757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809cd00 0x7fb76809d100 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb768071200 tx=0x7fb764009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9adb09ad5946ff0a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 1 -- 172.21.15.150:0/943552757 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764009e70 con 0x7fb76809cd00 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: dump: 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.903 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 5 monclient: authenticate success, global_id 15862 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 1 -- 172.21.15.150:0/943552757 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76400ba70 con 0x7fb76809cd00 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 4 set_mon_vals no callback set 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 1 -- 172.21.15.150:0/943552757 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76400be30 con 0x7fb76809cd00 2024-09-18T10:15:16.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb75ffff700 10 monclient: dump: 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 10 monclient: shutdown 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 -- 172.21.15.150:0/943552757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809cd00 msgr2=0x7fb76809d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 --2- 172.21.15.150:0/943552757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809cd00 0x7fb76809d100 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb768071200 tx=0x7fb764009250 comp rx=0 tx=0).stop 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 -- 172.21.15.150:0/943552757 shutdown_connections 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 --2- 172.21.15.150:0/943552757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809cd00 0x7fb76809d100 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 --2- 172.21.15.150:0/943552757 >> v2:172.21.15.142:3300/0 conn(0x7fb76809c930 0x7fb7680977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 -- 172.21.15.150:0/943552757 >> 172.21.15.150:0/943552757 conn(0x7fb768068360 msgr2=0x7fb7680a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 -- 172.21.15.150:0/943552757 shutdown_connections 2024-09-18T10:15:16.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 1 -- 172.21.15.150:0/943552757 wait complete. 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 10 monclient: build_initial_monmap 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.881+0000 7fb76f369700 10 monclient: monmap: 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.907 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 Processor -- start 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 -- start start 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient: init 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:16.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-b con 0x7fb76809c930 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb76809cd00 0x7fb76811f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-a con 0x7fb76809cd00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb768136760 0x7fb76811fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): picked mon.noname-c con 0x7fb768136760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76809e3f0 con 0x7fb76809cd00 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76809bbe0 con 0x7fb76809c930 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb768122710 con 0x7fb768136760 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76f369700 10 monclient(hunting): authenticate will time out at 1198.981127s 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53264/0 (socket says 172.21.15.150:53264) 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 -- 172.21.15.150:0/965908628 learned_addr learned my addr 172.21.15.150:0/965908628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): get_auth_request con 0x7fb76809c930 auth_method 0 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb768136760 0x7fb76811fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76c904700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb76809cd00 0x7fb76811f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): get_auth_request con 0x7fb768136760 auth_method 0 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76c904700 10 monclient(hunting): get_auth_request con 0x7fb76809cd00 auth_method 0 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76c904700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76c904700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76c904700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:16.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d906700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient(hunting): handle_auth_done global_id 15865 payload 824 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 -- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb768136760 msgr2=0x7fb76811fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb768136760 0x7fb76811fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 -- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb76809cd00 msgr2=0x7fb76811f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb76809cd00 0x7fb76811f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient: _finish_hunting 0 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 monclient: found mon.noname-b 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.885+0000 7fb76d105700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7640094f0 con 0x7fb76809c930 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76d105700 1 -- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 msgr2=0x7fb76811f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:16.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76d105700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e930 tx=0x7fb7580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758013890 con 0x7fb76809c930 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: dump: 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.913 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 5 monclient: authenticate success, global_id 15865 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7580139f0 con 0x7fb76809c930 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _renew_subs 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb768122430 con 0x7fb76809c930 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758013db0 con 0x7fb76809c930 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 10 monclient: dump: 2024-09-18T10:15:16.914 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75f7fe700 4 set_mon_vals no callback set 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _renew_subs 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb768121540 con 0x7fb76809c930 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb768063ed0 con 0x7fb76809c930 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb75800c610 con 0x7fb76809c930 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 0x7fb74c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(182..182 src has 58..182) v4 ==== 8105+0+0 (secure 0 0 0) 0x7fb758049290 con 0x7fb76809c930 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76c904700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 0x7fb74c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.889+0000 7fb76c904700 10 monclient: get_auth_request con 0x7fb74c035ac0 auth_method 0 2024-09-18T10:15:16.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.893+0000 7fb76c904700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 0x7fb74c037f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb768071200 tx=0x7fb764016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:16.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.897+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb758023ad0 con 0x7fb76809c930 2024-09-18T10:15:16.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.897+0000 7fb75d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:16.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:16.897+0000 7fb75d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837788157}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240142692}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837788157}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.897+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780682109}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358510022}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330454190}) 2024-09-18T10:15:16.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358510022}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748803281}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728737944}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2845331247}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728737944}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.901+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590462131}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482739133}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835818359}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482739133}) 2024-09-18T10:15:16.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991026505}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065632132}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908476101}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065632132}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.905+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952031730}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534603099}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896006350}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534603099}) 2024-09-18T10:15:16.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000760880}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258651893}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474965266}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258651893}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376125981}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283471680}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401594152}) 2024-09-18T10:15:16.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283471680}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.909+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745088580}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384773570}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2188186170}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1384773570}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441468675}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342525100}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244537533}) 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342525100}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011708602}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338187866}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250840811}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338187866}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.913+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979469063}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068169242}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570313203}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068169242}) 2024-09-18T10:15:16.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251306950}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552609024}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394115210}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552609024}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655889095}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116430306}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468027056}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116430306}) 2024-09-18T10:15:16.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815130610}) 2024-09-18T10:15:16.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567643824}) 2024-09-18T10:15:16.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792108860}) 2024-09-18T10:15:16.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567643824}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.917+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719063141}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892428432}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139830813}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892428432}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645902249}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399873720}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027267845}) 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399873720}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186141896}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924808181}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913091979}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924808181}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.921+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835581302}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737501404}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512264477}) 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737501404}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233841698}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713395042}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2658874388}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713395042}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938515421}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112012116}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3249814446}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112012116}) 2024-09-18T10:15:16.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860184542}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112911229}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468712152}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112911229}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.925+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3135432488}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2342799524,mdsmap=1531151339}) 2024-09-18T10:15:16.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=140771603,mdsmap=3585761405}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2342799524,mdsmap=1531151339}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1134781742,mdsmap=1531151339,mgr=3964223516}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=62} crc {mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2684201678,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=413841784,osd_snap=454132730,osdmap=1715849316}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=63} crc {mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2684201678,mon_config_key=620546692,monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=413841784,osd_snap=454132730,osdmap=4284290278}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=62} crc {mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2684201678,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=413841784,osd_snap=454132730,osdmap=1715849316}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.929+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=378242252,mon_config_key=2836485344,monmap=3505902565,osd_metadata=1788507861,osd_pg_creating=413841784,osd_snap=454132730,osdmap=611877542}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3556666175}) 2024-09-18T10:15:16.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1000570547}) 2024-09-18T10:15:16.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3556666175}) 2024-09-18T10:15:16.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.933+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1249341823}) 2024-09-18T10:15:16.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=878615293}) 2024-09-18T10:15:16.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=35} crc {osdmap=1649405395}) 2024-09-18T10:15:16.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:15:16.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=878615293}) 2024-09-18T10:15:16.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:15:16.937+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=35} crc {osdmap=2231449073}) 2024-09-18T10:15:17.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.065+0000 7fb76f369700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.065+0000 7fb76f369700 10 monclient: _send_command 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.065+0000 7fb76f369700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:17.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.065+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fb768071930 con 0x7fb76809c930 2024-09-18T10:15:17.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.869+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 moved item id -16 name 'fooo' to location {rack=barr} in crush map v183) v1 ==== 167+0+0 (secure 0 0 0) 0x7fb7580232f0 con 0x7fb76809c930 2024-09-18T10:15:17.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.869+0000 7fb75d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.869+0000 7fb75d7fa700 10 monclient: _finish_command 2 = system:0 moved item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T10:15:17.901 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:15:17.901 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:15:17.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.977+0000 7fb76f369700 10 start_mon_command cmd=[{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.977+0000 7fb76f369700 10 monclient: _send_command 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.977+0000 7fb76f369700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:17.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.977+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]} v 0) v1 -- 0x7fb76811ffd0 con 0x7fb76809c930 2024-09-18T10:15:17.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb75d7fa700 1 -- 172.21.15.150:0/965908628 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}]=0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map v183) v1 ==== 177+0+0 (secure 0 0 0) 0x7fb7580234d0 con 0x7fb76809c930 2024-09-18T10:15:17.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb75d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush move", "name": "fooo", "args": ["rack=barr"]}] 2024-09-18T10:15:17.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb75d7fa700 10 monclient: _finish_command 3 = system:0 no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T10:15:17.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 msgr2=0x7fb74c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:17.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 0x7fb74c037f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb768071200 tx=0x7fb764016000 comp rx=0 tx=0).stop 2024-09-18T10:15:17.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 10 monclient: shutdown 2024-09-18T10:15:17.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:17.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 msgr2=0x7fb76811f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:17.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e930 tx=0x7fb7580091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:17.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 shutdown_connections 2024-09-18T10:15:17.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb74c035ac0 0x7fb74c037f70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb768136760 0x7fb76811fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb76809cd00 0x7fb76811f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 --2- 172.21.15.150:0/965908628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb76809c930 0x7fb76811f010 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 >> 172.21.15.150:0/965908628 conn(0x7fb768068360 msgr2=0x7fb76809a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 shutdown_connections 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:17.981+0000 7fb76f369700 1 -- 172.21.15.150:0/965908628 wait complete. 2024-09-18T10:15:17.989 INFO:tasks.workunit.client.0.smithi150.stderr:no need to move item id -16 name 'fooo' to location {rack=barr} in crush map 2024-09-18T10:15:17.998 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm fooo 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: monmap: 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.005 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:18.003967+0000 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:18.003967+0000 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:18.006 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 Processor -- start 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 -- start start 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: init 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c0a2f20 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9d0c09ced0 0x7f9d0c09d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c09ced0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d0c09d810 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:18.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0c0a32f0 con 0x7f9d0c09d810 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c09abd0 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0c099fe0 con 0x7f9d0c09ced0 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0b59e700 10 monclient(hunting): authenticate will time out at 1206.930960s 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36254/0 (socket says 172.21.15.142:36254) 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 -- 172.21.15.142:0/2499265305 learned_addr learned my addr 172.21.15.142:0/2499265305 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): get_auth_request con 0x7f9d0c0a2f20 auth_method 0 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 1 --2- 172.21.15.142:0/2499265305 >> v2:172.21.15.150:3300/0 conn(0x7f9d0c09ced0 0x7f9d0c09d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): get_auth_request con 0x7f9d0c09ced0 auth_method 0 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_done global_id 54657 payload 274 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 -- 172.21.15.142:0/2499265305 >> v2:172.21.15.150:3300/0 conn(0x7f9d0c09ced0 msgr2=0x7f9d0c09d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 --2- 172.21.15.142:0/2499265305 >> v2:172.21.15.150:3300/0 conn(0x7f9d0c09ced0 0x7f9d0c09d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 -- 172.21.15.142:0/2499265305 >> v1:172.21.15.150:6789/0 conn(0x7f9d0c09d810 legacy=0x7f9d0c135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 monclient: found mon.noname-a 2024-09-18T10:15:18.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:18.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.001+0000 7f9d0a59c700 1 -- 172.21.15.142:0/2499265305 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c09a250 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0a59c700 1 -- 172.21.15.142:0/2499265305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 msgr2=0x7f9d0c099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0a59c700 1 --2- 172.21.15.142:0/2499265305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c06b440 tx=0x7f9cfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc022f90b713fcc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 1 -- 172.21.15.142:0/2499265305 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc009ea0 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: dump: 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.018 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 5 monclient: authenticate success, global_id 54657 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 1 -- 172.21.15.142:0/2499265305 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc00ba30 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 4 set_mon_vals no callback set 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 1 -- 172.21.15.142:0/2499265305 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00bdf0 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d09d9b700 10 monclient: dump: 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 10 monclient: shutdown 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 -- 172.21.15.142:0/2499265305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 msgr2=0x7f9d0c099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/2499265305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c06b440 tx=0x7f9cfc009250 comp rx=0 tx=0).stop 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 -- 172.21.15.142:0/2499265305 shutdown_connections 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/2499265305 >> v2:172.21.15.150:3300/0 conn(0x7f9d0c09ced0 0x7f9d0c09d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/2499265305 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c099aa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 -- 172.21.15.142:0/2499265305 >> 172.21.15.142:0/2499265305 conn(0x7f9d0c06a2e0 msgr2=0x7f9d0c0a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 -- 172.21.15.142:0/2499265305 shutdown_connections 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.005+0000 7f9d0b59e700 1 -- 172.21.15.142:0/2499265305 wait complete. 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient: monmap: 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:18.021 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 Processor -- start 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 -- start start 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient: init 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:18.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d0c09ced0 0x7f9d0c08b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9d0c09ced0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9d0c09d810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c088960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9d0c0a2f20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c08a930 con 0x7f9d0c0a2f20 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c09b8c0 con 0x7f9d0c09ced0 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c08c1b0 con 0x7f9d0c09d810 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0b59e700 10 monclient(hunting): authenticate will time out at 1206.938306s 2024-09-18T10:15:18.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38932/0 (socket says 172.21.15.142:38932) 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 -- 172.21.15.142:0/3749840884 learned_addr learned my addr 172.21.15.142:0/3749840884 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c088960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): get_auth_request con 0x7f9d0c09d810 auth_method 0 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d0c09ced0 0x7f9d0c08b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): get_auth_request con 0x7f9d0c09ced0 auth_method 0 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): get_auth_request con 0x7f9d0c0a2f20 auth_method 0 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d0ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient(hunting): handle_auth_done global_id 74123 payload 824 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 -- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d0c09ced0 msgr2=0x7f9d0c08b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d0c09ced0 0x7f9d0c08b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 -- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 msgr2=0x7f9d0c088960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c088960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.009+0000 7f9d03fff700 1 -- 172.21.15.142:0/3749840884 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0094f0 con 0x7f9d0c09d810 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d03fff700 1 -- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 msgr2=0x7f9d0c093fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d03fff700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400ece0 tx=0x7f9cf40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400ca60 con 0x7f9d0c09d810 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: dump: 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.028 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf400cbc0 con 0x7f9d0c09d810 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 5 monclient: authenticate success, global_id 74123 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _renew_subs 2024-09-18T10:15:18.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c08a410 con 0x7f9d0c09d810 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4013470 con 0x7f9d0c09d810 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 10 monclient: dump: 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.030 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0959a700 4 set_mon_vals no callback set 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _renew_subs 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c094510 con 0x7f9d0c09d810 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9cf40137e0 con 0x7f9d0c09d810 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c072a80 con 0x7f9d0c09d810 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 0x7f9cec037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(183..183 src has 58..183) v4 ==== 8105+0+0 (secure 0 0 0) 0x7f9cf4049330 con 0x7f9d0c09d810 2024-09-18T10:15:18.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0a59c700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 0x7f9cec037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.013+0000 7f9d0a59c700 10 monclient: get_auth_request con 0x7f9cec035ac0 auth_method 0 2024-09-18T10:15:18.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.017+0000 7f9d0a59c700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 0x7f9cec037f70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c06b440 tx=0x7f9cfc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.021+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9cf40234d0 con 0x7f9d0c09d810 2024-09-18T10:15:18.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.021+0000 7f9d027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.021+0000 7f9d027fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:18.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: get_monmap_and_config 2024-09-18T10:15:18.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: monmap: 2024-09-18T10:15:18.101 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:18.100178+0000 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:18.100178+0000 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.102 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:18.103 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:18.103 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:18.103 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:18.103 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:18.103 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 Processor -- start 2024-09-18T10:15:18.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 -- start start 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: init 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6260071240 0x7f6260096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-b con 0x7f6260071240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f626009e1e0 0x7f6260096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-c con 0x7f626009e1e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f62600994f0 0x7f62600974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-a con 0x7f62600994f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626009e640 con 0x7f62600994f0 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6260098cf0 con 0x7f626009e1e0 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626009df80 con 0x7f6260071240 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6267d18700 10 monclient(hunting): authenticate will time out at 1200.194689s 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f626009e1e0 0x7f6260096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f626009e1e0 0x7f6260096fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53272/0 (socket says 172.21.15.150:53272) 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 -- 172.21.15.150:0/1345667044 learned_addr learned my addr 172.21.15.150:0/1345667044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): get_auth_request con 0x7f626009e1e0 auth_method 0 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62662b5700 1 --2- 172.21.15.150:0/1345667044 >> v2:172.21.15.142:3300/0 conn(0x7f62600994f0 0x7f62600974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6265ab4700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6260071240 0x7f6260096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6265ab4700 10 monclient(hunting): get_auth_request con 0x7f6260071240 auth_method 0 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6265ab4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6265ab4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f6265ab4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62662b5700 10 monclient(hunting): get_auth_request con 0x7f62600994f0 auth_method 0 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62662b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62662b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62662b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient(hunting): handle_auth_done global_id 15874 payload 274 2024-09-18T10:15:18.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 -- 172.21.15.150:0/1345667044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6260071240 msgr2=0x7f6260096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6260071240 0x7f6260096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 -- 172.21.15.150:0/1345667044 >> v2:172.21.15.142:3300/0 conn(0x7f62600994f0 msgr2=0x7f62600974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 --2- 172.21.15.150:0/1345667044 >> v2:172.21.15.142:3300/0 conn(0x7f62600994f0 0x7f62600974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.097+0000 7f62652b3700 1 -- 172.21.15.150:0/1345667044 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260098a10 con 0x7f626009e1e0 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f62652b3700 1 -- 172.21.15.150:0/1345667044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f6260096fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f62652b3700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f6260096fb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f626006a7e0 tx=0x7f6250009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ed2887064adc5c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 1 -- 172.21.15.150:0/1345667044 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625000ba70 con 0x7f626009e1e0 2024-09-18T10:15:18.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: dump: 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.115 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 5 monclient: authenticate success, global_id 15874 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 1 -- 172.21.15.150:0/1345667044 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625000bbd0 con 0x7f626009e1e0 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 4 set_mon_vals no callback set 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 1 -- 172.21.15.150:0/1345667044 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250011420 con 0x7f626009e1e0 2024-09-18T10:15:18.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6264ab2700 10 monclient: dump: 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.117 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.118 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 10 monclient: shutdown 2024-09-18T10:15:18.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 -- 172.21.15.150:0/1345667044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f6260096fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f6260096fb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f626006a7e0 tx=0x7f6250009250 comp rx=0 tx=0).stop 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 -- 172.21.15.150:0/1345667044 shutdown_connections 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 --2- 172.21.15.150:0/1345667044 >> v2:172.21.15.142:3300/0 conn(0x7f62600994f0 0x7f62600974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f6260096fb0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 --2- 172.21.15.150:0/1345667044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6260071240 0x7f6260096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 -- 172.21.15.150:0/1345667044 >> 172.21.15.150:0/1345667044 conn(0x7f62600a1cc0 msgr2=0x7f62600a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 -- 172.21.15.150:0/1345667044 shutdown_connections 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.101+0000 7f6267d18700 1 -- 172.21.15.150:0/1345667044 wait complete. 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient: monmap: 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:18.119 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:18.120 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 Processor -- start 2024-09-18T10:15:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 -- start start 2024-09-18T10:15:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient: init 2024-09-18T10:15:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f626009c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-a con 0x7f6260071240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f626011dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-c con 0x7f62600994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): picked mon.noname-b con 0x7f626009e1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62601351b0 con 0x7f6260071240 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626011f1e0 con 0x7f626009e1e0 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62600980f0 con 0x7f62600994f0 2024-09-18T10:15:18.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6267d18700 10 monclient(hunting): authenticate will time out at 1200.202416s 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f626009c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f626011dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f626009c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:37874/0 (socket says 172.21.15.150:37874) 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53286/0 (socket says 172.21.15.150:53286) 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 1 -- 172.21.15.150:0/3850018861 learned_addr learned my addr 172.21.15.150:0/3850018861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): get_auth_request con 0x7f6260071240 auth_method 0 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): get_auth_request con 0x7f626009e1e0 auth_method 0 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): get_auth_request con 0x7f62600994f0 auth_method 0 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient(hunting): handle_auth_done global_id 15877 payload 824 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 msgr2=0x7f626011dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f626011dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 msgr2=0x7f626009c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f626009c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 monclient: found mon.noname-b 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.105+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500094f0 con 0x7f626009e1e0 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f626011e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6258007ed0 tx=0x7f6258009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625800eda0 con 0x7f626009e1e0 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: dump: 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.129 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 5 monclient: authenticate success, global_id 15877 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6258014420 con 0x7f626009e1e0 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _renew_subs 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626009b120 con 0x7f626009e1e0 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62580147e0 con 0x7f626009e1e0 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 10 monclient: dump: 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.131 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6257fff700 4 set_mon_vals no callback set 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _renew_subs 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626009d1f0 con 0x7f626009e1e0 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6260120830 con 0x7f626009e1e0 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6258007ed0 tx=0x7f6258009850 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f62580494e0 osd_map(183..183 src has 58..183) v4 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6258003020 con 0x7f626009e1e0 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.109+0000 7f6255ffb700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 0x7f6244037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.113+0000 7f6265ab4700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 0x7f6244037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.113+0000 7f6265ab4700 10 monclient: get_auth_request con 0x7f6244035ac0 auth_method 0 2024-09-18T10:15:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.113+0000 7f6265ab4700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 0x7f6244037f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f625c006fd0 tx=0x7f625c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.121+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(183..183 src has 58..183) v4 ==== 8105+0+0 (secure 0 0 0) 0x7f62580494e0 con 0x7f626009e1e0 2024-09-18T10:15:18.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d0b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d0b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d0b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d0c067950 con 0x7f9d0c09d810 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d027fc700 1 -- 172.21.15.142:0/3749840884 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9cf40232f0 con 0x7f9d0c09d810 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:18.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.249+0000 7f9d027fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:18.257 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 msgr2=0x7f9cec037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 0x7f9cec037f70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c06b440 tx=0x7f9cfc016000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 10 monclient: shutdown 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 msgr2=0x7f9d0c093fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400ece0 tx=0x7f9cf40091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 shutdown_connections 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9cec035ac0 0x7f9cec037f70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d0c0a2f20 0x7f9d0c088960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d0c09d810 0x7f9d0c093fd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 --2- 172.21.15.142:0/3749840884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d0c09ced0 0x7f9d0c08b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 >> 172.21.15.142:0/3749840884 conn(0x7f9d0c06a2e0 msgr2=0x7f9d0c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 shutdown_connections 2024-09-18T10:15:18.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.253+0000 7f9d0b59e700 1 -- 172.21.15.142:0/3749840884 wait complete. 2024-09-18T10:15:18.270 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:15:18.270 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:15:18.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:15:18.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:15:18.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 10 monclient: get_monmap_and_config 2024-09-18T10:15:18.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 10 monclient: monmap: 2024-09-18T10:15:18.375 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:18.373669+0000 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:18.373669+0000 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.376 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:18.377 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:18.377 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:18.377 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:18.377 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:18.377 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 1 Processor -- start 2024-09-18T10:15:18.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 1 -- start start 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.369+0000 7fb1e6e86700 10 monclient: init 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 0x7fb1e00991f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-b con 0x7fb1e00a2620 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1e009dd40 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1e0097160 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:18.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e00a29f0 con 0x7fb1e009dd40 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e0099730 con 0x7fb1e0097160 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00983f0 con 0x7fb1e00a2620 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e6e86700 10 monclient(hunting): authenticate will time out at 1207.300753s 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 0x7fb1e00991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5683700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb1e009dd40 0x7fb1e0096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41556/0 (socket says 172.21.15.142:41556) 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5683700 1 -- 172.21.15.142:0/2396040110 learned_addr learned my addr 172.21.15.142:0/2396040110 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request con 0x7fb1e00a2620 auth_method 0 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_done global_id 74132 payload 274 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 -- 172.21.15.142:0/2396040110 >> v1:172.21.15.150:6789/0 conn(0x7fb1e0097160 legacy=0x7fb1e0136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 -- 172.21.15.142:0/2396040110 >> v1:172.21.15.142:6789/0 conn(0x7fb1e009dd40 legacy=0x7fb1e0096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 monclient: found mon.noname-b 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 -- 172.21.15.142:0/2396040110 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e009e500 con 0x7fb1e00a2620 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 -- 172.21.15.142:0/2396040110 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 msgr2=0x7fb1e00991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.373+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/2396040110 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 0x7fb1e00991f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009e90 tx=0x7fb1d0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863e1b88960f317 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 1 -- 172.21.15.142:0/2396040110 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000e070 con 0x7fb1e00a2620 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: dump: 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 1 -- 172.21.15.142:0/2396040110 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d0012430 con 0x7fb1e00a2620 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 1 -- 172.21.15.142:0/2396040110 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d00127d0 con 0x7fb1e00a2620 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 5 monclient: authenticate success, global_id 74132 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 4 set_mon_vals no callback set 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e4e82700 10 monclient: dump: 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.396 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: shutdown 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- 172.21.15.142:0/2396040110 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 msgr2=0x7fb1e00991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/2396040110 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 0x7fb1e00991f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009e90 tx=0x7fb1d0009780 comp rx=0 tx=0).stop 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- 172.21.15.142:0/2396040110 shutdown_connections 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/2396040110 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e00a2620 0x7fb1e00991f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- 172.21.15.142:0/2396040110 >> 172.21.15.142:0/2396040110 conn(0x7fb1e00716a0 msgr2=0x7fb1e00a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- 172.21.15.142:0/2396040110 shutdown_connections 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- 172.21.15.142:0/2396040110 wait complete. 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: monmap: 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:18.398 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 Processor -- start 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- start start 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: init 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 0x7fb1e013e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-a con 0x7fb1e0097160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 0x7fb1e013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-c con 0x7fb1e009dd40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e00a2620 0x7fb1e0141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): picked mon.noname-b con 0x7fb1e00a2620 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e013ec40 con 0x7fb1e0097160 2024-09-18T10:15:18.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0123200 con 0x7fb1e00a2620 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0122c90 con 0x7fb1e009dd40 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6e86700 10 monclient(hunting): authenticate will time out at 1207.307369s 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 0x7fb1e013e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 0x7fb1e013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 0x7fb1e013e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36270/0 (socket says 172.21.15.142:36270) 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): get_auth_request con 0x7fb1e009dd40 auth_method 0 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 1 -- 172.21.15.142:0/3860235467 learned_addr learned my addr 172.21.15.142:0/3860235467 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request con 0x7fb1e0097160 auth_method 0 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6685700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e00a2620 0x7fb1e0141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5683700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6685700 10 monclient(hunting): get_auth_request con 0x7fb1e00a2620 auth_method 0 2024-09-18T10:15:18.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6685700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6685700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.377+0000 7fb1e6685700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6685700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6685700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6685700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 10 monclient(hunting): handle_auth_done global_id 74135 payload 824 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e00a2620 msgr2=0x7fb1e0141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e00a2620 0x7fb1e0141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 msgr2=0x7fb1e013e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 0x7fb1e013e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0009a20 con 0x7fb1e009dd40 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0097160 0x7fb1e013e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 msgr2=0x7fb1e013d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e5683700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 0x7fb1e013d890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c530 tx=0x7fb1d4007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4005d40 con 0x7fb1e009dd40 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: dump: 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.403 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 5 monclient: authenticate success, global_id 74135 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d40070d0 con 0x7fb1e009dd40 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4007490 con 0x7fb1e009dd40 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 monclient: _renew_subs 2024-09-18T10:15:18.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e0122930 con 0x7fb1e009dd40 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1dffff700 4 set_mon_vals no callback set 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1ddffb700 10 monclient: dump: 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 monclient: _renew_subs 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e013ddd0 con 0x7fb1e009dd40 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 0x7fb1e0141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.381+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0123090 con 0x7fb1e0121a60 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 0x7fb1e0141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient: get_auth_request con 0x7fb1e0121a60 auth_method 0 2024-09-18T10:15:18.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb1d401b3b0 con 0x7fb1e009dd40 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_done global_id 54672 payload 824 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1e0064680 con 0x7fb1e0121a60 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1cc035af0 0x7fb1cc037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(183..183 src has 58..183) v4 ==== 8105+0+0 (secure 0 0 0) 0x7fb1d4048480 con 0x7fb1e009dd40 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e6685700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1cc035af0 0x7fb1cc037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e6685700 10 monclient: get_auth_request con 0x7fb1cc035af0 auth_method 0 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 msgr2=0x7fb1e0141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 0x7fb1e0141c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009de0 tx=0x7fb1d0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0022070 con 0x7fb1e0121a60 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 10 monclient: dump: 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.408 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb1d001d920 con 0x7fb1e0121a60 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 msgr2=0x7fb1e0141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 0x7fb1e0141c90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009de0 tx=0x7fb1d0018000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 msgr2=0x7fb1e0141c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.385+0000 7fb1ddffb700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e0121a60 0x7fb1e0141c90 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009de0 tx=0x7fb1d0018000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.389+0000 7fb1e6685700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1cc035af0 0x7fb1cc037fa0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80039b0 tx=0x7fb1d8007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e6e86700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 0x7fb1e01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e6e86700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e013eea0 con 0x7fb1e00a0e80 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 0x7fb1e01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient: get_auth_request con 0x7fb1e00a0e80 auth_method 0 2024-09-18T10:15:18.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 10 monclient(hunting): handle_auth_done global_id 54675 payload 824 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 1 -- 172.21.15.142:0/3860235467 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb1e0072a80 con 0x7fb1e00a0e80 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 msgr2=0x7fb1e01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.397+0000 7fb1e5e84700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 0x7fb1e01421d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00099f0 tx=0x7fb1d001f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0034800 con 0x7fb1e00a0e80 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 10 monclient: dump: 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fb1d0034960 con 0x7fb1e00a0e80 2024-09-18T10:15:18.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 msgr2=0x7fb1e01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 0x7fb1e01421d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00099f0 tx=0x7fb1d001f070 comp rx=0 tx=0).stop 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 msgr2=0x7fb1e01421d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1ddffb700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1e00a0e80 0x7fb1e01421d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:15:18.415 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.416 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.417 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.418 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.419 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:18.420 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.421 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.422 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.423 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.426 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:18.427 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:18.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1cc035af0 msgr2=0x7fb1cc037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1cc035af0 0x7fb1cc037fa0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80039b0 tx=0x7fb1d8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 10 monclient: shutdown 2024-09-18T10:15:18.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 msgr2=0x7fb1e013d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 --2- 172.21.15.142:0/3860235467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1e009dd40 0x7fb1e013d890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c530 tx=0x7fb1d4007f90 comp rx=0 tx=0).stop 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6685700 1 -- 172.21.15.142:0/3860235467 reap_dead start 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6685700 1 -- 172.21.15.142:0/3860235467 reap_dead start 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 shutdown_connections 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 >> 172.21.15.142:0/3860235467 conn(0x7fb1e00716a0 msgr2=0x7fb1e006cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 shutdown_connections 2024-09-18T10:15:18.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.401+0000 7fb1e6e86700 1 -- 172.21.15.142:0/3860235467 wait complete. 2024-09-18T10:15:18.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:18.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:15:18.431 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:15:18.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: monmap: 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:18.544707+0000 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:18.544707+0000 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:18.547 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:18.547 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:18.547 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 Processor -- start 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 -- start start 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: init 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f983c09abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 0x7f983c134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f983c09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f983c134bb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c09dc90 con 0x7f983c09abd0 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983c09afa0 con 0x7f983c134bb0 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c09b500 con 0x7f983c09b740 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983bfff700 10 monclient(hunting): authenticate will time out at 1207.470234s 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 0x7f983c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983affd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f983c09abd0 0x7f983c0960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41562/0 (socket says 172.21.15.142:41562) 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983affd700 1 -- 172.21.15.142:0/539653530 learned_addr learned my addr 172.21.15.142:0/539653530 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): get_auth_request con 0x7f983c09b740 auth_method 0 2024-09-18T10:15:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient(hunting): handle_auth_done global_id 74138 payload 274 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 1 -- 172.21.15.142:0/539653530 >> v1:172.21.15.150:6789/0 conn(0x7f983c134bb0 legacy=0x7f983c136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 1 -- 172.21.15.142:0/539653530 >> v1:172.21.15.142:6789/0 conn(0x7f983c09abd0 legacy=0x7f983c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 1 monclient: found mon.noname-b 2024-09-18T10:15:18.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.541+0000 7f983a7fc700 1 -- 172.21.15.142:0/539653530 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983c06f810 con 0x7f983c09b740 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983a7fc700 1 -- 172.21.15.142:0/539653530 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 msgr2=0x7f983c134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983a7fc700 1 --2- 172.21.15.142:0/539653530 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 0x7f983c134670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f983c06fd20 tx=0x7f98300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f52f5be81bbe659 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 1 -- 172.21.15.142:0/539653530 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000ba70 con 0x7f983c09b740 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: dump: 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.556 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 5 monclient: authenticate success, global_id 74138 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 1 -- 172.21.15.142:0/539653530 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983000bbd0 con 0x7f983c09b740 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 4 set_mon_vals no callback set 2024-09-18T10:15:18.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 1 -- 172.21.15.142:0/539653530 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830011420 con 0x7f983c09b740 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f9839ffb700 10 monclient: dump: 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.558 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 10 monclient: shutdown 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 -- 172.21.15.142:0/539653530 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 msgr2=0x7f983c134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 --2- 172.21.15.142:0/539653530 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 0x7f983c134670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f983c06fd20 tx=0x7f98300091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 -- 172.21.15.142:0/539653530 shutdown_connections 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 --2- 172.21.15.142:0/539653530 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c09b740 0x7f983c134670 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 -- 172.21.15.142:0/539653530 >> 172.21.15.142:0/539653530 conn(0x7f983c070ef0 msgr2=0x7f983c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 -- 172.21.15.142:0/539653530 shutdown_connections 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 1 -- 172.21.15.142:0/539653530 wait complete. 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.545+0000 7f983bfff700 10 monclient: monmap: 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:18.559 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 Processor -- start 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 -- start start 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient: init 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:18.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09abd0 0x7f983c087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f983c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c134bb0 0x7f983c088520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f983c134bb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f983c088a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c08aff0 con 0x7f983c134bb0 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c09d240 con 0x7f983c09abd0 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c09fb30 con 0x7f983c088a60 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983bfff700 10 monclient(hunting): authenticate will time out at 1207.477064s 2024-09-18T10:15:18.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c134bb0 0x7f983c088520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38972/0 (socket says 172.21.15.142:38972) 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 learned_addr learned my addr 172.21.15.142:0/3979896678 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): get_auth_request con 0x7f983c088a60 auth_method 0 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): get_auth_request con 0x7f983c134bb0 auth_method 0 2024-09-18T10:15:18.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09abd0 0x7f983c087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): get_auth_request con 0x7f983c09abd0 auth_method 0 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient(hunting): handle_auth_done global_id 74141 payload 824 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09abd0 msgr2=0x7f983c087fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f983c09abd0 0x7f983c087fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c134bb0 msgr2=0x7f983c088520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f983c134bb0 0x7f983c088520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.549+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830009420 con 0x7f983c088a60 2024-09-18T10:15:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 msgr2=0x7f983c088e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983b7fe700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f983400c890 tx=0x7f98340026a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834012840 con 0x7f983c088a60 2024-09-18T10:15:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: dump: 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 5 monclient: authenticate success, global_id 74141 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98340129c0 con 0x7f983c088a60 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 monclient: _renew_subs 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c0893d0 con 0x7f983c088a60 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834012d80 con 0x7f983c088a60 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 10 monclient: dump: 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f98397fa700 4 set_mon_vals no callback set 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 monclient: _renew_subs 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c08b2a0 con 0x7f983c088a60 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 0x7f983c06b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c0648a0 con 0x7f983c06b370 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9834011590 con 0x7f983c088a60 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 0x7f983c06b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient: get_auth_request con 0x7f983c06b370 auth_method 0 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9828035ac0 0x7f9828037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(183..183 src has 58..183) v4 ==== 8105+0+0 (secure 0 0 0) 0x7f9834049400 con 0x7f983c088a60 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 10 monclient(hunting): handle_auth_done global_id 74144 payload 824 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983affd700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f983c08b720 con 0x7f983c06b370 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983a7fc700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9828035ac0 0x7f9828037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.553+0000 7f983a7fc700 10 monclient: get_auth_request con 0x7f9828035ac0 auth_method 0 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f983affd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 msgr2=0x7f983c06b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f983affd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 0x7f983c06b740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f982c00eac0 tx=0x7f982c007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c00cc40 con 0x7f983c06b370 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 10 monclient: dump: 2024-09-18T10:15:18.576 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f983a7fc700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9828035ac0 0x7f9828037f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f983c095630 tx=0x7f9830016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f982c00cda0 con 0x7f983c06b370 2024-09-18T10:15:18.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:18.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 msgr2=0x7f983c06b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 0x7f983c06b740 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f982c00eac0 tx=0x7f982c007f90 comp rx=0 tx=0).stop 2024-09-18T10:15:18.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 msgr2=0x7f983c06b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.557+0000 7f9822ffd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06b370 0x7f983c06b740 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.565+0000 7f983bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:18.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.565+0000 7f983bfff700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 0x7f983c149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.565+0000 7f983bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.565+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f983c072a80 con 0x7f983c06bc80 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 0x7f983c149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient: get_auth_request con 0x7f983c06bc80 auth_method 0 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 10 monclient(hunting): handle_auth_done global_id 74147 payload 824 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 1 -- 172.21.15.142:0/3979896678 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f983c08a3b0 con 0x7f983c06bc80 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 msgr2=0x7f983c149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f983affd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 0x7f983c149c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f982c007360 tx=0x7f982c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c029410 con 0x7f983c06bc80 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f9822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f9822ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.569+0000 7f9822ffd700 10 monclient: dump: 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.582 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f982c00ca30 con 0x7f983c06bc80 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 msgr2=0x7f983c149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 0x7f983c149c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f982c007360 tx=0x7f982c016000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 msgr2=0x7f983c149c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f9822ffd700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c06bc80 0x7f983c149c80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:18.583 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.584 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:18.585 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:18.586 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:18.587 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.588 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.590 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.592 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.594 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.595 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.596 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.597 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:18.597 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:18.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9828035ac0 msgr2=0x7f9828037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9828035ac0 0x7f9828037f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f983c095630 tx=0x7f9830016000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 10 monclient: shutdown 2024-09-18T10:15:18.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983b7fe700 1 -- 172.21.15.142:0/3979896678 reap_dead start 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 msgr2=0x7f983c088e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f983400c890 tx=0x7f98340026a0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 shutdown_connections 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 --2- 172.21.15.142:0/3979896678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f983c088a60 0x7f983c088e90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 >> 172.21.15.142:0/3979896678 conn(0x7f983c070ef0 msgr2=0x7f983c09bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 shutdown_connections 2024-09-18T10:15:18.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.573+0000 7f983bfff700 1 -- 172.21.15.142:0/3979896678 wait complete. 2024-09-18T10:15:18.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:18.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:15:18.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: monmap: 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:18.713913+0000 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:18.713913+0000 2024-09-18T10:15:18.715 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:18.716 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 Processor -- start 2024-09-18T10:15:18.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 -- start start 2024-09-18T10:15:18.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: init 2024-09-18T10:15:18.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5b809c760 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5b809cb30 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5b8134880 0x7fd5b8136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-c con 0x7fd5b8134880 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b8098b00 con 0x7fd5b809cb30 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b8098d20 con 0x7fd5b809c760 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.709+0000 7fd5bce0e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b809a100 con 0x7fd5b8134880 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 10 monclient(hunting): authenticate will time out at 1207.640086s 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b6ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd5b809cb30 0x7fd5b809cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41572/0 (socket says 172.21.15.142:41572) 2024-09-18T10:15:18.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/1677307589 learned_addr learned my addr 172.21.15.142:0/1677307589 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/1677307589 >> v2:172.21.15.150:3300/0 conn(0x7fd5b8134880 0x7fd5b8136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): get_auth_request con 0x7fd5b8134880 auth_method 0 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_done global_id 15886 payload 274 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 -- 172.21.15.142:0/1677307589 >> v1:172.21.15.150:6789/0 conn(0x7fd5b809c760 legacy=0x7fd5b809b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 -- 172.21.15.142:0/1677307589 >> v1:172.21.15.142:6789/0 conn(0x7fd5b809cb30 legacy=0x7fd5b809cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:15:18.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 -- 172.21.15.142:0/1677307589 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b809a3e0 con 0x7fd5b8134880 2024-09-18T10:15:18.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 -- 172.21.15.142:0/1677307589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 msgr2=0x7fd5b8136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/1677307589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b8136c60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00098e0 tx=0x7fd5b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90151b99733be635 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 1 -- 172.21.15.142:0/1677307589 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b000ba70 con 0x7fd5b8134880 2024-09-18T10:15:18.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: dump: 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.724 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 5 monclient: authenticate success, global_id 15886 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 1 -- 172.21.15.142:0/1677307589 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b000bbd0 con 0x7fd5b8134880 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5bce0e700 4 set_mon_vals no callback set 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 1 -- 172.21.15.142:0/1677307589 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0011420 con 0x7fd5b8134880 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.713+0000 7fd5b67fc700 10 monclient: dump: 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.727 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.728 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.728 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.728 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.728 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.728 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: shutdown 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- 172.21.15.142:0/1677307589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 msgr2=0x7fd5b8136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/1677307589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b8136c60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00098e0 tx=0x7fd5b00091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- 172.21.15.142:0/1677307589 shutdown_connections 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/1677307589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b8136c60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- 172.21.15.142:0/1677307589 >> 172.21.15.142:0/1677307589 conn(0x7fd5b8070ef0 msgr2=0x7fd5b80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- 172.21.15.142:0/1677307589 shutdown_connections 2024-09-18T10:15:18.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- 172.21.15.142:0/1677307589 wait complete. 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: build_initial_monmap 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: monmap: 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:18.731 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:18.732 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:18.732 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.732 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:18.732 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:18.732 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:18.733 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 Processor -- start 2024-09-18T10:15:18.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- start start 2024-09-18T10:15:18.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: init 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 0x7fd5b8130090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-c con 0x7fd5b809c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 0x7fd5b806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-a con 0x7fd5b809cb30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): picked mon.noname-b con 0x7fd5b8134880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b812b400 con 0x7fd5b809cb30 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b806e2a0 con 0x7fd5b8134880 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b812b120 con 0x7fd5b809c760 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5bce0e700 10 monclient(hunting): authenticate will time out at 1207.647230s 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 0x7fd5b806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 0x7fd5b8130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 0x7fd5b806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36324/0 (socket says 172.21.15.142:36324) 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/4021033297 learned_addr learned my addr 172.21.15.142:0/4021033297 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): get_auth_request con 0x7fd5b809cb30 auth_method 0 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): get_auth_request con 0x7fd5b809c760 auth_method 0 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): get_auth_request con 0x7fd5b8134880 auth_method 0 2024-09-18T10:15:18.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.717+0000 7fd5b77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 10 monclient(hunting): handle_auth_done global_id 74150 payload 824 2024-09-18T10:15:18.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 msgr2=0x7fd5b806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 msgr2=0x7fd5b806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 0x7fd5b806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 monclient: found mon.noname-c 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b0009420 con 0x7fd5b809c760 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b8134880 0x7fd5b806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b6ffd700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5b809cb30 0x7fd5b806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 msgr2=0x7fd5b8130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b77fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 0x7fd5b8130090 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000b4c0 tx=0x7fd5a0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000be70 con 0x7fd5b809c760 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: dump: 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.750 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.751 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.751 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 5 monclient: authenticate success, global_id 74150 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a000f720 con 0x7fd5b809c760 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 10 monclient: _renew_subs 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b812b6e0 con 0x7fd5b809c760 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000fb00 con 0x7fd5b809c760 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5b57fa700 4 set_mon_vals no callback set 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5af7fe700 10 monclient: dump: 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.753 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 10 monclient: _renew_subs 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.721+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b806ecd0 con 0x7fd5b809c760 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5bce0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 0x7fd5b806d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b80648a0 con 0x7fd5b806e590 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd5a001a400 con 0x7fd5b809c760 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5af7fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd598035ac0 0x7fd598037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 0x7fd5b806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient: get_auth_request con 0x7fd5b806e590 auth_method 0 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(183..183 src has 58..183) v4 ==== 8105+0+0 (secure 0 0 0) 0x7fd5a0048b20 con 0x7fd5b809c760 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd598035ac0 0x7fd598037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b7fff700 10 monclient: get_auth_request con 0x7fd598035ac0 auth_method 0 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_done global_id 15892 payload 824 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.725+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5b806ef20 con 0x7fd5b806e590 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5b7fff700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd598035ac0 0x7fd598037f70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00093f0 tx=0x7fd5b0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 msgr2=0x7fd5b806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5b6ffd700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 0x7fd5b806d240 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800dfd0 tx=0x7fd5a8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80074f0 con 0x7fd5b806e590 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 10 monclient: dump: 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.757 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd5a8007650 con 0x7fd5b806e590 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 msgr2=0x7fd5b806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 0x7fd5b806d240 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800dfd0 tx=0x7fd5a8009040 comp rx=0 tx=0).stop 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 msgr2=0x7fd5b806d240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.729+0000 7fd5af7fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b806e590 0x7fd5b806d240 secure :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800dfd0 tx=0x7fd5a8009040 comp rx=0 tx=0).stop 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5bce0e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 0x7fd5b805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5bce0e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b81306b0 con 0x7fd5b802ae80 2024-09-18T10:15:18.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 0x7fd5b805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient: get_auth_request con 0x7fd5b802ae80 auth_method 0 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 10 monclient(hunting): handle_auth_done global_id 15895 payload 824 2024-09-18T10:15:18.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.737+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/4021033297 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd5b802b280 con 0x7fd5b802ae80 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5b6ffd700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 msgr2=0x7fd5b805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5b6ffd700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 0x7fd5b805e890 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8017000 tx=0x7fd5a8018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8021830 con 0x7fd5b802ae80 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 10 monclient: dump: 2024-09-18T10:15:18.760 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fd5a8021990 con 0x7fd5b802ae80 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 msgr2=0x7fd5b805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 0x7fd5b805e890 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8017000 tx=0x7fd5a8018000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 msgr2=0x7fd5b805e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.741+0000 7fd5af7fe700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5b802ae80 0x7fd5b805e890 secure :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8017000 tx=0x7fd5a8018000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 68, 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:18.762 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 8, 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.763 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.764 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.765 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.765 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:18.765 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:18.765 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:18.768 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:18.768 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:18.769 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:18.769 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.769 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:18.769 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:18.769 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:18.770 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:18.771 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.772 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:18.773 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.774 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.775 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:18.776 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:18.777 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.778 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:18.779 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:18.780 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd598035ac0 msgr2=0x7fd598037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd598035ac0 0x7fd598037f70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00093f0 tx=0x7fd5b0016000 comp rx=0 tx=0).stop 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 10 monclient: shutdown 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 msgr2=0x7fd5b8130090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 0x7fd5b8130090 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000b4c0 tx=0x7fd5a0009060 comp rx=0 tx=0).stop 2024-09-18T10:15:18.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5b7fff700 1 -- 172.21.15.142:0/4021033297 reap_dead start 2024-09-18T10:15:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 shutdown_connections 2024-09-18T10:15:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 --2- 172.21.15.142:0/4021033297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5b809c760 0x7fd5b8130090 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 >> 172.21.15.142:0/4021033297 conn(0x7fd5b8070ef0 msgr2=0x7fd5b8133b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 shutdown_connections 2024-09-18T10:15:18.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:18.745+0000 7fd5bce0e700 1 -- 172.21.15.142:0/4021033297 wait complete. 2024-09-18T10:15:18.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:15:18.782 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:15:18.782 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-18T10:15:18.782 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:15:18.783 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:15:18.783 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:15:18.783 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f626011e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f626011e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6258007ed0 tx=0x7f6258009850 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6258007ed0 tx=0x7f6258009850 comp rx=0 tx=0).stop 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:18.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 msgr2=0x7f626011e100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f626009e1e0 0x7f626011e100 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f62440413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): picked mon.a con 0x7f6260071240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 0x7f62440441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 msgr2=0x7f62440441d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f62662b5700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 0x7f62440441d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): picked mon.b con 0x7f6244041d50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f6244046d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): picked mon.c con 0x7f62600994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6244047680 con 0x7f6260071240 2024-09-18T10:15:18.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62440477c0 con 0x7f6244041d50 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.785+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6244047980 con 0x7f62600994f0 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6255ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f62440413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 0 0 0) 0x7f62580098e0 con 0x7f626009e1e0 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6255ffb700 10 monclient(hunting): discarding stray monitor message mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): get_auth_request con 0x7f6260071240 auth_method 0 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f6244046d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:18.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): get_auth_request con 0x7f62600994f0 auth_method 0 2024-09-18T10:15:18.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:18.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:18.796 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f6265ab4700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient(hunting): handle_auth_done global_id 15877 payload 293 2024-09-18T10:15:18.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 msgr2=0x7f6244046d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f6244046d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 msgr2=0x7f62440441d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 0x7f62440441d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient: _finish_hunting 0 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 monclient: found mon.a 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f6244047c00 con 0x7f6260071240 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6260098a10 con 0x7f6260071240 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 10 monclient: _finish_auth 0 2024-09-18T10:15:18.798 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.789+0000 7f62652b3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:18.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f62652b3700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 msgr2=0x7f62440413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:18.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f62652b3700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f62440413f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f625000baf0 tx=0x7f6250012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:18.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250000c90 con 0x7f6260071240 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 10 monclient: dump: 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:18.800 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:18.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250021040 con 0x7f6260071240 2024-09-18T10:15:18.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f6255ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:18.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.793+0000 7f62577fe700 4 set_mon_vals no callback set 2024-09-18T10:15:18.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:18.797+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f625000bd40 con 0x7f6260071240 2024-09-18T10:15:18.804 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:15:18.804 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T10:15:18.804 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T10:15:18.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:15:21.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:21.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:21.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:21.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:21.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:21.108+0000 7f6254ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:22.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:22.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:22.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:22.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:23.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:23.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:23.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:23.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:24.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:24.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:24.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:24.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:25.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:25.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:25.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:25.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:26.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:26.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:26.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:26.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:27.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:27.104+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:27.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:27.104+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:28.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:28.104+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:28.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:28.104+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:29.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:29.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:29.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:29.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:30.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:30.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:30.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:30.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:31.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:31.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:31.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:31.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:31.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:31.108+0000 7f6254ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:32.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:32.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:32.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:32.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:33.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:33.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:33.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:33.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:34.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:34.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:34.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:34.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:35.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:35.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:35.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:36.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:36.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:36.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:36.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:37.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:37.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:37.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:37.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:38.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.112+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:38.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.112+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:38.805 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T10:15:38.806 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T10:15:38.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:15:38.807 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:15:38.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.812+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6250015070 con 0x7f6260071240 2024-09-18T10:15:38.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.812+0000 7f6255ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:38.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.812+0000 7f6255ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: get_monmap_and_config 2024-09-18T10:15:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: build_initial_monmap 2024-09-18T10:15:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: monmap: 2024-09-18T10:15:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:38.916016+0000 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:38.916016+0000 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:38.918 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 Processor -- start 2024-09-18T10:15:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 -- start start 2024-09-18T10:15:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: init 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63bc09b8c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-b con 0x7f63bc09bc90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-a con 0x7f63bc133a40 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0a2c30 con 0x7f63bc09b8c0 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc097eb0 con 0x7f63bc133a40 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc09e440 con 0x7f63bc09bc90 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63c0a5b700 10 monclient(hunting): authenticate will time out at 1227.843272s 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bad9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f63bc09b8c0 0x7f63bc096e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:44820/0 (socket says 172.21.15.142:44820) 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 learned_addr learned my addr 172.21.15.142:0/1493503240 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63bc133a40 auth_method 0 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 10 monclient(hunting): get_auth_request con 0x7f63bc09bc90 auth_method 0 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.912+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:38.916+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.142:6789/0 conn(0x7f63bc09b8c0 legacy=0x7f63bc096e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:38.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.980+0000 7f6267d18700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:38.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.980+0000 7f6267d18700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:38.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.980+0000 7f6267d18700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:38.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:38.980+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f6260063ed0 con 0x7f6260071240 2024-09-18T10:15:39.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:39.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:39.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:39.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63bc133a40 auth_method 0 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 10 monclient(hunting): get_auth_request con 0x7f63bc09bc90 auth_method 0 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:39.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:15:39.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.116+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:15:39.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 10 monclient(hunting): get_auth_request con 0x7f63bc09bc90 auth_method 0 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63bc133a40 auth_method 0 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:39.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:15:39.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:39.520+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:15:40.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:40.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:40.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:40.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63bc133a40 auth_method 0 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 10 monclient(hunting): get_auth_request con 0x7f63bc09bc90 auth_method 0 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:40.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:15:40.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:40.324+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:15:41.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:41.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:41.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:41.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:41.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:41.108+0000 7f6254ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:41.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): tick 2024-09-18T10:15:41.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 monclient(hunting): continuing hunt 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 msgr2=0x7f63bc133500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63bc09bc90 0x7f63bc133500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 msgr2=0x7f63bc135e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63bc133a40 0x7f63bc135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.142:6789/0 conn(0x7f63bc09b8c0 legacy=0x7f63bc096e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient: _add_conns ranks=[1,4] 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63a40059a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): picked mon.noname-c con 0x7f63a40096c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc097bd0 con 0x7f63a40059a0 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc097eb0 con 0x7f63a40096c0 2024-09-18T10:15:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63b8d99700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 msgr2=0x7f63a400bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a40059a0 legacy=0x7f63a4007df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:41.912+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:42.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:42.112+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:42.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:42.112+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:42.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.116+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 msgr2=0x7f63a400bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:42.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.116+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a40059a0 legacy=0x7f63a4007df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:42.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.116+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:15:42.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.520+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 msgr2=0x7f63a400bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:42.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.520+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a40059a0 legacy=0x7f63a4007df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:42.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:42.520+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:15:43.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:43.108+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:43.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:43.108+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:43.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:43.324+0000 7f63bad9d700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a40059a0 legacy=0x7f63a4007df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:43.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:43.324+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 msgr2=0x7f63a400bb70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:43.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:43.324+0000 7f63ba59c700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:15:44.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.112+0000 7f6254ff9700 10 monclient: tick 2024-09-18T10:15:44.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.112+0000 7f6254ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:44.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.836+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 removed item id -16 name 'fooo' from crush map v184) v1 ==== 122+0+0 (secure 0 0 0) 0x7f625000fcf0 con 0x7f6260071240 2024-09-18T10:15:44.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.836+0000 7f6255ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:44.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.836+0000 7f6255ffb700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'fooo' from crush map 2024-09-18T10:15:44.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): tick 2024-09-18T10:15:44.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 monclient(hunting): continuing hunt 2024-09-18T10:15:44.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 msgr2=0x7f63a400bb70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.150:3300/0 conn(0x7f63a40096c0 0x7f63a400bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a40059a0 legacy=0x7f63a4007df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 reap_dead start 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63bc09b8c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63a400c970 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63a400bb70 0x7f63a4008cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): picked mon.noname-a con 0x7f63a400bb70 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a400d240 con 0x7f63bc09b8c0 2024-09-18T10:15:44.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc097bd0 con 0x7f63a400c970 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 1 -- 172.21.15.142:0/1493503240 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc097eb0 con 0x7f63a400bb70 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.912+0000 7f63b8d99700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63ba59c700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a400c970 legacy=0x7f63a40085c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> v2:172.21.15.142:3300/0 conn(0x7f63a400bb70 0x7f63a4008cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63a400bb70 auth_method 0 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient(hunting): handle_auth_done global_id 54684 payload 274 2024-09-18T10:15:44.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.150:6789/0 conn(0x7f63a400c970 legacy=0x7f63a40085c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> v1:172.21.15.142:6789/0 conn(0x7f63bc09b8c0 legacy=0x7f63a400c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient: _finish_hunting 0 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 monclient: found mon.noname-a 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:44.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a400d040 con 0x7f63a400bb70 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a400bb70 msgr2=0x7f63a4008cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63bb59e700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a400bb70 0x7f63a4008cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0a1c20 tx=0x7f63b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade7139603d42aee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63b9d9b700 1 -- 172.21.15.142:0/1493503240 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007e80 con 0x7f63a400bb70 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63b9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:44.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.916+0000 7f63b9d9b700 10 monclient: dump: 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 5 monclient: authenticate success, global_id 54684 2024-09-18T10:15:44.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 1 -- 172.21.15.142:0/1493503240 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4017680 con 0x7f63a400bb70 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 4 set_mon_vals no callback set 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 1 -- 172.21.15.142:0/1493503240 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4017a20 con 0x7f63a400bb70 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63b9d9b700 10 monclient: dump: 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.927 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:44.928 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:44.928 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:44.928 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:44.928 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:44.928 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 10 monclient: shutdown 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 -- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a400bb70 msgr2=0x7f63a4008cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a400bb70 0x7f63a4008cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0a1c20 tx=0x7f63b400b040 comp rx=0 tx=0).stop 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 -- 172.21.15.142:0/1493503240 shutdown_connections 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/1493503240 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a400bb70 0x7f63a4008cd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 -- 172.21.15.142:0/1493503240 >> 172.21.15.142:0/1493503240 conn(0x7f63bc0718c0 msgr2=0x7f63bc0a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 -- 172.21.15.142:0/1493503240 shutdown_connections 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 -- 172.21.15.142:0/1493503240 wait complete. 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 10 monclient: build_initial_monmap 2024-09-18T10:15:44.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 10 monclient: monmap: 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.920+0000 7f63c0a5b700 1 Processor -- start 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 -- start start 2024-09-18T10:15:44.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient: init 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 0x7f63bc141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-b con 0x7f63a400bb70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-c con 0x7f63a400c970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 0x7f63bc06bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): picked mon.noname-a con 0x7f63bc09b8c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc11d200 con 0x7f63bc09b8c0 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc09a470 con 0x7f63a400bb70 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:44.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc11cf20 con 0x7f63a400c970 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63c0a5b700 10 monclient(hunting): authenticate will time out at 1233.853232s 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bad9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 msgr2=0x7f63bc141350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 0x7f63bc141350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 0x7f63bc06bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 0x7f63bc06bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44374/0 (socket says 172.21.15.142:44374) 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43106/0 (socket says 172.21.15.142:43106) 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 1 -- 172.21.15.142:0/699852271 learned_addr learned my addr 172.21.15.142:0/699852271 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): get_auth_request con 0x7f63bc09b8c0 auth_method 0 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:44.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): get_auth_request con 0x7f63a400c970 auth_method 0 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63bb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient(hunting): handle_auth_done global_id 74153 payload 824 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 -- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 msgr2=0x7f63bc141350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 0x7f63bc141350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 -- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 msgr2=0x7f63bc06bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 0x7f63bc06bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 monclient: found mon.noname-c 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:44.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.924+0000 7f63ba59c700 1 -- 172.21.15.142:0/699852271 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40099a0 con 0x7f63a400c970 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63ba59c700 1 -- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 msgr2=0x7f63bc12db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63ba59c700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63a8012980 tx=0x7f63a8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8012e10 con 0x7f63a400c970 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: dump: 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 5 monclient: authenticate success, global_id 74153 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a8010b60 con 0x7f63a400c970 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _renew_subs 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc06d670 con 0x7f63a400c970 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:44.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8015450 con 0x7f63a400c970 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 10 monclient: dump: 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:44.943 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b959a700 4 set_mon_vals no callback set 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _renew_subs 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc142130 con 0x7f63a400c970 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f63a80158c0 con 0x7f63a400c970 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.928+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc072a80 con 0x7f63a400c970 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.932+0000 7f63b2ffd700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 0x7f63a40395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.932+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(184..184 src has 58..184) v4 ==== 7889+0+0 (secure 0 0 0) 0x7f63a8049230 con 0x7f63a400c970 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.932+0000 7f63bad9d700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 0x7f63a40395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:44.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.932+0000 7f63bad9d700 10 monclient: get_auth_request con 0x7f63a4037120 auth_method 0 2024-09-18T10:15:44.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.932+0000 7f63bad9d700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 0x7f63a40395d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f63ac007fd0 tx=0x7f63ac007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:44.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.940+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f63a801d6d0 con 0x7f63a400c970 2024-09-18T10:15:44.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.940+0000 7f63b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:44.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:44.940+0000 7f63b2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6267d18700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6267d18700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6267d18700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:44.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "fooo"} v 0) v1 -- 0x7f6260071930 con 0x7f6260071240 2024-09-18T10:15:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6255ffb700 1 -- 172.21.15.150:0/3850018861 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "fooo"}]=0 device 'fooo' does not appear in the crush map v184) v1 ==== 122+0+0 (secure 0 0 0) 0x7f625000fcf0 con 0x7f6260071240 2024-09-18T10:15:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6255ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "fooo"}] 2024-09-18T10:15:44.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.944+0000 7f6255ffb700 10 monclient: _finish_command 3 = system:0 device 'fooo' does not appear in the crush map 2024-09-18T10:15:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 msgr2=0x7f6244037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 0x7f6244037f70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f625c006fd0 tx=0x7f625c008040 comp rx=0 tx=0).stop 2024-09-18T10:15:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 10 monclient: shutdown 2024-09-18T10:15:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:44.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 msgr2=0x7f62440413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f62440413f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f625000baf0 tx=0x7f6250012000 comp rx=0 tx=0).stop 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 shutdown_connections 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62600994f0 0x7f6244046d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6244041d50 0x7f62440441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6260071240 0x7f62440413f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 --2- 172.21.15.150:0/3850018861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6244035ac0 0x7f6244037f70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 >> 172.21.15.150:0/3850018861 conn(0x7f62600a1cc0 msgr2=0x7f626009a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 shutdown_connections 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:44.948+0000 7f6267d18700 1 -- 172.21.15.150:0/3850018861 wait complete. 2024-09-18T10:15:44.956 INFO:tasks.workunit.client.0.smithi150.stderr:device 'fooo' does not appear in the crush map 2024-09-18T10:15:44.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm barr 2024-09-18T10:15:45.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: get_monmap_and_config 2024-09-18T10:15:45.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: monmap: 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:45.065741+0000 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:45.065741+0000 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:45.067 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 Processor -- start 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 -- start start 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: init 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c80977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-b con 0x7f93c809c930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 0x7f93c809d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-c con 0x7f93c809cd00 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93c8136760 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c8098f00 con 0x7f93c8136760 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c8097d00 con 0x7f93c809cd00 2024-09-18T10:15:45.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c8097030 con 0x7f93c809c930 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93cd6e3700 10 monclient(hunting): authenticate will time out at 1227.159785s 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93c67fc700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 msgr2=0x7f93c809d0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.060+0000 7f93c67fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 0x7f93c809d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f93c8136760 0x7f93c8138b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:44550/0 (socket says 172.21.15.150:44550) 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c77fe700 1 -- 172.21.15.150:0/2397653091 learned_addr learned my addr 172.21.15.150:0/2397653091 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c80977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): get_auth_request con 0x7f93c809c930 auth_method 0 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_done global_id 74156 payload 274 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2397653091 >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 msgr2=0x7f93c809d0d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2397653091 >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 0x7f93c809d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2397653091 >> v1:172.21.15.142:6789/0 conn(0x7f93c8136760 legacy=0x7f93c8138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2397653091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c806f730 con 0x7f93c809c930 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 msgr2=0x7f93c80977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c80977c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93b80098e0 tx=0x7f93b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9cd92dd279b153 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 1 -- 172.21.15.150:0/2397653091 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ba70 con 0x7f93c809c930 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:45.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: dump: 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 5 monclient: authenticate success, global_id 74156 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 1 -- 172.21.15.150:0/2397653091 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b800bbf0 con 0x7f93c809c930 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 4 set_mon_vals no callback set 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 1 -- 172.21.15.150:0/2397653091 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b8011420 con 0x7f93c809c930 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93c5ffb700 10 monclient: dump: 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.078 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 10 monclient: shutdown 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 msgr2=0x7f93c80977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c80977c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93b80098e0 tx=0x7f93b80091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2397653091 shutdown_connections 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2397653091 >> v2:172.21.15.150:3300/0 conn(0x7f93c809cd00 0x7f93c809d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2397653091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c80977c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.064+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2397653091 >> 172.21.15.150:0/2397653091 conn(0x7f93c8068360 msgr2=0x7f93c80a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2397653091 shutdown_connections 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2397653091 wait complete. 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient: monmap: 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:45.080 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 Processor -- start 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- start start 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient: init 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-c con 0x7f93c809c930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 0x7f93c811f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-a con 0x7f93c809cd00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 0x7f93c811f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): picked mon.noname-b con 0x7f93c8136760 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:45.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c809e3f0 con 0x7f93c809cd00 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c809baa0 con 0x7f93c8136760 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c8121660 con 0x7f93c809c930 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93cd6e3700 10 monclient(hunting): authenticate will time out at 1227.165366s 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c77fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 msgr2=0x7f93c811f930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c77fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 0x7f93c811f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 0x7f93c811f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60780/0 (socket says 172.21.15.150:60780) 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2249993925 learned_addr learned my addr 172.21.15.150:0/2249993925 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): get_auth_request con 0x7f93c809c930 auth_method 0 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): get_auth_request con 0x7f93c809cd00 auth_method 0 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient(hunting): handle_auth_done global_id 74159 payload 824 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 msgr2=0x7f93c811f930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 0x7f93c811f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 msgr2=0x7f93c811f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 0x7f93c811f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 monclient: found mon.noname-c 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8009420 con 0x7f93c809c930 2024-09-18T10:15:45.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.068+0000 7f93c67fc700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 0x7f93c811f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93c6ffd700 1 -- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 msgr2=0x7f93c811eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93c6ffd700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93c80971c0 tx=0x7f93b8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b8011e30 con 0x7f93c809c930 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: dump: 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:45.086 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b8020070 con 0x7f93c809c930 2024-09-18T10:15:45.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b801b680 con 0x7f93c809c930 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 10 monclient: dump: 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.088 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 5 monclient: authenticate success, global_id 74159 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _renew_subs 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c8120910 con 0x7f93c809c930 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93c57fa700 4 set_mon_vals no callback set 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _renew_subs 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c81364a0 con 0x7f93c809c930 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8096a70 con 0x7f93c809c930 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.072+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f93b8022030 con 0x7f93c809c930 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.076+0000 7f93aeffd700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 0x7f93b4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.076+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(184..184 src has 58..184) v4 ==== 7889+0+0 (secure 0 0 0) 0x7f93b804ae90 con 0x7f93c809c930 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.076+0000 7f93c67fc700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 0x7f93b4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.076+0000 7f93c67fc700 10 monclient: get_auth_request con 0x7f93b4035a70 auth_method 0 2024-09-18T10:15:45.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.076+0000 7f93c67fc700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 0x7f93b4037f20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f93bc003fd0 tx=0x7f93bc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.084+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f93b801ba70 con 0x7f93c809c930 2024-09-18T10:15:45.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.084+0000 7f93aeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.084+0000 7f93aeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:45.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:15:45.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:45.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f63bc068360 con 0x7f63a400c970 2024-09-18T10:15:45.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63b2ffd700 1 -- 172.21.15.142:0/699852271 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f63a801d310 con 0x7f63a400c970 2024-09-18T10:15:45.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63b2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:15:45.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63b2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:45.178 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:15:45.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 msgr2=0x7f63a40395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 0x7f63a40395d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f63ac007fd0 tx=0x7f63ac007b00 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 10 monclient: shutdown 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 msgr2=0x7f63bc12db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63a8012980 tx=0x7f63a8008040 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 shutdown_connections 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63a4037120 0x7f63a40395d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63bc09b8c0 0x7f63bc06bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63a400c970 0x7f63bc12db00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 --2- 172.21.15.142:0/699852271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63a400bb70 0x7f63bc141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 >> 172.21.15.142:0/699852271 conn(0x7f63bc0718c0 msgr2=0x7f63bc0a09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 shutdown_connections 2024-09-18T10:15:45.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.172+0000 7f63c0a5b700 1 -- 172.21.15.142:0/699852271 wait complete. 2024-09-18T10:15:45.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:15:45.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:15:45.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:15:45.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.256+0000 7f93cd6e3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.256+0000 7f93cd6e3700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.256+0000 7f93cd6e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.256+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f93c8096a70 con 0x7f93c809c930 2024-09-18T10:15:45.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: get_monmap_and_config 2024-09-18T10:15:45.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: monmap: 2024-09-18T10:15:45.298 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:45.298 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:45.297283+0000 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:45.297283+0000 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:45.299 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 Processor -- start 2024-09-18T10:15:45.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 -- start start 2024-09-18T10:15:45.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: init 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff86409abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): picked mon.noname-b con 0x7ff86409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 0x7ff864136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): picked mon.noname-c con 0x7ff864134bb0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86409dc90 con 0x7ff86409abd0 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86409b500 con 0x7ff864134bb0 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640967f0 con 0x7ff86409b740 2024-09-18T10:15:45.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff869685700 10 monclient(hunting): authenticate will time out at 1234.222798s 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff863fff700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7ff86409abd0 legacy=0x7ff8640960e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff868e84700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 msgr2=0x7ff864136f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff868e84700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 0x7ff864136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43112/0 (socket says 172.21.15.142:43112) 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 -- 172.21.15.142:0/509014080 learned_addr learned my addr 172.21.15.142:0/509014080 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff86409b740 auth_method 0 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient(hunting): handle_auth_done global_id 74162 payload 274 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 -- 172.21.15.142:0/509014080 >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 msgr2=0x7ff864136f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 --2- 172.21.15.142:0/509014080 >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 0x7ff864136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 -- 172.21.15.142:0/509014080 >> v1:172.21.15.150:6789/0 conn(0x7ff86409abd0 legacy=0x7ff8640960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 monclient: found mon.noname-b 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.292+0000 7ff8637fe700 1 -- 172.21.15.142:0/509014080 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86409afa0 con 0x7ff86409b740 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff8637fe700 1 -- 172.21.15.142:0/509014080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 msgr2=0x7ff864134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff8637fe700 1 --2- 172.21.15.142:0/509014080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff86406fd20 tx=0x7ff858009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b4392ceb5df7a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 1 -- 172.21.15.142:0/509014080 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85800ba70 con 0x7ff86409b740 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: dump: 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 5 monclient: authenticate success, global_id 74162 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 1 -- 172.21.15.142:0/509014080 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85800bbd0 con 0x7ff86409b740 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 4 set_mon_vals no callback set 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 1 -- 172.21.15.142:0/509014080 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858011420 con 0x7ff86409b740 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff862ffd700 10 monclient: dump: 2024-09-18T10:15:45.308 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 10 monclient: shutdown 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 -- 172.21.15.142:0/509014080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 msgr2=0x7ff864134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 --2- 172.21.15.142:0/509014080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff86406fd20 tx=0x7ff858009250 comp rx=0 tx=0).stop 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 -- 172.21.15.142:0/509014080 shutdown_connections 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 --2- 172.21.15.142:0/509014080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff864134670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 --2- 172.21.15.142:0/509014080 >> v2:172.21.15.150:3300/0 conn(0x7ff864134bb0 0x7ff864136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 -- 172.21.15.142:0/509014080 >> 172.21.15.142:0/509014080 conn(0x7ff864070ef0 msgr2=0x7ff8640712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 -- 172.21.15.142:0/509014080 shutdown_connections 2024-09-18T10:15:45.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.296+0000 7ff869685700 1 -- 172.21.15.142:0/509014080 wait complete. 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient: monmap: 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:45.315 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 Processor -- start 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 -- start start 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient: init 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86409abd0 0x7ff86413f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): picked mon.noname-b con 0x7ff86409abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): picked mon.noname-c con 0x7ff86409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff864134bb0 0x7ff864140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): picked mon.noname-a con 0x7ff864134bb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8641276a0 con 0x7ff864134bb0 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff864127cd0 con 0x7ff86409abd0 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86406f050 con 0x7ff86409b740 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff863fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86409abd0 msgr2=0x7ff86413f9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff863fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86409abd0 0x7ff86413f9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:45.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43118/0 (socket says 172.21.15.142:43118) 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 -- 172.21.15.142:0/353482459 learned_addr learned my addr 172.21.15.142:0/353482459 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff864134bb0 0x7ff864140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff869685700 10 monclient(hunting): authenticate will time out at 1234.230923s 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): get_auth_request con 0x7ff86409b740 auth_method 0 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): get_auth_request con 0x7ff864134bb0 auth_method 0 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff868e84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient(hunting): handle_auth_done global_id 74165 payload 824 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86409abd0 msgr2=0x7ff86413f9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff86409abd0 0x7ff86413f9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff864134bb0 msgr2=0x7ff864140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff864134bb0 0x7ff864140440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 monclient: found mon.noname-c 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.300+0000 7ff8637fe700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8580094f0 con 0x7ff86409b740 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff8637fe700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 msgr2=0x7ff86413ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff8637fe700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff85800b3c0 tx=0x7ff858017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85801a420 con 0x7ff86409b740 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: dump: 2024-09-18T10:15:45.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 5 monclient: authenticate success, global_id 74165 2024-09-18T10:15:45.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85801a580 con 0x7ff86409b740 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 monclient: _renew_subs 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86409d240 con 0x7ff86409b740 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85801a940 con 0x7ff86409b740 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 10 monclient: dump: 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.322 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff8627fc700 4 set_mon_vals no callback set 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 monclient: _renew_subs 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86406ead0 con 0x7ff86409b740 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 0x7ff86405e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8640648a0 con 0x7ff86406e1a0 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff85801acf0 con 0x7ff86409b740 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 0x7ff86405e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient: get_auth_request con 0x7ff86406e1a0 auth_method 0 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff84bfff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff850035a70 0x7ff850037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 10 monclient(hunting): handle_auth_done global_id 54696 payload 824 2024-09-18T10:15:45.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.304+0000 7ff863fff700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff864128a20 con 0x7ff86406e1a0 2024-09-18T10:15:45.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(184..184 src has 58..184) v4 ==== 7889+0+0 (secure 0 0 0) 0x7ff85804ab00 con 0x7ff86409b740 2024-09-18T10:15:45.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff868e84700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff850035a70 0x7ff850037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff868e84700 10 monclient: get_auth_request con 0x7ff850035a70 auth_method 0 2024-09-18T10:15:45.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff863fff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 msgr2=0x7ff86405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff863fff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 0x7ff86405e890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00bce0 tx=0x7ff85c009690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c005300 con 0x7ff86406e1a0 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 10 monclient: dump: 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.327 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff85c005460 con 0x7ff86406e1a0 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 msgr2=0x7ff86405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 0x7ff86405e890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00bce0 tx=0x7ff85c009690 comp rx=0 tx=0).stop 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 msgr2=0x7ff86405e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff84bfff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86406e1a0 0x7ff86405e890 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00bce0 tx=0x7ff85c009690 comp rx=0 tx=0).stop 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.308+0000 7ff868e84700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff850035a70 0x7ff850037f20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff86409d490 tx=0x7ff854006f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff869685700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:45.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff869685700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 0x7ff864029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff869685700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8641346f0 con 0x7ff86405edd0 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 0x7ff864029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient: get_auth_request con 0x7ff86405edd0 auth_method 0 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 10 monclient(hunting): handle_auth_done global_id 54699 payload 824 2024-09-18T10:15:45.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 1 -- 172.21.15.142:0/353482459 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff864072a80 con 0x7ff86405edd0 2024-09-18T10:15:45.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 msgr2=0x7ff864029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff863fff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 0x7ff864029d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009fd0 tx=0x7ff85c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85c019070 con 0x7ff86405edd0 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff84bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff84bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.320+0000 7ff84bfff700 10 monclient: dump: 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.331 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7ff85c0143d0 con 0x7ff86405edd0 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 msgr2=0x7ff864029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 0x7ff864029d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009fd0 tx=0x7ff85c015000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 msgr2=0x7ff864029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff84bfff700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff86405edd0 0x7ff864029d00 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff85c009fd0 tx=0x7ff85c015000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.332 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 72, 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:45.333 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:45.334 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:45.335 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:45.336 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.337 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:45.338 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:45.339 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.340 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.341 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.342 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:45.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.344 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:45.345 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:45.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff850035a70 msgr2=0x7ff850037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff850035a70 0x7ff850037f20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff86409d490 tx=0x7ff854006f90 comp rx=0 tx=0).stop 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 10 monclient: shutdown 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 msgr2=0x7ff86413ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff85800b3c0 tx=0x7ff858017000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff868e84700 1 -- 172.21.15.142:0/353482459 reap_dead start 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 shutdown_connections 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 --2- 172.21.15.142:0/353482459 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff86409b740 0x7ff86413ff00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 >> 172.21.15.142:0/353482459 conn(0x7ff864070ef0 msgr2=0x7ff864131870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 shutdown_connections 2024-09-18T10:15:45.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.324+0000 7ff869685700 1 -- 172.21.15.142:0/353482459 wait complete. 2024-09-18T10:15:45.348 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:15:45.348 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:15:45.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: get_monmap_and_config 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: monmap: 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.462 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:15:45.460913+0000 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:15:45.460913+0000 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:45.463 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 Processor -- start 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 -- start start 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: init 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:15:45.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f169009c760 0x7f169009b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f169009c760 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f169009b630 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f169009d240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1690098b00 con 0x7f169009b630 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1690098d20 con 0x7f169009c760 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169009a100 con 0x7f169009d240 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168ffff700 10 monclient(hunting): authenticate will time out at 1234.386992s 2024-09-18T10:15:45.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43130/0 (socket says 172.21.15.142:43130) 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 -- 172.21.15.142:0/3378960503 learned_addr learned my addr 172.21.15.142:0/3378960503 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): get_auth_request con 0x7f169009d240 auth_method 0 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 1 --2- 172.21.15.142:0/3378960503 >> v2:172.21.15.142:3300/0 conn(0x7f169009c760 0x7f169009b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): get_auth_request con 0x7f169009c760 auth_method 0 2024-09-18T10:15:45.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient(hunting): handle_auth_done global_id 74168 payload 274 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 -- 172.21.15.142:0/3378960503 >> v2:172.21.15.142:3300/0 conn(0x7f169009c760 msgr2=0x7f169009b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 --2- 172.21.15.142:0/3378960503 >> v2:172.21.15.142:3300/0 conn(0x7f169009c760 0x7f169009b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 -- 172.21.15.142:0/3378960503 >> v1:172.21.15.142:6789/0 conn(0x7f169009b630 legacy=0x7f169009cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.456+0000 7f168f7fe700 1 -- 172.21.15.142:0/3378960503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169009a3e0 con 0x7f169009d240 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168f7fe700 1 -- 172.21.15.142:0/3378960503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 msgr2=0x7f1690135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168f7fe700 1 --2- 172.21.15.142:0/3378960503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1688009f20 tx=0x7f1688009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34af141cd1e760b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 1 -- 172.21.15.142:0/3378960503 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168800e070 con 0x7f169009d240 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:45.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: dump: 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 1 -- 172.21.15.142:0/3378960503 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168800bbe0 con 0x7f169009d240 2024-09-18T10:15:45.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 5 monclient: authenticate success, global_id 74168 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 1 -- 172.21.15.142:0/3378960503 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16880092e0 con 0x7f169009d240 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 4 set_mon_vals no callback set 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168dffb700 10 monclient: dump: 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.476 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 10 monclient: shutdown 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 -- 172.21.15.142:0/3378960503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 msgr2=0x7f1690135880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 --2- 172.21.15.142:0/3378960503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1688009f20 tx=0x7f1688009840 comp rx=0 tx=0).stop 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 -- 172.21.15.142:0/3378960503 shutdown_connections 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 --2- 172.21.15.142:0/3378960503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009d240 0x7f1690135880 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 --2- 172.21.15.142:0/3378960503 >> v2:172.21.15.142:3300/0 conn(0x7f169009c760 0x7f169009b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 -- 172.21.15.142:0/3378960503 >> 172.21.15.142:0/3378960503 conn(0x7f1690070ef0 msgr2=0x7f16900712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 -- 172.21.15.142:0/3378960503 shutdown_connections 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.460+0000 7f168ffff700 1 -- 172.21.15.142:0/3378960503 wait complete. 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient: build_initial_monmap 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient: monmap: 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.478 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 Processor -- start 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 -- start start 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient: init 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:45.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f169009b630 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f169009c760 0x7f16900897e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f169009c760 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f169009d240 0x7f1690089d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f169009d240 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169008acc0 con 0x7f169009c760 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169008cd20 con 0x7f169009d240 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169008d370 con 0x7f169009b630 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:45.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168ffff700 10 monclient(hunting): authenticate will time out at 1234.395009s 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168f7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f169009d240 msgr2=0x7f1690089d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f169009d240 0x7f1690089d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f169009c760 0x7f16900897e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43136/0 (socket says 172.21.15.142:43136) 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 -- 172.21.15.142:0/3191896591 learned_addr learned my addr 172.21.15.142:0/3191896591 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): get_auth_request con 0x7f169009b630 auth_method 0 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): get_auth_request con 0x7f169009c760 auth_method 0 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient(hunting): handle_auth_done global_id 74171 payload 824 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f169009d240 msgr2=0x7f1690089d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f169009d240 0x7f1690089d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f169009c760 msgr2=0x7f16900897e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f169009c760 0x7f16900897e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:45.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 1 monclient: found mon.noname-c 2024-09-18T10:15:45.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:45.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.464+0000 7f168effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168effd700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009ae0 con 0x7f169009b630 2024-09-18T10:15:45.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168e7fc700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f169009c760 0x7f16900897e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168effd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 msgr2=0x7f1690093b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168effd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f168000ec80 tx=0x7f16800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000ca60 con 0x7f169009b630 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: dump: 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000cbc0 con 0x7f169009b630 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 5 monclient: authenticate success, global_id 74171 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 10 monclient: _renew_subs 2024-09-18T10:15:45.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169006f810 con 0x7f169009b630 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680013470 con 0x7f169009b630 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f1676ffd700 10 monclient: dump: 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168d7fa700 4 set_mon_vals no callback set 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 10 monclient: _renew_subs 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.468+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169008b1f0 con 0x7f169009b630 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168ffff700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 0x7f169008a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f16800137e0 con 0x7f169009b630 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900648a0 con 0x7f169008bdc0 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 0x7f169008a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient: get_auth_request con 0x7f169008bdc0 auth_method 0 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f167c035a70 0x7f167c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 10 monclient(hunting): handle_auth_done global_id 74174 payload 824 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(184..184 src has 58..184) v4 ==== 7889+0+0 (secure 0 0 0) 0x7f1680049350 con 0x7f169009b630 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1690072a80 con 0x7f169008bdc0 2024-09-18T10:15:45.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168f7fe700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f167c035a70 0x7f167c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168f7fe700 10 monclient: get_auth_request con 0x7f167c035a70 auth_method 0 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 msgr2=0x7f169008a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f168e7fc700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 0x7f169008a260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f168400fb30 tx=0x7f1684007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168400dbc0 con 0x7f169008bdc0 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 10 monclient: dump: 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f168400dd20 con 0x7f169008bdc0 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 msgr2=0x7f169008a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 0x7f169008a260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f168400fb30 tx=0x7f1684007ea0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.472+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 msgr2=0x7f169008a260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:45.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.476+0000 7f1676ffd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169008bdc0 0x7f169008a260 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f168400fb30 tx=0x7f1684007ea0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.476+0000 7f168f7fe700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f167c035a70 0x7f167c037f20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f168800bf50 tx=0x7f1688017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168ffff700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 0x7f169006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900678a0 con 0x7f16900947d0 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 0x7f169006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient: get_auth_request con 0x7f16900947d0 auth_method 0 2024-09-18T10:15:45.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 10 monclient(hunting): handle_auth_done global_id 74177 payload 824 2024-09-18T10:15:45.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.484+0000 7f168e7fc700 1 -- 172.21.15.142:0/3191896591 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f16900940e0 con 0x7f16900947d0 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168e7fc700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 msgr2=0x7f169006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168e7fc700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 0x7f169006c370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1684019000 tx=0x7f168401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168401eb70 con 0x7f16900947d0 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 10 monclient: dump: 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:45.496 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:15:45.497 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 72, 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:15:45.498 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:15:45.499 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.500 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:15:45.501 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:15:45.502 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.503 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.504 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:15:45.505 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.506 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:15:45.507 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:15:45.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:15:45.509 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:15:45.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:15:45.511 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f168401ecd0 con 0x7f16900947d0 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 msgr2=0x7f169006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 0x7f169006c370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1684019000 tx=0x7f168401a000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 msgr2=0x7f169006c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f1676ffd700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f16900947d0 0x7f169006c370 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1684019000 tx=0x7f168401a000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f167c035a70 msgr2=0x7f167c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f167c035a70 0x7f167c037f20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f168800bf50 tx=0x7f1688017000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 10 monclient: shutdown 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 msgr2=0x7f1690093b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f168000ec80 tx=0x7f16800091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168f7fe700 1 -- 172.21.15.142:0/3191896591 reap_dead start 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 shutdown_connections 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 --2- 172.21.15.142:0/3191896591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f169009b630 0x7f1690093b40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 >> 172.21.15.142:0/3191896591 conn(0x7f1690070ef0 msgr2=0x7f1690097220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 shutdown_connections 2024-09-18T10:15:45.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:15:45.488+0000 7f168ffff700 1 -- 172.21.15.142:0/3191896591 wait complete. 2024-09-18T10:15:45.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:15:45.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:15:45.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.844+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 removed item id -19 name 'barr' from crush map v185) v1 ==== 122+0+0 (secure 0 0 0) 0x7f93b802cb30 con 0x7f93c809c930 2024-09-18T10:15:45.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.844+0000 7f93aeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.844+0000 7f93aeffd700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'barr' from crush map 2024-09-18T10:15:45.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.952+0000 7f93cd6e3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.952+0000 7f93cd6e3700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.952+0000 7f93cd6e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:45.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.952+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "barr"} v 0) v1 -- 0x7f93c8096a70 con 0x7f93c809c930 2024-09-18T10:15:45.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.956+0000 7f93aeffd700 1 -- 172.21.15.150:0/2249993925 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "barr"}]=0 device 'barr' does not appear in the crush map v185) v1 ==== 122+0+0 (secure 0 0 0) 0x7f93b801ba70 con 0x7f93c809c930 2024-09-18T10:15:45.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.956+0000 7f93aeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "barr"}] 2024-09-18T10:15:45.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.956+0000 7f93aeffd700 10 monclient: _finish_command 3 = system:0 device 'barr' does not appear in the crush map 2024-09-18T10:15:45.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 msgr2=0x7f93b4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 0x7f93b4037f20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f93bc003fd0 tx=0x7f93bc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 10 monclient: shutdown 2024-09-18T10:15:45.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 msgr2=0x7f93c811eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93c80971c0 tx=0x7f93b8016000 comp rx=0 tx=0).stop 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 shutdown_connections 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f93b4035a70 0x7f93b4037f20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93c8136760 0x7f93c811f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93c809cd00 0x7f93c811f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 --2- 172.21.15.150:0/2249993925 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93c809c930 0x7f93c811eeb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 >> 172.21.15.150:0/2249993925 conn(0x7f93c8068360 msgr2=0x7f93c809bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 shutdown_connections 2024-09-18T10:15:45.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:45.960+0000 7f93cd6e3700 1 -- 172.21.15.150:0/2249993925 wait complete. 2024-09-18T10:15:45.967 INFO:tasks.workunit.client.0.smithi150.stderr:device 'barr' does not appear in the crush map 2024-09-18T10:15:45.976 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T10:15:46.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: get_monmap_and_config 2024-09-18T10:15:46.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: build_initial_monmap 2024-09-18T10:15:46.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: monmap: 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:46.076613+0000 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:46.076613+0000 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:46.078 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 Processor -- start 2024-09-18T10:15:46.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 -- start start 2024-09-18T10:15:46.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: init 2024-09-18T10:15:46.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:46.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:15:46.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4dc09e6d0 0x7fd4dc0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-a con 0x7fd4dc09e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 0x7fd4dc0a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-c con 0x7fd4dc0a07d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4dc136a10 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc0a1110 con 0x7fd4dc136a10 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc09ee30 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc09e220 con 0x7fd4dc0a07d0 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:46.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1264700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 msgr2=0x7fd4dc0a0bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e3cc9700 10 monclient(hunting): authenticate will time out at 1228.171285s 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1264700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 0x7fd4dc0a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4dc09e6d0 0x7fd4dc0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e2266700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd4dc136a10 0x7fd4dc138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:44552/0 (socket says 172.21.15.150:44552) 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): get_auth_request con 0x7fd4dc09e6d0 auth_method 0 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e2266700 1 -- 172.21.15.150:0/3873877600 learned_addr learned my addr 172.21.15.150:0/3873877600 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:46.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.072+0000 7fd4e1a65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 10 monclient(hunting): handle_auth_done global_id 54708 payload 274 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 -- 172.21.15.150:0/3873877600 >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 msgr2=0x7fd4dc0a0bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 --2- 172.21.15.150:0/3873877600 >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 0x7fd4dc0a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 -- 172.21.15.150:0/3873877600 >> v1:172.21.15.142:6789/0 conn(0x7fd4dc136a10 legacy=0x7fd4dc138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 10 monclient: _finish_hunting 0 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 monclient: found mon.noname-a 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 -- 172.21.15.150:0/3873877600 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc09eaa0 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 -- 172.21.15.150:0/3873877600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc09e6d0 msgr2=0x7fd4dc0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e1a65700 1 --2- 172.21.15.150:0/3873877600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc0a0290 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc069dd0 tx=0x7fd4d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42420ce673b2b70b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 1 -- 172.21.15.150:0/3873877600 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800b9a0 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: dump: 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:46.087 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: _finish_auth 0 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e3cc9700 5 monclient: authenticate success, global_id 54708 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 1 -- 172.21.15.150:0/3873877600 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d800bb20 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e3cc9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e3cc9700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e3cc9700 4 set_mon_vals no callback set 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 1 -- 172.21.15.150:0/3873877600 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8011420 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:46.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.076+0000 7fd4e0a63700 10 monclient: dump: 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: shutdown 2024-09-18T10:15:46.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/3873877600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc09e6d0 msgr2=0x7fd4dc0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/3873877600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc0a0290 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc069dd0 tx=0x7fd4d80091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/3873877600 shutdown_connections 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/3873877600 >> v2:172.21.15.150:3300/0 conn(0x7fd4dc0a07d0 0x7fd4dc0a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/3873877600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc0a0290 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/3873877600 >> 172.21.15.150:0/3873877600 conn(0x7fd4dc0a58a0 msgr2=0x7fd4dc0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/3873877600 shutdown_connections 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/3873877600 wait complete. 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: build_initial_monmap 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: monmap: 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:46.090 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:46.091 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 Processor -- start 2024-09-18T10:15:46.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- start start 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: init 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc12e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-b con 0x7fd4dc09e6d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 0x7fd4dc130380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-c con 0x7fd4dc0a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): picked mon.noname-a con 0x7fd4dc136a10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:46.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc12dcd0 con 0x7fd4dc136a10 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc12f9a0 con 0x7fd4dc09e6d0 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.080+0000 7fd4e3cc9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e3cc9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc12ff60 con 0x7fd4dc0a07d0 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e3cc9700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e3cc9700 10 monclient(hunting): authenticate will time out at 1228.180033s 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1a65700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 msgr2=0x7fd4dc12e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1a65700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc12e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 0x7fd4dc130380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 0x7fd4dc130380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60788/0 (socket says 172.21.15.150:60788) 2024-09-18T10:15:46.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:43128/0 (socket says 172.21.15.150:43128) 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 1 -- 172.21.15.150:0/101531907 learned_addr learned my addr 172.21.15.150:0/101531907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): get_auth_request con 0x7fd4dc0a07d0 auth_method 0 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): get_auth_request con 0x7fd4dc136a10 auth_method 0 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:46.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e1264700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient(hunting): handle_auth_done global_id 54711 payload 824 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 -- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 msgr2=0x7fd4dc130380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 0x7fd4dc130380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 -- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 msgr2=0x7fd4dc12e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc12e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient: _finish_hunting 0 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 monclient: found mon.noname-a 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d8009420 con 0x7fd4dc136a10 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 -- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 msgr2=0x7fd4dc12c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4e2266700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400cb90 tx=0x7fd4d4005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d400aa80 con 0x7fd4dc136a10 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 10 monclient: dump: 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.100 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.084+0000 7fd4d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 5 monclient: authenticate success, global_id 54711 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _renew_subs 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d400ac00 con 0x7fd4dc136a10 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc12fc80 con 0x7fd4dc136a10 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40070f0 con 0x7fd4dc136a10 2024-09-18T10:15:46.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 10 monclient: dump: 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d3fff700 4 set_mon_vals no callback set 2024-09-18T10:15:46.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _renew_subs 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc12c7a0 con 0x7fd4dc136a10 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc063ed0 con 0x7fd4dc136a10 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd4d40074a0 con 0x7fd4dc136a10 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 0x7fd4c0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(185..185 src has 58..185) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fd4d40492e0 con 0x7fd4dc136a10 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e1a65700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 0x7fd4c0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:46.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.088+0000 7fd4e1a65700 10 monclient: get_auth_request con 0x7fd4c0035a70 auth_method 0 2024-09-18T10:15:46.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.092+0000 7fd4e1a65700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 0x7fd4c0037f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc069dd0 tx=0x7fd4d8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.100+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd4d4007750 con 0x7fd4dc136a10 2024-09-18T10:15:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.100+0000 7fd4d1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.100+0000 7fd4d1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:46.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.276+0000 7fd4e3cc9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:46.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.276+0000 7fd4e3cc9700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:46.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.276+0000 7fd4e3cc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.276+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fd4dc067b10 con 0x7fd4dc136a10 2024-09-18T10:15:46.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.844+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v186) v1 ==== 78+0+0 (secure 0 0 0) 0x7fd4d4010af0 con 0x7fd4dc136a10 2024-09-18T10:15:46.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.844+0000 7fd4d1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:46.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.844+0000 7fd4d1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:46.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.956+0000 7fd4e3cc9700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:46.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.956+0000 7fd4e3cc9700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:46.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.956+0000 7fd4e3cc9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:46.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:46.956+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7fd4dc06dd70 con 0x7fd4dc136a10 2024-09-18T10:15:47.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.860+0000 7fd4d1ffb700 1 -- 172.21.15.150:0/101531907 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v187) v1 ==== 78+0+0 (secure 0 0 0) 0x7fd4d401a360 con 0x7fd4dc136a10 2024-09-18T10:15:47.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.860+0000 7fd4d1ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:15:47.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.860+0000 7fd4d1ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:15:47.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 msgr2=0x7fd4c0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:47.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 0x7fd4c0037f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc069dd0 tx=0x7fd4d8016000 comp rx=0 tx=0).stop 2024-09-18T10:15:47.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 10 monclient: shutdown 2024-09-18T10:15:47.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:47.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 msgr2=0x7fd4dc12c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400cb90 tx=0x7fd4d4005340 comp rx=0 tx=0).stop 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 shutdown_connections 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4c0035a70 0x7fd4c0037f20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4dc136a10 0x7fd4dc12c260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4dc0a07d0 0x7fd4dc130380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 --2- 172.21.15.150:0/101531907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4dc09e6d0 0x7fd4dc12e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 >> 172.21.15.150:0/101531907 conn(0x7fd4dc0a58a0 msgr2=0x7fd4dc09b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 shutdown_connections 2024-09-18T10:15:47.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.864+0000 7fd4e3cc9700 1 -- 172.21.15.150:0/101531907 wait complete. 2024-09-18T10:15:47.886 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set create-compat 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: get_monmap_and_config 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: build_initial_monmap 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: monmap: 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:47.994093+0000 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:47.994093+0000 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:47.995 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:47.996 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 Processor -- start 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 -- start start 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: init 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cd0067b10 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:47.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd009c590 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-a con 0x7f6cd0137c00 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd00999f0 con 0x7f6cd0067b10 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd009e8e0 con 0x7f6cd0137c00 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd0099cb0 con 0x7f6cd009c590 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.988+0000 7f6cd5c62700 10 monclient(hunting): authenticate will time out at 1230.087848s 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:47.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60794/0 (socket says 172.21.15.150:60794) 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:43132/0 (socket says 172.21.15.150:43132) 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 -- 172.21.15.150:0/2071707689 learned_addr learned my addr 172.21.15.150:0/2071707689 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): get_auth_request con 0x7f6cd009c590 auth_method 0 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): get_auth_request con 0x7f6cd0137c00 auth_method 0 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:48.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient(hunting): handle_auth_done global_id 74183 payload 274 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 -- 172.21.15.150:0/2071707689 >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 msgr2=0x7f6cd0139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 --2- 172.21.15.150:0/2071707689 >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 -- 172.21.15.150:0/2071707689 >> v1:172.21.15.142:6789/0 conn(0x7f6cd0067b10 legacy=0x7f6cd009be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:48.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 -- 172.21.15.150:0/2071707689 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd0067ee0 con 0x7f6cd009c590 2024-09-18T10:15:48.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6ccffff700 1 --2- 172.21.15.150:0/2071707689 >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:48.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 -- 172.21.15.150:0/2071707689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 msgr2=0x7f6cd01376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cceffd700 1 --2- 172.21.15.150:0/2071707689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00099b0 tx=0x7f6cc0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5aec11e317cfbe server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 1 -- 172.21.15.150:0/2071707689 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0009ea0 con 0x7f6cd009c590 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: dump: 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:48.006 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cd5c62700 5 monclient: authenticate success, global_id 74183 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 1 -- 172.21.15.150:0/2071707689 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc000ba70 con 0x7f6cd009c590 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cd5c62700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cd5c62700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cd5c62700 4 set_mon_vals no callback set 2024-09-18T10:15:48.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 1 -- 172.21.15.150:0/2071707689 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc000be30 con 0x7f6cd009c590 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.992+0000 7f6cce7fc700 10 monclient: dump: 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:48.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: shutdown 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- 172.21.15.150:0/2071707689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 msgr2=0x7f6cd01376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/2071707689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00099b0 tx=0x7f6cc0009250 comp rx=0 tx=0).stop 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- 172.21.15.150:0/2071707689 shutdown_connections 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/2071707689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd009c590 0x7f6cd01376c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/2071707689 >> v2:172.21.15.142:3300/0 conn(0x7f6cd0137c00 0x7f6cd0139fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- 172.21.15.150:0/2071707689 >> 172.21.15.150:0/2071707689 conn(0x7f6cd00a2510 msgr2=0x7f6cd00a28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- 172.21.15.150:0/2071707689 shutdown_connections 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- 172.21.15.150:0/2071707689 wait complete. 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: build_initial_monmap 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: monmap: 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:48.009 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 Processor -- start 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- start start 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: init 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:48.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 0x7f6cd008fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-a con 0x7f6cd0067b10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 0x7f6cd008ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd009c590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): picked mon.noname-c con 0x7f6cd0137c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd00956d0 con 0x7f6cd0067b10 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd00a1d50 con 0x7f6cd009c590 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd0094b70 con 0x7f6cd0137c00 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:48.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cd5c62700 10 monclient(hunting): authenticate will time out at 1230.095130s 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cceffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 msgr2=0x7f6cd008ce90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6cceffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 0x7f6cd008ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 0x7f6cd008fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60804/0 (socket says 172.21.15.150:60804) 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 1 -- 172.21.15.150:0/659579776 learned_addr learned my addr 172.21.15.150:0/659579776 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): get_auth_request con 0x7f6cd0137c00 auth_method 0 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): get_auth_request con 0x7f6cd0067b10 auth_method 0 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:48.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:47.996+0000 7f6ccffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 10 monclient(hunting): handle_auth_done global_id 74186 payload 824 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 -- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 msgr2=0x7f6cd008ce90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 0x7f6cd008ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 -- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 msgr2=0x7f6cd008fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 0x7f6cd008fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:48.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 monclient: found mon.noname-c 2024-09-18T10:15:48.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:48.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc8009a20 con 0x7f6cd0137c00 2024-09-18T10:15:48.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccf7fe700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 0x7f6cd008fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 -- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 msgr2=0x7f6cd008d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccffff700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ef90 tx=0x7f6cc800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc800ce90 con 0x7f6cd0137c00 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: dump: 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:48.016 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 5 monclient: authenticate success, global_id 74186 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc800f040 con 0x7f6cd0137c00 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 10 monclient: _renew_subs 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc00094f0 con 0x7f6cd0137c00 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80146d0 con 0x7f6cd0137c00 2024-09-18T10:15:48.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cc77fe700 10 monclient: dump: 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:48.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6ccd7fa700 4 set_mon_vals no callback set 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 10 monclient: _renew_subs 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.000+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd008e980 con 0x7f6cd0137c00 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cd5c62700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cd5c62700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cd5c62700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd0063ed0 con 0x7f6cd0137c00 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6cc8014b00 con 0x7f6cd0137c00 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cc77fe700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 0x7f6cb0037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(187..187 src has 58..187) v4 ==== 7345+0+0 (secure 0 0 0) 0x7f6cc8049d10 con 0x7f6cd0137c00 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6ccf7fe700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 0x7f6cb0037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.004+0000 7f6ccf7fe700 10 monclient: get_auth_request con 0x7f6cb0035aa0 auth_method 0 2024-09-18T10:15:48.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.008+0000 7f6ccf7fe700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 0x7f6cb0037f50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009e40 tx=0x7f6cb8009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:48.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.016+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6cc8050080 con 0x7f6cd0137c00 2024-09-18T10:15:48.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.016+0000 7f6cc77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:48.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.016+0000 7f6cc77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.188+0000 7f6cd5c62700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.188+0000 7f6cd5c62700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.188+0000 7f6cd5c62700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.188+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f6cd008f550 con 0x7f6cd0137c00 2024-09-18T10:15:48.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.820+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 v188) v1 ==== 82+0+0 (secure 0 0 0) 0x7f6cc8018030 con 0x7f6cd0137c00 2024-09-18T10:15:48.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.820+0000 7f6cc77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.820+0000 7f6cc77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:15:48.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cd5c62700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cd5c62700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cd5c62700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:48.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set create-compat"} v 0) v1 -- 0x7f6cd0090930 con 0x7f6cd0137c00 2024-09-18T10:15:48.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cc77fe700 1 -- 172.21.15.150:0/659579776 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set create-compat"}]=0 compat weight-set already created v188) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6cc8018020 con 0x7f6cd0137c00 2024-09-18T10:15:48.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cc77fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set create-compat"}] 2024-09-18T10:15:48.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.932+0000 7f6cc77fe700 10 monclient: _finish_command 3 = system:0 compat weight-set already created 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 msgr2=0x7f6cb0037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 0x7f6cb0037f50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009e40 tx=0x7f6cb8009060 comp rx=0 tx=0).stop 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 10 monclient: shutdown 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 msgr2=0x7f6cd008d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800ef90 tx=0x7f6cc800b040 comp rx=0 tx=0).stop 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 shutdown_connections 2024-09-18T10:15:48.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cb0035aa0 0x7f6cb0037f50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd0137c00 0x7f6cd008d3d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd009c590 0x7f6cd008ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 --2- 172.21.15.150:0/659579776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd0067b10 0x7f6cd008fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 >> 172.21.15.150:0/659579776 conn(0x7f6cd00a2510 msgr2=0x7f6cd009abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 shutdown_connections 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:48.936+0000 7f6cd5c62700 1 -- 172.21.15.150:0/659579776 wait complete. 2024-09-18T10:15:48.943 INFO:tasks.workunit.client.0.smithi150.stderr:compat weight-set already created 2024-09-18T10:15:48.957 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket r1 rack root=default 2024-09-18T10:15:49.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 10 monclient: get_monmap_and_config 2024-09-18T10:15:49.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 10 monclient: build_initial_monmap 2024-09-18T10:15:49.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 10 monclient: monmap: 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:49.058621+0000 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:49.058621+0000 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:49.060 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:49.061 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:49.061 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:49.061 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 1 Processor -- start 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 1 -- start start 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.052+0000 7f8cce416700 10 monclient: init 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:15:49.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8cc809ad00 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8cc80991f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8cc80995f0 0x7f8cc813ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-a con 0x7f8cc80995f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc8098be0 con 0x7f8cc809ad00 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc8099ff0 con 0x7f8cc80991f0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc809d440 con 0x7f8cc80995f0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cce416700 10 monclient(hunting): authenticate will time out at 1231.152522s 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cc77fe700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f8cc80991f0 legacy=0x7f8cc813b7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8cc80995f0 0x7f8cc813ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:49.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cc7fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8cc809ad00 0x7f8cc809b0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:41190/0 (socket says 172.21.15.150:41190) 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8cc80995f0 0x7f8cc813ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57662/0 (socket says 172.21.15.150:57662) 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8cc7fff700 1 -- 172.21.15.150:0/391736116 learned_addr learned my addr 172.21.15.150:0/391736116 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): get_auth_request con 0x7f8cc80995f0 auth_method 0 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:49.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient(hunting): handle_auth_done global_id 54720 payload 274 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 -- 172.21.15.150:0/391736116 >> v1:172.21.15.150:6789/0 conn(0x7f8cc80991f0 legacy=0x7f8cc813b7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 -- 172.21.15.150:0/391736116 >> v1:172.21.15.142:6789/0 conn(0x7f8cc809ad00 legacy=0x7f8cc809b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient: _finish_hunting 0 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 monclient: found mon.noname-a 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:49.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.056+0000 7f8ccc9b3700 1 -- 172.21.15.150:0/391736116 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc809dd90 con 0x7f8cc80995f0 2024-09-18T10:15:49.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8ccc9b3700 1 -- 172.21.15.150:0/391736116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80995f0 msgr2=0x7f8cc813ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:49.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8ccc9b3700 1 --2- 172.21.15.150:0/391736116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80995f0 0x7f8cc813ef10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009e60 tx=0x7f8cc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb5b1749c54994f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:49.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 1 -- 172.21.15.150:0/391736116 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc000e070 con 0x7f8cc80995f0 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: dump: 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.069 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 5 monclient: authenticate success, global_id 54720 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 1 -- 172.21.15.150:0/391736116 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0012430 con 0x7f8cc80995f0 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 4 set_mon_vals no callback set 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 1 -- 172.21.15.150:0/391736116 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00127d0 con 0x7f8cc80995f0 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cc6ffd700 10 monclient: dump: 2024-09-18T10:15:49.071 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 10 monclient: shutdown 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- 172.21.15.150:0/391736116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80995f0 msgr2=0x7f8cc813ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 --2- 172.21.15.150:0/391736116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80995f0 0x7f8cc813ef10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009e60 tx=0x7f8cc0009780 comp rx=0 tx=0).stop 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- 172.21.15.150:0/391736116 shutdown_connections 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 --2- 172.21.15.150:0/391736116 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80995f0 0x7f8cc813ef10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- 172.21.15.150:0/391736116 >> 172.21.15.150:0/391736116 conn(0x7f8cc80a15e0 msgr2=0x7f8cc80a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- 172.21.15.150:0/391736116 shutdown_connections 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- 172.21.15.150:0/391736116 wait complete. 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 10 monclient: build_initial_monmap 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 10 monclient: monmap: 2024-09-18T10:15:49.073 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 Processor -- start 2024-09-18T10:15:49.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 1 -- start start 2024-09-18T10:15:49.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.060+0000 7f8cce416700 10 monclient: init 2024-09-18T10:15:49.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:49.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 0x7f8cc813cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-a con 0x7f8cc80991f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-c con 0x7f8cc80995f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 0x7f8cc813d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): picked mon.noname-b con 0x7f8cc809ad00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc813ec10 con 0x7f8cc80991f0 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc809b7f0 con 0x7f8cc809ad00 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:49.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80a0980 con 0x7f8cc80995f0 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cce416700 10 monclient(hunting): authenticate will time out at 1231.160655s 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8ccc9b3700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 msgr2=0x7f8cc813d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8ccc9b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 0x7f8cc813d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 0x7f8cc813cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60178/0 (socket says 172.21.15.150:60178) 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 -- 172.21.15.150:0/1314214854 learned_addr learned my addr 172.21.15.150:0/1314214854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): get_auth_request con 0x7f8cc80995f0 auth_method 0 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:49.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): get_auth_request con 0x7f8cc80991f0 auth_method 0 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient(hunting): handle_auth_done global_id 74189 payload 824 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 -- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 msgr2=0x7f8cc813d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 0x7f8cc813d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 -- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 msgr2=0x7f8cc813cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 0x7f8cc813cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 monclient: found mon.noname-c 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc77fe700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0009a20 con 0x7f8cc80995f0 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.064+0000 7f8cc7fff700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 0x7f8cc813cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cc77fe700 1 -- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 msgr2=0x7f8cc813d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:49.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cc77fe700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800b810 tx=0x7f8cb8009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8003e20 con 0x7f8cc80995f0 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: dump: 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:49.083 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: _finish_auth 0 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8006150 con 0x7f8cc80995f0 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 5 monclient: authenticate success, global_id 74189 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _renew_subs 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8006510 con 0x7f8cc80995f0 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc813e690 con 0x7f8cc80995f0 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:49.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cc67fc700 4 set_mon_vals no callback set 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 10 monclient: dump: 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _renew_subs 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc806f180 con 0x7f8cc80995f0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8cb801b3b0 con 0x7f8cc80995f0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 0x7f8cb4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(188..188 src has 58..188) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f8cb8048460 con 0x7f8cc80995f0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc8094fc0 con 0x7f8cc80995f0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cc7fff700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 0x7f8cb4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.068+0000 7f8cc7fff700 10 monclient: get_auth_request con 0x7f8cb4035a70 auth_method 0 2024-09-18T10:15:49.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.072+0000 7f8cc7fff700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 0x7f8cb4037f20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc005220 tx=0x7f8cbc007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:49.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.080+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8cb8017030 con 0x7f8cc80995f0 2024-09-18T10:15:49.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.080+0000 7f8caffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:49.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.080+0000 7f8caffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:49.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.252+0000 7f8cce416700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.252+0000 7f8cce416700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.252+0000 7f8cce416700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.252+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f8cc8063ed0 con 0x7f8cc80995f0 2024-09-18T10:15:49.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.872+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 added bucket r1 type rack to location {root=default} v189) v1 ==== 176+0+0 (secure 0 0 0) 0x7f8cb8017020 con 0x7f8cc80995f0 2024-09-18T10:15:49.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.872+0000 7f8caffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.872+0000 7f8caffff700 10 monclient: _finish_command 2 = system:0 added bucket r1 type rack to location {root=default} 2024-09-18T10:15:49.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.980+0000 7f8cce416700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.980+0000 7f8cce416700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.980+0000 7f8cce416700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:49.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.980+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]} v 0) v1 -- 0x7f8cc809ba00 con 0x7f8cc80995f0 2024-09-18T10:15:49.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.984+0000 7f8caffff700 1 -- 172.21.15.150:0/1314214854 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}]=0 bucket 'r1' already exists v189) v1 ==== 150+0+0 (secure 0 0 0) 0x7f8cb800e820 con 0x7f8cc80995f0 2024-09-18T10:15:49.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.984+0000 7f8caffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "r1", "type": "rack", "args": ["root=default"]}] 2024-09-18T10:15:49.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.984+0000 7f8caffff700 10 monclient: _finish_command 3 = system:0 bucket 'r1' already exists 2024-09-18T10:15:49.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 msgr2=0x7f8cb4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:49.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 0x7f8cb4037f20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc005220 tx=0x7f8cbc007f90 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 10 monclient: shutdown 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 msgr2=0x7f8cc813d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800b810 tx=0x7f8cb8009220 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 shutdown_connections 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8cb4035a70 0x7f8cb4037f20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cc809ad00 0x7f8cc813d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8cc80995f0 0x7f8cc813d060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 --2- 172.21.15.150:0/1314214854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc80991f0 0x7f8cc813cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 >> 172.21.15.150:0/1314214854 conn(0x7f8cc80a15e0 msgr2=0x7f8cc809f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 shutdown_connections 2024-09-18T10:15:49.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:49.988+0000 7f8cce416700 1 -- 172.21.15.150:0/1314214854 wait complete. 2024-09-18T10:15:49.996 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'r1' already exists 2024-09-18T10:15:50.004 INFO:tasks.workunit.client.0.smithi150.stderr:++ seq 1 32 2024-09-18T10:15:50.017 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:50.017 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h1 host rack=r1 2024-09-18T10:15:50.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 10 monclient: get_monmap_and_config 2024-09-18T10:15:50.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 10 monclient: build_initial_monmap 2024-09-18T10:15:50.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 10 monclient: monmap: 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:50.118669+0000 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:50.118669+0000 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:50.120 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 1 Processor -- start 2024-09-18T10:15:50.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 1 -- start start 2024-09-18T10:15:50.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.112+0000 7fdee8a3a700 10 monclient: init 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdee40a0290 0x7fdee409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-a con 0x7fdee40a0290 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 0x7fdee40a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-c con 0x7fdee40a0660 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 0x7fdee4138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-b con 0x7fdee4136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee409ef80 con 0x7fdee40a0290 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee40a1fb0 con 0x7fdee40a0660 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee409e360 con 0x7fdee4136a10 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee8a3a700 10 monclient(hunting): authenticate will time out at 1232.212203s 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee1d9b700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 msgr2=0x7fdee40a0a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee1d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 0x7fdee40a0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdee40a0290 0x7fdee409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:50.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 0x7fdee4138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:50.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdee40a0290 0x7fdee409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57694/0 (socket says 172.21.15.150:57694) 2024-09-18T10:15:50.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 -- 172.21.15.150:0/3431772105 learned_addr learned my addr 172.21.15.150:0/3431772105 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:50.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): get_auth_request con 0x7fdee40a0290 auth_method 0 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): get_auth_request con 0x7fdee4136a10 auth_method 0 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:50.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient(hunting): handle_auth_done global_id 54726 payload 274 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 -- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 msgr2=0x7fdee4138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 0x7fdee4138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 -- 172.21.15.150:0/3431772105 >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 msgr2=0x7fdee40a0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 --2- 172.21.15.150:0/3431772105 >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 0x7fdee40a0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 monclient: found mon.noname-a 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 -- 172.21.15.150:0/3431772105 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee409da50 con 0x7fdee40a0290 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee2d9d700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 0x7fdee4138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 -- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0290 msgr2=0x7fdee409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.116+0000 7fdee259c700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0290 0x7fdee409ea40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdee4064fb0 tx=0x7fded4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc94ec2077e455a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:50.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 1 -- 172.21.15.150:0/3431772105 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded400e070 con 0x7fdee40a0290 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: dump: 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: _finish_auth 0 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 5 monclient: authenticate success, global_id 54726 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 1 -- 172.21.15.150:0/3431772105 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded400bc30 con 0x7fdee40a0290 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 4 set_mon_vals no callback set 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 1 -- 172.21.15.150:0/3431772105 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40092e0 con 0x7fdee40a0290 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee159a700 10 monclient: dump: 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:50.133 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: shutdown 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0290 msgr2=0x7fdee409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0290 0x7fdee409ea40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdee4064fb0 tx=0x7fded4009840 comp rx=0 tx=0).stop 2024-09-18T10:15:50.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- 172.21.15.150:0/3431772105 shutdown_connections 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee4136a10 0x7fdee4138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/3431772105 >> v2:172.21.15.150:3300/0 conn(0x7fdee40a0660 0x7fdee40a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/3431772105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0290 0x7fdee409ea40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- 172.21.15.150:0/3431772105 >> 172.21.15.150:0/3431772105 conn(0x7fdee40a58a0 msgr2=0x7fdee40a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- 172.21.15.150:0/3431772105 shutdown_connections 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- 172.21.15.150:0/3431772105 wait complete. 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: build_initial_monmap 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: monmap: 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:50.135 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.136 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:50.136 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:50.136 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:50.136 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 Processor -- start 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- start start 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: init 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 0x7fdee4098700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-c con 0x7fdee40a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-a con 0x7fdee40a0660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 0x7fdee412c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): picked mon.noname-b con 0x7fdee4136a10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee41300d0 con 0x7fdee40a0660 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4099580 con 0x7fdee4136a10 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee412fa70 con 0x7fdee40a0290 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee8a3a700 10 monclient(hunting): authenticate will time out at 1232.219104s 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee2d9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 msgr2=0x7fdee412c680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee2d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 0x7fdee412c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:50.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 0x7fdee4098700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57702/0 (socket says 172.21.15.150:57702) 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 0x7fdee4098700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60190/0 (socket says 172.21.15.150:60190) 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 1 -- 172.21.15.150:0/452459800 learned_addr learned my addr 172.21.15.150:0/452459800 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): get_auth_request con 0x7fdee40a0660 auth_method 0 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): get_auth_request con 0x7fdee40a0290 auth_method 0 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:50.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.120+0000 7fdee259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 10 monclient(hunting): handle_auth_done global_id 54729 payload 824 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 -- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 msgr2=0x7fdee4098700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 0x7fdee4098700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 -- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 msgr2=0x7fdee412c680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 0x7fdee412c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 monclient: found mon.noname-a 2024-09-18T10:15:50.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc009420 con 0x7fdee40a0660 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 -- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 msgr2=0x7fdee4098c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee1d9b700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00b840 tx=0x7fdecc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc003e20 con 0x7fdee40a0660 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: dump: 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc006150 con 0x7fdee40a0660 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 5 monclient: authenticate success, global_id 54729 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 10 monclient: _renew_subs 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded4009ae0 con 0x7fdee40a0660 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc006510 con 0x7fdee40a0660 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdeda7fc700 10 monclient: dump: 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:50.146 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee0d99700 4 set_mon_vals no callback set 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 10 monclient: _renew_subs 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.124+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee412fcf0 con 0x7fdee40a0660 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee8a3a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee8a3a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee409bfa0 con 0x7fdee40a0660 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdecc01b3b0 con 0x7fdee40a0660 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdeda7fc700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 0x7fdec4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(189..189 src has 58..189) v4 ==== 7883+0+0 (secure 0 0 0) 0x7fdecc0484e0 con 0x7fdee40a0660 2024-09-18T10:15:50.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee259c700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 0x7fdec4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:50.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.128+0000 7fdee259c700 10 monclient: get_auth_request con 0x7fdec4035a70 auth_method 0 2024-09-18T10:15:50.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.132+0000 7fdee259c700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 0x7fdec4037f20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fded4009fd0 tx=0x7fded4018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:50.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.140+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdecc0196e0 con 0x7fdee40a0660 2024-09-18T10:15:50.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.140+0000 7fdeda7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:50.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.140+0000 7fdeda7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.308+0000 7fdee8a3a700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.308+0000 7fdee8a3a700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.308+0000 7fdee8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.308+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdee40a1850 con 0x7fdee40a0660 2024-09-18T10:15:50.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.884+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 added bucket h1 type host to location {rack=r1} v190) v1 ==== 166+0+0 (secure 0 0 0) 0x7fdecc017020 con 0x7fdee40a0660 2024-09-18T10:15:50.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.884+0000 7fdeda7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:50.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.884+0000 7fdeda7fc700 10 monclient: _finish_command 2 = system:0 added bucket h1 type host to location {rack=r1} 2024-09-18T10:15:51.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.992+0000 7fdee8a3a700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.992+0000 7fdee8a3a700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.992+0000 7fdee8a3a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.992+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdee409d6c0 con 0x7fdee40a0660 2024-09-18T10:15:51.000 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdeda7fc700 1 -- 172.21.15.150:0/452459800 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h1' already exists v190) v1 ==== 145+0+0 (secure 0 0 0) 0x7fdecc00d290 con 0x7fdee40a0660 2024-09-18T10:15:51.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdeda7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h1", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdeda7fc700 10 monclient: _finish_command 3 = system:0 bucket 'h1' already exists 2024-09-18T10:15:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 msgr2=0x7fdec4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 0x7fdec4037f20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fded4009fd0 tx=0x7fded4018000 comp rx=0 tx=0).stop 2024-09-18T10:15:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 10 monclient: shutdown 2024-09-18T10:15:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:51.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 msgr2=0x7fdee4098c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdecc00b840 tx=0x7fdecc0091d0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 shutdown_connections 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdec4035a70 0x7fdec4037f20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee4136a10 0x7fdee412c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdee40a0660 0x7fdee4098c40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 --2- 172.21.15.150:0/452459800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdee40a0290 0x7fdee4098700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 >> 172.21.15.150:0/452459800 conn(0x7fdee40a58a0 msgr2=0x7fdee409abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 shutdown_connections 2024-09-18T10:15:51.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:50.996+0000 7fdee8a3a700 1 -- 172.21.15.150:0/452459800 wait complete. 2024-09-18T10:15:51.005 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h1' already exists 2024-09-18T10:15:51.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:51.013 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h2 host rack=r1 2024-09-18T10:15:51.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: get_monmap_and_config 2024-09-18T10:15:51.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: build_initial_monmap 2024-09-18T10:15:51.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: monmap: 2024-09-18T10:15:51.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:51.120540+0000 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:51.120540+0000 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:51.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 Processor -- start 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 -- start start 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: init 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63ac0a5fb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63ac09fb20 0x7f63ac09e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-a con 0x7f63ac09fb20 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:51.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63ac0a09a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac09df10 con 0x7f63ac0a09a0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ac09dca0 con 0x7f63ac0a5fb0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ac09d1e0 con 0x7f63ac09fb20 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b3e7f700 10 monclient(hunting): authenticate will time out at 1233.214675s 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b1c1b700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f63ac0a5fb0 legacy=0x7f63ac0a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63ac09fb20 0x7f63ac09e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b241c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f63ac0a09a0 0x7f63ac139a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:41196/0 (socket says 172.21.15.150:41196) 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b241c700 1 -- 172.21.15.150:0/524376731 learned_addr learned my addr 172.21.15.150:0/524376731 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): get_auth_request con 0x7f63ac09fb20 auth_method 0 2024-09-18T10:15:51.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient(hunting): handle_auth_done global_id 54732 payload 274 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 1 -- 172.21.15.150:0/524376731 >> v1:172.21.15.150:6789/0 conn(0x7f63ac0a5fb0 legacy=0x7f63ac0a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 1 -- 172.21.15.150:0/524376731 >> v1:172.21.15.142:6789/0 conn(0x7f63ac0a09a0 legacy=0x7f63ac139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient: _finish_hunting 0 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 1 monclient: found mon.noname-a 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.116+0000 7f63b141a700 1 -- 172.21.15.150:0/524376731 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0a6380 con 0x7f63ac09fb20 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b141a700 1 -- 172.21.15.150:0/524376731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac09fb20 msgr2=0x7f63ac09e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b141a700 1 --2- 172.21.15.150:0/524376731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac09fb20 0x7f63ac09e610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0a1c50 tx=0x7f63a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292c80e72cd4254e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 1 -- 172.21.15.150:0/524376731 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800b9a0 con 0x7f63ac09fb20 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: dump: 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.130 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: _finish_auth 0 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 5 monclient: authenticate success, global_id 54732 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 1 -- 172.21.15.150:0/524376731 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a800bb00 con 0x7f63ac09fb20 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:51.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 4 set_mon_vals no callback set 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 1 -- 172.21.15.150:0/524376731 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800bea0 con 0x7f63ac09fb20 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b0c19700 10 monclient: dump: 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.132 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 10 monclient: shutdown 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 1 -- 172.21.15.150:0/524376731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac09fb20 msgr2=0x7f63ac09e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/524376731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac09fb20 0x7f63ac09e610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0a1c50 tx=0x7f63a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 1 -- 172.21.15.150:0/524376731 shutdown_connections 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/524376731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac09fb20 0x7f63ac09e610 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.120+0000 7f63b3e7f700 1 -- 172.21.15.150:0/524376731 >> 172.21.15.150:0/524376731 conn(0x7f63ac071ae0 msgr2=0x7f63ac0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- 172.21.15.150:0/524376731 shutdown_connections 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- 172.21.15.150:0/524376731 wait complete. 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient: build_initial_monmap 2024-09-18T10:15:51.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient: monmap: 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 Processor -- start 2024-09-18T10:15:51.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- start start 2024-09-18T10:15:51.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient: init 2024-09-18T10:15:51.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-a con 0x7f63ac0a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 0x7f63ac142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-b con 0x7f63ac0a5fb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63ac142560 0x7f63ac08e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): picked mon.noname-c con 0x7f63ac142560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac1429a0 con 0x7f63ac0a09a0 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac099510 con 0x7f63ac0a5fb0 2024-09-18T10:15:51.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ac090210 con 0x7f63ac142560 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b3e7f700 10 monclient(hunting): authenticate will time out at 1233.223005s 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b141a700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 msgr2=0x7f63ac142020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b141a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 0x7f63ac142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63ac142560 0x7f63ac08e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57710/0 (socket says 172.21.15.150:57710) 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 1 -- 172.21.15.150:0/526565871 learned_addr learned my addr 172.21.15.150:0/526565871 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): get_auth_request con 0x7f63ac0a09a0 auth_method 0 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:51.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): get_auth_request con 0x7f63ac142560 auth_method 0 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b241c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 10 monclient(hunting): handle_auth_done global_id 54735 payload 824 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.124+0000 7f63b1c1b700 1 -- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63ac142560 msgr2=0x7f63ac08e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:51.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63ac142560 0x7f63ac08e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 -- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 msgr2=0x7f63ac142020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:51.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 0x7f63ac142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 monclient: found mon.noname-a 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a8009420 con 0x7f63ac0a09a0 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 -- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 msgr2=0x7f63ac141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b1c1b700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b840 tx=0x7f63a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0003e20 con 0x7f63ac0a09a0 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: dump: 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:51.144 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:51.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a0006150 con 0x7f63ac0a09a0 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0006510 con 0x7f63ac0a09a0 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639dffb700 10 monclient: dump: 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:51.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 5 monclient: authenticate success, global_id 54735 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 10 monclient: _renew_subs 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f639ffff700 4 set_mon_vals no callback set 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ac09c2d0 con 0x7f63ac0a09a0 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 10 monclient: _renew_subs 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.128+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ac091750 con 0x7f63ac0a09a0 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f63a001b3b0 con 0x7f63ac0a09a0 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b3e7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:51.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b3e7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b3e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ac092a80 con 0x7f63ac0a09a0 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f639dffb700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 0x7f6390037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(190..190 src has 58..190) v4 ==== 8093+0+0 (secure 0 0 0) 0x7f63a0048600 con 0x7f63ac0a09a0 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b141a700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 0x7f6390037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.132+0000 7f63b141a700 10 monclient: get_auth_request con 0x7f6390035a70 auth_method 0 2024-09-18T10:15:51.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.136+0000 7f63b141a700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 0x7f6390037f20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63a80093a0 tx=0x7f63a8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:51.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.144+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f63a00196e0 con 0x7f63ac0a09a0 2024-09-18T10:15:51.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.144+0000 7f639dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:51.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.144+0000 7f639dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:51.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.312+0000 7f63b3e7f700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.312+0000 7f63b3e7f700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.312+0000 7f63b3e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:51.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.312+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63ac09e1f0 con 0x7f63ac0a09a0 2024-09-18T10:15:51.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.892+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 added bucket h2 type host to location {rack=r1} v191) v1 ==== 166+0+0 (secure 0 0 0) 0x7f63a0017030 con 0x7f63ac0a09a0 2024-09-18T10:15:51.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.892+0000 7f639dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:51.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:51.892+0000 7f639dffb700 10 monclient: _finish_command 2 = system:0 added bucket h2 type host to location {rack=r1} 2024-09-18T10:15:52.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:52.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f63ac063ed0 con 0x7f63ac0a09a0 2024-09-18T10:15:52.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f639dffb700 1 -- 172.21.15.150:0/526565871 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h2' already exists v191) v1 ==== 145+0+0 (secure 0 0 0) 0x7f63a0017030 con 0x7f63ac0a09a0 2024-09-18T10:15:52.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f639dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h2", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f639dffb700 10 monclient: _finish_command 3 = system:0 bucket 'h2' already exists 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 msgr2=0x7f6390037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 0x7f6390037f20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f63a80093a0 tx=0x7f63a8016000 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 10 monclient: shutdown 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 msgr2=0x7f63ac141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b840 tx=0x7f63a00091d0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 shutdown_connections 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6390035a70 0x7f6390037f20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63ac142560 0x7f63ac08e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63ac0a5fb0 0x7f63ac142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 --2- 172.21.15.150:0/526565871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63ac0a09a0 0x7f63ac141ae0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 >> 172.21.15.150:0/526565871 conn(0x7f63ac071ae0 msgr2=0x7f63ac09a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:52.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 shutdown_connections 2024-09-18T10:15:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.004+0000 7f63b3e7f700 1 -- 172.21.15.150:0/526565871 wait complete. 2024-09-18T10:15:52.014 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h2' already exists 2024-09-18T10:15:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:52.023 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h3 host rack=r1 2024-09-18T10:15:52.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: get_monmap_and_config 2024-09-18T10:15:52.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: build_initial_monmap 2024-09-18T10:15:52.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: monmap: 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:52.125754+0000 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:52.125754+0000 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:52.127 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 Processor -- start 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 -- start start 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: init 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 0x7f349009b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): picked mon.noname-b con 0x7f349009ad00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:52.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34900991f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34900995c0 0x7f349009c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): picked mon.noname-a con 0x7f34900995c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3490098be0 con 0x7f34900991f0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349009d440 con 0x7f34900995c0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3490099ff0 con 0x7f349009ad00 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3496922700 10 monclient(hunting): authenticate will time out at 1234.219238s 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348f7fe700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f34900991f0 legacy=0x7f349009bfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34900995c0 0x7f349009c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 0x7f349009b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:52.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34900995c0 0x7f349009c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57716/0 (socket says 172.21.15.150:57716) 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 1 -- 172.21.15.150:0/3566922140 learned_addr learned my addr 172.21.15.150:0/3566922140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): get_auth_request con 0x7f34900995c0 auth_method 0 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): get_auth_request con 0x7f349009ad00 auth_method 0 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:52.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.120+0000 7f348ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 10 monclient(hunting): handle_auth_done global_id 54738 payload 274 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 -- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 msgr2=0x7f349009b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 0x7f349009b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 -- 172.21.15.150:0/3566922140 >> v1:172.21.15.150:6789/0 conn(0x7f34900991f0 legacy=0x7f349009bfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 10 monclient: _finish_hunting 0 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 monclient: found mon.noname-a 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 -- 172.21.15.150:0/3566922140 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349009b610 con 0x7f34900995c0 2024-09-18T10:15:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348ffff700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 0x7f349009b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 -- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34900995c0 msgr2=0x7f349009c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3494ebf700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34900995c0 0x7f349009c6e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f20 tx=0x7f3488009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a64ab2124abf8e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 1 -- 172.21.15.150:0/3566922140 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348800e070 con 0x7f34900995c0 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: dump: 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:52.137 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: _finish_auth 0 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3496922700 5 monclient: authenticate success, global_id 54738 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 1 -- 172.21.15.150:0/3566922140 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348800bbe0 con 0x7f34900995c0 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3496922700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3496922700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:52.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 1 -- 172.21.15.150:0/3566922140 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880092e0 con 0x7f34900995c0 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f3496922700 4 set_mon_vals no callback set 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.124+0000 7f348effd700 10 monclient: dump: 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:52.139 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: shutdown 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34900995c0 msgr2=0x7f349009c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34900995c0 0x7f349009c6e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f20 tx=0x7f3488009840 comp rx=0 tx=0).stop 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- 172.21.15.150:0/3566922140 shutdown_connections 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34900995c0 0x7f349009c6e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- 172.21.15.150:0/3566922140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f349009ad00 0x7f349009b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- 172.21.15.150:0/3566922140 >> 172.21.15.150:0/3566922140 conn(0x7f34900a15e0 msgr2=0x7f34900a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- 172.21.15.150:0/3566922140 shutdown_connections 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- 172.21.15.150:0/3566922140 wait complete. 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: build_initial_monmap 2024-09-18T10:15:52.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: monmap: 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 Processor -- start 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- start start 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: init 2024-09-18T10:15:52.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 0x7f3490145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): picked mon.noname-b con 0x7f34900991f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): picked mon.noname-c con 0x7f34900995c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 0x7f3490143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): picked mon.noname-a con 0x7f349009ad00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349006e720 con 0x7f349009ad00 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349009b7f0 con 0x7f34900991f0 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:52.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3490143870 con 0x7f34900995c0 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3496922700 10 monclient(hunting): authenticate will time out at 1234.226744s 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348ffff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 msgr2=0x7f3490145e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 0x7f3490145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 0x7f3490143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60204/0 (socket says 172.21.15.150:60204) 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 -- 172.21.15.150:0/1754228140 learned_addr learned my addr 172.21.15.150:0/1754228140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): get_auth_request con 0x7f34900995c0 auth_method 0 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:52.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): get_auth_request con 0x7f349009ad00 auth_method 0 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient(hunting): handle_auth_done global_id 74204 payload 824 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 -- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 msgr2=0x7f3490145e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 0x7f3490145e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 -- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 msgr2=0x7f3490143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 0x7f3490143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f348f7fe700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484009420 con 0x7f34900995c0 2024-09-18T10:15:52.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.128+0000 7f3494ebf700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 0x7f3490143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f348f7fe700 1 -- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 msgr2=0x7f3490142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f348f7fe700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f348000b810 tx=0x7f34800091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480003e20 con 0x7f34900995c0 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:52.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: dump: 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: _finish_auth 0 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:52.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3480006150 con 0x7f34900995c0 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 5 monclient: authenticate success, global_id 74204 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3480006510 con 0x7f34900995c0 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 10 monclient: _renew_subs 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3488009ae0 con 0x7f34900995c0 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3477fff700 10 monclient: dump: 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:52.149 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f348e7fc700 4 set_mon_vals no callback set 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 10 monclient: _renew_subs 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.132+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3490126f30 con 0x7f34900995c0 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3496922700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:52.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3496922700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3496922700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349006ee40 con 0x7f34900995c0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f348001b3b0 con 0x7f34900995c0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3477fff700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 0x7f347c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(191..191 src has 58..191) v4 ==== 8303+0+0 (secure 0 0 0) 0x7f3480048770 con 0x7f34900995c0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f348ffff700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 0x7f347c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.136+0000 7f348ffff700 10 monclient: get_auth_request con 0x7f347c035a70 auth_method 0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.140+0000 7f348ffff700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 0x7f347c037f20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f34840096a0 tx=0x7f34840091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:52.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.144+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3480019880 con 0x7f34900995c0 2024-09-18T10:15:52.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.144+0000 7f3477fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:52.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.144+0000 7f3477fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:52.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.312+0000 7f3496922700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.312+0000 7f3496922700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.312+0000 7f3496922700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:52.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.312+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f349009ba20 con 0x7f34900995c0 2024-09-18T10:15:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.908+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 added bucket h3 type host to location {rack=r1} v192) v1 ==== 166+0+0 (secure 0 0 0) 0x7f3480017030 con 0x7f34900995c0 2024-09-18T10:15:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.908+0000 7f3477fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:52.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:52.908+0000 7f3477fff700 10 monclient: _finish_command 2 = system:0 added bucket h3 type host to location {rack=r1} 2024-09-18T10:15:53.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3496922700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3496922700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3496922700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3490063ed0 con 0x7f34900995c0 2024-09-18T10:15:53.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3477fff700 1 -- 172.21.15.150:0/1754228140 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h3' already exists v192) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3480017020 con 0x7f34900995c0 2024-09-18T10:15:53.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3477fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h3", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.008+0000 7f3477fff700 10 monclient: _finish_command 3 = system:0 bucket 'h3' already exists 2024-09-18T10:15:53.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 msgr2=0x7f347c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:53.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 0x7f347c037f20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f34840096a0 tx=0x7f34840091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 10 monclient: shutdown 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 msgr2=0x7f3490142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f348000b810 tx=0x7f34800091d0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 shutdown_connections 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f347c035a70 0x7f347c037f20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f349009ad00 0x7f3490143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34900995c0 0x7f3490142df0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 --2- 172.21.15.150:0/1754228140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34900991f0 0x7f3490145e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 >> 172.21.15.150:0/1754228140 conn(0x7f34900a15e0 msgr2=0x7f349009f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 shutdown_connections 2024-09-18T10:15:53.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.012+0000 7f3496922700 1 -- 172.21.15.150:0/1754228140 wait complete. 2024-09-18T10:15:53.020 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h3' already exists 2024-09-18T10:15:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:53.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h4 host rack=r1 2024-09-18T10:15:53.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: get_monmap_and_config 2024-09-18T10:15:53.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: build_initial_monmap 2024-09-18T10:15:53.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: monmap: 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:53.120663+0000 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:53.120663+0000 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:53.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 Processor -- start 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 -- start start 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: init 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 0x7f03180a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-c con 0x7f031809d000 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-b con 0x7f031809e780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03180a0ba0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180699a0 con 0x7f03180a0ba0 2024-09-18T10:15:53.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03180a1640 con 0x7f031809d000 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031809e3d0 con 0x7f031809e780 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f031e1f8700 10 monclient(hunting): authenticate will time out at 1235.214693s 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f03177fe700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 msgr2=0x7f03180a0120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0317fff700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f03180a0ba0 legacy=0x7f03181389b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f03177fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 0x7f03180a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60210/0 (socket says 172.21.15.150:60210) 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 -- 172.21.15.150:0/1842397469 learned_addr learned my addr 172.21.15.150:0/1842397469 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): get_auth_request con 0x7f031809e780 auth_method 0 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:53.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient(hunting): handle_auth_done global_id 74207 payload 274 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 -- 172.21.15.150:0/1842397469 >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 msgr2=0x7f03180a0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 --2- 172.21.15.150:0/1842397469 >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 0x7f03180a0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 -- 172.21.15.150:0/1842397469 >> v1:172.21.15.150:6789/0 conn(0x7f03180a0ba0 legacy=0x7f03181389b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.116+0000 7f0316ffd700 1 -- 172.21.15.150:0/1842397469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031809ebf0 con 0x7f031809e780 2024-09-18T10:15:53.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f0316ffd700 1 -- 172.21.15.150:0/1842397469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 msgr2=0x7f03180a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:53.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f0316ffd700 1 --2- 172.21.15.150:0/1842397469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03080099b0 tx=0x7f0308009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e974f4a058bc9b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:53.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 1 -- 172.21.15.150:0/1842397469 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800bb50 con 0x7f031809e780 2024-09-18T10:15:53.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: dump: 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:53.130 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 1 -- 172.21.15.150:0/1842397469 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030800bcb0 con 0x7f031809e780 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 1 -- 172.21.15.150:0/1842397469 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0308011670 con 0x7f031809e780 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f03167fc700 10 monclient: dump: 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.131 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 5 monclient: authenticate success, global_id 74207 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 4 set_mon_vals no callback set 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 10 monclient: shutdown 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 -- 172.21.15.150:0/1842397469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 msgr2=0x7f03180a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:53.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 --2- 172.21.15.150:0/1842397469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f03080099b0 tx=0x7f0308009250 comp rx=0 tx=0).stop 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 -- 172.21.15.150:0/1842397469 shutdown_connections 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 --2- 172.21.15.150:0/1842397469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f031809e780 0x7f03180a0660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 --2- 172.21.15.150:0/1842397469 >> v2:172.21.15.150:3300/0 conn(0x7f031809d000 0x7f03180a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 -- 172.21.15.150:0/1842397469 >> 172.21.15.150:0/1842397469 conn(0x7f03180a5830 msgr2=0x7f03180a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 -- 172.21.15.150:0/1842397469 shutdown_connections 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.120+0000 7f031e1f8700 1 -- 172.21.15.150:0/1842397469 wait complete. 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient: build_initial_monmap 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient: monmap: 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:53.133 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:53.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:53.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:53.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:53.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 Processor -- start 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 -- start start 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient: init 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f031809d000 0x7f0318141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-a con 0x7f031809d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 0x7f0318141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-b con 0x7f031809e780 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): picked mon.noname-c con 0x7f03180a0ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:53.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0318090700 con 0x7f031809d000 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031809c220 con 0x7f031809e780 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0318091940 con 0x7f03180a0ba0 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f031e1f8700 10 monclient(hunting): authenticate will time out at 1235.223118s 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0316ffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 msgr2=0x7f0318141fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0316ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 0x7f0318141fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f031809d000 0x7f0318141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60220/0 (socket says 172.21.15.150:60220) 2024-09-18T10:15:53.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 1 -- 172.21.15.150:0/3733169833 learned_addr learned my addr 172.21.15.150:0/3733169833 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): get_auth_request con 0x7f03180a0ba0 auth_method 0 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): get_auth_request con 0x7f031809d000 auth_method 0 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f0317fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:53.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.124+0000 7f03177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 10 monclient(hunting): handle_auth_done global_id 74210 payload 824 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 -- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 msgr2=0x7f0318141fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 0x7f0318141fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 -- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f031809d000 msgr2=0x7f0318141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f031809d000 0x7f0318141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 monclient: found mon.noname-c 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080094f0 con 0x7f03180a0ba0 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 -- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 msgr2=0x7f03181424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:53.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0317fff700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0310009fc0 tx=0x7f0310009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031000ce20 con 0x7f03180a0ba0 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: dump: 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:53.144 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: _finish_auth 0 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0310016430 con 0x7f03180a0ba0 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03100165e0 con 0x7f03180a0ba0 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 5 monclient: authenticate success, global_id 74210 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 10 monclient: _renew_subs 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0318090040 con 0x7f03180a0ba0 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f0315ffb700 4 set_mon_vals no callback set 2024-09-18T10:15:53.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f02ff7fe700 10 monclient: dump: 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 10 monclient: _renew_subs 2024-09-18T10:15:53.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.128+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0318099450 con 0x7f03180a0ba0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f031e1f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f031e1f8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f031e1f8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0318091b00 con 0x7f03180a0ba0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0310012020 con 0x7f03180a0ba0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f02ff7fe700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 0x7f0304037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(192..192 src has 58..192) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f03100492f0 con 0x7f03180a0ba0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f03177fe700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 0x7f0304037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:53.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f03177fe700 10 monclient: get_auth_request con 0x7f0304035a70 auth_method 0 2024-09-18T10:15:53.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.132+0000 7f03177fe700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 0x7f0304037f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f030c0039d0 tx=0x7f030c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:53.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.140+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0310016a50 con 0x7f03180a0ba0 2024-09-18T10:15:53.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.140+0000 7f02ff7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:53.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.140+0000 7f02ff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:53.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.308+0000 7f031e1f8700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.308+0000 7f031e1f8700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.308+0000 7f031e1f8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:53.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.308+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0318063ed0 con 0x7f03180a0ba0 2024-09-18T10:15:53.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.920+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 added bucket h4 type host to location {rack=r1} v193) v1 ==== 166+0+0 (secure 0 0 0) 0x7f03100072f0 con 0x7f03180a0ba0 2024-09-18T10:15:53.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.920+0000 7f02ff7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:53.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:53.920+0000 7f02ff7fe700 10 monclient: _finish_command 2 = system:0 added bucket h4 type host to location {rack=r1} 2024-09-18T10:15:54.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f031e1f8700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f031e1f8700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f031e1f8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:54.034 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0318067b10 con 0x7f03180a0ba0 2024-09-18T10:15:54.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f02ff7fe700 1 -- 172.21.15.150:0/3733169833 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h4' already exists v193) v1 ==== 145+0+0 (secure 0 0 0) 0x7f03100240b0 con 0x7f03180a0ba0 2024-09-18T10:15:54.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f02ff7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h4", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.036 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.028+0000 7f02ff7fe700 10 monclient: _finish_command 3 = system:0 bucket 'h4' already exists 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 msgr2=0x7f0304037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 0x7f0304037f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f030c0039d0 tx=0x7f030c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 10 monclient: shutdown 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 msgr2=0x7f03181424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0310009fc0 tx=0x7f0310009730 comp rx=0 tx=0).stop 2024-09-18T10:15:54.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 shutdown_connections 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0304035a70 0x7f0304037f20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03180a0ba0 0x7f03181424f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f031809e780 0x7f0318141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 --2- 172.21.15.150:0/3733169833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f031809d000 0x7f0318141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 >> 172.21.15.150:0/3733169833 conn(0x7f03180a5830 msgr2=0x7f03180a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 shutdown_connections 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.032+0000 7f031e1f8700 1 -- 172.21.15.150:0/3733169833 wait complete. 2024-09-18T10:15:54.040 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h4' already exists 2024-09-18T10:15:54.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:54.049 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h5 host rack=r1 2024-09-18T10:15:54.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: get_monmap_and_config 2024-09-18T10:15:54.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: build_initial_monmap 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: monmap: 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:54.149781+0000 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:54.149781+0000 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:54.151 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 Processor -- start 2024-09-18T10:15:54.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 -- start start 2024-09-18T10:15:54.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: init 2024-09-18T10:15:54.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:54.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34100a0290 0x7f341009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): picked mon.noname-a con 0x7f34100a0290 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34100a0660 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 0x7f3410139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): picked mon.noname-c con 0x7f3410137a20 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341009e360 con 0x7f34100a0660 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341009ef80 con 0x7f34100a0290 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34100a1fb0 con 0x7f3410137a20 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:54.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f3414990700 10 monclient(hunting): authenticate will time out at 1236.243164s 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340ed9d700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 msgr2=0x7f3410139e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340ed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 0x7f3410139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34100a0290 0x7f341009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340dd9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f34100a0660 0x7f34100a0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:41208/0 (socket says 172.21.15.150:41208) 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f34100a0290 0x7f341009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57744/0 (socket says 172.21.15.150:57744) 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340dd9b700 1 -- 172.21.15.150:0/3395805602 learned_addr learned my addr 172.21.15.150:0/3395805602 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): get_auth_request con 0x7f34100a0290 auth_method 0 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:54.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.144+0000 7f340e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 10 monclient(hunting): handle_auth_done global_id 54747 payload 274 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 -- 172.21.15.150:0/3395805602 >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 msgr2=0x7f3410139e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 --2- 172.21.15.150:0/3395805602 >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 0x7f3410139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 -- 172.21.15.150:0/3395805602 >> v1:172.21.15.142:6789/0 conn(0x7f34100a0660 legacy=0x7f34100a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 monclient: found mon.noname-a 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 -- 172.21.15.150:0/3395805602 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341009da50 con 0x7f34100a0290 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 -- 172.21.15.150:0/3395805602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34100a0290 msgr2=0x7f341009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340e59c700 1 --2- 172.21.15.150:0/3395805602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34100a0290 0x7f341009ea40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f341006fdc0 tx=0x7f34040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9798ce3f03bb147 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 1 -- 172.21.15.150:0/3395805602 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340400b9a0 con 0x7f34100a0290 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: dump: 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.163 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: _finish_auth 0 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f3414990700 5 monclient: authenticate success, global_id 54747 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 1 -- 172.21.15.150:0/3395805602 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340400bb20 con 0x7f34100a0290 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f3414990700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f3414990700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:54.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 1 -- 172.21.15.150:0/3395805602 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404011420 con 0x7f34100a0290 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f3414990700 4 set_mon_vals no callback set 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.148+0000 7f340d59a700 10 monclient: dump: 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:54.165 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: shutdown 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- 172.21.15.150:0/3395805602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34100a0290 msgr2=0x7f341009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- 172.21.15.150:0/3395805602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34100a0290 0x7f341009ea40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f341006fdc0 tx=0x7f34040091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- 172.21.15.150:0/3395805602 shutdown_connections 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- 172.21.15.150:0/3395805602 >> v2:172.21.15.150:3300/0 conn(0x7f3410137a20 0x7f3410139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- 172.21.15.150:0/3395805602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34100a0290 0x7f341009ea40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- 172.21.15.150:0/3395805602 >> 172.21.15.150:0/3395805602 conn(0x7f34100a58a0 msgr2=0x7f34100a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- 172.21.15.150:0/3395805602 shutdown_connections 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- 172.21.15.150:0/3395805602 wait complete. 2024-09-18T10:15:54.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: build_initial_monmap 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: monmap: 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 Processor -- start 2024-09-18T10:15:54.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- start start 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: init 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 0x7f3410098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): picked mon.noname-b con 0x7f34100a0290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 0x7f3410098d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): picked mon.noname-c con 0x7f34100a0660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): picked mon.noname-a con 0x7f3410137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341012ebb0 con 0x7f3410137a20 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:54.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341009c0e0 con 0x7f34100a0290 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341009d3f0 con 0x7f34100a0660 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f3414990700 10 monclient(hunting): authenticate will time out at 1236.250883s 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340e59c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 msgr2=0x7f3410098800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 0x7f3410098800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 0x7f3410098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57756/0 (socket says 172.21.15.150:57756) 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 -- 172.21.15.150:0/3649112074 learned_addr learned my addr 172.21.15.150:0/3649112074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): get_auth_request con 0x7f3410137a20 auth_method 0 2024-09-18T10:15:54.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 0x7f3410098d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60222/0 (socket says 172.21.15.150:60222) 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): get_auth_request con 0x7f34100a0660 auth_method 0 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:54.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 10 monclient(hunting): handle_auth_done global_id 54750 payload 824 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 -- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 msgr2=0x7f3410098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 0x7f3410098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.152+0000 7f340ed9d700 1 -- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 msgr2=0x7f3410098800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 0x7f3410098800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 1 monclient: found mon.noname-a 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404009420 con 0x7f3410137a20 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 1 -- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 msgr2=0x7f341012c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:54.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340ed9d700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f340800b5e0 tx=0x7f34080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:54.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408003e20 con 0x7f3410137a20 2024-09-18T10:15:54.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: dump: 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 5 monclient: authenticate success, global_id 54750 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408006150 con 0x7f3410137a20 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 10 monclient: _renew_subs 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341012efb0 con 0x7f3410137a20 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f340cd99700 4 set_mon_vals no callback set 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 10 monclient: _renew_subs 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3410131150 con 0x7f3410137a20 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408006510 con 0x7f3410137a20 2024-09-18T10:15:54.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.156+0000 7f33fe7fc700 10 monclient: dump: 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f340801a360 con 0x7f3410137a20 2024-09-18T10:15:54.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f3414990700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f3414990700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f3414990700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34100a6030 con 0x7f3410137a20 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f33fe7fc700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 0x7f33f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(193..193 src has 167..193) v4 ==== 8723+0+0 (secure 0 0 0) 0x7f3408048410 con 0x7f3410137a20 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f340e59c700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 0x7f33f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.160+0000 7f340e59c700 10 monclient: get_auth_request con 0x7f33f0035a70 auth_method 0 2024-09-18T10:15:54.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.164+0000 7f340e59c700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 0x7f33f0037f20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f341006fdc0 tx=0x7f3404016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:54.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.172+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3408018690 con 0x7f3410137a20 2024-09-18T10:15:54.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.172+0000 7f33fe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:54.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.172+0000 7f33fe7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:54.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.340+0000 7f3414990700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.340+0000 7f3414990700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.340+0000 7f3414990700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:54.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.340+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f341009d6c0 con 0x7f3410137a20 2024-09-18T10:15:54.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.920+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 added bucket h5 type host to location {rack=r1} v194) v1 ==== 166+0+0 (secure 0 0 0) 0x7f3408017020 con 0x7f3410137a20 2024-09-18T10:15:54.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.920+0000 7f33fe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:54.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:54.920+0000 7f33fe7fc700 10 monclient: _finish_command 2 = system:0 added bucket h5 type host to location {rack=r1} 2024-09-18T10:15:55.041 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f3414990700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f3414990700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f3414990700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3410063ed0 con 0x7f3410137a20 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f33fe7fc700 1 -- 172.21.15.150:0/3649112074 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h5' already exists v194) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3408017dd0 con 0x7f3410137a20 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f33fe7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h5", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:55.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.036+0000 7f33fe7fc700 10 monclient: _finish_command 3 = system:0 bucket 'h5' already exists 2024-09-18T10:15:55.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 msgr2=0x7f33f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:55.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 0x7f33f0037f20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f341006fdc0 tx=0x7f3404016000 comp rx=0 tx=0).stop 2024-09-18T10:15:55.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 10 monclient: shutdown 2024-09-18T10:15:55.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 msgr2=0x7f341012c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f340800b5e0 tx=0x7f34080090f0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 shutdown_connections 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f33f0035a70 0x7f33f0037f20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3410137a20 0x7f341012c8e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34100a0660 0x7f3410098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 --2- 172.21.15.150:0/3649112074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34100a0290 0x7f3410098800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 >> 172.21.15.150:0/3649112074 conn(0x7f34100a58a0 msgr2=0x7f34100a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 shutdown_connections 2024-09-18T10:15:55.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.040+0000 7f3414990700 1 -- 172.21.15.150:0/3649112074 wait complete. 2024-09-18T10:15:55.047 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h5' already exists 2024-09-18T10:15:55.059 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:55.059 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h6 host rack=r1 2024-09-18T10:15:55.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: get_monmap_and_config 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: build_initial_monmap 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: monmap: 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:55.161945+0000 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:55.161945+0000 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:55.163 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:55.164 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:55.164 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:55.164 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 Processor -- start 2024-09-18T10:15:55.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 -- start start 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: init 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9ac0a40d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9ac0a1be0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9ac138bf0 0x7fa9ac13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-c con 0x7fa9ac138bf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac06ac90 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac0a44a0 con 0x7fa9ac0a40d0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac0a20b0 con 0x7fa9ac138bf0 2024-09-18T10:15:55.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9b0b5e700 10 monclient(hunting): authenticate will time out at 1237.255852s 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9aad9d700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fa9ac138bf0 msgr2=0x7fa9ac13afd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9aa59c700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fa9ac0a40d0 legacy=0x7fa9ac06e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.156+0000 7fa9aad9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9ac138bf0 0x7fa9ac13afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a9d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa9ac0a1be0 0x7fa9ac1374d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:41214/0 (socket says 172.21.15.150:41214) 2024-09-18T10:15:55.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/3226437916 learned_addr learned my addr 172.21.15.150:0/3226437916 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/3226437916 >> v1:172.21.15.142:6789/0 conn(0x7fa9ac0a1be0 legacy=0x7fa9ac1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502232103 0 0) 0x7fa9ac06ac90 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): my global_id is 54753 2024-09-18T10:15:55.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa994003570 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1716744908 0 0) 0x7fa994003570 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 >> v2:172.21.15.150:3300/0 conn(0x7fa9ac138bf0 msgr2=0x7fa9ac13afd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 --2- 172.21.15.150:0/3226437916 >> v2:172.21.15.150:3300/0 conn(0x7fa9ac138bf0 0x7fa9ac13afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 >> v1:172.21.15.150:6789/0 conn(0x7fa9ac0a40d0 legacy=0x7fa9ac06e710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: _finish_hunting 0 2024-09-18T10:15:55.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac06b920 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: _finish_auth 0 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fa9a00017a0 con 0x7fa9ac0a1be0 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9b0b5e700 5 monclient: authenticate success, global_id 54753 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: dump: 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:55.175 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 >> v1:172.21.15.142:6789/0 conn(0x7fa9ac0a1be0 legacy=0x7fa9ac1374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:55.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa994003a60 0x7fa994003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): picked mon.c con 0x7fa994003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa994004810 0x7fa994004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): picked mon.b con 0x7fa994004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 0x7fa99400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): picked mon.a con 0x7fa9940053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0a20b0 con 0x7fa9940053c0 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99400c380 con 0x7fa994004810 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.160+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99400c540 con 0x7fa994003a60 2024-09-18T10:15:55.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9b0b5e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/3226437916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa994004810 msgr2=0x7fa994004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a9d9b700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa994004810 0x7fa994004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 0x7fa99400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): get_auth_request con 0x7fa9940053c0 auth_method 0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa994003a60 0x7fa994003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): get_auth_request con 0x7fa994003a60 auth_method 0 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:55.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient(hunting): handle_auth_done global_id 54753 payload 293 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 -- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa994003a60 msgr2=0x7fa994003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa994003a60 0x7fa994003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 -- 172.21.15.150:0/3226437916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa994004810 msgr2=0x7fa994004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa994004810 0x7fa994004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 monclient: found mon.a 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 -- 172.21.15.150:0/3226437916 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994003570 con 0x7fa9940053c0 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 10 monclient: _finish_auth 0 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 -- 172.21.15.150:0/3226437916 reap_dead start 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 -- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 msgr2=0x7fa99400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9aad9d700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 0x7fa99400baa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4003d50 tx=0x7fa9a40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7033f96c87aa5a33 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:55.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4009920 con 0x7fa9940053c0 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: dump: 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:55.181 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9b0b5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4015470 con 0x7fa9940053c0 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9b0b5e700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9b0b5e700 4 set_mon_vals no callback set 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 1 -- 172.21.15.150:0/3226437916 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4015810 con 0x7fa9940053c0 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.164+0000 7fa9a959a700 10 monclient: dump: 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.182 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: shutdown 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 msgr2=0x7fa99400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 0x7fa99400baa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4003d50 tx=0x7fa9a40095f0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/3226437916 shutdown_connections 2024-09-18T10:15:55.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/3226437916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9940053c0 0x7fa99400baa0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/3226437916 >> 172.21.15.150:0/3226437916 conn(0x7fa9ac0716e0 msgr2=0x7fa9ac0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/3226437916 shutdown_connections 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/3226437916 wait complete. 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: build_initial_monmap 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: monmap: 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.184 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:55.185 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:55.185 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:55.185 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 Processor -- start 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- start start 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: init 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 0x7fa9ac141430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-c con 0x7fa9940053c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-a con 0x7fa9ac141970 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 0x7fa9ac09c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): picked mon.noname-b con 0x7fa9ac137d60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac06f110 con 0x7fa9ac141970 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac134d20 con 0x7fa9ac137d60 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac138630 con 0x7fa9940053c0 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9b0b5e700 10 monclient(hunting): authenticate will time out at 1237.267316s 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aad9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 msgr2=0x7fa9ac09c1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:55.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 0x7fa9ac09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa99a7fc700 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 0x7fa9ac141430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 0x7fa9ac141430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60254/0 (socket says 172.21.15.150:60254) 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 1 -- 172.21.15.150:0/35667509 learned_addr learned my addr 172.21.15.150:0/35667509 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): get_auth_request con 0x7fa9ac141970 auth_method 0 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): get_auth_request con 0x7fa9940053c0 auth_method 0 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:55.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9a9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.168+0000 7fa9aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 10 monclient(hunting): handle_auth_done global_id 54756 payload 824 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 msgr2=0x7fa9ac141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 0x7fa9ac141430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 msgr2=0x7fa9ac09c1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 0x7fa9ac09c1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:55.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 monclient: found mon.noname-a 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac142310 con 0x7fa9ac141970 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 -- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 msgr2=0x7fa9ac137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0001ed0 tx=0x7fa9a00018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0007570 con 0x7fa9ac141970 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a9d9b700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0001ed0 tx=0x7fa9a00018e0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fa9a00076d0 config(0 keys) v1 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 10 monclient: dump: 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:55.193 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 5 monclient: authenticate success, global_id 54756 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a00076d0 con 0x7fa9ac141970 2024-09-18T10:15:55.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 10 monclient: _renew_subs 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac134a40 con 0x7fa9ac141970 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa99a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9a8d99700 4 set_mon_vals no callback set 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 10 monclient: _renew_subs 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.172+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac133c80 con 0x7fa9ac141970 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9b0b5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9b0b5e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9b0b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac06a9b0 con 0x7fa9ac141970 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0004a60 con 0x7fa9ac141970 2024-09-18T10:15:55.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 10 monclient: dump: 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:55.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa9a00133c0 con 0x7fa9ac141970 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa99a7fc700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 0x7fa98c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9aa59c700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 0x7fa98c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.176+0000 7fa9aa59c700 10 monclient: get_auth_request con 0x7fa98c035a70 auth_method 0 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.180+0000 7fa9aa59c700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 0x7fa98c037f20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa99c005470 tx=0x7fa99c0090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:55.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:55.188+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(194..194 src has 167..194) v4 ==== 9061+0+0 (secure 0 0 0) 0x7fa9a00490c0 con 0x7fa9ac141970 2024-09-18T10:15:56.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.004+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa9a00108c0 con 0x7fa9ac141970 2024-09-18T10:15:56.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.004+0000 7fa99a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:56.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.004+0000 7fa99a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:56.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.172+0000 7fa9b0b5e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:56.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.172+0000 7fa9b0b5e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:56.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.172+0000 7fa9b0b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:56.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.172+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa9ac133c80 con 0x7fa9ac141970 2024-09-18T10:15:56.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.940+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 added bucket h6 type host to location {rack=r1} v195) v1 ==== 166+0+0 (secure 0 0 0) 0x7fa9ac133c80 con 0x7fa9ac141970 2024-09-18T10:15:56.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.940+0000 7fa99a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:56.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:56.940+0000 7fa99a7fc700 10 monclient: _finish_command 2 = system:0 added bucket h6 type host to location {rack=r1} 2024-09-18T10:15:57.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa9b0b5e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa9b0b5e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa9b0b5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:57.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa9ac133c80 con 0x7fa9ac141970 2024-09-18T10:15:57.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa99a7fc700 1 -- 172.21.15.150:0/35667509 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h6' already exists v195) v1 ==== 145+0+0 (secure 0 0 0) 0x7fa9ac133c80 con 0x7fa9ac141970 2024-09-18T10:15:57.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa99a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h6", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.048+0000 7fa99a7fc700 10 monclient: _finish_command 3 = system:0 bucket 'h6' already exists 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 msgr2=0x7fa98c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 0x7fa98c037f20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa99c005470 tx=0x7fa99c0090f0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 10 monclient: shutdown 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 msgr2=0x7fa9ac137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0001ed0 tx=0x7fa9a00018e0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 shutdown_connections 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa98c035a70 0x7fa98c037f20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9ac137d60 0x7fa9ac09c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9ac141970 0x7fa9ac137820 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 --2- 172.21.15.150:0/35667509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9940053c0 0x7fa9ac141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 >> 172.21.15.150:0/35667509 conn(0x7fa9ac0716e0 msgr2=0x7fa9ac06d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:57.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 shutdown_connections 2024-09-18T10:15:57.060 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.052+0000 7fa9b0b5e700 1 -- 172.21.15.150:0/35667509 wait complete. 2024-09-18T10:15:57.060 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h6' already exists 2024-09-18T10:15:57.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:57.069 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h7 host rack=r1 2024-09-18T10:15:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.164+0000 7ff694a9b700 10 monclient: get_monmap_and_config 2024-09-18T10:15:57.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.164+0000 7ff694a9b700 10 monclient: build_initial_monmap 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.164+0000 7ff694a9b700 10 monclient: monmap: 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:57.171584+0000 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:57.171584+0000 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:57.173 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:57.174 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:57.174 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 Processor -- start 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 -- start start 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient: init 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6900a40d0 0x7ff69006e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-c con 0x7ff6900a40d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6900a1be0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff69006ec50 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff69006ac90 con 0x7ff6900a1be0 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6900a44a0 con 0x7ff69006ec50 2024-09-18T10:15:57.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff690069cf0 con 0x7ff6900a40d0 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff694a9b700 10 monclient(hunting): authenticate will time out at 1239.265438s 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff68ed9d700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7ff69006ec50 legacy=0x7ff690132870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff68e59c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7ff6900a40d0 msgr2=0x7ff69006e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff68e59c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6900a40d0 0x7ff69006e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff687fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff6900a1be0 0x7ff690132160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:41226/0 (socket says 172.21.15.150:41226) 2024-09-18T10:15:57.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.168+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 learned_addr learned my addr 172.21.15.150:0/1872806384 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 >> v1:172.21.15.142:6789/0 conn(0x7ff6900a1be0 legacy=0x7ff690132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603370338 0 0) 0x7ff69006ac90 con 0x7ff6900a1be0 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): my global_id is 54759 2024-09-18T10:15:57.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff670003570 con 0x7ff6900a1be0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464251487 0 0) 0x7ff670003570 con 0x7ff6900a1be0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 >> v2:172.21.15.150:3300/0 conn(0x7ff6900a40d0 msgr2=0x7ff69006e710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 --2- 172.21.15.150:0/1872806384 >> v2:172.21.15.150:3300/0 conn(0x7ff6900a40d0 0x7ff69006e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 >> v1:172.21.15.150:6789/0 conn(0x7ff69006ec50 legacy=0x7ff690132870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690132f80 con 0x7ff6900a1be0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff694a9b700 5 monclient: authenticate success, global_id 54759 2024-09-18T10:15:57.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff694a9b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff678001790 con 0x7ff6900a1be0 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: dump: 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:57.183 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 >> v1:172.21.15.142:6789/0 conn(0x7ff6900a1be0 legacy=0x7ff690132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670003b30 0x7ff670003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): picked mon.b con 0x7ff670003b30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 0x7ff670004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): picked mon.a con 0x7ff670004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68e59c700 1 -- 172.21.15.150:0/1872806384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670003b30 msgr2=0x7ff670003f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68e59c700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670003b30 0x7ff670003f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:57.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670005410 0x7ff67000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): picked mon.c con 0x7ff670005410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67000c370 con 0x7ff670004810 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67000c4e0 con 0x7ff670003b30 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67000c6a0 con 0x7ff670005410 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68dd9b700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff694a9b700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 0x7ff670004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): get_auth_request con 0x7ff670004810 auth_method 0 2024-09-18T10:15:57.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670005410 0x7ff67000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): get_auth_request con 0x7ff670005410 auth_method 0 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:57.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient(hunting): handle_auth_done global_id 54759 payload 293 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670005410 msgr2=0x7ff67000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff670005410 0x7ff67000bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670003b30 msgr2=0x7ff670003f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670003b30 0x7ff670003f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient: _finish_hunting 0 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 monclient: found mon.a 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1872806384 reap_dead start 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670003570 con 0x7ff670004810 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 10 monclient: _finish_auth 0 2024-09-18T10:15:57.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.172+0000 7ff687fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:57.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff687fff700 1 -- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 msgr2=0x7ff670004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff687fff700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 0x7ff670004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff678001970 tx=0x7ff678007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d5d8a935b2d5a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678007e50 con 0x7ff670004810 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: dump: 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.194 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67800cdf0 con 0x7ff670004810 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 4 set_mon_vals no callback set 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 1 -- 172.21.15.150:0/1872806384 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678004b40 con 0x7ff670004810 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff68dd9b700 10 monclient: dump: 2024-09-18T10:15:57.195 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 10 monclient: shutdown 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:57.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 msgr2=0x7ff670004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 0x7ff670004c80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff678001970 tx=0x7ff678007b20 comp rx=0 tx=0).stop 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- 172.21.15.150:0/1872806384 shutdown_connections 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1872806384 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff670004810 0x7ff670004c80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- 172.21.15.150:0/1872806384 >> 172.21.15.150:0/1872806384 conn(0x7ff6900716e0 msgr2=0x7ff6900a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- 172.21.15.150:0/1872806384 shutdown_connections 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- 172.21.15.150:0/1872806384 wait complete. 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 10 monclient: build_initial_monmap 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 10 monclient: monmap: 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:57.197 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 Processor -- start 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.176+0000 7ff694a9b700 1 -- start start 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient: init 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 0x7ff690141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-b con 0x7ff670004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-a con 0x7ff6901321c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): picked mon.noname-c con 0x7ff69011ed40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff690132ad0 con 0x7ff6901321c0 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6901225f0 con 0x7ff670004810 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69006c9c0 con 0x7ff69011ed40 2024-09-18T10:15:57.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff694a9b700 10 monclient(hunting): authenticate will time out at 1239.276869s 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68e59c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 msgr2=0x7ff690141350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 0x7ff690141350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff6867fc700 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60272/0 (socket says 172.21.15.150:60272) 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57788/0 (socket says 172.21.15.150:57788) 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1840726966 learned_addr learned my addr 172.21.15.150:0/1840726966 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): get_auth_request con 0x7ff69011ed40 auth_method 0 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:57.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): get_auth_request con 0x7ff6901321c0 auth_method 0 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:57.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient(hunting): handle_auth_done global_id 74219 payload 824 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 msgr2=0x7ff690141350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 0x7ff690141350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 msgr2=0x7ff690132590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6901421c0 con 0x7ff69011ed40 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.180+0000 7ff687fff700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:57.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff68ed9d700 1 -- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 msgr2=0x7ff69011f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff68ed9d700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff69006f070 tx=0x7ff6880095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688011070 con 0x7ff69011ed40 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: dump: 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:57.203 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: _finish_auth 0 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68800be20 con 0x7ff69011ed40 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 5 monclient: authenticate success, global_id 74219 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688005530 con 0x7ff69011ed40 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _renew_subs 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff68d59a700 4 set_mon_vals no callback set 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff690122310 con 0x7ff69011ed40 2024-09-18T10:15:57.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 10 monclient: dump: 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:57.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _renew_subs 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff690141a30 con 0x7ff69011ed40 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff688005960 con 0x7ff69011ed40 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69011b6d0 con 0x7ff69011ed40 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 0x7ff674037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(195..195 src has 167..195) v4 ==== 9271+0+0 (secure 0 0 0) 0x7ff688049a40 con 0x7ff69011ed40 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff68e59c700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 0x7ff674037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:57.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.184+0000 7ff68e59c700 10 monclient: get_auth_request con 0x7ff674035a70 auth_method 0 2024-09-18T10:15:57.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.188+0000 7ff68e59c700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 0x7ff674037f20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff680006fd0 tx=0x7ff680006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:57.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.196+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff6880153c0 con 0x7ff69011ed40 2024-09-18T10:15:57.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.196+0000 7ff6867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:57.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.196+0000 7ff6867fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.364+0000 7ff694a9b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.364+0000 7ff694a9b700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.364+0000 7ff694a9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.364+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff69006b920 con 0x7ff69011ed40 2024-09-18T10:15:57.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.948+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 added bucket h7 type host to location {rack=r1} v196) v1 ==== 166+0+0 (secure 0 0 0) 0x7ff688005e60 con 0x7ff69011ed40 2024-09-18T10:15:57.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.948+0000 7ff6867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:57.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:57.948+0000 7ff6867fc700 10 monclient: _finish_command 2 = system:0 added bucket h7 type host to location {rack=r1} 2024-09-18T10:15:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff694a9b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff694a9b700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff694a9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:58.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7ff690063ed0 con 0x7ff69011ed40 2024-09-18T10:15:58.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff6867fc700 1 -- 172.21.15.150:0/1840726966 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h7' already exists v196) v1 ==== 145+0+0 (secure 0 0 0) 0x7ff68801d340 con 0x7ff69011ed40 2024-09-18T10:15:58.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff6867fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h7", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.063 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.056+0000 7ff6867fc700 10 monclient: _finish_command 3 = system:0 bucket 'h7' already exists 2024-09-18T10:15:58.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 msgr2=0x7ff674037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 0x7ff674037f20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff680006fd0 tx=0x7ff680006e10 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 10 monclient: shutdown 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 msgr2=0x7ff69011f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff69006f070 tx=0x7ff6880095f0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 shutdown_connections 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff674035a70 0x7ff674037f20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff69011ed40 0x7ff69011f1c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff6901321c0 0x7ff690132590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 --2- 172.21.15.150:0/1840726966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff670004810 0x7ff690141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 >> 172.21.15.150:0/1840726966 conn(0x7ff6900716e0 msgr2=0x7ff69009ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:58.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 shutdown_connections 2024-09-18T10:15:58.069 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.060+0000 7ff694a9b700 1 -- 172.21.15.150:0/1840726966 wait complete. 2024-09-18T10:15:58.069 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h7' already exists 2024-09-18T10:15:58.082 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:58.082 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h8 host rack=r1 2024-09-18T10:15:58.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: get_monmap_and_config 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: build_initial_monmap 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: monmap: 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:58.184914+0000 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:58.184914+0000 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:58.186 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:58.187 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:58.187 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:58.187 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:58.187 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 Processor -- start 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 -- start start 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: init 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4c809c750 0x7fd4c80972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-a con 0x7fd4c809c750 addr v2:172.21.15.142:3300/0 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-b con 0x7fd4c809cb20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4c8132560 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:58.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c809e680 con 0x7fd4c8132560 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c80a2610 con 0x7fd4c809c750 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c80a28b0 con 0x7fd4c809cb20 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4cda9c700 10 monclient(hunting): authenticate will time out at 1240.278866s 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c7fff700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fd4c8132560 legacy=0x7fd4c8132980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:60278/0 (socket says 172.21.15.150:60278) 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 -- 172.21.15.150:0/3744117900 learned_addr learned my addr 172.21.15.150:0/3744117900 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): get_auth_request con 0x7fd4c809cb20 auth_method 0 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:58.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 1 --2- 172.21.15.150:0/3744117900 >> v2:172.21.15.142:3300/0 conn(0x7fd4c809c750 0x7fd4c80972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): get_auth_request con 0x7fd4c809c750 auth_method 0 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient(hunting): handle_auth_done global_id 74222 payload 274 2024-09-18T10:15:58.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 -- 172.21.15.150:0/3744117900 >> v2:172.21.15.142:3300/0 conn(0x7fd4c809c750 msgr2=0x7fd4c80972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 --2- 172.21.15.150:0/3744117900 >> v2:172.21.15.142:3300/0 conn(0x7fd4c809c750 0x7fd4c80972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 -- 172.21.15.150:0/3744117900 >> v1:172.21.15.150:6789/0 conn(0x7fd4c8132560 legacy=0x7fd4c8132980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient: _finish_hunting 0 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 monclient: found mon.noname-b 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:58.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.180+0000 7fd4beffd700 1 -- 172.21.15.150:0/3744117900 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c809e3a0 con 0x7fd4c809cb20 2024-09-18T10:15:58.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4beffd700 1 -- 172.21.15.150:0/3744117900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 msgr2=0x7fd4c809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4beffd700 1 --2- 172.21.15.150:0/3744117900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00099b0 tx=0x7fd4b0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27ec84531215653e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 1 -- 172.21.15.150:0/3744117900 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0009ea0 con 0x7fd4c809cb20 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: dump: 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:58.201 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 5 monclient: authenticate success, global_id 74222 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 1 -- 172.21.15.150:0/3744117900 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b000ba70 con 0x7fd4c809cb20 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 4 set_mon_vals no callback set 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 1 -- 172.21.15.150:0/3744117900 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b000be30 con 0x7fd4c809cb20 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:58.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4c6ffd700 10 monclient: dump: 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:58.203 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 10 monclient: shutdown 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 -- 172.21.15.150:0/3744117900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 msgr2=0x7fd4c809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/3744117900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00099b0 tx=0x7fd4b0009250 comp rx=0 tx=0).stop 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 -- 172.21.15.150:0/3744117900 shutdown_connections 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/3744117900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c809cb20 0x7fd4c809cef0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/3744117900 >> v2:172.21.15.142:3300/0 conn(0x7fd4c809c750 0x7fd4c80972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 -- 172.21.15.150:0/3744117900 >> 172.21.15.150:0/3744117900 conn(0x7fd4c80681c0 msgr2=0x7fd4c80a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 -- 172.21.15.150:0/3744117900 shutdown_connections 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.184+0000 7fd4cda9c700 1 -- 172.21.15.150:0/3744117900 wait complete. 2024-09-18T10:15:58.204 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient: build_initial_monmap 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient: monmap: 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 Processor -- start 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 -- start start 2024-09-18T10:15:58.205 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient: init 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-a con 0x7fd4c809c750 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 0x7fd4c8131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-b con 0x7fd4c809cb20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c8132560 0x7fd4c8131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): picked mon.noname-c con 0x7fd4c8132560 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c809ded0 con 0x7fd4c809c750 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c809bc70 con 0x7fd4c809cb20 2024-09-18T10:15:58.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8120980 con 0x7fd4c8132560 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4cda9c700 10 monclient(hunting): authenticate will time out at 1240.286178s 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4beffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 msgr2=0x7fd4c8131320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 0x7fd4c8131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:57802/0 (socket says 172.21.15.150:57802) 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 -- 172.21.15.150:0/4029654164 learned_addr learned my addr 172.21.15.150:0/4029654164 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): get_auth_request con 0x7fd4c809c750 auth_method 0 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:58.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c8132560 0x7fd4c8131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): get_auth_request con 0x7fd4c8132560 auth_method 0 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:58.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient(hunting): handle_auth_done global_id 54768 payload 824 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 -- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c8132560 msgr2=0x7fd4c8131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c8132560 0x7fd4c8131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 -- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 msgr2=0x7fd4c8131320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 0x7fd4c8131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 monclient: found mon.noname-a 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.188+0000 7fd4c77fe700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b00094f0 con 0x7fd4c809c750 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4c77fe700 1 -- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 msgr2=0x7fd4c8123730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4c77fe700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800ec80 tx=0x7fd4b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:58.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b800ca60 con 0x7fd4c809c750 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: dump: 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:58.210 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: _finish_auth 0 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 5 monclient: authenticate success, global_id 54768 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _renew_subs 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b800cbc0 con 0x7fd4c809c750 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c81212d0 con 0x7fd4c809c750 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8013470 con 0x7fd4c809c750 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4c67fc700 4 set_mon_vals no callback set 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _renew_subs 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:58.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c811ee40 con 0x7fd4c809c750 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4bffff700 10 monclient: dump: 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:58.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:58.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:58.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:58.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.192+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c8122920 con 0x7fd4c809c750 2024-09-18T10:15:58.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd4b80137e0 con 0x7fd4c809c750 2024-09-18T10:15:58.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4bffff700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 0x7fd4ac037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:58.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(196..196 src has 167..196) v4 ==== 9481+0+0 (secure 0 0 0) 0x7fd4b8049930 con 0x7fd4c809c750 2024-09-18T10:15:58.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4beffd700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 0x7fd4ac037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:58.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4beffd700 10 monclient: get_auth_request con 0x7fd4ac035a70 auth_method 0 2024-09-18T10:15:58.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.196+0000 7fd4beffd700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 0x7fd4ac037f20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00099b0 tx=0x7fd4b0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:58.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.208+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd4b8023ad0 con 0x7fd4c809c750 2024-09-18T10:15:58.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.208+0000 7fd4bffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:58.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.208+0000 7fd4bffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:58.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.376+0000 7fd4cda9c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.376+0000 7fd4cda9c700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.376+0000 7fd4cda9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:58.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.376+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd4c8120240 con 0x7fd4c809c750 2024-09-18T10:15:58.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.956+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 added bucket h8 type host to location {rack=r1} v197) v1 ==== 166+0+0 (secure 0 0 0) 0x7fd4b80232f0 con 0x7fd4c809c750 2024-09-18T10:15:58.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.956+0000 7fd4bffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:58.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:58.956+0000 7fd4bffff700 10 monclient: _finish_command 2 = system:0 added bucket h8 type host to location {rack=r1} 2024-09-18T10:15:59.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.064+0000 7fd4cda9c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:59.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.064+0000 7fd4cda9c700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:59.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.064+0000 7fd4cda9c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:15:59.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.064+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd4c8132340 con 0x7fd4c809c750 2024-09-18T10:15:59.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4bffff700 1 -- 172.21.15.150:0/4029654164 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h8' already exists v197) v1 ==== 145+0+0 (secure 0 0 0) 0x7fd4b80234d0 con 0x7fd4c809c750 2024-09-18T10:15:59.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4bffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h8", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:59.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4bffff700 10 monclient: _finish_command 3 = system:0 bucket 'h8' already exists 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 msgr2=0x7fd4ac037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 0x7fd4ac037f20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00099b0 tx=0x7fd4b0016000 comp rx=0 tx=0).stop 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 10 monclient: shutdown 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 msgr2=0x7fd4c8123730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:59.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800ec80 tx=0x7fd4b80091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 shutdown_connections 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4ac035a70 0x7fd4ac037f20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4c8132560 0x7fd4c8131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4c809cb20 0x7fd4c8131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 --2- 172.21.15.150:0/4029654164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4c809c750 0x7fd4c8123730 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 >> 172.21.15.150:0/4029654164 conn(0x7fd4c80681c0 msgr2=0x7fd4c809a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:59.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 shutdown_connections 2024-09-18T10:15:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.068+0000 7fd4cda9c700 1 -- 172.21.15.150:0/4029654164 wait complete. 2024-09-18T10:15:59.077 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h8' already exists 2024-09-18T10:15:59.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:15:59.086 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h9 host rack=r1 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: get_monmap_and_config 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: build_initial_monmap 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: monmap: 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:15:59.188220+0000 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:15:59.188220+0000 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:59.189 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:15:59.190 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 Processor -- start 2024-09-18T10:15:59.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 -- start start 2024-09-18T10:15:59.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: init 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08c409acd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c409c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-b con 0x7f08c4096b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08c4135780 addr v1:172.21.15.150:6789/0 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c4096f10 con 0x7f08c409acd0 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08c4095e70 con 0x7f08c4135780 2024-09-18T10:15:59.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c4096250 con 0x7f08c4096b40 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08cbdf0700 10 monclient(hunting): authenticate will time out at 1241.281387s 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08ca38d700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f08c4135780 legacy=0x7f08c4137b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c9b8c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f08c409acd0 0x7f08c409bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:53900/0 (socket says 172.21.15.150:53900) 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c9b8c700 1 -- 172.21.15.150:0/701281705 learned_addr learned my addr 172.21.15.150:0/701281705 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 1 --2- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c409c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): get_auth_request con 0x7f08c4096b40 auth_method 0 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:59.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient(hunting): handle_auth_done global_id 74228 payload 274 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 1 -- 172.21.15.150:0/701281705 >> v1:172.21.15.150:6789/0 conn(0x7f08c4135780 legacy=0x7f08c4137b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 1 -- 172.21.15.150:0/701281705 >> v1:172.21.15.142:6789/0 conn(0x7f08c409acd0 legacy=0x7f08c409bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient: _finish_hunting 0 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 1 monclient: found mon.noname-b 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.184+0000 7f08c938b700 1 -- 172.21.15.150:0/701281705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c4095920 con 0x7f08c4096b40 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c938b700 1 -- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 msgr2=0x7f08c409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c938b700 1 --2- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c409c6e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009910 tx=0x7f08b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c28852de014070 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 1 -- 172.21.15.150:0/701281705 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b800ba70 con 0x7f08c4096b40 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: dump: 2024-09-18T10:15:59.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: _finish_auth 0 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 1 -- 172.21.15.150:0/701281705 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b800bbd0 con 0x7f08c4096b40 2024-09-18T10:15:59.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 1 -- 172.21.15.150:0/701281705 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b8011420 con 0x7f08c4096b40 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08c8b8a700 10 monclient: dump: 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:59.199 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 5 monclient: authenticate success, global_id 74228 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 10 monclient: get_monmap_and_config success 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 4 set_mon_vals no callback set 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 10 monclient: shutdown 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 -- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 msgr2=0x7f08c409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c409c6e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08b8009910 tx=0x7f08b80091a0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 -- 172.21.15.150:0/701281705 shutdown_connections 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/701281705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c409c6e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 -- 172.21.15.150:0/701281705 >> 172.21.15.150:0/701281705 conn(0x7f08c40a15e0 msgr2=0x7f08c40a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 -- 172.21.15.150:0/701281705 shutdown_connections 2024-09-18T10:15:59.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.188+0000 7f08cbdf0700 1 -- 172.21.15.150:0/701281705 wait complete. 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient: build_initial_monmap 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient: monmap: 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:15:59.201 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 Processor -- start 2024-09-18T10:15:59.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 -- start start 2024-09-18T10:15:59.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient: init 2024-09-18T10:15:59.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient: _reopen_session rank -1 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-c con 0x7f08c4096b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 0x7f08c4094230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-b con 0x7f08c409acd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): picked mon.noname-a con 0x7f08c4135780 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c408dea0 con 0x7f08c4135780 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:59.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c408d9f0 con 0x7f08c409acd0 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08c406eb60 con 0x7f08c4096b40 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): _renew_subs 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08cbdf0700 10 monclient(hunting): authenticate will time out at 1241.290261s 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c938b700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 msgr2=0x7f08c4094230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c938b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 0x7f08c4094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56662/0 (socket says 172.21.15.150:56662) 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 1 -- 172.21.15.150:0/835781913 learned_addr learned my addr 172.21.15.150:0/835781913 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): get_auth_request con 0x7f08c4135780 auth_method 0 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:59.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): get_auth_request con 0x7f08c4096b40 auth_method 0 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:15:59.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient(hunting): handle_auth_done global_id 74231 payload 824 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 -- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 msgr2=0x7f08c4094230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 0x7f08c4094230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 -- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 msgr2=0x7f08c408bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient: _finish_hunting 0 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 monclient: found mon.noname-c 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08c9b8c700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b8009420 con 0x7f08c4096b40 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.192+0000 7f08ca38d700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c9b8c700 1 -- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 msgr2=0x7f08c4134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c9b8c700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c4134a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c590 tx=0x7f08bc007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:59.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc005d40 con 0x7f08c4096b40 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: dump: 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:59.211 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08bc0070d0 con 0x7f08c4096b40 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 5 monclient: authenticate success, global_id 74231 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc007490 con 0x7f08c4096b40 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c3fff700 4 set_mon_vals no callback set 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 10 monclient: dump: 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:15:59.212 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _renew_subs 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c408f710 con 0x7f08c4096b40 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _renew_subs 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c4134f80 con 0x7f08c4096b40 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f08bc01b3b0 con 0x7f08c4096b40 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.196+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4095b90 con 0x7f08c4096b40 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.200+0000 7f08c1ffb700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 0x7f08b0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.200+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(197..197 src has 167..197) v4 ==== 9691+0+0 (secure 0 0 0) 0x7f08bc048b60 con 0x7f08c4096b40 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.200+0000 7f08c938b700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 0x7f08b0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:15:59.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.200+0000 7f08c938b700 10 monclient: get_auth_request con 0x7f08b0035a70 auth_method 0 2024-09-18T10:15:59.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.200+0000 7f08c938b700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 0x7f08b0037f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f08b800a010 tx=0x7f08b8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:15:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.212+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f08bc0198a0 con 0x7f08c4096b40 2024-09-18T10:15:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.212+0000 7f08c1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:15:59.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.212+0000 7f08c1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:15:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.380+0000 7f08cbdf0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.380+0000 7f08cbdf0700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:15:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.380+0000 7f08cbdf0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:15:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:15:59.380+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f08c409f7d0 con 0x7f08c4096b40 2024-09-18T10:16:00.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.692+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 added bucket h9 type host to location {rack=r1} v198) v1 ==== 166+0+0 (secure 0 0 0) 0x7f08bc017030 con 0x7f08c4096b40 2024-09-18T10:16:00.698 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.692+0000 7f08c1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:00.699 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.692+0000 7f08c1ffb700 10 monclient: _finish_command 2 = system:0 added bucket h9 type host to location {rack=r1} 2024-09-18T10:16:00.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08cbdf0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:00.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08cbdf0700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:00.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08cbdf0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.805 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f08c4063ed0 con 0x7f08c4096b40 2024-09-18T10:16:00.807 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08c1ffb700 1 -- 172.21.15.150:0/835781913 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h9' already exists v198) v1 ==== 145+0+0 (secure 0 0 0) 0x7f08bc017020 con 0x7f08c4096b40 2024-09-18T10:16:00.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08c1ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h9", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:00.808 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.800+0000 7f08c1ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h9' already exists 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 msgr2=0x7f08b0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 0x7f08b0037f20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f08b800a010 tx=0x7f08b8016000 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 10 monclient: shutdown 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 msgr2=0x7f08c4134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c4134a40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00c590 tx=0x7f08bc007f90 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 shutdown_connections 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f08b0035a70 0x7f08b0037f20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08c4135780 0x7f08c408bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08c409acd0 0x7f08c4094230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 --2- 172.21.15.150:0/835781913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08c4096b40 0x7f08c4134a40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.811 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 >> 172.21.15.150:0/835781913 conn(0x7f08c40a15e0 msgr2=0x7f08c406c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:00.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 shutdown_connections 2024-09-18T10:16:00.812 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.804+0000 7f08cbdf0700 1 -- 172.21.15.150:0/835781913 wait complete. 2024-09-18T10:16:00.812 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h9' already exists 2024-09-18T10:16:00.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:00.820 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h10 host rack=r1 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: get_monmap_and_config 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: build_initial_monmap 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: monmap: 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:00.924068+0000 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:00.924068+0000 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:00.926 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:00.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:00.926 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:00.926 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:00.926 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 Processor -- start 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 -- start start 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: init 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-b con 0x7fdad00a40d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:00.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 0x7fdad0132160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-a con 0x7fdad00a1be0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdad006ec50 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad006ac90 con 0x7fdad006ec50 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad0069cd0 con 0x7fdad00a1be0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad00a44a0 con 0x7fdad00a40d0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad815b700 10 monclient(hunting): authenticate will time out at 1243.017664s 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad66f8700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fdad006ec50 legacy=0x7fdad01326a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 0x7fdad0132160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:00.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41414/0 (socket says 172.21.15.150:41414) 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 -- 172.21.15.150:0/3780280008 learned_addr learned my addr 172.21.15.150:0/3780280008 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): get_auth_request con 0x7fdad00a40d0 auth_method 0 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): get_auth_request con 0x7fdad00a1be0 auth_method 0 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:00.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient(hunting): handle_auth_done global_id 74234 payload 274 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 -- 172.21.15.150:0/3780280008 >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 msgr2=0x7fdad0132160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 --2- 172.21.15.150:0/3780280008 >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 0x7fdad0132160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 -- 172.21.15.150:0/3780280008 >> v1:172.21.15.150:6789/0 conn(0x7fdad006ec50 legacy=0x7fdad01326a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient: _finish_hunting 0 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 monclient: found mon.noname-b 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad5ef7700 1 -- 172.21.15.150:0/3780280008 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad00a20b0 con 0x7fdad00a40d0 2024-09-18T10:16:00.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.920+0000 7fdad56f6700 1 --2- 172.21.15.150:0/3780280008 >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 0x7fdad0132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:00.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad5ef7700 1 -- 172.21.15.150:0/3780280008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 msgr2=0x7fdad006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:00.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad5ef7700 1 --2- 172.21.15.150:0/3780280008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdad0069e10 tx=0x7fdac0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7750609b9ead4214 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 1 -- 172.21.15.150:0/3780280008 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac0009e70 con 0x7fdad00a40d0 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: dump: 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:00.935 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: _finish_auth 0 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 5 monclient: authenticate success, global_id 74234 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 1 -- 172.21.15.150:0/3780280008 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac000ba70 con 0x7fdad00a40d0 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 4 set_mon_vals no callback set 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 1 -- 172.21.15.150:0/3780280008 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac000be30 con 0x7fdad00a40d0 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad4ef5700 10 monclient: dump: 2024-09-18T10:16:00.936 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 10 monclient: shutdown 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- 172.21.15.150:0/3780280008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 msgr2=0x7fdad006e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:00.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 --2- 172.21.15.150:0/3780280008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdad0069e10 tx=0x7fdac0009250 comp rx=0 tx=0).stop 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- 172.21.15.150:0/3780280008 shutdown_connections 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 --2- 172.21.15.150:0/3780280008 >> v2:172.21.15.142:3300/0 conn(0x7fdad00a1be0 0x7fdad0132160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 --2- 172.21.15.150:0/3780280008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a40d0 0x7fdad006e710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- 172.21.15.150:0/3780280008 >> 172.21.15.150:0/3780280008 conn(0x7fdad00716e0 msgr2=0x7fdad00a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- 172.21.15.150:0/3780280008 shutdown_connections 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- 172.21.15.150:0/3780280008 wait complete. 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 10 monclient: build_initial_monmap 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 10 monclient: monmap: 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:00.938 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 Processor -- start 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 1 -- start start 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.924+0000 7fdad815b700 10 monclient: init 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:00.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 0x7fdad0132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-b con 0x7fdad006ec50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-c con 0x7fdad00a1be0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad00a40d0 0x7fdad011c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): picked mon.noname-a con 0x7fdad00a40d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad011f990 con 0x7fdad00a40d0 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad011f620 con 0x7fdad006ec50 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad01425a0 con 0x7fdad00a1be0 2024-09-18T10:16:00.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad815b700 10 monclient(hunting): authenticate will time out at 1243.024759s 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad5ef7700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 msgr2=0x7fdad0132220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad5ef7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 0x7fdad0132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad00a40d0 0x7fdad011c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41428/0 (socket says 172.21.15.150:41428) 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 -- 172.21.15.150:0/1833539322 learned_addr learned my addr 172.21.15.150:0/1833539322 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): get_auth_request con 0x7fdad00a1be0 auth_method 0 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:00.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): get_auth_request con 0x7fdad00a40d0 auth_method 0 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad66f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient(hunting): handle_auth_done global_id 74237 payload 824 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 -- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 msgr2=0x7fdad0132220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:00.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 0x7fdad0132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 -- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad00a40d0 msgr2=0x7fdad011c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad00a40d0 0x7fdad011c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient: _finish_hunting 0 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 monclient: found mon.noname-c 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.928+0000 7fdad56f6700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc009a20 con 0x7fdad00a1be0 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad56f6700 1 -- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 msgr2=0x7fdad011be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad56f6700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c0f0 tx=0x7fdacc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc014480 con 0x7fdad00a1be0 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: dump: 2024-09-18T10:16:00.945 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc00f040 con 0x7fdad00a1be0 2024-09-18T10:16:00.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 5 monclient: authenticate success, global_id 74237 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _renew_subs 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac00094f0 con 0x7fdad00a1be0 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _renew_subs 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac7fff700 4 set_mon_vals no callback set 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad011f340 con 0x7fdad00a1be0 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc014810 con 0x7fdad00a1be0 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 10 monclient: dump: 2024-09-18T10:16:00.947 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdacc014b80 con 0x7fdad00a1be0 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:00.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad011e500 con 0x7fdad00a1be0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 0x7fdabc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(198..198 src has 167..198) v4 ==== 9901+0+0 (secure 0 0 0) 0x7fdacc04a610 con 0x7fdad00a1be0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad5ef7700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 0x7fdabc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad5ef7700 10 monclient: get_auth_request con 0x7fdabc035a70 auth_method 0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.936+0000 7fdad5ef7700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 0x7fdabc037f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdad0069e10 tx=0x7fdac0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.948+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdacc024900 con 0x7fdad00a1be0 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.948+0000 7fdac5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:00.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:00.948+0000 7fdac5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:01.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.108+0000 7fdad815b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.108+0000 7fdad815b700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.108+0000 7fdad815b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:01.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.108+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdad0063ed0 con 0x7fdad00a1be0 2024-09-18T10:16:01.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.704+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 added bucket h10 type host to location {rack=r1} v199) v1 ==== 168+0+0 (secure 0 0 0) 0x7fdacc012340 con 0x7fdad00a1be0 2024-09-18T10:16:01.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.704+0000 7fdac5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.704+0000 7fdac5ffb700 10 monclient: _finish_command 2 = system:0 added bucket h10 type host to location {rack=r1} 2024-09-18T10:16:01.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.808+0000 7fdad815b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.808+0000 7fdad815b700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.808+0000 7fdad815b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:01.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.808+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fdad006a990 con 0x7fdad00a1be0 2024-09-18T10:16:01.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdac5ffb700 1 -- 172.21.15.150:0/1833539322 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h10' already exists v199) v1 ==== 147+0+0 (secure 0 0 0) 0x7fdacc024300 con 0x7fdad00a1be0 2024-09-18T10:16:01.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdac5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h10", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:01.818 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdac5ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h10' already exists 2024-09-18T10:16:01.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 msgr2=0x7fdabc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:01.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 0x7fdabc037f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdad0069e10 tx=0x7fdac0017000 comp rx=0 tx=0).stop 2024-09-18T10:16:01.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 10 monclient: shutdown 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 msgr2=0x7fdad011be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00c0f0 tx=0x7fdacc00b040 comp rx=0 tx=0).stop 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 shutdown_connections 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdabc035a70 0x7fdabc037f20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdad00a40d0 0x7fdad011c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdad00a1be0 0x7fdad011be10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 --2- 172.21.15.150:0/1833539322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdad006ec50 0x7fdad0132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 >> 172.21.15.150:0/1833539322 conn(0x7fdad00716e0 msgr2=0x7fdad006d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 shutdown_connections 2024-09-18T10:16:01.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.816+0000 7fdad815b700 1 -- 172.21.15.150:0/1833539322 wait complete. 2024-09-18T10:16:01.823 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h10' already exists 2024-09-18T10:16:01.832 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:01.832 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h11 host rack=r1 2024-09-18T10:16:01.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: get_monmap_and_config 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: build_initial_monmap 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: monmap: 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:01.929750+0000 2024-09-18T10:16:01.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:01.929750+0000 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:01.933 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 Processor -- start 2024-09-18T10:16:01.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 -- start start 2024-09-18T10:16:01.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: init 2024-09-18T10:16:01.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f985c09acd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f985c096b40 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f985c136790 0x7f985c138b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): picked mon.noname-c con 0x7f985c136790 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c095e70 con 0x7f985c09acd0 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985c095920 con 0x7f985c096b40 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985c096250 con 0x7f985c136790 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 10 monclient(hunting): authenticate will time out at 1244.023064s 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9860821700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f985c096b40 legacy=0x7f985c09c6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:16:01.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f985c136790 msgr2=0x7f985c138b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f985c136790 0x7f985c138b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f985c09acd0 0x7f985c09bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:53906/0 (socket says 172.21.15.150:53906) 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 1 -- 172.21.15.150:0/2696118265 learned_addr learned my addr 172.21.15.150:0/2696118265 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 1 -- 172.21.15.150:0/2696118265 >> v1:172.21.15.142:6789/0 conn(0x7f985c09acd0 legacy=0x7f985c09bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633228635 0 0) 0x7f985c095e70 con 0x7f985c09acd0 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): my global_id is 54780 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9844003570 con 0x7f985c09acd0 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4252191910 0 0) 0x7f9844003570 con 0x7f985c09acd0 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 >> v2:172.21.15.150:3300/0 conn(0x7f985c136790 msgr2=0x7f985c138b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:01.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 --2- 172.21.15.150:0/2696118265 >> v2:172.21.15.150:3300/0 conn(0x7f985c136790 0x7f985c138b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 >> v1:172.21.15.150:6789/0 conn(0x7f985c096b40 legacy=0x7f985c09c6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c09b0a0 con 0x7f985c09acd0 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: _finish_auth 0 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 5 monclient: authenticate success, global_id 54780 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f984c001b10 con 0x7f985c09acd0 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:01.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: dump: 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:16:01.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 >> v1:172.21.15.142:6789/0 conn(0x7f985c09acd0 legacy=0x7f985c09bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844003b30 0x7f9844003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): picked mon.c con 0x7f9844003b30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9844004810 0x7f9844004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): picked mon.b con 0x7f9844004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 0x7f984400be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): picked mon.a con 0x7f9844009a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c096250 con 0x7f9844009a50 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f984400c450 con 0x7f9844004810 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984400c610 con 0x7f9844003b30 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f985bfff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9863286700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9860821700 1 -- 172.21.15.150:0/2696118265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9844004810 msgr2=0x7f9844004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9860821700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9844004810 0x7f9844004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 0x7f984400be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844003b30 0x7f9844003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): get_auth_request con 0x7f9844009a50 auth_method 0 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:01.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): get_auth_request con 0x7f9844003b30 auth_method 0 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient(hunting): handle_auth_done global_id 54780 payload 293 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 -- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844003b30 msgr2=0x7f9844003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:01.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844003b30 0x7f9844003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 -- 172.21.15.150:0/2696118265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9844004810 msgr2=0x7f9844004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9844004810 0x7f9844004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient: _finish_hunting 0 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 monclient: found mon.a 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 -- 172.21.15.150:0/2696118265 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844003570 con 0x7f9844009a50 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 10 monclient: _finish_auth 0 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:01.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.928+0000 7f9861823700 1 -- 172.21.15.150:0/2696118265 reap_dead start 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9861823700 1 -- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 msgr2=0x7f984400be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9861823700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 0x7f984400be30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9854009970 tx=0x7f9854006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546565c14862cb9a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854003b30 con 0x7f9844009a50 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: dump: 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.955 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854003cb0 con 0x7f9844009a50 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:01.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 4 set_mon_vals no callback set 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 1 -- 172.21.15.150:0/2696118265 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854005320 con 0x7f9844009a50 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f985bfff700 10 monclient: dump: 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:01.957 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 10 monclient: shutdown 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 -- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 msgr2=0x7f984400be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 0x7f984400be30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9854009970 tx=0x7f9854006f50 comp rx=0 tx=0).stop 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 -- 172.21.15.150:0/2696118265 shutdown_connections 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 --2- 172.21.15.150:0/2696118265 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9844009a50 0x7f984400be30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 -- 172.21.15.150:0/2696118265 >> 172.21.15.150:0/2696118265 conn(0x7f985c0a15e0 msgr2=0x7f985c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 -- 172.21.15.150:0/2696118265 shutdown_connections 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.932+0000 7f9863286700 1 -- 172.21.15.150:0/2696118265 wait complete. 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient: build_initial_monmap 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient: monmap: 2024-09-18T10:16:01.958 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 Processor -- start 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 -- start start 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient: init 2024-09-18T10:16:01.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844009a50 0x7f985c135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): picked mon.noname-c con 0x7f9844009a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): picked mon.noname-a con 0x7f985c0915e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 0x7f985c08d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): picked mon.noname-b con 0x7f985c08ce00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c06f530 con 0x7f985c0915e0 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c06e7e0 con 0x7f985c08ce00 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985c090730 con 0x7f9844009a50 2024-09-18T10:16:01.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9863286700 10 monclient(hunting): authenticate will time out at 1244.033481s 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861823700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 msgr2=0x7f985c08d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861823700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 0x7f985c08d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844009a50 0x7f985c135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56712/0 (socket says 172.21.15.150:56712) 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 -- 172.21.15.150:0/598756427 learned_addr learned my addr 172.21.15.150:0/598756427 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): get_auth_request con 0x7f985c0915e0 auth_method 0 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:01.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): get_auth_request con 0x7f9844009a50 auth_method 0 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9861022700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient(hunting): handle_auth_done global_id 54783 payload 824 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 -- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844009a50 msgr2=0x7f985c135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:01.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844009a50 0x7f985c135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 -- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 msgr2=0x7f985c08d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 0x7f985c08d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient: _finish_hunting 0 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 monclient: found mon.noname-a 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.936+0000 7f9860821700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c08f560 con 0x7f985c0915e0 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9860821700 1 -- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 msgr2=0x7f985c136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9860821700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f985000b840 tx=0x7f9850009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850003e70 con 0x7f985c0915e0 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:01.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: dump: 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:01.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 5 monclient: authenticate success, global_id 54783 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9850006150 con 0x7f985c0915e0 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _renew_subs 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c090450 con 0x7f985c0915e0 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850006510 con 0x7f985c0915e0 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 10 monclient: dump: 2024-09-18T10:16:01.965 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _renew_subs 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f985b7fe700 4 set_mon_vals no callback set 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:01.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c08d7b0 con 0x7f985c0915e0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f98500103e0 con 0x7f985c0915e0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c095b90 con 0x7f985c0915e0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 0x7f9848037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(199..199 src has 167..199) v4 ==== 10114+0+0 (secure 0 0 0) 0x7f98500490c0 con 0x7f985c0915e0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9861022700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 0x7f9848037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.940+0000 7f9861022700 10 monclient: get_auth_request con 0x7f9848035a70 auth_method 0 2024-09-18T10:16:01.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.944+0000 7f9861022700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 0x7f9848037f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f984c00a190 tx=0x7f984c006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:01.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.956+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f985000ed00 con 0x7f985c0915e0 2024-09-18T10:16:01.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.956+0000 7f98597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:01.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:01.956+0000 7f98597fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:02.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.124+0000 7f9863286700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.124+0000 7f9863286700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.124+0000 7f9863286700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.124+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f985c063ed0 con 0x7f985c0915e0 2024-09-18T10:16:02.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.707+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 added bucket h11 type host to location {rack=r1} v200) v1 ==== 168+0+0 (secure 0 0 0) 0x7f9850017070 con 0x7f985c0915e0 2024-09-18T10:16:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.707+0000 7f98597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.707+0000 7f98597fa700 10 monclient: _finish_command 2 = system:0 added bucket h11 type host to location {rack=r1} 2024-09-18T10:16:02.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f9863286700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f9863286700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f9863286700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f985c067b10 con 0x7f985c0915e0 2024-09-18T10:16:02.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f98597fa700 1 -- 172.21.15.150:0/598756427 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h11' already exists v200) v1 ==== 147+0+0 (secure 0 0 0) 0x7f985000e700 con 0x7f985c0915e0 2024-09-18T10:16:02.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f98597fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h11", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:02.817 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.811+0000 7f98597fa700 10 monclient: _finish_command 3 = system:0 bucket 'h11' already exists 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 msgr2=0x7f9848037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 0x7f9848037f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f984c00a190 tx=0x7f984c006040 comp rx=0 tx=0).stop 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 10 monclient: shutdown 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 msgr2=0x7f985c136140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f985000b840 tx=0x7f9850009220 comp rx=0 tx=0).stop 2024-09-18T10:16:02.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 shutdown_connections 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9848035a70 0x7f9848037f20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f985c08ce00 0x7f985c08d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f985c0915e0 0x7f985c136140 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 --2- 172.21.15.150:0/598756427 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9844009a50 0x7f985c135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 >> 172.21.15.150:0/598756427 conn(0x7f985c0a15e0 msgr2=0x7f985c0a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 shutdown_connections 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.815+0000 7f9863286700 1 -- 172.21.15.150:0/598756427 wait complete. 2024-09-18T10:16:02.820 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h11' already exists 2024-09-18T10:16:02.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:02.829 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h12 host rack=r1 2024-09-18T10:16:02.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: get_monmap_and_config 2024-09-18T10:16:02.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: build_initial_monmap 2024-09-18T10:16:02.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: monmap: 2024-09-18T10:16:02.930 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:02.929062+0000 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:02.929062+0000 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:02.931 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:02.932 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:02.932 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:02.932 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 Processor -- start 2024-09-18T10:16:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 -- start start 2024-09-18T10:16:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: init 2024-09-18T10:16:02.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-b con 0x7f6a8409fbe0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6a84068050 0x7f6a84068440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-a con 0x7f6a84068050 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a84130010 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a84068980 con 0x7f6a84130010 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a84096f80 con 0x7f6a84068050 2024-09-18T10:16:02.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8406e9e0 con 0x7f6a8409fbe0 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.927+0000 7f6a8bac3700 10 monclient(hunting): authenticate will time out at 1245.023876s 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6a84068050 0x7f6a84068440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6a84068050 0x7f6a84068440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56720/0 (socket says 172.21.15.150:56720) 2024-09-18T10:16:02.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41456/0 (socket says 172.21.15.150:41456) 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8a060700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6a84130010 0x7f6a84130490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:53918/0 (socket says 172.21.15.150:53918) 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 -- 172.21.15.150:0/1688874358 learned_addr learned my addr 172.21.15.150:0/1688874358 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): get_auth_request con 0x7f6a84068050 auth_method 0 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): get_auth_request con 0x7f6a8409fbe0 auth_method 0 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:02.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient(hunting): handle_auth_done global_id 54786 payload 274 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 -- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 msgr2=0x7f6a84067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 -- 172.21.15.150:0/1688874358 >> v1:172.21.15.142:6789/0 conn(0x7f6a84130010 legacy=0x7f6a84130490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient: _finish_hunting 0 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 monclient: found mon.noname-a 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 -- 172.21.15.150:0/1688874358 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8409ffb0 con 0x7f6a84068050 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8985f700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 -- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a84068050 msgr2=0x7f6a84068440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8905e700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a84068050 0x7f6a84068440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6a740099b0 tx=0x7f6a74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed4fb14860dcb5b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 1 -- 172.21.15.150:0/1688874358 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74009ea0 con 0x7f6a84068050 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: dump: 2024-09-18T10:16:02.943 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:02.944 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:02.944 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.944 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.944 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:02.944 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.945 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:02.946 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:02.946 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:02.946 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: _finish_auth 0 2024-09-18T10:16:02.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8bac3700 5 monclient: authenticate success, global_id 54786 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8bac3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 1 -- 172.21.15.150:0/1688874358 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7400ba70 con 0x7f6a84068050 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8bac3700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:02.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8bac3700 4 set_mon_vals no callback set 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 1 -- 172.21.15.150:0/1688874358 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7400be30 con 0x7f6a84068050 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.931+0000 7f6a8885d700 10 monclient: dump: 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:02.948 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 10 monclient: shutdown 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a84068050 msgr2=0x7f6a84068440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a84068050 0x7f6a84068440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6a740099b0 tx=0x7f6a74009250 comp rx=0 tx=0).stop 2024-09-18T10:16:02.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/1688874358 shutdown_connections 2024-09-18T10:16:02.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a84068050 0x7f6a84068440 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/1688874358 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a8409fbe0 0x7f6a84067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/1688874358 >> 172.21.15.150:0/1688874358 conn(0x7f6a840a31e0 msgr2=0x7f6a8409f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:02.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/1688874358 shutdown_connections 2024-09-18T10:16:02.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/1688874358 wait complete. 2024-09-18T10:16:02.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 10 monclient: build_initial_monmap 2024-09-18T10:16:02.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 10 monclient: monmap: 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 Processor -- start 2024-09-18T10:16:02.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 1 -- start start 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.935+0000 7f6a8bac3700 10 monclient: init 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a84068050 0x7f6a8413f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-c con 0x7f6a84068050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-a con 0x7f6a8409fbe0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 0x7f6a8411cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): picked mon.noname-b con 0x7f6a84130010 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:02.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8411e810 con 0x7f6a8409fbe0 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8409da40 con 0x7f6a84130010 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84120630 con 0x7f6a84068050 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8bac3700 10 monclient(hunting): authenticate will time out at 1245.032686s 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8a060700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 msgr2=0x7f6a8411cc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:02.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8a060700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 0x7f6a8411cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56730/0 (socket says 172.21.15.150:56730) 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a84068050 0x7f6a8413f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 -- 172.21.15.150:0/2484002767 learned_addr learned my addr 172.21.15.150:0/2484002767 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): get_auth_request con 0x7f6a8409fbe0 auth_method 0 2024-09-18T10:16:02.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): get_auth_request con 0x7f6a84068050 auth_method 0 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:02.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8985f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient(hunting): handle_auth_done global_id 54789 payload 824 2024-09-18T10:16:02.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 -- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a84068050 msgr2=0x7f6a8413f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:02.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a84068050 0x7f6a8413f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 -- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 msgr2=0x7f6a8411cc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:02.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 0x7f6a8411cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:02.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient: _finish_hunting 0 2024-09-18T10:16:02.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 monclient: found mon.noname-a 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.939+0000 7f6a8905e700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a80009420 con 0x7f6a8409fbe0 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8905e700 1 -- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 msgr2=0x7f6a8412f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8905e700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6a8406f940 tx=0x7f6a74016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74006b60 con 0x7f6a8409fbe0 2024-09-18T10:16:02.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: dump: 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 5 monclient: authenticate success, global_id 54789 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _renew_subs 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a74020070 con 0x7f6a8409fbe0 2024-09-18T10:16:02.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740094f0 con 0x7f6a8409fbe0 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74011d80 con 0x7f6a8409fbe0 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 10 monclient: dump: 2024-09-18T10:16:02.967 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:02.968 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a7b7fe700 4 set_mon_vals no callback set 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _renew_subs 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8413fb60 con 0x7f6a8409fbe0 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:02.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6a74022030 con 0x7f6a8409fbe0 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8413fd50 con 0x7f6a8409fbe0 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 0x7f6a68037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(200..200 src has 167..200) v4 ==== 10327+0+0 (secure 0 0 0) 0x7f6a7404b300 con 0x7f6a8409fbe0 2024-09-18T10:16:02.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8985f700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 0x7f6a68037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:02.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.943+0000 7f6a8985f700 10 monclient: get_auth_request con 0x7f6a68035a70 auth_method 0 2024-09-18T10:16:02.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.947+0000 7f6a8985f700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 0x7f6a68037f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009e10 tx=0x7f6a800091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:02.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.951+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6a7402dbc0 con 0x7f6a8409fbe0 2024-09-18T10:16:02.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.951+0000 7f6a79ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:02.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:02.951+0000 7f6a79ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:03.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.123+0000 7f6a8bac3700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.123+0000 7f6a8bac3700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.123+0000 7f6a8bac3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:03.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.123+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6a84068980 con 0x7f6a8409fbe0 2024-09-18T10:16:03.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.827+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 added bucket h12 type host to location {rack=r1} v201) v1 ==== 168+0+0 (secure 0 0 0) 0x7f6a84068980 con 0x7f6a8409fbe0 2024-09-18T10:16:03.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.827+0000 7f6a79ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.827+0000 7f6a79ffb700 10 monclient: _finish_command 2 = system:0 added bucket h12 type host to location {rack=r1} 2024-09-18T10:16:03.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a8bac3700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a8bac3700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a8bac3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:03.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f6a84068980 con 0x7f6a8409fbe0 2024-09-18T10:16:03.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a79ffb700 1 -- 172.21.15.150:0/2484002767 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h12' already exists v201) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6a84068980 con 0x7f6a8409fbe0 2024-09-18T10:16:03.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a79ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h12", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:03.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.935+0000 7f6a79ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h12' already exists 2024-09-18T10:16:03.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 msgr2=0x7f6a68037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 0x7f6a68037f20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009e10 tx=0x7f6a800091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 10 monclient: shutdown 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 msgr2=0x7f6a8412f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6a8406f940 tx=0x7f6a74016000 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 shutdown_connections 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a68035a70 0x7f6a68037f20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a84130010 0x7f6a8411cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a8409fbe0 0x7f6a8412f170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 --2- 172.21.15.150:0/2484002767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a84068050 0x7f6a8413f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 >> 172.21.15.150:0/2484002767 conn(0x7f6a840a31e0 msgr2=0x7f6a840a0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:03.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 shutdown_connections 2024-09-18T10:16:03.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:03.939+0000 7f6a8bac3700 1 -- 172.21.15.150:0/2484002767 wait complete. 2024-09-18T10:16:03.943 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h12' already exists 2024-09-18T10:16:03.957 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:03.957 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h13 host rack=r1 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: get_monmap_and_config 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: build_initial_monmap 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: monmap: 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:04.060134+0000 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:04.060134+0000 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:04.061 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:04.062 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 Processor -- start 2024-09-18T10:16:04.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 -- start start 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: init 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f10fc0a5fb0 0x7f10fc09e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): picked mon.noname-a con 0x7f10fc0a5fb0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10fc09fb20 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): picked mon.noname-b con 0x7f10fc0a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc09df10 con 0x7f10fc09fb20 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc09d1e0 con 0x7f10fc0a5fb0 2024-09-18T10:16:04.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc09dca0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f1101332700 10 monclient(hunting): authenticate will time out at 1246.153546s 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fa7fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f10fc09fb20 legacy=0x7f10fc0a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41478/0 (socket says 172.21.15.150:41478) 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 -- 172.21.15.150:0/3959304477 learned_addr learned my addr 172.21.15.150:0/3959304477 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): get_auth_request con 0x7f10fc0a09a0 auth_method 0 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 1 --2- 172.21.15.150:0/3959304477 >> v2:172.21.15.142:3300/0 conn(0x7f10fc0a5fb0 0x7f10fc09e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:04.066 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): get_auth_request con 0x7f10fc0a5fb0 auth_method 0 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient(hunting): handle_auth_done global_id 74249 payload 274 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 -- 172.21.15.150:0/3959304477 >> v2:172.21.15.142:3300/0 conn(0x7f10fc0a5fb0 msgr2=0x7f10fc09e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:04.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 --2- 172.21.15.150:0/3959304477 >> v2:172.21.15.142:3300/0 conn(0x7f10fc0a5fb0 0x7f10fc09e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 -- 172.21.15.150:0/3959304477 >> v1:172.21.15.150:6789/0 conn(0x7f10fc09fb20 legacy=0x7f10fc0a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 monclient: found mon.noname-b 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:04.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.059+0000 7f10fb7fe700 1 -- 172.21.15.150:0/3959304477 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0a6380 con 0x7f10fc0a09a0 2024-09-18T10:16:04.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10fb7fe700 1 -- 172.21.15.150:0/3959304477 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 msgr2=0x7f10fc139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:04.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10fb7fe700 1 --2- 172.21.15.150:0/3959304477 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0a65d0 tx=0x7f10f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba3eea040802e6f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:04.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 1 -- 172.21.15.150:0/3959304477 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f400e070 con 0x7f10fc0a09a0 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: dump: 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 5 monclient: authenticate success, global_id 74249 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 1 -- 172.21.15.150:0/3959304477 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400bc30 con 0x7f10fc0a09a0 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 4 set_mon_vals no callback set 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 1 -- 172.21.15.150:0/3959304477 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f40092e0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f10f9ffb700 10 monclient: dump: 2024-09-18T10:16:04.075 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 10 monclient: shutdown 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- 172.21.15.150:0/3959304477 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 msgr2=0x7f10fc139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:04.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 --2- 172.21.15.150:0/3959304477 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0a65d0 tx=0x7f10f4009840 comp rx=0 tx=0).stop 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- 172.21.15.150:0/3959304477 shutdown_connections 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 --2- 172.21.15.150:0/3959304477 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a09a0 0x7f10fc139a30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 --2- 172.21.15.150:0/3959304477 >> v2:172.21.15.142:3300/0 conn(0x7f10fc0a5fb0 0x7f10fc09e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- 172.21.15.150:0/3959304477 >> 172.21.15.150:0/3959304477 conn(0x7f10fc071ae0 msgr2=0x7f10fc0a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- 172.21.15.150:0/3959304477 shutdown_connections 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- 172.21.15.150:0/3959304477 wait complete. 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 10 monclient: build_initial_monmap 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 10 monclient: monmap: 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:04.077 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 Processor -- start 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 1 -- start start 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.063+0000 7f1101332700 10 monclient: init 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): picked mon.noname-a con 0x7f10fc0a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a5fb0 0x7f10fc08e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): picked mon.noname-c con 0x7f10fc0a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 0x7f10fc08e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): picked mon.noname-b con 0x7f10fc090260 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0987e0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc08f720 con 0x7f10fc090260 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc08fc10 con 0x7f10fc0a5fb0 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f1101332700 10 monclient(hunting): authenticate will time out at 1246.160346s 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fb7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 msgr2=0x7f10fc08e8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fb7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 0x7f10fc08e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:04.079 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56758/0 (socket says 172.21.15.150:56758) 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 -- 172.21.15.150:0/3921696348 learned_addr learned my addr 172.21.15.150:0/3921696348 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a5fb0 0x7f10fc08e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): get_auth_request con 0x7f10fc0a09a0 auth_method 0 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): get_auth_request con 0x7f10fc0a5fb0 auth_method 0 2024-09-18T10:16:04.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10fa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient(hunting): handle_auth_done global_id 54795 payload 824 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 -- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a5fb0 msgr2=0x7f10fc08e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:04.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a5fb0 0x7f10fc08e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 -- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 msgr2=0x7f10fc08e8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 0x7f10fc08e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 monclient: found mon.noname-a 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009ae0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 -- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 msgr2=0x7f10fc08de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.067+0000 7f10faffd700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f10f000ecb0 tx=0x7f10f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000cb30 con 0x7f10fc0a09a0 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:04.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: dump: 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 5 monclient: authenticate success, global_id 54795 2024-09-18T10:16:04.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _renew_subs 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f000cc90 con 0x7f10fc0a09a0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc09bf10 con 0x7f10fc0a09a0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _renew_subs 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc091f60 con 0x7f10fc0a09a0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10f97fa700 4 set_mon_vals no callback set 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0013470 con 0x7f10fc0a09a0 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 10 monclient: dump: 2024-09-18T10:16:04.084 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:04.085 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc09e1f0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f10f00137e0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 0x7f10e8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(201..201 src has 176..201) v4 ==== 10540+0+0 (secure 0 0 0) 0x7f10f0049da0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10fa7fc700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 0x7f10e8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.071+0000 7f10fa7fc700 10 monclient: get_auth_request con 0x7f10e8035a70 auth_method 0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.075+0000 7f10fa7fc700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 0x7f10e8037f20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0038e0 tx=0x7f10ec009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:04.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.083+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f10f00236b0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.083+0000 7f10e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:04.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.083+0000 7f10e2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:04.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.247+0000 7f1101332700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.247+0000 7f1101332700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.247+0000 7f1101332700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.247+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f10fc063ed0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.839+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 added bucket h13 type host to location {rack=r1} v202) v1 ==== 168+0+0 (secure 0 0 0) 0x7f10f00232f0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.839+0000 7f10e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.839+0000 7f10e2ffd700 10 monclient: _finish_command 2 = system:0 added bucket h13 type host to location {rack=r1} 2024-09-18T10:16:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.947+0000 7f1101332700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.947+0000 7f1101332700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.947+0000 7f1101332700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:04.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.947+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f10fc091500 con 0x7f10fc0a09a0 2024-09-18T10:16:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f10e2ffd700 1 -- 172.21.15.150:0/3921696348 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h13' already exists v202) v1 ==== 147+0+0 (secure 0 0 0) 0x7f10f00234d0 con 0x7f10fc0a09a0 2024-09-18T10:16:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f10e2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h13", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:04.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f10e2ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h13' already exists 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 msgr2=0x7f10e8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 0x7f10e8037f20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0038e0 tx=0x7f10ec009120 comp rx=0 tx=0).stop 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 10 monclient: shutdown 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 msgr2=0x7f10fc08de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f10f000ecb0 tx=0x7f10f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 shutdown_connections 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f10e8035a70 0x7f10e8037f20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10fc090260 0x7f10fc08e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f10fc0a5fb0 0x7f10fc08e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 --2- 172.21.15.150:0/3921696348 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10fc0a09a0 0x7f10fc08de60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 >> 172.21.15.150:0/3921696348 conn(0x7f10fc071ae0 msgr2=0x7f10fc09b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 shutdown_connections 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:04.951+0000 7f1101332700 1 -- 172.21.15.150:0/3921696348 wait complete. 2024-09-18T10:16:04.956 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h13' already exists 2024-09-18T10:16:04.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:04.969 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h14 host rack=r1 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: get_monmap_and_config 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: build_initial_monmap 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: monmap: 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.071 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:05.065368+0000 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:05.065368+0000 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 Processor -- start 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 -- start start 2024-09-18T10:16:05.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: init 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efdc8071a50 0x7efdc8130470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-a con 0x7efdc8071a50 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdc806e850 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-b con 0x7efdc81309b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc8097820 con 0x7efdc806e850 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc8096f80 con 0x7efdc8071a50 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdc7fff700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7efdc806e850 legacy=0x7efdc806c680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc8071e20 con 0x7efdc81309b0 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdceda7700 10 monclient(hunting): authenticate will time out at 1247.159268s 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41488/0 (socket says 172.21.15.150:41488) 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 1 -- 172.21.15.150:0/3060458684 learned_addr learned my addr 172.21.15.150:0/3060458684 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): get_auth_request con 0x7efdc81309b0 auth_method 0 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 1 --2- 172.21.15.150:0/3060458684 >> v2:172.21.15.142:3300/0 conn(0x7efdc8071a50 0x7efdc8130470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): get_auth_request con 0x7efdc8071a50 auth_method 0 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdcd344700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.063+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 10 monclient(hunting): handle_auth_done global_id 74255 payload 274 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 -- 172.21.15.150:0/3060458684 >> v2:172.21.15.142:3300/0 conn(0x7efdc8071a50 msgr2=0x7efdc8130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 --2- 172.21.15.150:0/3060458684 >> v2:172.21.15.142:3300/0 conn(0x7efdc8071a50 0x7efdc8130470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 -- 172.21.15.150:0/3060458684 >> v1:172.21.15.150:6789/0 conn(0x7efdc806e850 legacy=0x7efdc806c680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 10 monclient: _finish_hunting 0 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 monclient: found mon.noname-b 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 -- 172.21.15.150:0/3060458684 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8097540 con 0x7efdc81309b0 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 -- 172.21.15.150:0/3060458684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 msgr2=0x7efdc806cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:05.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdcd344700 1 --2- 172.21.15.150:0/3060458684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc00099b0 tx=0x7efdc0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f46475830d2fcdb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 1 -- 172.21.15.150:0/3060458684 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0009ea0 con 0x7efdc81309b0 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: dump: 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 5 monclient: authenticate success, global_id 74255 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 1 -- 172.21.15.150:0/3060458684 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc000ba70 con 0x7efdc81309b0 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 1 -- 172.21.15.150:0/3060458684 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc000be30 con 0x7efdc81309b0 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdc77fe700 10 monclient: dump: 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 4 set_mon_vals no callback set 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 10 monclient: shutdown 2024-09-18T10:16:05.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 1 -- 172.21.15.150:0/3060458684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 msgr2=0x7efdc806cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 1 --2- 172.21.15.150:0/3060458684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc00099b0 tx=0x7efdc0009250 comp rx=0 tx=0).stop 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 1 -- 172.21.15.150:0/3060458684 shutdown_connections 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 1 --2- 172.21.15.150:0/3060458684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc81309b0 0x7efdc806cd90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.067+0000 7efdceda7700 1 --2- 172.21.15.150:0/3060458684 >> v2:172.21.15.142:3300/0 conn(0x7efdc8071a50 0x7efdc8130470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- 172.21.15.150:0/3060458684 >> 172.21.15.150:0/3060458684 conn(0x7efdc80a41f0 msgr2=0x7efdc80710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- 172.21.15.150:0/3060458684 shutdown_connections 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- 172.21.15.150:0/3060458684 wait complete. 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient: build_initial_monmap 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient: monmap: 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 Processor -- start 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- start start 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient: init 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-a con 0x7efdc806e850 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efdc813f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-c con 0x7efdc8071a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efdc813f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): picked mon.noname-b con 0x7efdc81309b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc811b870 con 0x7efdc806e850 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc811d580 con 0x7efdc81309b0 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc813fd30 con 0x7efdc8071a50 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdceda7700 10 monclient(hunting): authenticate will time out at 1247.166978s 2024-09-18T10:16:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdcd344700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 msgr2=0x7efdc813f7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdcd344700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efdc813f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efdc813f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:56776/0 (socket says 172.21.15.150:56776) 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 learned_addr learned my addr 172.21.15.150:0/3648940075 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): get_auth_request con 0x7efdc806e850 auth_method 0 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): get_auth_request con 0x7efdc8071a50 auth_method 0 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.071+0000 7efdccb43700 10 monclient(hunting): handle_auth_done global_id 54801 payload 824 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 msgr2=0x7efdc813f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efdc813f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 msgr2=0x7efdc813f7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efdc813f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 10 monclient: _finish_hunting 0 2024-09-18T10:16:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 monclient: found mon.noname-a 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00094f0 con 0x7efdc806e850 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc7fff700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efdc813f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 msgr2=0x7efdc812f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009ef0 tx=0x7efdb800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb800ce30 con 0x7efdc806e850 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: dump: 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.095 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb800f040 con 0x7efdc806e850 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80146b0 con 0x7efdc806e850 2024-09-18T10:16:05.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: dump: 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 5 monclient: authenticate success, global_id 54801 2024-09-18T10:16:05.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 10 monclient: _renew_subs 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc811c560 con 0x7efdc806e850 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc67fc700 4 set_mon_vals no callback set 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 10 monclient: _renew_subs 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc812f6b0 con 0x7efdc806e850 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 msgr2=0x7efdc812f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 msgr2=0x7efdc812f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009ef0 tx=0x7efdb800b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efdb8009ef0 tx=0x7efdb800b040 comp rx=0 tx=0).stop 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 msgr2=0x7efdc812f170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdc806e850 0x7efdc812f170 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efda80057e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): picked mon.c con 0x7efdc8071a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 0x7efda8008560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): picked mon.a con 0x7efda80060e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efda800b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): picked mon.b con 0x7efdc81309b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda800bab0 con 0x7efda80060e0 2024-09-18T10:16:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda800bbf0 con 0x7efdc81309b0 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda800bd30 con 0x7efdc8071a50 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdc4ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdcd344700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 msgr2=0x7efda800b0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.075+0000 7efdcd344700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efda800b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 0x7efda8008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efda80057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): get_auth_request con 0x7efda80060e0 auth_method 0 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): get_auth_request con 0x7efdc8071a50 auth_method 0 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdccb43700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdceda7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient(hunting): handle_auth_done global_id 54801 payload 293 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 msgr2=0x7efda80057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efda80057e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 msgr2=0x7efda800b0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efda800b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 monclient: found mon.a 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7efda800bfc0 con 0x7efda80060e0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 10 monclient: _finish_auth 0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdceda7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdceda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc811c3b0 con 0x7efda80060e0 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 msgr2=0x7efda8008560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc7fff700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 0x7efda8008560 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efdc811dc80 tx=0x7efdb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb0003ba0 con 0x7efda80060e0 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 10 monclient: dump: 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb00066b0 con 0x7efda80060e0 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.079+0000 7efdc6ffd700 4 set_mon_vals no callback set 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.083+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efdb00123f0 con 0x7efda80060e0 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.083+0000 7efdc4ff9700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 0x7efda80409b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.083+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 4 ==== osd_map(202..202 src has 176..202) v4 ==== 10753+0+0 (secure 0 0 0) 0x7efdb0047e30 con 0x7efda80060e0 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.083+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 0x7efda80409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.083+0000 7efdccb43700 10 monclient: get_auth_request con 0x7efda803e500 auth_method 0 2024-09-18T10:16:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.087+0000 7efdccb43700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 0x7efda80409b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efdb801f040 tx=0x7efdb801a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.091+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7efdb0019310 con 0x7efda80060e0 2024-09-18T10:16:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.091+0000 7efdc4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.091+0000 7efdc4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:05.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.255+0000 7efdceda7700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:05.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.255+0000 7efdceda7700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:05.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.255+0000 7efdceda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:05.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:05.255+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efdc811d8c0 con 0x7efda80060e0 2024-09-18T10:16:05.515 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:16:05.516 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:16:05.516 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:16:05.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:16:05.519 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:16:05.519 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-18T10:16:05.519 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-18T10:16:05.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:16:08.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:08.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:08.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:08.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:08.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:08.071+0000 7efdbf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:09.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:09.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:09.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:10.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:10.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:10.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:10.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:11.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:11.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:11.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:11.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:12.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:12.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:12.076 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:12.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:13.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:13.071+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:13.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:13.071+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:14.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:14.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:14.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:14.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:15.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:15.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:15.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:15.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:16.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:16.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:16.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:16.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:17.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:17.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:17.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:17.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:18.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:18.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:18.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:18.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:18.080 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:18.075+0000 7efdbf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:19.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:19.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:19.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:19.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:20.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:20.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:20.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:20.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:21.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:21.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:21.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:21.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:22.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:22.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:22.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:22.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:23.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:23.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:23.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:24.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:24.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:24.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:24.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.075+0000 7efdbf7fe700 10 monclient: tick 2024-09-18T10:16:25.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.075+0000 7efdbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.520 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-18T10:16:25.521 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-18T10:16:25.521 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:16:25.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:25.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.627+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 added bucket h14 type host to location {rack=r1} v203) v1 ==== 168+0+0 (secure 0 0 0) 0x7efdb0016050 con 0x7efda80060e0 2024-09-18T10:16:25.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.631+0000 7efdc4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:25.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.631+0000 7efdc4ff9700 10 monclient: _finish_command 2 = system:0 added bucket h14 type host to location {rack=r1} 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.627+0000 7fc218e4f700 10 monclient: get_monmap_and_config 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.627+0000 7fc218e4f700 10 monclient: build_initial_monmap 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient: monmap: 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:25.635725+0000 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:25.635725+0000 2024-09-18T10:16:25.637 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:25.638 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 Processor -- start 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 -- start start 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient: init 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-b con 0x7fc2140a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:25.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc21412f810 0x7fc21409f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-c con 0x7fc21412f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:25.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2140a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:25.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21409d990 con 0x7fc2140a02d0 2024-09-18T10:16:25.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21409dc70 con 0x7fc21412f810 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a20c0 con 0x7fc2140a1cf0 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc218e4f700 10 monclient(hunting): authenticate will time out at 1274.561350s 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50490/0 (socket says 172.21.15.142:50490) 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 -- 172.21.15.142:0/3494500178 learned_addr learned my addr 172.21.15.142:0/3494500178 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:25.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): get_auth_request con 0x7fc2140a1cf0 auth_method 0 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc20affd700 1 --2- 172.21.15.142:0/3494500178 >> v2:172.21.15.150:3300/0 conn(0x7fc21412f810 0x7fc21409f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc20affd700 10 monclient(hunting): get_auth_request con 0x7fc21412f810 auth_method 0 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc20affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc20affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc20affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient(hunting): handle_auth_done global_id 74261 payload 274 2024-09-18T10:16:25.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 -- 172.21.15.142:0/3494500178 >> v2:172.21.15.150:3300/0 conn(0x7fc21412f810 msgr2=0x7fc21409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 --2- 172.21.15.142:0/3494500178 >> v2:172.21.15.150:3300/0 conn(0x7fc21412f810 0x7fc21409f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 -- 172.21.15.142:0/3494500178 >> v1:172.21.15.150:6789/0 conn(0x7fc2140a02d0 legacy=0x7fc2141320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 monclient: found mon.noname-b 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.631+0000 7fc2137fe700 1 -- 172.21.15.142:0/3494500178 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21409f9c0 con 0x7fc2140a1cf0 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc2137fe700 1 -- 172.21.15.142:0/3494500178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 msgr2=0x7fc21412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc2137fe700 1 --2- 172.21.15.142:0/3494500178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc2040099b0 tx=0x7fc204009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1810c19187e5c2e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 1 -- 172.21.15.142:0/3494500178 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20400ba70 con 0x7fc2140a1cf0 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:25.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: dump: 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 5 monclient: authenticate success, global_id 74261 2024-09-18T10:16:25.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 1 -- 172.21.15.142:0/3494500178 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20400bbd0 con 0x7fc2140a1cf0 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 4 set_mon_vals no callback set 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 1 -- 172.21.15.142:0/3494500178 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204011420 con 0x7fc2140a1cf0 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc212ffd700 10 monclient: dump: 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.652 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 10 monclient: shutdown 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- 172.21.15.142:0/3494500178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 msgr2=0x7fc21412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3494500178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc2040099b0 tx=0x7fc204009250 comp rx=0 tx=0).stop 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- 172.21.15.142:0/3494500178 shutdown_connections 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3494500178 >> v2:172.21.15.150:3300/0 conn(0x7fc21412f810 0x7fc21409f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3494500178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a1cf0 0x7fc21412f030 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- 172.21.15.142:0/3494500178 >> 172.21.15.142:0/3494500178 conn(0x7fc214071820 msgr2=0x7fc2140a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- 172.21.15.142:0/3494500178 shutdown_connections 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- 172.21.15.142:0/3494500178 wait complete. 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 10 monclient: build_initial_monmap 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 10 monclient: monmap: 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:25.654 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:25.655 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:25.655 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 Processor -- start 2024-09-18T10:16:25.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 1 -- start start 2024-09-18T10:16:25.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.635+0000 7fc218e4f700 10 monclient: init 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 0x7fc21409b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-c con 0x7fc2140a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2140a1cf0 0x7fc21409b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-b con 0x7fc2140a1cf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): picked mon.noname-a con 0x7fc21412f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21409cdc0 con 0x7fc21412f810 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21412eec0 con 0x7fc2140a1cf0 2024-09-18T10:16:25.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21406baf0 con 0x7fc2140a02d0 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient(hunting): authenticate will time out at 1274.568130s 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 0x7fc21409b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47244/0 (socket says 172.21.15.142:47244) 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 1 -- 172.21.15.142:0/3428742022 learned_addr learned my addr 172.21.15.142:0/3428742022 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): get_auth_request con 0x7fc21412f810 auth_method 0 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): get_auth_request con 0x7fc2140a02d0 auth_method 0 2024-09-18T10:16:25.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2140a1cf0 0x7fc21409b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): get_auth_request con 0x7fc2140a1cf0 auth_method 0 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient(hunting): handle_auth_done global_id 74264 payload 824 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 -- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2140a1cf0 msgr2=0x7fc21409b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2140a1cf0 0x7fc21409b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 -- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 msgr2=0x7fc21409bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 -- 172.21.15.142:0/3428742022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040094f0 con 0x7fc2140a02d0 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc213fff700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 -- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 msgr2=0x7fc21409b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2137fe700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 0x7fc21409b060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc204002000 tx=0x7fc204016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204000c90 con 0x7fc2140a02d0 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: dump: 2024-09-18T10:16:25.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: _finish_auth 0 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 5 monclient: authenticate success, global_id 74264 2024-09-18T10:16:25.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20401b680 con 0x7fc2140a02d0 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _renew_subs 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20401ba40 con 0x7fc2140a02d0 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21406c9b0 con 0x7fc2140a02d0 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc2127fc700 4 set_mon_vals no callback set 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc20bfff700 10 monclient: dump: 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.667 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.667 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:25.667 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _renew_subs 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.639+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21406f900 con 0x7fc2140a02d0 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc20401c450 con 0x7fc2140a02d0 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc218e4f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc218e4f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc218e4f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21406ce50 con 0x7fc2140a02d0 2024-09-18T10:16:25.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc20bfff700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 0x7fc1f4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7fc20404b670 con 0x7fc2140a02d0 2024-09-18T10:16:25.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc20affd700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 0x7fc1f4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.643+0000 7fc20affd700 10 monclient: get_auth_request con 0x7fc1f4035ac0 auth_method 0 2024-09-18T10:16:25.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.647+0000 7fc20affd700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 0x7fc1f4037f70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc006fd0 tx=0x7fc1fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.647+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc204018950 con 0x7fc2140a02d0 2024-09-18T10:16:25.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.647+0000 7fc20bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.647+0000 7fc20bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:25.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.739+0000 7efdceda7700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:25.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.739+0000 7efdceda7700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:25.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.739+0000 7efdceda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:25.744 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.739+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7efdc8097260 con 0x7efda80060e0 2024-09-18T10:16:25.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdc4ff9700 1 -- 172.21.15.150:0/3648940075 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h14' already exists v203) v1 ==== 147+0+0 (secure 0 0 0) 0x7efdb001f0b0 con 0x7efda80060e0 2024-09-18T10:16:25.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdc4ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h14", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:25.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdc4ff9700 10 monclient: _finish_command 3 = system:0 bucket 'h14' already exists 2024-09-18T10:16:25.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 msgr2=0x7efda80409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 0x7efda80409b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7efdb801f040 tx=0x7efdb801a000 comp rx=0 tx=0).stop 2024-09-18T10:16:25.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 10 monclient: shutdown 2024-09-18T10:16:25.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:25.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 msgr2=0x7efda8008560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 0x7efda8008560 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7efdc811dc80 tx=0x7efdb00091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 shutdown_connections 2024-09-18T10:16:25.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efda803e500 0x7efda80409b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdc81309b0 0x7efda800b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efda80060e0 0x7efda8008560 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 --2- 172.21.15.150:0/3648940075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdc8071a50 0x7efda80057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 >> 172.21.15.150:0/3648940075 conn(0x7efdc80a41f0 msgr2=0x7efdc80980c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 shutdown_connections 2024-09-18T10:16:25.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.743+0000 7efdceda7700 1 -- 172.21.15.150:0/3648940075 wait complete. 2024-09-18T10:16:25.754 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h14' already exists 2024-09-18T10:16:25.757 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:25.758 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h15 host rack=r1 2024-09-18T10:16:25.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: get_monmap_and_config 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: build_initial_monmap 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: monmap: 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:25.861695+0000 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:25.861695+0000 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:25.863 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:25.864 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:25.864 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 Processor -- start 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 -- start start 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: init 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19d80a0290 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:25.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19d80a0d70 0x7f19d809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-c con 0x7f19d80a0d70 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d8137a20 0x7f19d8139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-b con 0x7f19d8137a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d809e360 con 0x7f19d80a0290 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d809ef80 con 0x7f19d80a0d70 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80a1fb0 con 0x7f19d8137a20 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19dced1700 10 monclient(hunting): authenticate will time out at 1267.955329s 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d659c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f19d80a0290 0x7f19d80a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:54824/0 (socket says 172.21.15.150:54824) 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19d80a0d70 0x7f19d809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d659c700 1 -- 172.21.15.150:0/3356796089 learned_addr learned my addr 172.21.15.150:0/3356796089 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): get_auth_request con 0x7f19d80a0d70 auth_method 0 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d8137a20 0x7f19d8139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): get_auth_request con 0x7f19d8137a20 auth_method 0 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.859+0000 7f19d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 10 monclient(hunting): handle_auth_done global_id 84106 payload 274 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 -- 172.21.15.150:0/3356796089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d8137a20 msgr2=0x7f19d8139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d8137a20 0x7f19d8139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 -- 172.21.15.150:0/3356796089 >> v1:172.21.15.150:6789/0 conn(0x7f19d80a0290 legacy=0x7f19d80a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 monclient: found mon.noname-c 2024-09-18T10:16:25.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:25.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:25.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 -- 172.21.15.150:0/3356796089 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d809da50 con 0x7f19d80a0d70 2024-09-18T10:16:25.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 -- 172.21.15.150:0/3356796089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0d70 msgr2=0x7f19d809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:25.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d5d9b700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0d70 0x7f19d809ea40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009e60 tx=0x7f19c8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f391193ca7df0cf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 1 -- 172.21.15.150:0/3356796089 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c800e070 con 0x7f19d80a0d70 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: dump: 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.877 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: _finish_auth 0 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19dced1700 5 monclient: authenticate success, global_id 84106 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19dced1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 1 -- 172.21.15.150:0/3356796089 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c800bd10 con 0x7f19d80a0d70 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 1 -- 172.21.15.150:0/3356796089 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c8012660 con 0x7f19d80a0d70 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19dced1700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19dced1700 4 set_mon_vals no callback set 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:25.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.863+0000 7f19d559a700 10 monclient: dump: 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: shutdown 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- 172.21.15.150:0/3356796089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0d70 msgr2=0x7f19d809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0d70 0x7f19d809ea40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009e60 tx=0x7f19c8009780 comp rx=0 tx=0).stop 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- 172.21.15.150:0/3356796089 shutdown_connections 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d8137a20 0x7f19d8139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- 172.21.15.150:0/3356796089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0d70 0x7f19d809ea40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- 172.21.15.150:0/3356796089 >> 172.21.15.150:0/3356796089 conn(0x7f19d80a58a0 msgr2=0x7f19d80a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- 172.21.15.150:0/3356796089 shutdown_connections 2024-09-18T10:16:25.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- 172.21.15.150:0/3356796089 wait complete. 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: build_initial_monmap 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: monmap: 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:25.881 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 Processor -- start 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- start start 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: init 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-b con 0x7f19d80a0290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d80a0d70 0x7f19d8142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-c con 0x7f19d80a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 0x7f19d812c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): picked mon.noname-a con 0x7f19d8137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d809d3f0 con 0x7f19d8137a20 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d809c080 con 0x7f19d80a0290 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d812ecc0 con 0x7f19d80a0d70 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19dced1700 10 monclient(hunting): authenticate will time out at 1267.961858s 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60984/0 (socket says 172.21.15.150:60984) 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 -- 172.21.15.150:0/2373090691 learned_addr learned my addr 172.21.15.150:0/2373090691 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): get_auth_request con 0x7f19d80a0290 auth_method 0 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 0x7f19d812c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d5d9b700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d80a0d70 0x7f19d8142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 10 monclient(hunting): get_auth_request con 0x7f19d8137a20 auth_method 0 2024-09-18T10:16:25.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d5d9b700 10 monclient(hunting): get_auth_request con 0x7f19d80a0d70 auth_method 0 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient(hunting): handle_auth_done global_id 84109 payload 824 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 -- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d80a0d70 msgr2=0x7f19d8142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d80a0d70 0x7f19d8142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 -- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 msgr2=0x7f19d812c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 0x7f19d812c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:25.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 monclient: found mon.noname-b 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d659c700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8009a20 con 0x7f19d80a0290 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.867+0000 7f19d6d9d700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 0x7f19d812c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19d659c700 1 -- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 msgr2=0x7f19d81420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19d659c700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00b5e0 tx=0x7f19cc009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc011840 con 0x7f19d80a0290 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:25.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: dump: 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:25.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 5 monclient: authenticate success, global_id 84109 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc0119c0 con 0x7f19d80a0290 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _renew_subs 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc011d80 con 0x7f19d80a0290 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d81301e0 con 0x7f19d80a0290 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19d4d99700 4 set_mon_vals no callback set 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 10 monclient: dump: 2024-09-18T10:16:25.888 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _renew_subs 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d812e7f0 con 0x7f19d80a0290 2024-09-18T10:16:25.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d809d6c0 con 0x7f19d80a0290 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f19cc010590 con 0x7f19d80a0290 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 0x7f19b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7f19cc049500 con 0x7f19d80a0290 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19d5d9b700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 0x7f19b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:25.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.871+0000 7f19d5d9b700 10 monclient: get_auth_request con 0x7f19b8035ac0 auth_method 0 2024-09-18T10:16:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.875+0000 7f19d5d9b700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 0x7f19b8037f70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f19c80099a0 tx=0x7f19c8018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:25.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.883+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f19cc00d860 con 0x7f19d80a0290 2024-09-18T10:16:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.883+0000 7f19c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:25.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:25.883+0000 7f19c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc214072a80 con 0x7fc2140a02d0 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc20bfff700 1 -- 172.21.15.142:0/3428742022 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fc20402d350 con 0x7fc2140a02d0 2024-09-18T10:16:25.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc20bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:25.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc20bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:25.893 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 msgr2=0x7fc1f4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 0x7fc1f4037f70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc006fd0 tx=0x7fc1fc008040 comp rx=0 tx=0).stop 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 10 monclient: shutdown 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 msgr2=0x7fc21409b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 0x7fc21409b060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc204002000 tx=0x7fc204016000 comp rx=0 tx=0).stop 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 shutdown_connections 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f4035ac0 0x7fc1f4037f70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc21412f810 0x7fc21409bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2140a1cf0 0x7fc21409b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 --2- 172.21.15.142:0/3428742022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2140a02d0 0x7fc21409b060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:25.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 >> 172.21.15.142:0/3428742022 conn(0x7fc214071820 msgr2=0x7fc214131f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:25.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 shutdown_connections 2024-09-18T10:16:25.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.871+0000 7fc218e4f700 1 -- 172.21.15.142:0/3428742022 wait complete. 2024-09-18T10:16:25.896 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:16:25.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:16:25.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:16:26.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: get_monmap_and_config 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: monmap: 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:26.004351+0000 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:26.004351+0000 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:26.006 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:26.007 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 Processor -- start 2024-09-18T10:16:26.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 -- start start 2024-09-18T10:16:26.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: init 2024-09-18T10:16:26.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5a409abd0 0x7fd5a40960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd5a409abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd5a409b740 0x7fd5a409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd5a409b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5a409c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a409dc90 con 0x7fd5a409c410 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a409b500 con 0x7fd5a409b740 2024-09-18T10:16:26.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a409afa0 con 0x7fd5a409abd0 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a359e700 10 monclient(hunting): authenticate will time out at 1274.929949s 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a2d9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd5a409c410 0x7fd5a4136680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57926/0 (socket says 172.21.15.142:57926) 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a2d9d700 1 -- 172.21.15.142:0/374427555 learned_addr learned my addr 172.21.15.142:0/374427555 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 --2- 172.21.15.142:0/374427555 >> v2:172.21.15.142:3300/0 conn(0x7fd5a409b740 0x7fd5a409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): get_auth_request con 0x7fd5a409b740 auth_method 0 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 1 --2- 172.21.15.142:0/374427555 >> v2:172.21.15.150:3300/0 conn(0x7fd5a409abd0 0x7fd5a40960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): get_auth_request con 0x7fd5a409abd0 auth_method 0 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient(hunting): handle_auth_done global_id 54810 payload 274 2024-09-18T10:16:26.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 -- 172.21.15.142:0/374427555 >> v2:172.21.15.150:3300/0 conn(0x7fd5a409abd0 msgr2=0x7fd5a40960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 --2- 172.21.15.142:0/374427555 >> v2:172.21.15.150:3300/0 conn(0x7fd5a409abd0 0x7fd5a40960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 -- 172.21.15.142:0/374427555 >> v1:172.21.15.142:6789/0 conn(0x7fd5a409c410 legacy=0x7fd5a4136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 monclient: found mon.noname-a 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:25.999+0000 7fd599d9b700 1 -- 172.21.15.142:0/374427555 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a406f810 con 0x7fd5a409b740 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd599d9b700 1 -- 172.21.15.142:0/374427555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409b740 msgr2=0x7fd5a409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd599d9b700 1 --2- 172.21.15.142:0/374427555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409b740 0x7fd5a409bed0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd5a406a310 tx=0x7fd5940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce82d10aa2e1fd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 1 -- 172.21.15.142:0/374427555 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594011840 con 0x7fd5a409b740 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: dump: 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.017 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 5 monclient: authenticate success, global_id 54810 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 1 -- 172.21.15.142:0/374427555 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5940119c0 con 0x7fd5a409b740 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 4 set_mon_vals no callback set 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 1 -- 172.21.15.142:0/374427555 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594011d80 con 0x7fd5a409b740 2024-09-18T10:16:26.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a1d9b700 10 monclient: dump: 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 10 monclient: shutdown 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- 172.21.15.142:0/374427555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409b740 msgr2=0x7fd5a409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 --2- 172.21.15.142:0/374427555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409b740 0x7fd5a409bed0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd5a406a310 tx=0x7fd5940091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- 172.21.15.142:0/374427555 shutdown_connections 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 --2- 172.21.15.142:0/374427555 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409b740 0x7fd5a409bed0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 --2- 172.21.15.142:0/374427555 >> v2:172.21.15.150:3300/0 conn(0x7fd5a409abd0 0x7fd5a40960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- 172.21.15.142:0/374427555 >> 172.21.15.142:0/374427555 conn(0x7fd5a4070ef0 msgr2=0x7fd5a40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- 172.21.15.142:0/374427555 shutdown_connections 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- 172.21.15.142:0/374427555 wait complete. 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 10 monclient: monmap: 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:26.021 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 Processor -- start 2024-09-18T10:16:26.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 1 -- start start 2024-09-18T10:16:26.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.003+0000 7fd5a359e700 10 monclient: init 2024-09-18T10:16:26.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 0x7fd5a413f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd5a409abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5a409b740 0x7fd5a413feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd5a409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a409c410 0x7fd5a408bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd5a409c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a409d240 con 0x7fd5a409abd0 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a408dcc0 con 0x7fd5a409c410 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a409ffc0 con 0x7fd5a409b740 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient(hunting): authenticate will time out at 1274.936669s 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 0x7fd5a413f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5a409b740 0x7fd5a413feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 0x7fd5a413f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47258/0 (socket says 172.21.15.142:47258) 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 -- 172.21.15.142:0/1623495529 learned_addr learned my addr 172.21.15.142:0/1623495529 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): get_auth_request con 0x7fd5a409abd0 auth_method 0 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): get_auth_request con 0x7fd5a409b740 auth_method 0 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a409c410 0x7fd5a408bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): get_auth_request con 0x7fd5a409c410 auth_method 0 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient(hunting): handle_auth_done global_id 54813 payload 824 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5a409b740 msgr2=0x7fd5a413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5a409b740 0x7fd5a413feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a409c410 msgr2=0x7fd5a408bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a409c410 0x7fd5a408bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 monclient: found mon.noname-a 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd594009420 con 0x7fd5a409abd0 2024-09-18T10:16:26.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 msgr2=0x7fd5a413f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a259c700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 0x7fd5a413f970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00e950 tx=0x7fd58c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c013890 con 0x7fd5a409abd0 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: dump: 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.033 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c013a10 con 0x7fd5a409abd0 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 5 monclient: authenticate success, global_id 54813 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient: _renew_subs 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a408f2d0 con 0x7fd5a409abd0 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c013dd0 con 0x7fd5a409abd0 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd5a0d99700 4 set_mon_vals no callback set 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.007+0000 7fd59affd700 10 monclient: dump: 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 10 monclient: _renew_subs 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a4090500 con 0x7fd5a409abd0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 0x7fd5a408c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd58c0125e0 con 0x7fd5a409abd0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a40648a0 con 0x7fd5a408c1f0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 0x7fd5a408c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd59affd700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd584035ac0 0x7fd584037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient: get_auth_request con 0x7fd5a408c1f0 auth_method 0 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7fd58c049d50 con 0x7fd5a409abd0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a2d9d700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd584035ac0 0x7fd584037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd5a2d9d700 10 monclient: get_auth_request con 0x7fd584035ac0 auth_method 0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 10 monclient(hunting): handle_auth_done global_id 54816 payload 824 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.011+0000 7fd599d9b700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5a40a2310 con 0x7fd5a408c1f0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd599d9b700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 msgr2=0x7fd5a408c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd599d9b700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 0x7fd5a408c5c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd594009fd0 tx=0x7fd594017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59400be50 con 0x7fd5a408c1f0 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 10 monclient: dump: 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd5a2d9d700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd584035ac0 0x7fd584037f70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4090000 tx=0x7fd59c005340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd594020040 con 0x7fd5a408c1f0 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 msgr2=0x7fd5a408c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 0x7fd5a408c5c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd594009fd0 tx=0x7fd594017000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 msgr2=0x7fd5a408c5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.015+0000 7fd59affd700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408c1f0 0x7fd5a408c5c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd5a359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd5a359e700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 0x7fd5a4087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd5a359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a408f6e0 con 0x7fd5a408cb00 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 0x7fd5a4087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient: get_auth_request con 0x7fd5a408cb00 auth_method 0 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 10 monclient(hunting): handle_auth_done global_id 54819 payload 824 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.023+0000 7fd599d9b700 1 -- 172.21.15.142:0/1623495529 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd5a4072a80 con 0x7fd5a408cb00 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd599d9b700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 msgr2=0x7fd5a4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd599d9b700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 0x7fd5a4087fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd5a408e600 tx=0x7fd594021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd594038070 con 0x7fd5a408cb00 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 10 monclient: dump: 2024-09-18T10:16:26.042 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fd59400bc40 con 0x7fd5a408cb00 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:26.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 msgr2=0x7fd5a4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 0x7fd5a4087fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd5a408e600 tx=0x7fd594021000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 msgr2=0x7fd5a4087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd59affd700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a408cb00 0x7fd5a4087fe0 secure :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd5a408e600 tx=0x7fd594021000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:26.044 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.045 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.046 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.047 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.048 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:26.049 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.050 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.052 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:26.053 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.054 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.055 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.056 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:26.057 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:26.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.051+0000 7f19dced1700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.051+0000 7f19dced1700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.051+0000 7f19dced1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:26.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.051+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f19d8063ed0 con 0x7f19d80a0290 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd584035ac0 msgr2=0x7fd584037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd584035ac0 0x7fd584037f70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4090000 tx=0x7fd59c005340 comp rx=0 tx=0).stop 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 10 monclient: shutdown 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 msgr2=0x7fd5a413f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 --2- 172.21.15.142:0/1623495529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5a409abd0 0x7fd5a413f970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00e950 tx=0x7fd58c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a2d9d700 1 -- 172.21.15.142:0/1623495529 reap_dead start 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a2d9d700 1 -- 172.21.15.142:0/1623495529 reap_dead start 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 shutdown_connections 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.027+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 >> 172.21.15.142:0/1623495529 conn(0x7fd5a4070ef0 msgr2=0x7fd5a409de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.031+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 shutdown_connections 2024-09-18T10:16:26.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.031+0000 7fd5a359e700 1 -- 172.21.15.142:0/1623495529 wait complete. 2024-09-18T10:16:26.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:26.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:16:26.060 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:16:26.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.163+0000 7f88a4b2f700 10 monclient: get_monmap_and_config 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.163+0000 7f88a4b2f700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.163+0000 7f88a4b2f700 10 monclient: monmap: 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:26.171451+0000 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:26.171451+0000 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:26.173 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:26.174 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:26.174 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:26.174 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 Processor -- start 2024-09-18T10:16:26.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 -- start start 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient: init 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88a009eee0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a009f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-b con 0x7f88a009f2b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88a0137660 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00a0660 con 0x7f88a009eee0 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a00a1c40 con 0x7f88a0137660 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00a1600 con 0x7f88a009f2b0 2024-09-18T10:16:26.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f88a4b2f700 10 monclient(hunting): authenticate will time out at 1275.097068s 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f88a009eee0 0x7f88a01325c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57936/0 (socket says 172.21.15.142:57936) 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889f7fe700 1 -- 172.21.15.142:0/3489069441 learned_addr learned my addr 172.21.15.142:0/3489069441 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 1 --2- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a009f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): get_auth_request con 0x7f88a009f2b0 auth_method 0 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient(hunting): handle_auth_done global_id 74276 payload 274 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 1 -- 172.21.15.142:0/3489069441 >> v1:172.21.15.150:6789/0 conn(0x7f88a0137660 legacy=0x7f88a0139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 1 -- 172.21.15.142:0/3489069441 >> v1:172.21.15.142:6789/0 conn(0x7f88a009eee0 legacy=0x7f88a01325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 1 monclient: found mon.noname-b 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.167+0000 7f889effd700 1 -- 172.21.15.142:0/3489069441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00a1f20 con 0x7f88a009f2b0 2024-09-18T10:16:26.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889effd700 1 -- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 msgr2=0x7f88a009f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889effd700 1 --2- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a009f6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8890009e90 tx=0x7f8890009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b32378c8ef25841 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 1 -- 172.21.15.142:0/3489069441 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000e070 con 0x7f88a009f2b0 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: dump: 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 1 -- 172.21.15.142:0/3489069441 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890012430 con 0x7f88a009f2b0 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 5 monclient: authenticate success, global_id 74276 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 1 -- 172.21.15.142:0/3489069441 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900127f0 con 0x7f88a009f2b0 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 4 set_mon_vals no callback set 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f889e7fc700 10 monclient: dump: 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.188 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 10 monclient: shutdown 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 msgr2=0x7f88a009f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a009f6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8890009e90 tx=0x7f8890009780 comp rx=0 tx=0).stop 2024-09-18T10:16:26.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- 172.21.15.142:0/3489069441 shutdown_connections 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/3489069441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a009f6c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- 172.21.15.142:0/3489069441 >> 172.21.15.142:0/3489069441 conn(0x7f88a00a5700 msgr2=0x7f88a00a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- 172.21.15.142:0/3489069441 shutdown_connections 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- 172.21.15.142:0/3489069441 wait complete. 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 10 monclient: monmap: 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:26.190 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 Processor -- start 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 1 -- start start 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.171+0000 7f88a4b2f700 10 monclient: init 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88a009eee0 0x7f88a006e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-b con 0x7f88a009eee0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-c con 0x7f88a009f2b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): picked mon.noname-a con 0x7f88a0137660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:26.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00a11c0 con 0x7f88a0137660 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a009dc30 con 0x7f88a009eee0 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a006dac0 con 0x7f88a009f2b0 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient(hunting): authenticate will time out at 1275.104175s 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47276/0 (socket says 172.21.15.142:47276) 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a006aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50514/0 (socket says 172.21.15.142:50514) 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 learned_addr learned my addr 172.21.15.142:0/2229011405 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:26.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): get_auth_request con 0x7f88a0137660 auth_method 0 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): get_auth_request con 0x7f88a009f2b0 auth_method 0 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88a009eee0 0x7f88a006e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): get_auth_request con 0x7f88a009eee0 auth_method 0 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient(hunting): handle_auth_done global_id 54822 payload 824 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 msgr2=0x7f88a006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a006aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88a009eee0 msgr2=0x7f88a006e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f88a009eee0 0x7f88a006e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 monclient: found mon.noname-a 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009a20 con 0x7f88a0137660 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889effd700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a009f2b0 0x7f88a006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 msgr2=0x7f88a006af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889ffff700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f888800c5b0 tx=0x7f8888007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88880117a0 con 0x7f88a0137660 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: dump: 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 5 monclient: authenticate success, global_id 54822 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _renew_subs 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888011900 con 0x7f88a0137660 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a006cb30 con 0x7f88a0137660 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888011cc0 con 0x7f88a0137660 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88977fe700 10 monclient: dump: 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.197 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _renew_subs 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a006c040 con 0x7f88a0137660 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.175+0000 7f889dffb700 4 set_mon_vals no callback set 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88a4b2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 0x7f88a0133390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00647a0 con 0x7f88a006b4a0 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8888012630 con 0x7f88a0137660 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 0x7f88a0133390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient: get_auth_request con 0x7f88a006b4a0 auth_method 0 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 10 monclient(hunting): handle_auth_done global_id 74282 payload 824 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88a006b870 con 0x7f88a006b4a0 2024-09-18T10:16:26.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88977fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8880035af0 0x7f8880037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7f8888049600 con 0x7f88a0137660 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889effd700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8880035af0 0x7f8880037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889effd700 10 monclient: get_auth_request con 0x7f8880035af0 auth_method 0 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 msgr2=0x7f88a0133390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.179+0000 7f889f7fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 0x7f88a0133390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f889800eac0 tx=0x7f8898007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889800cc40 con 0x7f88a006b4a0 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 10 monclient: dump: 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.202 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f889800cda0 con 0x7f88a006b4a0 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 msgr2=0x7f88a0133390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 0x7f88a0133390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f889800eac0 tx=0x7f8898007f90 comp rx=0 tx=0).stop 2024-09-18T10:16:26.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 msgr2=0x7f88a0133390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f88977fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a006b4a0 0x7f88a0133390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.183+0000 7f889effd700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8880035af0 0x7f8880037fa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f8890018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f88a4b2f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 0x7f88a0133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f88a4b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a006c040 con 0x7f88a01338d0 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 0x7f88a0133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient: get_auth_request con 0x7f88a01338d0 auth_method 0 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.191+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f889f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f889f7fe700 10 monclient(hunting): handle_auth_done global_id 74285 payload 824 2024-09-18T10:16:26.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f889f7fe700 1 -- 172.21.15.142:0/2229011405 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f88a0072a80 con 0x7f88a01338d0 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f889f7fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 msgr2=0x7f88a0133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f889f7fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 0x7f88a0133cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8898007360 tx=0x7f8898016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898029410 con 0x7f88a01338d0 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 10 monclient: dump: 2024-09-18T10:16:26.206 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f889800ca30 con 0x7f88a01338d0 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:26.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 msgr2=0x7f88a0133cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 0x7f88a0133cd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8898007360 tx=0x7f8898016000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 msgr2=0x7f88a0133cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.195+0000 7f88977fe700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88a01338d0 0x7f88a0133cd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:26.208 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.209 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.210 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.211 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.212 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:26.213 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.214 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:26.215 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.216 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.217 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.218 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.219 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:26.220 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:26.221 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8880035af0 msgr2=0x7f8880037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8880035af0 0x7f8880037fa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f8890018000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 10 monclient: shutdown 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f889ffff700 1 -- 172.21.15.142:0/2229011405 reap_dead start 2024-09-18T10:16:26.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 msgr2=0x7f88a006af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f888800c5b0 tx=0x7f8888007f60 comp rx=0 tx=0).stop 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 shutdown_connections 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 --2- 172.21.15.142:0/2229011405 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88a0137660 0x7f88a006af60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 >> 172.21.15.142:0/2229011405 conn(0x7f88a00a5700 msgr2=0x7f88a00a3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 shutdown_connections 2024-09-18T10:16:26.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.199+0000 7f88a4b2f700 1 -- 172.21.15.142:0/2229011405 wait complete. 2024-09-18T10:16:26.224 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:26.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:16:26.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: get_monmap_and_config 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: monmap: 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:26.340897+0000 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:26.340897+0000 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:26.343 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 Processor -- start 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 -- start start 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: init 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafbc0999e0 0x7fafbc0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-c con 0x7fafbc0999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fafbc09c410 0x7fafbc0a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-a con 0x7fafbc09c410 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafbc0a0c00 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafbc0a20c0 con 0x7fafbc0a0c00 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc0a1040 con 0x7fafbc09c410 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafbc09a000 con 0x7fafbc0999e0 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc2246700 10 monclient(hunting): authenticate will time out at 1275.266462s 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fafbc09c410 0x7fafbc0a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fafbc09c410 0x7fafbc0a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47290/0 (socket says 172.21.15.142:47290) 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 -- 172.21.15.142:0/1573539236 learned_addr learned my addr 172.21.15.142:0/1573539236 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): get_auth_request con 0x7fafbc09c410 auth_method 0 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1a45700 1 --1- 172.21.15.142:0/1573539236 >> v1:172.21.15.142:6789/0 conn(0x7fafbc0a0c00 0x7fafbc138910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57952/0 (socket says 172.21.15.142:57952) 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 1 --2- 172.21.15.142:0/1573539236 >> v2:172.21.15.150:3300/0 conn(0x7fafbc0999e0 0x7fafbc0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): get_auth_request con 0x7fafbc0999e0 auth_method 0 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient(hunting): handle_auth_done global_id 54825 payload 274 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 -- 172.21.15.142:0/1573539236 >> v2:172.21.15.150:3300/0 conn(0x7fafbc0999e0 msgr2=0x7fafbc0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 --2- 172.21.15.142:0/1573539236 >> v2:172.21.15.150:3300/0 conn(0x7fafbc0999e0 0x7fafbc0a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 -- 172.21.15.142:0/1573539236 >> v1:172.21.15.142:6789/0 conn(0x7fafbc0a0c00 legacy=0x7fafbc138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 monclient: found mon.noname-a 2024-09-18T10:16:26.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.335+0000 7fafc0a43700 1 -- 172.21.15.142:0/1573539236 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc0a1c80 con 0x7fafbc09c410 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc0a43700 1 -- 172.21.15.142:0/1573539236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 msgr2=0x7fafbc0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc0a43700 1 --2- 172.21.15.142:0/1573539236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc0a06c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafbc069c60 tx=0x7fafb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c098f5abf5e961 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 1 -- 172.21.15.142:0/1573539236 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8009ea0 con 0x7fafbc09c410 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: dump: 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 5 monclient: authenticate success, global_id 54825 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 1 -- 172.21.15.142:0/1573539236 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb800ba30 con 0x7fafbc09c410 2024-09-18T10:16:26.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 4 set_mon_vals no callback set 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 1 -- 172.21.15.142:0/1573539236 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb800bdf0 con 0x7fafbc09c410 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafb3fff700 10 monclient: dump: 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.360 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 10 monclient: shutdown 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 -- 172.21.15.142:0/1573539236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 msgr2=0x7fafbc0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 --2- 172.21.15.142:0/1573539236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc0a06c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafbc069c60 tx=0x7fafb8009250 comp rx=0 tx=0).stop 2024-09-18T10:16:26.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 -- 172.21.15.142:0/1573539236 shutdown_connections 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 --2- 172.21.15.142:0/1573539236 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc0a06c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 --2- 172.21.15.142:0/1573539236 >> v2:172.21.15.150:3300/0 conn(0x7fafbc0999e0 0x7fafbc0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 -- 172.21.15.142:0/1573539236 >> 172.21.15.142:0/1573539236 conn(0x7fafbc0a5790 msgr2=0x7fafbc0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 -- 172.21.15.142:0/1573539236 shutdown_connections 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 1 -- 172.21.15.142:0/1573539236 wait complete. 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.339+0000 7fafc2246700 10 monclient: monmap: 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:26.362 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 Processor -- start 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 -- start start 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient: init 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafbc0999e0 0x7fafbc08e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-c con 0x7fafbc0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc08e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-a con 0x7fafbc09c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc0a0c00 0x7fafbc08eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): picked mon.noname-b con 0x7fafbc0a0c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0a4dc0 con 0x7fafbc09c410 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc092630 con 0x7fafbc0a0c00 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc08fda0 con 0x7fafbc0999e0 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc2246700 10 monclient(hunting): authenticate will time out at 1275.273696s 2024-09-18T10:16:26.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc08e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafbc0999e0 0x7fafbc08e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafbc0999e0 0x7fafbc08e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50544/0 (socket says 172.21.15.142:50544) 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc08e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47304/0 (socket says 172.21.15.142:47304) 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 1 -- 172.21.15.142:0/1821609897 learned_addr learned my addr 172.21.15.142:0/1821609897 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): get_auth_request con 0x7fafbc0999e0 auth_method 0 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): get_auth_request con 0x7fafbc09c410 auth_method 0 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1a45700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc0a0c00 0x7fafbc08eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1a45700 10 monclient(hunting): get_auth_request con 0x7fafbc0a0c00 auth_method 0 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1a45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1a45700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc1a45700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient(hunting): handle_auth_done global_id 54828 payload 824 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafbc0999e0 msgr2=0x7fafbc08e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafbc0999e0 0x7fafbc08e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc0a0c00 msgr2=0x7fafbc08eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc0a0c00 0x7fafbc08eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 monclient: found mon.noname-a 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.343+0000 7fafc0a43700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb80094f0 con 0x7fafbc09c410 2024-09-18T10:16:26.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc0a43700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 msgr2=0x7fafbc08e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc0a43700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc08e560 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fafbc09fda0 tx=0x7fafb8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80119a0 con 0x7fafbc09c410 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: dump: 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.368 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 5 monclient: authenticate success, global_id 54828 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 monclient: _renew_subs 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8020070 con 0x7fafbc09c410 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc0913c0 con 0x7fafbc09c410 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8011db0 con 0x7fafbc09c410 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb37fe700 4 set_mon_vals no callback set 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 10 monclient: dump: 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.370 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.372 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.372 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.373 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 monclient: _renew_subs 2024-09-18T10:16:26.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc090460 con 0x7fafbc09c410 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 0x7fafbc02ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc0647f0 con 0x7fafbc08efe0 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fafb8022030 con 0x7fafbc09c410 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 0x7fafbc02ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient: get_auth_request con 0x7fafbc08efe0 auth_method 0 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fafa0035ac0 0x7fafa0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7fafb804bac0 con 0x7fafbc09c410 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 10 monclient(hunting): handle_auth_done global_id 84127 payload 824 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1244700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafbc06e220 con 0x7fafbc08efe0 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1a45700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fafa0035ac0 0x7fafa0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.347+0000 7fafc1a45700 10 monclient: get_auth_request con 0x7fafa0035ac0 auth_method 0 2024-09-18T10:16:26.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafc1244700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 msgr2=0x7fafbc02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafc1244700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 0x7fafbc02ae80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafac011bf0 tx=0x7fafac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac014440 con 0x7fafbc08efe0 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 10 monclient: dump: 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.378 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafc1a45700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fafa0035ac0 0x7fafa0037f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fafbc091de0 tx=0x7fafb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fafac0145a0 con 0x7fafbc08efe0 2024-09-18T10:16:26.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 msgr2=0x7fafbc02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 0x7fafbc02ae80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafac011bf0 tx=0x7fafac0091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 msgr2=0x7fafbc02ae80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.351+0000 7fafb17fa700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc08efe0 0x7fafbc02ae80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc2246700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc2246700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 0x7fafbc05ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc2246700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc072a80 con 0x7fafbc05e890 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 0x7fafbc05ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient: get_auth_request con 0x7fafbc05e890 auth_method 0 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 10 monclient(hunting): handle_auth_done global_id 84130 payload 824 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.363+0000 7fafc1244700 1 -- 172.21.15.142:0/1821609897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fafbc0918a0 con 0x7fafbc05e890 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc1244700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 msgr2=0x7fafbc05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc1244700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 0x7fafbc05ec90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafac011bf0 tx=0x7fafac019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac01de70 con 0x7fafbc05e890 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 10 monclient: dump: 2024-09-18T10:16:26.381 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fafac00fd40 con 0x7fafbc05e890 2024-09-18T10:16:26.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:26.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:26.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 msgr2=0x7fafbc05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 0x7fafbc05ec90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafac011bf0 tx=0x7fafac019000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 msgr2=0x7fafbc05ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:26.384 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.385 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:26.387 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:26.388 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:26.389 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.390 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.391 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:26.392 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.392 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.392 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.392 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.393 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.395 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.395 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.395 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.395 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:26.395 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:26.396 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:26.397 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.398 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:26.399 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:26.400 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafb17fa700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafbc05e890 0x7fafbc05ec90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fafa0035ac0 msgr2=0x7fafa0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fafa0035ac0 0x7fafa0037f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fafbc091de0 tx=0x7fafb4009040 comp rx=0 tx=0).stop 2024-09-18T10:16:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 10 monclient: shutdown 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 msgr2=0x7fafbc08e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 --2- 172.21.15.142:0/1821609897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafbc09c410 0x7fafbc08e560 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fafbc09fda0 tx=0x7fafb8016000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc1a45700 1 -- 172.21.15.142:0/1821609897 reap_dead start 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc1a45700 1 -- 172.21.15.142:0/1821609897 reap_dead start 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 shutdown_connections 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 >> 172.21.15.142:0/1821609897 conn(0x7fafbc0a5790 msgr2=0x7fafbc09dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 shutdown_connections 2024-09-18T10:16:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.367+0000 7fafc2246700 1 -- 172.21.15.142:0/1821609897 wait complete. 2024-09-18T10:16:26.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:26.404 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:16:26.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:16:26.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:16:26.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: monmap: 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:26.518269+0000 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:26.518269+0000 2024-09-18T10:16:26.520 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:26.521 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:26.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 Processor -- start 2024-09-18T10:16:26.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 -- start start 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: init 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 0x7fe73c098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fe73c09a7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe73c099220 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe73c09c4f0 0x7fe73c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fe73c09c4f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73c09abc0 con 0x7fe73c099220 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73c09d950 con 0x7fe73c09a7f0 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73c09db90 con 0x7fe73c09c4f0 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.511+0000 7fe73bfff700 10 monclient(hunting): authenticate will time out at 1275.443878s 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 0x7fe73c098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73a7fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe73c099220 0x7fe73c09bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57962/0 (socket says 172.21.15.142:57962) 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73a7fc700 1 -- 172.21.15.142:0/2015234860 learned_addr learned my addr 172.21.15.142:0/2015234860 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): get_auth_request con 0x7fe73c09a7f0 auth_method 0 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 --2- 172.21.15.142:0/2015234860 >> v2:172.21.15.150:3300/0 conn(0x7fe73c09c4f0 0x7fe73c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): get_auth_request con 0x7fe73c09c4f0 auth_method 0 2024-09-18T10:16:26.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_done global_id 84133 payload 274 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 -- 172.21.15.142:0/2015234860 >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 msgr2=0x7fe73c098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 --2- 172.21.15.142:0/2015234860 >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 0x7fe73c098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 -- 172.21.15.142:0/2015234860 >> v1:172.21.15.142:6789/0 conn(0x7fe73c099220 legacy=0x7fe73c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:26.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 -- 172.21.15.142:0/2015234860 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c099a30 con 0x7fe73c09c4f0 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73affd700 1 --2- 172.21.15.142:0/2015234860 >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 0x7fe73c098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 -- 172.21.15.142:0/2015234860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c09c4f0 msgr2=0x7fe73c13e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe73b7fe700 1 --2- 172.21.15.142:0/2015234860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c09c4f0 0x7fe73c13e610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe73c06fc80 tx=0x7fe7340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ade6c12959953b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe739ffb700 1 -- 172.21.15.142:0/2015234860 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734011840 con 0x7fe73c09c4f0 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe739ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.515+0000 7fe739ffb700 10 monclient: dump: 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.540 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 5 monclient: authenticate success, global_id 84133 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 1 -- 172.21.15.142:0/2015234860 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7340119a0 con 0x7fe73c09c4f0 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:26.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 4 set_mon_vals no callback set 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 1 -- 172.21.15.142:0/2015234860 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734011d40 con 0x7fe73c09c4f0 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe739ffb700 10 monclient: dump: 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.542 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: shutdown 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- 172.21.15.142:0/2015234860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c09c4f0 msgr2=0x7fe73c13e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- 172.21.15.142:0/2015234860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c09c4f0 0x7fe73c13e610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe73c06fc80 tx=0x7fe7340091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- 172.21.15.142:0/2015234860 shutdown_connections 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- 172.21.15.142:0/2015234860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c09c4f0 0x7fe73c13e610 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- 172.21.15.142:0/2015234860 >> v2:172.21.15.142:3300/0 conn(0x7fe73c09a7f0 0x7fe73c098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- 172.21.15.142:0/2015234860 >> 172.21.15.142:0/2015234860 conn(0x7fe73c0a1fe0 msgr2=0x7fe73c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- 172.21.15.142:0/2015234860 shutdown_connections 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- 172.21.15.142:0/2015234860 wait complete. 2024-09-18T10:16:26.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: monmap: 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 Processor -- start 2024-09-18T10:16:26.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- start start 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: init 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c099220 0x7fe73c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fe73c099220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fe73c09a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fe73c09c4f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c06f970 con 0x7fe73c09c4f0 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c06e7f0 con 0x7fe73c099220 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c126880 con 0x7fe73c09a7f0 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73bfff700 10 monclient(hunting): authenticate will time out at 1275.451212s 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50558/0 (socket says 172.21.15.142:50558) 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 1 -- 172.21.15.142:0/381719599 learned_addr learned my addr 172.21.15.142:0/381719599 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47322/0 (socket says 172.21.15.142:47322) 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): get_auth_request con 0x7fe73c09a7f0 auth_method 0 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): get_auth_request con 0x7fe73c09c4f0 auth_method 0 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c099220 0x7fe73c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): get_auth_request con 0x7fe73c099220 auth_method 0 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:26.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.519+0000 7fe73affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 10 monclient(hunting): handle_auth_done global_id 74291 payload 824 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 -- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c099220 msgr2=0x7fe73c145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c099220 0x7fe73c145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 -- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 msgr2=0x7fe73c06c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 monclient: found mon.noname-c 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 -- 172.21.15.142:0/381719599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009420 con 0x7fe73c09a7f0 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73b7fe700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 -- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 msgr2=0x7fe73c138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73a7fc700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00cca0 tx=0x7fe72c005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00ab20 con 0x7fe73c09a7f0 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: dump: 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.549 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 5 monclient: authenticate success, global_id 74291 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 10 monclient: _renew_subs 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe734009420 con 0x7fe73c09a7f0 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00aca0 con 0x7fe73c09a7f0 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe738ff9700 4 set_mon_vals no callback set 2024-09-18T10:16:26.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 10 monclient: _renew_subs 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c126e40 con 0x7fe73c09a7f0 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c007340 con 0x7fe73c09a7f0 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.523+0000 7fe732ffd700 10 monclient: dump: 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:26.551 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe72c0076f0 con 0x7fe73c09a7f0 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe732ffd700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 0x7fe71c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(203..203 src has 176..203) v4 ==== 10966+0+0 (secure 0 0 0) 0x7fe72c04a150 con 0x7fe73c09a7f0 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73affd700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 0x7fe71c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73affd700 10 monclient: get_auth_request con 0x7fe71c035ac0 auth_method 0 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c06db40 con 0x7fe73c09a7f0 2024-09-18T10:16:26.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.527+0000 7fe73affd700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 0x7fe71c037f70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe724009f40 tx=0x7fe7240091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:26.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.535+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe72c01a360 con 0x7fe73c09a7f0 2024-09-18T10:16:26.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.535+0000 7fe732ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:26.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.535+0000 7fe732ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:26.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.755+0000 7fe73bfff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:16:26.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.755+0000 7fe73bfff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:16:26.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.755+0000 7fe73bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:26.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.755+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe73c072a80 con 0x7fe73c09a7f0 2024-09-18T10:16:26.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.763+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 added bucket h15 type host to location {rack=r1} v204) v1 ==== 168+0+0 (secure 0 0 0) 0x7f19cc015070 con 0x7f19d80a0290 2024-09-18T10:16:26.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.763+0000 7f19c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.763+0000 7f19c67fc700 10 monclient: _finish_command 2 = system:0 added bucket h15 type host to location {rack=r1} 2024-09-18T10:16:26.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe732ffd700 1 -- 172.21.15.142:0/381719599 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fe72c010af0 con 0x7fe73c09a7f0 2024-09-18T10:16:26.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe732ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:16:26.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe732ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:26.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 msgr2=0x7fe71c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 0x7fe71c037f70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe724009f40 tx=0x7fe7240091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 10 monclient: shutdown 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 msgr2=0x7fe73c138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.759+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00cca0 tx=0x7fe72c005480 comp rx=0 tx=0).stop 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 shutdown_connections 2024-09-18T10:16:26.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe71c035ac0 0x7fe71c037f70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe73c09c4f0 0x7fe73c06c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe73c09a7f0 0x7fe73c138750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 --2- 172.21.15.142:0/381719599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe73c099220 0x7fe73c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 >> 172.21.15.142:0/381719599 conn(0x7fe73c0a1fe0 msgr2=0x7fe73c09bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 shutdown_connections 2024-09-18T10:16:26.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:26.763+0000 7fe73bfff700 1 -- 172.21.15.142:0/381719599 wait complete. 2024-09-18T10:16:26.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:16:26.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.871+0000 7f19dced1700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.871+0000 7f19dced1700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.871+0000 7f19dced1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:26.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.871+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f19d812ff70 con 0x7f19d80a0290 2024-09-18T10:16:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=12,kv=7,logm=66} crc {auth=178536557,config=1228700967,health=1555756363,kv=2693826500,logm=41881909}) 2024-09-18T10:16:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=351051815,config=3140514009,health=1555756363,kv=4261215878,logm=1747707336}) 2024-09-18T10:16:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=12,kv=7,logm=66} crc {auth=178536557,config=1228700967,health=1555756363,kv=2693826500,logm=41881909}) 2024-09-18T10:16:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:26.867+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=1,health=11,kv=8,logm=66} crc {auth=1652297357,config=1228700967,health=1051828215,kv=4261215878,logm=144126824}) 2024-09-18T10:16:26.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19c67fc700 1 -- 172.21.15.150:0/2373090691 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h15' already exists v204) v1 ==== 147+0+0 (secure 0 0 0) 0x7f19cc00d290 con 0x7f19d80a0290 2024-09-18T10:16:26.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19c67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h15", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:26.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19c67fc700 10 monclient: _finish_command 3 = system:0 bucket 'h15' already exists 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 msgr2=0x7f19b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 0x7f19b8037f70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f19c80099a0 tx=0x7f19c8018000 comp rx=0 tx=0).stop 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 10 monclient: shutdown 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 msgr2=0x7f19d81420c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00b5e0 tx=0x7f19cc009120 comp rx=0 tx=0).stop 2024-09-18T10:16:26.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 shutdown_connections 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f19b8035ac0 0x7f19b8037f70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19d8137a20 0x7f19d812c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19d80a0d70 0x7f19d8142600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 --2- 172.21.15.150:0/2373090691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d80a0290 0x7f19d81420c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 >> 172.21.15.150:0/2373090691 conn(0x7f19d80a58a0 msgr2=0x7f19d809ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 shutdown_connections 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.875+0000 7f19dced1700 1 -- 172.21.15.150:0/2373090691 wait complete. 2024-09-18T10:16:26.881 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h15' already exists 2024-09-18T10:16:26.891 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:26.891 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h16 host rack=r1 2024-09-18T10:16:26.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 10 monclient: get_monmap_and_config 2024-09-18T10:16:26.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 10 monclient: build_initial_monmap 2024-09-18T10:16:26.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 10 monclient: monmap: 2024-09-18T10:16:26.995 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:26.996 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:26.994145+0000 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:26.994145+0000 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:26.997 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:26.998 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:26.998 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:26.998 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:26.998 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 1 Processor -- start 2024-09-18T10:16:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 1 -- start start 2024-09-18T10:16:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.991+0000 7f2784a9b700 10 monclient: init 2024-09-18T10:16:27.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27800989b0 0x7f278009bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-c con 0x7f27800989b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 0x7f278009c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-b con 0x7f278009bff0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f278009ce00 0x7f278009d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-a con 0x7f278009ce00 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2780098fd0 con 0x7f278009ce00 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278009c3f0 con 0x7f27800989b0 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2780096e80 con 0x7f278009bff0 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2784a9b700 10 monclient(hunting): authenticate will time out at 1269.089258s 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27800989b0 0x7f278009bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27800989b0 0x7f278009bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60986/0 (socket says 172.21.15.150:60986) 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 -- 172.21.15.150:0/912382800 learned_addr learned my addr 172.21.15.150:0/912382800 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): get_auth_request con 0x7f27800989b0 auth_method 0 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 1 --2- 172.21.15.150:0/912382800 >> v2:172.21.15.142:3300/0 conn(0x7f278009ce00 0x7f278009d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 0x7f278009c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): get_auth_request con 0x7f278009ce00 auth_method 0 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 10 monclient(hunting): get_auth_request con 0x7f278009bff0 auth_method 0 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient(hunting): handle_auth_done global_id 84139 payload 274 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 -- 172.21.15.150:0/912382800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 msgr2=0x7f278009c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 0x7f278009c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 -- 172.21.15.150:0/912382800 >> v2:172.21.15.142:3300/0 conn(0x7f278009ce00 msgr2=0x7f278009d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 --2- 172.21.15.150:0/912382800 >> v2:172.21.15.142:3300/0 conn(0x7f278009ce00 0x7f278009d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 monclient: found mon.noname-c 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f277e59c700 1 -- 172.21.15.150:0/912382800 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780096a70 con 0x7f27800989b0 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.995+0000 7f2777fff700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 0x7f278009c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277e59c700 1 -- 172.21.15.150:0/912382800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27800989b0 msgr2=0x7f278009bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277e59c700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27800989b0 0x7f278009bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f278004cf90 tx=0x7f277000b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=201d15a82a8175f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 1 -- 172.21.15.150:0/912382800 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000dc60 con 0x7f27800989b0 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.015 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: dump: 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.016 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 5 monclient: authenticate success, global_id 84139 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 1 -- 172.21.15.150:0/912382800 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277000ddc0 con 0x7f27800989b0 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 4 set_mon_vals no callback set 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 1 -- 172.21.15.150:0/912382800 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770017670 con 0x7f27800989b0 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:27.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f277dd9b700 10 monclient: dump: 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.018 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 10 monclient: shutdown 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- 172.21.15.150:0/912382800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27800989b0 msgr2=0x7f278009bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27800989b0 0x7f278009bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f278004cf90 tx=0x7f277000b4e0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- 172.21.15.150:0/912382800 shutdown_connections 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 --2- 172.21.15.150:0/912382800 >> v2:172.21.15.142:3300/0 conn(0x7f278009ce00 0x7f278009d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009bff0 0x7f278009c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 --2- 172.21.15.150:0/912382800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27800989b0 0x7f278009bab0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- 172.21.15.150:0/912382800 >> 172.21.15.150:0/912382800 conn(0x7f2780068730 msgr2=0x7f27800a1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- 172.21.15.150:0/912382800 shutdown_connections 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- 172.21.15.150:0/912382800 wait complete. 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 10 monclient: build_initial_monmap 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 10 monclient: monmap: 2024-09-18T10:16:27.019 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 Processor -- start 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 1 -- start start 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:26.999+0000 7f2784a9b700 10 monclient: init 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 0x7f2780139e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-a con 0x7f27800989b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-b con 0x7f278009bff0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:27.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009ce00 0x7f278013d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): picked mon.noname-c con 0x7f278009ce00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278011ee10 con 0x7f27800989b0 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278011e850 con 0x7f278009bff0 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278011c330 con 0x7f278009ce00 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2784a9b700 10 monclient(hunting): authenticate will time out at 1269.096599s 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32770/0 (socket says 172.21.15.150:32770) 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 -- 172.21.15.150:0/2154255408 learned_addr learned my addr 172.21.15.150:0/2154255408 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): get_auth_request con 0x7f278009bff0 auth_method 0 2024-09-18T10:16:27.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 0x7f2780139e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): get_auth_request con 0x7f27800989b0 auth_method 0 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009ce00 0x7f278013d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): get_auth_request con 0x7f278009ce00 auth_method 0 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient(hunting): handle_auth_done global_id 84142 payload 824 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 -- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009ce00 msgr2=0x7f278013d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009ce00 0x7f278013d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 -- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 msgr2=0x7f2780139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 0x7f2780139e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 monclient: found mon.noname-b 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277000b780 con 0x7f278009bff0 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f277e59c700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 0x7f2780139e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 -- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 msgr2=0x7f27801394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f2777fff700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f276800e8d0 tx=0x7f2768008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.003+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27680093b0 con 0x7f278009bff0 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: dump: 2024-09-18T10:16:27.026 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 5 monclient: authenticate success, global_id 84142 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2768013620 con 0x7f278009bff0 2024-09-18T10:16:27.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27680139c0 con 0x7f278009bff0 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 10 monclient: dump: 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.028 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _renew_subs 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f277d59a700 4 set_mon_vals no callback set 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2780097970 con 0x7f278009bff0 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _renew_subs 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2780139a10 con 0x7f278009bff0 2024-09-18T10:16:27.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2768012370 con 0x7f278009bff0 2024-09-18T10:16:27.030 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2780063ed0 con 0x7f278009bff0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 0x7f2760037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(204..204 src has 176..204) v4 ==== 11179+0+0 (secure 0 0 0) 0x7f2768049990 con 0x7f278009bff0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f277e59c700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 0x7f2760037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.007+0000 7f277e59c700 10 monclient: get_auth_request con 0x7f2760035ac0 auth_method 0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.011+0000 7f277e59c700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 0x7f2760037f70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f277000bbc0 tx=0x7f277001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.015+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2768012d00 con 0x7f278009bff0 2024-09-18T10:16:27.031 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.015+0000 7f27767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.015+0000 7f27767fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:27.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.183+0000 7f2784a9b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.183+0000 7f2784a9b700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.183+0000 7f2784a9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.183+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f278011e5d0 con 0x7f278009bff0 2024-09-18T10:16:27.553 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.553 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051879028}) 2024-09-18T10:16:27.553 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133240810}) 2024-09-18T10:16:27.554 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.554 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051879028}) 2024-09-18T10:16:27.554 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.547+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169595515}) 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972080019}) 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134904387}) 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972080019}) 2024-09-18T10:16:27.558 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.551+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638053379}) 2024-09-18T10:16:27.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180432166}) 2024-09-18T10:16:27.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266868169}) 2024-09-18T10:16:27.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180432166}) 2024-09-18T10:16:27.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.559+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737970500}) 2024-09-18T10:16:27.569 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334380954}) 2024-09-18T10:16:27.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507746138}) 2024-09-18T10:16:27.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334380954}) 2024-09-18T10:16:27.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499284100}) 2024-09-18T10:16:27.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854855024}) 2024-09-18T10:16:27.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101992269}) 2024-09-18T10:16:27.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.572 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854855024}) 2024-09-18T10:16:27.572 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.563+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255925688}) 2024-09-18T10:16:27.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213173674}) 2024-09-18T10:16:27.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717613283}) 2024-09-18T10:16:27.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213173674}) 2024-09-18T10:16:27.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041726832}) 2024-09-18T10:16:27.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684946649}) 2024-09-18T10:16:27.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3614392012}) 2024-09-18T10:16:27.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684946649}) 2024-09-18T10:16:27.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.575+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124281927}) 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227829382}) 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1431331498}) 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227829382}) 2024-09-18T10:16:27.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507253507}) 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842523725}) 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065752912}) 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842523725}) 2024-09-18T10:16:27.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114840263}) 2024-09-18T10:16:27.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216930377}) 2024-09-18T10:16:27.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2424124587}) 2024-09-18T10:16:27.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2216930377}) 2024-09-18T10:16:27.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.579+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592474632}) 2024-09-18T10:16:27.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051058588}) 2024-09-18T10:16:27.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449393833}) 2024-09-18T10:16:27.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2051058588}) 2024-09-18T10:16:27.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237504698}) 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3992195654}) 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157669086}) 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3992195654}) 2024-09-18T10:16:27.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.583+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309726336}) 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310519216}) 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188590359}) 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=310519216}) 2024-09-18T10:16:27.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3655187236}) 2024-09-18T10:16:27.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353640953}) 2024-09-18T10:16:27.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996089069}) 2024-09-18T10:16:27.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353640953}) 2024-09-18T10:16:27.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535481756}) 2024-09-18T10:16:27.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767870592}) 2024-09-18T10:16:27.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470952279}) 2024-09-18T10:16:27.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3767870592}) 2024-09-18T10:16:27.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.587+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061481288}) 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983814215}) 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688382148}) 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983814215}) 2024-09-18T10:16:27.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881539098}) 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957661198}) 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414877109}) 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957661198}) 2024-09-18T10:16:27.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613090882}) 2024-09-18T10:16:27.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843167507}) 2024-09-18T10:16:27.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939479009}) 2024-09-18T10:16:27.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843167507}) 2024-09-18T10:16:27.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.591+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955454140}) 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359103136}) 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396471146}) 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359103136}) 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3872982391}) 2024-09-18T10:16:27.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936311726}) 2024-09-18T10:16:27.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1171470764}) 2024-09-18T10:16:27.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936311726}) 2024-09-18T10:16:27.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429911623}) 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940856295}) 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3810294971}) 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940856295}) 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.595+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481368813}) 2024-09-18T10:16:27.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420520877}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1842610873}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420520877}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=41005459}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986381105}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080708274}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=986381105}) 2024-09-18T10:16:27.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.599+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902067165}) 2024-09-18T10:16:27.655 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.655 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340956099}) 2024-09-18T10:16:27.655 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443697040}) 2024-09-18T10:16:27.655 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.655 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340956099}) 2024-09-18T10:16:27.656 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.647+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522382320}) 2024-09-18T10:16:27.662 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.662 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982947165}) 2024-09-18T10:16:27.662 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062663330}) 2024-09-18T10:16:27.663 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.663 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982947165}) 2024-09-18T10:16:27.663 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.655+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=17949383}) 2024-09-18T10:16:27.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271798962}) 2024-09-18T10:16:27.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4231283745}) 2024-09-18T10:16:27.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271798962}) 2024-09-18T10:16:27.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.671+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554160815}) 2024-09-18T10:16:27.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.675+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 added bucket h16 type host to location {rack=r1} v205) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2768020070 con 0x7f278009bff0 2024-09-18T10:16:27.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.675+0000 7f27767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.675+0000 7f27767fc700 10 monclient: _finish_command 2 = system:0 added bucket h16 type host to location {rack=r1} 2024-09-18T10:16:27.783 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:16:27.784 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:27.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.783+0000 7f2784a9b700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.783+0000 7f2784a9b700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.783+0000 7f2784a9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.783+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f27800008d0 con 0x7f278009bff0 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085621924}) 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878073268}) 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085621924}) 2024-09-18T10:16:27.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.779+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152578455}) 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547614504}) 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878730851}) 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547614504}) 2024-09-18T10:16:27.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1691436386}) 2024-09-18T10:16:27.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f27767fc700 1 -- 172.21.15.150:0/2154255408 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h16' already exists v205) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2768010320 con 0x7f278009bff0 2024-09-18T10:16:27.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f27767fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h16", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:27.799 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f27767fc700 10 monclient: _finish_command 3 = system:0 bucket 'h16' already exists 2024-09-18T10:16:27.800 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.800 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332262720}) 2024-09-18T10:16:27.800 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1609205489}) 2024-09-18T10:16:27.800 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.801 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3332262720}) 2024-09-18T10:16:27.801 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.791+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003876053}) 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 msgr2=0x7f2760037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 0x7f2760037f70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f277000bbc0 tx=0x7f277001a000 comp rx=0 tx=0).stop 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 10 monclient: shutdown 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 msgr2=0x7f27801394d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.795+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f276800e8d0 tx=0x7f2768008040 comp rx=0 tx=0).stop 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 shutdown_connections 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2760035ac0 0x7f2760037f70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f278009ce00 0x7f278013d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278009bff0 0x7f27801394d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 --2- 172.21.15.150:0/2154255408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27800989b0 0x7f2780139e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 >> 172.21.15.150:0/2154255408 conn(0x7f2780068730 msgr2=0x7f278009fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:27.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 shutdown_connections 2024-09-18T10:16:27.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.799+0000 7f2784a9b700 1 -- 172.21.15.150:0/2154255408 wait complete. 2024-09-18T10:16:27.803 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h16' already exists 2024-09-18T10:16:27.803 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.804 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291185560}) 2024-09-18T10:16:27.804 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502819357}) 2024-09-18T10:16:27.804 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.804 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291185560}) 2024-09-18T10:16:27.804 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=203442693}) 2024-09-18T10:16:27.805 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.805 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783722497}) 2024-09-18T10:16:27.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=913509496}) 2024-09-18T10:16:27.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783722497}) 2024-09-18T10:16:27.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.795+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939780440}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294315612}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455925718}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=294315612}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320730481}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924081016}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693132794}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924081016}) 2024-09-18T10:16:27.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.799+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174495603}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174363902}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030898427}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174363902}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059997827}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665316530}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2891176620}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665316530}) 2024-09-18T10:16:27.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110528920}) 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464958094}) 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2536100638}) 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464958094}) 2024-09-18T10:16:27.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.803+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278004254}) 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=1811288124,mdsmap=1531151339,mgr=273474444,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=4060897197,mon_config_key=3383142877,monmap=3505902565,osd_metadata=361655855,osd_pg_creating=3127103657,osd_snap=454132730,osdmap=3350915012}) 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=232021386,mdsmap=1531151339,mgr=3316658068,mgr_metadata=2738435100,mgrstat=2513860653,mon_config_key=620546692,monmap=3505902565,osd_metadata=3738634618,osd_pg_creating=3127103657,osd_snap=454132730,osdmap=3579365995}) 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=36} crc {logm=1811288124,mdsmap=1531151339,mgr=273474444,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=4060897197,mon_config_key=3383142877,monmap=3505902565,osd_metadata=361655855,osd_pg_creating=3127103657,osd_snap=454132730,osdmap=3350915012}) 2024-09-18T10:16:27.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.807+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=25,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=39} crc {logm=1916556955,mdsmap=1531151339,mgr=3895449980,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3526570239,mon_config_key=620546692,monmap=4028308432,osd_metadata=3738634618,osd_pg_creating=3127103657,osd_snap=454132730,osdmap=176629798}) 2024-09-18T10:16:27.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:27.817 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h17 host rack=r1 2024-09-18T10:16:27.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=2534328478}) 2024-09-18T10:16:27.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=3553356028}) 2024-09-18T10:16:27.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:16:27.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=2534328478}) 2024-09-18T10:16:27.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:16:27.811+0000 7fc32aa98700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=1998935509}) 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.883+0000 7fdfd170e700 10 monclient: get_monmap_and_config 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.883+0000 7fdfd170e700 10 monclient: build_initial_monmap 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.883+0000 7fdfd170e700 10 monclient: monmap: 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:27.891552+0000 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:27.891552+0000 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.893 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:27.894 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:27.894 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:27.894 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:27.894 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:27.894 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 Processor -- start 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 -- start start 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient: init 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfcc0a2670 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 0x7fdfcc097160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc06f2d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfcc1344a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc09de90 con 0x7fdfcc1344a0 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc09b910 con 0x7fdfcc0a2670 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc098630 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd170e700 10 monclient(hunting): authenticate will time out at 1276.818399s 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 0x7fdfcc097160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd0f0d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdfcc1344a0 0x7fdfcc136880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57964/0 (socket says 172.21.15.142:57964) 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/4175400800 learned_addr learned my addr 172.21.15.142:0/4175400800 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): get_auth_request con 0x7fdfcc06f2d0 auth_method 0 2024-09-18T10:16:27.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_done global_id 74300 payload 274 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 1 -- 172.21.15.142:0/4175400800 >> v1:172.21.15.150:6789/0 conn(0x7fdfcc0a2670 legacy=0x7fdfcc096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 1 -- 172.21.15.142:0/4175400800 >> v1:172.21.15.142:6789/0 conn(0x7fdfcc1344a0 legacy=0x7fdfcc136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 1 monclient: found mon.noname-b 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.887+0000 7fdfc3fff700 1 -- 172.21.15.142:0/4175400800 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0a2a40 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfc3fff700 1 -- 172.21.15.142:0/4175400800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 msgr2=0x7fdfcc097160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfc3fff700 1 --2- 172.21.15.142:0/4175400800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 0x7fdfcc097160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009910 tx=0x7fdfb40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b0b501efb465989 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 1 -- 172.21.15.142:0/4175400800 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb400ba70 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: dump: 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.900 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 5 monclient: authenticate success, global_id 74300 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 1 -- 172.21.15.142:0/4175400800 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb400bbd0 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 4 set_mon_vals no callback set 2024-09-18T10:16:27.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 1 -- 172.21.15.142:0/4175400800 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4011420 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfcb7fe700 10 monclient: dump: 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 10 monclient: shutdown 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 -- 172.21.15.142:0/4175400800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 msgr2=0x7fdfcc097160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 --2- 172.21.15.142:0/4175400800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 0x7fdfcc097160 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009910 tx=0x7fdfb40091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 -- 172.21.15.142:0/4175400800 shutdown_connections 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 --2- 172.21.15.142:0/4175400800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc06f2d0 0x7fdfcc097160 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 -- 172.21.15.142:0/4175400800 >> 172.21.15.142:0/4175400800 conn(0x7fdfcc071700 msgr2=0x7fdfcc0a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 -- 172.21.15.142:0/4175400800 shutdown_connections 2024-09-18T10:16:27.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 1 -- 172.21.15.142:0/4175400800 wait complete. 2024-09-18T10:16:27.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 10 monclient: build_initial_monmap 2024-09-18T10:16:27.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.891+0000 7fdfd170e700 10 monclient: monmap: 2024-09-18T10:16:27.906 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:27.906 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:27.906 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 Processor -- start 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 -- start start 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient: init 2024-09-18T10:16:27.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc06f2d0 0x7fdfcc1317b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc06f2d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfcc0a2670 0x7fdfcc11d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc0a2670 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): picked mon.noname-c con 0x7fdfcc1344a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc097ec0 con 0x7fdfcc06f2d0 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0a17c0 con 0x7fdfcc0a2670 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc120980 con 0x7fdfcc1344a0 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd170e700 10 monclient(hunting): authenticate will time out at 1276.825023s 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc06f2d0 0x7fdfcc1317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50578/0 (socket says 172.21.15.142:50578) 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/3147654721 learned_addr learned my addr 172.21.15.142:0/3147654721 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): get_auth_request con 0x7fdfcc1344a0 auth_method 0 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): get_auth_request con 0x7fdfcc06f2d0 auth_method 0 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfcc0a2670 0x7fdfcc11d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): get_auth_request con 0x7fdfcc0a2670 auth_method 0 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient(hunting): handle_auth_done global_id 74303 payload 824 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfcc0a2670 msgr2=0x7fdfcc11d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfcc0a2670 0x7fdfcc11d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc06f2d0 msgr2=0x7fdfcc1317b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc06f2d0 0x7fdfcc1317b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 monclient: found mon.noname-c 2024-09-18T10:16:27.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.895+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/3147654721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb4009420 con 0x7fdfcc1344a0 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd0f0d700 1 -- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 msgr2=0x7fdfcc11d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd0f0d700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400ccf0 tx=0x7fdfc4005400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc400aa80 con 0x7fdfcc1344a0 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: dump: 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.912 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 5 monclient: authenticate success, global_id 74303 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400ac00 con 0x7fdfcc1344a0 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _renew_subs 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc134d40 con 0x7fdfcc1344a0 2024-09-18T10:16:27.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40070f0 con 0x7fdfcc1344a0 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 10 monclient: dump: 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:27.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfca7fc700 4 set_mon_vals no callback set 2024-09-18T10:16:27.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _renew_subs 2024-09-18T10:16:27.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc11f990 con 0x7fdfcc1344a0 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdfc4007460 con 0x7fdfcc1344a0 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc072a80 con 0x7fdfcc1344a0 2024-09-18T10:16:27.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 0x7fdfb0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fdfc404a240 con 0x7fdfcc1344a0 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfcbfff700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 0x7fdfb0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.899+0000 7fdfcbfff700 10 monclient: get_auth_request con 0x7fdfb0035ac0 auth_method 0 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.903+0000 7fdfcbfff700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 0x7fdfb0037f70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc003a30 tx=0x7fdfbc007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.907+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdfc401a360 con 0x7fdfcc1344a0 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.907+0000 7fdfc8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:27.907+0000 7fdfc8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:27.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: get_monmap_and_config 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: build_initial_monmap 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: monmap: 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:27.916910+0000 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:27.916910+0000 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:27.918 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:27.919 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 Processor -- start 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 -- start start 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: init 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa91c0a0290 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-b con 0x7fa91c0a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 0x7fa91c139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-a con 0x7fa91c137a20 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c09e360 con 0x7fa91c0a0290 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c09ef80 con 0x7fa91c137a20 2024-09-18T10:16:27.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0a1fb0 con 0x7fa91c0a0d70 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa921dcb700 10 monclient(hunting): authenticate will time out at 1270.010912s 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 0x7fa91c139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa91c0a0290 0x7fa91c0a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:40168/0 (socket says 172.21.15.150:40168) 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:41556/0 (socket says 172.21.15.150:41556) 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 -- 172.21.15.150:0/294425 learned_addr learned my addr 172.21.15.150:0/294425 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): get_auth_request con 0x7fa91c0a0d70 auth_method 0 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): get_auth_request con 0x7fa91c137a20 auth_method 0 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient(hunting): handle_auth_done global_id 74306 payload 274 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 -- 172.21.15.150:0/294425 >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 msgr2=0x7fa91c139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 --2- 172.21.15.150:0/294425 >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 0x7fa91c139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 -- 172.21.15.150:0/294425 >> v1:172.21.15.142:6789/0 conn(0x7fa91c0a0290 legacy=0x7fa91c0a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 monclient: found mon.noname-b 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91affd700 1 -- 172.21.15.150:0/294425 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c09da50 con 0x7fa91c0a0d70 2024-09-18T10:16:27.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.915+0000 7fa91bfff700 1 --2- 172.21.15.150:0/294425 >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 0x7fa91c139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91affd700 1 -- 172.21.15.150:0/294425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 msgr2=0x7fa91c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91affd700 1 --2- 172.21.15.150:0/294425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0099b0 tx=0x7fa90c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6609b950f8a5b553 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 1 -- 172.21.15.150:0/294425 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c009ea0 con 0x7fa91c0a0d70 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: dump: 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.926 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 5 monclient: authenticate success, global_id 74306 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 1 -- 172.21.15.150:0/294425 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c00ba70 con 0x7fa91c0a0d70 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:27.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 4 set_mon_vals no callback set 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 1 -- 172.21.15.150:0/294425 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00be30 con 0x7fa91c0a0d70 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa91a7fc700 10 monclient: dump: 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.928 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 10 monclient: shutdown 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 -- 172.21.15.150:0/294425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 msgr2=0x7fa91c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 --2- 172.21.15.150:0/294425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0099b0 tx=0x7fa90c009250 comp rx=0 tx=0).stop 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 -- 172.21.15.150:0/294425 shutdown_connections 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 --2- 172.21.15.150:0/294425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0d70 0x7fa91c09ea40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 --2- 172.21.15.150:0/294425 >> v2:172.21.15.142:3300/0 conn(0x7fa91c137a20 0x7fa91c139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 -- 172.21.15.150:0/294425 >> 172.21.15.150:0/294425 conn(0x7fa91c0a58a0 msgr2=0x7fa91c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 -- 172.21.15.150:0/294425 shutdown_connections 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.919+0000 7fa921dcb700 1 -- 172.21.15.150:0/294425 wait complete. 2024-09-18T10:16:27.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient: build_initial_monmap 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient: monmap: 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 Processor -- start 2024-09-18T10:16:27.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 -- start start 2024-09-18T10:16:27.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient: init 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0290 0x7fa91c12c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-c con 0x7fa91c0a0290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 0x7fa91c12cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-a con 0x7fa91c0a0d70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): picked mon.noname-b con 0x7fa91c137a20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c130cf0 con 0x7fa91c0a0d70 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c09d420 con 0x7fa91c137a20 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c12f800 con 0x7fa91c0a0290 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa921dcb700 10 monclient(hunting): authenticate will time out at 1270.018238s 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32776/0 (socket says 172.21.15.150:32776) 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 -- 172.21.15.150:0/733112162 learned_addr learned my addr 172.21.15.150:0/733112162 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): get_auth_request con 0x7fa91c137a20 auth_method 0 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0290 0x7fa91c12c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 0x7fa91c12cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): get_auth_request con 0x7fa91c0a0290 auth_method 0 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): get_auth_request con 0x7fa91c0a0d70 auth_method 0 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:27.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient(hunting): handle_auth_done global_id 84148 payload 824 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 -- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0290 msgr2=0x7fa91c12c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0290 0x7fa91c12c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 -- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 msgr2=0x7fa91c12cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 0x7fa91c12cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 monclient: found mon.noname-b 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91bfff700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009a20 con 0x7fa91c137a20 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.923+0000 7fa91affd700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 0x7fa91c12cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:27.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa91bfff700 1 -- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 msgr2=0x7fa91c12d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa91bfff700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ef90 tx=0x7fa91400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914009d00 con 0x7fa91c137a20 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: dump: 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.940 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 5 monclient: authenticate success, global_id 84148 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91400f040 con 0x7fa91c137a20 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914014480 con 0x7fa91c137a20 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 10 monclient: _renew_subs 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c0094f0 con 0x7fa91c137a20 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa919ffb700 4 set_mon_vals no callback set 2024-09-18T10:16:27.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 10 monclient: dump: 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 10 monclient: _renew_subs 2024-09-18T10:16:27.942 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c12dfe0 con 0x7fa91c137a20 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.927+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa9140148f0 con 0x7fa91c137a20 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa9137fe700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 0x7fa8fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fa91404ac30 con 0x7fa91c137a20 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa91b7fe700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 0x7fa8fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa91b7fe700 10 monclient: get_auth_request con 0x7fa8fc035ac0 auth_method 0 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa921dcb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa921dcb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa921dcb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:27.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c09d6c0 con 0x7fa91c137a20 2024-09-18T10:16:27.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.931+0000 7fa91b7fe700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 0x7fa8fc037f70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa904009ad0 tx=0x7fa904009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:27.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.939+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa914047e60 con 0x7fa91c137a20 2024-09-18T10:16:27.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.939+0000 7fa9137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:27.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:27.939+0000 7fa9137fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:28.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:28.107+0000 7fa921dcb700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:28.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:28.107+0000 7fa921dcb700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:28.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:28.107+0000 7fa921dcb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:28.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:28.107+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa91c063ed0 con 0x7fa91c137a20 2024-09-18T10:16:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfd170e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfd170e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfd170e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdfcc120df0 con 0x7fdfcc1344a0 2024-09-18T10:16:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfc8ff9700 1 -- 172.21.15.142:0/3147654721 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fdfc4010af0 con 0x7fdfcc1344a0 2024-09-18T10:16:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfc8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:28.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.135+0000 7fdfc8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:28.145 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 msgr2=0x7fdfb0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 0x7fdfb0037f70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc003a30 tx=0x7fdfbc007ea0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 10 monclient: shutdown 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 msgr2=0x7fdfcc11d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400ccf0 tx=0x7fdfc4005400 comp rx=0 tx=0).stop 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 shutdown_connections 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdfb0035ac0 0x7fdfb0037f70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdfcc1344a0 0x7fdfcc11d810 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfcc0a2670 0x7fdfcc11d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 --2- 172.21.15.142:0/3147654721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdfcc06f2d0 0x7fdfcc1317b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 >> 172.21.15.142:0/3147654721 conn(0x7fdfcc071700 msgr2=0x7fdfcc132c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.139+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 shutdown_connections 2024-09-18T10:16:28.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.143+0000 7fdfd170e700 1 -- 172.21.15.142:0/3147654721 wait complete. 2024-09-18T10:16:28.164 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:16:28.164 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:16:28.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:16:28.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.263+0000 7f1dd68ab700 10 monclient: get_monmap_and_config 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.263+0000 7f1dd68ab700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.263+0000 7f1dd68ab700 10 monclient: monmap: 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:28.271515+0000 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:28.271515+0000 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.273 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:28.274 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:28.274 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:28.274 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:28.274 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:28.274 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 Processor -- start 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 -- start start 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient: init 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 0x7f1dd009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-c con 0x7f1dd009abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-b con 0x7f1dd009db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 0x7f1dd0096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-a con 0x7f1dd009c950 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd006fa60 con 0x7f1dd009c950 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd00958c0 con 0x7f1dd009abd0 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0095ef0 con 0x7f1dd009db80 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd68ab700 10 monclient(hunting): authenticate will time out at 1277.196928s 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 0x7f1dd0096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50584/0 (socket says 172.21.15.142:50584) 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 0x7f1dd0096870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47364/0 (socket says 172.21.15.142:47364) 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2308250399 learned_addr learned my addr 172.21.15.142:0/2308250399 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): get_auth_request con 0x7f1dd009db80 auth_method 0 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): get_auth_request con 0x7f1dd009c950 auth_method 0 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 0x7f1dd009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 10 monclient(hunting): get_auth_request con 0x7f1dd009abd0 auth_method 0 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_done global_id 74312 payload 274 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2308250399 >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 msgr2=0x7f1dd009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 0x7f1dd009bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2308250399 >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 msgr2=0x7f1dd0096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 0x7f1dd0096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 monclient: found mon.noname-b 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2308250399 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd009afa0 con 0x7f1dd009db80 2024-09-18T10:16:28.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.267+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 0x7f1dd009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2308250399 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 msgr2=0x7f1dd009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2308250399 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0076e90 tx=0x7f1db8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=198c78eac47b90ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 1 -- 172.21.15.142:0/2308250399 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8009e70 con 0x7f1dd009db80 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: dump: 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.285 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 5 monclient: authenticate success, global_id 74312 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 1 -- 172.21.15.142:0/2308250399 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db800ba70 con 0x7f1dd009db80 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:28.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 4 set_mon_vals no callback set 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 1 -- 172.21.15.142:0/2308250399 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db800be30 con 0x7f1dd009db80 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dcf7fe700 10 monclient: dump: 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.287 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 10 monclient: shutdown 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2308250399 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 msgr2=0x7f1dd009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2308250399 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0076e90 tx=0x7f1db8009250 comp rx=0 tx=0).stop 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2308250399 shutdown_connections 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.142:3300/0 conn(0x7f1dd009c950 0x7f1dd0096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2308250399 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009db80 0x7f1dd009c410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2308250399 >> v2:172.21.15.150:3300/0 conn(0x7f1dd009abd0 0x7f1dd009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2308250399 >> 172.21.15.142:0/2308250399 conn(0x7f1dd00a14e0 msgr2=0x7f1dd00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2308250399 shutdown_connections 2024-09-18T10:16:28.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2308250399 wait complete. 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 10 monclient: monmap: 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 Processor -- start 2024-09-18T10:16:28.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 1 -- start start 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.271+0000 7f1dd68ab700 10 monclient: init 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd009abd0 0x7f1dd008bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-a con 0x7f1dd009abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-c con 0x7f1dd009c950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dd009db80 0x7f1dd008c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): picked mon.noname-b con 0x7f1dd009db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0097c90 con 0x7f1dd009abd0 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00974a0 con 0x7f1dd009db80 2024-09-18T10:16:28.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00901c0 con 0x7f1dd009c950 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient(hunting): authenticate will time out at 1277.204143s 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd009abd0 0x7f1dd008bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50600/0 (socket says 172.21.15.142:50600) 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2381836756 learned_addr learned my addr 172.21.15.142:0/2381836756 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): get_auth_request con 0x7f1dd009c950 auth_method 0 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): get_auth_request con 0x7f1dd009abd0 auth_method 0 2024-09-18T10:16:28.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dd009db80 0x7f1dd008c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): get_auth_request con 0x7f1dd009db80 auth_method 0 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd4e48700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient(hunting): handle_auth_done global_id 74315 payload 824 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dd009db80 msgr2=0x7f1dd008c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1dd009db80 0x7f1dd008c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd009abd0 msgr2=0x7f1dd008bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd009abd0 0x7f1dd008bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80094f0 con 0x7f1dd009c950 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 msgr2=0x7f1dd008c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0096030 tx=0x7f1db8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8006a90 con 0x7f1dd009c950 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: dump: 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.294 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 5 monclient: authenticate success, global_id 74315 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _renew_subs 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80118c0 con 0x7f1dd009c950 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd008efa0 con 0x7f1dd009c950 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8011c80 con 0x7f1dd009c950 2024-09-18T10:16:28.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dceffd700 4 set_mon_vals no callback set 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dccff9700 10 monclient: dump: 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.296 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.297 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _renew_subs 2024-09-18T10:16:28.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.275+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd008e040 con 0x7f1dd009c950 2024-09-18T10:16:28.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd68ab700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 0x7f1dd0087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0064950 con 0x7f1dd008cb20 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 0x7f1dd0087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dc77fe700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0096030 tx=0x7f1db8017000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f1db8023070 mgrmap(e 5) v1 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient: get_auth_request con 0x7f1dd008cb20 auth_method 0 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1db8023070 con 0x7f1dd009c950 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1db0035ac0 0x7f1db0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 10 monclient(hunting): handle_auth_done global_id 54858 payload 824 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dd00696a0 con 0x7f1dd008cb20 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd4e48700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1db0035ac0 0x7f1db0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dd4e48700 10 monclient: get_auth_request con 0x7f1db0035ac0 auth_method 0 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 msgr2=0x7f1dd0087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 0x7f1dd0087fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0006fd0 tx=0x7f1dc0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc000fa00 con 0x7f1dd008cb20 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.279+0000 7f1dccff9700 10 monclient: dump: 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1dc000fb60 con 0x7f1dd008cb20 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 msgr2=0x7f1dd0087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 0x7f1dd0087fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0006fd0 tx=0x7f1dc0008040 comp rx=0 tx=0).stop 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 msgr2=0x7f1dd0087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dccff9700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd008cb20 0x7f1dd0087fe0 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0006fd0 tx=0x7f1dc0008040 comp rx=0 tx=0).stop 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.283+0000 7f1dd4e48700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1db0035ac0 0x7f1db0037f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800f4d0 tx=0x7f1dc800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dd68ab700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 0x7f1dd0088920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dd68ab700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd008f5f0 con 0x7f1dd0088520 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 0x7f1dd0088920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient: get_auth_request con 0x7f1dd0088520 auth_method 0 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 10 monclient(hunting): handle_auth_done global_id 54861 payload 824 2024-09-18T10:16:28.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.291+0000 7f1dcffff700 1 -- 172.21.15.142:0/2381836756 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1dd008ec80 con 0x7f1dd0088520 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dcffff700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 msgr2=0x7f1dd0088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dcffff700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 0x7f1dd0088920 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1dc001a000 tx=0x7f1dc001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0015a50 con 0x7f1dd0088520 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 10 monclient: dump: 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.306 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f1dc0015bb0 con 0x7f1dd0088520 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 msgr2=0x7f1dd0088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 0x7f1dd0088920 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1dc001a000 tx=0x7f1dc001d000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 msgr2=0x7f1dd0088920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1dd0088520 0x7f1dd0088920 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1dc001a000 tx=0x7f1dc001d000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:28.308 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.309 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.310 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.311 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.312 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.313 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.314 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:28.315 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.316 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.317 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.318 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:28.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.320 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:28.321 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1db0035ac0 msgr2=0x7f1db0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1db0035ac0 0x7f1db0037f70 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800f4d0 tx=0x7f1dc800b040 comp rx=0 tx=0).stop 2024-09-18T10:16:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 10 monclient: shutdown 2024-09-18T10:16:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd4e48700 1 -- 172.21.15.142:0/2381836756 reap_dead start 2024-09-18T10:16:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 msgr2=0x7f1dd008c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0096030 tx=0x7f1db8017000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dccff9700 1 -- 172.21.15.142:0/2381836756 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (unknown 0 0 0) 0x7f1db804b040 con 0x7f1dd009c950 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 shutdown_connections 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 --2- 172.21.15.142:0/2381836756 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1dd009c950 0x7f1dd008c0a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 >> 172.21.15.142:0/2381836756 conn(0x7f1dd00a14e0 msgr2=0x7f1dd009fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 shutdown_connections 2024-09-18T10:16:28.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.295+0000 7f1dd68ab700 1 -- 172.21.15.142:0/2381836756 wait complete. 2024-09-18T10:16:28.323 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:28.323 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:16:28.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.427+0000 7fa7ebfff700 10 monclient: get_monmap_and_config 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.427+0000 7fa7ebfff700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.427+0000 7fa7ebfff700 10 monclient: monmap: 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:28.435111+0000 2024-09-18T10:16:28.437 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:28.435111+0000 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 Processor -- start 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 -- start start 2024-09-18T10:16:28.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient: init 2024-09-18T10:16:28.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T10:16:28.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7ec0975d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:28.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7ec132500 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:28.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7ec136000 0x7fa7ec1383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-c con 0x7fa7ec136000 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec0979a0 con 0x7fa7ec0975d0 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec098860 con 0x7fa7ec132500 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec09dc00 con 0x7fa7ec136000 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7ebfff700 10 monclient(hunting): authenticate will time out at 1277.360635s 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eaffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa7ec0975d0 0x7fa7ec099cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57970/0 (socket says 172.21.15.142:57970) 2024-09-18T10:16:28.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eaffd700 1 -- 172.21.15.142:0/3849118386 learned_addr learned my addr 172.21.15.142:0/3849118386 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 1 --2- 172.21.15.142:0/3849118386 >> v2:172.21.15.150:3300/0 conn(0x7fa7ec136000 0x7fa7ec1383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): get_auth_request con 0x7fa7ec136000 auth_method 0 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_done global_id 84157 payload 274 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/3849118386 >> v1:172.21.15.150:6789/0 conn(0x7fa7ec132500 legacy=0x7fa7ec1348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/3849118386 >> v1:172.21.15.142:6789/0 conn(0x7fa7ec0975d0 legacy=0x7fa7ec099cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.431+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/3849118386 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec098b60 con 0x7fa7ec136000 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/3849118386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec136000 msgr2=0x7fa7ec1383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7eb7fe700 1 --2- 172.21.15.142:0/3849118386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec136000 0x7fa7ec1383e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e60 tx=0x7fa7e4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fea17aa83c7bad2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 1 -- 172.21.15.142:0/3849118386 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e400e070 con 0x7fa7ec136000 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: dump: 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.450 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 5 monclient: authenticate success, global_id 84157 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 1 -- 172.21.15.142:0/3849118386 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4012430 con 0x7fa7ec136000 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 4 set_mon_vals no callback set 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 1 -- 172.21.15.142:0/3849118386 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40127d0 con 0x7fa7ec136000 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ea7fc700 10 monclient: dump: 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.452 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: shutdown 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- 172.21.15.142:0/3849118386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec136000 msgr2=0x7fa7ec1383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/3849118386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec136000 0x7fa7ec1383e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009e60 tx=0x7fa7e4009780 comp rx=0 tx=0).stop 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- 172.21.15.142:0/3849118386 shutdown_connections 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/3849118386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec136000 0x7fa7ec1383e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- 172.21.15.142:0/3849118386 >> 172.21.15.142:0/3849118386 conn(0x7fa7ec071ae0 msgr2=0x7fa7ec0a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- 172.21.15.142:0/3849118386 shutdown_connections 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- 172.21.15.142:0/3849118386 wait complete. 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: monmap: 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:28.454 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 Processor -- start 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- start start 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: init 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 0x7fa7ec13d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-a con 0x7fa7ec0975d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec132500 0x7fa7ec13d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-b con 0x7fa7ec132500 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:28.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 0x7fa7ec13dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): picked mon.noname-c con 0x7fa7ec136000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1270f0 con 0x7fa7ec0975d0 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec127670 con 0x7fa7ec132500 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1287e0 con 0x7fa7ec136000 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.435+0000 7fa7ebfff700 10 monclient(hunting): authenticate will time out at 1277.367978s 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 0x7fa7ec13d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 0x7fa7ec13dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 0x7fa7ec13d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47400/0 (socket says 172.21.15.142:47400) 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 0x7fa7ec13dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50602/0 (socket says 172.21.15.142:50602) 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 1 -- 172.21.15.142:0/328763440 learned_addr learned my addr 172.21.15.142:0/328763440 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): get_auth_request con 0x7fa7ec0975d0 auth_method 0 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): get_auth_request con 0x7fa7ec136000 auth_method 0 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec132500 0x7fa7ec13d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): get_auth_request con 0x7fa7ec132500 auth_method 0 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient(hunting): handle_auth_done global_id 74318 payload 824 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec132500 msgr2=0x7fa7ec13d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7ec132500 0x7fa7ec13d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 msgr2=0x7fa7ec13d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 0x7fa7ec13d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4009a20 con 0x7fa7ec136000 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eaffd700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0975d0 0x7fa7ec13d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 msgr2=0x7fa7ec13dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7eb7fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 0x7fa7ec13dec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4015000 tx=0x7fa7e4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4026430 con 0x7fa7ec136000 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: dump: 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.460 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 5 monclient: authenticate success, global_id 74318 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 10 monclient: _renew_subs 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec127390 con 0x7fa7ec136000 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 10 monclient: _renew_subs 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec126450 con 0x7fa7ec136000 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4021070 con 0x7fa7ec136000 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e40267e0 con 0x7fa7ec136000 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e9ffb700 4 set_mon_vals no callback set 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.439+0000 7fa7e37fe700 10 monclient: dump: 2024-09-18T10:16:28.462 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7ebfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 0x7fa7d0004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0007cd0 con 0x7fa7d0002860 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa7e4023030 con 0x7fa7ec136000 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 0x7fa7d0004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient: get_auth_request con 0x7fa7d0002860 auth_method 0 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7e37fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7cc035ac0 0x7fa7cc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_done global_id 74321 payload 824 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fa7e404c1c0 con 0x7fa7ec136000 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7eaffd700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7d0007e90 con 0x7fa7d0002860 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7e27fc700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7cc035ac0 0x7fa7cc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.443+0000 7fa7e27fc700 10 monclient: get_auth_request con 0x7fa7cc035ac0 auth_method 0 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7eaffd700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 msgr2=0x7fa7d0004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7eaffd700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 0x7fa7d0004cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0073d0 tx=0x7fa7dc004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00fb70 con 0x7fa7d0002860 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 10 monclient: dump: 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.469 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa7dc00fcd0 con 0x7fa7d0002860 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 msgr2=0x7fa7d0004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 0x7fa7d0004cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0073d0 tx=0x7fa7dc004a80 comp rx=0 tx=0).stop 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 msgr2=0x7fa7d0004cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e37fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0002860 0x7fa7d0004cf0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0073d0 tx=0x7fa7dc004a80 comp rx=0 tx=0).stop 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.447+0000 7fa7e27fc700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7cc035ac0 0x7fa7cc037f70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4003a30 tx=0x7fa7d4007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7ebfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 0x7fa7d0007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7ebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d00077a0 con 0x7fa7d0006cc0 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 0x7fa7d0007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient: get_auth_request con 0x7fa7d0006cc0 auth_method 0 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 10 monclient(hunting): handle_auth_done global_id 74324 payload 824 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.459+0000 7fa7eaffd700 1 -- 172.21.15.142:0/328763440 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa7d0007960 con 0x7fa7d0006cc0 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7eaffd700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 msgr2=0x7fa7d0007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7eaffd700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 0x7fa7d0007140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc007970 tx=0x7fa7dc018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc01c980 con 0x7fa7d0006cc0 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 10 monclient: dump: 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.473 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fa7dc01cae0 con 0x7fa7d0006cc0 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 msgr2=0x7fa7d0007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 0x7fa7d0007140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc007970 tx=0x7fa7dc018000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 msgr2=0x7fa7d0007140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7e37fe700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d0006cc0 0x7fa7d0007140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:28.475 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:28.476 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:28.477 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.478 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:28.479 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.480 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.481 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.482 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.483 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.484 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.485 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.486 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.487 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:28.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.489 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.490 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:28.491 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7cc035ac0 msgr2=0x7fa7cc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7cc035ac0 0x7fa7cc037f70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4003a30 tx=0x7fa7d4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 10 monclient: shutdown 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 msgr2=0x7fa7ec13dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 --2- 172.21.15.142:0/328763440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec136000 0x7fa7ec13dec0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4015000 tx=0x7fa7e4018000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 reap_dead start 2024-09-18T10:16:28.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7eb7fe700 1 -- 172.21.15.142:0/328763440 reap_dead start 2024-09-18T10:16:28.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 shutdown_connections 2024-09-18T10:16:28.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 >> 172.21.15.142:0/328763440 conn(0x7fa7ec071ae0 msgr2=0x7fa7ec1348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 shutdown_connections 2024-09-18T10:16:28.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.463+0000 7fa7ebfff700 1 -- 172.21.15.142:0/328763440 wait complete. 2024-09-18T10:16:28.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:28.493 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:16:28.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:16:28.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 10 monclient: get_monmap_and_config 2024-09-18T10:16:28.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 10 monclient: monmap: 2024-09-18T10:16:28.607 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.607 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:28.606026+0000 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:28.606026+0000 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:28.608 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 1 Processor -- start 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 1 -- start start 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.599+0000 7f1d6f2c6700 10 monclient: init 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1d6806f3b0 0x7f1d680989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-a con 0x7f1d6806f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d68098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:28.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d68099320 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6809c330 con 0x7f1d68098f00 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6809c530 con 0x7f1d68099320 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6809b030 con 0x7f1d6806f3b0 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6f2c6700 10 monclient(hunting): authenticate will time out at 1277.532880s 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1d6806f3b0 0x7f1d680989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6dac3700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f1d68098f00 0x7f1d68133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57984/0 (socket says 172.21.15.142:57984) 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1d6806f3b0 0x7f1d680989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47410/0 (socket says 172.21.15.142:47410) 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6dac3700 1 -- 172.21.15.142:0/3620055732 learned_addr learned my addr 172.21.15.142:0/3620055732 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): get_auth_request con 0x7f1d6806f3b0 auth_method 0 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_done global_id 54867 payload 274 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/3620055732 >> v1:172.21.15.150:6789/0 conn(0x7f1d68099320 legacy=0x7f1d68136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/3620055732 >> v1:172.21.15.142:6789/0 conn(0x7f1d68098f00 legacy=0x7f1d68133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 monclient: found mon.noname-a 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:16:28.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/3620055732 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6809c0f0 con 0x7f1d6806f3b0 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/3620055732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d6806f3b0 msgr2=0x7f1d680989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6e2c4700 1 --2- 172.21.15.142:0/3620055732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d6806f3b0 0x7f1d680989c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00bb90 tx=0x7f1d5c00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18bcee9dd4f578b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.603+0000 7f1d6d2c2700 1 -- 172.21.15.142:0/3620055732 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00de90 con 0x7f1d6806f3b0 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:16:28.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: dump: 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 5 monclient: authenticate success, global_id 54867 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 1 -- 172.21.15.142:0/3620055732 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c010470 con 0x7f1d6806f3b0 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 4 set_mon_vals no callback set 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 1 -- 172.21.15.142:0/3620055732 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c010830 con 0x7f1d6806f3b0 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6d2c2700 10 monclient: dump: 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.620 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: shutdown 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/3620055732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d6806f3b0 msgr2=0x7f1d680989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/3620055732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d6806f3b0 0x7f1d680989c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00bb90 tx=0x7f1d5c00b430 comp rx=0 tx=0).stop 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/3620055732 shutdown_connections 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/3620055732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d6806f3b0 0x7f1d680989c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/3620055732 >> 172.21.15.142:0/3620055732 conn(0x7f1d68070ef0 msgr2=0x7f1d680712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/3620055732 shutdown_connections 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/3620055732 wait complete. 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: monmap: 2024-09-18T10:16:28.624 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 Processor -- start 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- start start 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: init 2024-09-18T10:16:28.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-c con 0x7f1d68098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68099320 0x7f1d680943a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-b con 0x7f1d68099320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 0x7f1d6808e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): picked mon.noname-a con 0x7f1d68090660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6808fb20 con 0x7f1d68090660 2024-09-18T10:16:28.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806fa20 con 0x7f1d68099320 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806eaa0 con 0x7f1d68098f00 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6f2c6700 10 monclient(hunting): authenticate will time out at 1277.538774s 2024-09-18T10:16:28.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 0x7f1d6808e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50630/0 (socket says 172.21.15.142:50630) 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 0x7f1d6808e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:47412/0 (socket says 172.21.15.142:47412) 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/1496783829 learned_addr learned my addr 172.21.15.142:0/1496783829 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): get_auth_request con 0x7f1d68098f00 auth_method 0 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): get_auth_request con 0x7f1d68090660 auth_method 0 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68099320 0x7f1d680943a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): get_auth_request con 0x7f1d68099320 auth_method 0 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6dac3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient(hunting): handle_auth_done global_id 74327 payload 824 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68099320 msgr2=0x7f1d680943a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68099320 0x7f1d680943a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 msgr2=0x7f1d6808e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 0x7f1d6808e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 monclient: found mon.noname-c 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c00b6d0 con 0x7f1d68098f00 2024-09-18T10:16:28.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.607+0000 7f1d6eac5700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1d68090660 0x7f1d6808e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:28.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6e2c4700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 msgr2=0x7f1d681419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6e2c4700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00d5d0 tx=0x7f1d5c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c010e90 con 0x7f1d68098f00 2024-09-18T10:16:28.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: dump: 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.636 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c018840 con 0x7f1d68098f00 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 5 monclient: authenticate success, global_id 74327 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c018be0 con 0x7f1d68098f00 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 10 monclient: _renew_subs 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d680918a0 con 0x7f1d68098f00 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6cac1700 4 set_mon_vals no callback set 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 10 monclient: dump: 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.638 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 10 monclient: _renew_subs 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d68141f20 con 0x7f1d68098f00 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.611+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1d5c00f3d0 con 0x7f1d68098f00 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d667fc700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d54035af0 0x7f1d54037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7f1d5c014070 con 0x7f1d68098f00 2024-09-18T10:16:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6f2c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6dac3700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d54035af0 0x7f1d54037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 0x7f1d6808e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680648a0 con 0x7f1d680948e0 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6dac3700 10 monclient: get_auth_request con 0x7f1d54035af0 auth_method 0 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 0x7f1d6808e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient: get_auth_request con 0x7f1d680948e0 auth_method 0 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_done global_id 84166 payload 824 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6eac5700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d68142140 con 0x7f1d680948e0 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.615+0000 7f1d6dac3700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d54035af0 0x7f1d54037fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006e80 tx=0x7f1d580053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d6eac5700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 msgr2=0x7f1d6808e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d6eac5700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 0x7f1d6808e5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000fb30 tx=0x7f1d60007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6000db60 con 0x7f1d680948e0 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 10 monclient: dump: 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1d6000dcc0 con 0x7f1d680948e0 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 msgr2=0x7f1d6808e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 0x7f1d6808e5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000fb30 tx=0x7f1d60007ea0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 msgr2=0x7f1d6808e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.619+0000 7f1d667fc700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d680948e0 0x7f1d6808e5c0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000fb30 tx=0x7f1d60007ea0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.627+0000 7f1d6f2c6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.627+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 0x7f1d68089ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.627+0000 7f1d6f2c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.627+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68094cb0 con 0x7f1d6808eb00 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 0x7f1d68089ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient: get_auth_request con 0x7f1d6808eb00 auth_method 0 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 10 monclient(hunting): handle_auth_done global_id 84169 payload 824 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 1 -- 172.21.15.142:0/1496783829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1d68091d50 con 0x7f1d6808eb00 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 msgr2=0x7f1d68089ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d6eac5700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 0x7f1d68089ff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000fb30 tx=0x7f1d60019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6001d950 con 0x7f1d6808eb00 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 10 monclient: dump: 2024-09-18T10:16:28.645 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f1d60026070 con 0x7f1d6808eb00 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 msgr2=0x7f1d68089ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 0x7f1d68089ff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d6000fb30 tx=0x7f1d60019000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 msgr2=0x7f1d68089ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.631+0000 7f1d667fc700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d6808eb00 0x7f1d68089ff0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 80, 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:28.647 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.648 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.649 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:28.650 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:28.651 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.652 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.653 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.654 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:28.656 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:28.657 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.658 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.659 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:28.660 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:28.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d54035af0 msgr2=0x7f1d54037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1d54035af0 0x7f1d54037fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f1d58006e80 tx=0x7f1d580053c0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 10 monclient: shutdown 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6eac5700 1 -- 172.21.15.142:0/1496783829 reap_dead start 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 msgr2=0x7f1d681419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00d5d0 tx=0x7f1d5c01a000 comp rx=0 tx=0).stop 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 shutdown_connections 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 --2- 172.21.15.142:0/1496783829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1d68098f00 0x7f1d681419e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 >> 172.21.15.142:0/1496783829 conn(0x7f1d68070ef0 msgr2=0x7f1d6809f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 shutdown_connections 2024-09-18T10:16:28.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.635+0000 7f1d6f2c6700 1 -- 172.21.15.142:0/1496783829 wait complete. 2024-09-18T10:16:28.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:16:28.664 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:16:28.664 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:16:28.664 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:16:28.665 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:16:28.665 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:16:28.665 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:16:28.687 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:16:28.687 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:16:28.688 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:28.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: get_monmap_and_config 2024-09-18T10:16:28.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: monmap: 2024-09-18T10:16:28.790 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.790 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:28.789080+0000 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:28.789080+0000 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:28.791 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 Processor -- start 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 -- start start 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: init 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc0c09cdd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-b con 0x7fdc0c134a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 0x7fdc0c139750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-c con 0x7fdc0c137370 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:28.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c09d1a0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c09d780 con 0x7fdc0c137370 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c09dd40 con 0x7fdc0c134a50 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc11458700 10 monclient(hunting): authenticate will time out at 1277.714777s 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50638/0 (socket says 172.21.15.142:50638) 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 -- 172.21.15.142:0/2764022207 learned_addr learned my addr 172.21.15.142:0/2764022207 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): get_auth_request con 0x7fdc0c134a50 auth_method 0 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0b7fe700 1 --2- 172.21.15.142:0/2764022207 >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 0x7fdc0c139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_done global_id 74330 payload 274 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 -- 172.21.15.142:0/2764022207 >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 msgr2=0x7fdc0c139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 --2- 172.21.15.142:0/2764022207 >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 0x7fdc0c139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 -- 172.21.15.142:0/2764022207 >> v1:172.21.15.150:6789/0 conn(0x7fdc0c09cdd0 legacy=0x7fdc0c09e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 monclient: found mon.noname-b 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0a7fc700 1 -- 172.21.15.142:0/2764022207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0a6710 con 0x7fdc0c134a50 2024-09-18T10:16:28.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0b7fe700 10 monclient: get_auth_request con 0x7fdc0c137370 auth_method 0 2024-09-18T10:16:28.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.783+0000 7fdc0b7fe700 1 --2- 172.21.15.142:0/2764022207 >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 0x7fdc0c139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc0a7fc700 1 -- 172.21.15.142:0/2764022207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 msgr2=0x7fdc0c136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc0a7fc700 1 --2- 172.21.15.142:0/2764022207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009f20 tx=0x7fdbfc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebbb52c7ff8609c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 1 -- 172.21.15.142:0/2764022207 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc00e070 con 0x7fdc0c134a50 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: dump: 2024-09-18T10:16:28.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 5 monclient: authenticate success, global_id 74330 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 1 -- 172.21.15.142:0/2764022207 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc00bc90 con 0x7fdc0c134a50 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 4 set_mon_vals no callback set 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 1 -- 172.21.15.142:0/2764022207 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc012660 con 0x7fdc0c134a50 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc09ffb700 10 monclient: dump: 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.806 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 10 monclient: shutdown 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- 172.21.15.142:0/2764022207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 msgr2=0x7fdc0c136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 --2- 172.21.15.142:0/2764022207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009f20 tx=0x7fdbfc009840 comp rx=0 tx=0).stop 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- 172.21.15.142:0/2764022207 shutdown_connections 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 --2- 172.21.15.142:0/2764022207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c134a50 0x7fdc0c136e30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 --2- 172.21.15.142:0/2764022207 >> v2:172.21.15.150:3300/0 conn(0x7fdc0c137370 0x7fdc0c139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- 172.21.15.142:0/2764022207 >> 172.21.15.142:0/2764022207 conn(0x7fdc0c071680 msgr2=0x7fdc0c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- 172.21.15.142:0/2764022207 shutdown_connections 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- 172.21.15.142:0/2764022207 wait complete. 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 10 monclient: build_initial_monmap 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 10 monclient: monmap: 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:28.808 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 Processor -- start 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 1 -- start start 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.787+0000 7fdc11458700 10 monclient: init 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-c con 0x7fdc0c09cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc0c134a50 0x7fdc0c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-b con 0x7fdc0c134a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 0x7fdc0c06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): picked mon.noname-a con 0x7fdc0c137370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0a2f00 con 0x7fdc0c137370 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c06ed70 con 0x7fdc0c134a50 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c06e0a0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc11458700 10 monclient(hunting): authenticate will time out at 1277.720579s 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0b7fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 msgr2=0x7fdc0c06b0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 0x7fdc0c06b0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50646/0 (socket says 172.21.15.142:50646) 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 -- 172.21.15.142:0/1494621387 learned_addr learned my addr 172.21.15.142:0/1494621387 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): get_auth_request con 0x7fdc0c09cdd0 auth_method 0 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc0c134a50 0x7fdc0c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): get_auth_request con 0x7fdc0c134a50 auth_method 0 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient(hunting): handle_auth_done global_id 74333 payload 824 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 -- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc0c134a50 msgr2=0x7fdc0c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc0c134a50 0x7fdc0c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 -- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 msgr2=0x7fdc0c06b0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 0x7fdc0c06b0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:28.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 monclient: found mon.noname-c 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 -- 172.21.15.142:0/1494621387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc009ae0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 -- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 msgr2=0x7fdc0c09c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.791+0000 7fdc0affd700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c5d0 tx=0x7fdbf4007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40117a0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: dump: 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.814 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 5 monclient: authenticate success, global_id 74333 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _renew_subs 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4011920 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c06c8a0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4011ce0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 10 monclient: dump: 2024-09-18T10:16:28.815 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc097fa700 4 set_mon_vals no callback set 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _renew_subs 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c06bf10 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c06dca0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdbf4012630 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 0x7fdbf0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fdbf4049cb0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc0a7fc700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 0x7fdbf0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:28.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.795+0000 7fdc0a7fc700 10 monclient: get_auth_request con 0x7fdbf0035a70 auth_method 0 2024-09-18T10:16:28.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.799+0000 7fdc0a7fc700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 0x7fdbf0037f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009f20 tx=0x7fdbfc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:28.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.807+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdbf40108e0 con 0x7fdc0c09cdd0 2024-09-18T10:16:28.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.807+0000 7fdc02ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:28.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:28.807+0000 7fdc02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:29.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:29.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:29.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:29.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc0c138ea0 con 0x7fdc0c09cdd0 2024-09-18T10:16:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc02ffd700 1 -- 172.21.15.142:0/1494621387 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fdbf40102e0 con 0x7fdc0c09cdd0 2024-09-18T10:16:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc02ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc02ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:29.046 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:29.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 msgr2=0x7fdbf0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:29.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 0x7fdbf0037f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009f20 tx=0x7fdbfc017000 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 10 monclient: shutdown 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 msgr2=0x7fdc0c09c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400c5d0 tx=0x7fdbf4007f60 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 shutdown_connections 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdbf0035a70 0x7fdbf0037f20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc0c137370 0x7fdc0c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc0c134a50 0x7fdc0c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 --2- 172.21.15.142:0/1494621387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc0c09cdd0 0x7fdc0c09c230 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 >> 172.21.15.142:0/1494621387 conn(0x7fdc0c071680 msgr2=0x7fdc0c1340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 shutdown_connections 2024-09-18T10:16:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:29.039+0000 7fdc11458700 1 -- 172.21.15.142:0/1494621387 wait complete. 2024-09-18T10:16:30.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:30.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:30.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:30.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:30.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:30.923+0000 7fa9127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:31.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:31.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:31.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:32.061 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:32.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: get_monmap_and_config 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: build_initial_monmap 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: monmap: 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:32.173260+0000 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:32.173260+0000 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:32.175 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:32.176 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:32.176 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:32.176 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:32.176 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 Processor -- start 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 -- start start 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: init 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02f809abd0 0x7f02f809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-c con 0x7f02f809abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02f809db80 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02f8097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f80958c0 con 0x7f02f8097880 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f8095ef0 con 0x7f02f809db80 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f806fa60 con 0x7f02f809abd0 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:32.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02fcae3700 10 monclient(hunting): authenticate will time out at 1281.098744s 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f6d9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f02f8097880 legacy=0x7f02f8137690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02f809abd0 0x7f02f809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02edd9b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f02f809db80 0x7f02f809c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:45458/0 (socket says 172.21.15.142:45458) 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02edd9b700 1 -- 172.21.15.142:0/3042870360 learned_addr learned my addr 172.21.15.142:0/3042870360 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): get_auth_request con 0x7f02f809abd0 auth_method 0 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:32.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.167+0000 7f02f659c700 10 monclient(hunting): handle_auth_done global_id 84175 payload 274 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 -- 172.21.15.142:0/3042870360 >> v1:172.21.15.150:6789/0 conn(0x7f02f809db80 legacy=0x7f02f809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:32.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 -- 172.21.15.142:0/3042870360 >> v1:172.21.15.142:6789/0 conn(0x7f02f8097880 legacy=0x7f02f8137690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:32.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:32.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 monclient: found mon.noname-c 2024-09-18T10:16:32.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:32.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:32.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 -- 172.21.15.142:0/3042870360 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f809afa0 con 0x7f02f809abd0 2024-09-18T10:16:32.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 -- 172.21.15.142:0/3042870360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 msgr2=0x7f02f809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:32.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f659c700 1 --2- 172.21.15.142:0/3042870360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f809bed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a2460 tx=0x7f02e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43fb08cd9c13003b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:32.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 1 -- 172.21.15.142:0/3042870360 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e800b9a0 con 0x7f02f809abd0 2024-09-18T10:16:32.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: dump: 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:32.187 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: _finish_auth 0 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02fcae3700 5 monclient: authenticate success, global_id 84175 2024-09-18T10:16:32.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 1 -- 172.21.15.142:0/3042870360 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e800bb20 con 0x7f02f809abd0 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02fcae3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02fcae3700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 1 -- 172.21.15.142:0/3042870360 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e8011420 con 0x7f02f809abd0 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02fcae3700 4 set_mon_vals no callback set 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.171+0000 7f02f5d9b700 10 monclient: dump: 2024-09-18T10:16:32.189 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: shutdown 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:32.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- 172.21.15.142:0/3042870360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 msgr2=0x7f02f809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3042870360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f809bed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a2460 tx=0x7f02e80091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- 172.21.15.142:0/3042870360 shutdown_connections 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3042870360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f809bed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- 172.21.15.142:0/3042870360 >> 172.21.15.142:0/3042870360 conn(0x7f02f80a14e0 msgr2=0x7f02f80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- 172.21.15.142:0/3042870360 shutdown_connections 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- 172.21.15.142:0/3042870360 wait complete. 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: build_initial_monmap 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: monmap: 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 Processor -- start 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- start start 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: init 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-c con 0x7f02f8097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f80943b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-b con 0x7f02f809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 0x7f02f80989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): picked mon.noname-a con 0x7f02f809db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f809a9f0 con 0x7f02f809db80 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f8091e70 con 0x7f02f809abd0 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f80918b0 con 0x7f02f8097880 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02fcae3700 10 monclient(hunting): authenticate will time out at 1281.106426s 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f6d9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 msgr2=0x7f02f80989c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f6d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 0x7f02f80989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58408/0 (socket says 172.21.15.142:58408) 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 -- 172.21.15.142:0/3218435432 learned_addr learned my addr 172.21.15.142:0/3218435432 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): get_auth_request con 0x7f02f8097880 auth_method 0 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f80943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): get_auth_request con 0x7f02f809abd0 auth_method 0 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient(hunting): handle_auth_done global_id 74336 payload 824 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 -- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 msgr2=0x7f02f80943b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f80943b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 -- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 msgr2=0x7f02f80989c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 0x7f02f80989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 monclient: found mon.noname-c 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.175+0000 7f02f659c700 1 -- 172.21.15.142:0/3218435432 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8009420 con 0x7f02f8097880 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02f659c700 1 -- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 msgr2=0x7f02f8141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:32.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02f659c700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a2530 tx=0x7f02e8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:32.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e800bea0 con 0x7f02f8097880 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: dump: 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:32.200 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e8020070 con 0x7f02f8097880 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 5 monclient: authenticate success, global_id 74336 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e801b6a0 con 0x7f02f8097880 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _renew_subs 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02f559a700 4 set_mon_vals no callback set 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f80915d0 con 0x7f02f8097880 2024-09-18T10:16:32.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 10 monclient: dump: 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:32.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _renew_subs 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f8142770 con 0x7f02f8097880 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f02e8022030 con 0x7f02f8097880 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.179+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f8142960 con 0x7f02f8097880 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.183+0000 7f02eeffd700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 0x7f02d8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.183+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7f02e804bcf0 con 0x7f02f8097880 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.183+0000 7f02edd9b700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 0x7f02d8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.183+0000 7f02edd9b700 10 monclient: get_auth_request con 0x7f02d8035a70 auth_method 0 2024-09-18T10:16:32.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.183+0000 7f02edd9b700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 0x7f02d8037f20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f02f809c850 tx=0x7f02e0007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:32.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.191+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f02e8051020 con 0x7f02f8097880 2024-09-18T10:16:32.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.191+0000 7f02eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:32.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.191+0000 7f02eeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:32.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02fcae3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:32.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02fcae3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:32.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02fcae3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:32.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f02f8142960 con 0x7f02f8097880 2024-09-18T10:16:32.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02eeffd700 1 -- 172.21.15.142:0/3218435432 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f02e8011e30 con 0x7f02f8097880 2024-09-18T10:16:32.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:32.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.415+0000 7f02eeffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:32.424 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:32.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 msgr2=0x7f02d8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 0x7f02d8037f20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f02f809c850 tx=0x7f02e0007f90 comp rx=0 tx=0).stop 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 10 monclient: shutdown 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 msgr2=0x7f02f8141a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02f80a2530 tx=0x7f02e8016000 comp rx=0 tx=0).stop 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 shutdown_connections 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f02d8035a70 0x7f02d8037f20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f02f809db80 0x7f02f80989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f809abd0 0x7f02f80943b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 --2- 172.21.15.142:0/3218435432 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02f8097880 0x7f02f8141a50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:32.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 >> 172.21.15.142:0/3218435432 conn(0x7f02f80a14e0 msgr2=0x7f02f806d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:32.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 shutdown_connections 2024-09-18T10:16:32.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:32.419+0000 7f02fcae3700 1 -- 172.21.15.142:0/3218435432 wait complete. 2024-09-18T10:16:32.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:32.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:32.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:32.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:33.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:33.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:33.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:33.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:34.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:34.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:34.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:34.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:35.442 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: get_monmap_and_config 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: build_initial_monmap 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: monmap: 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:35.556 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:35.554803+0000 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:35.554803+0000 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:35.557 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 Processor -- start 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 -- start start 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: init 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-b con 0x7fe3a006e860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3a006f460 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:35.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3a009ca30 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:35.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a006f220 con 0x7fe3a006f460 2024-09-18T10:16:35.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a0099780 con 0x7fe3a009ca30 2024-09-18T10:16:35.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:35.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a0099910 con 0x7fe3a006e860 2024-09-18T10:16:35.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3977fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fe3a006f460 legacy=0x7fe3a009c320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe3a5417700 10 monclient(hunting): authenticate will time out at 1284.481355s 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58420/0 (socket says 172.21.15.142:58420) 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 -- 172.21.15.142:0/3374722506 learned_addr learned my addr 172.21.15.142:0/3374722506 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): get_auth_request con 0x7fe3a006e860 auth_method 0 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:35.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient(hunting): handle_auth_done global_id 74339 payload 274 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 -- 172.21.15.142:0/3374722506 >> v1:172.21.15.150:6789/0 conn(0x7fe3a009ca30 legacy=0x7fe3a01395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 -- 172.21.15.142:0/3374722506 >> v1:172.21.15.142:6789/0 conn(0x7fe3a006f460 legacy=0x7fe3a009c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 monclient: found mon.noname-b 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.551+0000 7fe39ffff700 1 -- 172.21.15.142:0/3374722506 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a009b770 con 0x7fe3a006e860 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39ffff700 1 -- 172.21.15.142:0/3374722506 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 msgr2=0x7fe3a009bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39ffff700 1 --2- 172.21.15.142:0/3374722506 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe3900099e0 tx=0x7fe390009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ae9da68325e6208 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 1 -- 172.21.15.142:0/3374722506 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39000baf0 con 0x7fe3a006e860 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: dump: 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:35.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 1 -- 172.21.15.142:0/3374722506 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39000bc50 con 0x7fe3a006e860 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 5 monclient: authenticate success, global_id 74339 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:35.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 4 set_mon_vals no callback set 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 1 -- 172.21.15.142:0/3374722506 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390011420 con 0x7fe3a006e860 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe39f7fe700 10 monclient: dump: 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:35.579 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 10 monclient: shutdown 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 -- 172.21.15.142:0/3374722506 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 msgr2=0x7fe3a009bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 --2- 172.21.15.142:0/3374722506 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe3900099e0 tx=0x7fe390009250 comp rx=0 tx=0).stop 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 -- 172.21.15.142:0/3374722506 shutdown_connections 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 --2- 172.21.15.142:0/3374722506 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006e860 0x7fe3a009bde0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 -- 172.21.15.142:0/3374722506 >> 172.21.15.142:0/3374722506 conn(0x7fe3a00a1fe0 msgr2=0x7fe3a0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 -- 172.21.15.142:0/3374722506 shutdown_connections 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 1 -- 172.21.15.142:0/3374722506 wait complete. 2024-09-18T10:16:35.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 10 monclient: build_initial_monmap 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.555+0000 7fe3a5417700 10 monclient: monmap: 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 Processor -- start 2024-09-18T10:16:35.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 -- start start 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient: init 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 0x7fe3a008fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-a con 0x7fe3a006e860 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-c con 0x7fe3a006f460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a009ca30 0x7fe3a00908c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): picked mon.noname-b con 0x7fe3a009ca30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe39ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 msgr2=0x7fe3a008fe40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a0091980 con 0x7fe3a006e860 2024-09-18T10:16:35.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a009d230 con 0x7fe3a009ca30 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a009fff0 con 0x7fe3a006f460 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe39ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 0x7fe3a008fe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a5417700 10 monclient(hunting): authenticate will time out at 1284.490180s 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58434/0 (socket says 172.21.15.142:58434) 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 -- 172.21.15.142:0/2428546579 learned_addr learned my addr 172.21.15.142:0/2428546579 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): get_auth_request con 0x7fe3a006f460 auth_method 0 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:35.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a009ca30 0x7fe3a00908c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): get_auth_request con 0x7fe3a009ca30 auth_method 0 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3a4c16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient(hunting): handle_auth_done global_id 74342 payload 824 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 -- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a009ca30 msgr2=0x7fe3a00908c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:35.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a009ca30 0x7fe3a00908c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 -- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 msgr2=0x7fe3a008fe40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 0x7fe3a008fe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.559+0000 7fe3977fe700 1 -- 172.21.15.142:0/2428546579 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3900094f0 con 0x7fe3a006f460 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3977fe700 1 -- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 msgr2=0x7fe3a0090380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3977fe700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe38800b810 tx=0x7fe388009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388003e20 con 0x7fe3a006f460 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:35.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: dump: 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: _finish_auth 0 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe388006150 con 0x7fe3a006f460 2024-09-18T10:16:35.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 5 monclient: authenticate success, global_id 74342 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _renew_subs 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388006510 con 0x7fe3a006f460 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a0093700 con 0x7fe3a006f460 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39effd700 4 set_mon_vals no callback set 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 10 monclient: dump: 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.587 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _renew_subs 2024-09-18T10:16:35.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a0092f00 con 0x7fe3a006f460 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a0093180 con 0x7fe3a006f460 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.563+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe38801b3b0 con 0x7fe3a006f460 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.567+0000 7fe39cff9700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 0x7fe380037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.567+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fe388049460 con 0x7fe3a006f460 2024-09-18T10:16:35.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.567+0000 7fe39ffff700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 0x7fe380037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:35.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.567+0000 7fe39ffff700 10 monclient: get_auth_request con 0x7fe380035a70 auth_method 0 2024-09-18T10:16:35.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.567+0000 7fe39ffff700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 0x7fe380037f20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0067590 tx=0x7fe390016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:35.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.575+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe38800e860 con 0x7fe3a006f460 2024-09-18T10:16:35.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.575+0000 7fe39cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:35.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.575+0000 7fe39cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:35.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.795+0000 7fe3a5417700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:35.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.795+0000 7fe3a5417700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:35.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.795+0000 7fe3a5417700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.795+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe3a0072a80 con 0x7fe3a006f460 2024-09-18T10:16:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe39cff9700 1 -- 172.21.15.142:0/2428546579 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fe388017030 con 0x7fe3a006f460 2024-09-18T10:16:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe39cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe39cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:35.805 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:35.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 msgr2=0x7fe380037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:35.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 0x7fe380037f20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0067590 tx=0x7fe390016000 comp rx=0 tx=0).stop 2024-09-18T10:16:35.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 10 monclient: shutdown 2024-09-18T10:16:35.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 msgr2=0x7fe3a0090380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe38800b810 tx=0x7fe388009220 comp rx=0 tx=0).stop 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 shutdown_connections 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe380035a70 0x7fe380037f20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3a009ca30 0x7fe3a00908c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3a006f460 0x7fe3a0090380 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 --2- 172.21.15.142:0/2428546579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3a006e860 0x7fe3a008fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 >> 172.21.15.142:0/2428546579 conn(0x7fe3a00a1fe0 msgr2=0x7fe3a009dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 shutdown_connections 2024-09-18T10:16:35.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:35.803+0000 7fe3a5417700 1 -- 172.21.15.142:0/2428546579 wait complete. 2024-09-18T10:16:35.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:35.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:35.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:35.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:36.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:36.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:36.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:36.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:37.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:37.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:37.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:37.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:38.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:16:38.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:38.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:38.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:38.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 10 monclient: get_monmap_and_config 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 10 monclient: build_initial_monmap 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 10 monclient: monmap: 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:38.930552+0000 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:38.930552+0000 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:38.933 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 1 Processor -- start 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 1 -- start start 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.927+0000 7fe91730a700 10 monclient: init 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-c con 0x7fe91009d920 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-b con 0x7fe9100953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:38.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9100957d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91009b0b0 con 0x7fe9100957d0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91006fba0 con 0x7fe91009d920 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91009a7e0 con 0x7fe9100953c0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe91730a700 10 monclient(hunting): authenticate will time out at 1287.856615s 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916b09700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fe9100957d0 legacy=0x7fe9100971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:38.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58446/0 (socket says 172.21.15.142:58446) 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 learned_addr learned my addr 172.21.15.142:0/2042641575 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe9100953c0 auth_method 0 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe91009d920 auth_method 0 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:38.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:16:38.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:38.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:38.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:38.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:38.931+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:39.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916b09700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.142:6789/0 conn(0x7fe9100957d0 legacy=0x7fe9100971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:39.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:39.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe9100953c0 auth_method 0 2024-09-18T10:16:39.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe91009d920 auth_method 0 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:16:39.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:39.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:39.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.135+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916b09700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.142:6789/0 conn(0x7fe9100957d0 legacy=0x7fe9100971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe9100953c0 auth_method 0 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:39.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe91009d920 auth_method 0 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:39.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:39.539+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:16:39.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:39.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:39.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:39.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:40.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916b09700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.142:6789/0 conn(0x7fe9100957d0 legacy=0x7fe9100971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:40.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:40.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe9100953c0 auth_method 0 2024-09-18T10:16:40.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:40.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe91009d920 auth_method 0 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:16:40.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:16:40.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:40.343+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:16:40.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:40.923+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:40.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:40.923+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:40.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:40.923+0000 7fa9127fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:41.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:41.927+0000 7fa9127fc700 10 monclient: tick 2024-09-18T10:16:41.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:41.927+0000 7fa9127fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:41.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): tick 2024-09-18T10:16:41.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:16:41.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:16:41.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 msgr2=0x7fe910096cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:41.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe9100953c0 0x7fe910096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 msgr2=0x7fe91006f660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.150:3300/0 conn(0x7fe91009d920 0x7fe91006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.142:6789/0 conn(0x7fe9100957d0 legacy=0x7fe9100971f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient: _add_conns ranks=[1,3] 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8f40036f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 0x7fe8f40098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe8f4007410 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 -- 172.21.15.142:0/2042641575 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe910097900 con 0x7fe8f40036f0 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:41.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 1 -- 172.21.15.142:0/2042641575 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91006fba0 con 0x7fe8f4007410 2024-09-18T10:16:41.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe907fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:41.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 msgr2=0x7fe8f40098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:41.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 0x7fe8f40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:41.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:41.931+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.150:6789/0 conn(0x7fe8f40036f0 legacy=0x7fe8f4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:42.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.135+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 msgr2=0x7fe8f40098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:42.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.135+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 0x7fe8f40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:16:42.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.539+0000 7fe915b07700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 msgr2=0x7fe8f40098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:42.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.539+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 0x7fe8f40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:16:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.707+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929785407 0 0) 0x7fe910097900 con 0x7fe8f40036f0 2024-09-18T10:16:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.707+0000 7fe915306700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.707+0000 7fe915306700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.707+0000 7fe915306700 10 monclient(hunting): my global_id is 84184 2024-09-18T10:16:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.707+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8fc003570 con 0x7fe8f40036f0 2024-09-18T10:16:42.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342820121 0 0) 0x7fe8fc003570 con 0x7fe8f40036f0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 msgr2=0x7fe8f40098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 --2- 172.21.15.142:0/2042641575 >> v2:172.21.15.142:3300/0 conn(0x7fe8f4007410 0x7fe8f40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 10 monclient: _finish_hunting 0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91009b0b0 con 0x7fe8f40036f0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 10 monclient: _finish_auth 0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fe90c00d480 con 0x7fe8f40036f0 2024-09-18T10:16:42.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe91730a700 5 monclient: authenticate success, global_id 84184 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.711+0000 7fe91730a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient: dump: 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:42.721 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 >> v1:172.21.15.150:6789/0 conn(0x7fe8f40036f0 legacy=0x7fe8f4005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916b09700 1 -- 172.21.15.142:0/2042641575 reap_dead start 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 0x7fe8fc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): picked mon.c con 0x7fe8fc003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 0x7fe8fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): picked mon.b con 0x7fe8fc004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe8fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): picked mon.a con 0x7fe8fc0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91006fba0 con 0x7fe8fc0053c0 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc00c320 con 0x7fe8fc004810 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc00c4e0 con 0x7fe8fc003a60 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915306700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916b09700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 msgr2=0x7fe8fc00ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe91730a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916b09700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe8fc00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 0x7fe8fc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:42.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe8fc003a60 auth_method 0 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 0x7fe8fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe8fc004810 auth_method 0 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:42.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient(hunting): handle_auth_done global_id 84184 payload 293 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 msgr2=0x7fe8fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 0x7fe8fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 msgr2=0x7fe8fc00ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe8fc00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient: _finish_hunting 0 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 monclient: found mon.c 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc003570 con 0x7fe8fc003a60 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 10 monclient: _finish_auth 0 2024-09-18T10:16:42.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe916308700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:42.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.715+0000 7fe915b07700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 0x7fe8fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:42.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe916308700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 msgr2=0x7fe8fc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe916308700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 0x7fe8fc003e50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe90c006f00 tx=0x7fe90c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2bd822e4468838a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c00dc10 con 0x7fe8fc003a60 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: dump: 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:42.732 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c013d80 con 0x7fe8fc003a60 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 1 -- 172.21.15.142:0/2042641575 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c016670 con 0x7fe8fc003a60 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 4 set_mon_vals no callback set 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:42.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe915306700 10 monclient: dump: 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 10 monclient: shutdown 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:42.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 -- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 msgr2=0x7fe8fc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 0x7fe8fc003e50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe90c006f00 tx=0x7fe90c008040 comp rx=0 tx=0).stop 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 -- 172.21.15.142:0/2042641575 shutdown_connections 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe8fc00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc004810 0x7fe8fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 --2- 172.21.15.142:0/2042641575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc003a60 0x7fe8fc003e50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 -- 172.21.15.142:0/2042641575 >> 172.21.15.142:0/2042641575 conn(0x7fe9100a1fe0 msgr2=0x7fe910070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 -- 172.21.15.142:0/2042641575 shutdown_connections 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 1 -- 172.21.15.142:0/2042641575 wait complete. 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 10 monclient: build_initial_monmap 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.719+0000 7fe91730a700 10 monclient: monmap: 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:42.735 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 Processor -- start 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 -- start start 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient: init 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:42.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc003a60 0x7fe910129b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-b con 0x7fe8fc003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-c con 0x7fe8fc004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe9101307e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): picked mon.noname-a con 0x7fe8fc0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91012bcc0 con 0x7fe8fc0053c0 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91012a160 con 0x7fe8fc003a60 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91012b9e0 con 0x7fe8fc004810 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:42.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe91730a700 10 monclient(hunting): authenticate will time out at 1291.649356s 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916b09700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 msgr2=0x7fe9101307e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916b09700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe9101307e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc003a60 0x7fe910129b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42856/0 (socket says 172.21.15.142:42856) 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 -- 172.21.15.142:0/3974545357 learned_addr learned my addr 172.21.15.142:0/3974545357 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): get_auth_request con 0x7fe8fc004810 auth_method 0 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): get_auth_request con 0x7fe8fc003a60 auth_method 0 2024-09-18T10:16:42.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe916308700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient(hunting): handle_auth_done global_id 74345 payload 824 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 -- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc003a60 msgr2=0x7fe910129b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc003a60 0x7fe910129b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 -- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 msgr2=0x7fe9101307e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:42.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe9101307e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient: _finish_hunting 0 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 monclient: found mon.noname-c 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.723+0000 7fe915b07700 1 -- 172.21.15.142:0/3974545357 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91012bfa0 con 0x7fe8fc004810 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe915b07700 1 -- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 msgr2=0x7fe91009c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe915b07700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe900010680 tx=0x7fe900008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900015df0 con 0x7fe8fc004810 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: dump: 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:42.740 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 5 monclient: authenticate success, global_id 74345 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _renew_subs 2024-09-18T10:16:42.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90001c070 con 0x7fe8fc004810 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe910130d20 con 0x7fe8fc004810 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe914b05700 4 set_mon_vals no callback set 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _renew_subs 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe900018680 con 0x7fe8fc004810 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91013ec00 con 0x7fe8fc004810 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 10 monclient: dump: 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:42.742 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe910072a80 con 0x7fe8fc004810 2024-09-18T10:16:42.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe90001e030 con 0x7fe8fc004810 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 0x7fe8f40385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.727+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(205..205 src has 176..205) v4 ==== 11648+0+0 (secure 0 0 0) 0x7fe90004ae20 con 0x7fe8fc004810 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.731+0000 7fe916308700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 0x7fe8f40385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.731+0000 7fe916308700 10 monclient: get_auth_request con 0x7fe8f4036100 auth_method 0 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.731+0000 7fe916308700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 0x7fe8f40385b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe90c006f00 tx=0x7fe90c01e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.739+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe9000149a0 con 0x7fe8fc004810 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.739+0000 7fe9067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:42.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.739+0000 7fe9067fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:42.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.779+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 added bucket h17 type host to location {rack=r1} v206) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa914018030 con 0x7fa91c137a20 2024-09-18T10:16:42.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.779+0000 7fa9137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:42.784 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.779+0000 7fa9137fe700 10 monclient: _finish_command 2 = system:0 added bucket h17 type host to location {rack=r1} 2024-09-18T10:16:42.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa921dcb700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:42.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa921dcb700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:42.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa921dcb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:42.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa91c12fc40 con 0x7fa91c137a20 2024-09-18T10:16:42.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa9137fe700 1 -- 172.21.15.150:0/733112162 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h17' already exists v206) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa914018030 con 0x7fa91c137a20 2024-09-18T10:16:42.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa9137fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h17", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:42.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.887+0000 7fa9137fe700 10 monclient: _finish_command 3 = system:0 bucket 'h17' already exists 2024-09-18T10:16:42.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 msgr2=0x7fa8fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 0x7fa8fc037f70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa904009ad0 tx=0x7fa904009060 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 10 monclient: shutdown 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 msgr2=0x7fa91c12d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ef90 tx=0x7fa91400b040 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 shutdown_connections 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8fc035ac0 0x7fa8fc037f70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c137a20 0x7fa91c12d140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa91c0a0d70 0x7fa91c12cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 --2- 172.21.15.150:0/733112162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91c0a0290 0x7fa91c12c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 >> 172.21.15.150:0/733112162 conn(0x7fa91c0a58a0 msgr2=0x7fa91c09ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:42.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 shutdown_connections 2024-09-18T10:16:42.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.891+0000 7fa921dcb700 1 -- 172.21.15.150:0/733112162 wait complete. 2024-09-18T10:16:42.895 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h17' already exists 2024-09-18T10:16:42.903 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:42.904 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h18 host rack=r1 2024-09-18T10:16:42.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.955+0000 7fe91730a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.955+0000 7fe91730a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.955+0000 7fe91730a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.955+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe91012aaa0 con 0x7fe8fc004810 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe9067fc700 1 -- 172.21.15.142:0/3974545357 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fe9000147c0 con 0x7fe8fc004810 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe9067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:16:42.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe9067fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:42.962 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:16:42.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 msgr2=0x7fe8f40385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 0x7fe8f40385b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe90c006f00 tx=0x7fe90c01e000 comp rx=0 tx=0).stop 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 10 monclient: shutdown 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 msgr2=0x7fe91009c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe900010680 tx=0x7fe900008040 comp rx=0 tx=0).stop 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 shutdown_connections 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8f4036100 0x7fe8f40385b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8fc0053c0 0x7fe9101307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8fc004810 0x7fe91009c670 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 --2- 172.21.15.142:0/3974545357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8fc003a60 0x7fe910129b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:42.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 >> 172.21.15.142:0/3974545357 conn(0x7fe9100a1fe0 msgr2=0x7fe9101318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:42.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 shutdown_connections 2024-09-18T10:16:42.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:42.959+0000 7fe91730a700 1 -- 172.21.15.142:0/3974545357 wait complete. 2024-09-18T10:16:42.974 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:16:42.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:16:42.975 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:16:43.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: get_monmap_and_config 2024-09-18T10:16:43.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.003 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: monmap: 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:43.002399+0000 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:43.002399+0000 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:43.004 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 1 Processor -- start 2024-09-18T10:16:43.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 1 -- start start 2024-09-18T10:16:43.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: init 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:42.999+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f347409acd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3474096b40 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 0x7f3474138b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-b con 0x7f3474136790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3474095e70 con 0x7f347409acd0 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3474095920 con 0x7f3474096b40 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3474096250 con 0x7f3474136790 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient(hunting): authenticate will time out at 1285.096162s 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3472ffd700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f347409acd0 legacy=0x7f347409bfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:43.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34727fc700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3474096b40 0x7f347409c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40918/0 (socket says 172.21.15.150:40918) 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34727fc700 1 -- 172.21.15.150:0/1092580913 learned_addr learned my addr 172.21.15.150:0/1092580913 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 --2- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 0x7f3474138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): get_auth_request con 0x7f3474136790 auth_method 0 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient(hunting): handle_auth_done global_id 74348 payload 274 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 -- 172.21.15.150:0/1092580913 >> v1:172.21.15.150:6789/0 conn(0x7f3474096b40 legacy=0x7f347409c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 -- 172.21.15.150:0/1092580913 >> v1:172.21.15.142:6789/0 conn(0x7f347409acd0 legacy=0x7f347409bfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:43.008 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 monclient: found mon.noname-b 2024-09-18T10:16:43.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 -- 172.21.15.150:0/1092580913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347409b0a0 con 0x7f3474136790 2024-09-18T10:16:43.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 -- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 msgr2=0x7f3474138b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34737fe700 1 --2- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 0x7f3474138b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f347409d630 tx=0x7f34680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585759fe3c4ea048 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 1 -- 172.21.15.150:0/1092580913 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346800ba10 con 0x7f3474136790 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: dump: 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.011 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 1 -- 172.21.15.150:0/1092580913 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346800bb90 con 0x7f3474136790 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 5 monclient: authenticate success, global_id 74348 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 1 -- 172.21.15.150:0/1092580913 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011420 con 0x7f3474136790 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f34799fc700 4 set_mon_vals no callback set 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.003+0000 7f3471ffb700 10 monclient: dump: 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:43.012 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: shutdown 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 msgr2=0x7f3474138b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 --2- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 0x7f3474138b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f347409d630 tx=0x7f34680091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- 172.21.15.150:0/1092580913 shutdown_connections 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 --2- 172.21.15.150:0/1092580913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474136790 0x7f3474138b70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- 172.21.15.150:0/1092580913 >> 172.21.15.150:0/1092580913 conn(0x7f34740a15e0 msgr2=0x7f34740a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- 172.21.15.150:0/1092580913 shutdown_connections 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- 172.21.15.150:0/1092580913 wait complete. 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: monmap: 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.014 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:43.015 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:43.015 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:43.015 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 Processor -- start 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- start start 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: init 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-c con 0x7f3474096b40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 0x7f3474135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-a con 0x7f347409acd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:43.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 0x7f347408b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): picked mon.noname-b con 0x7f3474136790 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34740944b0 con 0x7f347409acd0 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347408d9e0 con 0x7f3474136790 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3474094f10 con 0x7f3474096b40 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34799fc700 10 monclient(hunting): authenticate will time out at 1285.103159s 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34727fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 msgr2=0x7f3474135f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34727fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 0x7f3474135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 0x7f347408b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 0x7f347408b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56654/0 (socket says 172.21.15.150:56654) 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44598/0 (socket says 172.21.15.150:44598) 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 1 -- 172.21.15.150:0/2701269513 learned_addr learned my addr 172.21.15.150:0/2701269513 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): get_auth_request con 0x7f3474136790 auth_method 0 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): get_auth_request con 0x7f3474096b40 auth_method 0 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f34737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient(hunting): handle_auth_done global_id 74351 payload 824 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 -- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 msgr2=0x7f347408b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 0x7f347408b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 -- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 msgr2=0x7f3474135f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 0x7f3474135f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 monclient: found mon.noname-c 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.007+0000 7f3472ffd700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468009420 con 0x7f3474096b40 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f3472ffd700 1 -- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 msgr2=0x7f34741359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f3472ffd700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f346c00bad0 tx=0x7f346c009760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c00ca50 con 0x7f3474096b40 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: dump: 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 5 monclient: authenticate success, global_id 74351 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 10 monclient: _renew_subs 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347406e8a0 con 0x7f3474096b40 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 10 monclient: _renew_subs 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347408d460 con 0x7f3474096b40 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c00cbb0 con 0x7f3474096b40 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c00cd60 con 0x7f3474096b40 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f345affd700 10 monclient: dump: 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f3470ff9700 4 set_mon_vals no callback set 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.011+0000 7f34799fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34799fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34799fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3474063ed0 con 0x7f3474096b40 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f346c012030 con 0x7f3474096b40 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f345affd700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 0x7f3460037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(206..206 src has 176..206) v4 ==== 11861+0+0 (secure 0 0 0) 0x7f346c049820 con 0x7f3474096b40 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34727fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 0x7f3460037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34727fc700 10 monclient: get_auth_request con 0x7f3460035aa0 auth_method 0 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.015+0000 7f34727fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 0x7f3460037f50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3464004860 tx=0x7f3464007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.023+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f346c0138e0 con 0x7f3474096b40 2024-09-18T10:16:43.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.023+0000 7f345affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.023+0000 7f345affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:43.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:16:43.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 10 monclient: monmap: 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:43.073352+0000 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:43.073352+0000 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:43.075 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:43.076 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 1 Processor -- start 2024-09-18T10:16:43.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 1 -- start start 2024-09-18T10:16:43.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.071+0000 7f193b59e700 10 monclient: init 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f193c0676f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 0x7f193c0975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f193c097160 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f193c1344a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1933fff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 msgr2=0x7f193c0975a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c098990 con 0x7f193c0676f0 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c098140 con 0x7f193c097160 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1933fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 0x7f193c0975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c09b670 con 0x7f193c1344a0 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient(hunting): authenticate will time out at 1292.000548s 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42858/0 (socket says 172.21.15.142:42858) 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 -- 172.21.15.142:0/1575161083 learned_addr learned my addr 172.21.15.142:0/1575161083 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:43.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): get_auth_request con 0x7f193c1344a0 auth_method 0 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient(hunting): handle_auth_done global_id 74354 payload 274 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 -- 172.21.15.142:0/1575161083 >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 msgr2=0x7f193c0975a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 --2- 172.21.15.142:0/1575161083 >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 0x7f193c0975a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 -- 172.21.15.142:0/1575161083 >> v1:172.21.15.150:6789/0 conn(0x7f193c0676f0 legacy=0x7f193c096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 monclient: found mon.noname-b 2024-09-18T10:16:43.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 -- 172.21.15.142:0/1575161083 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c067ac0 con 0x7f193c1344a0 2024-09-18T10:16:43.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 -- 172.21.15.142:0/1575161083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 msgr2=0x7f193c136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193ad9d700 1 --2- 172.21.15.142:0/1575161083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f193400d1a0 tx=0x7f193400ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98335fbcc8d9cb4c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 1 -- 172.21.15.142:0/1575161083 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934010070 con 0x7f193c1344a0 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: dump: 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.087 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 5 monclient: authenticate success, global_id 74354 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 1 -- 172.21.15.142:0/1575161083 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193400b2e0 con 0x7f193c1344a0 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 1 -- 172.21.15.142:0/1575161083 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934009860 con 0x7f193c1344a0 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f193b59e700 4 set_mon_vals no callback set 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.075+0000 7f1939d9b700 10 monclient: dump: 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: shutdown 2024-09-18T10:16:43.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- 172.21.15.142:0/1575161083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 msgr2=0x7f193c136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- 172.21.15.142:0/1575161083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f193400d1a0 tx=0x7f193400ba50 comp rx=0 tx=0).stop 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- 172.21.15.142:0/1575161083 shutdown_connections 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- 172.21.15.142:0/1575161083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c136880 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- 172.21.15.142:0/1575161083 >> v2:172.21.15.142:3300/0 conn(0x7f193c097160 0x7f193c0975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- 172.21.15.142:0/1575161083 >> 172.21.15.142:0/1575161083 conn(0x7f193c0714d0 msgr2=0x7f193c0a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- 172.21.15.142:0/1575161083 shutdown_connections 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- 172.21.15.142:0/1575161083 wait complete. 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: monmap: 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:43.090 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 Processor -- start 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- start start 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: init 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:16:43.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c0676f0 0x7f193c141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f193c0676f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c097160 0x7f193c141e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f193c097160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f193c1344a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c11cbb0 con 0x7f193c097160 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c09e400 con 0x7f193c0676f0 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c09a740 con 0x7f193c1344a0 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193b59e700 10 monclient(hunting): authenticate will time out at 1292.007247s 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f1933fff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c097160 msgr2=0x7f193c141e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f1933fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c097160 0x7f193c141e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42870/0 (socket says 172.21.15.142:42870) 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 learned_addr learned my addr 172.21.15.142:0/3975563181 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): get_auth_request con 0x7f193c1344a0 auth_method 0 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c0676f0 0x7f193c141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): get_auth_request con 0x7f193c0676f0 auth_method 0 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient(hunting): handle_auth_done global_id 74357 payload 824 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c0676f0 msgr2=0x7f193c141940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c0676f0 0x7f193c141940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c097160 msgr2=0x7f193c141e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c097160 0x7f193c141e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 1 monclient: found mon.noname-c 2024-09-18T10:16:43.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.079+0000 7f193ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193400bcf0 con 0x7f193c1344a0 2024-09-18T10:16:43.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 msgr2=0x7f193c1423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193ad9d700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f193400bba0 tx=0x7f1934019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193ad9d700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f193400bba0 tx=0x7f1934019000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f1934009de0 mon_map magic: 0 v1 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934009de0 con 0x7f193c1344a0 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.098 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1934014430 con 0x7f193c1344a0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f19327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 5 monclient: authenticate success, global_id 74357 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 10 monclient: _renew_subs 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193959a700 4 set_mon_vals no callback set 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c06d210 con 0x7f193c1344a0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 10 monclient: _renew_subs 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c11d510 con 0x7f193c1344a0 2024-09-18T10:16:43.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.083+0000 7f193b59e700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 0x7f193c0994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c064680 con 0x7f193c0990f0 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 0x7f193c0994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient: get_auth_request con 0x7f193c0990f0 auth_method 0 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 10 monclient(hunting): handle_auth_done global_id 74360 payload 824 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f193c06d930 con 0x7f193c0990f0 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19340147f0 con 0x7f193c1344a0 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.101 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1934014ba0 con 0x7f193c1344a0 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f191c035ac0 0x7f191c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f1933fff700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f191c035ac0 0x7f191c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f1933fff700 10 monclient: get_auth_request con 0x7f191c035ac0 auth_method 0 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 msgr2=0x7f193c0994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f193a59c700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 0x7f193c0994c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e7b0 tx=0x7f192c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c012940 con 0x7f193c0990f0 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f192c012aa0 con 0x7f193c0990f0 2024-09-18T10:16:43.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 msgr2=0x7f193c0994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 0x7f193c0994c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e7b0 tx=0x7f192c009060 comp rx=0 tx=0).stop 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 msgr2=0x7f193c0994c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.087+0000 7f19327fc700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c0990f0 0x7f193c0994c0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f192c00e7b0 tx=0x7f192c009060 comp rx=0 tx=0).stop 2024-09-18T10:16:43.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.091+0000 7f1933fff700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f191c035ac0 0x7f191c037f70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:43.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193b59e700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 0x7f193c06b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c072a80 con 0x7f193c099a00 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 0x7f193c06b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient: get_auth_request con 0x7f193c099a00 auth_method 0 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 10 monclient(hunting): handle_auth_done global_id 74363 payload 824 2024-09-18T10:16:43.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.099+0000 7f193a59c700 1 -- 172.21.15.142:0/3975563181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f193c11c780 con 0x7f193c099a00 2024-09-18T10:16:43.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f193a59c700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 msgr2=0x7f193c06b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f193a59c700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 0x7f193c06b0a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f192c017000 tx=0x7f192c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c01e860 con 0x7f193c099a00 2024-09-18T10:16:43.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.108 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f192c01e9c0 con 0x7f193c099a00 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 msgr2=0x7f193c06b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 0x7f193c06b0a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f192c017000 tx=0x7f192c015000 comp rx=0 tx=0).stop 2024-09-18T10:16:43.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 msgr2=0x7f193c06b0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f19327fc700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c099a00 0x7f193c06b0a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 82, 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:43.110 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.111 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.112 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:43.113 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.114 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.115 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.116 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.117 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.118 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.119 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.120 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.121 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:43.122 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f191c035ac0 msgr2=0x7f191c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.103+0000 7f193b59e700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f191c035ac0 0x7f191c037f70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1924006fd0 tx=0x7f1924006e10 comp rx=0 tx=0).stop 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193ad9d700 1 -- 172.21.15.142:0/3975563181 reap_dead start 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 10 monclient: shutdown 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:43.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 msgr2=0x7f193c1423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f193400bba0 tx=0x7f1934019000 comp rx=0 tx=0).stop 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f19327fc700 1 -- 172.21.15.142:0/3975563181 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(206..206 src has 176..206) v4 ==== 11861+0+0 (unknown 0 0 0) 0x7f193404bf10 con 0x7f193c1344a0 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 shutdown_connections 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 --2- 172.21.15.142:0/3975563181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c1344a0 0x7f193c1423c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 >> 172.21.15.142:0/3975563181 conn(0x7f193c0714d0 msgr2=0x7f193c130720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 shutdown_connections 2024-09-18T10:16:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.107+0000 7f193b59e700 1 -- 172.21.15.142:0/3975563181 wait complete. 2024-09-18T10:16:43.125 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:16:43.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:16:43.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:16:43.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.195+0000 7f34799fc700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:43.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.195+0000 7f34799fc700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:43.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.195+0000 7f34799fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:43.195+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f347408e0f0 con 0x7f3474096b40 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: get_monmap_and_config 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: monmap: 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:16:43.228375+0000 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:16:43.228375+0000 2024-09-18T10:16:43.230 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 Processor -- start 2024-09-18T10:16:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 -- start start 2024-09-18T10:16:43.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: init 2024-09-18T10:16:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:16:43.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12c009c760 0x7f12c009b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-c con 0x7f12c009c760 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12c009b630 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 0x7f12c0135880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-a con 0x7f12c009d240 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c0098b00 con 0x7f12c009b630 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c0098d20 con 0x7f12c009d240 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12c009a100 con 0x7f12c009c760 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12c4b67700 10 monclient(hunting): authenticate will time out at 1292.154093s 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12b6ffd700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f12c009b630 legacy=0x7f12c009cb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:43.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bffff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 msgr2=0x7f12c0135880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 0x7f12c0135880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12c009c760 0x7f12c009b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f12c009c760 0x7f12c009b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:57098/0 (socket says 172.21.15.142:57098) 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 -- 172.21.15.142:0/4109054282 learned_addr learned my addr 172.21.15.142:0/4109054282 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request con 0x7f12c009c760 auth_method 0 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_done global_id 84196 payload 274 2024-09-18T10:16:43.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 -- 172.21.15.142:0/4109054282 >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 msgr2=0x7f12c0135880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/4109054282 >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 0x7f12c0135880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 -- 172.21.15.142:0/4109054282 >> v1:172.21.15.142:6789/0 conn(0x7f12c009b630 legacy=0x7f12c009cb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.227+0000 7f12bf7fe700 1 -- 172.21.15.142:0/4109054282 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c009a3e0 con 0x7f12c009c760 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12bf7fe700 1 -- 172.21.15.142:0/4109054282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009c760 msgr2=0x7f12c009b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/4109054282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009c760 0x7f12c009b0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12c0076de0 tx=0x7f12b0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d7487ea853ea5ed server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 1 -- 172.21.15.142:0/4109054282 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b000ba70 con 0x7f12c009c760 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:43.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: dump: 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 5 monclient: authenticate success, global_id 84196 2024-09-18T10:16:43.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 1 -- 172.21.15.142:0/4109054282 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b000bbd0 con 0x7f12c009c760 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 1 -- 172.21.15.142:0/4109054282 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0011420 con 0x7f12c009c760 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 4 set_mon_vals no callback set 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12beffd700 10 monclient: dump: 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.247 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 10 monclient: shutdown 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- 172.21.15.142:0/4109054282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009c760 msgr2=0x7f12c009b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 --2- 172.21.15.142:0/4109054282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009c760 0x7f12c009b0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12c0076de0 tx=0x7f12b0009250 comp rx=0 tx=0).stop 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- 172.21.15.142:0/4109054282 shutdown_connections 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 --2- 172.21.15.142:0/4109054282 >> v2:172.21.15.142:3300/0 conn(0x7f12c009d240 0x7f12c0135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 --2- 172.21.15.142:0/4109054282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009c760 0x7f12c009b0f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- 172.21.15.142:0/4109054282 >> 172.21.15.142:0/4109054282 conn(0x7f12c0070ef0 msgr2=0x7f12c00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- 172.21.15.142:0/4109054282 shutdown_connections 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- 172.21.15.142:0/4109054282 wait complete. 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 10 monclient: build_initial_monmap 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 10 monclient: monmap: 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:43.249 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 Processor -- start 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 1 -- start start 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.231+0000 7f12c4b67700 10 monclient: init 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009b630 0x7f12c0141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-b con 0x7f12c009b630 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 0x7f12c0141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-c con 0x7f12c009c760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12c009d240 0x7f12c008dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): picked mon.noname-a con 0x7f12c009d240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c009e710 con 0x7f12c009d240 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0091690 con 0x7f12c009b630 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c006f810 con 0x7f12c009c760 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12c4b67700 10 monclient(hunting): authenticate will time out at 1292.160496s 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12c009d240 msgr2=0x7f12c008dd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12c009d240 0x7f12c008dd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 0x7f12c0141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009b630 0x7f12c0141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 0x7f12c0141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42896/0 (socket says 172.21.15.142:42896) 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request con 0x7f12c009b630 auth_method 0 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 -- 172.21.15.142:0/1337127141 learned_addr learned my addr 172.21.15.142:0/1337127141 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): get_auth_request con 0x7f12c009c760 auth_method 0 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient(hunting): handle_auth_done global_id 74366 payload 824 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009b630 msgr2=0x7f12c0141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009b630 0x7f12c0141990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12c009d240 msgr2=0x7f12c008dd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12c009d240 0x7f12c008dd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 monclient: found mon.noname-c 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00094f0 con 0x7f12c009c760 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c009b630 0x7f12c0141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 msgr2=0x7f12c0141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.235+0000 7f12b6ffd700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 0x7f12c0141ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12a800b7e0 tx=0x7f12a8009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003e20 con 0x7f12c009c760 2024-09-18T10:16:43.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: dump: 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: _finish_auth 0 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 5 monclient: authenticate success, global_id 74366 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 monclient: _renew_subs 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c0091c50 con 0x7f12c009c760 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8006150 con 0x7f12c009c760 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a80064d0 con 0x7f12c009c760 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 10 monclient: dump: 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.255 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 monclient: _renew_subs 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c0142410 con 0x7f12c009c760 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bdffb700 4 set_mon_vals no callback set 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 0x7f12c008e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00648a0 con 0x7f12c0092600 2024-09-18T10:16:43.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f12a80103c0 con 0x7f12c009c760 2024-09-18T10:16:43.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 0x7f12c008e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient: get_auth_request con 0x7f12c0092600 auth_method 0 2024-09-18T10:16:43.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12a0035a70 0x7f12a0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(206..206 src has 176..206) v4 ==== 11861+0+0 (secure 0 0 0) 0x7f12a8049780 con 0x7f12c009c760 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bffff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12a0035a70 0x7f12a0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bffff700 10 monclient: get_auth_request con 0x7f12a0035a70 auth_method 0 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_done global_id 84202 payload 824 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.239+0000 7f12bf7fe700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12c0072a80 con 0x7f12c0092600 2024-09-18T10:16:43.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12bf7fe700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 msgr2=0x7f12c008e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 0x7f12c008e2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12c009ba90 tx=0x7f12b0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0021070 con 0x7f12c0092600 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12bffff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12a0035a70 0x7f12a0037f20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f12b8003d50 tx=0x7f12b8009370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 10 monclient: dump: 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f12b001c870 con 0x7f12c0092600 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 msgr2=0x7f12c008e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 0x7f12c008e2c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12c009ba90 tx=0x7f12b0016000 comp rx=0 tx=0).stop 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 msgr2=0x7f12c008e2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:43.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.243+0000 7f12b7fff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c0092600 0x7f12c008e2c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12c4b67700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12c4b67700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 0x7f12c008ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12c4b67700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c00913b0 con 0x7f12c008e800 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 0x7f12c008ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient: get_auth_request con 0x7f12c008e800 auth_method 0 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 10 monclient(hunting): handle_auth_done global_id 84205 payload 824 2024-09-18T10:16:43.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.255+0000 7f12bf7fe700 1 -- 172.21.15.142:0/1337127141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f12c008f110 con 0x7f12c008e800 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12bf7fe700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 msgr2=0x7f12c008ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12bf7fe700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 0x7f12c008ebd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b0003080 tx=0x7f12b001e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b0038070 con 0x7f12c008e800 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 10 monclient: dump: 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:43.265 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f12b001c660 con 0x7f12c008e800 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 msgr2=0x7f12c008ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 0x7f12c008ebd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b0003080 tx=0x7f12b001e070 comp rx=0 tx=0).stop 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 msgr2=0x7f12c008ebd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:16:43.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12b7fff700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12c008e800 0x7f12c008ebd0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f12b0003080 tx=0x7f12b001e070 comp rx=0 tx=0).stop 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 82, 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.267 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.268 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:16:43.270 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:16:43.271 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.272 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.274 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.275 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:16:43.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.278 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:16:43.279 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:16:43.280 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:16:43.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12a0035a70 msgr2=0x7f12a0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12a0035a70 0x7f12a0037f20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f12b8003d50 tx=0x7f12b8009370 comp rx=0 tx=0).stop 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 10 monclient: shutdown 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 msgr2=0x7f12c0141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 --2- 172.21.15.142:0/1337127141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12c009c760 0x7f12c0141ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12a800b7e0 tx=0x7f12a8009220 comp rx=0 tx=0).stop 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12bffff700 1 -- 172.21.15.142:0/1337127141 reap_dead start 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12bffff700 1 -- 172.21.15.142:0/1337127141 reap_dead start 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 shutdown_connections 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 >> 172.21.15.142:0/1337127141 conn(0x7f12c0070ef0 msgr2=0x7f12c0096350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 shutdown_connections 2024-09-18T10:16:43.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:16:43.259+0000 7f12c4b67700 1 -- 172.21.15.142:0/1337127141 wait complete. 2024-09-18T10:16:43.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:16:43.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:16:44.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.634+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 added bucket h18 type host to location {rack=r1} v207) v1 ==== 168+0+0 (secure 0 0 0) 0x7f346c01d2f0 con 0x7f3474096b40 2024-09-18T10:16:44.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.634+0000 7f345affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:44.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.634+0000 7f345affd700 10 monclient: _finish_command 2 = system:0 added bucket h18 type host to location {rack=r1} 2024-09-18T10:16:44.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.738+0000 7f34799fc700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:44.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.738+0000 7f34799fc700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:44.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.738+0000 7f34799fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:44.743 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.738+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f3474067b10 con 0x7f3474096b40 2024-09-18T10:16:44.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f345affd700 1 -- 172.21.15.150:0/2701269513 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h18' already exists v207) v1 ==== 147+0+0 (secure 0 0 0) 0x7f346c0132e0 con 0x7f3474096b40 2024-09-18T10:16:44.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f345affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h18", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:44.745 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f345affd700 10 monclient: _finish_command 3 = system:0 bucket 'h18' already exists 2024-09-18T10:16:44.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 msgr2=0x7f3460037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 0x7f3460037f50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3464004860 tx=0x7f3464007c10 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 10 monclient: shutdown 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 msgr2=0x7f34741359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f346c00bad0 tx=0x7f346c009760 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 shutdown_connections 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3460035aa0 0x7f3460037f50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3474136790 0x7f347408b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f347409acd0 0x7f3474135f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 --2- 172.21.15.150:0/2701269513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3474096b40 0x7f34741359d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 >> 172.21.15.150:0/2701269513 conn(0x7f34740a15e0 msgr2=0x7f34740a00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:44.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 shutdown_connections 2024-09-18T10:16:44.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.742+0000 7f34799fc700 1 -- 172.21.15.150:0/2701269513 wait complete. 2024-09-18T10:16:44.749 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h18' already exists 2024-09-18T10:16:44.761 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:44.762 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h19 host rack=r1 2024-09-18T10:16:44.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 10 monclient: get_monmap_and_config 2024-09-18T10:16:44.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 10 monclient: build_initial_monmap 2024-09-18T10:16:44.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 10 monclient: monmap: 2024-09-18T10:16:44.863 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:44.862390+0000 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:44.862390+0000 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:44.864 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 1 Processor -- start 2024-09-18T10:16:44.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 1 -- start start 2024-09-18T10:16:44.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.858+0000 7fd424c6e700 10 monclient: init 2024-09-18T10:16:44.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:44.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T10:16:44.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4200710a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 0x7fd4200975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-a con 0x7fd420097180 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4201345d0 0x7fd4201369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-c con 0x7fd4201345d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420071470 con 0x7fd4200710a0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd420099320 con 0x7fd420097180 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd420098290 con 0x7fd4201345d0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd424c6e700 10 monclient(hunting): authenticate will time out at 1286.957369s 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4201345d0 0x7fd4201369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:44.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41dd9b700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 msgr2=0x7fd4200975e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41e59c700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fd4200710a0 legacy=0x7fd420096a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41dd9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 0x7fd4200975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4201345d0 0x7fd4201369b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56668/0 (socket says 172.21.15.150:56668) 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 -- 172.21.15.150:0/328826464 learned_addr learned my addr 172.21.15.150:0/328826464 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): get_auth_request con 0x7fd4201345d0 auth_method 0 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_done global_id 84208 payload 274 2024-09-18T10:16:44.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 -- 172.21.15.150:0/328826464 >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 msgr2=0x7fd4200975e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 --2- 172.21.15.150:0/328826464 >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 0x7fd4200975e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 -- 172.21.15.150:0/328826464 >> v1:172.21.15.142:6789/0 conn(0x7fd4200710a0 legacy=0x7fd420096a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:44.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.862+0000 7fd41ed9d700 1 -- 172.21.15.150:0/328826464 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd420098570 con 0x7fd4201345d0 2024-09-18T10:16:44.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41ed9d700 1 -- 172.21.15.150:0/328826464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4201345d0 msgr2=0x7fd4201369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:44.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41ed9d700 1 --2- 172.21.15.150:0/328826464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4201345d0 0x7fd4201369b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd42009dce0 tx=0x7fd418009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24ce321e27407fb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:44.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 1 -- 172.21.15.150:0/328826464 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd41800e070 con 0x7fd4201345d0 2024-09-18T10:16:44.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: dump: 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: _finish_auth 0 2024-09-18T10:16:44.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 5 monclient: authenticate success, global_id 84208 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 1 -- 172.21.15.150:0/328826464 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41800bc60 con 0x7fd4201345d0 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 4 set_mon_vals no callback set 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 1 -- 172.21.15.150:0/328826464 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd418012430 con 0x7fd4201345d0 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd41d59a700 10 monclient: dump: 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:44.874 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 10 monclient: shutdown 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- 172.21.15.150:0/328826464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4201345d0 msgr2=0x7fd4201369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 --2- 172.21.15.150:0/328826464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4201345d0 0x7fd4201369b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd42009dce0 tx=0x7fd418009840 comp rx=0 tx=0).stop 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- 172.21.15.150:0/328826464 shutdown_connections 2024-09-18T10:16:44.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 --2- 172.21.15.150:0/328826464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4201345d0 0x7fd4201369b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 --2- 172.21.15.150:0/328826464 >> v2:172.21.15.142:3300/0 conn(0x7fd420097180 0x7fd4200975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- 172.21.15.150:0/328826464 >> 172.21.15.150:0/328826464 conn(0x7fd420068590 msgr2=0x7fd4200a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- 172.21.15.150:0/328826464 shutdown_connections 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- 172.21.15.150:0/328826464 wait complete. 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 10 monclient: build_initial_monmap 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 10 monclient: monmap: 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:44.876 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 Processor -- start 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 1 -- start start 2024-09-18T10:16:44.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.866+0000 7fd424c6e700 10 monclient: init 2024-09-18T10:16:44.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:44.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:44.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 0x7fd42009c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-a con 0x7fd4200710a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-b con 0x7fd420097180 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4201345d0 0x7fd42006d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): picked mon.noname-c con 0x7fd4201345d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4200a1090 con 0x7fd4200710a0 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42011b6f0 con 0x7fd420097180 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:44.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42011b410 con 0x7fd4201345d0 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient(hunting): authenticate will time out at 1286.964305s 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41e59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 msgr2=0x7fd42009c840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 0x7fd42009c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56682/0 (socket says 172.21.15.150:56682) 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 -- 172.21.15.150:0/4220481733 learned_addr learned my addr 172.21.15.150:0/4220481733 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): get_auth_request con 0x7fd420097180 auth_method 0 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:44.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4201345d0 0x7fd42006d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): get_auth_request con 0x7fd4201345d0 auth_method 0 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:44.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient(hunting): handle_auth_done global_id 84211 payload 824 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 -- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4201345d0 msgr2=0x7fd42006d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4201345d0 0x7fd42006d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 -- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 msgr2=0x7fd42009c840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 0x7fd42009c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 monclient: found mon.noname-b 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd418009ae0 con 0x7fd420097180 2024-09-18T10:16:44.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 -- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 msgr2=0x7fd42006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41dd9b700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd41400b810 tx=0x7fd414009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414003e20 con 0x7fd420097180 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: dump: 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:44.889 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 5 monclient: authenticate success, global_id 84211 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414006150 con 0x7fd420097180 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _renew_subs 2024-09-18T10:16:44.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42006eb30 con 0x7fd420097180 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd414006510 con 0x7fd420097180 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:44.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd40e7fc700 10 monclient: dump: 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:44.892 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:44.893 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:44.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _renew_subs 2024-09-18T10:16:44.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:44.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42011b130 con 0x7fd420097180 2024-09-18T10:16:44.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.870+0000 7fd41cd99700 4 set_mon_vals no callback set 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd41401b3b0 con 0x7fd420097180 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd424c6e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd424c6e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd424c6e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd420063ed0 con 0x7fd420097180 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd40e7fc700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 0x7fd400037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(207..207 src has 176..207) v4 ==== 12074+0+0 (secure 0 0 0) 0x7fd414049600 con 0x7fd420097180 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd41e59c700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 0x7fd400037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.874+0000 7fd41e59c700 10 monclient: get_auth_request con 0x7fd400035ac0 auth_method 0 2024-09-18T10:16:44.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.878+0000 7fd41e59c700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 0x7fd400037f70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd410003900 tx=0x7fd410006dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:44.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.882+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd41400e820 con 0x7fd420097180 2024-09-18T10:16:44.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.882+0000 7fd40e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:44.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:44.882+0000 7fd40e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:45.055 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:45.050+0000 7fd424c6e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:45.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:45.050+0000 7fd424c6e700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:45.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:45.050+0000 7fd424c6e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:45.056 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:45.050+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd42011b8a0 con 0x7fd420097180 2024-09-18T10:16:46.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.042+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 added bucket h19 type host to location {rack=r1} v208) v1 ==== 168+0+0 (secure 0 0 0) 0x7fd414017070 con 0x7fd420097180 2024-09-18T10:16:46.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.042+0000 7fd40e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.042+0000 7fd40e7fc700 10 monclient: _finish_command 2 = system:0 added bucket h19 type host to location {rack=r1} 2024-09-18T10:16:46.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd424c6e700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd424c6e700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd424c6e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fd420071900 con 0x7fd420097180 2024-09-18T10:16:46.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd40e7fc700 1 -- 172.21.15.150:0/4220481733 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h19' already exists v208) v1 ==== 147+0+0 (secure 0 0 0) 0x7fd41400e260 con 0x7fd420097180 2024-09-18T10:16:46.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd40e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h19", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.150+0000 7fd40e7fc700 10 monclient: _finish_command 3 = system:0 bucket 'h19' already exists 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 msgr2=0x7fd400037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 0x7fd400037f70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd410003900 tx=0x7fd410006dc0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 10 monclient: shutdown 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 msgr2=0x7fd42006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd41400b810 tx=0x7fd414009220 comp rx=0 tx=0).stop 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 shutdown_connections 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd400035ac0 0x7fd400037f70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4201345d0 0x7fd42006d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd420097180 0x7fd42006d360 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 --2- 172.21.15.150:0/4220481733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4200710a0 0x7fd42009c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 >> 172.21.15.150:0/4220481733 conn(0x7fd420068590 msgr2=0x7fd420099c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:46.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 shutdown_connections 2024-09-18T10:16:46.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.154+0000 7fd424c6e700 1 -- 172.21.15.150:0/4220481733 wait complete. 2024-09-18T10:16:46.159 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h19' already exists 2024-09-18T10:16:46.168 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:46.168 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h20 host rack=r1 2024-09-18T10:16:46.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: get_monmap_and_config 2024-09-18T10:16:46.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: build_initial_monmap 2024-09-18T10:16:46.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: monmap: 2024-09-18T10:16:46.274 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:46.273406+0000 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:46.273406+0000 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:46.275 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 Processor -- start 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 -- start start 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: init 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 0x7f7f5c09b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-a con 0x7f7f5c09b2b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7f5c0955b0 0x7f7f5c097a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-c con 0x7f7f5c0955b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:46.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f5c097fb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c09aab0 con 0x7f7f5c097fb0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c09a7d0 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0992f0 con 0x7f7f5c0955b0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f629fb700 10 monclient(hunting): authenticate will time out at 1288.366733s 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5bfff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 msgr2=0x7f7f5c09b680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f60f98700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7f5c097fb0 0x7f7f5c09c860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40928/0 (socket says 172.21.15.150:40928) 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f60f98700 1 -- 172.21.15.150:0/1682475635 learned_addr learned my addr 172.21.15.150:0/1682475635 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5bfff700 1 --2- 172.21.15.150:0/1682475635 >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 0x7f7f5c09b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 --2- 172.21.15.150:0/1682475635 >> v2:172.21.15.150:3300/0 conn(0x7f7f5c0955b0 0x7f7f5c097a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:46.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): get_auth_request con 0x7f7f5c0955b0 auth_method 0 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient(hunting): handle_auth_done global_id 84214 payload 274 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 -- 172.21.15.150:0/1682475635 >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 msgr2=0x7f7f5c09b680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 --2- 172.21.15.150:0/1682475635 >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 0x7f7f5c09b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 -- 172.21.15.150:0/1682475635 >> v1:172.21.15.150:6789/0 conn(0x7f7f5c097fb0 legacy=0x7f7f5c09c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:46.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:16:46.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:46.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.270+0000 7f7f5b7fe700 1 -- 172.21.15.150:0/1682475635 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c09a4f0 con 0x7f7f5c0955b0 2024-09-18T10:16:46.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5b7fe700 1 -- 172.21.15.150:0/1682475635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c0955b0 msgr2=0x7f7f5c097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5b7fe700 1 --2- 172.21.15.150:0/1682475635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c0955b0 0x7f7f5c097a70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f440098e0 tx=0x7f7f440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34804c735435ed0a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 1 -- 172.21.15.150:0/1682475635 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4400ba30 con 0x7f7f5c0955b0 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: dump: 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:46.284 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: _finish_auth 0 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 5 monclient: authenticate success, global_id 84214 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 1 -- 172.21.15.150:0/1682475635 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4400bbb0 con 0x7f7f5c0955b0 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 4 set_mon_vals no callback set 2024-09-18T10:16:46.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 1 -- 172.21.15.150:0/1682475635 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f44011420 con 0x7f7f5c0955b0 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f5affd700 10 monclient: dump: 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:46.286 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 10 monclient: shutdown 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 -- 172.21.15.150:0/1682475635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c0955b0 msgr2=0x7f7f5c097a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 --2- 172.21.15.150:0/1682475635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c0955b0 0x7f7f5c097a70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f440098e0 tx=0x7f7f440091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 -- 172.21.15.150:0/1682475635 shutdown_connections 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 --2- 172.21.15.150:0/1682475635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c0955b0 0x7f7f5c097a70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 --2- 172.21.15.150:0/1682475635 >> v2:172.21.15.142:3300/0 conn(0x7f7f5c09b2b0 0x7f7f5c09b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 -- 172.21.15.150:0/1682475635 >> 172.21.15.150:0/1682475635 conn(0x7f7f5c0a15e0 msgr2=0x7f7f5c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 -- 172.21.15.150:0/1682475635 shutdown_connections 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 -- 172.21.15.150:0/1682475635 wait complete. 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 10 monclient: build_initial_monmap 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 10 monclient: monmap: 2024-09-18T10:16:46.287 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.274+0000 7f7f629fb700 1 Processor -- start 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 -- start start 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient: init 2024-09-18T10:16:46.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 0x7f7f5c093c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-c con 0x7f7f5c0955b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 0x7f7f5c08e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-a con 0x7f7f5c097fb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): picked mon.noname-b con 0x7f7f5c09b2b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c091dc0 con 0x7f7f5c097fb0 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0904c0 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:46.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c091ae0 con 0x7f7f5c0955b0 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient(hunting): authenticate will time out at 1288.372539s 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5b7fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 msgr2=0x7f7f5c08e640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 0x7f7f5c08e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56694/0 (socket says 172.21.15.150:56694) 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 -- 172.21.15.150:0/2602311144 learned_addr learned my addr 172.21.15.150:0/2602311144 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): get_auth_request con 0x7f7f5c09b2b0 auth_method 0 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:46.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 0x7f7f5c093c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 10 monclient(hunting): get_auth_request con 0x7f7f5c0955b0 auth_method 0 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient(hunting): handle_auth_done global_id 84217 payload 824 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 -- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 msgr2=0x7f7f5c093c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 0x7f7f5c093c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 -- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 msgr2=0x7f7f5c08e640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 0x7f7f5c08e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient: _finish_hunting 0 2024-09-18T10:16:46.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 monclient: found mon.noname-b 2024-09-18T10:16:46.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:46.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f44009420 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5bfff700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 0x7f7f5c093c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 -- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 msgr2=0x7f7f5c08eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f60f98700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400ccc0 tx=0x7f7f54005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400ce70 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: dump: 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:46.294 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: _finish_auth 0 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 5 monclient: authenticate success, global_id 84217 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient: _renew_subs 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5400ad80 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c0920a0 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54007360 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f5a7fc700 4 set_mon_vals no callback set 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.278+0000 7f7f53fff700 10 monclient: dump: 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:46.296 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 10 monclient: _renew_subs 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c094c60 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c0983d0 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7f54007710 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f53fff700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 0x7f7f3c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(208..208 src has 176..208) v4 ==== 12287+0+0 (secure 0 0 0) 0x7f7f5404a6a0 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f5bfff700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 0x7f7f3c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.282+0000 7f7f5bfff700 10 monclient: get_auth_request con 0x7f7f3c035a70 auth_method 0 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.286+0000 7f7f5bfff700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 0x7f7f3c037f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c004fd0 tx=0x7f7f4c004aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.290+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7f54019900 con 0x7f7f5c09b2b0 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.290+0000 7f7f53fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:46.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.290+0000 7f7f53fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:46.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.454+0000 7f7f629fb700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.454+0000 7f7f629fb700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:46.460 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.454+0000 7f7f629fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:46.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:46.454+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7f5c09d000 con 0x7f7f5c09b2b0 2024-09-18T10:16:47.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.046+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 added bucket h20 type host to location {rack=r1} v209) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7f5401a360 con 0x7f7f5c09b2b0 2024-09-18T10:16:47.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.046+0000 7f7f53fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.046+0000 7f7f53fff700 10 monclient: _finish_command 2 = system:0 added bucket h20 type host to location {rack=r1} 2024-09-18T10:16:47.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.150+0000 7f7f629fb700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.150+0000 7f7f629fb700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.150+0000 7f7f629fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.150+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7f5c092230 con 0x7f7f5c09b2b0 2024-09-18T10:16:47.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f53fff700 1 -- 172.21.15.150:0/2602311144 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h20' already exists v209) v1 ==== 147+0+0 (secure 0 0 0) 0x7f7f54019300 con 0x7f7f5c09b2b0 2024-09-18T10:16:47.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f53fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h20", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f53fff700 10 monclient: _finish_command 3 = system:0 bucket 'h20' already exists 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 msgr2=0x7f7f3c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 0x7f7f3c037f20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c004fd0 tx=0x7f7f4c004aa0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 10 monclient: shutdown 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 msgr2=0x7f7f5c08eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400ccc0 tx=0x7f7f54005400 comp rx=0 tx=0).stop 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 shutdown_connections 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f3c035a70 0x7f7f3c037f20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f5c09b2b0 0x7f7f5c08eb80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f5c097fb0 0x7f7f5c08e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 --2- 172.21.15.150:0/2602311144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f5c0955b0 0x7f7f5c093c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 >> 172.21.15.150:0/2602311144 conn(0x7f7f5c0a15e0 msgr2=0x7f7f5c09f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 shutdown_connections 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.154+0000 7f7f629fb700 1 -- 172.21.15.150:0/2602311144 wait complete. 2024-09-18T10:16:47.161 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h20' already exists 2024-09-18T10:16:47.170 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:47.170 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h21 host rack=r1 2024-09-18T10:16:47.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: get_monmap_and_config 2024-09-18T10:16:47.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: build_initial_monmap 2024-09-18T10:16:47.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: monmap: 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:47.273442+0000 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:47.273442+0000 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:47.275 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 Processor -- start 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 -- start start 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: init 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:16:47.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 0x7fe4ac096b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-a con 0x7fe4ac09acd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-c con 0x7fe4ac09bfd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac136780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-b con 0x7fe4ac09c8e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac096250 con 0x7fe4ac09acd0 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac095e70 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac095920 con 0x7fe4ac09c8e0 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:47.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4b13ee700 10 monclient(hunting): authenticate will time out at 1289.367447s 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4aaffd700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 msgr2=0x7fe4ac096b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4aaffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 0x7fe4ac096b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56706/0 (socket says 172.21.15.150:56706) 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 1 -- 172.21.15.150:0/2200102142 learned_addr learned my addr 172.21.15.150:0/2200102142 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): get_auth_request con 0x7fe4ac09bfd0 auth_method 0 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4ab7fe700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac136780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:47.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4ab7fe700 10 monclient(hunting): get_auth_request con 0x7fe4ac09c8e0 auth_method 0 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.270+0000 7fe4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_done global_id 84220 payload 274 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 -- 172.21.15.150:0/2200102142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 msgr2=0x7fe4ac136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac136780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 -- 172.21.15.150:0/2200102142 >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 msgr2=0x7fe4ac096b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/2200102142 >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 0x7fe4ac096b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:47.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 monclient: found mon.noname-c 2024-09-18T10:16:47.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:47.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 -- 172.21.15.150:0/2200102142 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac09b0a0 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 -- 172.21.15.150:0/2200102142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 msgr2=0x7fe4ac09c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:47.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac09d630 tx=0x7fe494009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136f7482a6d6cacf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 1 -- 172.21.15.150:0/2200102142 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494009e70 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: dump: 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:47.286 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4b13ee700 5 monclient: authenticate success, global_id 84220 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4b13ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 1 -- 172.21.15.150:0/2200102142 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49400ba70 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4b13ee700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4b13ee700 4 set_mon_vals no callback set 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 1 -- 172.21.15.150:0/2200102142 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400be30 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.274+0000 7fe4aa7fc700 10 monclient: dump: 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:47.288 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: shutdown 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- 172.21.15.150:0/2200102142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 msgr2=0x7fe4ac09c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac09d630 tx=0x7fe494009250 comp rx=0 tx=0).stop 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- 172.21.15.150:0/2200102142 shutdown_connections 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac136780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/2200102142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac09c3a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/2200102142 >> v2:172.21.15.142:3300/0 conn(0x7fe4ac09acd0 0x7fe4ac096b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- 172.21.15.150:0/2200102142 >> 172.21.15.150:0/2200102142 conn(0x7fe4ac0a15e0 msgr2=0x7fe4ac0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- 172.21.15.150:0/2200102142 shutdown_connections 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- 172.21.15.150:0/2200102142 wait complete. 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: build_initial_monmap 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: monmap: 2024-09-18T10:16:47.289 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 Processor -- start 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- start start 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: init 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:47.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 0x7fe4ac089ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-a con 0x7fe4ac09acd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-b con 0x7fe4ac09bfd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac08a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): picked mon.noname-c con 0x7fe4ac09c8e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac08bdd0 con 0x7fe4ac09acd0 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06f530 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac09a2f0 con 0x7fe4ac09c8e0 2024-09-18T10:16:47.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4b13ee700 10 monclient(hunting): authenticate will time out at 1289.374434s 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4aaffd700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 msgr2=0x7fe4ac089ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4aaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 0x7fe4ac089ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56722/0 (socket says 172.21.15.150:56722) 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 -- 172.21.15.150:0/3438286210 learned_addr learned my addr 172.21.15.150:0/3438286210 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): get_auth_request con 0x7fe4ac09bfd0 auth_method 0 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac08a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:47.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): get_auth_request con 0x7fe4ac09c8e0 auth_method 0 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient(hunting): handle_auth_done global_id 84223 payload 824 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 -- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 msgr2=0x7fe4ac08a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac08a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 -- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 msgr2=0x7fe4ac089ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 0x7fe4ac089ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 monclient: found mon.noname-b 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.278+0000 7fe4a3fff700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4940094f0 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a3fff700 1 -- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 msgr2=0x7fe4ac08a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:47.296 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a3fff700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac095fb0 tx=0x7fe494017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49401f070 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: dump: 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:47.297 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 5 monclient: authenticate success, global_id 84223 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 10 monclient: _renew_subs 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac08d380 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940117b0 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494011b90 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:47.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a9ffb700 4 set_mon_vals no callback set 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4a2ffd700 10 monclient: dump: 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 10 monclient: _renew_subs 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.282+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac08ce40 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4b13ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4b13ee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4b13ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac095b90 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe494023070 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4a2ffd700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 0x7fe490037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(209..209 src has 176..209) v4 ==== 12500+0+0 (secure 0 0 0) 0x7fe49404b450 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4aaffd700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 0x7fe490037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.286+0000 7fe4aaffd700 10 monclient: get_auth_request con 0x7fe490035a70 auth_method 0 2024-09-18T10:16:47.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.290+0000 7fe4aaffd700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 0x7fe490037f20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe49c006fd0 tx=0x7fe49c006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:47.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.294+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe494023320 con 0x7fe4ac09bfd0 2024-09-18T10:16:47.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.294+0000 7fe4a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:47.301 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.294+0000 7fe4a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:47.464 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.458+0000 7fe4b13ee700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.458+0000 7fe4b13ee700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:47.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.458+0000 7fe4b13ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:47.465 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:47.458+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe4ac063ed0 con 0x7fe4ac09bfd0 2024-09-18T10:16:48.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.062+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 added bucket h21 type host to location {rack=r1} v210) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe49402a020 con 0x7fe4ac09bfd0 2024-09-18T10:16:48.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.062+0000 7fe4a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:48.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.062+0000 7fe4a2ffd700 10 monclient: _finish_command 2 = system:0 added bucket h21 type host to location {rack=r1} 2024-09-18T10:16:48.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.166+0000 7fe4b13ee700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:48.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.166+0000 7fe4b13ee700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:48.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.166+0000 7fe4b13ee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:48.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.166+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fe4ac08de60 con 0x7fe4ac09bfd0 2024-09-18T10:16:48.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4a2ffd700 1 -- 172.21.15.150:0/3438286210 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h21' already exists v210) v1 ==== 147+0+0 (secure 0 0 0) 0x7fe4ac08de60 con 0x7fe4ac09bfd0 2024-09-18T10:16:48.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4a2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h21", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:48.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4a2ffd700 10 monclient: _finish_command 3 = system:0 bucket 'h21' already exists 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 msgr2=0x7fe490037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 0x7fe490037f20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe49c006fd0 tx=0x7fe49c006e10 comp rx=0 tx=0).stop 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 10 monclient: shutdown 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 msgr2=0x7fe4ac08a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac095fb0 tx=0x7fe494017000 comp rx=0 tx=0).stop 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 shutdown_connections 2024-09-18T10:16:48.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe490035a70 0x7fe490037f20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4ac09c8e0 0x7fe4ac08a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4ac09bfd0 0x7fe4ac08a410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 --2- 172.21.15.150:0/3438286210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4ac09acd0 0x7fe4ac089ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 >> 172.21.15.150:0/3438286210 conn(0x7fe4ac0a15e0 msgr2=0x7fe4ac0989c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 shutdown_connections 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.170+0000 7fe4b13ee700 1 -- 172.21.15.150:0/3438286210 wait complete. 2024-09-18T10:16:48.177 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h21' already exists 2024-09-18T10:16:48.185 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:48.186 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h22 host rack=r1 2024-09-18T10:16:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: get_monmap_and_config 2024-09-18T10:16:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: build_initial_monmap 2024-09-18T10:16:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: monmap: 2024-09-18T10:16:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:48.287 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:48.286357+0000 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:48.286357+0000 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:48.288 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 Processor -- start 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 -- start start 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: init 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-b con 0x7f5b0409c450 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b0409c820 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 0x7f5b04139b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-a con 0x7f5b041377a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0409bda0 con 0x7f5b0409c820 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:48.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0409d320 con 0x7f5b041377a0 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0409b5c0 con 0x7f5b0409c450 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b08ca0700 10 monclient(hunting): authenticate will time out at 1290.379753s 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b02d9d700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 msgr2=0x7f5b04139b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b01d9b700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f5b0409c820 legacy=0x7f5b0409cc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.282+0000 7f5b02d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 0x7f5b04139b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44650/0 (socket says 172.21.15.150:44650) 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 -- 172.21.15.150:0/196779145 learned_addr learned my addr 172.21.15.150:0/196779145 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): get_auth_request con 0x7f5b0409c450 auth_method 0 2024-09-18T10:16:48.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient(hunting): handle_auth_done global_id 74381 payload 274 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 -- 172.21.15.150:0/196779145 >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 msgr2=0x7f5b04139b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 --2- 172.21.15.150:0/196779145 >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 0x7f5b04139b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 -- 172.21.15.150:0/196779145 >> v1:172.21.15.142:6789/0 conn(0x7f5b0409c820 legacy=0x7f5b0409cc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 monclient: found mon.noname-b 2024-09-18T10:16:48.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:48.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:48.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 -- 172.21.15.150:0/196779145 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0409b800 con 0x7f5b0409c450 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 -- 172.21.15.150:0/196779145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 msgr2=0x7f5b0406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0259c700 1 --2- 172.21.15.150:0/196779145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5af80099b0 tx=0x7f5af8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca19723b70feb16 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 1 -- 172.21.15.150:0/196779145 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af800bb50 con 0x7f5b0409c450 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: dump: 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:48.302 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: _finish_auth 0 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b08ca0700 5 monclient: authenticate success, global_id 74381 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b08ca0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:48.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 1 -- 172.21.15.150:0/196779145 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af800bcb0 con 0x7f5b0409c450 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b08ca0700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b08ca0700 4 set_mon_vals no callback set 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 1 -- 172.21.15.150:0/196779145 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8011670 con 0x7f5b0409c450 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.286+0000 7f5b0159a700 10 monclient: dump: 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.304 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: shutdown 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- 172.21.15.150:0/196779145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 msgr2=0x7f5b0406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/196779145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5af80099b0 tx=0x7f5af8009250 comp rx=0 tx=0).stop 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- 172.21.15.150:0/196779145 shutdown_connections 2024-09-18T10:16:48.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/196779145 >> v2:172.21.15.142:3300/0 conn(0x7f5b041377a0 0x7f5b04139b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/196779145 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b0406b0f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- 172.21.15.150:0/196779145 >> 172.21.15.150:0/196779145 conn(0x7f5b040a2690 msgr2=0x7f5b0409e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- 172.21.15.150:0/196779145 shutdown_connections 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- 172.21.15.150:0/196779145 wait complete. 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: build_initial_monmap 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: monmap: 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:48.306 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 Processor -- start 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- start start 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: init 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-c con 0x7f5b0409c450 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:48.307 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-b con 0x7f5b0409c820 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 0x7f5b04142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): picked mon.noname-a con 0x7f5b041377a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0411e0f0 con 0x7f5b041377a0 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0409d9e0 con 0x7f5b0409c820 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0411cb40 con 0x7f5b0409c450 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b08ca0700 10 monclient(hunting): authenticate will time out at 1290.386968s 2024-09-18T10:16:48.308 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b02d9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 msgr2=0x7f5b04142690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b02d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 0x7f5b04142690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56736/0 (socket says 172.21.15.150:56736) 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 1 -- 172.21.15.150:0/1557690642 learned_addr learned my addr 172.21.15.150:0/1557690642 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): get_auth_request con 0x7f5b0409c820 auth_method 0 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): get_auth_request con 0x7f5b0409c450 auth_method 0 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:48.309 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b01d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 10 monclient(hunting): handle_auth_done global_id 74384 payload 824 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.290+0000 7f5b0259c700 1 -- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 msgr2=0x7f5b04142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 -- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 msgr2=0x7f5b04142690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 0x7f5b04142690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:48.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 monclient: found mon.noname-c 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af80094f0 con 0x7f5b0409c450 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b01d9b700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 -- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 msgr2=0x7f5b04141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5af8015000 tx=0x7f5af8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b0259c700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5af8015000 tx=0x7f5af8016000 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f5af801c450 mon_map magic: 0 v1 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af801c450 con 0x7f5b0409c450 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 10 monclient: dump: 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:48.311 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 5 monclient: authenticate success, global_id 74384 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af801c5b0 con 0x7f5b0409c450 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 10 monclient: _renew_subs 2024-09-18T10:16:48.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b040719b0 con 0x7f5b0409c450 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5af27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 10 monclient: _renew_subs 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b00d99700 4 set_mon_vals no callback set 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.294+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0411c630 con 0x7f5b0409c450 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af801c970 con 0x7f5b0409c450 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 10 monclient: dump: 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:48.313 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5af801cce0 con 0x7f5b0409c450 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b08ca0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b08ca0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b08ca0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b04063ed0 con 0x7f5b0409c450 2024-09-18T10:16:48.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5af27fc700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 0x7f5ae4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:48.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b01d9b700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 0x7f5ae4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:48.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.298+0000 7f5b01d9b700 10 monclient: get_auth_request con 0x7f5ae4035ac0 auth_method 0 2024-09-18T10:16:48.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.302+0000 7f5b01d9b700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 0x7f5ae4037f70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5b0411c230 tx=0x7f5af4009220 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:48.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.310+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(210..210 src has 176..210) v4 ==== 12713+0+0 (secure 0 0 0) 0x7f5af804b920 con 0x7f5b0409c450 2024-09-18T10:16:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.918+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5af8023b30 con 0x7f5b0409c450 2024-09-18T10:16:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.918+0000 7f5af27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:48.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:48.918+0000 7f5af27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:49.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:49.090+0000 7f5b08ca0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:49.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:49.090+0000 7f5b08ca0700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:49.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:49.090+0000 7f5b08ca0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:49.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:49.090+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5b04067ea0 con 0x7f5b0409c450 2024-09-18T10:16:50.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.074+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 added bucket h22 type host to location {rack=r1} v211) v1 ==== 168+0+0 (secure 0 0 0) 0x7f5af8023350 con 0x7f5b0409c450 2024-09-18T10:16:50.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.074+0000 7f5af27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.074+0000 7f5af27fc700 10 monclient: _finish_command 2 = system:0 added bucket h22 type host to location {rack=r1} 2024-09-18T10:16:50.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.178+0000 7f5b08ca0700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.178+0000 7f5b08ca0700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.182+0000 7f5b08ca0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:16:50.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.182+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5b04067ea0 con 0x7f5b0409c450 2024-09-18T10:16:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.182+0000 7f5af27fc700 1 -- 172.21.15.150:0/1557690642 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h22' already exists v211) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5af8023530 con 0x7f5b0409c450 2024-09-18T10:16:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.182+0000 7f5af27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h22", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.182+0000 7f5af27fc700 10 monclient: _finish_command 3 = system:0 bucket 'h22' already exists 2024-09-18T10:16:50.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 msgr2=0x7f5ae4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 0x7f5ae4037f70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5b0411c230 tx=0x7f5af4009220 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 10 monclient: shutdown 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 msgr2=0x7f5b04141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5af8015000 tx=0x7f5af8016000 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 shutdown_connections 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ae4035ac0 0x7f5ae4037f70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b041377a0 0x7f5b04142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b0409c820 0x7f5b04142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 --2- 172.21.15.150:0/1557690642 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b0409c450 0x7f5b04141c10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 >> 172.21.15.150:0/1557690642 conn(0x7f5b040a2690 msgr2=0x7f5b0406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:50.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 shutdown_connections 2024-09-18T10:16:50.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.186+0000 7f5b08ca0700 1 -- 172.21.15.150:0/1557690642 wait complete. 2024-09-18T10:16:50.191 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h22' already exists 2024-09-18T10:16:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:50.200 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h23 host rack=r1 2024-09-18T10:16:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 10 monclient: get_monmap_and_config 2024-09-18T10:16:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 10 monclient: build_initial_monmap 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 10 monclient: monmap: 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:50.298457+0000 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:50.298457+0000 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 1 Processor -- start 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 1 -- start start 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 10 monclient: init 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.294+0000 7f7a7612c700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a70099b90 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 0x7f7a7009d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-a con 0x7f7a70099f60 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-b con 0x7f7a70134620 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7009add0 con 0x7f7a70099b90 2024-09-18T10:16:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7009a7a0 con 0x7f7a70099f60 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70064bf0 con 0x7f7a70134620 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a7612c700 10 monclient(hunting): authenticate will time out at 1292.392146s 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6effd700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 msgr2=0x7f7a7009d5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 0x7f7a7009d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6f7fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f7a70099b90 legacy=0x7f7a7009ce90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:40426/0 (socket says 172.21.15.150:40426) 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 -- 172.21.15.150:0/3778176910 learned_addr learned my addr 172.21.15.150:0/3778176910 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): get_auth_request con 0x7f7a70134620 auth_method 0 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient(hunting): handle_auth_done global_id 74387 payload 274 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 -- 172.21.15.150:0/3778176910 >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 msgr2=0x7f7a7009d5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 --2- 172.21.15.150:0/3778176910 >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 0x7f7a7009d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 -- 172.21.15.150:0/3778176910 >> v1:172.21.15.142:6789/0 conn(0x7f7a70099b90 legacy=0x7f7a7009ce90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 monclient: found mon.noname-b 2024-09-18T10:16:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 -- 172.21.15.150:0/3778176910 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7009c8f0 con 0x7f7a70134620 2024-09-18T10:16:50.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 -- 172.21.15.150:0/3778176910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 msgr2=0x7f7a70136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:50.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.298+0000 7f7a6ffff700 1 --2- 172.21.15.150:0/3778176910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009f20 tx=0x7f7a68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a30d2328de6eb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:50.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 1 -- 172.21.15.150:0/3778176910 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800e070 con 0x7f7a70134620 2024-09-18T10:16:50.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:50.310 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: dump: 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:50.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 5 monclient: authenticate success, global_id 74387 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 1 -- 172.21.15.150:0/3778176910 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6800bc90 con 0x7f7a70134620 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 4 set_mon_vals no callback set 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 1 -- 172.21.15.150:0/3778176910 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68012660 con 0x7f7a70134620 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6e7fc700 10 monclient: dump: 2024-09-18T10:16:50.312 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.313 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: shutdown 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- 172.21.15.150:0/3778176910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 msgr2=0x7f7a70136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- 172.21.15.150:0/3778176910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009f20 tx=0x7f7a68009840 comp rx=0 tx=0).stop 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- 172.21.15.150:0/3778176910 shutdown_connections 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- 172.21.15.150:0/3778176910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70134620 0x7f7a70136a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- 172.21.15.150:0/3778176910 >> v2:172.21.15.142:3300/0 conn(0x7f7a70099f60 0x7f7a7009d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- 172.21.15.150:0/3778176910 >> 172.21.15.150:0/3778176910 conn(0x7f7a700a24a0 msgr2=0x7f7a700a2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- 172.21.15.150:0/3778176910 shutdown_connections 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- 172.21.15.150:0/3778176910 wait complete. 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: build_initial_monmap 2024-09-18T10:16:50.314 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: monmap: 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 Processor -- start 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- start start 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: init 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-b con 0x7f7a70099b90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70099f60 0x7f7a7012c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-c con 0x7f7a70099f60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 0x7f7a7012cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): picked mon.noname-a con 0x7f7a70134620 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:50.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7012e5f0 con 0x7f7a70134620 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7012f1e0 con 0x7f7a70099b90 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7012e8d0 con 0x7f7a70099f60 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a7612c700 10 monclient(hunting): authenticate will time out at 1292.399777s 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 msgr2=0x7f7a7012cd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.302+0000 7f7a6ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 0x7f7a7012cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45470/0 (socket says 172.21.15.150:45470) 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 -- 172.21.15.150:0/645748132 learned_addr learned my addr 172.21.15.150:0/645748132 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:50.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): get_auth_request con 0x7f7a70099b90 auth_method 0 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70099f60 0x7f7a7012c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): get_auth_request con 0x7f7a70099f60 auth_method 0 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:50.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient(hunting): handle_auth_done global_id 84229 payload 824 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 -- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70099f60 msgr2=0x7f7a7012c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70099f60 0x7f7a7012c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 -- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 msgr2=0x7f7a7012cd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 0x7f7a7012cd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:50.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68009ae0 con 0x7f7a70099b90 2024-09-18T10:16:50.322 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 -- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 msgr2=0x7f7a7012c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6f7fe700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400b810 tx=0x7f7a640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64003dd0 con 0x7f7a70099b90 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: dump: 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.323 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: _finish_auth 0 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 5 monclient: authenticate success, global_id 84229 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 10 monclient: _renew_subs 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7012ebb0 con 0x7f7a70099b90 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a64006150 con 0x7f7a70099b90 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:50.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64006510 con 0x7f7a70099b90 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a577fe700 10 monclient: dump: 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:50.325 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 10 monclient: _renew_subs 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a6dffb700 4 set_mon_vals no callback set 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.306+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7012f8a0 con 0x7f7a70099b90 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a7612c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a7612c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a7612c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7012edf0 con 0x7f7a70099b90 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7a6401b3b0 con 0x7f7a70099b90 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a577fe700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 0x7f7a5c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(211..211 src has 176..211) v4 ==== 12926+0+0 (secure 0 0 0) 0x7f7a64049820 con 0x7f7a70099b90 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a6effd700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 0x7f7a5c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.310+0000 7f7a6effd700 10 monclient: get_auth_request con 0x7f7a5c035a70 auth_method 0 2024-09-18T10:16:50.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.314+0000 7f7a6effd700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 0x7f7a5c037f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7a70098d10 tx=0x7f7a600091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:50.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.322+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7a6400e820 con 0x7f7a70099b90 2024-09-18T10:16:50.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.322+0000 7f7a577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:50.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.322+0000 7f7a577fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.486+0000 7f7a7612c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.486+0000 7f7a7612c700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.486+0000 7f7a7612c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:50.493 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:50.486+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7a70098750 con 0x7f7a70099b90 2024-09-18T10:16:51.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.082+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 added bucket h23 type host to location {rack=r1} v212) v1 ==== 168+0+0 (secure 0 0 0) 0x7f7a64017020 con 0x7f7a70099b90 2024-09-18T10:16:51.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.082+0000 7f7a577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:51.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.082+0000 7f7a577fe700 10 monclient: _finish_command 2 = system:0 added bucket h23 type host to location {rack=r1} 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f7a7009e870 con 0x7f7a70099b90 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a577fe700 1 -- 172.21.15.150:0/645748132 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h23' already exists v212) v1 ==== 147+0+0 (secure 0 0 0) 0x7f7a6400e260 con 0x7f7a70099b90 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a577fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h23", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:51.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a577fe700 10 monclient: _finish_command 3 = system:0 bucket 'h23' already exists 2024-09-18T10:16:51.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 msgr2=0x7f7a5c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:51.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 0x7f7a5c037f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7a70098d10 tx=0x7f7a600091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 10 monclient: shutdown 2024-09-18T10:16:51.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:51.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 msgr2=0x7f7a7012c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.194+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a6400b810 tx=0x7f7a640091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 shutdown_connections 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a5c035a70 0x7f7a5c037f20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a70134620 0x7f7a7012cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a70099f60 0x7f7a7012c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 --2- 172.21.15.150:0/645748132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a70099b90 0x7f7a7012c2b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 >> 172.21.15.150:0/645748132 conn(0x7f7a700a24a0 msgr2=0x7f7a700a0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 shutdown_connections 2024-09-18T10:16:51.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.198+0000 7f7a7612c700 1 -- 172.21.15.150:0/645748132 wait complete. 2024-09-18T10:16:51.203 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h23' already exists 2024-09-18T10:16:51.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:51.211 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h24 host rack=r1 2024-09-18T10:16:51.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: get_monmap_and_config 2024-09-18T10:16:51.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: build_initial_monmap 2024-09-18T10:16:51.311 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: monmap: 2024-09-18T10:16:51.311 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:51.311 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:51.310196+0000 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:51.310196+0000 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:51.312 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:51.313 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 Processor -- start 2024-09-18T10:16:51.315 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 -- start start 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: init 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd806b5a0 0x7fedd809e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): picked mon.noname-c con 0x7fedd806b5a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 0x7fedd806e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): picked mon.noname-a con 0x7fedd809ec70 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedd806ec50 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd806b970 con 0x7fedd806ec50 2024-09-18T10:16:51.316 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd809fb10 con 0x7fedd809ec70 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd806a7e0 con 0x7fedd806b5a0 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.306+0000 7feddf303700 10 monclient(hunting): authenticate will time out at 1293.403908s 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddc89e700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 msgr2=0x7fedd806e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd8a0700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fedd806ec50 legacy=0x7fedd81321c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:51.317 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddc89e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 0x7fedd806e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd806b5a0 0x7fedd809e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fedd806b5a0 0x7fedd809e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45486/0 (socket says 172.21.15.150:45486) 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 -- 172.21.15.150:0/3180034318 learned_addr learned my addr 172.21.15.150:0/3180034318 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): get_auth_request con 0x7fedd806b5a0 auth_method 0 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:51.318 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient(hunting): handle_auth_done global_id 84232 payload 274 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 -- 172.21.15.150:0/3180034318 >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 msgr2=0x7fedd806e710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 --2- 172.21.15.150:0/3180034318 >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 0x7fedd806e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 -- 172.21.15.150:0/3180034318 >> v1:172.21.15.142:6789/0 conn(0x7fedd806ec50 legacy=0x7fedd81321c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient: _finish_hunting 0 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 monclient: found mon.noname-c 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.319 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 -- 172.21.15.150:0/3180034318 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd809fd50 con 0x7fedd806b5a0 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 -- 172.21.15.150:0/3180034318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806b5a0 msgr2=0x7fedd809e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7feddd09f700 1 --2- 172.21.15.150:0/3180034318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806b5a0 0x7fedd809e730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedd40099e0 tx=0x7fedd4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba68ea71924fee0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 1 -- 172.21.15.150:0/3180034318 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400bad0 con 0x7fedd806b5a0 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: dump: 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.324 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: _finish_auth 0 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 1 -- 172.21.15.150:0/3180034318 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd400bc30 con 0x7fedd806b5a0 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 1 -- 172.21.15.150:0/3180034318 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4011420 con 0x7fedd806b5a0 2024-09-18T10:16:51.325 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.310+0000 7fedcffff700 10 monclient: dump: 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:51.326 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 5 monclient: authenticate success, global_id 84232 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 4 set_mon_vals no callback set 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 10 monclient: shutdown 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- 172.21.15.150:0/3180034318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806b5a0 msgr2=0x7fedd809e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 --2- 172.21.15.150:0/3180034318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806b5a0 0x7fedd809e730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedd40099e0 tx=0x7fedd4009250 comp rx=0 tx=0).stop 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- 172.21.15.150:0/3180034318 shutdown_connections 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 --2- 172.21.15.150:0/3180034318 >> v2:172.21.15.142:3300/0 conn(0x7fedd809ec70 0x7fedd806e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 --2- 172.21.15.150:0/3180034318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806b5a0 0x7fedd809e730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.327 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- 172.21.15.150:0/3180034318 >> 172.21.15.150:0/3180034318 conn(0x7fedd8071b60 msgr2=0x7fedd80a3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- 172.21.15.150:0/3180034318 shutdown_connections 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- 172.21.15.150:0/3180034318 wait complete. 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 10 monclient: build_initial_monmap 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 10 monclient: monmap: 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.328 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 Processor -- start 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 1 -- start start 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.314+0000 7feddf303700 10 monclient: init 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 0x7fedd8141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): picked mon.noname-a con 0x7fedd806b5a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:51.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): picked mon.noname-b con 0x7fedd806ec50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 0x7fedd811e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): picked mon.noname-c con 0x7fedd8142550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8122560 con 0x7fedd806b5a0 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81203e0 con 0x7fedd806ec50 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd09f700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 msgr2=0x7fedd8141ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8069b30 con 0x7fedd8142550 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd09f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 0x7fedd8141ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:51.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient(hunting): authenticate will time out at 1293.412618s 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45500/0 (socket says 172.21.15.150:45500) 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 -- 172.21.15.150:0/1954254829 learned_addr learned my addr 172.21.15.150:0/1954254829 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): get_auth_request con 0x7fedd806ec50 auth_method 0 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:51.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 0x7fedd811e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 10 monclient(hunting): get_auth_request con 0x7fedd8142550 auth_method 0 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient(hunting): handle_auth_done global_id 84235 payload 824 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 -- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 msgr2=0x7fedd811e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 0x7fedd811e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 -- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 msgr2=0x7fedd8141ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 0x7fedd8141ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:51.332 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient: _finish_hunting 0 2024-09-18T10:16:51.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 monclient: found mon.noname-b 2024-09-18T10:16:51.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:51.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40094f0 con 0x7fedd806ec50 2024-09-18T10:16:51.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddd8a0700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 0x7fedd811e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 -- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 msgr2=0x7fedd8142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc800b840 tx=0x7fedc8009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8015070 con 0x7fedd806ec50 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:51.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 10 monclient: dump: 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddc89e700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc800b840 tx=0x7fedc8009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fedc8003e20 config(0 keys) v1 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 5 monclient: authenticate success, global_id 84235 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _renew_subs 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8132050 con 0x7fedd806ec50 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8003e20 con 0x7fedd806ec50 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7fedcf7fe700 4 set_mon_vals no callback set 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _renew_subs 2024-09-18T10:16:51.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.318+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8120d00 con 0x7fedd806ec50 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc80063a0 con 0x7fedd806ec50 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 10 monclient: dump: 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:51.338 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fedc8006710 con 0x7fedd806ec50 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddf303700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddf303700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7fedcd7fa700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 0x7fedbc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddf303700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8063ed0 con 0x7fedd806ec50 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddd09f700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 0x7fedbc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.322+0000 7feddd09f700 10 monclient: get_auth_request con 0x7fedbc035a70 auth_method 0 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.326+0000 7feddd09f700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 0x7fedbc037f20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fedd40094c0 tx=0x7fedd4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:51.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.326+0000 7feddc89e700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc800b840 tx=0x7fedc8009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7fedc8018af0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:16:51.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:51.942+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(212..212 src has 176..212) v4 ==== 13139+0+0 (secure 0 0 0) 0x7fedc8049fd0 con 0x7fedd806ec50 2024-09-18T10:16:52.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.154+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fedc8018af0 con 0x7fedd806ec50 2024-09-18T10:16:52.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.154+0000 7fedcd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:52.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.154+0000 7fedcd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:52.329 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.326+0000 7feddf303700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:52.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.326+0000 7feddf303700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:52.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.326+0000 7feddf303700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:52.330 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:52.326+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fedd8067b10 con 0x7fedd806ec50 2024-09-18T10:16:53.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.102+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 added bucket h24 type host to location {rack=r1} v213) v1 ==== 168+0+0 (secure 0 0 0) 0x7fedc8018310 con 0x7fedd806ec50 2024-09-18T10:16:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.102+0000 7fedcd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:53.482 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.102+0000 7fedcd7fa700 10 monclient: _finish_command 2 = system:0 added bucket h24 type host to location {rack=r1} 2024-09-18T10:16:53.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7feddf303700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:53.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7feddf303700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:53.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7feddf303700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fedd8120fb0 con 0x7fedd806ec50 2024-09-18T10:16:53.497 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7fedcd7fa700 1 -- 172.21.15.150:0/1954254829 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h24' already exists v213) v1 ==== 147+0+0 (secure 0 0 0) 0x7fedc80184f0 con 0x7fedd806ec50 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7fedcd7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h24", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.206+0000 7fedcd7fa700 10 monclient: _finish_command 3 = system:0 bucket 'h24' already exists 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 msgr2=0x7fedbc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 0x7fedbc037f20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fedd40094c0 tx=0x7fedd4016000 comp rx=0 tx=0).stop 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 10 monclient: shutdown 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 msgr2=0x7fedd8142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:53.498 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fedc800b840 tx=0x7fedc8009220 comp rx=0 tx=0).stop 2024-09-18T10:16:53.499 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 shutdown_connections 2024-09-18T10:16:53.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fedbc035a70 0x7fedbc037f20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fedd8142550 0x7fedd811e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedd806ec50 0x7fedd8142010 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 --2- 172.21.15.150:0/1954254829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fedd806b5a0 0x7fedd8141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.500 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 >> 172.21.15.150:0/1954254829 conn(0x7fedd8071b60 msgr2=0x7fedd806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:53.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 shutdown_connections 2024-09-18T10:16:53.501 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.210+0000 7feddf303700 1 -- 172.21.15.150:0/1954254829 wait complete. 2024-09-18T10:16:53.501 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h24' already exists 2024-09-18T10:16:53.501 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h25 host rack=r1 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: get_monmap_and_config 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: build_initial_monmap 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: monmap: 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.502 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:53.325371+0000 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:53.325371+0000 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 Processor -- start 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 -- start start 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: init 2024-09-18T10:16:53.503 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52b009d970 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52b0131750 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f52b0135250 0x7f52b0137630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-a con 0x7f52b0135250 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b009b0a0 con 0x7f52b009d970 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b009dd40 con 0x7f52b0131750 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b00982f0 con 0x7f52b0135250 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b807d700 10 monclient(hunting): authenticate will time out at 1295.418906s 2024-09-18T10:16:53.504 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b5e19700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f52b009d970 legacy=0x7f52b00955b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b5618700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f52b0131750 0x7f52b0133b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35656/0 (socket says 172.21.15.150:35656) 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b5618700 1 -- 172.21.15.150:0/2879851507 learned_addr learned my addr 172.21.15.150:0/2879851507 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 >> v2:172.21.15.142:3300/0 conn(0x7f52b0135250 msgr2=0x7f52b0137630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.322+0000 7f52b661a700 1 --2- 172.21.15.150:0/2879851507 >> v2:172.21.15.142:3300/0 conn(0x7f52b0135250 0x7f52b0137630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5618700 1 -- 172.21.15.150:0/2879851507 >> v1:172.21.15.150:6789/0 conn(0x7f52b0131750 legacy=0x7f52b0133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22077092 0 0) 0x7f52b009dd40 con 0x7f52b0131750 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): my global_id is 84238 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f529c003570 con 0x7f52b0131750 2024-09-18T10:16:53.505 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1319674258 0 0) 0x7f529c003570 con 0x7f52b0131750 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 >> v2:172.21.15.142:3300/0 conn(0x7f52b0135250 msgr2=0x7f52b0137630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 --2- 172.21.15.150:0/2879851507 >> v2:172.21.15.142:3300/0 conn(0x7f52b0135250 0x7f52b0137630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 >> v1:172.21.15.142:6789/0 conn(0x7f52b009d970 legacy=0x7f52b00955b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: _finish_hunting 0 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b009c560 con 0x7f52b0131750 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: _finish_auth 0 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:53.506 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f52ac0017a0 con 0x7f52b0131750 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b807d700 5 monclient: authenticate success, global_id 84238 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b807d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: dump: 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:53.507 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 >> v1:172.21.15.150:6789/0 conn(0x7f52b0131750 legacy=0x7f52b0133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f529c003a60 0x7f529c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.508 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): picked mon.c con 0x7f529c003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f529c004810 0x7f529c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): picked mon.a con 0x7f529c004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f529c00baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): picked mon.b con 0x7f529c0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00982f0 con 0x7f529c004810 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c00e590 con 0x7f529c0053c0 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c00e750 con 0x7f529c003a60 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b4e17700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:16:53.509 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b807d700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5618700 1 -- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f529c004810 msgr2=0x7f529c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5618700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f529c004810 0x7f529c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f529c00baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): get_auth_request con 0x7f529c0053c0 auth_method 0 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:53.510 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5e19700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f529c003a60 0x7f529c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5e19700 10 monclient(hunting): get_auth_request con 0x7f529c003a60 auth_method 0 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5e19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5e19700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b5e19700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient(hunting): handle_auth_done global_id 84238 payload 293 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f529c003a60 msgr2=0x7f529c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f529c003a60 0x7f529c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.511 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f529c004810 msgr2=0x7f529c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f529c004810 0x7f529c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient: _finish_hunting 0 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 monclient: found mon.b 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c003570 con 0x7f529c0053c0 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 10 monclient: _finish_auth 0 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.326+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 reap_dead start 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b661a700 1 -- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 msgr2=0x7f529c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b661a700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f529c00baa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009f90 tx=0x7f52a80095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5504bfa80173bc5e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:53.512 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8009900 con 0x7f529c0053c0 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: dump: 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:53.513 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a800be70 con 0x7f529c0053c0 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 1 -- 172.21.15.150:0/2879851507 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80156a0 con 0x7f529c0053c0 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 4 set_mon_vals no callback set 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b4e17700 10 monclient: dump: 2024-09-18T10:16:53.514 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.515 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 10 monclient: shutdown 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 -- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 msgr2=0x7f529c00baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f529c00baa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f52a8009f90 tx=0x7f52a80095d0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 -- 172.21.15.150:0/2879851507 shutdown_connections 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 --2- 172.21.15.150:0/2879851507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f529c00baa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 -- 172.21.15.150:0/2879851507 >> 172.21.15.150:0/2879851507 conn(0x7f52b00a15e0 msgr2=0x7f52b00a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:53.516 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 -- 172.21.15.150:0/2879851507 shutdown_connections 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 1 -- 172.21.15.150:0/2879851507 wait complete. 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 10 monclient: build_initial_monmap 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.330+0000 7f52b807d700 10 monclient: monmap: 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:53.517 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 Processor -- start 2024-09-18T10:16:53.518 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 -- start start 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient: init 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-b con 0x7f529c0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52b006e740 0x7f52b0130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-c con 0x7f52b006e740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:53.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 0x7f52b0140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): picked mon.noname-a con 0x7f52b012b100 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b012bff0 con 0x7f52b012b100 2024-09-18T10:16:53.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b012c5b0 con 0x7f529c0053c0 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b012c2d0 con 0x7f52b006e740 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b807d700 10 monclient(hunting): authenticate will time out at 1295.429507s 2024-09-18T10:16:53.521 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b661a700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 msgr2=0x7f52b0140b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b661a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 0x7f52b0140b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45512/0 (socket says 172.21.15.150:45512) 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 -- 172.21.15.150:0/3336697861 learned_addr learned my addr 172.21.15.150:0/3336697861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): get_auth_request con 0x7f529c0053c0 auth_method 0 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52b006e740 0x7f52b0130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): get_auth_request con 0x7f52b006e740 auth_method 0 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:53.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5618700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient(hunting): handle_auth_done global_id 84241 payload 824 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 -- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52b006e740 msgr2=0x7f52b0130df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52b006e740 0x7f52b0130df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 -- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 msgr2=0x7f52b0140b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 0x7f52b0140b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient: _finish_hunting 0 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 monclient: found mon.noname-b 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.334+0000 7f52b5e19700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b012ac50 con 0x7f529c0053c0 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b5e19700 1 -- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 msgr2=0x7f52b013fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b5e19700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52a000d7d0 tx=0x7f52a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0003e20 con 0x7f529c0053c0 2024-09-18T10:16:53.524 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b5e19700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52a000d7d0 tx=0x7f52a0008040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f52a0005320 mon_map magic: 0 v1 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: dump: 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.525 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a000f040 con 0x7f529c0053c0 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 5 monclient: authenticate success, global_id 84241 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _renew_subs 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b01405f0 con 0x7f529c0053c0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _renew_subs 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a77fe700 4 set_mon_vals no callback set 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b01314d0 con 0x7f529c0053c0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b0096b90 con 0x7f529c0053c0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0005320 con 0x7f529c0053c0 2024-09-18T10:16:53.527 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 10 monclient: dump: 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:53.528 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.338+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f52a0005690 con 0x7f529c0053c0 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.342+0000 7f52a5ffb700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 0x7f5294037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.342+0000 7f52b5618700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 0x7f5294037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.342+0000 7f52b5618700 10 monclient: get_auth_request con 0x7f5294035a70 auth_method 0 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.342+0000 7f52b5618700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 0x7f5294037f20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f52ac000f60 tx=0x7f52ac001890 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:53.529 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:53.350+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(213..213 src has 176..213) v4 ==== 13352+0+0 (secure 0 0 0) 0x7f52a0049890 con 0x7f529c0053c0 2024-09-18T10:16:54.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.170+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f52a00194e0 con 0x7f529c0053c0 2024-09-18T10:16:54.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.170+0000 7f52a5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:54.173 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.170+0000 7f52a5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:54.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.342+0000 7f52b807d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:54.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.342+0000 7f52b807d700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:54.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.342+0000 7f52b807d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:54.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:54.342+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f52b0097770 con 0x7f529c0053c0 2024-09-18T10:16:55.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.118+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 added bucket h25 type host to location {rack=r1} v214) v1 ==== 168+0+0 (secure 0 0 0) 0x7f52a0017070 con 0x7f529c0053c0 2024-09-18T10:16:55.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.118+0000 7f52a5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.118+0000 7f52a5ffb700 10 monclient: _finish_command 2 = system:0 added bucket h25 type host to location {rack=r1} 2024-09-18T10:16:55.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52b807d700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52b807d700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52b807d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f52b0063ed0 con 0x7f529c0053c0 2024-09-18T10:16:55.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52a5ffb700 1 -- 172.21.15.150:0/3336697861 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h25' already exists v214) v1 ==== 147+0+0 (secure 0 0 0) 0x7f52a0011ae0 con 0x7f529c0053c0 2024-09-18T10:16:55.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52a5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h25", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.222+0000 7f52a5ffb700 10 monclient: _finish_command 3 = system:0 bucket 'h25' already exists 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 msgr2=0x7f5294037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 0x7f5294037f20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f52ac000f60 tx=0x7f52ac001890 comp rx=0 tx=0).stop 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 10 monclient: shutdown 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 msgr2=0x7f52b013fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52a000d7d0 tx=0x7f52a0008040 comp rx=0 tx=0).stop 2024-09-18T10:16:55.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 shutdown_connections 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5294035a70 0x7f5294037f20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f52b012b100 0x7f52b0140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f52b006e740 0x7f52b0130df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 --2- 172.21.15.150:0/3336697861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f529c0053c0 0x7f52b013fb00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 >> 172.21.15.150:0/3336697861 conn(0x7f52b00a15e0 msgr2=0x7f52b009c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 shutdown_connections 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.226+0000 7f52b807d700 1 -- 172.21.15.150:0/3336697861 wait complete. 2024-09-18T10:16:55.232 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h25' already exists 2024-09-18T10:16:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:55.241 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h26 host rack=r1 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: get_monmap_and_config 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: build_initial_monmap 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: monmap: 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:55.342 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:55.341215+0000 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:55.341215+0000 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:55.343 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.345 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 Processor -- start 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 -- start start 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: init 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-b con 0x7fa7d006f160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7d009cc90 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 0x7fa7d0132c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-a con 0x7fa7d009d060 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d009e170 con 0x7fa7d009cc90 2024-09-18T10:16:55.346 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7d0071250 con 0x7fa7d009d060 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0098ab0 con 0x7fa7d006f160 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d79b5700 10 monclient(hunting): authenticate will time out at 1297.434673s 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5f52700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 msgr2=0x7fa7d0132c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d4f50700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fa7d009cc90 legacy=0x7fa7d0132560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5f52700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 0x7fa7d0132c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:40480/0 (socket says 172.21.15.150:40480) 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 -- 172.21.15.150:0/4102716003 learned_addr learned my addr 172.21.15.150:0/4102716003 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): get_auth_request con 0x7fa7d006f160 auth_method 0 2024-09-18T10:16:55.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient(hunting): handle_auth_done global_id 74399 payload 274 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 -- 172.21.15.150:0/4102716003 >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 msgr2=0x7fa7d0132c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 --2- 172.21.15.150:0/4102716003 >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 0x7fa7d0132c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 -- 172.21.15.150:0/4102716003 >> v1:172.21.15.142:6789/0 conn(0x7fa7d009cc90 legacy=0x7fa7d0132560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient: _finish_hunting 0 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 monclient: found mon.noname-b 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:55.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:55.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.338+0000 7fa7d5751700 1 -- 172.21.15.150:0/4102716003 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d006f530 con 0x7fa7d006f160 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d5751700 1 -- 172.21.15.150:0/4102716003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 msgr2=0x7fa7d009c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d5751700 1 --2- 172.21.15.150:0/4102716003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0099b0 tx=0x7fa7cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac508cea07673795 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 1 -- 172.21.15.150:0/4102716003 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00bb50 con 0x7fa7d006f160 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: dump: 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:55.350 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: _finish_auth 0 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 5 monclient: authenticate success, global_id 74399 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 1 -- 172.21.15.150:0/4102716003 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc00bcb0 con 0x7fa7d006f160 2024-09-18T10:16:55.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 4 set_mon_vals no callback set 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 1 -- 172.21.15.150:0/4102716003 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc011670 con 0x7fa7d006f160 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7c7fff700 10 monclient: dump: 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:55.352 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 10 monclient: shutdown 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 -- 172.21.15.150:0/4102716003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 msgr2=0x7fa7d009c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/4102716003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0099b0 tx=0x7fa7cc009250 comp rx=0 tx=0).stop 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 -- 172.21.15.150:0/4102716003 shutdown_connections 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/4102716003 >> v2:172.21.15.142:3300/0 conn(0x7fa7d009d060 0x7fa7d0132c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/4102716003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d006f160 0x7fa7d009c750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 -- 172.21.15.150:0/4102716003 >> 172.21.15.150:0/4102716003 conn(0x7fa7d00681c0 msgr2=0x7fa7d00a1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 -- 172.21.15.150:0/4102716003 shutdown_connections 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.342+0000 7fa7d79b5700 1 -- 172.21.15.150:0/4102716003 wait complete. 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient: build_initial_monmap 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient: monmap: 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:55.354 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:55.355 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 Processor -- start 2024-09-18T10:16:55.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 -- start start 2024-09-18T10:16:55.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient: init 2024-09-18T10:16:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:16:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 0x7fa7d0141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-a con 0x7fa7d006f160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:55.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-b con 0x7fa7d009cc90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d009d060 0x7fa7d011e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): picked mon.noname-c con 0x7fa7d009d060 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d009df90 con 0x7fa7d006f160 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d00a21c0 con 0x7fa7d009cc90 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d0121990 con 0x7fa7d009d060 2024-09-18T10:16:55.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d79b5700 10 monclient(hunting): authenticate will time out at 1297.442003s 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5751700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 msgr2=0x7fa7d0141aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5751700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 0x7fa7d0141aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45516/0 (socket says 172.21.15.150:45516) 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 -- 172.21.15.150:0/2789823113 learned_addr learned my addr 172.21.15.150:0/2789823113 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): get_auth_request con 0x7fa7d009cc90 auth_method 0 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:55.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d009d060 0x7fa7d011e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): get_auth_request con 0x7fa7d009d060 auth_method 0 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:55.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d5f52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient(hunting): handle_auth_done global_id 84244 payload 824 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 -- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d009d060 msgr2=0x7fa7d011e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d009d060 0x7fa7d011e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 -- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 msgr2=0x7fa7d0141aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 0x7fa7d0141aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient: _finish_hunting 0 2024-09-18T10:16:55.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 monclient: found mon.noname-b 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.346+0000 7fa7d4f50700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0094f0 con 0x7fa7d009cc90 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d4f50700 1 -- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 msgr2=0x7fa7d01347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d4f50700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b810 tx=0x7fa7c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0003dd0 con 0x7fa7d009cc90 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: dump: 2024-09-18T10:16:55.364 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:55.365 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 5 monclient: authenticate success, global_id 84244 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c0006150 con 0x7fa7d009cc90 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _renew_subs 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0006510 con 0x7fa7d009cc90 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d011fde0 con 0x7fa7d009cc90 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c77fe700 4 set_mon_vals no callback set 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7c57fa700 10 monclient: dump: 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:55.367 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _renew_subs 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d01427e0 con 0x7fa7d009cc90 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:55.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.350+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d0063ed0 con 0x7fa7d009cc90 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa7c001a360 con 0x7fa7d009cc90 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7c57fa700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 0x7fa7b4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(214..214 src has 176..214) v4 ==== 13565+0+0 (secure 0 0 0) 0x7fa7c0049750 con 0x7fa7d009cc90 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7d5751700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 0x7fa7b4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:55.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7d5751700 10 monclient: get_auth_request con 0x7fa7b4035a70 auth_method 0 2024-09-18T10:16:55.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.354+0000 7fa7d5751700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 0x7fa7b4037f20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0099b0 tx=0x7fa7cc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:55.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.362+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa7c00177d0 con 0x7fa7d009cc90 2024-09-18T10:16:55.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.362+0000 7fa7c57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:55.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.362+0000 7fa7c57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.538+0000 7fa7d79b5700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.538+0000 7fa7d79b5700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.538+0000 7fa7d79b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:55.538+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa7d0121dc0 con 0x7fa7d009cc90 2024-09-18T10:16:56.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.126+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 added bucket h26 type host to location {rack=r1} v215) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa7c0017070 con 0x7fa7d009cc90 2024-09-18T10:16:56.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.126+0000 7fa7c57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.126+0000 7fa7c57fa700 10 monclient: _finish_command 2 = system:0 added bucket h26 type host to location {rack=r1} 2024-09-18T10:16:56.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa7d00008d0 con 0x7fa7d009cc90 2024-09-18T10:16:56.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7c57fa700 1 -- 172.21.15.150:0/2789823113 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h26' already exists v215) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa7c00177d0 con 0x7fa7d009cc90 2024-09-18T10:16:56.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7c57fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h26", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7c57fa700 10 monclient: _finish_command 3 = system:0 bucket 'h26' already exists 2024-09-18T10:16:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 msgr2=0x7fa7b4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:56.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 0x7fa7b4037f20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0099b0 tx=0x7fa7cc016000 comp rx=0 tx=0).stop 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 10 monclient: shutdown 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 msgr2=0x7fa7d01347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000b810 tx=0x7fa7c00091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 shutdown_connections 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa7b4035a70 0x7fa7b4037f20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7d009d060 0x7fa7d011e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7d009cc90 0x7fa7d01347f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 --2- 172.21.15.150:0/2789823113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7d006f160 0x7fa7d0141aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 >> 172.21.15.150:0/2789823113 conn(0x7fa7d00681c0 msgr2=0x7fa7d0135670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:56.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.238+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 shutdown_connections 2024-09-18T10:16:56.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.242+0000 7fa7d79b5700 1 -- 172.21.15.150:0/2789823113 wait complete. 2024-09-18T10:16:56.248 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h26' already exists 2024-09-18T10:16:56.255 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:56.255 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h27 host rack=r1 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 10 monclient: get_monmap_and_config 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 10 monclient: build_initial_monmap 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 10 monclient: monmap: 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:56.357984+0000 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:56.357984+0000 2024-09-18T10:16:56.359 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:56.360 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 1 Processor -- start 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 1 -- start start 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.354+0000 7f080b5cf700 10 monclient: init 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08040a0290 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f080409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-b con 0x7f08040a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0804137a20 addr v1:172.21.15.150:6789/0 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080409e360 con 0x7f08040a0290 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080409da50 con 0x7f0804137a20 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:56.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080409ef80 con 0x7f08040a0d70 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080b5cf700 10 monclient(hunting): authenticate will time out at 1298.452653s 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f080936b700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f08040a0290 legacy=0x7f08040a0660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0809b6c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0804137a20 0x7f0804139e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35668/0 (socket says 172.21.15.150:35668) 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0809b6c700 1 -- 172.21.15.150:0/1701144750 learned_addr learned my addr 172.21.15.150:0/1701144750 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 1 --2- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f080409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): get_auth_request con 0x7f08040a0d70 auth_method 0 2024-09-18T10:16:56.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient(hunting): handle_auth_done global_id 74405 payload 274 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 1 -- 172.21.15.150:0/1701144750 >> v1:172.21.15.150:6789/0 conn(0x7f0804137a20 legacy=0x7f0804139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 1 -- 172.21.15.150:0/1701144750 >> v1:172.21.15.142:6789/0 conn(0x7f08040a0290 legacy=0x7f08040a0660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient: _finish_hunting 0 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 1 monclient: found mon.noname-b 2024-09-18T10:16:56.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:56.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:56.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.358+0000 7f0808b6a700 1 -- 172.21.15.150:0/1701144750 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080409dd30 con 0x7f08040a0d70 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f0808b6a700 1 -- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 msgr2=0x7f080409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f0808b6a700 1 --2- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f080409ea40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009910 tx=0x7f07f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=530c6524595845d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 1 -- 172.21.15.150:0/1701144750 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f400ba70 con 0x7f08040a0d70 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:56.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: dump: 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: _finish_auth 0 2024-09-18T10:16:56.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 5 monclient: authenticate success, global_id 74405 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 1 -- 172.21.15.150:0/1701144750 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f400bbd0 con 0x7f08040a0d70 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 4 set_mon_vals no callback set 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 1 -- 172.21.15.150:0/1701144750 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4011420 con 0x7f08040a0d70 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f07fbfff700 10 monclient: dump: 2024-09-18T10:16:56.373 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 10 monclient: shutdown 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:56.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 msgr2=0x7f080409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 --2- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f080409ea40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009910 tx=0x7f07f40091a0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- 172.21.15.150:0/1701144750 shutdown_connections 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 --2- 172.21.15.150:0/1701144750 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f080409ea40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- 172.21.15.150:0/1701144750 >> 172.21.15.150:0/1701144750 conn(0x7f08040a58a0 msgr2=0x7f08040a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- 172.21.15.150:0/1701144750 shutdown_connections 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- 172.21.15.150:0/1701144750 wait complete. 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 10 monclient: build_initial_monmap 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 10 monclient: monmap: 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:56.375 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 Processor -- start 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.362+0000 7f080b5cf700 1 -- start start 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient: init 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:56.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-b con 0x7f08040a0290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f08041423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-c con 0x7f08040a0d70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 0x7f0804142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): picked mon.noname-a con 0x7f0804137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080412e120 con 0x7f0804137a20 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080409bfc0 con 0x7f08040a0290 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:56.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080409d3f0 con 0x7f08040a0d70 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080b5cf700 10 monclient(hunting): authenticate will time out at 1298.460579s 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0809b6c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 msgr2=0x7f0804142910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45518/0 (socket says 172.21.15.150:45518) 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0809b6c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 0x7f0804142910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 -- 172.21.15.150:0/3930283301 learned_addr learned my addr 172.21.15.150:0/3930283301 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): get_auth_request con 0x7f08040a0290 auth_method 0 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:56.378 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f08041423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): get_auth_request con 0x7f08040a0d70 auth_method 0 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f0808b6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:56.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient(hunting): handle_auth_done global_id 84247 payload 824 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 -- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 msgr2=0x7f08041423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f08041423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 -- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 msgr2=0x7f0804142910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 0x7f0804142910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient: _finish_hunting 0 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 monclient: found mon.noname-b 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f4009420 con 0x7f08040a0290 2024-09-18T10:16:56.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 -- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 msgr2=0x7f0804141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:56.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.366+0000 7f080936b700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f080000b7e0 tx=0x7f0800009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:56.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800003e20 con 0x7f08040a0290 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: dump: 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.385 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:56.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 5 monclient: authenticate success, global_id 84247 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _renew_subs 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0800006150 con 0x7f08040a0290 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080412e850 con 0x7f08040a0290 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _renew_subs 2024-09-18T10:16:56.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080412ecb0 con 0x7f08040a0290 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07fb7fe700 4 set_mon_vals no callback set 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800006510 con 0x7f08040a0290 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 10 monclient: dump: 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:56.388 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:56.389 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080409d6c0 con 0x7f08040a0290 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f08000103c0 con 0x7f08040a0290 2024-09-18T10:16:56.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 0x7f07e8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(215..215 src has 176..215) v4 ==== 13778+0+0 (secure 0 0 0) 0x7f0800049e80 con 0x7f08040a0290 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f0808b6a700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 0x7f07e8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.370+0000 7f0808b6a700 10 monclient: get_auth_request con 0x7f07e8035a70 auth_method 0 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.374+0000 7f0808b6a700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 0x7f07e8037f20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f07f40093f0 tx=0x7f07f4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.378+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f080000e8c0 con 0x7f08040a0290 2024-09-18T10:16:56.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.378+0000 7f07f97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:56.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.378+0000 7f07f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.546+0000 7f080b5cf700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.546+0000 7f080b5cf700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.546+0000 7f080b5cf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:56.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:56.546+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0804063ed0 con 0x7f08040a0290 2024-09-18T10:16:57.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.134+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 added bucket h27 type host to location {rack=r1} v216) v1 ==== 168+0+0 (secure 0 0 0) 0x7f0800017070 con 0x7f08040a0290 2024-09-18T10:16:57.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.134+0000 7f07f97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.134+0000 7f07f97fa700 10 monclient: _finish_command 2 = system:0 added bucket h27 type host to location {rack=r1} 2024-09-18T10:16:57.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.242+0000 7f080b5cf700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.242+0000 7f080b5cf700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.242+0000 7f080b5cf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.242+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f0804067b10 con 0x7f08040a0290 2024-09-18T10:16:57.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f07f97fa700 1 -- 172.21.15.150:0/3930283301 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h27' already exists v216) v1 ==== 147+0+0 (secure 0 0 0) 0x7f080000d2f0 con 0x7f08040a0290 2024-09-18T10:16:57.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f07f97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h27", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f07f97fa700 10 monclient: _finish_command 3 = system:0 bucket 'h27' already exists 2024-09-18T10:16:57.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 msgr2=0x7f07e8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:57.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 0x7f07e8037f20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f07f40093f0 tx=0x7f07f4016000 comp rx=0 tx=0).stop 2024-09-18T10:16:57.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 10 monclient: shutdown 2024-09-18T10:16:57.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:57.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 msgr2=0x7f0804141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:57.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f080000b7e0 tx=0x7f0800009220 comp rx=0 tx=0).stop 2024-09-18T10:16:57.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 shutdown_connections 2024-09-18T10:16:57.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f07e8035a70 0x7f07e8037f20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0804137a20 0x7f0804142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f08040a0d70 0x7f08041423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 --2- 172.21.15.150:0/3930283301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08040a0290 0x7f0804141e90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 >> 172.21.15.150:0/3930283301 conn(0x7f08040a58a0 msgr2=0x7f08040a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:57.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 shutdown_connections 2024-09-18T10:16:57.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.246+0000 7f080b5cf700 1 -- 172.21.15.150:0/3930283301 wait complete. 2024-09-18T10:16:57.256 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h27' already exists 2024-09-18T10:16:57.262 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:57.263 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h28 host rack=r1 2024-09-18T10:16:57.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.358+0000 7fa0b4d19700 10 monclient: get_monmap_and_config 2024-09-18T10:16:57.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.358+0000 7fa0b4d19700 10 monclient: build_initial_monmap 2024-09-18T10:16:57.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.358+0000 7fa0b4d19700 10 monclient: monmap: 2024-09-18T10:16:57.364 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:57.363416+0000 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:57.363416+0000 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:57.365 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:57.366 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 Processor -- start 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 -- start start 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient: init 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-18T10:16:57.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-b con 0x7fa0b00a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 0x7fa0b00a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-a con 0x7fa0b00a07d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0b0136a10 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00a1100 con 0x7fa0b0136a10 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b00a6380 con 0x7fa0b00a07d0 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00a6770 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:57.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0b4d19700 10 monclient(hunting): authenticate will time out at 1299.457051s 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0aed9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fa0b0136a10 legacy=0x7fa0b0138df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0a7fff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 msgr2=0x7fa0b00a0bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0a7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 0x7fa0b00a0bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:40526/0 (socket says 172.21.15.150:40526) 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 -- 172.21.15.150:0/4292059961 learned_addr learned my addr 172.21.15.150:0/4292059961 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): get_auth_request con 0x7fa0b00a5fb0 auth_method 0 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:57.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_done global_id 74411 payload 274 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 -- 172.21.15.150:0/4292059961 >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 msgr2=0x7fa0b00a0bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 --2- 172.21.15.150:0/4292059961 >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 0x7fa0b00a0bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 -- 172.21.15.150:0/4292059961 >> v1:172.21.15.142:6789/0 conn(0x7fa0b0136a10 legacy=0x7fa0b0138df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient: _finish_hunting 0 2024-09-18T10:16:57.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 monclient: found mon.noname-b 2024-09-18T10:16:57.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:57.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:16:57.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.362+0000 7fa0ae59c700 1 -- 172.21.15.150:0/4292059961 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b009e660 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0ae59c700 1 -- 172.21.15.150:0/4292059961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 msgr2=0x7fa0b00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0ae59c700 1 --2- 172.21.15.150:0/4292059961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00099b0 tx=0x7fa0a0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4959dfc9317fdcf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 1 -- 172.21.15.150:0/4292059961 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000bb50 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:57.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: dump: 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: _finish_auth 0 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:57.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 1 -- 172.21.15.150:0/4292059961 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a000bcb0 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 5 monclient: authenticate success, global_id 74411 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 1 -- 172.21.15.150:0/4292059961 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0011650 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 4 set_mon_vals no callback set 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0add9b700 10 monclient: dump: 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.381 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 10 monclient: shutdown 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 -- 172.21.15.150:0/4292059961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 msgr2=0x7fa0b00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/4292059961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00099b0 tx=0x7fa0a0009250 comp rx=0 tx=0).stop 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 -- 172.21.15.150:0/4292059961 shutdown_connections 2024-09-18T10:16:57.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/4292059961 >> v2:172.21.15.142:3300/0 conn(0x7fa0b00a07d0 0x7fa0b00a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/4292059961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a5fb0 0x7fa0b00a0290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 -- 172.21.15.150:0/4292059961 >> 172.21.15.150:0/4292059961 conn(0x7fa0b0071ae0 msgr2=0x7fa0b00a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 -- 172.21.15.150:0/4292059961 shutdown_connections 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 1 -- 172.21.15.150:0/4292059961 wait complete. 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.366+0000 7fa0b4d19700 10 monclient: build_initial_monmap 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient: monmap: 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.383 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 Processor -- start 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 -- start start 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient: init 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a07d0 0x7fa0b012c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-c con 0x7fa0b00a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-b con 0x7fa0b00a5fb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:57.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 0x7fa0b012d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): picked mon.noname-a con 0x7fa0b0136a10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b009c010 con 0x7fa0b0136a10 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b012fee0 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b012f680 con 0x7fa0b00a07d0 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0b4d19700 10 monclient(hunting): authenticate will time out at 1299.466218s 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0aed9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 msgr2=0x7fa0b012d210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0aed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 0x7fa0b012d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:57.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45532/0 (socket says 172.21.15.150:45532) 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 -- 172.21.15.150:0/1480444014 learned_addr learned my addr 172.21.15.150:0/1480444014 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request con 0x7fa0b00a5fb0 auth_method 0 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:57.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a07d0 0x7fa0b012c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): get_auth_request con 0x7fa0b00a07d0 auth_method 0 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0ae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_done global_id 84250 payload 824 2024-09-18T10:16:57.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 -- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a07d0 msgr2=0x7fa0b012c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a07d0 0x7fa0b012c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 -- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 msgr2=0x7fa0b012d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 0x7fa0b012d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 monclient: found mon.noname-b 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.370+0000 7fa0a7fff700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a00094f0 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a7fff700 1 -- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 msgr2=0x7fa0b012ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a7fff700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa09800b810 tx=0x7fa098009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098003e20 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:57.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: dump: 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: _finish_auth 0 2024-09-18T10:16:57.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098006150 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098006510 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 5 monclient: authenticate success, global_id 84250 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _renew_subs 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b012f920 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0acd99700 4 set_mon_vals no callback set 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 10 monclient: dump: 2024-09-18T10:16:57.390 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _renew_subs 2024-09-18T10:16:57.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b0130bb0 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b012fb60 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa09801b3b0 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 0x7fa090037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:57.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(216..216 src has 176..216) v4 ==== 13991+0+0 (secure 0 0 0) 0x7fa098049d90 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0ae59c700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 0x7fa090037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.374+0000 7fa0ae59c700 10 monclient: get_auth_request con 0x7fa090035aa0 auth_method 0 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.378+0000 7fa0ae59c700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 0x7fa090037f50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00099b0 tx=0x7fa0a0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.386+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa09800e820 con 0x7fa0b00a5fb0 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.386+0000 7fa0a67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:57.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.386+0000 7fa0a67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:57.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.550+0000 7fa0b4d19700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.550+0000 7fa0b4d19700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:57.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.550+0000 7fa0b4d19700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:57.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:57.550+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa0b009e920 con 0x7fa0b00a5fb0 2024-09-18T10:16:58.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.150+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 added bucket h28 type host to location {rack=r1} v217) v1 ==== 168+0+0 (secure 0 0 0) 0x7fa098017030 con 0x7fa0b00a5fb0 2024-09-18T10:16:58.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.150+0000 7fa0a67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.150+0000 7fa0a67fc700 10 monclient: _finish_command 2 = system:0 added bucket h28 type host to location {rack=r1} 2024-09-18T10:16:58.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7fa0b009bb50 con 0x7fa0b00a5fb0 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0a67fc700 1 -- 172.21.15.150:0/1480444014 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h28' already exists v217) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa098017070 con 0x7fa0b00a5fb0 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0a67fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h28", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0a67fc700 10 monclient: _finish_command 3 = system:0 bucket 'h28' already exists 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 msgr2=0x7fa090037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 0x7fa090037f50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00099b0 tx=0x7fa0a0016000 comp rx=0 tx=0).stop 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 10 monclient: shutdown 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 msgr2=0x7fa0b012ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa09800b810 tx=0x7fa098009220 comp rx=0 tx=0).stop 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 shutdown_connections 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa090035aa0 0x7fa090037f50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa0b0136a10 0x7fa0b012d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0b00a5fb0 0x7fa0b012ccd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 --2- 172.21.15.150:0/1480444014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa0b00a07d0 0x7fa0b012c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 >> 172.21.15.150:0/1480444014 conn(0x7fa0b0071ae0 msgr2=0x7fa0b009ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:58.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 shutdown_connections 2024-09-18T10:16:58.262 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.254+0000 7fa0b4d19700 1 -- 172.21.15.150:0/1480444014 wait complete. 2024-09-18T10:16:58.262 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h28' already exists 2024-09-18T10:16:58.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:58.271 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h29 host rack=r1 2024-09-18T10:16:58.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: get_monmap_and_config 2024-09-18T10:16:58.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: build_initial_monmap 2024-09-18T10:16:58.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: monmap: 2024-09-18T10:16:58.370 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:58.370 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:58.369184+0000 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:58.369184+0000 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:58.371 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 Processor -- start 2024-09-18T10:16:58.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 -- start start 2024-09-18T10:16:58.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: init 2024-09-18T10:16:58.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:58.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:16:58.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 0x7f48d809c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-a con 0x7f48d80710a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48d809a8c0 0x7f48d809cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-c con 0x7f48d809a8c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48d809d240 addr v1:172.21.15.142:6789/0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d8099c70 con 0x7f48d809d240 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d8071470 con 0x7f48d80710a0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d809ad80 con 0x7f48d809a8c0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.366+0000 7f48dfb97700 10 monclient(hunting): authenticate will time out at 1300.463979s 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48d809a8c0 0x7f48d809cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:58.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48d809a8c0 0x7f48d809cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45540/0 (socket says 172.21.15.150:45540) 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48de134700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f48d809d240 legacy=0x7f48d81365e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 -- 172.21.15.150:0/2073516130 learned_addr learned my addr 172.21.15.150:0/2073516130 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): get_auth_request con 0x7f48d809a8c0 auth_method 0 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd933700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 msgr2=0x7f48d809c7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd933700 1 --2- 172.21.15.150:0/2073516130 >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 0x7f48d809c7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 1 -- 172.21.15.150:0/2073516130 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:58.376 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient(hunting): handle_auth_done global_id 84253 payload 274 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 -- 172.21.15.150:0/2073516130 >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 msgr2=0x7f48d809c7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 --2- 172.21.15.150:0/2073516130 >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 0x7f48d809c7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 -- 172.21.15.150:0/2073516130 >> v1:172.21.15.142:6789/0 conn(0x7f48d809d240 legacy=0x7f48d81365e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient: _finish_hunting 0 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 monclient: found mon.noname-c 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 -- 172.21.15.150:0/2073516130 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8099620 con 0x7f48d809a8c0 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 -- 172.21.15.150:0/2073516130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 msgr2=0x7f48d809cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dd132700 1 --2- 172.21.15.150:0/2073516130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d809cd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f48d809de20 tx=0x7f48c8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=643e3a28d1c3c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 1 -- 172.21.15.150:0/2073516130 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c800ba70 con 0x7f48d809a8c0 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: dump: 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.385 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:58.386 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.386 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:58.386 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: _finish_auth 0 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 5 monclient: authenticate success, global_id 84253 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 1 -- 172.21.15.150:0/2073516130 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c800bbd0 con 0x7f48d809a8c0 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 4 set_mon_vals no callback set 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 1 -- 172.21.15.150:0/2073516130 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8011420 con 0x7f48d809a8c0 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:58.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dc931700 10 monclient: dump: 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 10 monclient: shutdown 2024-09-18T10:16:58.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 -- 172.21.15.150:0/2073516130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 msgr2=0x7f48d809cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 --2- 172.21.15.150:0/2073516130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d809cd00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f48d809de20 tx=0x7f48c8009250 comp rx=0 tx=0).stop 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 -- 172.21.15.150:0/2073516130 shutdown_connections 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 --2- 172.21.15.150:0/2073516130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d809cd00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 --2- 172.21.15.150:0/2073516130 >> v2:172.21.15.142:3300/0 conn(0x7f48d80710a0 0x7f48d809c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 -- 172.21.15.150:0/2073516130 >> 172.21.15.150:0/2073516130 conn(0x7f48d8068590 msgr2=0x7f48d80a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.370+0000 7f48dfb97700 1 -- 172.21.15.150:0/2073516130 shutdown_connections 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 -- 172.21.15.150:0/2073516130 wait complete. 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient: build_initial_monmap 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient: monmap: 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:58.389 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 Processor -- start 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 -- start start 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient: init 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48d80710a0 0x7f48d813c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-c con 0x7f48d80710a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:58.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-b con 0x7f48d809a8c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 0x7f48d813f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): picked mon.noname-a con 0x7f48d809d240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8121250 con 0x7f48d809d240 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8120c90 con 0x7f48d809a8c0 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d811e720 con 0x7f48d80710a0 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dfb97700 10 monclient(hunting): authenticate will time out at 1300.470364s 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:58.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48de134700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 msgr2=0x7f48d813f7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48de134700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 0x7f48d813f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45546/0 (socket says 172.21.15.150:45546) 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 -- 172.21.15.150:0/1995431457 learned_addr learned my addr 172.21.15.150:0/1995431457 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): get_auth_request con 0x7f48d809a8c0 auth_method 0 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd933700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48d80710a0 0x7f48d813c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd933700 10 monclient(hunting): get_auth_request con 0x7f48d80710a0 auth_method 0 2024-09-18T10:16:58.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd933700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd933700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd933700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient(hunting): handle_auth_done global_id 84256 payload 824 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 -- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48d80710a0 msgr2=0x7f48d813c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48d80710a0 0x7f48d813c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 -- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 msgr2=0x7f48d813f7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 0x7f48d813f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient: _finish_hunting 0 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 monclient: found mon.noname-b 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.374+0000 7f48dd132700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c80094f0 con 0x7f48d809a8c0 2024-09-18T10:16:58.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dd132700 1 -- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 msgr2=0x7f48d813b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dd132700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48d809aec0 tx=0x7f48c8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8006b80 con 0x7f48d809a8c0 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: dump: 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:58.394 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: _finish_auth 0 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c8020070 con 0x7f48d809a8c0 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 5 monclient: authenticate success, global_id 84256 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _renew_subs 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d81209b0 con 0x7f48d809a8c0 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c801b6a0 con 0x7f48d809a8c0 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:58.395 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 10 monclient: dump: 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cffff700 4 set_mon_vals no callback set 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _renew_subs 2024-09-18T10:16:58.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d813be50 con 0x7f48d809a8c0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f48c8022030 con 0x7f48d809a8c0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d813c040 con 0x7f48d809a8c0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 0x7f48bc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.378+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(217..217 src has 176..217) v4 ==== 14204+0+0 (secure 0 0 0) 0x7f48c804c790 con 0x7f48d809a8c0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.382+0000 7f48dd933700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 0x7f48bc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:58.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.382+0000 7f48dd933700 10 monclient: get_auth_request con 0x7f48bc035a70 auth_method 0 2024-09-18T10:16:58.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.382+0000 7f48dd933700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 0x7f48bc037f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f48d4003c80 tx=0x7f48d4009570 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:58.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.390+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f48c801ba90 con 0x7f48d809a8c0 2024-09-18T10:16:58.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.390+0000 7f48cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:58.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.390+0000 7f48cdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:58.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.550+0000 7f48dfb97700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.550+0000 7f48dfb97700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:58.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.550+0000 7f48dfb97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:58.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:58.550+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f48d813c040 con 0x7f48d809a8c0 2024-09-18T10:16:59.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.146+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 added bucket h29 type host to location {rack=r1} v218) v1 ==== 168+0+0 (secure 0 0 0) 0x7f48c802ee60 con 0x7f48d809a8c0 2024-09-18T10:16:59.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.146+0000 7f48cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.146+0000 7f48cdffb700 10 monclient: _finish_command 2 = system:0 added bucket h29 type host to location {rack=r1} 2024-09-18T10:16:59.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.254+0000 7f48dfb97700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.254+0000 7f48dfb97700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.254+0000 7f48dfb97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.254+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f48d813c040 con 0x7f48d809a8c0 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48cdffb700 1 -- 172.21.15.150:0/1995431457 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h29' already exists v218) v1 ==== 147+0+0 (secure 0 0 0) 0x7f48c801ab80 con 0x7f48d809a8c0 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48cdffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h29", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.261 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48cdffb700 10 monclient: _finish_command 3 = system:0 bucket 'h29' already exists 2024-09-18T10:16:59.263 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 msgr2=0x7f48bc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 0x7f48bc037f20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f48d4003c80 tx=0x7f48d4009570 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 10 monclient: shutdown 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 msgr2=0x7f48d813b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f48d809aec0 tx=0x7f48c8016000 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 shutdown_connections 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc035a70 0x7f48bc037f20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48d809d240 0x7f48d813f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48d809a8c0 0x7f48d813b910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 --2- 172.21.15.150:0/1995431457 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48d80710a0 0x7f48d813c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 >> 172.21.15.150:0/1995431457 conn(0x7f48d8068590 msgr2=0x7f48d806d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:59.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 shutdown_connections 2024-09-18T10:16:59.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.258+0000 7f48dfb97700 1 -- 172.21.15.150:0/1995431457 wait complete. 2024-09-18T10:16:59.265 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h29' already exists 2024-09-18T10:16:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:16:59.277 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h30 host rack=r1 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 10 monclient: get_monmap_and_config 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 10 monclient: build_initial_monmap 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 10 monclient: monmap: 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:16:59.378678+0000 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:16:59.378678+0000 2024-09-18T10:16:59.380 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:16:59.381 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 1 Processor -- start 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 1 -- start start 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.374+0000 7f55dba43700 10 monclient: init 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 0x7f55d40a0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-a con 0x7f55d409d000 addr v2:172.21.15.142:3300/0 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 0x7f55d40a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.384 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-b con 0x7f55d40a0740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55d4136a80 0x7f55d4138e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-c con 0x7f55d4136a80 addr v2:172.21.15.150:3300/0 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d409e240 con 0x7f55d409d000 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40a1680 con 0x7f55d4136a80 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d409d3d0 con 0x7f55d40a0740 2024-09-18T10:16:59.385 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient(hunting): authenticate will time out at 1301.472262s 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d97df700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 msgr2=0x7f55d40a0200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d97df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 0x7f55d40a0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55d4136a80 0x7f55d4138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55d4136a80 0x7f55d4138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44036/0 (socket says 172.21.15.150:44036) 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 -- 172.21.15.150:0/1963422317 learned_addr learned my addr 172.21.15.150:0/1963422317 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): get_auth_request con 0x7f55d4136a80 auth_method 0 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:59.386 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 0x7f55d40a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 10 monclient(hunting): get_auth_request con 0x7f55d40a0740 auth_method 0 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient(hunting): handle_auth_done global_id 84259 payload 274 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 -- 172.21.15.150:0/1963422317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 msgr2=0x7f55d40a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 0x7f55d40a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 -- 172.21.15.150:0/1963422317 >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 msgr2=0x7f55d40a0200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 --2- 172.21.15.150:0/1963422317 >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 0x7f55d40a0200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient: _finish_hunting 0 2024-09-18T10:16:59.387 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 monclient: found mon.noname-c 2024-09-18T10:16:59.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:59.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 -- 172.21.15.150:0/1963422317 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40a1e60 con 0x7f55d4136a80 2024-09-18T10:16:59.388 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d8fde700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 0x7f55d40a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 -- 172.21.15.150:0/1963422317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d4136a80 msgr2=0x7f55d4138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55d9fe0700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d4136a80 0x7f55d4138e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f55d4069aa0 tx=0x7f55cc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b04d12e9e6f7f7ac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 1 -- 172.21.15.150:0/1963422317 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc00e070 con 0x7f55d4136a80 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: dump: 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.396 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: _finish_auth 0 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 5 monclient: authenticate success, global_id 84259 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 1 -- 172.21.15.150:0/1963422317 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc00bc60 con 0x7f55d4136a80 2024-09-18T10:16:59.397 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 1 -- 172.21.15.150:0/1963422317 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0092e0 con 0x7f55d4136a80 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 10 monclient: get_monmap_and_config success 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55dba43700 4 set_mon_vals no callback set 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.378+0000 7f55cbfff700 10 monclient: dump: 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:59.398 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: shutdown 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- 172.21.15.150:0/1963422317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d4136a80 msgr2=0x7f55d4138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d4136a80 0x7f55d4138e60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f55d4069aa0 tx=0x7f55cc009840 comp rx=0 tx=0).stop 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- 172.21.15.150:0/1963422317 shutdown_connections 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d4136a80 0x7f55d4138e60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- 172.21.15.150:0/1963422317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d40a0740 0x7f55d40a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- 172.21.15.150:0/1963422317 >> v2:172.21.15.142:3300/0 conn(0x7f55d409d000 0x7f55d40a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- 172.21.15.150:0/1963422317 >> 172.21.15.150:0/1963422317 conn(0x7f55d40a5910 msgr2=0x7f55d40a5d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- 172.21.15.150:0/1963422317 shutdown_connections 2024-09-18T10:16:59.399 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- 172.21.15.150:0/1963422317 wait complete. 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: build_initial_monmap 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: monmap: 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 Processor -- start 2024-09-18T10:16:59.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- start start 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: init 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: _reopen_session rank -1 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-b con 0x7f55d409d000 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 0x7f55d406e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-a con 0x7f55d40a0740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d4136a80 0x7f55d406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): picked mon.noname-c con 0x7f55d4136a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4135b30 con 0x7f55d40a0740 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d408dd20 con 0x7f55d409d000 2024-09-18T10:16:59.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): start opening mon connection 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d408e000 con 0x7f55d4136a80 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): _renew_subs 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55dba43700 10 monclient(hunting): authenticate will time out at 1301.479434s 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44038/0 (socket says 172.21.15.150:44038) 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 -- 172.21.15.150:0/1524674338 learned_addr learned my addr 172.21.15.150:0/1524674338 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:16:59.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): get_auth_request con 0x7f55d409d000 auth_method 0 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d8fde700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 msgr2=0x7f55d406e6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d8fde700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 0x7f55d406e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d9fe0700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d4136a80 0x7f55d406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d9fe0700 10 monclient(hunting): get_auth_request con 0x7f55d4136a80 auth_method 0 2024-09-18T10:16:59.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d9fe0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d9fe0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d9fe0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient(hunting): handle_auth_done global_id 84262 payload 824 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d4136a80 msgr2=0x7f55d406ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d4136a80 0x7f55d406ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 msgr2=0x7f55d406e6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 0x7f55d406e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 10 monclient: _finish_hunting 0 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.382+0000 7f55d97df700 1 monclient: found mon.noname-b 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55d97df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55d97df700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55d97df700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc009ae0 con 0x7f55d409d000 2024-09-18T10:16:59.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55d97df700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 msgr2=0x7f55d406e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55d97df700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55c400b7e0 tx=0x7f55c4009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4015070 con 0x7f55d409d000 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: dump: 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:59.405 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: _finish_auth 0 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 5 monclient: authenticate success, global_id 84262 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c4003e20 con 0x7f55d409d000 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 10 monclient: _renew_subs 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d4097160 con 0x7f55d409d000 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4006380 con 0x7f55d409d000 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:16:59.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55cb7fe700 4 set_mon_vals no callback set 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55c97fa700 10 monclient: dump: 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 10 monclient: _renew_subs 2024-09-18T10:16:59.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.386+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d408c6d0 con 0x7f55d409d000 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55dba43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55dba43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55dba43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d408c950 con 0x7f55d409d000 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f55c4006770 con 0x7f55d409d000 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55c97fa700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 0x7f55b8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(218..218 src has 176..218) v4 ==== 14417+0+0 (secure 0 0 0) 0x7f55c404a470 con 0x7f55d409d000 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55d8fde700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 0x7f55b8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:16:59.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.390+0000 7f55d8fde700 10 monclient: get_auth_request con 0x7f55b8035a70 auth_method 0 2024-09-18T10:16:59.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.394+0000 7f55d8fde700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 0x7f55b8037f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f55d0006f00 tx=0x7f55d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:16:59.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.402+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f55c401a410 con 0x7f55d409d000 2024-09-18T10:16:59.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.402+0000 7f55c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:16:59.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.402+0000 7f55c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:16:59.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.570+0000 7f55dba43700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.570+0000 7f55dba43700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:16:59.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.570+0000 7f55dba43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:16:59.576 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:16:59.570+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f55d4063ed0 con 0x7f55d409d000 2024-09-18T10:17:00.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.162+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 added bucket h30 type host to location {rack=r1} v219) v1 ==== 168+0+0 (secure 0 0 0) 0x7f55c4018310 con 0x7f55d409d000 2024-09-18T10:17:00.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.162+0000 7f55c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.162+0000 7f55c97fa700 10 monclient: _finish_command 2 = system:0 added bucket h30 type host to location {rack=r1} 2024-09-18T10:17:00.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f55d40008d0 con 0x7f55d409d000 2024-09-18T10:17:00.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55c97fa700 1 -- 172.21.15.150:0/1524674338 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h30' already exists v219) v1 ==== 147+0+0 (secure 0 0 0) 0x7f55c40184f0 con 0x7f55d409d000 2024-09-18T10:17:00.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55c97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h30", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55c97fa700 10 monclient: _finish_command 3 = system:0 bucket 'h30' already exists 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 msgr2=0x7f55b8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 0x7f55b8037f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f55d0006f00 tx=0x7f55d0008040 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 10 monclient: shutdown 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 msgr2=0x7f55d406e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55c400b7e0 tx=0x7f55c4009220 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 shutdown_connections 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55b8035a70 0x7f55b8037f20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55d4136a80 0x7f55d406ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d40a0740 0x7f55d406e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 --2- 172.21.15.150:0/1524674338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d409d000 0x7f55d406e180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 >> 172.21.15.150:0/1524674338 conn(0x7f55d40a5910 msgr2=0x7f55d4098880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:00.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 shutdown_connections 2024-09-18T10:17:00.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.270+0000 7f55dba43700 1 -- 172.21.15.150:0/1524674338 wait complete. 2024-09-18T10:17:00.278 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h30' already exists 2024-09-18T10:17:00.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:00.286 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h31 host rack=r1 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: get_monmap_and_config 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: build_initial_monmap 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: monmap: 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:00.384957+0000 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:00.384957+0000 2024-09-18T10:17:00.390 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 Processor -- start 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 -- start start 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: init 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:17:00.391 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5cf40a5fb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 0x7f5cf409e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-b con 0x7f5cf409fb20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40a09a0 0x7f5cf4139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-c con 0x7f5cf40a09a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf409df10 con 0x7f5cf40a5fb0 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf409d1e0 con 0x7f5cf40a09a0 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf409dca0 con 0x7f5cf409fb20 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cfaa27700 10 monclient(hunting): authenticate will time out at 1302.478524s 2024-09-18T10:17:00.392 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40a09a0 0x7f5cf4139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf3fff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f5cf40a5fb0 legacy=0x7f5cf40a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40a09a0 0x7f5cf4139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44048/0 (socket says 172.21.15.150:44048) 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/957779763 learned_addr learned my addr 172.21.15.150:0/957779763 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): get_auth_request con 0x7f5cf40a09a0 auth_method 0 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf37fe700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 0x7f5cf409e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_done global_id 84265 payload 274 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/957779763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 msgr2=0x7f5cf409e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:00.393 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 0x7f5cf409e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/957779763 >> v1:172.21.15.142:6789/0 conn(0x7f5cf40a5fb0 legacy=0x7f5cf40a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient: _finish_hunting 0 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 monclient: found mon.noname-c 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/957779763 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40a6380 con 0x7f5cf40a09a0 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf37fe700 10 monclient: get_auth_request con 0x7f5cf409fb20 auth_method 0 2024-09-18T10:17:00.394 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.382+0000 7f5cf37fe700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 0x7f5cf409e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:17:00.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/957779763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40a09a0 msgr2=0x7f5cf4139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:00.400 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf8fc4700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40a09a0 0x7f5cf4139a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009f20 tx=0x7f5cec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327b9ff05a95041a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 1 -- 172.21.15.150:0/957779763 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec00e070 con 0x7f5cf40a09a0 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: dump: 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.401 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 5 monclient: authenticate success, global_id 84265 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 1 -- 172.21.15.150:0/957779763 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec00bc90 con 0x7f5cf40a09a0 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 4 set_mon_vals no callback set 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 1 -- 172.21.15.150:0/957779763 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec012660 con 0x7f5cf40a09a0 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:00.402 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cf2ffd700 10 monclient: dump: 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 10 monclient: shutdown 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:00.403 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 -- 172.21.15.150:0/957779763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40a09a0 msgr2=0x7f5cf4139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40a09a0 0x7f5cf4139a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009f20 tx=0x7f5cec009840 comp rx=0 tx=0).stop 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 -- 172.21.15.150:0/957779763 shutdown_connections 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40a09a0 0x7f5cf4139a30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/957779763 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf409fb20 0x7f5cf409e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 -- 172.21.15.150:0/957779763 >> 172.21.15.150:0/957779763 conn(0x7f5cf4071ae0 msgr2=0x7f5cf40a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 -- 172.21.15.150:0/957779763 shutdown_connections 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 1 -- 172.21.15.150:0/957779763 wait complete. 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 10 monclient: build_initial_monmap 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.386+0000 7f5cfaa27700 10 monclient: monmap: 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:00.404 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 Processor -- start 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 -- start start 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient: init 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 0x7f5cf412da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-a con 0x7f5cf40a09a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf40a5fb0 0x7f5cf406f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-c con 0x7f5cf40a5fb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): picked mon.noname-b con 0x7f5cf406f640 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf409be50 con 0x7f5cf40a09a0 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40994e0 con 0x7f5cf406f640 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf406bd10 con 0x7f5cf40a5fb0 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cfaa27700 10 monclient(hunting): authenticate will time out at 1302.485166s 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf3fff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 msgr2=0x7f5cf412da20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 0x7f5cf412da20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:00.406 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf40a5fb0 0x7f5cf406f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44064/0 (socket says 172.21.15.150:44064) 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/2414330639 learned_addr learned my addr 172.21.15.150:0/2414330639 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): get_auth_request con 0x7f5cf406f640 auth_method 0 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): get_auth_request con 0x7f5cf40a5fb0 auth_method 0 2024-09-18T10:17:00.407 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient(hunting): handle_auth_done global_id 84268 payload 824 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf40a5fb0 msgr2=0x7f5cf406f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf40a5fb0 0x7f5cf406f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 msgr2=0x7f5cf412da20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:00.408 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 0x7f5cf412da20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient: _finish_hunting 0 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 monclient: found mon.noname-b 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.390+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec009ae0 con 0x7f5cf406f640 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cf8fc4700 1 -- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 msgr2=0x7f5cf406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cf8fc4700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00b6c0 tx=0x7f5cec017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec012e70 con 0x7f5cf406f640 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: dump: 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.409 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: _finish_auth 0 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 5 monclient: authenticate success, global_id 84268 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _renew_subs 2024-09-18T10:17:00.410 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec012690 con 0x7f5cf406f640 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf406c3d0 con 0x7f5cf406f640 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec01d650 con 0x7f5cf406f640 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 10 monclient: dump: 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.411 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _renew_subs 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf406d330 con 0x7f5cf406f640 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cf27fc700 4 set_mon_vals no callback set 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.412 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5cec01da40 con 0x7f5cf406f640 2024-09-18T10:17:00.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 0x7f5ce0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(219..219 src has 176..219) v4 ==== 14630+0+0 (secure 0 0 0) 0x7f5cec04c250 con 0x7f5cf406f640 2024-09-18T10:17:00.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cf3fff700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 0x7f5ce0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.394+0000 7f5cf3fff700 10 monclient: get_auth_request con 0x7f5ce0035a70 auth_method 0 2024-09-18T10:17:00.413 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.398+0000 7f5cf3fff700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 0x7f5ce0037f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8003980 tx=0x7f5ce8006dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.398+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:17:00.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.402+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.402+0000 7f5cdbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.402+0000 7f5cdbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:00.571 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.566+0000 7f5cfaa27700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.566+0000 7f5cfaa27700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.566+0000 7f5cfaa27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.572 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.566+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.638+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 added bucket h31 type host to location {rack=r1} v220) v1 ==== 168+0+0 (secure 0 0 0) 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.638+0000 7f5cdbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.638+0000 7f5cdbfff700 10 monclient: _finish_command 2 = system:0 added bucket h31 type host to location {rack=r1} 2024-09-18T10:17:00.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cfaa27700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cfaa27700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cfaa27700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:00.750 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cdbfff700 1 -- 172.21.15.150:0/2414330639 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h31' already exists v220) v1 ==== 147+0+0 (secure 0 0 0) 0x7f5cf406d670 con 0x7f5cf406f640 2024-09-18T10:17:00.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cdbfff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h31", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:00.751 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.746+0000 7f5cdbfff700 10 monclient: _finish_command 3 = system:0 bucket 'h31' already exists 2024-09-18T10:17:00.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 msgr2=0x7f5ce0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 0x7f5ce0037f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8003980 tx=0x7f5ce8006dc0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 10 monclient: shutdown 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 msgr2=0x7f5cf406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5cec00b6c0 tx=0x7f5cec017000 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 shutdown_connections 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5ce0035a70 0x7f5ce0037f20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf406f640 0x7f5cf406ac50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5cf40a5fb0 0x7f5cf406f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 --2- 172.21.15.150:0/2414330639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5cf40a09a0 0x7f5cf412da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 >> 172.21.15.150:0/2414330639 conn(0x7f5cf4071ae0 msgr2=0x7f5cf40a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 shutdown_connections 2024-09-18T10:17:00.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.750+0000 7f5cfaa27700 1 -- 172.21.15.150:0/2414330639 wait complete. 2024-09-18T10:17:00.755 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h31' already exists 2024-09-18T10:17:00.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:00.763 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush add-bucket h32 host rack=r1 2024-09-18T10:17:00.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: get_monmap_and_config 2024-09-18T10:17:00.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: build_initial_monmap 2024-09-18T10:17:00.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: monmap: 2024-09-18T10:17:00.865 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:00.865 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:00.864329+0000 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:00.864329+0000 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:00.866 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 Processor -- start 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 -- start start 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: init 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 0x7f8d640a0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-b con 0x7f8d6409d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d6409e780 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 0x7f8d64139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-a con 0x7f8d641379b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:00.869 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640699a0 con 0x7f8d6409e780 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640a1640 con 0x7f8d641379b0 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6409e3d0 con 0x7f8d6409d000 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6c79c700 10 monclient(hunting): authenticate will time out at 1302.957876s 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d69d37700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8d6409e780 0x7f8d640a0660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34434/0 (socket says 172.21.15.150:34434) 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d69d37700 1 -- 172.21.15.150:0/2223347614 learned_addr learned my addr 172.21.15.150:0/2223347614 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6ad39700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 msgr2=0x7f8d64139d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6ad39700 1 --2- 172.21.15.150:0/2223347614 >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 0x7f8d64139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 --2- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 0x7f8d640a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.870 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): get_auth_request con 0x7f8d6409d000 auth_method 0 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient(hunting): handle_auth_done global_id 74429 payload 274 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 -- 172.21.15.150:0/2223347614 >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 msgr2=0x7f8d64139d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 --2- 172.21.15.150:0/2223347614 >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 0x7f8d64139d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 -- 172.21.15.150:0/2223347614 >> v1:172.21.15.150:6789/0 conn(0x7f8d6409e780 legacy=0x7f8d640a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient: _finish_hunting 0 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 monclient: found mon.noname-b 2024-09-18T10:17:00.871 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:00.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:00.872 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.862+0000 7f8d6a538700 1 -- 172.21.15.150:0/2223347614 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6409ebf0 con 0x7f8d6409d000 2024-09-18T10:17:00.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6a538700 1 -- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 msgr2=0x7f8d640a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6a538700 1 --2- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 0x7f8d640a0120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d6409ede0 tx=0x7f8d540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0709b8b402d8e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 1 -- 172.21.15.150:0/2223347614 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400b9a0 con 0x7f8d6409d000 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: dump: 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.874 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: _finish_auth 0 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 5 monclient: authenticate success, global_id 74429 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 1 -- 172.21.15.150:0/2223347614 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5400bb20 con 0x7f8d6409d000 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:00.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 4 set_mon_vals no callback set 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 1 -- 172.21.15.150:0/2223347614 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54011420 con 0x7f8d6409d000 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d69536700 10 monclient: dump: 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.876 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 10 monclient: shutdown 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 -- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 msgr2=0x7f8d640a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 0x7f8d640a0120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d6409ede0 tx=0x7f8d540091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 -- 172.21.15.150:0/2223347614 shutdown_connections 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/2223347614 >> v2:172.21.15.142:3300/0 conn(0x7f8d641379b0 0x7f8d64139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/2223347614 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6409d000 0x7f8d640a0120 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 -- 172.21.15.150:0/2223347614 >> 172.21.15.150:0/2223347614 conn(0x7f8d640a5830 msgr2=0x7f8d640a5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:00.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 -- 172.21.15.150:0/2223347614 shutdown_connections 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 1 -- 172.21.15.150:0/2223347614 wait complete. 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 10 monclient: build_initial_monmap 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.866+0000 7f8d6c79c700 10 monclient: monmap: 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 Processor -- start 2024-09-18T10:17:00.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 -- start start 2024-09-18T10:17:00.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient: init 2024-09-18T10:17:00.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:00.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:00.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 0x7f8d6412c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-a con 0x7f8d6409d000 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 0x7f8d6412cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-b con 0x7f8d6409e780 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 0x7f8d6412d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): picked mon.noname-c con 0x7f8d641379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d64130160 con 0x7f8d6409d000 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6412e640 con 0x7f8d6409e780 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6412f570 con 0x7f8d641379b0 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6c79c700 10 monclient(hunting): authenticate will time out at 1302.965453s 2024-09-18T10:17:00.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6a538700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 msgr2=0x7f8d6412c760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6a538700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 0x7f8d6412c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 0x7f8d6412cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 0x7f8d6412cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44066/0 (socket says 172.21.15.150:44066) 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 0x7f8d6412d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 1 -- 172.21.15.150:0/4067223515 learned_addr learned my addr 172.21.15.150:0/4067223515 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): get_auth_request con 0x7f8d6409e780 auth_method 0 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): get_auth_request con 0x7f8d641379b0 auth_method 0 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:00.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d69d37700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient(hunting): handle_auth_done global_id 74432 payload 824 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 -- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 msgr2=0x7f8d6412cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 0x7f8d6412cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 -- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 msgr2=0x7f8d6412c760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 0x7f8d6412c760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient: _finish_hunting 0 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 monclient: found mon.noname-c 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.870+0000 7f8d6ad39700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54009420 con 0x7f8d641379b0 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6ad39700 1 -- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 msgr2=0x7f8d6412d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6ad39700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 0x7f8d6412d1e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c003d00 tx=0x7f8d5c0096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00ce90 con 0x7f8d641379b0 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: dump: 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.886 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 5 monclient: authenticate success, global_id 74432 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _renew_subs 2024-09-18T10:17:00.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d64130aa0 con 0x7f8d641379b0 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c006360 con 0x7f8d641379b0 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c006510 con 0x7f8d641379b0 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 10 monclient: dump: 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:00.888 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d68d35700 4 set_mon_vals no callback set 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _renew_subs 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6412de90 con 0x7f8d641379b0 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8d5c012020 con 0x7f8d641379b0 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:00.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d64063ed0 con 0x7f8d641379b0 2024-09-18T10:17:00.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 0x7f8d48037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:00.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(220..220 src has 176..220) v4 ==== 14843+0+0 (secure 0 0 0) 0x7f8d5c04ad70 con 0x7f8d641379b0 2024-09-18T10:17:00.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6a538700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 0x7f8d48037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:00.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.874+0000 7f8d6a538700 10 monclient: get_auth_request con 0x7f8d48035aa0 auth_method 0 2024-09-18T10:17:00.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.878+0000 7f8d6a538700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 0x7f8d48037f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8d6409ede0 tx=0x7f8d54017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:00.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.890+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8d5c012500 con 0x7f8d641379b0 2024-09-18T10:17:00.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.890+0000 7f8d5a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:00.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:00.890+0000 7f8d5a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:01.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.050+0000 7f8d6c79c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.054+0000 7f8d6c79c700 10 monclient: _send_command 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.057 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.054+0000 7f8d6c79c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:01.058 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.054+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8d64067b10 con 0x7f8d641379b0 2024-09-18T10:17:01.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.646+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 added bucket h32 type host to location {rack=r1} v221) v1 ==== 168+0+0 (secure 0 0 0) 0x7f8d5c013ad0 con 0x7f8d641379b0 2024-09-18T10:17:01.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.646+0000 7f8d5a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.646+0000 7f8d5a7fc700 10 monclient: _finish_command 2 = system:0 added bucket h32 type host to location {rack=r1} 2024-09-18T10:17:01.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.750+0000 7f8d6c79c700 10 start_mon_command cmd=[{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.750+0000 7f8d6c79c700 10 monclient: _send_command 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.750+0000 7f8d6c79c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:01.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.750+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]} v 0) v1 -- 0x7f8d6412d720 con 0x7f8d641379b0 2024-09-18T10:17:01.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.754+0000 7f8d5a7fc700 1 -- 172.21.15.150:0/4067223515 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}]=0 bucket 'h32' already exists v221) v1 ==== 147+0+0 (secure 0 0 0) 0x7f8d5c0122d0 con 0x7f8d641379b0 2024-09-18T10:17:01.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.754+0000 7f8d5a7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush add-bucket", "name": "h32", "type": "host", "args": ["rack=r1"]}] 2024-09-18T10:17:01.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.754+0000 7f8d5a7fc700 10 monclient: _finish_command 3 = system:0 bucket 'h32' already exists 2024-09-18T10:17:01.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 msgr2=0x7f8d48037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:01.762 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 0x7f8d48037f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8d6409ede0 tx=0x7f8d54017000 comp rx=0 tx=0).stop 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 10 monclient: shutdown 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 msgr2=0x7f8d6412d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 0x7f8d6412d1e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c003d00 tx=0x7f8d5c0096a0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 shutdown_connections 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8d48035aa0 0x7f8d48037f50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d641379b0 0x7f8d6412d1e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d6409e780 0x7f8d6412cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 --2- 172.21.15.150:0/4067223515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6409d000 0x7f8d6412c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 >> 172.21.15.150:0/4067223515 conn(0x7f8d640a5830 msgr2=0x7f8d6409abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:01.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 shutdown_connections 2024-09-18T10:17:01.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.758+0000 7f8d6c79c700 1 -- 172.21.15.150:0/4067223515 wait complete. 2024-09-18T10:17:01.765 INFO:tasks.workunit.client.0.smithi150.stderr:bucket 'h32' already exists 2024-09-18T10:17:01.773 INFO:tasks.workunit.client.0.smithi150.stderr:++ seq 1 32 2024-09-18T10:17:01.774 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:01.774 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h1 2024-09-18T10:17:01.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: get_monmap_and_config 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: build_initial_monmap 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: monmap: 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:01.866557+0000 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:01.866557+0000 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:01.868 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:01.869 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:01.869 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:01.869 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:01.869 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 Processor -- start 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 -- start start 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: init 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:17:01.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 0x7f5ea809b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-b con 0x7f5ea809ad30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ea8099180 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5ea80995b0 0x7f5ea809c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-c con 0x7f5ea80995b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea809a5b0 con 0x7f5ea8099180 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea809a080 con 0x7f5ea80995b0 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8099ca0 con 0x7f5ea809ad30 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.862+0000 7f5eb0205700 10 monclient(hunting): authenticate will time out at 1303.959909s 2024-09-18T10:17:01.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5ead7a0700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f5ea8099180 legacy=0x7f5ea809bfd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 0x7f5ea809b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5ea80995b0 0x7f5ea809c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 0x7f5ea809b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:58410/0 (socket says 172.21.15.150:58410) 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 1 -- 172.21.15.150:0/1568430195 learned_addr learned my addr 172.21.15.150:0/1568430195 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): get_auth_request con 0x7f5ea809ad30 auth_method 0 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): get_auth_request con 0x7f5ea80995b0 auth_method 0 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:01.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient(hunting): handle_auth_done global_id 84274 payload 274 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 -- 172.21.15.150:0/1568430195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 msgr2=0x7f5ea809b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 --2- 172.21.15.150:0/1568430195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 0x7f5ea809b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 -- 172.21.15.150:0/1568430195 >> v1:172.21.15.142:6789/0 conn(0x7f5ea8099180 legacy=0x7f5ea809bfd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient: _finish_hunting 0 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 monclient: found mon.noname-c 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 -- 172.21.15.150:0/1568430195 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea809dd00 con 0x7f5ea80995b0 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 -- 172.21.15.150:0/1568430195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea80995b0 msgr2=0x7f5ea809c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eae7a2700 1 --2- 172.21.15.150:0/1568430195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea80995b0 0x7f5ea809c6e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009e60 tx=0x7f5ea0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2258bdc656c3d3c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 1 -- 172.21.15.150:0/1568430195 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000e070 con 0x7f5ea80995b0 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: dump: 2024-09-18T10:17:01.878 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: _finish_auth 0 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eb0205700 5 monclient: authenticate success, global_id 84274 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eb0205700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 1 -- 172.21.15.150:0/1568430195 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea000bd70 con 0x7f5ea80995b0 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eb0205700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eb0205700 4 set_mon_vals no callback set 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 1 -- 172.21.15.150:0/1568430195 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0012660 con 0x7f5ea80995b0 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.866+0000 7f5eacf9f700 10 monclient: dump: 2024-09-18T10:17:01.880 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: shutdown 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- 172.21.15.150:0/1568430195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea80995b0 msgr2=0x7f5ea809c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:01.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- 172.21.15.150:0/1568430195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea80995b0 0x7f5ea809c6e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009e60 tx=0x7f5ea0009780 comp rx=0 tx=0).stop 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- 172.21.15.150:0/1568430195 shutdown_connections 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- 172.21.15.150:0/1568430195 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea80995b0 0x7f5ea809c6e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- 172.21.15.150:0/1568430195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea809ad30 0x7f5ea809b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- 172.21.15.150:0/1568430195 >> 172.21.15.150:0/1568430195 conn(0x7f5ea80a15e0 msgr2=0x7f5ea80a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- 172.21.15.150:0/1568430195 shutdown_connections 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- 172.21.15.150:0/1568430195 wait complete. 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: build_initial_monmap 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: monmap: 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:01.882 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:01.883 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 Processor -- start 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- start start 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: init 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-b con 0x7f5ea8099180 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 0x7f5ea8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-c con 0x7f5ea80995b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:01.888 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 0x7f5ea8145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): picked mon.noname-a con 0x7f5ea809ad30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea809b9b0 con 0x7f5ea809ad30 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8098350 con 0x7f5ea8099180 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8143000 con 0x7f5ea80995b0 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eb0205700 10 monclient(hunting): authenticate will time out at 1303.967055s 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:01.889 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44070/0 (socket says 172.21.15.150:44070) 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 -- 172.21.15.150:0/3267901564 learned_addr learned my addr 172.21.15.150:0/3267901564 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): get_auth_request con 0x7f5ea8099180 auth_method 0 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eae7a2700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 msgr2=0x7f5ea8145a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eae7a2700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 0x7f5ea8145a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5ead7a0700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 0x7f5ea8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:01.890 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient(hunting): handle_auth_done global_id 84277 payload 824 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 msgr2=0x7f5ea8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 0x7f5ea8141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 msgr2=0x7f5ea8145a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 0x7f5ea8145a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient: _finish_hunting 0 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 monclient: found mon.noname-b 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5eadfa1700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0009a20 con 0x7f5ea8099180 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5ead7a0700 10 monclient: get_auth_request con 0x7f5ea80995b0 auth_method 0 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.870+0000 7f5ead7a0700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 0x7f5ea8141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:17:01.891 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eadfa1700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 msgr2=0x7f5ea8142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eadfa1700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400ea20 tx=0x7f5ea4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea400ca60 con 0x7f5ea8099180 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: dump: 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:01.892 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:01.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 5 monclient: authenticate success, global_id 84277 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea400cbc0 con 0x7f5ea8099180 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 10 monclient: _renew_subs 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea8126f00 con 0x7f5ea8099180 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4013470 con 0x7f5ea8099180 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9ffff700 4 set_mon_vals no callback set 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 10 monclient: dump: 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:01.894 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 10 monclient: _renew_subs 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea806ecd0 con 0x7f5ea8099180 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.874+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5ea40138a0 con 0x7f5ea8099180 2024-09-18T10:17:01.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5eb0205700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5eb0205700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5eb0205700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5e9dffb700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 0x7f5e8c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea8063ed0 con 0x7f5ea8099180 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(221..221 src has 176..221) v4 ==== 15056+0+0 (secure 0 0 0) 0x7f5ea404b3e0 con 0x7f5ea8099180 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5ead7a0700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 0x7f5e8c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5ead7a0700 10 monclient: get_auth_request con 0x7f5e8c035a70 auth_method 0 2024-09-18T10:17:01.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.878+0000 7f5ead7a0700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 0x7f5e8c037f20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e98003980 tx=0x7f5e98008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:01.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.886+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5ea4023300 con 0x7f5ea8099180 2024-09-18T10:17:01.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.886+0000 7f5e9dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:01.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:01.886+0000 7f5e9dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:02.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.058+0000 7f5eb0205700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.058+0000 7f5eb0205700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.058+0000 7f5eb0205700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.062 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.058+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f5ea8126cc0 con 0x7f5ea8099180 2024-09-18T10:17:02.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.646+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 removed item id -19 name 'h1' from crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5ea40080a0 con 0x7f5ea8099180 2024-09-18T10:17:02.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.646+0000 7f5e9dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.646+0000 7f5e9dffb700 10 monclient: _finish_command 2 = system:0 removed item id -19 name 'h1' from crush map 2024-09-18T10:17:02.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5eb0205700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5eb0205700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5eb0205700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h1"} v 0) v1 -- 0x7f5ea80008d0 con 0x7f5ea8099180 2024-09-18T10:17:02.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5e9dffb700 1 -- 172.21.15.150:0/3267901564 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h1"}]=0 device 'h1' does not appear in the crush map v222) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5ea401b3b0 con 0x7f5ea8099180 2024-09-18T10:17:02.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5e9dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h1"}] 2024-09-18T10:17:02.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.754+0000 7f5e9dffb700 10 monclient: _finish_command 3 = system:0 device 'h1' does not appear in the crush map 2024-09-18T10:17:02.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 msgr2=0x7f5e8c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:02.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 0x7f5e8c037f20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5e98003980 tx=0x7f5e98008040 comp rx=0 tx=0).stop 2024-09-18T10:17:02.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 10 monclient: shutdown 2024-09-18T10:17:02.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:02.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 msgr2=0x7f5ea8142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400ea20 tx=0x7f5ea4009250 comp rx=0 tx=0).stop 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 shutdown_connections 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5e8c035a70 0x7f5e8c037f20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ea809ad30 0x7f5ea8145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5ea80995b0 0x7f5ea8141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 --2- 172.21.15.150:0/3267901564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ea8099180 0x7f5ea8142540 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 >> 172.21.15.150:0/3267901564 conn(0x7f5ea80a15e0 msgr2=0x7f5ea80a0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 shutdown_connections 2024-09-18T10:17:02.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.758+0000 7f5eb0205700 1 -- 172.21.15.150:0/3267901564 wait complete. 2024-09-18T10:17:02.765 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h1' does not appear in the crush map 2024-09-18T10:17:02.773 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:02.773 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h2 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 10 monclient: get_monmap_and_config 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 10 monclient: build_initial_monmap 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 10 monclient: monmap: 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:02.873740+0000 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:02.873740+0000 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:02.875 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:02.876 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:02.876 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:02.876 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 1 Processor -- start 2024-09-18T10:17:02.876 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 1 -- start start 2024-09-18T10:17:02.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.870+0000 7f155d52b700 10 monclient: init 2024-09-18T10:17:02.878 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15580a2740 0x7f1558096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-c con 0x7f15580a2740 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1558096fb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 0x7f15581365e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-a con 0x7f1558097ac0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580983a0 con 0x7f1558096fb0 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580a2b10 con 0x7f1558097ac0 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155806f570 con 0x7f15580a2740 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:02.879 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient(hunting): authenticate will time out at 1304.968533s 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15580a2740 0x7f1558096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f15567fc700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f1558096fb0 legacy=0x7f15580973b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f15577fe700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 msgr2=0x7f15581365e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15580a2740 0x7f1558096a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44074/0 (socket says 172.21.15.150:44074) 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 -- 172.21.15.150:0/3054314511 learned_addr learned my addr 172.21.15.150:0/3054314511 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): get_auth_request con 0x7f15580a2740 auth_method 0 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f15577fe700 1 --2- 172.21.15.150:0/3054314511 >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 0x7f15581365e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:02.880 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient(hunting): handle_auth_done global_id 84280 payload 274 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 -- 172.21.15.150:0/3054314511 >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 msgr2=0x7f15581365e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 --2- 172.21.15.150:0/3054314511 >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 0x7f15581365e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 -- 172.21.15.150:0/3054314511 >> v1:172.21.15.142:6789/0 conn(0x7f1558096fb0 legacy=0x7f15580973b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 monclient: found mon.noname-c 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.881 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 -- 172.21.15.150:0/3054314511 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1558098680 con 0x7f15580a2740 2024-09-18T10:17:02.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 -- 172.21.15.150:0/3054314511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15580a2740 msgr2=0x7f1558096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:02.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1556ffd700 1 --2- 172.21.15.150:0/3054314511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15580a2740 0x7f1558096a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f155809bcd0 tx=0x7f1548009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c6875c52a788792 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:02.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 1 -- 172.21.15.150:0/3054314511 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154800ba40 con 0x7f15580a2740 2024-09-18T10:17:02.883 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: dump: 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 5 monclient: authenticate success, global_id 84280 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 1 -- 172.21.15.150:0/3054314511 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154800bba0 con 0x7f15580a2740 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 1 -- 172.21.15.150:0/3054314511 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548011420 con 0x7f15580a2740 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f155d52b700 4 set_mon_vals no callback set 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.874+0000 7f1555ffb700 10 monclient: dump: 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:02.885 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 10 monclient: shutdown 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- 172.21.15.150:0/3054314511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15580a2740 msgr2=0x7f1558096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 --2- 172.21.15.150:0/3054314511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15580a2740 0x7f1558096a70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f155809bcd0 tx=0x7f1548009250 comp rx=0 tx=0).stop 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- 172.21.15.150:0/3054314511 shutdown_connections 2024-09-18T10:17:02.886 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 --2- 172.21.15.150:0/3054314511 >> v2:172.21.15.142:3300/0 conn(0x7f1558097ac0 0x7f15581365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 --2- 172.21.15.150:0/3054314511 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15580a2740 0x7f1558096a70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- 172.21.15.150:0/3054314511 >> 172.21.15.150:0/3054314511 conn(0x7f1558068220 msgr2=0x7f15580a1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- 172.21.15.150:0/3054314511 shutdown_connections 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- 172.21.15.150:0/3054314511 wait complete. 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 10 monclient: build_initial_monmap 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 10 monclient: monmap: 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:02.887 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:02.888 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:02.888 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 Processor -- start 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 1 -- start start 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 10 monclient: init 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.878+0000 7f155d52b700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-b con 0x7f1558096fb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 0x7f155813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-a con 0x7f1558097ac0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15580a2740 0x7f155813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.892 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): picked mon.noname-c con 0x7f15580a2740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155811ff30 con 0x7f1558097ac0 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155811f970 con 0x7f1558096fb0 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155811d400 con 0x7f15580a2740 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient(hunting): authenticate will time out at 1304.976210s 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15567fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 msgr2=0x7f155813a5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15567fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 0x7f155813a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44082/0 (socket says 172.21.15.150:44082) 2024-09-18T10:17:02.893 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 -- 172.21.15.150:0/3940719526 learned_addr learned my addr 172.21.15.150:0/3940719526 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): get_auth_request con 0x7f1558096fb0 auth_method 0 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15577fe700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15580a2740 0x7f155813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15577fe700 10 monclient(hunting): get_auth_request con 0x7f15580a2740 auth_method 0 2024-09-18T10:17:02.894 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15577fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient(hunting): handle_auth_done global_id 84283 payload 824 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 -- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15580a2740 msgr2=0x7f155813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15580a2740 0x7f155813e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 -- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 msgr2=0x7f155813a5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 0x7f155813a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 monclient: found mon.noname-b 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15480094f0 con 0x7f1558096fb0 2024-09-18T10:17:02.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 -- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 msgr2=0x7f155813af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f1556ffd700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548020070 con 0x7f1558096fb0 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: dump: 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:02.896 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: _finish_auth 0 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 5 monclient: authenticate success, global_id 84283 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548006b60 con 0x7f1558096fb0 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient: _renew_subs 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155811f690 con 0x7f1558096fb0 2024-09-18T10:17:02.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154801b7f0 con 0x7f1558096fb0 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f153effd700 10 monclient: dump: 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:02.898 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f15557fa700 4 set_mon_vals no callback set 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient: _renew_subs 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.882+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155813ab30 con 0x7f1558096fb0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f155d52b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f155d52b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f155d52b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1548024070 con 0x7f1558096fb0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155813ad20 con 0x7f1558096fb0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f153effd700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 0x7f1544037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(222..222 src has 176..222) v4 ==== 14846+0+0 (secure 0 0 0) 0x7f154804df50 con 0x7f1558096fb0 2024-09-18T10:17:02.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f15567fc700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 0x7f1544037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:02.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.886+0000 7f15567fc700 10 monclient: get_auth_request con 0x7f1544035a70 auth_method 0 2024-09-18T10:17:02.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.890+0000 7f15567fc700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 0x7f1544037f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f154c006fd0 tx=0x7f154c006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:02.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.894+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1548025560 con 0x7f1558096fb0 2024-09-18T10:17:02.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.894+0000 7f153effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:02.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:02.894+0000 7f153effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:03.067 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.062+0000 7f155d52b700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.062+0000 7f155d52b700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.062+0000 7f155d52b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.068 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.062+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f1558063ed0 con 0x7f1558096fb0 2024-09-18T10:17:03.283 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:17:03.283 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:17:03.283 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:17:03.284 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:17:03.286 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:17:03.287 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:17:03.287 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: get_monmap_and_config 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: build_initial_monmap 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: monmap: 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:03.401823+0000 2024-09-18T10:17:03.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:03.401823+0000 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 Processor -- start 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 -- start start 2024-09-18T10:17:03.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: init 2024-09-18T10:17:03.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:03.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:17:03.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-a con 0x7f1ac009d800 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1ac009b600 0x7f1ac012f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-c con 0x7f1ac009b600 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ac012f8a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac009dbd0 con 0x7f1ac012f8a0 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac009a1f0 con 0x7f1ac009d800 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac009a3e0 con 0x7f1ac009b600 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac754c700 10 monclient(hunting): authenticate will time out at 1312.327620s 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58390/0 (socket says 172.21.15.142:58390) 2024-09-18T10:17:03.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 -- 172.21.15.142:0/1344389886 learned_addr learned my addr 172.21.15.142:0/1344389886 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 10 monclient(hunting): get_auth_request con 0x7f1ac009d800 auth_method 0 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 -- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 msgr2=0x7f1ac012ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 -- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 msgr2=0x7f1ac012ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 --2- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac52e8700 1 --2- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.399+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/1344389886 >> v2:172.21.15.150:3300/0 conn(0x7f1ac009b600 0x7f1ac012f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): get_auth_request con 0x7f1ac009b600 auth_method 0 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_done global_id 84286 payload 274 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 msgr2=0x7f1ac012ee20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/1344389886 >> v1:172.21.15.142:6789/0 conn(0x7f1ac012f8a0 legacy=0x7f1ac012b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient: _finish_hunting 0 2024-09-18T10:17:03.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 monclient: found mon.noname-c 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/1344389886 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac009a6c0 con 0x7f1ac009b600 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/1344389886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 msgr2=0x7f1ac012f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/1344389886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac012f360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0068ee0 tx=0x7f1abc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bdc3184bafec445 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 1 -- 172.21.15.142:0/1344389886 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00f8c0 con 0x7f1ac009b600 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: dump: 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: _finish_auth 0 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac754c700 5 monclient: authenticate success, global_id 84286 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac754c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 1 -- 172.21.15.142:0/1344389886 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc00fa20 con 0x7f1ac009b600 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 1 -- 172.21.15.142:0/1344389886 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc00fdc0 con 0x7f1ac009b600 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac754c700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ac754c700 4 set_mon_vals no callback set 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.403+0000 7f1ab7fff700 10 monclient: dump: 2024-09-18T10:17:03.423 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: shutdown 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:03.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- 172.21.15.142:0/1344389886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 msgr2=0x7f1ac012f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- 172.21.15.142:0/1344389886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac012f360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0068ee0 tx=0x7f1abc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- 172.21.15.142:0/1344389886 shutdown_connections 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- 172.21.15.142:0/1344389886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac012f360 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- 172.21.15.142:0/1344389886 >> v2:172.21.15.142:3300/0 conn(0x7f1ac009d800 0x7f1ac012ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- 172.21.15.142:0/1344389886 >> 172.21.15.142:0/1344389886 conn(0x7f1ac0071680 msgr2=0x7f1ac00a1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- 172.21.15.142:0/1344389886 shutdown_connections 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- 172.21.15.142:0/1344389886 wait complete. 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: build_initial_monmap 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: monmap: 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:03.425 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 Processor -- start 2024-09-18T10:17:03.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- start start 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: init 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-b con 0x7f1ac009b600 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-c con 0x7f1ac009d800 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): picked mon.noname-a con 0x7f1ac012f8a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0127ee0 con 0x7f1ac012f8a0 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0099dc0 con 0x7f1ac009b600 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0129b40 con 0x7f1ac009d800 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac754c700 10 monclient(hunting): authenticate will time out at 1312.334084s 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59746/0 (socket says 172.21.15.142:59746) 2024-09-18T10:17:03.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/2456516865 learned_addr learned my addr 172.21.15.142:0/2456516865 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): get_auth_request con 0x7f1ac009d800 auth_method 0 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 10 monclient(hunting): get_auth_request con 0x7f1ac012f8a0 auth_method 0 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:03.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): get_auth_request con 0x7f1ac009b600 auth_method 0 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 msgr2=0x7f1ac0127100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 msgr2=0x7f1ac0127100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac5ae9700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac52e8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:03.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient(hunting): handle_auth_done global_id 74441 payload 824 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 msgr2=0x7f1ac006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac006f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 msgr2=0x7f1ac0127100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient: _finish_hunting 0 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 monclient: found mon.noname-c 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.407+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/2456516865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc009420 con 0x7f1ac009d800 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac4ae7700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 msgr2=0x7f1ac0126bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac4ae7700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b0e0 tx=0x7f1abc018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc01d070 con 0x7f1ac009d800 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: dump: 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: _finish_auth 0 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 5 monclient: authenticate success, global_id 74441 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _renew_subs 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc010e20 con 0x7f1ac009d800 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0128640 con 0x7f1ac009d800 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc006e50 con 0x7f1ac009d800 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:03.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 10 monclient: dump: 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab77fe700 4 set_mon_vals no callback set 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _renew_subs 2024-09-18T10:17:03.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac006fb60 con 0x7f1ac009d800 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00696a0 con 0x7f1ac009d800 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1abc018b10 con 0x7f1ac009d800 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 0x7f1aa4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(222..222 src has 176..222) v4 ==== 14846+0+0 (secure 0 0 0) 0x7f1abc014070 con 0x7f1ac009d800 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac52e8700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 0x7f1aa4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.411+0000 7f1ac52e8700 10 monclient: get_auth_request con 0x7f1aa4035ac0 auth_method 0 2024-09-18T10:17:03.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.415+0000 7f1ac52e8700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 0x7f1aa4037f70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.423+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1abc00b3d0 con 0x7f1ac009d800 2024-09-18T10:17:03.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.423+0000 7f1ab57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.423+0000 7f1ab57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:03.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ac754c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:03.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ac754c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:03.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ac754c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:03.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1ac009d140 con 0x7f1ac009d800 2024-09-18T10:17:03.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ab57fa700 1 -- 172.21.15.142:0/2456516865 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f1abc027020 con 0x7f1ac009d800 2024-09-18T10:17:03.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ab57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:03.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.647+0000 7f1ab57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:03.652 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 msgr2=0x7f1aa4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 0x7f1aa4037f70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0008040 comp rx=0 tx=0).stop 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 10 monclient: shutdown 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 msgr2=0x7f1ac0126bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b0e0 tx=0x7f1abc018000 comp rx=0 tx=0).stop 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 shutdown_connections 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1aa4035ac0 0x7f1aa4037f70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1ac012f8a0 0x7f1ac0127100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1ac009d800 0x7f1ac0126bc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 --2- 172.21.15.142:0/2456516865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ac009b600 0x7f1ac006f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 >> 172.21.15.142:0/2456516865 conn(0x7f1ac0071680 msgr2=0x7f1ac012ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:03.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 shutdown_connections 2024-09-18T10:17:03.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:03.651+0000 7f1ac754c700 1 -- 172.21.15.142:0/2456516865 wait complete. 2024-09-18T10:17:03.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.658+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 removed item id -22 name 'h2' from crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f154802b070 con 0x7f1558096fb0 2024-09-18T10:17:03.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.658+0000 7f153effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.658+0000 7f153effd700 10 monclient: _finish_command 2 = system:0 removed item id -22 name 'h2' from crush map 2024-09-18T10:17:03.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.766+0000 7f155d52b700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.766+0000 7f155d52b700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.766+0000 7f155d52b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.772 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.766+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h2"} v 0) v1 -- 0x7f15580008d0 con 0x7f1558096fb0 2024-09-18T10:17:03.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.766+0000 7f153effd700 1 -- 172.21.15.150:0/3940719526 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h2"}]=0 device 'h2' does not appear in the crush map v223) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1548025380 con 0x7f1558096fb0 2024-09-18T10:17:03.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f153effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h2"}] 2024-09-18T10:17:03.773 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f153effd700 10 monclient: _finish_command 3 = system:0 device 'h2' does not appear in the crush map 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 msgr2=0x7f1544037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 0x7f1544037f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f154c006fd0 tx=0x7f154c006e10 comp rx=0 tx=0).stop 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 10 monclient: shutdown 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 msgr2=0x7f155813af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1548006fd0 tx=0x7f1548017000 comp rx=0 tx=0).stop 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 shutdown_connections 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1544035a70 0x7f1544037f20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15580a2740 0x7f155813e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1558097ac0 0x7f155813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.776 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 --2- 172.21.15.150:0/3940719526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1558096fb0 0x7f155813af40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 >> 172.21.15.150:0/3940719526 conn(0x7f1558068220 msgr2=0x7f155809a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:03.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 shutdown_connections 2024-09-18T10:17:03.777 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.770+0000 7f155d52b700 1 -- 172.21.15.150:0/3940719526 wait complete. 2024-09-18T10:17:03.777 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h2' does not appear in the crush map 2024-09-18T10:17:03.786 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:03.786 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h3 2024-09-18T10:17:03.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 10 monclient: get_monmap_and_config 2024-09-18T10:17:03.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 10 monclient: build_initial_monmap 2024-09-18T10:17:03.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 10 monclient: monmap: 2024-09-18T10:17:03.884 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:03.885 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.885 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:03.882710+0000 2024-09-18T10:17:03.885 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:03.882710+0000 2024-09-18T10:17:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:03.886 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:03.887 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:03.887 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:03.887 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 1 Processor -- start 2024-09-18T10:17:03.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 1 -- start start 2024-09-18T10:17:03.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.878+0000 7f26762e8700 10 monclient: init 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f267009ad30 0x7f267009b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-c con 0x7f267009ad30 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2670099180 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26700995e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267009a5b0 con 0x7f2670099180 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267009cdf0 con 0x7f26700995e0 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.896 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267009a080 con 0x7f267009ad30 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 10 monclient(hunting): authenticate will time out at 1305.976629s 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f267009ad30 0x7f267009b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f2674885700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f26700995e0 0x7f267009c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34438/0 (socket says 172.21.15.150:34438) 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f2674885700 1 -- 172.21.15.150:0/1412293960 learned_addr learned my addr 172.21.15.150:0/1412293960 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 --2- 172.21.15.150:0/1412293960 >> v2:172.21.15.150:3300/0 conn(0x7f267009ad30 0x7f267009b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44090/0 (socket says 172.21.15.150:44090) 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): get_auth_request con 0x7f267009ad30 auth_method 0 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:03.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient(hunting): handle_auth_done global_id 84292 payload 274 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 -- 172.21.15.150:0/1412293960 >> v1:172.21.15.150:6789/0 conn(0x7f26700995e0 legacy=0x7f267009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 -- 172.21.15.150:0/1412293960 >> v1:172.21.15.142:6789/0 conn(0x7f2670099180 legacy=0x7f267009bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 monclient: found mon.noname-c 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 -- 172.21.15.150:0/1412293960 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670099ca0 con 0x7f267009ad30 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 -- 172.21.15.150:0/1412293960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 msgr2=0x7f267009b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266ffff700 1 --2- 172.21.15.150:0/1412293960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267009b100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26680098e0 tx=0x7f26680091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c35d0deb171e2a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 1 -- 172.21.15.150:0/1412293960 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266800ba70 con 0x7f267009ad30 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 10 monclient: dump: 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 10 monclient: _finish_auth 0 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f266effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 5 monclient: authenticate success, global_id 84292 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.882+0000 7f26762e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 1 -- 172.21.15.150:0/1412293960 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266800bbd0 con 0x7f267009ad30 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 4 set_mon_vals no callback set 2024-09-18T10:17:03.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 1 -- 172.21.15.150:0/1412293960 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668011420 con 0x7f267009ad30 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266effd700 10 monclient: dump: 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.901 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: shutdown 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- 172.21.15.150:0/1412293960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 msgr2=0x7f267009b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 --2- 172.21.15.150:0/1412293960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267009b100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26680098e0 tx=0x7f26680091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- 172.21.15.150:0/1412293960 shutdown_connections 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 --2- 172.21.15.150:0/1412293960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267009b100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- 172.21.15.150:0/1412293960 >> 172.21.15.150:0/1412293960 conn(0x7f26700a15e0 msgr2=0x7f26700a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- 172.21.15.150:0/1412293960 shutdown_connections 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- 172.21.15.150:0/1412293960 wait complete. 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: build_initial_monmap 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: monmap: 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:03.902 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 Processor -- start 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- start start 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: init 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:03.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2670099180 0x7f2670145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-a con 0x7f2670099180 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26700995e0 0x7f2670138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-c con 0x7f26700995e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): picked mon.noname-b con 0x7f267009ad30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267006f210 con 0x7f2670099180 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267006dd60 con 0x7f267009ad30 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:03.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267006ec50 con 0x7f26700995e0 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f26762e8700 10 monclient(hunting): authenticate will time out at 1305.982688s 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44092/0 (socket says 172.21.15.150:44092) 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 -- 172.21.15.150:0/2509739927 learned_addr learned my addr 172.21.15.150:0/2509739927 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): get_auth_request con 0x7f267009ad30 auth_method 0 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:03.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266f7fe700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26700995e0 0x7f2670138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266ffff700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2670099180 0x7f2670145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266ffff700 10 monclient(hunting): get_auth_request con 0x7f2670099180 auth_method 0 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266f7fe700 10 monclient(hunting): get_auth_request con 0x7f26700995e0 auth_method 0 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f266f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient(hunting): handle_auth_done global_id 84295 payload 824 2024-09-18T10:17:03.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 -- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26700995e0 msgr2=0x7f2670138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26700995e0 0x7f2670138c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 -- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2670099180 msgr2=0x7f2670145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2670099180 0x7f2670145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient: _finish_hunting 0 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 monclient: found mon.noname-b 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.886+0000 7f2674885700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2668009420 con 0x7f267009ad30 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2674885700 1 -- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 msgr2=0x7f267006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2674885700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f266000ccc0 tx=0x7f2660005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266000ce70 con 0x7f267009ad30 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: dump: 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: _finish_auth 0 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266000ad80 con 0x7f267009ad30 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 5 monclient: authenticate success, global_id 84295 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _renew_subs 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267006ef30 con 0x7f267009ad30 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2660007360 con 0x7f267009ad30 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 10 monclient: dump: 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:03.909 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _renew_subs 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f266e7fc700 4 set_mon_vals no callback set 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2670139300 con 0x7f267009ad30 2024-09-18T10:17:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2660007710 con 0x7f267009ad30 2024-09-18T10:17:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:03.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.890+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267009ba20 con 0x7f267009ad30 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.894+0000 7f2667fff700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 0x7f2650037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.894+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(223..223 src has 176..223) v4 ==== 14636+0+0 (secure 0 0 0) 0x7f266004b070 con 0x7f267009ad30 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.894+0000 7f266ffff700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 0x7f2650037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.894+0000 7f266ffff700 10 monclient: get_auth_request con 0x7f2650035ac0 auth_method 0 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.894+0000 7f266ffff700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 0x7f2650037f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f26680030c0 tx=0x7f2668016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.902+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f267009ba20 con 0x7f267009ad30 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.902+0000 7f2667fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:03.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:03.902+0000 7f2667fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:04.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.070+0000 7f26762e8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.070+0000 7f26762e8700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.070+0000 7f26762e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.070+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f267009ba20 con 0x7f267009ad30 2024-09-18T10:17:04.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.666+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 removed item id -25 name 'h3' from crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7f267009ba20 con 0x7f267009ad30 2024-09-18T10:17:04.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.666+0000 7f2667fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.666+0000 7f2667fff700 10 monclient: _finish_command 2 = system:0 removed item id -25 name 'h3' from crush map 2024-09-18T10:17:04.781 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h3"} v 0) v1 -- 0x7f2670139790 con 0x7f267009ad30 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f2667fff700 1 -- 172.21.15.150:0/2509739927 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h3"}]=0 device 'h3' does not appear in the crush map v224) v1 ==== 118+0+0 (secure 0 0 0) 0x7f267009ba20 con 0x7f267009ad30 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f2667fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h3"}] 2024-09-18T10:17:04.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f2667fff700 10 monclient: _finish_command 3 = system:0 device 'h3' does not appear in the crush map 2024-09-18T10:17:04.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 msgr2=0x7f2650037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:04.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 0x7f2650037f70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f26680030c0 tx=0x7f2668016000 comp rx=0 tx=0).stop 2024-09-18T10:17:04.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 10 monclient: shutdown 2024-09-18T10:17:04.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:04.785 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 msgr2=0x7f267006b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.778+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f266000ccc0 tx=0x7f2660005400 comp rx=0 tx=0).stop 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 shutdown_connections 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2650035ac0 0x7f2650037f70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f267009ad30 0x7f267006b620 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26700995e0 0x7f2670138c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 --2- 172.21.15.150:0/2509739927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2670099180 0x7f2670145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 >> 172.21.15.150:0/2509739927 conn(0x7f26700a15e0 msgr2=0x7f26701399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 shutdown_connections 2024-09-18T10:17:04.786 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.782+0000 7f26762e8700 1 -- 172.21.15.150:0/2509739927 wait complete. 2024-09-18T10:17:04.787 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h3' does not appear in the crush map 2024-09-18T10:17:04.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:04.795 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h4 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: get_monmap_and_config 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: build_initial_monmap 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: monmap: 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:04.897100+0000 2024-09-18T10:17:04.898 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:04.897100+0000 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.899 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 1 Processor -- start 2024-09-18T10:17:04.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 1 -- start start 2024-09-18T10:17:04.900 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: init 2024-09-18T10:17:04.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:04.902 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.894+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc096a70 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4fc096e40 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 0x7fd4fc136b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc134740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0a2c00 con 0x7fd4fc096e40 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc098f60 con 0x7fd4fc096a70 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0985a0 con 0x7fd4fc134740 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd5008a3700 10 monclient(hunting): authenticate will time out at 1306.992283s 2024-09-18T10:17:04.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4f1d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd4fc096e40 0x7fd4fc097240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:37218/0 (socket says 172.21.15.150:37218) 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4f1d9b700 1 -- 172.21.15.150:0/3821927314 learned_addr learned my addr 172.21.15.150:0/3821927314 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 10 monclient(hunting): get_auth_request con 0x7fd4fc096a70 auth_method 0 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 0x7fd4fc136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): get_auth_request con 0x7fd4fc134740 auth_method 0 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 1 -- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 msgr2=0x7fd4fc0998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 1 -- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 msgr2=0x7fd4fc0998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 1 --2- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fa59c700 1 --2- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient(hunting): handle_auth_done global_id 74447 payload 274 2024-09-18T10:17:04.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 -- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 msgr2=0x7fd4fc0998e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 -- 172.21.15.150:0/3821927314 >> v1:172.21.15.142:6789/0 conn(0x7fd4fc096e40 legacy=0x7fd4fc097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 monclient: found mon.noname-b 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:04.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 -- 172.21.15.150:0/3821927314 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc099e20 con 0x7fd4fc134740 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 -- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 msgr2=0x7fd4fc136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.898+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 0x7fd4fc136b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009e60 tx=0x7fd4f4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4514b1f18f6e0d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 1 -- 172.21.15.150:0/3821927314 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400e070 con 0x7fd4fc134740 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:04.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: dump: 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 5 monclient: authenticate success, global_id 74447 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 1 -- 172.21.15.150:0/3821927314 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f400be40 con 0x7fd4fc134740 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 1 -- 172.21.15.150:0/3821927314 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4012680 con 0x7fd4fc134740 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 4 set_mon_vals no callback set 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd4f9d9b700 10 monclient: dump: 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:04.911 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 10 monclient: shutdown 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 msgr2=0x7fd4fc136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 --2- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 0x7fd4fc136b20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4009e60 tx=0x7fd4f4009780 comp rx=0 tx=0).stop 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- 172.21.15.150:0/3821927314 shutdown_connections 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 --2- 172.21.15.150:0/3821927314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc134740 0x7fd4fc136b20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 --2- 172.21.15.150:0/3821927314 >> v2:172.21.15.142:3300/0 conn(0x7fd4fc096a70 0x7fd4fc0998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- 172.21.15.150:0/3821927314 >> 172.21.15.150:0/3821927314 conn(0x7fd4fc068430 msgr2=0x7fd4fc0a1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- 172.21.15.150:0/3821927314 shutdown_connections 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- 172.21.15.150:0/3821927314 wait complete. 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 10 monclient: build_initial_monmap 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 10 monclient: monmap: 2024-09-18T10:17:04.913 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 Processor -- start 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 1 -- start start 2024-09-18T10:17:04.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.902+0000 7fd5008a3700 10 monclient: init 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc096a70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 0x7fd4fc11e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-c con 0x7fd4fc096e40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc134740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:04.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc09b880 con 0x7fd4fc134740 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc121cd0 con 0x7fd4fc096a70 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1219f0 con 0x7fd4fc096e40 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd5008a3700 10 monclient(hunting): authenticate will time out at 1307.000815s 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44106/0 (socket says 172.21.15.150:44106) 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 -- 172.21.15.150:0/454688163 learned_addr learned my addr 172.21.15.150:0/454688163 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): get_auth_request con 0x7fd4fc096a70 auth_method 0 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:04.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 0x7fd4fc11e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): get_auth_request con 0x7fd4fc096e40 auth_method 0 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 10 monclient(hunting): get_auth_request con 0x7fd4fc134740 auth_method 0 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:04.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 msgr2=0x7fd4fc11ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 msgr2=0x7fd4fc11ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fad9d700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient(hunting): handle_auth_done global_id 84298 payload 824 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 msgr2=0x7fd4fc11e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 0x7fd4fc11e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 msgr2=0x7fd4fc11ed60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient: _finish_hunting 0 2024-09-18T10:17:04.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 monclient: found mon.noname-b 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4fa59c700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4009a20 con 0x7fd4fc096a70 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.906+0000 7fd4f1d9b700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 0x7fd4fc11e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4fa59c700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 msgr2=0x7fd4fc11e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4fa59c700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00d860 tx=0x7fd4ec008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec012800 con 0x7fd4fc096a70 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: dump: 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:04.923 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:04.924 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 5 monclient: authenticate success, global_id 84298 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec012980 con 0x7fd4fc096a70 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _renew_subs 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec012d40 con 0x7fd4fc096a70 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc121710 con 0x7fd4fc096a70 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f959a700 4 set_mon_vals no callback set 2024-09-18T10:17:04.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 10 monclient: dump: 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _renew_subs 2024-09-18T10:17:04.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc121400 con 0x7fd4fc096a70 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd4ec014640 con 0x7fd4fc096a70 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 0x7fd4dc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(224..224 src has 176..224) v4 ==== 14426+0+0 (secure 0 0 0) 0x7fd4ec04afd0 con 0x7fd4fc096a70 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f1d9b700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 0x7fd4dc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.910+0000 7fd4f1d9b700 10 monclient: get_auth_request con 0x7fd4dc035ac0 auth_method 0 2024-09-18T10:17:04.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.914+0000 7fd4f1d9b700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 0x7fd4dc037f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005220 tx=0x7fd4e4007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:04.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.918+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:04.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.918+0000 7fd4f2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:04.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:04.918+0000 7fd4f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.086+0000 7fd5008a3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.090+0000 7fd5008a3700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.090+0000 7fd5008a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.090+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:05.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.678+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 removed item id -28 name 'h4' from crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:05.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.678+0000 7fd4f2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.678+0000 7fd4f2ffd700 10 monclient: _finish_command 2 = system:0 removed item id -28 name 'h4' from crush map 2024-09-18T10:17:05.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.782+0000 7fd5008a3700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.782+0000 7fd5008a3700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.782+0000 7fd5008a3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.789 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.782+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h4"} v 0) v1 -- 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:05.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd4f2ffd700 1 -- 172.21.15.150:0/454688163 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h4"}]=0 device 'h4' does not appear in the crush map v225) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd4fc121e80 con 0x7fd4fc096a70 2024-09-18T10:17:05.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd4f2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h4"}] 2024-09-18T10:17:05.790 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd4f2ffd700 10 monclient: _finish_command 3 = system:0 device 'h4' does not appear in the crush map 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 msgr2=0x7fd4dc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 0x7fd4dc037f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005220 tx=0x7fd4e4007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 10 monclient: shutdown 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 msgr2=0x7fd4fc11e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00d860 tx=0x7fd4ec008040 comp rx=0 tx=0).stop 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 shutdown_connections 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd4dc035ac0 0x7fd4dc037f70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4fc134740 0x7fd4fc11ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4fc096e40 0x7fd4fc11e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 --2- 172.21.15.150:0/454688163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4fc096a70 0x7fd4fc11e2e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 >> 172.21.15.150:0/454688163 conn(0x7fd4fc068430 msgr2=0x7fd4fc133ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 shutdown_connections 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.786+0000 7fd5008a3700 1 -- 172.21.15.150:0/454688163 wait complete. 2024-09-18T10:17:05.794 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h4' does not appear in the crush map 2024-09-18T10:17:05.802 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:05.803 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h5 2024-09-18T10:17:05.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 10 monclient: get_monmap_and_config 2024-09-18T10:17:05.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 10 monclient: build_initial_monmap 2024-09-18T10:17:05.903 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 10 monclient: monmap: 2024-09-18T10:17:05.903 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:05.903 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:05.902156+0000 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:05.902156+0000 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:05.904 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 1 Processor -- start 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 1 -- start start 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.898+0000 7fbe7cc2a700 10 monclient: init 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-a con 0x7fbe7812e480 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe7806eca0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:05.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe7806f0a0 0x7fbe7809cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-c con 0x7fbe7806f0a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7806aaf0 con 0x7fbe7806eca0 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7812e850 con 0x7fbe7812e480 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7806bc90 con 0x7fbe7806f0a0 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe7cc2a700 10 monclient(hunting): authenticate will time out at 1307.997042s 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe7806f0a0 0x7fbe7809cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe7806f0a0 0x7fbe7809cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44114/0 (socket says 172.21.15.150:44114) 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 -- 172.21.15.150:0/1130446951 learned_addr learned my addr 172.21.15.150:0/1130446951 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): get_auth_request con 0x7fbe7806f0a0 auth_method 0 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 1 --2- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 10 monclient(hunting): get_auth_request con 0x7fbe7812e480 auth_method 0 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:05.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 1 -- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 msgr2=0x7fbe7812eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 1 -- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 msgr2=0x7fbe7812eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 1 --2- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe777fe700 1 --2- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient(hunting): handle_auth_done global_id 84301 payload 274 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 -- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 msgr2=0x7fbe7812eb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 --2- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 -- 172.21.15.150:0/1130446951 >> v1:172.21.15.142:6789/0 conn(0x7fbe7806eca0 legacy=0x7fbe7809c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:05.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:05.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 monclient: found mon.noname-c 2024-09-18T10:17:05.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:05.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.902+0000 7fbe77fff700 1 -- 172.21.15.150:0/1130446951 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7809e250 con 0x7fbe7806f0a0 2024-09-18T10:17:05.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 -- 172.21.15.150:0/1130446951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7806f0a0 msgr2=0x7fbe7809cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 --2- 172.21.15.150:0/1130446951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7806f0a0 0x7fbe7809cbe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe7812f100 tx=0x7fbe70009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25073483641eee8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 1 -- 172.21.15.150:0/1130446951 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7000e070 con 0x7fbe7806f0a0 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: dump: 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:05.921 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 5 monclient: authenticate success, global_id 84301 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 1 -- 172.21.15.150:0/1130446951 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7000bc60 con 0x7fbe7806f0a0 2024-09-18T10:17:05.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 4 set_mon_vals no callback set 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 1 -- 172.21.15.150:0/1130446951 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700092e0 con 0x7fbe7806f0a0 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe76ffd700 10 monclient: dump: 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: shutdown 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/1130446951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7806f0a0 msgr2=0x7fbe7809cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/1130446951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7806f0a0 0x7fbe7809cbe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbe7812f100 tx=0x7fbe70009840 comp rx=0 tx=0).stop 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/1130446951 shutdown_connections 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/1130446951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7806f0a0 0x7fbe7809cbe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/1130446951 >> v2:172.21.15.142:3300/0 conn(0x7fbe7812e480 0x7fbe7812eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/1130446951 >> 172.21.15.150:0/1130446951 conn(0x7fbe780a2390 msgr2=0x7fbe7809e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/1130446951 shutdown_connections 2024-09-18T10:17:05.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/1130446951 wait complete. 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: build_initial_monmap 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: monmap: 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 Processor -- start 2024-09-18T10:17:05.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- start start 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: init 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 0x7fbe78131180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-c con 0x7fbe7806eca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-a con 0x7fbe7806f0a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): picked mon.noname-b con 0x7fbe7812e480 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780a0cb0 con 0x7fbe7806f0a0 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7806ea70 con 0x7fbe7812e480 2024-09-18T10:17:05.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78132fa0 con 0x7fbe7806eca0 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe7cc2a700 10 monclient(hunting): authenticate will time out at 1308.003007s 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44128/0 (socket says 172.21.15.150:44128) 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 -- 172.21.15.150:0/2364628618 learned_addr learned my addr 172.21.15.150:0/2364628618 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): get_auth_request con 0x7fbe7812e480 auth_method 0 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 0x7fbe78131180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 10 monclient(hunting): get_auth_request con 0x7fbe7806eca0 auth_method 0 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 10 monclient(hunting): get_auth_request con 0x7fbe7806f0a0 auth_method 0 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 msgr2=0x7fbe781316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 msgr2=0x7fbe781316c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:05.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe6ffff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient(hunting): handle_auth_done global_id 84304 payload 824 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 msgr2=0x7fbe78131180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 0x7fbe78131180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 msgr2=0x7fbe781316c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 monclient: found mon.noname-b 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe77fff700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009ae0 con 0x7fbe7812e480 2024-09-18T10:17:05.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.906+0000 7fbe777fe700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 0x7fbe78131180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe77fff700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 msgr2=0x7fbe78131c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe77fff700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009fd0 tx=0x7fbe70017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70012be0 con 0x7fbe7812e480 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: dump: 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:05.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 5 monclient: authenticate success, global_id 84304 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70012d40 con 0x7fbe7812e480 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7001d670 con 0x7fbe7812e480 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe6f7fe700 10 monclient: dump: 2024-09-18T10:17:05.931 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe767fc700 4 set_mon_vals no callback set 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _renew_subs 2024-09-18T10:17:05.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe78132a90 con 0x7fbe7812e480 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _renew_subs 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe78135890 con 0x7fbe7812e480 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.910+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7809a1e0 con 0x7fbe7812e480 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbe7001daa0 con 0x7fbe7812e480 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe6f7fe700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 0x7fbe58037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(225..225 src has 176..225) v4 ==== 14216+0+0 (secure 0 0 0) 0x7fbe7004c060 con 0x7fbe7812e480 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe777fe700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 0x7fbe58037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:05.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe777fe700 10 monclient: get_auth_request con 0x7fbe58035ac0 auth_method 0 2024-09-18T10:17:05.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.914+0000 7fbe777fe700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 0x7fbe58037f70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006f30 tx=0x7fbe68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:05.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.922+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbe7001a330 con 0x7fbe7812e480 2024-09-18T10:17:05.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.922+0000 7fbe6f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:05.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:05.922+0000 7fbe6f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:06.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.094+0000 7fbe7cc2a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.094+0000 7fbe7cc2a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.094+0000 7fbe7cc2a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.094+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7fbe7804bbf0 con 0x7fbe7812e480 2024-09-18T10:17:06.667 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:06.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.678+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 removed item id -31 name 'h5' from crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe700269d0 con 0x7fbe7812e480 2024-09-18T10:17:06.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.678+0000 7fbe6f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.678+0000 7fbe6f7fe700 10 monclient: _finish_command 2 = system:0 removed item id -31 name 'h5' from crush map 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 10 monclient: get_monmap_and_config 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 10 monclient: build_initial_monmap 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 10 monclient: monmap: 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:06.774684+0000 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:06.774684+0000 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:06.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:06.777 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 1 Processor -- start 2024-09-18T10:17:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 1 -- start start 2024-09-18T10:17:06.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.771+0000 7f6481f52700 10 monclient: init 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f647c09a7f0 0x7f647c098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-c con 0x7f647c09a7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f647c099220 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-b con 0x7f647c09c4f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c09abc0 con 0x7f647c099220 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c09d950 con 0x7f647c09a7f0 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c09db90 con 0x7f647c09c4f0 2024-09-18T10:17:06.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient(hunting): authenticate will time out at 1315.700288s 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59798/0 (socket says 172.21.15.142:59798) 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 1 -- 172.21.15.142:0/2940771331 learned_addr learned my addr 172.21.15.142:0/2940771331 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 --2- 172.21.15.142:0/2940771331 >> v2:172.21.15.150:3300/0 conn(0x7f647c09a7f0 0x7f647c098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): get_auth_request con 0x7f647c09c4f0 auth_method 0 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): get_auth_request con 0x7f647c09a7f0 auth_method 0 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient(hunting): handle_auth_done global_id 84307 payload 274 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 -- 172.21.15.142:0/2940771331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 msgr2=0x7f647c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 --2- 172.21.15.142:0/2940771331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c13e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 -- 172.21.15.142:0/2940771331 >> v1:172.21.15.150:6789/0 conn(0x7f647c099220 legacy=0x7f647c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient: _finish_hunting 0 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 monclient: found mon.noname-c 2024-09-18T10:17:06.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:06.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 -- 172.21.15.142:0/2940771331 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c099a30 con 0x7f647c09a7f0 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 -- 172.21.15.142:0/2940771331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c09a7f0 msgr2=0x7f647c098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6480f50700 1 --2- 172.21.15.142:0/2940771331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c09a7f0 0x7f647c098ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f64700098e0 tx=0x7f64700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1db8cbe227e9c23 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 1 -- 172.21.15.142:0/2940771331 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000b850 con 0x7f647c09a7f0 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:06.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: dump: 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:06.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 5 monclient: authenticate success, global_id 84307 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 1 -- 172.21.15.142:0/2940771331 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647000b9b0 con 0x7f647c09a7f0 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 1 -- 172.21.15.142:0/2940771331 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000bd50 con 0x7f647c09a7f0 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f6481f52700 4 set_mon_vals no callback set 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.775+0000 7f647b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647b7fe700 10 monclient: dump: 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.790 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: shutdown 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- 172.21.15.142:0/2940771331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c09a7f0 msgr2=0x7f647c098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- 172.21.15.142:0/2940771331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c09a7f0 0x7f647c098ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f64700098e0 tx=0x7f64700091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- 172.21.15.142:0/2940771331 shutdown_connections 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- 172.21.15.142:0/2940771331 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c13e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- 172.21.15.142:0/2940771331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c09a7f0 0x7f647c098ce0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- 172.21.15.142:0/2940771331 >> 172.21.15.142:0/2940771331 conn(0x7f647c0a1fe0 msgr2=0x7f647c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- 172.21.15.142:0/2940771331 shutdown_connections 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- 172.21.15.142:0/2940771331 wait complete. 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: build_initial_monmap 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: monmap: 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:06.792 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 Processor -- start 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- start start 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: init 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c099220 0x7f647c145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-b con 0x7f647c099220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-a con 0x7f647c09a7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c06c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): picked mon.noname-c con 0x7f647c09c4f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:06.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c06f970 con 0x7f647c09a7f0 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c06e340 con 0x7f647c099220 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0975b0 con 0x7f647c09c4f0 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481f52700 10 monclient(hunting): authenticate will time out at 1315.706464s 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c06c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58398/0 (socket says 172.21.15.142:58398) 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 -- 172.21.15.142:0/2184697667 learned_addr learned my addr 172.21.15.142:0/2184697667 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 10 monclient(hunting): get_auth_request con 0x7f647c09a7f0 auth_method 0 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): get_auth_request con 0x7f647c09c4f0 auth_method 0 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c099220 0x7f647c145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 msgr2=0x7f647c1399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): get_auth_request con 0x7f647c099220 auth_method 0 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 msgr2=0x7f647c1399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:06.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f647bfff700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6480f50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient(hunting): handle_auth_done global_id 74459 payload 824 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c099220 msgr2=0x7f647c145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c099220 0x7f647c145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 msgr2=0x7f647c1399e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient: _finish_hunting 0 2024-09-18T10:17:06.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 monclient: found mon.noname-c 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.779+0000 7f6481751700 1 -- 172.21.15.142:0/2184697667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470009420 con 0x7f647c09c4f0 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481751700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 msgr2=0x7f647c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481751700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c06c2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6474006fd0 tx=0x7f6474009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474013890 con 0x7f647c09c4f0 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: dump: 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.797 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64740139f0 con 0x7f647c09c4f0 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 5 monclient: authenticate success, global_id 74459 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 10 monclient: _renew_subs 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c146a80 con 0x7f647c09c4f0 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474013d90 con 0x7f647c09c4f0 2024-09-18T10:17:06.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6478ff9700 10 monclient: dump: 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 10 monclient: _renew_subs 2024-09-18T10:17:06.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f647a7fc700 4 set_mon_vals no callback set 2024-09-18T10:17:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:06.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.783+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c06de30 con 0x7f647c09c4f0 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe7cc2a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe7cc2a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe7cc2a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h5"} v 0) v1 -- 0x7fbe7809bf20 con 0x7fbe7812e480 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe6f7fe700 1 -- 172.21.15.150:0/2364628618 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h5"}]=0 device 'h5' does not appear in the crush map v226) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbe70026bb0 con 0x7fbe7812e480 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe6f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h5"}] 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.790+0000 7fbe6f7fe700 10 monclient: _finish_command 3 = system:0 device 'h5' does not appear in the crush map 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 msgr2=0x7fbe58037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 0x7fbe58037f70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006f30 tx=0x7fbe68008040 comp rx=0 tx=0).stop 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 10 monclient: shutdown 2024-09-18T10:17:06.802 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 msgr2=0x7fbe78131c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009fd0 tx=0x7fbe70017000 comp rx=0 tx=0).stop 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 shutdown_connections 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbe58035ac0 0x7fbe58037f70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe7812e480 0x7fbe78131c00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe7806f0a0 0x7fbe781316c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 --2- 172.21.15.150:0/2364628618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbe7806eca0 0x7fbe78131180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 >> 172.21.15.150:0/2364628618 conn(0x7fbe780a2390 msgr2=0x7fbe7806d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 shutdown_connections 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.794+0000 7fbe7cc2a700 1 -- 172.21.15.150:0/2364628618 wait complete. 2024-09-18T10:17:06.803 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h5' does not appear in the crush map 2024-09-18T10:17:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6481f52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6481f52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6481f52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c072a80 con 0x7f647c09c4f0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f647400c5f0 con 0x7f647c09c4f0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6478ff9700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 0x7f6468037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(226..226 src has 176..226) v4 ==== 14006+0+0 (secure 0 0 0) 0x7f647404a920 con 0x7f647c09c4f0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6480f50700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 0x7f6468037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.787+0000 7f6480f50700 10 monclient: get_auth_request con 0x7f6468035ac0 auth_method 0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.791+0000 7f6480f50700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 0x7f6468037f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f64700030c0 tx=0x7f6470016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.795+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6474023ad0 con 0x7f647c09c4f0 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.795+0000 7f6478ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:06.795+0000 7f6478ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:06.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:06.808 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h6 2024-09-18T10:17:06.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: get_monmap_and_config 2024-09-18T10:17:06.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: build_initial_monmap 2024-09-18T10:17:06.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: monmap: 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:06.908281+0000 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:06.908281+0000 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:06.910 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 Processor -- start 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 -- start start 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: init 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40b409ff90 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-a con 0x7f40b409f830 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40b40a0a70 0x7f40b4139730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-c con 0x7f40b40a0a70 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b4068ca0 con 0x7f40b409ff90 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b40a1400 con 0x7f40b409f830 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b409d840 con 0x7f40b40a0a70 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bb488700 10 monclient(hunting): authenticate will time out at 1309.001554s 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40b40a0a70 0x7f40b4139730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40ba486700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f40b409ff90 0x7f40b40a0360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34444/0 (socket says 172.21.15.150:34444) 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40ba486700 1 -- 172.21.15.150:0/2159840732 learned_addr learned my addr 172.21.15.150:0/2159840732 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): get_auth_request con 0x7f40b40a0a70 auth_method 0 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 1 --2- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 10 monclient(hunting): get_auth_request con 0x7f40b409f830 auth_method 0 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 1 -- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 msgr2=0x7f40b409e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 1 -- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 msgr2=0x7f40b409e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 1 --2- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40b9c85700 1 --2- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient(hunting): handle_auth_done global_id 84313 payload 274 2024-09-18T10:17:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 -- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 msgr2=0x7f40b409e220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 --2- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 -- 172.21.15.150:0/2159840732 >> v1:172.21.15.150:6789/0 conn(0x7f40b409ff90 legacy=0x7f40b40a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient: _finish_hunting 0 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 monclient: found mon.noname-c 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.906+0000 7f40bac87700 1 -- 172.21.15.150:0/2159840732 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b409d560 con 0x7f40b40a0a70 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bac87700 1 -- 172.21.15.150:0/2159840732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 msgr2=0x7f40b4139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bac87700 1 --2- 172.21.15.150:0/2159840732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4139730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f40b406fcf0 tx=0x7f40ac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbc275080834793 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 1 -- 172.21.15.150:0/2159840732 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac00e070 con 0x7f40b40a0a70 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.920 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: dump: 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: _finish_auth 0 2024-09-18T10:17:06.921 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 1 -- 172.21.15.150:0/2159840732 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac00be40 con 0x7f40b40a0a70 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 5 monclient: authenticate success, global_id 84313 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 1 -- 172.21.15.150:0/2159840732 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac012660 con 0x7f40b40a0a70 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 4 set_mon_vals no callback set 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40b9484700 10 monclient: dump: 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.922 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 10 monclient: shutdown 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- 172.21.15.150:0/2159840732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 msgr2=0x7f40b4139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 --2- 172.21.15.150:0/2159840732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4139730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f40b406fcf0 tx=0x7f40ac009780 comp rx=0 tx=0).stop 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- 172.21.15.150:0/2159840732 shutdown_connections 2024-09-18T10:17:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 --2- 172.21.15.150:0/2159840732 >> v2:172.21.15.142:3300/0 conn(0x7f40b409f830 0x7f40b409e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 --2- 172.21.15.150:0/2159840732 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4139730 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- 172.21.15.150:0/2159840732 >> 172.21.15.150:0/2159840732 conn(0x7f40b40a55a0 msgr2=0x7f40b40a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- 172.21.15.150:0/2159840732 shutdown_connections 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- 172.21.15.150:0/2159840732 wait complete. 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 10 monclient: build_initial_monmap 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 10 monclient: monmap: 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 Processor -- start 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 1 -- start start 2024-09-18T10:17:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.910+0000 7f40bb488700 10 monclient: init 2024-09-18T10:17:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40b409f830 0x7f40b4141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-c con 0x7f40b409f830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-a con 0x7f40b409ff90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): picked mon.noname-b con 0x7f40b40a0a70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40a11d0 con 0x7f40b409ff90 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b406f520 con 0x7f40b40a0a70 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40942c0 con 0x7f40b409f830 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bb488700 10 monclient(hunting): authenticate will time out at 1309.008689s 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44146/0 (socket says 172.21.15.150:44146) 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40b409f830 0x7f40b4141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 -- 172.21.15.150:0/4039755767 learned_addr learned my addr 172.21.15.150:0/4039755767 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 10 monclient(hunting): get_auth_request con 0x7f40b409ff90 auth_method 0 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): get_auth_request con 0x7f40b40a0a70 auth_method 0 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): get_auth_request con 0x7f40b409f830 auth_method 0 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 msgr2=0x7f40b4141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 msgr2=0x7f40b4141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40b9c85700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40ba486700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient(hunting): handle_auth_done global_id 84316 payload 824 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40b409f830 msgr2=0x7f40b4141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40b409f830 0x7f40b4141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 msgr2=0x7f40b4141f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient: _finish_hunting 0 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 monclient: found mon.noname-b 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.914+0000 7f40bac87700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac009a20 con 0x7f40b40a0a70 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bac87700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 msgr2=0x7f40b4091090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:06.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bac87700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40b409d980 tx=0x7f40ac018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac026430 con 0x7f40b40a0a70 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: dump: 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.935 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 5 monclient: authenticate success, global_id 84316 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _renew_subs 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b40945d0 con 0x7f40b40a0a70 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _renew_subs 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b4092aa0 con 0x7f40b40a0a70 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac021070 con 0x7f40b40a0a70 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac026800 con 0x7f40b40a0a70 2024-09-18T10:17:06.936 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 10 monclient: dump: 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:06.937 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40b8c83700 4 set_mon_vals no callback set 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b404bbf0 con 0x7f40b40a0a70 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f40ac023030 con 0x7f40b40a0a70 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 0x7f4098037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(226..226 src has 176..226) v4 ==== 14006+0+0 (secure 0 0 0) 0x7f40ac04cb80 con 0x7f40b40a0a70 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40ba486700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 0x7f4098037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.918+0000 7f40ba486700 10 monclient: get_auth_request con 0x7f4098035ac0 auth_method 0 2024-09-18T10:17:06.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.922+0000 7f40ba486700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 0x7f4098037f70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007880 tx=0x7f40a4007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.930+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f40ac01d5e0 con 0x7f40b40a0a70 2024-09-18T10:17:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.930+0000 7f40aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:06.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:06.930+0000 7f40aa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:07.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.019+0000 7f6481f52700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:07.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.019+0000 7f6481f52700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:07.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.019+0000 7f6481f52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:07.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.019+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f647c09d730 con 0x7f647c09c4f0 2024-09-18T10:17:07.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6478ff9700 1 -- 172.21.15.142:0/2184697667 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f64740232f0 con 0x7f647c09c4f0 2024-09-18T10:17:07.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6478ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:07.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6478ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:07.026 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:07.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 msgr2=0x7f6468037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:07.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 0x7f6468037f70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f64700030c0 tx=0x7f6470016000 comp rx=0 tx=0).stop 2024-09-18T10:17:07.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 10 monclient: shutdown 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 msgr2=0x7f647c06c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c06c2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6474006fd0 tx=0x7f6474009250 comp rx=0 tx=0).stop 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 shutdown_connections 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6468035ac0 0x7f6468037f70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f647c09c4f0 0x7f647c06c2a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f647c09a7f0 0x7f647c1399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 --2- 172.21.15.142:0/2184697667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f647c099220 0x7f647c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 >> 172.21.15.142:0/2184697667 conn(0x7f647c0a1fe0 msgr2=0x7f647c09f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 shutdown_connections 2024-09-18T10:17:07.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:07.023+0000 7f6481f52700 1 -- 172.21.15.142:0/2184697667 wait complete. 2024-09-18T10:17:07.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.094+0000 7f40bb488700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.094+0000 7f40bb488700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.094+0000 7f40bb488700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:07.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.094+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f40b4063ed0 con 0x7f40b40a0a70 2024-09-18T10:17:07.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.878+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 removed item id -34 name 'h6' from crush map v227) v1 ==== 118+0+0 (secure 0 0 0) 0x7f40ac02e390 con 0x7f40b40a0a70 2024-09-18T10:17:07.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.878+0000 7f40aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.884 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.878+0000 7f40aa7fc700 10 monclient: _finish_command 2 = system:0 removed item id -34 name 'h6' from crush map 2024-09-18T10:17:07.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40bb488700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40bb488700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40bb488700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:07.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h6"} v 0) v1 -- 0x7f40b4063ed0 con 0x7f40b40a0a70 2024-09-18T10:17:07.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40aa7fc700 1 -- 172.21.15.150:0/4039755767 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h6"}]=0 device 'h6' does not appear in the crush map v227) v1 ==== 118+0+0 (secure 0 0 0) 0x7f40ac02e570 con 0x7f40b40a0a70 2024-09-18T10:17:07.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40aa7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h6"}] 2024-09-18T10:17:07.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.986+0000 7f40aa7fc700 10 monclient: _finish_command 3 = system:0 device 'h6' does not appear in the crush map 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 msgr2=0x7f4098037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 0x7f4098037f70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007880 tx=0x7f40a4007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 10 monclient: shutdown 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 msgr2=0x7f40b4091090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40b409d980 tx=0x7f40ac018000 comp rx=0 tx=0).stop 2024-09-18T10:17:07.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 shutdown_connections 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4098035ac0 0x7f4098037f70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40b40a0a70 0x7f40b4091090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f40b409ff90 0x7f40b4141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 --2- 172.21.15.150:0/4039755767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f40b409f830 0x7f40b4141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 >> 172.21.15.150:0/4039755767 conn(0x7f40b40a55a0 msgr2=0x7f40b40a4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 shutdown_connections 2024-09-18T10:17:07.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:07.990+0000 7f40bb488700 1 -- 172.21.15.150:0/4039755767 wait complete. 2024-09-18T10:17:07.997 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h6' does not appear in the crush map 2024-09-18T10:17:08.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:08.006 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h7 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 10 monclient: get_monmap_and_config 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 10 monclient: build_initial_monmap 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 10 monclient: monmap: 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:08.106014+0000 2024-09-18T10:17:08.107 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:08.106014+0000 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:08.108 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 1 Processor -- start 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 1 -- start start 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.102+0000 7f2f52c9b700 10 monclient: init 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c09e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:08.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c0a07d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f4c136a10 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f4c0a1110 con 0x7f2f4c136a10 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4c09ee30 con 0x7f2f4c09e6d0 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c09e220 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f52c9b700 10 monclient(hunting): authenticate will time out at 1310.200207s 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:58480/0 (socket says 172.21.15.150:58480) 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f51238700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2f4c136a10 0x7f2f4c138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:37234/0 (socket says 172.21.15.150:37234) 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 -- 172.21.15.150:0/3296474389 learned_addr learned my addr 172.21.15.150:0/3296474389 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:44338/0 (socket says 172.21.15.150:44338) 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): get_auth_request con 0x7f2f4c0a07d0 auth_method 0 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 10 monclient(hunting): get_auth_request con 0x7f2f4c09e6d0 auth_method 0 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:08.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 -- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 -- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c0a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient(hunting): handle_auth_done global_id 74465 payload 274 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 -- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c0a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 --2- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 -- 172.21.15.150:0/3296474389 >> v1:172.21.15.142:6789/0 conn(0x7f2f4c136a10 legacy=0x7f2f4c138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 monclient: found mon.noname-b 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:08.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:08.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 -- 172.21.15.150:0/3296474389 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c09eaa0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 -- 172.21.15.150:0/3296474389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 msgr2=0x7f2f4c0a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:08.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.106+0000 7f2f43fff700 1 --2- 172.21.15.150:0/3296474389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0099b0 tx=0x7f2f3c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a92f0885f90c476 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 1 -- 172.21.15.150:0/3296474389 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c00ba70 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: dump: 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:08.119 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 5 monclient: authenticate success, global_id 74465 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 1 -- 172.21.15.150:0/3296474389 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c00bbd0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 4 set_mon_vals no callback set 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 1 -- 172.21.15.150:0/3296474389 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c011420 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:08.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f437fe700 10 monclient: dump: 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: shutdown 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:08.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3296474389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 msgr2=0x7f2f4c0a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3296474389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0099b0 tx=0x7f2f3c009250 comp rx=0 tx=0).stop 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3296474389 shutdown_connections 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3296474389 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c0a07d0 0x7f2f4c0a0bd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3296474389 >> v2:172.21.15.142:3300/0 conn(0x7f2f4c09e6d0 0x7f2f4c0a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3296474389 >> 172.21.15.150:0/3296474389 conn(0x7f2f4c0a58a0 msgr2=0x7f2f4c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3296474389 shutdown_connections 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3296474389 wait complete. 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: build_initial_monmap 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: monmap: 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:08.122 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 Processor -- start 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- start start 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: init 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-a con 0x7f2f4c09e6d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:08.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-b con 0x7f2f4c0a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 0x7f2f4c12cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): picked mon.noname-c con 0x7f2f4c136a10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c09bdc0 con 0x7f2f4c09e6d0 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c12fb90 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4c12fe70 con 0x7f2f4c136a10 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f52c9b700 10 monclient(hunting): authenticate will time out at 1310.207195s 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44154/0 (socket says 172.21.15.150:44154) 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 1 -- 172.21.15.150:0/3834544680 learned_addr learned my addr 172.21.15.150:0/3834544680 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): get_auth_request con 0x7f2f4c0a07d0 auth_method 0 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f51238700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 0x7f2f4c12cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 10 monclient(hunting): get_auth_request con 0x7f2f4c09e6d0 auth_method 0 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f51238700 10 monclient(hunting): get_auth_request con 0x7f2f4c136a10 auth_method 0 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f51238700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f51238700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f51238700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c12c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c12c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.110+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 10 monclient(hunting): handle_auth_done global_id 84319 payload 824 2024-09-18T10:17:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 msgr2=0x7f2f4c12cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 0x7f2f4c12cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 msgr2=0x7f2f4c12c4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 monclient: found mon.noname-b 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0094f0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f51238700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 0x7f2f4c12cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 msgr2=0x7f2f4c12ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f43fff700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b3c0 tx=0x7f2f3c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c006ad0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:08.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: dump: 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 5 monclient: authenticate success, global_id 84319 2024-09-18T10:17:08.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 10 monclient: _renew_subs 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3c020070 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4c12ddb0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f42ffd700 4 set_mon_vals no callback set 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 10 monclient: _renew_subs 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4c12d480 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3c01b680 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.114+0000 7f2f40ff9700 10 monclient: dump: 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:08.131 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2f3c022030 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f40ff9700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 0x7f2f38037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(227..227 src has 176..227) v4 ==== 13796+0+0 (secure 0 0 0) 0x7f2f3c04c4d0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f52c9b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f52c9b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:08.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f52c9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 0x7f2f38037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4c12f990 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f50a37700 10 monclient: get_auth_request con 0x7f2f38035ac0 auth_method 0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.118+0000 7f2f50a37700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 0x7f2f38037f70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2f48006f30 tx=0x7f2f48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.122+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2f3c01ba70 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.122+0000 7f2f40ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:08.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.122+0000 7f2f40ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:08.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.294+0000 7f2f52c9b700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:08.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.294+0000 7f2f52c9b700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:08.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.294+0000 7f2f52c9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:08.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.294+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f2f4c12f990 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.890+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 removed item id -37 name 'h7' from crush map v228) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2f3c0243d0 con 0x7f2f4c0a07d0 2024-09-18T10:17:08.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.890+0000 7f2f40ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:08.895 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:08.890+0000 7f2f40ff9700 10 monclient: _finish_command 2 = system:0 removed item id -37 name 'h7' from crush map 2024-09-18T10:17:09.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f52c9b700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:09.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f52c9b700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:09.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f52c9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.006 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h7"} v 0) v1 -- 0x7f2f4c12fcd0 con 0x7f2f4c0a07d0 2024-09-18T10:17:09.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f40ff9700 1 -- 172.21.15.150:0/3834544680 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h7"}]=0 device 'h7' does not appear in the crush map v228) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2f4c12fcd0 con 0x7f2f4c0a07d0 2024-09-18T10:17:09.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f40ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h7"}] 2024-09-18T10:17:09.007 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f40ff9700 10 monclient: _finish_command 3 = system:0 device 'h7' does not appear in the crush map 2024-09-18T10:17:09.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.002+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 msgr2=0x7f2f38037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:09.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 0x7f2f38037f70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2f48006f30 tx=0x7f2f48008040 comp rx=0 tx=0).stop 2024-09-18T10:17:09.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 10 monclient: shutdown 2024-09-18T10:17:09.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:09.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 msgr2=0x7f2f4c12ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b3c0 tx=0x7f2f3c016000 comp rx=0 tx=0).stop 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 shutdown_connections 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f38035ac0 0x7f2f38037f70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f4c136a10 0x7f2f4c12cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f4c0a07d0 0x7f2f4c12ca00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 --2- 172.21.15.150:0/3834544680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f4c09e6d0 0x7f2f4c12c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 >> 172.21.15.150:0/3834544680 conn(0x7f2f4c0a58a0 msgr2=0x7f2f4c09b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 shutdown_connections 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.006+0000 7f2f52c9b700 1 -- 172.21.15.150:0/3834544680 wait complete. 2024-09-18T10:17:09.010 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h7' does not appear in the crush map 2024-09-18T10:17:09.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:09.019 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h8 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.114+0000 7f6a23fff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.114+0000 7f6a23fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.114+0000 7f6a23fff700 10 monclient: monmap: 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:09.119660+0000 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:09.119660+0000 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.121 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 Processor -- start 2024-09-18T10:17:09.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 -- start start 2024-09-18T10:17:09.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient: init 2024-09-18T10:17:09.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:09.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:17:09.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a24099900 0x7f6a2409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6a24099900 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2409d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6a2409b190 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6a2409c580 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2409a4d0 con 0x7f6a2409c580 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a24099cd0 con 0x7f6a24099900 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2409ecd0 con 0x7f6a2409b190 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a23fff700 10 monclient(hunting): authenticate will time out at 1311.214413s 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a24099900 0x7f6a2409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a24099900 0x7f6a2409ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48964/0 (socket says 172.21.15.150:48964) 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 -- 172.21.15.150:0/2962542900 learned_addr learned my addr 172.21.15.150:0/2962542900 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): get_auth_request con 0x7f6a24099900 auth_method 0 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 1 --2- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): get_auth_request con 0x7f6a2409b190 auth_method 0 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 10 monclient(hunting): get_auth_request con 0x7f6a2409c580 auth_method 0 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:09.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a227fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 1 -- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 msgr2=0x7f6a2409d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 1 -- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 msgr2=0x7f6a2409d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 1 --2- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a237fe700 1 --2- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient(hunting): handle_auth_done global_id 84322 payload 274 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 -- 172.21.15.150:0/2962542900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 msgr2=0x7f6a2409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:09.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 -- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 msgr2=0x7f6a2409d890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 monclient: found mon.noname-c 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.118+0000 7f6a22ffd700 1 -- 172.21.15.150:0/2962542900 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2409a7b0 con 0x7f6a24099900 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a22ffd700 1 -- 172.21.15.150:0/2962542900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a24099900 msgr2=0x7f6a2409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:09.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a24099900 0x7f6a2409ce10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6a2406a2c0 tx=0x7f6a14009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a5aa866a262d677 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 1 -- 172.21.15.150:0/2962542900 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14009e70 con 0x7f6a24099900 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: dump: 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.137 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 5 monclient: authenticate success, global_id 84322 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 1 -- 172.21.15.150:0/2962542900 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1400ba70 con 0x7f6a24099900 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:09.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 4 set_mon_vals no callback set 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 1 -- 172.21.15.150:0/2962542900 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1400be30 con 0x7f6a24099900 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a21ffb700 10 monclient: dump: 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:09.139 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 10 monclient: shutdown 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 -- 172.21.15.150:0/2962542900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a24099900 msgr2=0x7f6a2409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a24099900 0x7f6a2409ce10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6a2406a2c0 tx=0x7f6a14009250 comp rx=0 tx=0).stop 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 -- 172.21.15.150:0/2962542900 shutdown_connections 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 --2- 172.21.15.150:0/2962542900 >> v2:172.21.15.142:3300/0 conn(0x7f6a2409c580 0x7f6a2409d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2409d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 --2- 172.21.15.150:0/2962542900 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a24099900 0x7f6a2409ce10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 -- 172.21.15.150:0/2962542900 >> 172.21.15.150:0/2962542900 conn(0x7f6a240a2420 msgr2=0x7f6a240a2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:09.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 -- 172.21.15.150:0/2962542900 shutdown_connections 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 1 -- 172.21.15.150:0/2962542900 wait complete. 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.122+0000 7f6a23fff700 10 monclient: monmap: 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 Processor -- start 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 -- start start 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient: init 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6a24099900 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2408a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6a2409b190 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6a2409c580 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2409dfc0 con 0x7f6a24099900 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240a0eb0 con 0x7f6a2409c580 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2408e4c0 con 0x7f6a2409b190 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a23fff700 10 monclient(hunting): authenticate will time out at 1311.220888s 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48976/0 (socket says 172.21.15.150:48976) 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 -- 172.21.15.150:0/342415325 learned_addr learned my addr 172.21.15.150:0/342415325 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): get_auth_request con 0x7f6a2409c580 auth_method 0 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a227fc700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2408a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 10 monclient(hunting): get_auth_request con 0x7f6a24099900 auth_method 0 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a227fc700 10 monclient(hunting): get_auth_request con 0x7f6a2409b190 auth_method 0 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a227fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a227fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:09.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient(hunting): handle_auth_done global_id 84325 payload 824 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 msgr2=0x7f6a24089f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 msgr2=0x7f6a24089f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 msgr2=0x7f6a2408a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2408a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 msgr2=0x7f6a24089f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 monclient: found mon.noname-b 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140094f0 con 0x7f6a2409c580 2024-09-18T10:17:09.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 msgr2=0x7f6a2408a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.126+0000 7f6a237fe700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007ef0 tx=0x7f6a1c009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c00ee20 con 0x7f6a2409c580 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: dump: 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:09.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: _finish_auth 0 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 5 monclient: authenticate success, global_id 84325 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c014420 con 0x7f6a2409c580 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _renew_subs 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2408d7d0 con 0x7f6a2409c580 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c0147e0 con 0x7f6a2409c580 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:09.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 10 monclient: dump: 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a217fa700 4 set_mon_vals no callback set 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _renew_subs 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2408c2b0 con 0x7f6a2409c580 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2409aa90 con 0x7f6a2409c580 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6a1c014c50 con 0x7f6a2409c580 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 0x7f6a10037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(228..228 src has 176..228) v4 ==== 13586+0+0 (secure 0 0 0) 0x7f6a1c04ae50 con 0x7f6a2409c580 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 0x7f6a10037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:09.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.130+0000 7f6a22ffd700 10 monclient: get_auth_request con 0x7f6a10035ac0 auth_method 0 2024-09-18T10:17:09.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.134+0000 7f6a22ffd700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 0x7f6a10037f70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a2406a2c0 tx=0x7f6a14017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:09.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.138+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6a1c009900 con 0x7f6a2409c580 2024-09-18T10:17:09.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.138+0000 7f6a0affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:09.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.138+0000 7f6a0affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:09.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.298+0000 7f6a23fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:09.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.298+0000 7f6a23fff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:09.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.298+0000 7f6a23fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:09.305 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:09.298+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f6a2404bbf0 con 0x7f6a2409c580 2024-09-18T10:17:10.041 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.146+0000 7f12addea700 10 monclient: get_monmap_and_config 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.146+0000 7f12addea700 10 monclient: build_initial_monmap 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient: monmap: 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:10.153 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:10.151745+0000 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:10.151745+0000 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:10.154 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 Processor -- start 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 -- start start 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient: init 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12a809aae0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): picked mon.noname-a con 0x7f12a8132570 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12a8134e90 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a809aee0 con 0x7f12a809aae0 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a806f6d0 con 0x7f12a8134e90 2024-09-18T10:17:10.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80955a0 con 0x7f12a8132570 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12addea700 10 monclient(hunting): authenticate will time out at 1319.077333s 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12acde8700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f12a809aae0 0x7f12a8096260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:43288/0 (socket says 172.21.15.142:43288) 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 learned_addr learned my addr 172.21.15.142:0/741030214 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f12a8132570 auth_method 0 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:10.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:10.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:10.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.150+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.142:6789/0 conn(0x7f12a809aae0 legacy=0x7f12a8096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:10.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.154+0000 7f12ad5e9700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.150:6789/0 conn(0x7f12a8134e90 legacy=0x7f12a8137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f12a8132570 auth_method 0 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:10.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:10.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:10.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f12a8132570 auth_method 0 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:10.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:10.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f12a8132570 auth_method 0 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:11.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:17:12.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:12.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:12.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:12.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:12.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:12.126+0000 7f6a09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:13.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:13.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:13.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:13.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): tick 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 monclient(hunting): continuing hunt 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 msgr2=0x7f12a8134950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.142:3300/0 conn(0x7f12a8132570 0x7f12a8134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.150:6789/0 conn(0x7f12a8134e90 legacy=0x7f12a8137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.142:6789/0 conn(0x7f12a809aae0 legacy=0x7f12a8096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient: _add_conns ranks=[4,2] 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): picked mon.noname-c con 0x7f1294003830 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): picked mon.noname-b con 0x7f1294006360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a80955a0 con 0x7f1294003830 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1294008f00 con 0x7f1294006360 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a67fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294006360 auth_method 0 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 10 monclient(hunting): get_auth_request con 0x7f1294003830 auth_method 0 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.150+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294006360 auth_method 0 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:13.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 10 monclient(hunting): get_auth_request con 0x7f1294003830 auth_method 0 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:13.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.354+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:13.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294006360 auth_method 0 2024-09-18T10:17:13.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 10 monclient(hunting): get_auth_request con 0x7f1294003830 auth_method 0 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:13.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:13.758+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:14.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:14.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:14.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:14.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294006360 auth_method 0 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 10 monclient(hunting): get_auth_request con 0x7f1294003830 auth_method 0 2024-09-18T10:17:14.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:14.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:14.562+0000 7f12acde8700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:17:15.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:15.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:15.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:15.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:16.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:16.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:16.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): tick 2024-09-18T10:17:16.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 monclient(hunting): continuing hunt 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 msgr2=0x7f12940087d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006360 0x7f12940087d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 msgr2=0x7f1294005c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> v2:172.21.15.150:3300/0 conn(0x7f1294003830 0x7f1294005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12ad5e9700 1 -- 172.21.15.142:0/741030214 reap_dead start 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12a809aae0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 0x7f12940067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): picked mon.noname-b con 0x7f1294003c70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1294006d00 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:16.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a8095b60 con 0x7f12a809aae0 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1294008950 con 0x7f1294006d00 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 1 -- 172.21.15.142:0/741030214 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80955a0 con 0x7f1294003c70 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a67fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 0x7f12940067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294003c70 auth_method 0 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:16.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient(hunting): handle_auth_done global_id 74477 payload 274 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.150:6789/0 conn(0x7f1294006d00 legacy=0x7f1294007170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> v1:172.21.15.142:6789/0 conn(0x7f12a809aae0 legacy=0x7f129400b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 1 monclient: found mon.noname-b 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.150+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294010a80 con 0x7f1294003c70 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a7fff700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 msgr2=0x7f12940067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a7fff700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 0x7f12940067c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1298009590 tx=0x7f129800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfee6cc3fc968eb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 1 -- 172.21.15.142:0/741030214 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298019880 con 0x7f1294003c70 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: dump: 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 1 -- 172.21.15.142:0/741030214 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12980199e0 con 0x7f1294003c70 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 5 monclient: authenticate success, global_id 74477 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 1 -- 172.21.15.142:0/741030214 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298019da0 con 0x7f1294003c70 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 4 set_mon_vals no callback set 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12a77fe700 10 monclient: dump: 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.164 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.165 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 10 monclient: shutdown 2024-09-18T10:17:16.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:16.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 -- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 msgr2=0x7f12940067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 0x7f12940067c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1298009590 tx=0x7f129800b040 comp rx=0 tx=0).stop 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 -- 172.21.15.142:0/741030214 shutdown_connections 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 --2- 172.21.15.142:0/741030214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294003c70 0x7f12940067c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 -- 172.21.15.142:0/741030214 >> 172.21.15.142:0/741030214 conn(0x7f12a80a1fe0 msgr2=0x7f12a8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 -- 172.21.15.142:0/741030214 shutdown_connections 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.154+0000 7f12addea700 1 -- 172.21.15.142:0/741030214 wait complete. 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient: build_initial_monmap 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient: monmap: 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:16.170 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 Processor -- start 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 -- start start 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient: init 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:16.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1294003c70 0x7f12a813f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): picked mon.noname-b con 0x7f1294003c70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): picked mon.noname-c con 0x7f1294006d00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): picked mon.noname-a con 0x7f12a809aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a809fc00 con 0x7f12a809aae0 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8140300 con 0x7f1294003c70 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a806f8d0 con 0x7f1294006d00 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12addea700 10 monclient(hunting): authenticate will time out at 1325.087504s 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12ad5e9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12acde8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1294003c70 0x7f12a813f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40032/0 (socket says 172.21.15.142:40032) 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 1 -- 172.21.15.142:0/2343525194 learned_addr learned my addr 172.21.15.142:0/2343525194 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 10 monclient(hunting): get_auth_request con 0x7f1294006d00 auth_method 0 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.158+0000 7f12a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): get_auth_request con 0x7f1294003c70 auth_method 0 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 10 monclient(hunting): get_auth_request con 0x7f12a809aae0 auth_method 0 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:16.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12acde8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 msgr2=0x7f12a81264e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 msgr2=0x7f12a81264e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12ad5e9700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient(hunting): handle_auth_done global_id 74480 payload 824 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1294003c70 msgr2=0x7f12a813f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1294003c70 0x7f12a813f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 msgr2=0x7f12a81264e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 monclient: found mon.noname-c 2024-09-18T10:17:16.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:16.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 -- 172.21.15.142:0/2343525194 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298009c20 con 0x7f1294006d00 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 msgr2=0x7f12a8130850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a7fff700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f129801bb70 tx=0x7f1298025000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298024860 con 0x7f1294006d00 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: dump: 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.178 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.178 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.178 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.178 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.178 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 5 monclient: authenticate success, global_id 74480 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 10 monclient: _renew_subs 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a8128670 con 0x7f1294006d00 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298024a00 con 0x7f1294006d00 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 10 monclient: _renew_subs 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a6ffd700 4 set_mon_vals no callback set 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8129a30 con 0x7f1294006d00 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298024dc0 con 0x7f1294006d00 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.162+0000 7f12a4ff9700 10 monclient: dump: 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.180 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.181 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12addea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:16.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12addea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:16.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12addea700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8129a30 con 0x7f1294006d00 2024-09-18T10:17:16.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1298003040 con 0x7f1294006d00 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12a4ff9700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 0x7f1294039a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(228..228 src has 176..228) v4 ==== 13586+0+0 (secure 0 0 0) 0x7f129804e070 con 0x7f1294006d00 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12acde8700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 0x7f1294039a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.166+0000 7f12acde8700 10 monclient: get_auth_request con 0x7f12940375a0 auth_method 0 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.170+0000 7f12acde8700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 0x7f1294039a50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f129c007ce0 tx=0x7f129c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.174+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1298017df0 con 0x7f1294006d00 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.174+0000 7f12a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:16.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.174+0000 7f12a4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:16.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12addea700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:16.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12addea700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:16.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12addea700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f12a8129a30 con 0x7f1294006d00 2024-09-18T10:17:16.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12a4ff9700 1 -- 172.21.15.142:0/2343525194 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f1298017820 con 0x7f1294006d00 2024-09-18T10:17:16.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12a4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:16.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.398+0000 7f12a4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:16.405 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 msgr2=0x7f1294039a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 0x7f1294039a50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f129c007ce0 tx=0x7f129c009040 comp rx=0 tx=0).stop 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 10 monclient: shutdown 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 msgr2=0x7f12a8130850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f129801bb70 tx=0x7f1298025000 comp rx=0 tx=0).stop 2024-09-18T10:17:16.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 shutdown_connections 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f12940375a0 0x7f1294039a50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12a809aae0 0x7f12a81264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1294006d00 0x7f12a8130850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 --2- 172.21.15.142:0/2343525194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1294003c70 0x7f12a813f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 >> 172.21.15.142:0/2343525194 conn(0x7f12a80a1fe0 msgr2=0x7f12a8131780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 shutdown_connections 2024-09-18T10:17:16.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.402+0000 7f12addea700 1 -- 172.21.15.142:0/2343525194 wait complete. 2024-09-18T10:17:16.419 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:17:16.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:17:16.419 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:17:16.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: get_monmap_and_config 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: build_initial_monmap 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: monmap: 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:16.526267+0000 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:16.526267+0000 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:16.528 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:16.529 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 Processor -- start 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 -- start start 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: init 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-a con 0x7f106809a6a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f106809aa70 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:16.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f106809b580 0x7f106813d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-c con 0x7f106809b580 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106809dae0 con 0x7f106809aa70 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1068099bc0 con 0x7f106809a6a0 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10680998e0 con 0x7f106809b580 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.522+0000 7f106c88d700 10 monclient(hunting): authenticate will time out at 1325.451857s 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39714/0 (socket says 172.21.15.142:39714) 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 -- 172.21.15.142:0/2632165292 learned_addr learned my addr 172.21.15.142:0/2632165292 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:16.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 10 monclient(hunting): get_auth_request con 0x7f106809a6a0 auth_method 0 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 --2- 172.21.15.142:0/2632165292 >> v2:172.21.15.150:3300/0 conn(0x7f106809b580 0x7f106813d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): get_auth_request con 0x7f106809b580 auth_method 0 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 -- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 msgr2=0x7f106806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 -- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 msgr2=0x7f106806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:16.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient(hunting): handle_auth_done global_id 84331 payload 274 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 -- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 msgr2=0x7f106806f7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 --2- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 -- 172.21.15.142:0/2632165292 >> v1:172.21.15.150:6789/0 conn(0x7f106809aa70 legacy=0x7f106809ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient: _finish_hunting 0 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 monclient: found mon.noname-c 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 -- 172.21.15.142:0/2632165292 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106809bed0 con 0x7f106809b580 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 -- 172.21.15.142:0/2632165292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809b580 msgr2=0x7f106813d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:16.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106759e700 1 --2- 172.21.15.142:0/2632165292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809b580 0x7f106813d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1068099f00 tx=0x7f10580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64460614d15bee86 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 1 -- 172.21.15.142:0/2632165292 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800b8c0 con 0x7f106809b580 2024-09-18T10:17:16.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:16.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: dump: 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: _finish_auth 0 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106c88d700 5 monclient: authenticate success, global_id 84331 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 1 -- 172.21.15.142:0/2632165292 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800ba20 con 0x7f106809b580 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106c88d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106c88d700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 1 -- 172.21.15.142:0/2632165292 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800bdc0 con 0x7f106809b580 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f106c88d700 4 set_mon_vals no callback set 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.526+0000 7f1065d9b700 10 monclient: dump: 2024-09-18T10:17:16.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: shutdown 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- 172.21.15.142:0/2632165292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809b580 msgr2=0x7f106813d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- 172.21.15.142:0/2632165292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809b580 0x7f106813d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1068099f00 tx=0x7f10580091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- 172.21.15.142:0/2632165292 shutdown_connections 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- 172.21.15.142:0/2632165292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809b580 0x7f106813d6f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- 172.21.15.142:0/2632165292 >> v2:172.21.15.142:3300/0 conn(0x7f106809a6a0 0x7f106806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- 172.21.15.142:0/2632165292 >> 172.21.15.142:0/2632165292 conn(0x7f1068070ef0 msgr2=0x7f10680712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- 172.21.15.142:0/2632165292 shutdown_connections 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- 172.21.15.142:0/2632165292 wait complete. 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: build_initial_monmap 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: monmap: 2024-09-18T10:17:16.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 Processor -- start 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- start start 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: init 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:16.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-a con 0x7f106809a6a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809aa70 0x7f106806bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-b con 0x7f106809aa70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 0x7f106806c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): picked mon.noname-c con 0x7f106809b580 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068094830 con 0x7f106809a6a0 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068097e30 con 0x7f106809aa70 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1068126ae0 con 0x7f106809b580 2024-09-18T10:17:16.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106c88d700 10 monclient(hunting): authenticate will time out at 1325.458706s 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 0x7f106806c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39728/0 (socket says 172.21.15.142:39728) 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 learned_addr learned my addr 172.21.15.142:0/2795304435 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 10 monclient(hunting): get_auth_request con 0x7f106809a6a0 auth_method 0 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): get_auth_request con 0x7f106809b580 auth_method 0 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 msgr2=0x7f106813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 msgr2=0x7f106813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809aa70 0x7f106806bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 10 monclient(hunting): get_auth_request con 0x7f106809aa70 auth_method 0 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:16.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient(hunting): handle_auth_done global_id 74483 payload 824 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809aa70 msgr2=0x7f106806bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809aa70 0x7f106806bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 msgr2=0x7f106813d640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106809a6a0 0x7f106813d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient: _finish_hunting 0 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 monclient: found mon.noname-c 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058009420 con 0x7f106809b580 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.530+0000 7f106659c700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f106809aa70 0x7f106806bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 msgr2=0x7f106806c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106759e700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 0x7f106806c400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f106806fd60 tx=0x7f1058016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058011e50 con 0x7f106809b580 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: dump: 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.555 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: _finish_auth 0 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 5 monclient: authenticate success, global_id 74483 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 10 monclient: _renew_subs 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106806df50 con 0x7f106809b580 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1058020070 con 0x7f106809b580 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105801b6a0 con 0x7f106809b580 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f104effd700 10 monclient: dump: 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:16.556 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 10 monclient: _renew_subs 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106806e440 con 0x7f106809b580 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106559a700 4 set_mon_vals no callback set 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:16.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.534+0000 7f106c88d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680648a0 con 0x7f106806e610 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 10 monclient: get_auth_request con 0x7f106806e610 auth_method 0 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1058022030 con 0x7f106809b580 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:16.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f104effd700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1054035ac0 0x7f1054037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:16.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(228..228 src has 176..228) v4 ==== 13586+0+0 (secure 0 0 0) 0x7f105804c5c0 con 0x7f106809b580 2024-09-18T10:17:16.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f106659c700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1054035ac0 0x7f1054037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.538+0000 7f106659c700 10 monclient: get_auth_request con 0x7f1054035ac0 auth_method 0 2024-09-18T10:17:16.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.542+0000 7f106659c700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1054035ac0 0x7f1054037f70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1060007fd0 tx=0x7f1060009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:16.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 10 monclient: get_auth_request con 0x7f106806e610 auth_method 0 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:16.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:16.742+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:17.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.126+0000 7f6a09ffb700 10 monclient: tick 2024-09-18T10:17:17.131 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.126+0000 7f6a09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 10 monclient: get_auth_request con 0x7f106806e610 auth_method 0 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.146+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:17.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.834+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 removed item id -40 name 'h8' from crush map v229) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a1c0122e0 con 0x7f6a2409c580 2024-09-18T10:17:17.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.834+0000 7f6a0affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:17.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.834+0000 7f6a0affd700 10 monclient: _finish_command 2 = system:0 removed item id -40 name 'h8' from crush map 2024-09-18T10:17:17.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.942+0000 7f6a23fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:17.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.942+0000 7f6a23fff700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:17.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.942+0000 7f6a23fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:17.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.942+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h8"} v 0) v1 -- 0x7f6a24063ed0 con 0x7f6a2409c580 2024-09-18T10:17:17.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a0affd700 1 -- 172.21.15.150:0/342415325 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h8"}]=0 device 'h8' does not appear in the crush map v229) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6a1c02c090 con 0x7f6a2409c580 2024-09-18T10:17:17.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a0affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h8"}] 2024-09-18T10:17:17.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a0affd700 10 monclient: _finish_command 3 = system:0 device 'h8' does not appear in the crush map 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 msgr2=0x7f6a10037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 0x7f6a10037f70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6a2406a2c0 tx=0x7f6a14017000 comp rx=0 tx=0).stop 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 10 monclient: shutdown 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 msgr2=0x7f6a2408a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c007ef0 tx=0x7f6a1c009870 comp rx=0 tx=0).stop 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 shutdown_connections 2024-09-18T10:17:17.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6a10035ac0 0x7f6a10037f70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a2409c580 0x7f6a2408a9f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a2409b190 0x7f6a2408a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 --2- 172.21.15.150:0/342415325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a24099900 0x7f6a24089f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 >> 172.21.15.150:0/342415325 conn(0x7f6a240a2420 msgr2=0x7f6a2409eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 shutdown_connections 2024-09-18T10:17:17.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:17.946+0000 7f6a23fff700 1 -- 172.21.15.150:0/342415325 wait complete. 2024-09-18T10:17:17.954 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h8' does not appear in the crush map 2024-09-18T10:17:17.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient: get_auth_request con 0x7f106806e610 auth_method 0 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 10 monclient(hunting): handle_auth_done global_id 94098 payload 824 2024-09-18T10:17:17.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1068126740 con 0x7f106806e610 2024-09-18T10:17:17.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:17.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.950+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f105c016920 tx=0x7f105c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:17.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c014c90 con 0x7f106806e610 2024-09-18T10:17:17.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:17.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:17.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 10 monclient: dump: 2024-09-18T10:17:17.957 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:17.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f105c00c040 con 0x7f106806e610 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f105c016920 tx=0x7f105c008040 comp rx=0 tx=0).stop 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 msgr2=0x7f106806c940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:17.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.954+0000 7f104effd700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106806e610 0x7f106806c940 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:17.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:17.965 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h9 2024-09-18T10:17:17.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f106c88d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:17.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f106c88d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 0x7f106805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:17.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f106c88d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:17.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068093670 con 0x7f106802ae80 2024-09-18T10:17:17.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 0x7f106805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient: get_auth_request con 0x7f106802ae80 auth_method 0 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 10 monclient(hunting): handle_auth_done global_id 94101 payload 824 2024-09-18T10:17:17.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.966+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1068072a80 con 0x7f106802ae80 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f1066d9d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 msgr2=0x7f106805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f1066d9d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 0x7f106805e890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f105c016920 tx=0x7f105c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c022d70 con 0x7f106802ae80 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 10 monclient: dump: 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:17.976 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f105c014a80 con 0x7f106802ae80 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 msgr2=0x7f106805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 0x7f106805e890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f105c016920 tx=0x7f105c023000 comp rx=0 tx=0).stop 2024-09-18T10:17:17.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 msgr2=0x7f106805e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f104effd700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f106802ae80 0x7f106805e890 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f105c016920 tx=0x7f105c023000 comp rx=0 tx=0).stop 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:17.978 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:17.979 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:17.980 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:17.981 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:17.982 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.983 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.984 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:17.985 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:17.986 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.987 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:17.988 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:17.989 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:17.990 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1054035ac0 msgr2=0x7f1054037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1054035ac0 0x7f1054037f70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1060007fd0 tx=0x7f1060009040 comp rx=0 tx=0).stop 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 10 monclient: shutdown 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 msgr2=0x7f106806c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 --2- 172.21.15.142:0/2795304435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f106809b580 0x7f106806c400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f106806fd60 tx=0x7f1058016000 comp rx=0 tx=0).stop 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 reap_dead start 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106759e700 1 -- 172.21.15.142:0/2795304435 reap_dead start 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 shutdown_connections 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 >> 172.21.15.142:0/2795304435 conn(0x7f1068070ef0 msgr2=0x7f10680907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 shutdown_connections 2024-09-18T10:17:17.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:17.970+0000 7f106c88d700 1 -- 172.21.15.142:0/2795304435 wait complete. 2024-09-18T10:17:17.994 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:17.994 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:17:17.995 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: get_monmap_and_config 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: monmap: 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:18.064974+0000 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:18.064974+0000 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.070 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 Processor -- start 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 -- start start 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: init 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.071 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f974009d970 0x7f97400955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-c con 0x7f974009d970 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9740098ac0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97400995a0 0x7f9740136780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-b con 0x7f97400995a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974009b0a0 con 0x7f9740098ac0 2024-09-18T10:17:18.072 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9740099a00 con 0x7f974009d970 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974009ab50 con 0x7f97400995a0 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f97483a5700 10 monclient(hunting): authenticate will time out at 1320.158310s 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f974009d970 0x7f97400955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9745940700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9740098ac0 0x7f9740098e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41808/0 (socket says 172.21.15.150:41808) 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9745940700 1 -- 172.21.15.150:0/2476163627 learned_addr learned my addr 172.21.15.150:0/2476163627 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:18.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): get_auth_request con 0x7f974009d970 auth_method 0 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97400995a0 0x7f9740136780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): get_auth_request con 0x7f97400995a0 auth_method 0 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient(hunting): handle_auth_done global_id 84337 payload 274 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 -- 172.21.15.150:0/2476163627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97400995a0 msgr2=0x7f9740136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97400995a0 0x7f9740136780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 -- 172.21.15.150:0/2476163627 >> v1:172.21.15.150:6789/0 conn(0x7f9740098ac0 legacy=0x7f9740098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 monclient: found mon.noname-c 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:18.075 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.062+0000 7f9746141700 1 -- 172.21.15.150:0/2476163627 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974009ae30 con 0x7f974009d970 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f9746141700 1 -- 172.21.15.150:0/2476163627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f974009d970 msgr2=0x7f97400955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f9746141700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f974009d970 0x7f97400955b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f97300098e0 tx=0x7f97300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dc5dab69a1ea728 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 1 -- 172.21.15.150:0/2476163627 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973000b850 con 0x7f974009d970 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:18.081 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: dump: 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 5 monclient: authenticate success, global_id 84337 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 1 -- 172.21.15.150:0/2476163627 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973000b9b0 con 0x7f974009d970 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 4 set_mon_vals no callback set 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 1 -- 172.21.15.150:0/2476163627 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973000bd50 con 0x7f974009d970 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f974513f700 10 monclient: dump: 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.083 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 10 monclient: shutdown 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 -- 172.21.15.150:0/2476163627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f974009d970 msgr2=0x7f97400955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f974009d970 0x7f97400955b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f97300098e0 tx=0x7f97300091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 -- 172.21.15.150:0/2476163627 shutdown_connections 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97400995a0 0x7f9740136780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.084 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 --2- 172.21.15.150:0/2476163627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f974009d970 0x7f97400955b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 -- 172.21.15.150:0/2476163627 >> 172.21.15.150:0/2476163627 conn(0x7f97400a15e0 msgr2=0x7f97400a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 -- 172.21.15.150:0/2476163627 shutdown_connections 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 1 -- 172.21.15.150:0/2476163627 wait complete. 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.066+0000 7f97483a5700 10 monclient: monmap: 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:18.085 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 Processor -- start 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 -- start start 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient: init 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9740098ac0 0x7f974009c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-c con 0x7f9740098ac0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-b con 0x7f97400995a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f974009d970 0x7f974008d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): picked mon.noname-a con 0x7f974009d970 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974008e700 con 0x7f974009d970 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740098450 con 0x7f97400995a0 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9740097b50 con 0x7f9740098ac0 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f97483a5700 10 monclient(hunting): authenticate will time out at 1320.165135s 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48992/0 (socket says 172.21.15.150:48992) 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 -- 172.21.15.150:0/3456621170 learned_addr learned my addr 172.21.15.150:0/3456621170 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): get_auth_request con 0x7f97400995a0 auth_method 0 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9740098ac0 0x7f974009c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): get_auth_request con 0x7f9740098ac0 auth_method 0 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f974009d970 0x7f974008d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): get_auth_request con 0x7f974009d970 auth_method 0 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746141700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9746942700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient(hunting): handle_auth_done global_id 84340 payload 824 2024-09-18T10:17:18.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 -- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9740098ac0 msgr2=0x7f974009c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9740098ac0 0x7f974009c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 -- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f974009d970 msgr2=0x7f974008d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f974009d970 0x7f974008d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 monclient: found mon.noname-b 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.070+0000 7f9745940700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9730009420 con 0x7f97400995a0 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9745940700 1 -- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 msgr2=0x7f9740094630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9745940700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c5e0 tx=0x7f973c004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c005d40 con 0x7f97400995a0 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: dump: 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 5 monclient: authenticate success, global_id 84340 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c0070d0 con 0x7f97400995a0 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _renew_subs 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974008f830 con 0x7f97400995a0 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c007490 con 0x7f97400995a0 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 10 monclient: dump: 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.092 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9737fff700 4 set_mon_vals no callback set 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _renew_subs 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:18.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974009d2b0 con 0x7f97400995a0 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9745940700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c5e0 tx=0x7f973c004a80 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f973c0499f0 osd_map(229..229 src has 176..229) v4 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f973c01b3b0 con 0x7f97400995a0 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:18.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f97367fc700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 0x7f9724037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9746141700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 0x7f9724037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.074+0000 7f9746141700 10 monclient: get_auth_request con 0x7f9724035ac0 auth_method 0 2024-09-18T10:17:18.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.078+0000 7f9746141700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 0x7f9724037f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f97300030c0 tx=0x7f9730016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.086+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(229..229 src has 176..229) v4 ==== 13376+0+0 (secure 0 0 0) 0x7f973c0499f0 con 0x7f97400995a0 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: get_monmap_and_config 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: monmap: 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:18.105632+0000 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:18.105632+0000 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.107 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:18.108 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:18.108 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:18.108 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:18.108 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:18.108 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 Processor -- start 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 -- start start 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: init 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ea409b0b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea412edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-b con 0x7f4ea409b6f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ea41339d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea409b480 con 0x7f4ea409b0b0 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea4098cd0 con 0x7f4ea41339d0 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406f940 con 0x7f4ea409b6f0 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4eaa0f9700 10 monclient(hunting): authenticate will time out at 1327.031483s 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4ea37fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4ea409b0b0 0x7f4ea412e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33538/0 (socket says 172.21.15.142:33538) 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4ea37fe700 1 -- 172.21.15.142:0/299598382 learned_addr learned my addr 172.21.15.142:0/299598382 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea412edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request con 0x7f4ea409b6f0 auth_method 0 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.102+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_done global_id 74492 payload 274 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 -- 172.21.15.142:0/299598382 >> v1:172.21.15.150:6789/0 conn(0x7f4ea41339d0 legacy=0x7f4ea4135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 -- 172.21.15.142:0/299598382 >> v1:172.21.15.142:6789/0 conn(0x7f4ea409b0b0 legacy=0x7f4ea412e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 monclient: found mon.noname-b 2024-09-18T10:17:18.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:18.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 -- 172.21.15.142:0/299598382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4099110 con 0x7f4ea409b6f0 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 -- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 msgr2=0x7f4ea412edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea412edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009910 tx=0x7f4e8c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95f9dd8c36d1b30 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 1 -- 172.21.15.142:0/299598382 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c00ba70 con 0x7f4ea409b6f0 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: dump: 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.115 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4eaa0f9700 5 monclient: authenticate success, global_id 74492 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4eaa0f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 1 -- 172.21.15.142:0/299598382 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8c00bbd0 con 0x7f4ea409b6f0 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4eaa0f9700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:18.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4eaa0f9700 4 set_mon_vals no callback set 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 1 -- 172.21.15.142:0/299598382 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c011420 con 0x7f4ea409b6f0 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.106+0000 7f4ea2ffd700 10 monclient: dump: 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.117 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: shutdown 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 msgr2=0x7f4ea412edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea412edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009910 tx=0x7f4e8c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/299598382 shutdown_connections 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/299598382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea412edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/299598382 >> 172.21.15.142:0/299598382 conn(0x7f4ea40a14e0 msgr2=0x7f4ea40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/299598382 shutdown_connections 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/299598382 wait complete. 2024-09-18T10:17:18.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: monmap: 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 Processor -- start 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- start start 2024-09-18T10:17:18.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: init 2024-09-18T10:17:18.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:18.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-a con 0x7f4ea409b0b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea413fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-c con 0x7f4ea409b6f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ea41339d0 0x7f4ea41403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): picked mon.noname-b con 0x7f4ea41339d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406b680 con 0x7f4ea409b0b0 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406a6d0 con 0x7f4ea41339d0 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea409d0a0 con 0x7f4ea409b6f0 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4eaa0f9700 10 monclient(hunting): authenticate will time out at 1327.038732s 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39768/0 (socket says 172.21.15.142:39768) 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 -- 172.21.15.142:0/343109363 learned_addr learned my addr 172.21.15.142:0/343109363 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request con 0x7f4ea409b0b0 auth_method 0 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea413fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request con 0x7f4ea409b6f0 auth_method 0 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ea41339d0 0x7f4ea41403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request con 0x7f4ea41339d0 auth_method 0 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 msgr2=0x7f4ea413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea409b6f0 0x7f4ea413fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ea41339d0 msgr2=0x7f4ea41403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ea41339d0 0x7f4ea41403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.110+0000 7f4ea37fe700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c009420 con 0x7f4ea409b0b0 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4ea37fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 msgr2=0x7f4ea409c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4ea37fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c5e0 tx=0x7f4e94004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94005d40 con 0x7f4ea409b0b0 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: dump: 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 5 monclient: authenticate success, global_id 94107 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e940070d0 con 0x7f4ea409b0b0 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 monclient: _renew_subs 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea41335d0 con 0x7f4ea409b0b0 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94007490 con 0x7f4ea409b0b0 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9b7fe700 10 monclient: dump: 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.130 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4ea1ffb700 4 set_mon_vals no callback set 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 monclient: _renew_subs 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea406a3f0 con 0x7f4ea409b0b0 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 0x7f4ea41319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4064950 con 0x7f4ea41408e0 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 0x7f4ea41319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient: get_auth_request con 0x7f4ea41408e0 auth_method 0 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.114+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4ea37fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c5e0 tx=0x7f4e94004a80 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f4e9401b3b0 mgrmap(e 5) v1 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4e9401b3b0 con 0x7f4ea409b0b0 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_done global_id 74498 payload 824 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9bfff700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ea4140cb0 con 0x7f4ea41408e0 2024-09-18T10:17:18.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4e84035af0 0x7f4e84037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4ea3fff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4e84035af0 0x7f4e84037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4ea3fff700 10 monclient: get_auth_request con 0x7f4e84035af0 auth_method 0 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9bfff700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 msgr2=0x7f4ea41319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 0x7f4ea41319b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409bae0 tx=0x7f4e8c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c021070 con 0x7f4ea41408e0 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 10 monclient: dump: 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.133 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4e8c009200 con 0x7f4ea41408e0 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 msgr2=0x7f4ea41319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 0x7f4ea41319b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409bae0 tx=0x7f4e8c016000 comp rx=0 tx=0).stop 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 msgr2=0x7f4ea41319b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:18.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.118+0000 7f4e9b7fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea41408e0 0x7f4ea41319b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.122+0000 7f4ea3fff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4e84035af0 0x7f4e84037fa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409ce50 tx=0x7f4e9c002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4eaa0f9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 0x7f4ea41322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4eaa0f9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea406fa90 con 0x7f4ea4131ef0 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 0x7f4ea41322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient: get_auth_request con 0x7f4ea4131ef0 auth_method 0 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 10 monclient(hunting): handle_auth_done global_id 74501 payload 824 2024-09-18T10:17:18.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.130+0000 7f4e9bfff700 1 -- 172.21.15.142:0/343109363 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4ea40696a0 con 0x7f4ea4131ef0 2024-09-18T10:17:18.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9bfff700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 msgr2=0x7f4ea41322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9bfff700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 0x7f4ea41322c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009860 tx=0x7f4e8c01e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c011ea0 con 0x7f4ea4131ef0 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 10 monclient: dump: 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.139 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f4e8c01c450 con 0x7f4ea4131ef0 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 msgr2=0x7f4ea41322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 0x7f4ea41322c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009860 tx=0x7f4e8c01e070 comp rx=0 tx=0).stop 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 msgr2=0x7f4ea41322c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ea4131ef0 0x7f4ea41322c0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009860 tx=0x7f4e8c01e070 comp rx=0 tx=0).stop 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:18.140 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.141 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.142 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:18.143 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.144 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:18.145 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:18.146 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:18.147 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.149 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:18.150 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:18.151 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:18.152 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4e84035af0 msgr2=0x7f4e84037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4e84035af0 0x7f4e84037fa0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409ce50 tx=0x7f4e9c002620 comp rx=0 tx=0).stop 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 10 monclient: shutdown 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 msgr2=0x7f4ea409c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 --2- 172.21.15.142:0/343109363 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ea409b0b0 0x7f4ea409c760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400c5e0 tx=0x7f4e94004a80 comp rx=0 tx=0).stop 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4ea3fff700 1 -- 172.21.15.142:0/343109363 reap_dead start 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4e9b7fe700 1 -- 172.21.15.142:0/343109363 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(229..229 src has 176..229) v4 ==== 13376+0+0 (unknown 0 0 0) 0x7f4e940499e0 con 0x7f4ea409b0b0 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4ea3fff700 1 -- 172.21.15.142:0/343109363 reap_dead start 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 shutdown_connections 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 >> 172.21.15.142:0/343109363 conn(0x7f4ea40a14e0 msgr2=0x7f4ea4094940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 shutdown_connections 2024-09-18T10:17:18.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.134+0000 7f4eaa0f9700 1 -- 172.21.15.142:0/343109363 wait complete. 2024-09-18T10:17:18.155 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:18.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:17:18.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: monmap: 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:18.270550+0000 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:18.270550+0000 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.272 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:18.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 1 Processor -- start 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 1 -- start start 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: init 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdec0959b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efdec097f20 0x7efdec095d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.266+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-a con 0x7efdec097f20 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:18.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-b con 0x7efdec13b580 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdec096b20 con 0x7efdec0959b0 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdec0962c0 con 0x7efdec097f20 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec096840 con 0x7efdec13b580 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient(hunting): authenticate will time out at 1327.196064s 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efdec097f20 0x7efdec095d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efdec097f20 0x7efdec095d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39778/0 (socket says 172.21.15.142:39778) 2024-09-18T10:17:18.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 -- 172.21.15.142:0/1948096605 learned_addr learned my addr 172.21.15.142:0/1948096605 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): get_auth_request con 0x7efdec097f20 auth_method 0 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): get_auth_request con 0x7efdec13b580 auth_method 0 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient(hunting): handle_auth_done global_id 94110 payload 274 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 -- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 msgr2=0x7efdec13d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 -- 172.21.15.142:0/1948096605 >> v1:172.21.15.150:6789/0 conn(0x7efdec0959b0 legacy=0x7efdec097810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 monclient: found mon.noname-a 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 -- 172.21.15.142:0/1948096605 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec09daf0 con 0x7efdec097f20 2024-09-18T10:17:18.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdeb7fe700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:18.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 -- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec097f20 msgr2=0x7efdec095d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efde27fc700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec097f20 0x7efdec095d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efddc0099b0 tx=0x7efddc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79eea60f5df7b44c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 1 -- 172.21.15.142:0/1948096605 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc009ea0 con 0x7efdec097f20 2024-09-18T10:17:18.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: dump: 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 5 monclient: authenticate success, global_id 94110 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 1 -- 172.21.15.142:0/1948096605 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc00ba70 con 0x7efdec097f20 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdebfff700 4 set_mon_vals no callback set 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 1 -- 172.21.15.142:0/1948096605 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc00be10 con 0x7efdec097f20 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.270+0000 7efdea7fc700 10 monclient: dump: 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.289 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: shutdown 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec097f20 msgr2=0x7efdec095d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec097f20 0x7efdec095d80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efddc0099b0 tx=0x7efddc009250 comp rx=0 tx=0).stop 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- 172.21.15.142:0/1948096605 shutdown_connections 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- 172.21.15.142:0/1948096605 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec097f20 0x7efdec095d80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- 172.21.15.142:0/1948096605 >> 172.21.15.142:0/1948096605 conn(0x7efdec071ae0 msgr2=0x7efdec0a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- 172.21.15.142:0/1948096605 shutdown_connections 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- 172.21.15.142:0/1948096605 wait complete. 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: monmap: 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:18.291 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 Processor -- start 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- start start 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: init 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:18.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec0959b0 0x7efdec093fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-a con 0x7efdec0959b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-c con 0x7efdec13b580 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec13ad00 0x7efdec06a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): picked mon.noname-b con 0x7efdec13ad00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec06d750 con 0x7efdec0959b0 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec13d2a0 con 0x7efdec13ad00 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdec06c540 con 0x7efdec13b580 2024-09-18T10:17:18.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdebfff700 10 monclient(hunting): authenticate will time out at 1327.202780s 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec0959b0 0x7efdec093fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40116/0 (socket says 172.21.15.142:40116) 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 -- 172.21.15.142:0/2908645155 learned_addr learned my addr 172.21.15.142:0/2908645155 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): get_auth_request con 0x7efdec13b580 auth_method 0 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): get_auth_request con 0x7efdec0959b0 auth_method 0 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec13ad00 0x7efdec06a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 10 monclient(hunting): get_auth_request con 0x7efdec13ad00 auth_method 0 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient(hunting): handle_auth_done global_id 74507 payload 824 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec13ad00 msgr2=0x7efdec06a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec13ad00 0x7efdec06a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec0959b0 msgr2=0x7efdec093fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdec0959b0 0x7efdec093fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 monclient: found mon.noname-c 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efde27fc700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4009a20 con 0x7efdec13b580 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.274+0000 7efdeb7fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec13ad00 0x7efdec06a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde27fc700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 msgr2=0x7efdec13a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde27fc700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13a7c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efddc006fd0 tx=0x7efddc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc01b470 con 0x7efdec13b580 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: dump: 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efddc01b5d0 con 0x7efdec13b580 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 5 monclient: authenticate success, global_id 74507 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 monclient: _renew_subs 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc0094f0 con 0x7efdec13b580 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc01b990 con 0x7efdec13b580 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 10 monclient: dump: 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde97fa700 4 set_mon_vals no callback set 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 monclient: _renew_subs 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec06d190 con 0x7efdec13b580 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 0x7efdec06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efddc01c450 con 0x7efdec13b580 2024-09-18T10:17:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.278+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec064830 con 0x7efdec097f20 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdcc035ac0 0x7efdcc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(229..229 src has 176..229) v4 ==== 13376+0+0 (secure 0 0 0) 0x7efddc04c2c0 con 0x7efdec13b580 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 0x7efdec06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient: get_auth_request con 0x7efdec097f20 auth_method 0 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeb7fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdcc035ac0 0x7efdcc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeb7fe700 10 monclient: get_auth_request con 0x7efdcc035ac0 auth_method 0 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 10 monclient(hunting): handle_auth_done global_id 84349 payload 824 2024-09-18T10:17:18.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdec04c430 con 0x7efdec097f20 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 msgr2=0x7efdec06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeaffd700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 0x7efdec06a8a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efdd400f7b0 tx=0x7efdd4007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efdeb7fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdcc035ac0 0x7efdcc037f70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efde40099f0 tx=0x7efde400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4014890 con 0x7efdec097f20 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.282+0000 7efde37fe700 10 monclient: dump: 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efdd40149f0 con 0x7efdec097f20 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 msgr2=0x7efdec06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 0x7efdec06a8a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efdd400f7b0 tx=0x7efdd4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 msgr2=0x7efdec06a8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.286+0000 7efde37fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec097f20 0x7efdec06a8a0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efdd400f7b0 tx=0x7efdd4007ea0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdebfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdebfff700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 0x7efdec06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdebfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec0a0860 con 0x7efdec06d320 2024-09-18T10:17:18.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 0x7efdec06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient: get_auth_request con 0x7efdec06d320 auth_method 0 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.294+0000 7efdeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efdeaffd700 10 monclient(hunting): handle_auth_done global_id 84352 payload 824 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efdeaffd700 1 -- 172.21.15.142:0/2908645155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efdec067830 con 0x7efdec06d320 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efdeaffd700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 msgr2=0x7efdec06ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efdeaffd700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 0x7efdec06ade0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efdd4018000 tx=0x7efdd4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd401c740 con 0x7efdec06d320 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 10 monclient: dump: 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7efdd4024070 con 0x7efdec06d320 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 msgr2=0x7efdec06ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 0x7efdec06ade0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efdd4018000 tx=0x7efdd4019000 comp rx=0 tx=0).stop 2024-09-18T10:17:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 msgr2=0x7efdec06ade0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.298+0000 7efde37fe700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdec06d320 0x7efdec06ade0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efdd4018000 tx=0x7efdd4019000 comp rx=0 tx=0).stop 2024-09-18T10:17:18.316 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:18.316 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:17:18.317 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.318 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:18.319 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.320 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:18.321 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:18.322 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:18.323 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:18.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:18.325 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.326 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:18.328 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.331 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.332 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:18.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:18.334 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:18.334 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:18.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdcc035ac0 msgr2=0x7efdcc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efdcc035ac0 0x7efdcc037f70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efde40099f0 tx=0x7efde400b040 comp rx=0 tx=0).stop 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 10 monclient: shutdown 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 msgr2=0x7efdec13a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 --2- 172.21.15.142:0/2908645155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efdec13b580 0x7efdec13a7c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efddc006fd0 tx=0x7efddc016000 comp rx=0 tx=0).stop 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdeb7fe700 1 -- 172.21.15.142:0/2908645155 reap_dead start 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdeb7fe700 1 -- 172.21.15.142:0/2908645155 reap_dead start 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 shutdown_connections 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 >> 172.21.15.142:0/2908645155 conn(0x7efdec071ae0 msgr2=0x7efdec13b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 shutdown_connections 2024-09-18T10:17:18.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.302+0000 7efdebfff700 1 -- 172.21.15.142:0/2908645155 wait complete. 2024-09-18T10:17:18.337 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:18.337 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:17:18.337 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:17:18.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.442+0000 7fb5a4f70700 10 monclient: get_monmap_and_config 2024-09-18T10:17:18.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.442+0000 7fb5a4f70700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.442+0000 7fb5a4f70700 10 monclient: monmap: 2024-09-18T10:17:18.448 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:18.446966+0000 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:18.446966+0000 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:18.449 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 Processor -- start 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 -- start start 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient: init 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 0x7fb5a0099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-b con 0x7fb5a00a2f20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5a009ced0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb5a009d9b0 0x7fb5a0136670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-a con 0x7fb5a009d9b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00a32f0 con 0x7fb5a009ced0 2024-09-18T10:17:18.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a009abd0 con 0x7fb5a009d9b0 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a0099fe0 con 0x7fb5a00a2f20 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb5a4f70700 10 monclient(hunting): authenticate will time out at 1327.373023s 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59effd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb5a009ced0 0x7fb5a009d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33540/0 (socket says 172.21.15.142:33540) 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 0x7fb5a0099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59effd700 1 -- 172.21.15.142:0/663094878 learned_addr learned my addr 172.21.15.142:0/663094878 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:18.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 1 --2- 172.21.15.142:0/663094878 >> v2:172.21.15.142:3300/0 conn(0x7fb5a009d9b0 0x7fb5a0136670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): get_auth_request con 0x7fb5a00a2f20 auth_method 0 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): get_auth_request con 0x7fb5a009d9b0 auth_method 0 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient(hunting): handle_auth_done global_id 74510 payload 274 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 -- 172.21.15.142:0/663094878 >> v2:172.21.15.142:3300/0 conn(0x7fb5a009d9b0 msgr2=0x7fb5a0136670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/663094878 >> v2:172.21.15.142:3300/0 conn(0x7fb5a009d9b0 0x7fb5a0136670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 -- 172.21.15.142:0/663094878 >> v1:172.21.15.142:6789/0 conn(0x7fb5a009ced0 legacy=0x7fb5a009d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 -- 172.21.15.142:0/663094878 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a009a250 con 0x7fb5a00a2f20 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 -- 172.21.15.142:0/663094878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 msgr2=0x7fb5a0099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.446+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/663094878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 0x7fb5a0099aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5900098e0 tx=0x7fb5900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f965a6631ba0e30 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 1 -- 172.21.15.142:0/663094878 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59000b850 con 0x7fb5a00a2f20 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: dump: 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.459 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 5 monclient: authenticate success, global_id 74510 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 1 -- 172.21.15.142:0/663094878 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59000b9b0 con 0x7fb5a00a2f20 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 4 set_mon_vals no callback set 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 1 -- 172.21.15.142:0/663094878 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59000bd50 con 0x7fb5a00a2f20 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59e7fc700 10 monclient: dump: 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.461 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: shutdown 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- 172.21.15.142:0/663094878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 msgr2=0x7fb5a0099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/663094878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 0x7fb5a0099aa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb5900098e0 tx=0x7fb5900091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- 172.21.15.142:0/663094878 shutdown_connections 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/663094878 >> v2:172.21.15.142:3300/0 conn(0x7fb5a009d9b0 0x7fb5a0136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/663094878 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a00a2f20 0x7fb5a0099aa0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- 172.21.15.142:0/663094878 >> 172.21.15.142:0/663094878 conn(0x7fb5a006a2e0 msgr2=0x7fb5a00a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- 172.21.15.142:0/663094878 shutdown_connections 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- 172.21.15.142:0/663094878 wait complete. 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: build_initial_monmap 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: monmap: 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:18.463 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.464 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:18.464 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:18.464 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:18.464 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 Processor -- start 2024-09-18T10:17:18.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- start start 2024-09-18T10:17:18.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: init 2024-09-18T10:17:18.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:18.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 0x7fb5a012c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-b con 0x7fb5a009ced0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-c con 0x7fb5a009d9b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00a2f20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a006d670 con 0x7fb5a00a2f20 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a006f410 con 0x7fb5a009ced0 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a006f130 con 0x7fb5a009d9b0 2024-09-18T10:17:18.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb5a4f70700 10 monclient(hunting): authenticate will time out at 1327.378955s 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39816/0 (socket says 172.21.15.142:39816) 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 -- 172.21.15.142:0/2440869330 learned_addr learned my addr 172.21.15.142:0/2440869330 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40138/0 (socket says 172.21.15.142:40138) 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): get_auth_request con 0x7fb5a00a2f20 auth_method 0 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): get_auth_request con 0x7fb5a009d9b0 auth_method 0 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 0x7fb5a012c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 10 monclient(hunting): get_auth_request con 0x7fb5a009ced0 auth_method 0 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:18.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient(hunting): handle_auth_done global_id 94119 payload 824 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 -- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 msgr2=0x7fb5a013d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 -- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 msgr2=0x7fb5a012c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 0x7fb5a012c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 monclient: found mon.noname-a 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59ffff700 1 -- 172.21.15.142:0/2440869330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb590009420 con 0x7fb5a00a2f20 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59effd700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.450+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 0x7fb5a012c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb59ffff700 1 -- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 msgr2=0x7fb5a013d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb59ffff700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb598009e90 tx=0x7fb59800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59800cd60 con 0x7fb5a00a2f20 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:18.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: dump: 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.475 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59800f040 con 0x7fb5a00a2f20 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 5 monclient: authenticate success, global_id 94119 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 10 monclient: _renew_subs 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a006f6f0 con 0x7fb5a00a2f20 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5980146d0 con 0x7fb5a00a2f20 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 10 monclient: dump: 2024-09-18T10:17:18.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:18.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 10 monclient: _renew_subs 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb59dffb700 4 set_mon_vals no callback set 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a012cd60 con 0x7fb5a00a2f20 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb598014a80 con 0x7fb5a00a2f20 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.454+0000 7fb5a4f70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb5a4f70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb5a4f70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a0072a80 con 0x7fb5a00a2f20 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb5877fe700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 0x7fb58c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(230..230 src has 176..230) v4 ==== 13376+0+0 (secure 0 0 0) 0x7fb59804b360 con 0x7fb5a00a2f20 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 0x7fb58c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb59f7fe700 10 monclient: get_auth_request con 0x7fb58c035ac0 auth_method 0 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.458+0000 7fb59f7fe700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 0x7fb58c037f70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb5900093f0 tx=0x7fb590016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.462+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb598024300 con 0x7fb5a00a2f20 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.462+0000 7fb5877fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.462+0000 7fb5877fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:18.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.686+0000 7fb5a4f70700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:17:18.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.686+0000 7fb5a4f70700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:17:18.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.686+0000 7fb5a4f70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:18.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.686+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb5a0067950 con 0x7fb5a00a2f20 2024-09-18T10:17:18.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.690+0000 7fb5877fe700 1 -- 172.21.15.142:0/2440869330 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb598018030 con 0x7fb5a00a2f20 2024-09-18T10:17:18.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.690+0000 7fb5877fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:17:18.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.690+0000 7fb5877fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:18.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.696 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=8,logm=64} crc {auth=3704777740,config=3140514009,health=2401782148,kv=4261215878,logm=79461308}) 2024-09-18T10:17:18.696 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=3518798854,config=3140514009,health=73422447,kv=1528380980,logm=1883916965}) 2024-09-18T10:17:18.696 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=8,logm=64} crc {auth=3704777740,config=3140514009,health=2401782148,kv=4261215878,logm=79461308}) 2024-09-18T10:17:18.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.690+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=7,logm=63} crc {auth=2348792085,config=3140514009,health=1735394671,kv=3173669432,logm=4259537163}) 2024-09-18T10:17:18.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 msgr2=0x7fb58c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 0x7fb58c037f70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb5900093f0 tx=0x7fb590016000 comp rx=0 tx=0).stop 2024-09-18T10:17:18.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078713631}) 2024-09-18T10:17:18.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4098177491}) 2024-09-18T10:17:18.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 10 monclient: shutdown 2024-09-18T10:17:18.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:18.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 msgr2=0x7fb5a013d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:18.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb598009e90 tx=0x7fb59800b040 comp rx=0 tx=0).stop 2024-09-18T10:17:18.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078713631}) 2024-09-18T10:17:18.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960007664}) 2024-09-18T10:17:18.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 shutdown_connections 2024-09-18T10:17:18.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb58c035ac0 0x7fb58c037f70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a00a2f20 0x7fb5a013d570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5a009d9b0 0x7fb5a013d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 --2- 172.21.15.142:0/2440869330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5a009ced0 0x7fb5a012c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 >> 172.21.15.142:0/2440869330 conn(0x7fb5a006a2e0 msgr2=0x7fb5a0130b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 shutdown_connections 2024-09-18T10:17:18.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:18.694+0000 7fb5a4f70700 1 -- 172.21.15.142:0/2440869330 wait complete. 2024-09-18T10:17:18.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382401328}) 2024-09-18T10:17:18.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114954746}) 2024-09-18T10:17:18.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382401328}) 2024-09-18T10:17:18.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.694+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3338683799}) 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044240980}) 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978977376}) 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044240980}) 2024-09-18T10:17:18.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=689774664}) 2024-09-18T10:17:18.704 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824165343}) 2024-09-18T10:17:18.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210429681}) 2024-09-18T10:17:18.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824165343}) 2024-09-18T10:17:18.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.698+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635362374}) 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460622208}) 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65977}) 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460622208}) 2024-09-18T10:17:18.706 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899553110}) 2024-09-18T10:17:18.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176858524}) 2024-09-18T10:17:18.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522039}) 2024-09-18T10:17:18.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176858524}) 2024-09-18T10:17:18.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.702+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276998821}) 2024-09-18T10:17:18.712 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:17:18.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590555587}) 2024-09-18T10:17:18.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2598027889}) 2024-09-18T10:17:18.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590555587}) 2024-09-18T10:17:18.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098687646}) 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477638129}) 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202866929}) 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477638129}) 2024-09-18T10:17:18.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.710+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459907932}) 2024-09-18T10:17:18.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747543648}) 2024-09-18T10:17:18.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194314080}) 2024-09-18T10:17:18.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747543648}) 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403090662}) 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652572473}) 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730986113}) 2024-09-18T10:17:18.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652572473}) 2024-09-18T10:17:18.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410463054}) 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333114253}) 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153084418}) 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333114253}) 2024-09-18T10:17:18.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.714+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4121525949}) 2024-09-18T10:17:18.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2531225600}) 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584676178}) 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2531225600}) 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3986940344}) 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478188940}) 2024-09-18T10:17:18.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177866125}) 2024-09-18T10:17:18.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478188940}) 2024-09-18T10:17:18.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360345984}) 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896230036}) 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927641145}) 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896230036}) 2024-09-18T10:17:18.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.718+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=519995182}) 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667664290}) 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372400833}) 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667664290}) 2024-09-18T10:17:18.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712852116}) 2024-09-18T10:17:18.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753485960}) 2024-09-18T10:17:18.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145362171}) 2024-09-18T10:17:18.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753485960}) 2024-09-18T10:17:18.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.722+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043321968}) 2024-09-18T10:17:18.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602999506}) 2024-09-18T10:17:18.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552591761}) 2024-09-18T10:17:18.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602999506}) 2024-09-18T10:17:18.731 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299776726}) 2024-09-18T10:17:18.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169938878}) 2024-09-18T10:17:18.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583286078}) 2024-09-18T10:17:18.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169938878}) 2024-09-18T10:17:18.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287284694}) 2024-09-18T10:17:18.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594232243}) 2024-09-18T10:17:18.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1103955518}) 2024-09-18T10:17:18.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594232243}) 2024-09-18T10:17:18.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.726+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539073168}) 2024-09-18T10:17:18.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717484586}) 2024-09-18T10:17:18.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=207440795}) 2024-09-18T10:17:18.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717484586}) 2024-09-18T10:17:18.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661538365}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909884933}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2571238333}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909884933}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.730+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106629144}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305104195}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818037832}) 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305104195}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3189877337}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205931539}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076096110}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205931539}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677611874}) 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2256849692}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294394945}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2256849692}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228070419}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938991571}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3580745279}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938991571}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.734+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609270886}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390379661}) 2024-09-18T10:17:18.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148213534}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390379661}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3965211753}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296637597}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417661252}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296637597}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=567511047}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92265062}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2311083508}) 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92265062}) 2024-09-18T10:17:18.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.738+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372540521}) 2024-09-18T10:17:18.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707935103}) 2024-09-18T10:17:18.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400780298}) 2024-09-18T10:17:18.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1707935103}) 2024-09-18T10:17:18.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.830+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940877472}) 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789863117}) 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210843582}) 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789863117}) 2024-09-18T10:17:18.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.846+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056249074}) 2024-09-18T10:17:18.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.902+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:18.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.902+0000 7f97367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:18.906 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:18.902+0000 7f97367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:18.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806325136}) 2024-09-18T10:17:18.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3950593729}) 2024-09-18T10:17:18.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:18.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806325136}) 2024-09-18T10:17:18.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:18.902+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122300875}) 2024-09-18T10:17:19.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151665293}) 2024-09-18T10:17:19.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1207046425}) 2024-09-18T10:17:19.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151665293}) 2024-09-18T10:17:19.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.030+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480580432}) 2024-09-18T10:17:19.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469315751}) 2024-09-18T10:17:19.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113794213}) 2024-09-18T10:17:19.044 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.044 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469315751}) 2024-09-18T10:17:19.044 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.038+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636949804}) 2024-09-18T10:17:19.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.070+0000 7f97483a5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.070+0000 7f97483a5700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.070+0000 7f97483a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:19.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.070+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:19.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53470533}) 2024-09-18T10:17:19.078 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=637075612}) 2024-09-18T10:17:19.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53470533}) 2024-09-18T10:17:19.079 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.074+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912622880}) 2024-09-18T10:17:19.142 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896256766}) 2024-09-18T10:17:19.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479331690}) 2024-09-18T10:17:19.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3896256766}) 2024-09-18T10:17:19.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=410324992}) 2024-09-18T10:17:19.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130949118}) 2024-09-18T10:17:19.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1455673003}) 2024-09-18T10:17:19.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130949118}) 2024-09-18T10:17:19.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.138+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021409279}) 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975371892}) 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538684276}) 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975371892}) 2024-09-18T10:17:19.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.142+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152671084}) 2024-09-18T10:17:19.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551950854}) 2024-09-18T10:17:19.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2453897014}) 2024-09-18T10:17:19.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551950854}) 2024-09-18T10:17:19.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=861012650}) 2024-09-18T10:17:19.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735371052}) 2024-09-18T10:17:19.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=518760520}) 2024-09-18T10:17:19.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735371052}) 2024-09-18T10:17:19.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.146+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807506584}) 2024-09-18T10:17:19.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280456315}) 2024-09-18T10:17:19.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=561200226}) 2024-09-18T10:17:19.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280456315}) 2024-09-18T10:17:19.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649736588}) 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029639838}) 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1944479624}) 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029639838}) 2024-09-18T10:17:19.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616009894}) 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=1209827092,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=65180215,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=3423936030,osd_snap=454132730,osdmap=2062861164}) 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.150+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=44} crc {logm=2551742368,mdsmap=3585761405,mgr=3895449980,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=65180215,mon_config_key=620546692,monmap=4028308432,osd_metadata=1440571423,osd_pg_creating=3423936030,osd_snap=454132730,osdmap=14856266}) 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.154+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.154+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=1209827092,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=65180215,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=3423936030,osd_snap=454132730,osdmap=2062861164}) 2024-09-18T10:17:19.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.154+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=45} crc {logm=701403072,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgrstat=65180215,mon_config_key=620546692,monmap=4028308432,osd_metadata=1440571423,osd_pg_creating=3423936030,osd_snap=3823785925,osdmap=813622644}) 2024-09-18T10:17:19.322 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.322 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1465787996}) 2024-09-18T10:17:19.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=644173919}) 2024-09-18T10:17:19.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:17:19.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=1465787996}) 2024-09-18T10:17:19.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:17:19.318+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=2638434864}) 2024-09-18T10:17:19.712 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:17:19.713 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: get_monmap_and_config 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: build_initial_monmap 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: monmap: 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:19.817644+0000 2024-09-18T10:17:19.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:19.817644+0000 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:19.820 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 Processor -- start 2024-09-18T10:17:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 -- start start 2024-09-18T10:17:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: init 2024-09-18T10:17:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:19.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9fb409bed0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 0x7f9fb40992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-b con 0x7f9fb409c9b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fb41376a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb4098eb0 con 0x7f9fb409bed0 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb406f380 con 0x7f9fb41376a0 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb409de00 con 0x7f9fb409c9b0 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb9441700 10 monclient(hunting): authenticate will time out at 1328.743144s 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb2ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9fb409bed0 0x7f9fb409c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33550/0 (socket says 172.21.15.142:33550) 2024-09-18T10:17:19.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 0x7f9fb40992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb2ffd700 1 -- 172.21.15.142:0/3578777216 learned_addr learned my addr 172.21.15.142:0/3578777216 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): get_auth_request con 0x7f9fb409c9b0 auth_method 0 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_done global_id 74516 payload 274 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 1 -- 172.21.15.142:0/3578777216 >> v1:172.21.15.150:6789/0 conn(0x7f9fb41376a0 legacy=0x7f9fb4139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 1 -- 172.21.15.142:0/3578777216 >> v1:172.21.15.142:6789/0 conn(0x7f9fb409bed0 legacy=0x7f9fb409c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:19.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 1 monclient: found mon.noname-b 2024-09-18T10:17:19.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:19.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:19.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.814+0000 7f9fb27fc700 1 -- 172.21.15.142:0/3578777216 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4098b60 con 0x7f9fb409c9b0 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb27fc700 1 -- 172.21.15.142:0/3578777216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 msgr2=0x7f9fb40992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb27fc700 1 --2- 172.21.15.142:0/3578777216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 0x7f9fb40992f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009910 tx=0x7f9fa80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b1ad767b212747 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 1 -- 172.21.15.142:0/3578777216 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa800ba70 con 0x7f9fb409c9b0 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: dump: 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:19.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 5 monclient: authenticate success, global_id 74516 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 1 -- 172.21.15.142:0/3578777216 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800bbd0 con 0x7f9fb409c9b0 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 1 -- 172.21.15.142:0/3578777216 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011420 con 0x7f9fb409c9b0 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:19.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb1ffb700 10 monclient: dump: 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 4 set_mon_vals no callback set 2024-09-18T10:17:19.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 10 monclient: shutdown 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 1 -- 172.21.15.142:0/3578777216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 msgr2=0x7f9fb40992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.818+0000 7f9fb9441700 1 --2- 172.21.15.142:0/3578777216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 0x7f9fb40992f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009910 tx=0x7f9fa80091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- 172.21.15.142:0/3578777216 shutdown_connections 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 --2- 172.21.15.142:0/3578777216 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409c9b0 0x7f9fb40992f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- 172.21.15.142:0/3578777216 >> 172.21.15.142:0/3578777216 conn(0x7f9fb40a14e0 msgr2=0x7f9fb40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- 172.21.15.142:0/3578777216 shutdown_connections 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- 172.21.15.142:0/3578777216 wait complete. 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient: build_initial_monmap 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient: monmap: 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:19.830 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 Processor -- start 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- start start 2024-09-18T10:17:19.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient: init 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 0x7f9fb4136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-c con 0x7f9fb409bed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-a con 0x7f9fb409c9b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb41376a0 0x7f9fb408da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): picked mon.noname-b con 0x7f9fb41376a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4091700 con 0x7f9fb409c9b0 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:19.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb409d550 con 0x7f9fb41376a0 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4097cb0 con 0x7f9fb409bed0 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb9441700 10 monclient(hunting): authenticate will time out at 1328.750534s 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 0x7f9fb4136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39822/0 (socket says 172.21.15.142:39822) 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 1 -- 172.21.15.142:0/55499979 learned_addr learned my addr 172.21.15.142:0/55499979 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): get_auth_request con 0x7f9fb409c9b0 auth_method 0 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:19.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): get_auth_request con 0x7f9fb409bed0 auth_method 0 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb41376a0 0x7f9fb408da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): get_auth_request con 0x7f9fb41376a0 auth_method 0 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:19.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient(hunting): handle_auth_done global_id 74519 payload 824 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 -- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb41376a0 msgr2=0x7f9fb408da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb41376a0 0x7f9fb408da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 -- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 msgr2=0x7f9fb4137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 monclient: found mon.noname-c 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:19.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb2ffd700 1 -- 172.21.15.142:0/55499979 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8009420 con 0x7f9fb409bed0 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.822+0000 7f9fb27fc700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb2ffd700 1 -- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 msgr2=0x7f9fb4136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb2ffd700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 0x7f9fb4136b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400c560 tx=0x7f9fa4007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4005d40 con 0x7f9fb409bed0 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: dump: 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.840 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: _finish_auth 0 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa40070d0 con 0x7f9fb409bed0 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 5 monclient: authenticate success, global_id 74519 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 10 monclient: _renew_subs 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb40901c0 con 0x7f9fb409bed0 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:19.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4007490 con 0x7f9fb409bed0 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9f9affd700 10 monclient: dump: 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:19.842 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb17fa700 4 set_mon_vals no callback set 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 10 monclient: _renew_subs 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.826+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb408f5f0 con 0x7f9fb409bed0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9fa401b3b0 con 0x7f9fb409bed0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9f9affd700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 0x7f9fa0037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(230..230 src has 176..230) v4 ==== 13376+0+0 (secure 0 0 0) 0x7f9fa4049940 con 0x7f9fb409bed0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb9441700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb9441700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb9441700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb40696a0 con 0x7f9fb409bed0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb27fc700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 0x7f9fa0037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:19.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb27fc700 10 monclient: get_auth_request con 0x7f9fa0035af0 auth_method 0 2024-09-18T10:17:19.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.830+0000 7f9fb27fc700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 0x7f9fa0037fa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80091e0 tx=0x7f9fa8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:19.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.838+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9fa4017070 con 0x7f9fb409bed0 2024-09-18T10:17:19.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.838+0000 7f9f9affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:19.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:19.838+0000 7f9f9affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:19.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.870+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 removed item id -43 name 'h9' from crush map v231) v1 ==== 118+0+0 (secure 0 0 0) 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:19.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.874+0000 7f97367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.877 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.874+0000 7f97367fc700 10 monclient: _finish_command 2 = system:0 removed item id -43 name 'h9' from crush map 2024-09-18T10:17:19.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97483a5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97483a5700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97483a5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:19.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h9"} v 0) v1 -- 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:19.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97367fc700 1 -- 172.21.15.150:0/3456621170 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h9"}]=0 device 'h9' does not appear in the crush map v231) v1 ==== 118+0+0 (secure 0 0 0) 0x7f974008fb70 con 0x7f97400995a0 2024-09-18T10:17:19.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97367fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h9"}] 2024-09-18T10:17:19.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.978+0000 7f97367fc700 10 monclient: _finish_command 3 = system:0 device 'h9' does not appear in the crush map 2024-09-18T10:17:19.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 msgr2=0x7f9724037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:19.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 0x7f9724037f70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f97300030c0 tx=0x7f9730016000 comp rx=0 tx=0).stop 2024-09-18T10:17:19.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 10 monclient: shutdown 2024-09-18T10:17:19.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 msgr2=0x7f9740094630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c5e0 tx=0x7f973c004a80 comp rx=0 tx=0).stop 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 shutdown_connections 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9724035ac0 0x7f9724037f70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f974009d970 0x7f974008d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97400995a0 0x7f9740094630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 --2- 172.21.15.150:0/3456621170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9740098ac0 0x7f974009c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 >> 172.21.15.150:0/3456621170 conn(0x7f97400a15e0 msgr2=0x7f97400a0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:19.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 shutdown_connections 2024-09-18T10:17:19.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:19.982+0000 7f97483a5700 1 -- 172.21.15.150:0/3456621170 wait complete. 2024-09-18T10:17:19.989 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h9' does not appear in the crush map 2024-09-18T10:17:19.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:19.997 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h10 2024-09-18T10:17:20.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.066+0000 7f9fb9441700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:20.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.066+0000 7f9fb9441700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:20.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.066+0000 7f9fb9441700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:20.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.066+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9fb4067950 con 0x7f9fb409bed0 2024-09-18T10:17:20.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9f9affd700 1 -- 172.21.15.142:0/55499979 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f9fa401a6c0 con 0x7f9fb409bed0 2024-09-18T10:17:20.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9f9affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:20.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9f9affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:20.074 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:20.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 msgr2=0x7f9fa0037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 0x7f9fa0037fa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80091e0 tx=0x7f9fa8016000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 10 monclient: shutdown 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 msgr2=0x7f9fb4136b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 0x7f9fb4136b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400c560 tx=0x7f9fa4007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 shutdown_connections 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9fa0035af0 0x7f9fa0037fa0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb41376a0 0x7f9fb408da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb409c9b0 0x7f9fb4137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 --2- 172.21.15.142:0/55499979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb409bed0 0x7f9fb4136b10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 >> 172.21.15.142:0/55499979 conn(0x7f9fb40a14e0 msgr2=0x7f9fb409fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 shutdown_connections 2024-09-18T10:17:20.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.070+0000 7f9fb9441700 1 -- 172.21.15.142:0/55499979 wait complete. 2024-09-18T10:17:20.087 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:17:20.088 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:17:20.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:17:20.088 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:17:20.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.082+0000 7f46df576700 10 monclient: get_monmap_and_config 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.082+0000 7f46df576700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.082+0000 7f46df576700 10 monclient: monmap: 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:20.087418+0000 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:20.087418+0000 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:20.089 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:20.090 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:20.090 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:20.090 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:20.090 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 Processor -- start 2024-09-18T10:17:20.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 -- start start 2024-09-18T10:17:20.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient: init 2024-09-18T10:17:20.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d80955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): picked mon.noname-b con 0x7f46d809d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f46d8098ac0 0x7f46d8098e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): picked mon.noname-a con 0x7f46d8098ac0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f46d80993d0 0x7f46d8135770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): picked mon.noname-c con 0x7f46d80993d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d809b0a0 con 0x7f46d8098ac0 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d809ab50 con 0x7f46d80993d0 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d809ae30 con 0x7f46d809d970 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46df576700 10 monclient(hunting): authenticate will time out at 1322.181120s 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f46d80993d0 0x7f46d8135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f46d80993d0 0x7f46d8135770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36884/0 (socket says 172.21.15.150:36884) 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 -- 172.21.15.150:0/1904693370 learned_addr learned my addr 172.21.15.150:0/1904693370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): get_auth_request con 0x7f46d80993d0 auth_method 0 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d80955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): get_auth_request con 0x7f46d809d970 auth_method 0 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 1 --2- 172.21.15.150:0/1904693370 >> v2:172.21.15.142:3300/0 conn(0x7f46d8098ac0 0x7f46d8098e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): get_auth_request con 0x7f46d8098ac0 auth_method 0 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dd312700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient(hunting): handle_auth_done global_id 84361 payload 274 2024-09-18T10:17:20.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 -- 172.21.15.150:0/1904693370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 msgr2=0x7f46d80955b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d80955b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 -- 172.21.15.150:0/1904693370 >> v2:172.21.15.142:3300/0 conn(0x7f46d8098ac0 msgr2=0x7f46d8098e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 --2- 172.21.15.150:0/1904693370 >> v2:172.21.15.142:3300/0 conn(0x7f46d8098ac0 0x7f46d8098e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 monclient: found mon.noname-c 2024-09-18T10:17:20.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.086+0000 7f46ddb13700 1 -- 172.21.15.150:0/1904693370 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d809dd40 con 0x7f46d80993d0 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 1 -- 172.21.15.150:0/1904693370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d8135770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d8135770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46d809bab0 tx=0x7f46d0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf94a7e1f2e3ff08 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 1 -- 172.21.15.150:0/1904693370 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d000e070 con 0x7f46d80993d0 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: dump: 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:20.108 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.109 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 5 monclient: authenticate success, global_id 84361 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 1 -- 172.21.15.150:0/1904693370 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d000bb80 con 0x7f46d80993d0 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 4 set_mon_vals no callback set 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 1 -- 172.21.15.150:0/1904693370 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d00092e0 con 0x7f46d80993d0 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:20.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46cbfff700 10 monclient: dump: 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.111 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: shutdown 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- 172.21.15.150:0/1904693370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d8135770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d8135770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f46d809bab0 tx=0x7f46d0009840 comp rx=0 tx=0).stop 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- 172.21.15.150:0/1904693370 shutdown_connections 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d8135770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- 172.21.15.150:0/1904693370 >> v2:172.21.15.142:3300/0 conn(0x7f46d8098ac0 0x7f46d8098e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- 172.21.15.150:0/1904693370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d80955b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- 172.21.15.150:0/1904693370 >> 172.21.15.150:0/1904693370 conn(0x7f46d80a15e0 msgr2=0x7f46d80a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- 172.21.15.150:0/1904693370 shutdown_connections 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- 172.21.15.150:0/1904693370 wait complete. 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: monmap: 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.113 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 Processor -- start 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- start start 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: init 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46d813d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): picked mon.noname-a con 0x7f46d8098ac0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): picked mon.noname-b con 0x7f46d80993d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d806d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): picked mon.noname-c con 0x7f46d809d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8126f70 con 0x7f46d8098ac0 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8097760 con 0x7f46d80993d0 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8098450 con 0x7f46d809d970 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46df576700 10 monclient(hunting): authenticate will time out at 1322.187054s 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36898/0 (socket says 172.21.15.150:36898) 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 learned_addr learned my addr 172.21.15.150:0/2169780960 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): get_auth_request con 0x7f46d80993d0 auth_method 0 2024-09-18T10:17:20.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.117 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46d813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 10 monclient(hunting): get_auth_request con 0x7f46d8098ac0 auth_method 0 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 10 monclient(hunting): get_auth_request con 0x7f46d809d970 auth_method 0 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient(hunting): handle_auth_done global_id 84364 payload 824 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 msgr2=0x7f46d806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d806d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 msgr2=0x7f46d813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46d813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 monclient: found mon.noname-b 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d0009ae0 con 0x7f46d80993d0 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46d806d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.090+0000 7f46dd312700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46d813d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d812de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46cc012d90 tx=0x7f46cc00f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00d2e0 con 0x7f46d80993d0 2024-09-18T10:17:20.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: dump: 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.121 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 5 monclient: authenticate success, global_id 84364 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _renew_subs 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc013040 con 0x7f46d80993d0 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d81285f0 con 0x7f46d80993d0 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc010d60 con 0x7f46d80993d0 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.122 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46cb7fe700 4 set_mon_vals no callback set 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46c97fa700 10 monclient: dump: 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.123 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _renew_subs 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d813d7c0 con 0x7f46d80993d0 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.094+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d8063ed0 con 0x7f46d80993d0 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46cc012d90 tx=0x7f46cc00f040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f46cc04e6f0 osd_map(231..231 src has 176..231) v4 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f46cc005a00 con 0x7f46d80993d0 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46c97fa700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 0x7f46bc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46dd312700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 0x7f46bc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46dd312700 10 monclient: get_auth_request con 0x7f46bc035ac0 auth_method 0 2024-09-18T10:17:20.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.098+0000 7f46dd312700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 0x7f46bc037f70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009d80 tx=0x7f46d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.125 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.106+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f46cc04e6f0 con 0x7f46d80993d0 2024-09-18T10:17:20.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: get_monmap_and_config 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: monmap: 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:20.193148+0000 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:20.193148+0000 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:20.195 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:20.196 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:20.196 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:20.196 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:20.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 Processor -- start 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 -- start start 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: init 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafa809b0f0 0x7fafa809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-c con 0x7fafa809b0f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafa80940a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:20.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafa8094c10 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8098b00 con 0x7fafa8094c10 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8098d20 con 0x7fafa80940a0 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa809d5f0 con 0x7fafa809b0f0 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafadc66700 10 monclient(hunting): authenticate will time out at 1329.118615s 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafad465700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fafa8094c10 0x7fafa809cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33556/0 (socket says 172.21.15.142:33556) 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafad465700 1 -- 172.21.15.142:0/4156634794 learned_addr learned my addr 172.21.15.142:0/4156634794 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:20.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 1 --2- 172.21.15.142:0/4156634794 >> v2:172.21.15.150:3300/0 conn(0x7fafa809b0f0 0x7fafa809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): get_auth_request con 0x7fafa809b0f0 auth_method 0 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient(hunting): handle_auth_done global_id 84367 payload 274 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 1 -- 172.21.15.142:0/4156634794 >> v1:172.21.15.150:6789/0 conn(0x7fafa80940a0 legacy=0x7fafa8094500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 1 -- 172.21.15.142:0/4156634794 >> v1:172.21.15.142:6789/0 conn(0x7fafa8094c10 legacy=0x7fafa809cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 1 monclient: found mon.noname-c 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:17:20.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.190+0000 7fafacc64700 1 -- 172.21.15.142:0/4156634794 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa809a100 con 0x7fafa809b0f0 2024-09-18T10:17:20.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafacc64700 1 -- 172.21.15.142:0/4156634794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 msgr2=0x7fafa809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafacc64700 1 --2- 172.21.15.142:0/4156634794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa809b4c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faf980098e0 tx=0x7faf980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44f5369369d04cd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 1 -- 172.21.15.142:0/4156634794 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9800ba70 con 0x7fafa809b0f0 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: dump: 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.214 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.215 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 5 monclient: authenticate success, global_id 84367 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 1 -- 172.21.15.142:0/4156634794 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9800bbd0 con 0x7fafa809b0f0 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 4 set_mon_vals no callback set 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 1 -- 172.21.15.142:0/4156634794 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98011420 con 0x7fafa809b0f0 2024-09-18T10:17:20.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafa77fe700 10 monclient: dump: 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 10 monclient: shutdown 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 -- 172.21.15.142:0/4156634794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 msgr2=0x7fafa809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 --2- 172.21.15.142:0/4156634794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa809b4c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faf980098e0 tx=0x7faf980091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 -- 172.21.15.142:0/4156634794 shutdown_connections 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 --2- 172.21.15.142:0/4156634794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa809b4c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 -- 172.21.15.142:0/4156634794 >> 172.21.15.142:0/4156634794 conn(0x7fafa8070ef0 msgr2=0x7fafa80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 -- 172.21.15.142:0/4156634794 shutdown_connections 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.194+0000 7fafadc66700 1 -- 172.21.15.142:0/4156634794 wait complete. 2024-09-18T10:17:20.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient: monmap: 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:20.219 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 Processor -- start 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 -- start start 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient: init 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80940a0 0x7fafa806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-c con 0x7fafa80940a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 0x7fafa8130920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-a con 0x7fafa8094c10 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa8130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): picked mon.noname-b con 0x7fafa809b0f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8133ac0 con 0x7fafa8094c10 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8134080 con 0x7fafa809b0f0 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8133450 con 0x7fafa80940a0 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafadc66700 10 monclient(hunting): authenticate will time out at 1329.125989s 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 0x7fafa8130920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 0x7fafa8130920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39834/0 (socket says 172.21.15.142:39834) 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 -- 172.21.15.142:0/3078150949 learned_addr learned my addr 172.21.15.142:0/3078150949 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): get_auth_request con 0x7fafa8094c10 auth_method 0 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80940a0 0x7fafa806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): get_auth_request con 0x7fafa80940a0 auth_method 0 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafad465700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa8130e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafad465700 10 monclient(hunting): get_auth_request con 0x7fafa809b0f0 auth_method 0 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafad465700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafad465700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafad465700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient(hunting): handle_auth_done global_id 94131 payload 824 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80940a0 msgr2=0x7fafa806ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80940a0 0x7fafa806ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 msgr2=0x7fafa8130e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafa809b0f0 0x7fafa8130e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.198+0000 7fafa7fff700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf98009420 con 0x7fafa8094c10 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa7fff700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 msgr2=0x7fafa8130920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa7fff700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 0x7fafa8130920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf9000b5e0 tx=0x7faf90009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000f720 con 0x7fafa8094c10 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: dump: 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.224 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 5 monclient: authenticate success, global_id 94131 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000f8a0 con 0x7fafa8094c10 2024-09-18T10:17:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000fc60 con 0x7fafa8094c10 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 10 monclient: dump: 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa6ffd700 4 set_mon_vals no callback set 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 monclient: _renew_subs 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8131d10 con 0x7fafa8094c10 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 monclient: _renew_subs 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa81337e0 con 0x7fafa8094c10 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7faf90011610 con 0x7fafa8094c10 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 0x7fafa806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80648a0 con 0x7fafa81313a0 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035ac0 0x7faf88037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 0x7fafa806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7faf9004a040 con 0x7fafa8094c10 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient: get_auth_request con 0x7fafa81313a0 auth_method 0 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafad465700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035ac0 0x7faf88037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.202+0000 7fafad465700 10 monclient: get_auth_request con 0x7faf88035ac0 auth_method 0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafacc64700 10 monclient(hunting): handle_auth_done global_id 94134 payload 824 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafacc64700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafa8131770 con 0x7fafa81313a0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafacc64700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 msgr2=0x7fafa806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafacc64700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 0x7fafa806c370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf980098e0 tx=0x7faf98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafad465700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035ac0 0x7faf88037f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fafa8132270 tx=0x7fafa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98021070 con 0x7fafa81313a0 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 10 monclient: dump: 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7faf98011e20 con 0x7fafa81313a0 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 msgr2=0x7fafa806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 0x7fafa806c370 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf980098e0 tx=0x7faf98016000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 msgr2=0x7fafa806c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.206+0000 7fafa4ff9700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa81313a0 0x7fafa806c370 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faf980098e0 tx=0x7faf98016000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafadc66700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:20.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafadc66700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 0x7fafa806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafadc66700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8072a80 con 0x7fafa806c8b0 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafacc64700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 0x7fafa806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.218+0000 7fafacc64700 10 monclient: get_auth_request con 0x7fafa806c8b0 auth_method 0 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 10 monclient(hunting): handle_auth_done global_id 94137 payload 824 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 1 -- 172.21.15.142:0/3078150949 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fafa809d0c0 con 0x7fafa806c8b0 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 msgr2=0x7fafa806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafacc64700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 0x7fafa806ccb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fafa8072200 tx=0x7faf9801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98034440 con 0x7fafa806c8b0 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 10 monclient: dump: 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.235 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7faf98032410 con 0x7fafa806c8b0 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 msgr2=0x7fafa806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 0x7fafa806ccb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fafa8072200 tx=0x7faf9801e070 comp rx=0 tx=0).stop 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 msgr2=0x7fafa806ccb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafa4ff9700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa806c8b0 0x7fafa806ccb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:20.236 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.237 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:20.238 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.239 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:20.240 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.241 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:20.242 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.243 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:20.244 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.245 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.246 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.248 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.249 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.250 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:20.250 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:20.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035ac0 msgr2=0x7faf88037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf88035ac0 0x7faf88037f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fafa8132270 tx=0x7fafa0009040 comp rx=0 tx=0).stop 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 10 monclient: shutdown 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 msgr2=0x7fafa8130920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 --2- 172.21.15.142:0/3078150949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa8094c10 0x7fafa8130920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faf9000b5e0 tx=0x7faf90009120 comp rx=0 tx=0).stop 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafad465700 1 -- 172.21.15.142:0/3078150949 reap_dead start 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafad465700 1 -- 172.21.15.142:0/3078150949 reap_dead start 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 shutdown_connections 2024-09-18T10:17:20.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.222+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 >> 172.21.15.142:0/3078150949 conn(0x7fafa8070ef0 msgr2=0x7fafa809f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.226+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 shutdown_connections 2024-09-18T10:17:20.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.226+0000 7fafadc66700 1 -- 172.21.15.142:0/3078150949 wait complete. 2024-09-18T10:17:20.253 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:20.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:17:20.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.362+0000 7fc4fbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.362+0000 7fc4fbfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.362+0000 7fc4fbfff700 10 monclient: monmap: 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:20.367355+0000 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:20.367355+0000 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:20.369 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:20.370 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:20.370 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:20.370 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:20.370 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 Processor -- start 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 -- start start 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient: init 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:17:20.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4fc09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc4fc09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4fc134bb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc09dc90 con 0x7fc4fc134bb0 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc09afa0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc09b500 con 0x7fc4fc09b740 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fbfff700 10 monclient(hunting): authenticate will time out at 1329.292877s 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40160/0 (socket says 172.21.15.142:40160) 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 -- 172.21.15.142:0/176236390 learned_addr learned my addr 172.21.15.142:0/176236390 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request con 0x7fc4fc09b740 auth_method 0 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4fb7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc4fc134bb0 0x7fc4fc136f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33566/0 (socket says 172.21.15.142:33566) 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_done global_id 74531 payload 274 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 -- 172.21.15.142:0/176236390 >> v1:172.21.15.150:6789/0 conn(0x7fc4fc09abd0 legacy=0x7fc4fc0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 -- 172.21.15.142:0/176236390 >> v1:172.21.15.142:6789/0 conn(0x7fc4fc134bb0 legacy=0x7fc4fc136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 monclient: found mon.noname-b 2024-09-18T10:17:20.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:20.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.366+0000 7fc4f27fc700 1 -- 172.21.15.142:0/176236390 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc06f810 con 0x7fc4fc09b740 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 1 -- 172.21.15.142:0/176236390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 msgr2=0x7fc4fc134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 1 --2- 172.21.15.142:0/176236390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc076de0 tx=0x7fc4e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be4e14c877376bda server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 1 -- 172.21.15.142:0/176236390 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400bb50 con 0x7fc4fc09b740 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: dump: 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.381 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 1 -- 172.21.15.142:0/176236390 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e400bcb0 con 0x7fc4fc09b740 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 5 monclient: authenticate success, global_id 74531 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:20.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 4 set_mon_vals no callback set 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 1 -- 172.21.15.142:0/176236390 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4011670 con 0x7fc4fc09b740 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fa7fc700 10 monclient: dump: 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.383 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: shutdown 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- 172.21.15.142:0/176236390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 msgr2=0x7fc4fc134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/176236390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc076de0 tx=0x7fc4e4009250 comp rx=0 tx=0).stop 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- 172.21.15.142:0/176236390 shutdown_connections 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/176236390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc134670 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- 172.21.15.142:0/176236390 >> 172.21.15.142:0/176236390 conn(0x7fc4fc070ef0 msgr2=0x7fc4fc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- 172.21.15.142:0/176236390 shutdown_connections 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- 172.21.15.142:0/176236390 wait complete. 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: monmap: 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 Processor -- start 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- start start 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: init 2024-09-18T10:17:20.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 0x7fc4fc098c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fc4fc09abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc099190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fc4fc09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4fc134bb0 0x7fc4fc0996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fc4fc134bb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc08dfd0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc09d240 con 0x7fc4fc134bb0 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc09f860 con 0x7fc4fc09b740 2024-09-18T10:17:20.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fbfff700 10 monclient(hunting): authenticate will time out at 1329.298545s 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc099190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc099190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40166/0 (socket says 172.21.15.142:40166) 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 0x7fc4fc098c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 1 -- 172.21.15.142:0/3682521284 learned_addr learned my addr 172.21.15.142:0/3682521284 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request con 0x7fc4fc09b740 auth_method 0 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): get_auth_request con 0x7fc4fc09abd0 auth_method 0 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fb7fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4fc134bb0 0x7fc4fc0996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient(hunting): handle_auth_done global_id 94140 payload 824 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 msgr2=0x7fc4fc099190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc09b740 0x7fc4fc099190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4fc134bb0 msgr2=0x7fc4fc0996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4fc134bb0 0x7fc4fc0996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4faffd700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e40094f0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fb7fe700 10 monclient: get_auth_request con 0x7fc4fc134bb0 auth_method 0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.370+0000 7fc4fb7fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4fc134bb0 0x7fc4fc0996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4faffd700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 msgr2=0x7fc4fc098c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4faffd700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 0x7fc4fc098c50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c4e0 tx=0x7fc4ec007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0117a0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: dump: 2024-09-18T10:17:20.389 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 5 monclient: authenticate success, global_id 94140 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ec011920 con 0x7fc4fc09abd0 2024-09-18T10:17:20.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 monclient: _renew_subs 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc08dc80 con 0x7fc4fc09abd0 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec011ce0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 10 monclient: dump: 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.391 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f9ffb700 4 set_mon_vals no callback set 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 monclient: _renew_subs 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc099c10 con 0x7fc4fc09abd0 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc4ec012630 con 0x7fc4fc09abd0 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 0x7fc4fc06b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc099e30 con 0x7fc4fc08cf40 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 0x7fc4fc06b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient: get_auth_request con 0x7fc4fc08cf40 auth_method 0 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4dc035ac0 0x7fc4dc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_done global_id 74537 payload 824 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7fc4ec04a0f0 con 0x7fc4fc09abd0 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.374+0000 7fc4f27fc700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4fc08d850 con 0x7fc4fc08cf40 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4fb7fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4dc035ac0 0x7fc4dc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4fb7fe700 10 monclient: get_auth_request con 0x7fc4dc035ac0 auth_method 0 2024-09-18T10:17:20.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f27fc700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 msgr2=0x7fc4fc06b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f27fc700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 0x7fc4fc06b370 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009930 tx=0x7fc4e4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4021070 con 0x7fc4fc08cf40 2024-09-18T10:17:20.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 10 monclient: dump: 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc4e401c660 con 0x7fc4fc08cf40 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 msgr2=0x7fc4fc06b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 0x7fc4fc06b370 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009930 tx=0x7fc4e4016000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 msgr2=0x7fc4fc06b370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4f37fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc08cf40 0x7fc4fc06b370 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009930 tx=0x7fc4e4016000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.378+0000 7fc4fb7fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4dc035ac0 0x7fc4dc037f70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc08db80 tx=0x7fc4f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.386+0000 7fc4fbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.386+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 0x7fc4fc06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.386+0000 7fc4fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.386+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc0648a0 con 0x7fc4fc06b8b0 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 0x7fc4fc06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient: get_auth_request con 0x7fc4fc06b8b0 auth_method 0 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 10 monclient(hunting): handle_auth_done global_id 74540 payload 824 2024-09-18T10:17:20.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 1 -- 172.21.15.142:0/3682521284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4fc08d4f0 con 0x7fc4fc06b8b0 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 msgr2=0x7fc4fc06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f27fc700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 0x7fc4fc06bcb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc09a270 tx=0x7fc4e4021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4025e10 con 0x7fc4fc06b8b0 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.390+0000 7fc4f37fe700 10 monclient: dump: 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.399 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7fc4e401b440 con 0x7fc4fc06b8b0 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 msgr2=0x7fc4fc06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 0x7fc4fc06bcb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc09a270 tx=0x7fc4e4021000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 msgr2=0x7fc4fc06bcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4f37fe700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc4fc06b8b0 0x7fc4fc06bcb0 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc09a270 tx=0x7fc4e4021000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:20.401 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.402 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.403 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:20.404 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:20.405 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.406 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.407 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.408 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.409 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.410 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.411 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.412 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.413 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:20.414 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:20.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4dc035ac0 msgr2=0x7fc4dc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc4dc035ac0 0x7fc4dc037f70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc08db80 tx=0x7fc4f4009040 comp rx=0 tx=0).stop 2024-09-18T10:17:20.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 10 monclient: shutdown 2024-09-18T10:17:20.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 msgr2=0x7fc4fc098c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 --2- 172.21.15.142:0/3682521284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4fc09abd0 0x7fc4fc098c50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00c4e0 tx=0x7fc4ec007ea0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fb7fe700 1 -- 172.21.15.142:0/3682521284 reap_dead start 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fb7fe700 1 -- 172.21.15.142:0/3682521284 reap_dead start 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 shutdown_connections 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 >> 172.21.15.142:0/3682521284 conn(0x7fc4fc070ef0 msgr2=0x7fc4fc09bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 shutdown_connections 2024-09-18T10:17:20.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.394+0000 7fc4fbfff700 1 -- 172.21.15.142:0/3682521284 wait complete. 2024-09-18T10:17:20.417 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:20.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:17:20.417 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.522+0000 7f946e319700 10 monclient: get_monmap_and_config 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.522+0000 7f946e319700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.522+0000 7f946e319700 10 monclient: monmap: 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.528 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:20.526739+0000 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:20.526739+0000 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:20.529 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 Processor -- start 2024-09-18T10:17:20.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 -- start start 2024-09-18T10:17:20.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient: init 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f94680a2730 0x7f946809c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): picked mon.noname-a con 0x7f94680a2730 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946806f540 0x7f946809cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): picked mon.noname-b con 0x7f946806f540 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 0x7f9468136630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): picked mon.noname-c con 0x7f946809d290 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680999c0 con 0x7f94680a2730 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680a2b00 con 0x7f946809d290 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9468099400 con 0x7f946806f540 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946e319700 10 monclient(hunting): authenticate will time out at 1329.453840s 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f94680a2730 0x7f946809c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f94680a2730 0x7f946809c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39868/0 (socket says 172.21.15.142:39868) 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 -- 172.21.15.142:0/2020025488 learned_addr learned my addr 172.21.15.142:0/2020025488 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946806f540 0x7f946809cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): get_auth_request con 0x7f94680a2730 auth_method 0 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): get_auth_request con 0x7f946806f540 auth_method 0 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 1 --2- 172.21.15.142:0/2020025488 >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 0x7f9468136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 10 monclient(hunting): get_auth_request con 0x7f946809d290 auth_method 0 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient(hunting): handle_auth_done global_id 94143 payload 274 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 -- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946806f540 msgr2=0x7f946809cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 --2- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946806f540 0x7f946809cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 -- 172.21.15.142:0/2020025488 >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 msgr2=0x7f9468136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 --2- 172.21.15.142:0/2020025488 >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 0x7f9468136630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f9467fff700 1 -- 172.21.15.142:0/2020025488 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680996e0 con 0x7f94680a2730 2024-09-18T10:17:20.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.526+0000 7f946c8b6700 1 --2- 172.21.15.142:0/2020025488 >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 0x7f9468136630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:20.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9467fff700 1 -- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94680a2730 msgr2=0x7f946809c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9467fff700 1 --2- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94680a2730 0x7f946809c810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9468098d90 tx=0x7f9458009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cae306fcd396b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 1 -- 172.21.15.142:0/2020025488 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458009e70 con 0x7f94680a2730 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: dump: 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.545 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 5 monclient: authenticate success, global_id 94143 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 1 -- 172.21.15.142:0/2020025488 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945800ba70 con 0x7f94680a2730 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 4 set_mon_vals no callback set 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 1 -- 172.21.15.142:0/2020025488 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945800be30 con 0x7f94680a2730 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f9466ffd700 10 monclient: dump: 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 10 monclient: shutdown 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 -- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94680a2730 msgr2=0x7f946809c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 --2- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94680a2730 0x7f946809c810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9468098d90 tx=0x7f9458009250 comp rx=0 tx=0).stop 2024-09-18T10:17:20.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 -- 172.21.15.142:0/2020025488 shutdown_connections 2024-09-18T10:17:20.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 --2- 172.21.15.142:0/2020025488 >> v2:172.21.15.150:3300/0 conn(0x7f946809d290 0x7f9468136630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 --2- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946806f540 0x7f946809cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 --2- 172.21.15.142:0/2020025488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94680a2730 0x7f946809c810 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 -- 172.21.15.142:0/2020025488 >> 172.21.15.142:0/2020025488 conn(0x7f9468071740 msgr2=0x7f94680a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 -- 172.21.15.142:0/2020025488 shutdown_connections 2024-09-18T10:17:20.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 1 -- 172.21.15.142:0/2020025488 wait complete. 2024-09-18T10:17:20.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.530+0000 7f946e319700 10 monclient: monmap: 2024-09-18T10:17:20.552 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.552 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:20.552 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:20.552 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:20.552 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 Processor -- start 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 -- start start 2024-09-18T10:17:20.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient: init 2024-09-18T10:17:20.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:20.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): picked mon.noname-a con 0x7f946806f540 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946809d290 0x7f946813d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): picked mon.noname-c con 0x7f946809d290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94680a2730 0x7f94681414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): picked mon.noname-b con 0x7f94680a2730 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468098770 con 0x7f946806f540 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946809dfc0 con 0x7f94680a2730 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680a0720 con 0x7f946809d290 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946e319700 10 monclient(hunting): authenticate will time out at 1329.461001s 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946809d290 0x7f946813d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39876/0 (socket says 172.21.15.142:39876) 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 -- 172.21.15.142:0/3082934533 learned_addr learned my addr 172.21.15.142:0/3082934533 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): get_auth_request con 0x7f946806f540 auth_method 0 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): get_auth_request con 0x7f946809d290 auth_method 0 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94680a2730 0x7f94681414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): get_auth_request con 0x7f94680a2730 auth_method 0 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f946c8b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient(hunting): handle_auth_done global_id 94146 payload 824 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946809d290 msgr2=0x7f946813d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f946809d290 0x7f946813d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94680a2730 msgr2=0x7f94681414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94680a2730 0x7f94681414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.534+0000 7f9467fff700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580094f0 con 0x7f946806f540 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f9467fff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 msgr2=0x7f946813df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f9467fff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9458006fd0 tx=0x7f9458017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458006a90 con 0x7f946806f540 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: dump: 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.560 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94580118c0 con 0x7f946806f540 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 5 monclient: authenticate success, global_id 94146 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 monclient: _renew_subs 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946813ea80 con 0x7f946806f540 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458011c80 con 0x7f946806f540 2024-09-18T10:17:20.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 10 monclient: dump: 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.563 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94667fc700 4 set_mon_vals no callback set 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 monclient: _renew_subs 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9468121960 con 0x7f946806f540 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 0x7f9468029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468122b30 con 0x7f9468097d40 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9458023070 con 0x7f946806f540 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 0x7f9468029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient: get_auth_request con 0x7f9468097d40 auth_method 0 2024-09-18T10:17:20.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9454035ac0 0x7f9454037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f945804b670 con 0x7f946806f540 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 10 monclient(hunting): handle_auth_done global_id 84379 payload 824 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f94677fe700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9468122810 con 0x7f9468097d40 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946c8b6700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9454035ac0 0x7f9454037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.538+0000 7f946c8b6700 10 monclient: get_auth_request con 0x7f9454035ac0 auth_method 0 2024-09-18T10:17:20.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f94677fe700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 msgr2=0x7f9468029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f94677fe700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 0x7f9468029d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00fe80 con 0x7f9468097d40 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 10 monclient: dump: 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.569 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f946c8b6700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9454035ac0 0x7f9454037f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9468123090 tx=0x7f9460009990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f945c014440 con 0x7f9468097d40 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 msgr2=0x7f9468029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 0x7f9468029d00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 msgr2=0x7f9468029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.542+0000 7f944ffff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9468097d40 0x7f9468029d00 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f946e319700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f946e319700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 0x7f946802a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f946e319700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680647a0 con 0x7f946802a240 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 0x7f946802a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient: get_auth_request con 0x7f946802a240 auth_method 0 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 10 monclient(hunting): handle_auth_done global_id 84382 payload 824 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.558+0000 7f94677fe700 1 -- 172.21.15.142:0/3082934533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9468119be0 con 0x7f946802a240 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f94677fe700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 msgr2=0x7f946802a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f94677fe700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 0x7f946802a640 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c01de40 con 0x7f946802a240 2024-09-18T10:17:20.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:20.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 10 monclient: dump: 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.577 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f945c01c3e0 con 0x7f946802a240 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 msgr2=0x7f946802a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 0x7f946802a640 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c01c000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 msgr2=0x7f946802a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f944ffff700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946802a240 0x7f946802a640 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9468098b10 tx=0x7f945c01c000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.579 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:20.579 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 90, 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.580 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:20.581 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:20.582 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:20.583 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:20.584 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.585 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.586 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.587 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.588 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:20.589 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:20.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:20.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:20.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:20.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:20.594 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:20.594 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9454035ac0 msgr2=0x7f9454037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9454035ac0 0x7f9454037f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9468123090 tx=0x7f9460009990 comp rx=0 tx=0).stop 2024-09-18T10:17:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 10 monclient: shutdown 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 msgr2=0x7f946813df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9458006fd0 tx=0x7f9458017000 comp rx=0 tx=0).stop 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946c8b6700 1 -- 172.21.15.142:0/3082934533 reap_dead start 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 shutdown_connections 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 --2- 172.21.15.142:0/3082934533 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946806f540 0x7f946813df60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 >> 172.21.15.142:0/3082934533 conn(0x7f9468071740 msgr2=0x7f946806cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 shutdown_connections 2024-09-18T10:17:20.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.562+0000 7f946e319700 1 -- 172.21.15.142:0/3082934533 wait complete. 2024-09-18T10:17:20.597 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:17:20.597 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:17:20.597 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:17:20.597 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:17:20.597 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:17:20.597 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:17:20.597 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:17:20.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.598+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d812de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:20.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.598+0000 7f46dcb11700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d812de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:20.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.598+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46cc012d90 tx=0x7f46cc00f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:20.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.598+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f46cc012d90 tx=0x7f46cc00f040 comp rx=0 tx=0).stop 2024-09-18T10:17:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 msgr2=0x7f46d812de40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:20.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46d80993d0 0x7f46d812de40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46bc049520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): picked mon.c con 0x7f46d809d970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46bc04bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): picked mon.a con 0x7f46d8098ac0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 0x7f46bc04ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): picked mon.b con 0x7f46bc04c7b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc04f6a0 con 0x7f46d8098ac0 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc04f7e0 con 0x7f46bc04c7b0 2024-09-18T10:17:20.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc04f9a0 con 0x7f46d809d970 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dd312700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 msgr2=0x7f46bc04ec30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dd312700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 0x7f46bc04ec30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (unknown 0 0 0) 0x7f46cc016b20 con 0x7f46d80993d0 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46c97fa700 10 monclient(hunting): discarding stray monitor message mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46bc04bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46bc049520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): get_auth_request con 0x7f46d8098ac0 auth_method 0 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.609 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): get_auth_request con 0x7f46d809d970 auth_method 0 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46dcb11700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:20.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient(hunting): handle_auth_done global_id 84364 payload 293 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 msgr2=0x7f46bc049520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46bc049520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 msgr2=0x7f46bc04ec30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 0x7f46bc04ec30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 monclient: found mon.a 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f46bc04fbc0 con 0x7f46d8098ac0 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d809dd40 con 0x7f46d8098ac0 2024-09-18T10:17:20.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.602+0000 7f46ddb13700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.612 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46ddb13700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 msgr2=0x7f46bc04bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46ddb13700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46bc04bde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009340 tx=0x7f46d0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d000bcf0 con 0x7f46d8098ac0 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 10 monclient: dump: 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:20.613 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d000be50 con 0x7f46d8098ac0 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f46d0024070 con 0x7f46d8098ac0 2024-09-18T10:17:20.614 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.606+0000 7f46caffd700 4 set_mon_vals no callback set 2024-09-18T10:17:20.616 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:17:20.616 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:17:20.616 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:20.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.614+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f46d001bbb0 con 0x7f46d8098ac0 2024-09-18T10:17:20.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.614+0000 7f46c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.614+0000 7f46c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:20.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: get_monmap_and_config 2024-09-18T10:17:20.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: monmap: 2024-09-18T10:17:20.727 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.727 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:20.725852+0000 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:20.725852+0000 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:20.728 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 Processor -- start 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 -- start start 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: init 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c7409aae0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c74132570 0x7f7c74134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-a con 0x7f7c74132570 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 0x7f7c74137270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-c con 0x7f7c74134e90 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7409aee0 con 0x7f7c7409aae0 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7406f6d0 con 0x7f7c74132570 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c740955a0 con 0x7f7c74134e90 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c79ef2700 10 monclient(hunting): authenticate will time out at 1329.651550s 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c74132570 0x7f7c74134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c74132570 0x7f7c74134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39880/0 (socket says 172.21.15.142:39880) 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 1 -- 172.21.15.142:0/3330889130 learned_addr learned my addr 172.21.15.142:0/3330889130 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): get_auth_request con 0x7f7c74132570 auth_method 0 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c78ef0700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7c7409aae0 0x7f7c74096260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33576/0 (socket says 172.21.15.142:33576) 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c796f1700 1 -- 172.21.15.142:0/3330889130 >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 msgr2=0x7f7c74137270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c796f1700 1 --2- 172.21.15.142:0/3330889130 >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 0x7f7c74137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.722+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_done global_id 94149 payload 274 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 -- 172.21.15.142:0/3330889130 >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 msgr2=0x7f7c74137270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 --2- 172.21.15.142:0/3330889130 >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 0x7f7c74137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 -- 172.21.15.142:0/3330889130 >> v1:172.21.15.142:6789/0 conn(0x7f7c7409aae0 legacy=0x7f7c74096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:20.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 -- 172.21.15.142:0/3330889130 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74095b60 con 0x7f7c74132570 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 -- 172.21.15.142:0/3330889130 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74132570 msgr2=0x7f7c74134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6bfff700 1 --2- 172.21.15.142:0/3330889130 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74132570 0x7f7c74134950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c7409b9d0 tx=0x7f7c64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2505c2b0a685fb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 1 -- 172.21.15.142:0/3330889130 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400ba70 con 0x7f7c74132570 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: dump: 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.737 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 1 -- 172.21.15.142:0/3330889130 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6400bbd0 con 0x7f7c74132570 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 1 -- 172.21.15.142:0/3330889130 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64011420 con 0x7f7c74132570 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c79ef2700 5 monclient: authenticate success, global_id 94149 2024-09-18T10:17:20.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c79ef2700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c79ef2700 4 set_mon_vals no callback set 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.726+0000 7f7c6b7fe700 10 monclient: dump: 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.739 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: shutdown 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3330889130 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74132570 msgr2=0x7f7c74134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3330889130 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74132570 0x7f7c74134950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c7409b9d0 tx=0x7f7c64009250 comp rx=0 tx=0).stop 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3330889130 shutdown_connections 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3330889130 >> v2:172.21.15.150:3300/0 conn(0x7f7c74134e90 0x7f7c74137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3330889130 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74132570 0x7f7c74134950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3330889130 >> 172.21.15.142:0/3330889130 conn(0x7f7c740a1fe0 msgr2=0x7f7c74070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3330889130 shutdown_connections 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3330889130 wait complete. 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: build_initial_monmap 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: monmap: 2024-09-18T10:17:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 Processor -- start 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- start start 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: init 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 0x7f7c74141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-b con 0x7f7c7409aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c74132570 0x7f7c741420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-c con 0x7f7c74132570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): picked mon.noname-a con 0x7f7c74134e90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7412a970 con 0x7f7c74134e90 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74098dc0 con 0x7f7c7409aae0 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7409f540 con 0x7f7c74132570 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c79ef2700 10 monclient(hunting): authenticate will time out at 1329.658018s 2024-09-18T10:17:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c78ef0700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 msgr2=0x7f7c74141b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c78ef0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 0x7f7c74141b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39892/0 (socket says 172.21.15.142:39892) 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 -- 172.21.15.142:0/3631629801 learned_addr learned my addr 172.21.15.142:0/3631629801 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): get_auth_request con 0x7f7c74134e90 auth_method 0 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c74132570 0x7f7c741420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): get_auth_request con 0x7f7c74132570 auth_method 0 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c6bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient(hunting): handle_auth_done global_id 94152 payload 824 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 -- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c74132570 msgr2=0x7f7c741420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c74132570 0x7f7c741420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 -- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 msgr2=0x7f7c74141b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 0x7f7c74141b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient: _finish_hunting 0 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 monclient: found mon.noname-a 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:20.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.730+0000 7f7c796f1700 1 -- 172.21.15.142:0/3631629801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640094f0 con 0x7f7c74134e90 2024-09-18T10:17:20.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c796f1700 1 -- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 msgr2=0x7f7c7406de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c796f1700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00bb40 tx=0x7f7c6c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00ca50 con 0x7f7c74134e90 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: dump: 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 5 monclient: authenticate success, global_id 94152 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _renew_subs 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7406f8d0 con 0x7f7c74134e90 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c00cbd0 con 0x7f7c74134e90 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00cd80 con 0x7f7c74134e90 2024-09-18T10:17:20.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 10 monclient: dump: 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:20.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c6affd700 4 set_mon_vals no callback set 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _renew_subs 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7409a840 con 0x7f7c74134e90 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7c6c012020 con 0x7f7c74134e90 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 0x7f7c60037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f7c6c04a210 con 0x7f7c74134e90 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.734+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c740999a0 con 0x7f7c74134e90 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.738+0000 7f7c78ef0700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 0x7f7c60037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.738+0000 7f7c78ef0700 10 monclient: get_auth_request con 0x7f7c60035a70 auth_method 0 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.738+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:17:20.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.738+0000 7f7c78ef0700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 0x7f7c60037f20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7c70004910 tx=0x7f7c70007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:20.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.742+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7c6c01b030 con 0x7f7c74134e90 2024-09-18T10:17:20.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.742+0000 7f7c68ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:20.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.742+0000 7f7c68ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:20.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.786+0000 7f46df576700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:20.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.786+0000 7f46df576700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:20.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.786+0000 7f46df576700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.793 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:20.786+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f46d8067b10 con 0x7f46d8098ac0 2024-09-18T10:17:20.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c79ef2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:20.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c79ef2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:20.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c79ef2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:20.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7c74072a80 con 0x7f7c74134e90 2024-09-18T10:17:20.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c68ff9700 1 -- 172.21.15.142:0/3631629801 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f7c6c01b020 con 0x7f7c74134e90 2024-09-18T10:17:20.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c68ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:20.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.970+0000 7f7c68ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:20.977 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 msgr2=0x7f7c60037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 0x7f7c60037f20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7c70004910 tx=0x7f7c70007b50 comp rx=0 tx=0).stop 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 10 monclient: shutdown 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 msgr2=0x7f7c7406de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:20.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00bb40 tx=0x7f7c6c0097d0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 shutdown_connections 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c60035a70 0x7f7c60037f20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c74134e90 0x7f7c7406de70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c74132570 0x7f7c741420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 --2- 172.21.15.142:0/3631629801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c7409aae0 0x7f7c74141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 >> 172.21.15.142:0/3631629801 conn(0x7f7c740a1fe0 msgr2=0x7f7c7409deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 shutdown_connections 2024-09-18T10:17:20.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:20.974+0000 7f7c79ef2700 1 -- 172.21.15.142:0/3631629801 wait complete. 2024-09-18T10:17:23.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:23.094+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:23.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:23.094+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:23.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:23.094+0000 7f46c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:23.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:24.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:24.090+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:24.782 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:24.090+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: get_monmap_and_config 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: build_initial_monmap 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: monmap: 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:24.100605+0000 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:24.100605+0000 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:24.798 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 Processor -- start 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 -- start start 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: init 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8df409a830 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:24.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8df409ac00 0x7f8df40992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-a con 0x7f8df409ac00 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 0x7f8df413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-c con 0x7f8df413b6e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df4099820 con 0x7f8df409a830 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df4099d50 con 0x7f8df409ac00 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df409dca0 con 0x7f8df413b6e0 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df95da700 10 monclient(hunting): authenticate will time out at 1333.026117s 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df37fe700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 msgr2=0x7f8df413dac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df2ffd700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f8df409a830 legacy=0x7f8df4098bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:24.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8df409ac00 0x7f8df40992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8df37fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 0x7f8df413dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8df409ac00 0x7f8df40992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53248/0 (socket says 172.21.15.142:53248) 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 -- 172.21.15.142:0/3547225765 learned_addr learned my addr 172.21.15.142:0/3547225765 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): get_auth_request con 0x7f8df409ac00 auth_method 0 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient(hunting): handle_auth_done global_id 94155 payload 274 2024-09-18T10:17:24.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 -- 172.21.15.142:0/3547225765 >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 msgr2=0x7f8df413dac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 --2- 172.21.15.142:0/3547225765 >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 0x7f8df413dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 -- 172.21.15.142:0/3547225765 >> v1:172.21.15.150:6789/0 conn(0x7f8df409a830 legacy=0x7f8df4098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 monclient: found mon.noname-a 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.098+0000 7f8debfff700 1 -- 172.21.15.142:0/3547225765 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4099a70 con 0x7f8df409ac00 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8debfff700 1 -- 172.21.15.142:0/3547225765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 msgr2=0x7f8df40992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8debfff700 1 --2- 172.21.15.142:0/3547225765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df40992e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8df406fdb0 tx=0x7f8de4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e5a7d6b1459b83 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 1 -- 172.21.15.142:0/3547225765 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de400ba70 con 0x7f8df409ac00 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: dump: 2024-09-18T10:17:24.802 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 5 monclient: authenticate success, global_id 94155 2024-09-18T10:17:24.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 1 -- 172.21.15.142:0/3547225765 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de400bbd0 con 0x7f8df409ac00 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 4 set_mon_vals no callback set 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 1 -- 172.21.15.142:0/3547225765 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4011420 con 0x7f8df409ac00 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df27fc700 10 monclient: dump: 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.804 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 10 monclient: shutdown 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 -- 172.21.15.142:0/3547225765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 msgr2=0x7f8df40992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 --2- 172.21.15.142:0/3547225765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df40992e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8df406fdb0 tx=0x7f8de4009250 comp rx=0 tx=0).stop 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 -- 172.21.15.142:0/3547225765 shutdown_connections 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 --2- 172.21.15.142:0/3547225765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df40992e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 --2- 172.21.15.142:0/3547225765 >> v2:172.21.15.150:3300/0 conn(0x7f8df413b6e0 0x7f8df413dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 -- 172.21.15.142:0/3547225765 >> 172.21.15.142:0/3547225765 conn(0x7f8df40a14e0 msgr2=0x7f8df40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 -- 172.21.15.142:0/3547225765 shutdown_connections 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.102+0000 7f8df95da700 1 -- 172.21.15.142:0/3547225765 wait complete. 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient: build_initial_monmap 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient: monmap: 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 Processor -- start 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 -- start start 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient: init 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 0x7f8df4094180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-b con 0x7f8df409a830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-a con 0x7f8df409ac00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8df413b6e0 0x7f8df413ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): picked mon.noname-c con 0x7f8df413b6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df4094830 con 0x7f8df409ac00 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:24.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df413d510 con 0x7f8df409a830 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df406b7d0 con 0x7f8df413b6e0 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df95da700 10 monclient(hunting): authenticate will time out at 1333.034291s 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df2ffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 msgr2=0x7f8df4094180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 0x7f8df4094180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8df413b6e0 0x7f8df413ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53264/0 (socket says 172.21.15.142:53264) 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 -- 172.21.15.142:0/4234114197 learned_addr learned my addr 172.21.15.142:0/4234114197 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): get_auth_request con 0x7f8df409ac00 auth_method 0 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:24.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): get_auth_request con 0x7f8df413b6e0 auth_method 0 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8df37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient(hunting): handle_auth_done global_id 94158 payload 824 2024-09-18T10:17:24.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 -- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8df413b6e0 msgr2=0x7f8df413ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8df413b6e0 0x7f8df413ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 -- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 msgr2=0x7f8df4094180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 0x7f8df4094180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 monclient: found mon.noname-a 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.106+0000 7f8debfff700 1 -- 172.21.15.142:0/4234114197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40094f0 con 0x7f8df409ac00 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8debfff700 1 -- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 msgr2=0x7f8df413a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8debfff700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8df40a2530 tx=0x7f8de4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4006bc0 con 0x7f8df409ac00 2024-09-18T10:17:24.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: dump: 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 5 monclient: authenticate success, global_id 94158 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de401a670 con 0x7f8df409ac00 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _renew_subs 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df406c4e0 con 0x7f8df409ac00 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de401aa50 con 0x7f8df409ac00 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 10 monclient: dump: 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:24.812 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df17fa700 4 set_mon_vals no callback set 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _renew_subs 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df406f200 con 0x7f8df409ac00 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8de401b440 con 0x7f8df409ac00 2024-09-18T10:17:24.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 0x7f8dd4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f8de404c340 con 0x7f8df409ac00 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.110+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df40696a0 con 0x7f8df409ac00 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.114+0000 7f8df2ffd700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 0x7f8dd4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.114+0000 7f8df2ffd700 10 monclient: get_auth_request con 0x7f8dd4035a70 auth_method 0 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.114+0000 7f8df2ffd700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 0x7f8dd4037f20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc003980 tx=0x7f8ddc006dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.122+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8de4018dc0 con 0x7f8df409ac00 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.122+0000 7f8deaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.122+0000 7f8deaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.342+0000 7f8df95da700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.342+0000 7f8df95da700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.342+0000 7f8df95da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.342+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8df406d6e0 con 0x7f8df409ac00 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8deaffd700 1 -- 172.21.15.142:0/4234114197 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f8de404c020 con 0x7f8df409ac00 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8deaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8deaffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:24.815 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 msgr2=0x7f8dd4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 0x7f8dd4037f20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc003980 tx=0x7f8ddc006dc0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 10 monclient: shutdown 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 msgr2=0x7f8df413a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8df40a2530 tx=0x7f8de4016000 comp rx=0 tx=0).stop 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 shutdown_connections 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8dd4035a70 0x7f8dd4037f20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8df413b6e0 0x7f8df413ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8df409ac00 0x7f8df413a6f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 --2- 172.21.15.142:0/4234114197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8df409a830 0x7f8df4094180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:24.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 >> 172.21.15.142:0/4234114197 conn(0x7f8df40a14e0 msgr2=0x7f8df413bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:24.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 shutdown_connections 2024-09-18T10:17:24.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:24.346+0000 7f8df95da700 1 -- 172.21.15.142:0/4234114197 wait complete. 2024-09-18T10:17:25.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:25.090+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:25.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:25.090+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:26.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:26.094+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:26.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:26.094+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:27.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:27.089+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:27.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:27.089+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:27.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:27.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: monmap: 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:27.928225+0000 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:27.928225+0000 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:27.930 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:27.931 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 Processor -- start 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 -- start start 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: init 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d80953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f22d809b050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22d8097970 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:27.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22d8133780 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d809a570 con 0x7f22d8133780 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22d806f900 con 0x7f22d8097970 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d809b420 con 0x7f22d809b050 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d7fff700 10 monclient(hunting): authenticate will time out at 1336.854279s 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f22d8133780 0x7f22d8135b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:57872/0 (socket says 172.21.15.142:57872) 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d80953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d77fe700 1 -- 172.21.15.142:0/3248497817 learned_addr learned my addr 172.21.15.142:0/3248497817 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d67fc700 1 -- 172.21.15.142:0/3248497817 >> v1:172.21.15.150:6789/0 conn(0x7f22d8097970 legacy=0x7f22d8097d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): get_auth_request con 0x7f22d809b050 auth_method 0 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:27.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient(hunting): handle_auth_done global_id 74555 payload 274 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 1 -- 172.21.15.142:0/3248497817 >> v1:172.21.15.150:6789/0 conn(0x7f22d8097970 legacy=0x7f22d8097d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 1 -- 172.21.15.142:0/3248497817 >> v1:172.21.15.142:6789/0 conn(0x7f22d8133780 legacy=0x7f22d8135b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 1 monclient: found mon.noname-b 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:27.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.926+0000 7f22d6ffd700 1 -- 172.21.15.142:0/3248497817 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d809a290 con 0x7f22d809b050 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d6ffd700 1 -- 172.21.15.142:0/3248497817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 msgr2=0x7f22d80953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d6ffd700 1 --2- 172.21.15.142:0/3248497817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d80953c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22d809de40 tx=0x7f22c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc5979685b00f3cf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 1 -- 172.21.15.142:0/3248497817 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000ba70 con 0x7f22d809b050 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: dump: 2024-09-18T10:17:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 5 monclient: authenticate success, global_id 74555 2024-09-18T10:17:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 1 -- 172.21.15.142:0/3248497817 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c000bbf0 con 0x7f22d809b050 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 4 set_mon_vals no callback set 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 1 -- 172.21.15.142:0/3248497817 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0011420 con 0x7f22d809b050 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d5ffb700 10 monclient: dump: 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:27.939 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 10 monclient: shutdown 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 -- 172.21.15.142:0/3248497817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 msgr2=0x7f22d80953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 --2- 172.21.15.142:0/3248497817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d80953c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22d809de40 tx=0x7f22c00091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 -- 172.21.15.142:0/3248497817 shutdown_connections 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 --2- 172.21.15.142:0/3248497817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d80953c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 -- 172.21.15.142:0/3248497817 >> 172.21.15.142:0/3248497817 conn(0x7f22d80a1fe0 msgr2=0x7f22d8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 -- 172.21.15.142:0/3248497817 shutdown_connections 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.930+0000 7f22d7fff700 1 -- 172.21.15.142:0/3248497817 wait complete. 2024-09-18T10:17:27.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient: monmap: 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:27.941 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 Processor -- start 2024-09-18T10:17:27.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 -- start start 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient: init 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 0x7f22d80884b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f22d8097970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d81392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f22d809b050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f22d8133780 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d808ac10 con 0x7f22d8133780 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:27.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d809d0d0 con 0x7f22d8097970 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22d8089560 con 0x7f22d809b050 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d7fff700 10 monclient(hunting): authenticate will time out at 1336.861656s 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d6ffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 msgr2=0x7f22d80884b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d6ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 0x7f22d80884b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d81392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53270/0 (socket says 172.21.15.142:53270) 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d81392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44106/0 (socket says 172.21.15.142:44106) 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 -- 172.21.15.142:0/2952596390 learned_addr learned my addr 172.21.15.142:0/2952596390 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): get_auth_request con 0x7f22d8133780 auth_method 0 2024-09-18T10:17:27.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): get_auth_request con 0x7f22d809b050 auth_method 0 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient(hunting): handle_auth_done global_id 94161 payload 824 2024-09-18T10:17:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 -- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 msgr2=0x7f22d81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d81392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 -- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 msgr2=0x7f22d80884b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 0x7f22d80884b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.934+0000 7f22d77fe700 1 -- 172.21.15.142:0/2952596390 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0009420 con 0x7f22d8133780 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d77fe700 1 -- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 msgr2=0x7f22d8094780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d77fe700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cbf0 tx=0x7f22d00053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000ab20 con 0x7f22d8133780 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: dump: 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 5 monclient: authenticate success, global_id 94161 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _renew_subs 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d808a930 con 0x7f22d8133780 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _renew_subs 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d8089c30 con 0x7f22d8133780 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d000aca0 con 0x7f22d8133780 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0007340 con 0x7f22d8133780 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d57fa700 4 set_mon_vals no callback set 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 10 monclient: dump: 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc0050b0 con 0x7f22d8133780 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f22d00076b0 con 0x7f22d8133780 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 0x7f22b8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.938+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f22d004a900 con 0x7f22d8133780 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.942+0000 7f22d6ffd700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 0x7f22b8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.942+0000 7f22d6ffd700 10 monclient: get_auth_request con 0x7f22b8035a70 auth_method 0 2024-09-18T10:17:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.942+0000 7f22d6ffd700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 0x7f22b8037f20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f22d809de40 tx=0x7f22c0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.950+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f22d001a360 con 0x7f22d8133780 2024-09-18T10:17:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.950+0000 7f22ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:27.950+0000 7f22ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:28.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:28.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:28.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:28.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:28.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22d7fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22d7fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22d7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f22bc0058e0 con 0x7f22d8133780 2024-09-18T10:17:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22ceffd700 1 -- 172.21.15.142:0/2952596390 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f22d0010300 con 0x7f22d8133780 2024-09-18T10:17:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22ceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.178+0000 7f22ceffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:28.185 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":90,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 msgr2=0x7f22b8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 0x7f22b8037f20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f22d809de40 tx=0x7f22c0016000 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 10 monclient: shutdown 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 msgr2=0x7f22d8094780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.182+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22d000cbf0 tx=0x7f22d00053d0 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 shutdown_connections 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22b8035a70 0x7f22b8037f20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22d8133780 0x7f22d8094780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22d809b050 0x7f22d81392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 --2- 172.21.15.142:0/2952596390 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22d8097970 0x7f22d80884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 >> 172.21.15.142:0/2952596390 conn(0x7f22d80a1fe0 msgr2=0x7f22d809fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 shutdown_connections 2024-09-18T10:17:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:28.186+0000 7f22d7fff700 1 -- 172.21.15.142:0/2952596390 wait complete. 2024-09-18T10:17:29.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:29.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:29.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:29.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:30.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:30.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:30.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:30.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:31.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:31.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:31.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:31.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:31.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:31.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.314+0000 7f215bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.314+0000 7f215bfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.314+0000 7f215bfff700 10 monclient: monmap: 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:31.319055+0000 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:31.319055+0000 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:31.321 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:31.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:31.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 Processor -- start 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 -- start start 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient: init 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f215c0a2c70 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f215c09de10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f215c071970 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c0a3040 con 0x7f215c071970 2024-09-18T10:17:31.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c071db0 con 0x7f215c0a2c70 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c09a620 con 0x7f215c09de10 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215bfff700 10 monclient(hunting): authenticate will time out at 1340.245133s 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215affd700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 msgr2=0x7f215c070ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44108/0 (socket says 172.21.15.142:44108) 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 learned_addr learned my addr 172.21.15.142:0/118601026 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 10 monclient(hunting): get_auth_request con 0x7f215c09de10 auth_method 0 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:31.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:31.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.318+0000 7f215b7fe700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.142:6789/0 conn(0x7f215c071970 legacy=0x7f215c138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 msgr2=0x7f215c070ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 10 monclient(hunting): get_auth_request con 0x7f215c09de10 auth_method 0 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.522+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 msgr2=0x7f215c070ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 10 monclient(hunting): get_auth_request con 0x7f215c09de10 auth_method 0 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:31.926+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:17:32.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:32.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:32.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:32.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 msgr2=0x7f215c070ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 10 monclient(hunting): get_auth_request con 0x7f215c09de10 auth_method 0 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:32.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:32.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:32.730+0000 7f215a7fc700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:17:33.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:33.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:33.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:33.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:33.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:33.093+0000 7f46c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:34.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:34.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:34.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:34.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:34.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): tick 2024-09-18T10:17:34.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 monclient(hunting): continuing hunt 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 msgr2=0x7f215c071430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c09de10 0x7f215c071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 msgr2=0x7f215c070ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.150:3300/0 conn(0x7f215c0a2c70 0x7f215c070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.142:6789/0 conn(0x7f215c071970 legacy=0x7f215c138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient: _add_conns ranks=[3,1] 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f2148003790 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21480062f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 -- 172.21.15.142:0/118601026 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c09e760 con 0x7f21480062f0 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:34.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 1 -- 172.21.15.142:0/118601026 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c09a620 con 0x7f2148003790 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f2158ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.150:6789/0 conn(0x7f21480062f0 legacy=0x7f21480087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 10 monclient(hunting): get_auth_request con 0x7f2148003790 auth_method 0 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 msgr2=0x7f2148005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 msgr2=0x7f2148005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:34.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.318+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.150:6789/0 conn(0x7f21480062f0 legacy=0x7f21480087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 10 monclient(hunting): get_auth_request con 0x7f2148003790 auth_method 0 2024-09-18T10:17:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 msgr2=0x7f2148005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:34.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 msgr2=0x7f2148005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:34.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:34.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.522+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:17:34.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215a7fc700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.150:6789/0 conn(0x7f21480062f0 legacy=0x7f21480087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:34.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> v2:172.21.15.142:3300/0 conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): get_auth_request con 0x7f2148003790 auth_method 0 2024-09-18T10:17:34.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:34.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient(hunting): handle_auth_done global_id 94164 payload 274 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> v1:172.21.15.150:6789/0 conn(0x7f21480062f0 legacy=0x7f21480087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 monclient: found mon.noname-a 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:34.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214800b050 con 0x7f2148003790 2024-09-18T10:17:34.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 msgr2=0x7f2148005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:34.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215affd700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f2148005c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2150003980 tx=0x7f2150009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911ab4031bd7f240 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:34.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 1 -- 172.21.15.142:0/118601026 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150016430 con 0x7f2148003790 2024-09-18T10:17:34.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:34.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:34.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: dump: 2024-09-18T10:17:34.933 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215bfff700 5 monclient: authenticate success, global_id 94164 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:34.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 1 -- 172.21.15.142:0/118601026 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150016590 con 0x7f2148003790 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f215bfff700 4 set_mon_vals no callback set 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 1 -- 172.21.15.142:0/118601026 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150016910 con 0x7f2148003790 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.926+0000 7f2159ffb700 10 monclient: dump: 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:34.935 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:34.936 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:34.936 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:34.936 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:34.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: shutdown 2024-09-18T10:17:34.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 msgr2=0x7f2148005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f2148005c00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2150003980 tx=0x7f2150009800 comp rx=0 tx=0).stop 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215b7fe700 1 -- 172.21.15.142:0/118601026 reap_dead start 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- 172.21.15.142:0/118601026 shutdown_connections 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 --2- 172.21.15.142:0/118601026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f2148005c00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- 172.21.15.142:0/118601026 >> 172.21.15.142:0/118601026 conn(0x7f215c06a140 msgr2=0x7f215c0a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- 172.21.15.142:0/118601026 shutdown_connections 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- 172.21.15.142:0/118601026 wait complete. 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: build_initial_monmap 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: monmap: 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:34.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 Processor -- start 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- start start 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: init 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:34.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f215c1417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f2148003790 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f215c141d10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 0x7f215c08d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f215c08cfa0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c090c60 con 0x7f2148003790 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c142660 con 0x7f215c08cfa0 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c09e2f0 con 0x7f215c141d10 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215bfff700 10 monclient(hunting): authenticate will time out at 1343.859144s 2024-09-18T10:17:34.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215b7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 msgr2=0x7f215c08d3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f215c1417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33160/0 (socket says 172.21.15.142:33160) 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 0x7f215c08d3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 -- 172.21.15.142:0/2044792846 learned_addr learned my addr 172.21.15.142:0/2044792846 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): get_auth_request con 0x7f215c141d10 auth_method 0 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): get_auth_request con 0x7f2148003790 auth_method 0 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:34.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient(hunting): handle_auth_done global_id 74561 payload 824 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 -- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 msgr2=0x7f215c08d3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 0x7f215c08d3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 -- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 msgr2=0x7f215c1417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f215c1417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 monclient: found mon.noname-c 2024-09-18T10:17:34.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215a7fc700 1 -- 172.21.15.142:0/2044792846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150011040 con 0x7f215c141d10 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.930+0000 7f215affd700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f215c1417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215a7fc700 1 -- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 msgr2=0x7f215c142120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215a7fc700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2154007d90 tx=0x7f2154009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154012cd0 con 0x7f215c141d10 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: dump: 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.946 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215400f040 con 0x7f215c141d10 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 5 monclient: authenticate success, global_id 74561 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 10 monclient: _renew_subs 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c09c240 con 0x7f215c141d10 2024-09-18T10:17:34.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215400b320 con 0x7f215c141d10 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2142ffd700 10 monclient: dump: 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:34.948 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:34.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f2158ff9700 4 set_mon_vals no callback set 2024-09-18T10:17:34.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 10 monclient: _renew_subs 2024-09-18T10:17:34.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:34.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.934+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c091930 con 0x7f215c141d10 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c072a80 con 0x7f215c141d10 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2154009720 con 0x7f215c141d10 2024-09-18T10:17:34.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f2142ffd700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 0x7f2148038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7f215404b3c0 con 0x7f215c141d10 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215affd700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 0x7f2148038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.938+0000 7f215affd700 10 monclient: get_auth_request con 0x7f2148036830 auth_method 0 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.942+0000 7f215affd700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 0x7f2148038ce0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2150009840 tx=0x7f2150015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.942+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f215401d360 con 0x7f215c141d10 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.942+0000 7f2142ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:34.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:34.942+0000 7f2142ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:35.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:35.093+0000 7f46c3fff700 10 monclient: tick 2024-09-18T10:17:35.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:35.093+0000 7f46c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f215bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f215bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f215bfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f215c067960 con 0x7f215c141d10 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f2142ffd700 1 -- 172.21.15.142:0/2044792846 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f215401de80 con 0x7f215c141d10 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f2142ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:35.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.166+0000 7f2142ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:35.173 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 msgr2=0x7f2148038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 0x7f2148038ce0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2150009840 tx=0x7f2150015000 comp rx=0 tx=0).stop 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 10 monclient: shutdown 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 msgr2=0x7f215c142120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2154007d90 tx=0x7f2154009040 comp rx=0 tx=0).stop 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 shutdown_connections 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2148036830 0x7f2148038ce0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f215c08cfa0 0x7f215c08d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f215c141d10 0x7f215c142120 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 --2- 172.21.15.142:0/2044792846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2148003790 0x7f215c1417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 >> 172.21.15.142:0/2044792846 conn(0x7f215c06a140 msgr2=0x7f215c0956b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:35.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 shutdown_connections 2024-09-18T10:17:35.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.170+0000 7f215bfff700 1 -- 172.21.15.142:0/2044792846 wait complete. 2024-09-18T10:17:35.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:17:35.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:17:35.186 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: monmap: 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:35.288772+0000 2024-09-18T10:17:35.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:35.288772+0000 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:35.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 Processor -- start 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 -- start start 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: init 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6a409abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a409b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:35.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6a4134bb0 0x7fa6a4136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa6a4134bb0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a409dc90 con 0x7fa6a409abd0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a409b500 con 0x7fa6a4134bb0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40967f0 con 0x7fa6a409b740 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a3fff700 10 monclient(hunting): authenticate will time out at 1344.214218s 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a2ffd700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fa6a409abd0 legacy=0x7fa6a40960e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6a4134bb0 0x7fa6a4136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33166/0 (socket says 172.21.15.142:33166) 2024-09-18T10:17:35.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 -- 172.21.15.142:0/1823277441 learned_addr learned my addr 172.21.15.142:0/1823277441 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): get_auth_request con 0x7fa6a409b740 auth_method 0 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): get_auth_request con 0x7fa6a4134bb0 auth_method 0 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient(hunting): handle_auth_done global_id 74564 payload 274 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 -- 172.21.15.142:0/1823277441 >> v2:172.21.15.142:3300/0 conn(0x7fa6a4134bb0 msgr2=0x7fa6a4136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 --2- 172.21.15.142:0/1823277441 >> v2:172.21.15.142:3300/0 conn(0x7fa6a4134bb0 0x7fa6a4136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 -- 172.21.15.142:0/1823277441 >> v1:172.21.15.150:6789/0 conn(0x7fa6a409abd0 legacy=0x7fa6a40960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 monclient: found mon.noname-b 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:35.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.286+0000 7fa6a27fc700 1 -- 172.21.15.142:0/1823277441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a409afa0 con 0x7fa6a409b740 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a27fc700 1 -- 172.21.15.142:0/1823277441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 msgr2=0x7fa6a4134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a27fc700 1 --2- 172.21.15.142:0/1823277441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa6a406fd20 tx=0x7fa694009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2791d7bb118db8f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 1 -- 172.21.15.142:0/1823277441 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694009e70 con 0x7fa6a409b740 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:35.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: dump: 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 5 monclient: authenticate success, global_id 74564 2024-09-18T10:17:35.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 1 -- 172.21.15.142:0/1823277441 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69400ba70 con 0x7fa6a409b740 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 4 set_mon_vals no callback set 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 1 -- 172.21.15.142:0/1823277441 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69400be30 con 0x7fa6a409b740 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a1ffb700 10 monclient: dump: 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.301 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: shutdown 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- 172.21.15.142:0/1823277441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 msgr2=0x7fa6a4134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/1823277441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa6a406fd20 tx=0x7fa694009250 comp rx=0 tx=0).stop 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- 172.21.15.142:0/1823277441 shutdown_connections 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/1823277441 >> v2:172.21.15.142:3300/0 conn(0x7fa6a4134bb0 0x7fa6a4136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/1823277441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409b740 0x7fa6a4134670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- 172.21.15.142:0/1823277441 >> 172.21.15.142:0/1823277441 conn(0x7fa6a4070ef0 msgr2=0x7fa6a40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:35.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- 172.21.15.142:0/1823277441 shutdown_connections 2024-09-18T10:17:35.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- 172.21.15.142:0/1823277441 wait complete. 2024-09-18T10:17:35.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:35.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: monmap: 2024-09-18T10:17:35.303 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:35.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 Processor -- start 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- start start 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: init 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:35.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409abd0 0x7fa6a4088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa6a409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a409b740 0x7fa6a40894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a409b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 0x7fa6a4089a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa6a4134bb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a409d240 con 0x7fa6a4134bb0 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a409fa60 con 0x7fa6a409b740 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a408a8b0 con 0x7fa6a409abd0 2024-09-18T10:17:35.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.290+0000 7fa6a3fff700 10 monclient(hunting): authenticate will time out at 1344.219931s 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 0x7fa6a4089a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409abd0 0x7fa6a4088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a27fc700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a409b740 msgr2=0x7fa6a40894f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a27fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a409b740 0x7fa6a40894f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 0x7fa6a4089a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38644/0 (socket says 172.21.15.142:38644) 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 learned_addr learned my addr 172.21.15.142:0/4131030238 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): get_auth_request con 0x7fa6a4134bb0 auth_method 0 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): get_auth_request con 0x7fa6a409abd0 auth_method 0 2024-09-18T10:17:35.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient(hunting): handle_auth_done global_id 94173 payload 824 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409abd0 msgr2=0x7fa6a4088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409abd0 0x7fa6a4088fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a409b740 msgr2=0x7fa6a40894f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a409b740 0x7fa6a40894f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:35.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:35.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:35.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940094f0 con 0x7fa6a4134bb0 2024-09-18T10:17:35.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 msgr2=0x7fa6a4089a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a37fe700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 0x7fa6a4089a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009ef0 tx=0x7fa69c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c00ce60 con 0x7fa6a4134bb0 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: dump: 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: _finish_auth 0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 5 monclient: authenticate success, global_id 94173 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 monclient: _renew_subs 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a408c3e0 con 0x7fa6a4134bb0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 monclient: _renew_subs 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a4089f70 con 0x7fa6a4134bb0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c00f040 con 0x7fa6a4134bb0 2024-09-18T10:17:35.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c014690 con 0x7fa6a4134bb0 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a17fa700 4 set_mon_vals no callback set 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa69affd700 10 monclient: dump: 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 0x7fa688004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.294+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa688007cd0 con 0x7fa688002860 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 0x7fa688004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient: get_auth_request con 0x7fa688002860 auth_method 0 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa69c014830 con 0x7fa6a4134bb0 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_done global_id 94176 payload 824 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa688007e90 con 0x7fa688002860 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa684035a70 0x7fa684037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7fa69c04b060 con 0x7fa6a4134bb0 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a27fc700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa684035a70 0x7fa684037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a27fc700 10 monclient: get_auth_request con 0x7fa684035a70 auth_method 0 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 msgr2=0x7fa688004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa6a2ffd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 0x7fa688004cf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009630 tx=0x7fa68c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c012940 con 0x7fa688002860 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:35.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 10 monclient: dump: 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa68c012aa0 con 0x7fa688002860 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:35.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:35.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 msgr2=0x7fa688004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 0x7fa688004cf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009630 tx=0x7fa68c009120 comp rx=0 tx=0).stop 2024-09-18T10:17:35.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 msgr2=0x7fa688004cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:35.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.298+0000 7fa69affd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688002860 0x7fa688004cf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.302+0000 7fa6a27fc700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa684035a70 0x7fa684037f20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa6a406fd20 tx=0x7fa694017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 0x7fa688007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a3fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6880077a0 con 0x7fa688006cc0 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 0x7fa688007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient: get_auth_request con 0x7fa688006cc0 auth_method 0 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 10 monclient(hunting): handle_auth_done global_id 94179 payload 824 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.310+0000 7fa6a2ffd700 1 -- 172.21.15.142:0/4131030238 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa688007960 con 0x7fa688006cc0 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a2ffd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 msgr2=0x7fa688007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a2ffd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 0x7fa688007140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa68c017000 tx=0x7fa68c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c01cde0 con 0x7fa688006cc0 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 10 monclient: dump: 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.320 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7fa68c012730 con 0x7fa688006cc0 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:35.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 msgr2=0x7fa688007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 0x7fa688007140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa68c017000 tx=0x7fa68c018000 comp rx=0 tx=0).stop 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 msgr2=0x7fa688007140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa69affd700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa688006cc0 0x7fa688007140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:35.322 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 94, 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.323 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.324 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:35.325 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.326 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:35.327 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.329 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:35.331 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.333 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.334 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.335 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:17:35.336 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.336 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.336 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.336 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:35.336 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:35.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa684035a70 msgr2=0x7fa684037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa684035a70 0x7fa684037f20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa6a406fd20 tx=0x7fa694017000 comp rx=0 tx=0).stop 2024-09-18T10:17:35.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 10 monclient: shutdown 2024-09-18T10:17:35.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:35.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 msgr2=0x7fa6a4089a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 --2- 172.21.15.142:0/4131030238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4134bb0 0x7fa6a4089a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009ef0 tx=0x7fa69c00b040 comp rx=0 tx=0).stop 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 reap_dead start 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a37fe700 1 -- 172.21.15.142:0/4131030238 reap_dead start 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 shutdown_connections 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 >> 172.21.15.142:0/4131030238 conn(0x7fa6a4070ef0 msgr2=0x7fa6a40a0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 shutdown_connections 2024-09-18T10:17:35.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.314+0000 7fa6a3fff700 1 -- 172.21.15.142:0/4131030238 wait complete. 2024-09-18T10:17:35.338 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:17:35.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:17:35.338 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: get_monmap_and_config 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: build_initial_monmap 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: monmap: 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:35.445774+0000 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:35.445774+0000 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:35.448 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 1 Processor -- start 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 1 -- start start 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: init 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.442+0000 7feff16c9700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefec0999e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefec09c410 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fefec138920 0x7fefec13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-a con 0x7fefec138920 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0a20c0 con 0x7fefec0999e0 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefec0a1c80 con 0x7fefec09c410 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefec09a000 con 0x7fefec138920 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient(hunting): authenticate will time out at 1344.372208s 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeb7fe700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fefec09c410 legacy=0x7fefec0a0890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fefec138920 0x7fefec13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fefec138920 0x7fefec13ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38664/0 (socket says 172.21.15.142:38664) 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 -- 172.21.15.142:0/3761551365 learned_addr learned my addr 172.21.15.142:0/3761551365 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): get_auth_request con 0x7fefec138920 auth_method 0 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefebfff700 1 --1- 172.21.15.142:0/3761551365 >> v1:172.21.15.142:6789/0 conn(0x7fefec0999e0 0x7fefec0a0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48570/0 (socket says 172.21.15.142:48570) 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient(hunting): handle_auth_done global_id 94182 payload 274 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 -- 172.21.15.142:0/3761551365 >> v1:172.21.15.150:6789/0 conn(0x7fefec09c410 legacy=0x7fefec0a0890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 -- 172.21.15.142:0/3761551365 >> v1:172.21.15.142:6789/0 conn(0x7fefec0999e0 legacy=0x7fefec0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient: _finish_hunting 0 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 monclient: found mon.noname-a 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 -- 172.21.15.142:0/3761551365 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec09a580 con 0x7fefec138920 2024-09-18T10:17:35.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 -- 172.21.15.142:0/3761551365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec138920 msgr2=0x7fefec13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff0ec8700 1 --2- 172.21.15.142:0/3761551365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec138920 0x7fefec13ad00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f20 tx=0x7fefe4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd5659ea6b39585 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 1 -- 172.21.15.142:0/3761551365 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe400e070 con 0x7fefec138920 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: dump: 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.456 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 1 -- 172.21.15.142:0/3761551365 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe400bc90 con 0x7fefec138920 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 5 monclient: authenticate success, global_id 94182 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7feff16c9700 4 set_mon_vals no callback set 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 1 -- 172.21.15.142:0/3761551365 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4012660 con 0x7fefec138920 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:35.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.446+0000 7fefeaffd700 10 monclient: dump: 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 10 monclient: shutdown 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- 172.21.15.142:0/3761551365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec138920 msgr2=0x7fefec13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 --2- 172.21.15.142:0/3761551365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec138920 0x7fefec13ad00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f20 tx=0x7fefe4009840 comp rx=0 tx=0).stop 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- 172.21.15.142:0/3761551365 shutdown_connections 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 --2- 172.21.15.142:0/3761551365 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec138920 0x7fefec13ad00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- 172.21.15.142:0/3761551365 >> 172.21.15.142:0/3761551365 conn(0x7fefec0a5790 msgr2=0x7fefec0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- 172.21.15.142:0/3761551365 shutdown_connections 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- 172.21.15.142:0/3761551365 wait complete. 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 10 monclient: build_initial_monmap 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 10 monclient: monmap: 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:35.459 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.460 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:35.460 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:35.460 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:35.460 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 Processor -- start 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 1 -- start start 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.450+0000 7feff16c9700 10 monclient: init 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 0x7fefec06f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-a con 0x7fefec0999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec09c410 0x7fefec06b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-c con 0x7fefec09c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefec138920 0x7fefec06b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): picked mon.noname-b con 0x7fefec138920 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec06f8e0 con 0x7fefec0999e0 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec06e9c0 con 0x7fefec138920 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec06cd00 con 0x7fefec09c410 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff16c9700 10 monclient(hunting): authenticate will time out at 1344.380613s 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 0x7fefec06f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec09c410 0x7fefec06b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff0ec8700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefec138920 msgr2=0x7fefec06b740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 0x7fefec06f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38672/0 (socket says 172.21.15.142:38672) 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7feff0ec8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefec138920 0x7fefec06b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:35.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 -- 172.21.15.142:0/2274506144 learned_addr learned my addr 172.21.15.142:0/2274506144 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): get_auth_request con 0x7fefec0999e0 auth_method 0 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): get_auth_request con 0x7fefec09c410 auth_method 0 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient(hunting): handle_auth_done global_id 94185 payload 824 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec09c410 msgr2=0x7fefec06b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec09c410 0x7fefec06b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefec138920 msgr2=0x7fefec06b740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefec138920 0x7fefec06b740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 monclient: found mon.noname-a 2024-09-18T10:17:35.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4009ae0 con 0x7fefec0999e0 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 msgr2=0x7fefec06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefebfff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 0x7fefec06f3a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c5e0 tx=0x7fefdc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.454+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc005c80 con 0x7fefec0999e0 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: dump: 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.469 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: _finish_auth 0 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 5 monclient: authenticate success, global_id 94185 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 monclient: _renew_subs 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec06e740 con 0x7fefec0999e0 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefdc005e00 con 0x7fefec0999e0 2024-09-18T10:17:35.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefdc007320 con 0x7fefec0999e0 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 10 monclient: dump: 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.471 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefea7fc700 4 set_mon_vals no callback set 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 monclient: _renew_subs 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec1325a0 con 0x7fefec0999e0 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 0x7fefec132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec06daa0 con 0x7fefec06bc80 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fefdc01b3b0 con 0x7fefec0999e0 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 0x7fefec132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient: get_auth_request con 0x7fefec06bc80 auth_method 0 2024-09-18T10:17:35.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_done global_id 74573 payload 824 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefeb7fe700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefec06d090 con 0x7fefec06bc80 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fefd8035aa0 0x7fefd8037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(231..231 src has 176..231) v4 ==== 13166+0+0 (secure 0 0 0) 0x7fefdc049a20 con 0x7fefec0999e0 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff0ec8700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fefd8035aa0 0x7fefd8037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.458+0000 7feff0ec8700 10 monclient: get_auth_request con 0x7fefd8035aa0 auth_method 0 2024-09-18T10:17:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefeb7fe700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 msgr2=0x7fefec132af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefeb7fe700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 0x7fefec132af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009650 tx=0x7fefe00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0012940 con 0x7fefec06bc80 2024-09-18T10:17:35.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 10 monclient: dump: 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.475 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fefe0012aa0 con 0x7fefec06bc80 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 msgr2=0x7fefec132af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 0x7fefec132af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009650 tx=0x7fefe00091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 msgr2=0x7fefec132af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7fefd3fff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06bc80 0x7fefec132af0 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009650 tx=0x7fefe00091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.462+0000 7feff0ec8700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fefd8035aa0 0x7fefd8037f50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f20 tx=0x7fefe4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7feff16c9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7feff16c9700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 0x7fefec133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7feff16c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec0647f0 con 0x7fefec06c050 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 0x7fefec133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient: get_auth_request con 0x7fefec06c050 auth_method 0 2024-09-18T10:17:35.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 10 monclient(hunting): handle_auth_done global_id 74576 payload 824 2024-09-18T10:17:35.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 1 -- 172.21.15.142:0/2274506144 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fefec072a80 con 0x7fefec06c050 2024-09-18T10:17:35.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 msgr2=0x7fefec133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:35.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.474+0000 7fefeb7fe700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 0x7fefec133030 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fefe00093d0 tx=0x7fefe0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0021990 con 0x7fefec06c050 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 10 monclient: dump: 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:35.482 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7fefe0021af0 con 0x7fefec06c050 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 msgr2=0x7fefec133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 0x7fefec133030 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fefe00093d0 tx=0x7fefe0015000 comp rx=0 tx=0).stop 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 msgr2=0x7fefec133030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7fefd3fff700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefec06c050 0x7fefec133030 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:17:35.483 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 94, 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.484 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.485 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:17:35.486 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:17:35.487 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.488 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:17:35.491 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:17:35.492 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.494 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.495 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:17:35.496 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:17:35.497 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fefd8035aa0 msgr2=0x7fefd8037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fefd8035aa0 0x7fefd8037f50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009f20 tx=0x7fefe4017000 comp rx=0 tx=0).stop 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 10 monclient: shutdown 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 msgr2=0x7fefec06f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:35.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 --2- 172.21.15.142:0/2274506144 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefec0999e0 0x7fefec06f3a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c5e0 tx=0x7fefdc007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff0ec8700 1 -- 172.21.15.142:0/2274506144 reap_dead start 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff0ec8700 1 -- 172.21.15.142:0/2274506144 reap_dead start 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 shutdown_connections 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 >> 172.21.15.142:0/2274506144 conn(0x7fefec0a5790 msgr2=0x7fefec137e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 shutdown_connections 2024-09-18T10:17:35.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:35.478+0000 7feff16c9700 1 -- 172.21.15.142:0/2274506144 wait complete. 2024-09-18T10:17:35.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:17:35.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:17:35.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:35.893+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 5 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 removed item id -46 name 'h10' from crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7f46d002a050 con 0x7f46d8098ac0 2024-09-18T10:17:35.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:35.893+0000 7f46c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:35.901 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:35.893+0000 7f46c97fa700 10 monclient: _finish_command 2 = system:0 removed item id -46 name 'h10' from crush map 2024-09-18T10:17:36.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.001+0000 7f46df576700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:36.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.001+0000 7f46df576700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:36.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.001+0000 7f46df576700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.009 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.001+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h10"} v 0) v1 -- 0x7f46d8068c60 con 0x7f46d8098ac0 2024-09-18T10:17:36.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46c97fa700 1 -- 172.21.15.150:0/2169780960 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h10"}]=0 device 'h10' does not appear in the crush map v232) v1 ==== 120+0+0 (secure 0 0 0) 0x7f46d0032020 con 0x7f46d8098ac0 2024-09-18T10:17:36.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46c97fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h10"}] 2024-09-18T10:17:36.010 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46c97fa700 10 monclient: _finish_command 3 = system:0 device 'h10' does not appear in the crush map 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 msgr2=0x7f46bc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 0x7f46bc037f70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009d80 tx=0x7f46d4008040 comp rx=0 tx=0).stop 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 10 monclient: shutdown 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 msgr2=0x7f46bc04bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46bc04bde0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009340 tx=0x7f46d0013000 comp rx=0 tx=0).stop 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 shutdown_connections 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46bc04c7b0 0x7f46bc04ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46d8098ac0 0x7f46bc04bde0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.013 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46d809d970 0x7f46bc049520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 --2- 172.21.15.150:0/2169780960 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46bc035ac0 0x7f46bc037f70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 >> 172.21.15.150:0/2169780960 conn(0x7f46d80a15e0 msgr2=0x7f46d809f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:36.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 shutdown_connections 2024-09-18T10:17:36.014 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.005+0000 7f46df576700 1 -- 172.21.15.150:0/2169780960 wait complete. 2024-09-18T10:17:36.014 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h10' does not appear in the crush map 2024-09-18T10:17:36.028 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:36.029 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h11 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 10 monclient: get_monmap_and_config 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 10 monclient: build_initial_monmap 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 10 monclient: monmap: 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:36.122769+0000 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:36.122769+0000 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.124 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:36.125 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:36.125 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:36.125 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:36.125 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:36.125 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 1 Processor -- start 2024-09-18T10:17:36.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 1 -- start start 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.117+0000 7f6691006700 10 monclient: init 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f668c09cf90 0x7f668c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): picked mon.noname-a con 0x7f668c09cf90 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f668c09e480 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 0x7f668c139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): picked mon.noname-c con 0x7f668c0990a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c09e060 con 0x7f668c09e480 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c0a66c0 con 0x7f668c09cf90 2024-09-18T10:17:36.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c09d750 con 0x7f668c0990a0 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f6691006700 10 monclient(hunting): authenticate will time out at 1338.216251s 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668b59e700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 msgr2=0x7f668c139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 0x7f668c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f668c09cf90 0x7f668c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668a59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f668c09e480 0x7f668c0a07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:42050/0 (socket says 172.21.15.150:42050) 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668a59c700 1 -- 172.21.15.150:0/3454558752 learned_addr learned my addr 172.21.15.150:0/3454558752 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): get_auth_request con 0x7f668c09cf90 auth_method 0 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:36.129 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient(hunting): handle_auth_done global_id 94188 payload 274 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 -- 172.21.15.150:0/3454558752 >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 msgr2=0x7f668c139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 --2- 172.21.15.150:0/3454558752 >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 0x7f668c139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 -- 172.21.15.150:0/3454558752 >> v1:172.21.15.142:6789/0 conn(0x7f668c09e480 legacy=0x7f668c0a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 monclient: found mon.noname-a 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.130 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.121+0000 7f668ad9d700 1 -- 172.21.15.150:0/3454558752 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c09d360 con 0x7f668c09cf90 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668ad9d700 1 -- 172.21.15.150:0/3454558752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09cf90 msgr2=0x7f668c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668ad9d700 1 --2- 172.21.15.150:0/3454558752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09cf90 0x7f668c0a0290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f668c06fe50 tx=0x7f66800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e7a61486f3e23e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 1 -- 172.21.15.150:0/3454558752 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668000b9a0 con 0x7f668c09cf90 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: dump: 2024-09-18T10:17:36.134 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: _finish_auth 0 2024-09-18T10:17:36.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 5 monclient: authenticate success, global_id 94188 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 1 -- 172.21.15.150:0/3454558752 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668000bb20 con 0x7f668c09cf90 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 4 set_mon_vals no callback set 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 1 -- 172.21.15.150:0/3454558752 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680011420 con 0x7f668c09cf90 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:36.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6689d9b700 10 monclient: dump: 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: shutdown 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:36.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- 172.21.15.150:0/3454558752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09cf90 msgr2=0x7f668c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- 172.21.15.150:0/3454558752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09cf90 0x7f668c0a0290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f668c06fe50 tx=0x7f66800091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- 172.21.15.150:0/3454558752 shutdown_connections 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- 172.21.15.150:0/3454558752 >> v2:172.21.15.150:3300/0 conn(0x7f668c0990a0 0x7f668c139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- 172.21.15.150:0/3454558752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09cf90 0x7f668c0a0290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- 172.21.15.150:0/3454558752 >> 172.21.15.150:0/3454558752 conn(0x7f668c0a58a0 msgr2=0x7f668c0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- 172.21.15.150:0/3454558752 shutdown_connections 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- 172.21.15.150:0/3454558752 wait complete. 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: build_initial_monmap 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: monmap: 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:36.138 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 Processor -- start 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- start start 2024-09-18T10:17:36.139 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: init 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 0x7f668c08d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): picked mon.noname-b con 0x7f668c0990a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 0x7f668c08d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): picked mon.noname-c con 0x7f668c09cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): picked mon.noname-a con 0x7f668c09e480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c0a4280 con 0x7f668c09e480 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c090c00 con 0x7f668c0990a0 2024-09-18T10:17:36.140 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c090890 con 0x7f668c09cf90 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f6691006700 10 monclient(hunting): authenticate will time out at 1338.223637s 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668ad9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 msgr2=0x7f668c08d3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668ad9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 0x7f668c08d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 0x7f668c08d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.125+0000 7f668b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:49254/0 (socket says 172.21.15.150:49254) 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 -- 172.21.15.150:0/1100697167 learned_addr learned my addr 172.21.15.150:0/1100697167 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): get_auth_request con 0x7f668c09e480 auth_method 0 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 0x7f668c08d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:47848/0 (socket says 172.21.15.150:47848) 2024-09-18T10:17:36.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): get_auth_request con 0x7f668c09cf90 auth_method 0 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient(hunting): handle_auth_done global_id 94191 payload 824 2024-09-18T10:17:36.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 -- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 msgr2=0x7f668c08d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 0x7f668c08d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 -- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 msgr2=0x7f668c08d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 0x7f668c08d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient: _finish_hunting 0 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 monclient: found mon.noname-a 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680009420 con 0x7f668c09e480 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 -- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 msgr2=0x7f668c08de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f668b59e700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f668400beb0 tx=0x7f66840096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684006360 con 0x7f668c09e480 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: dump: 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:36.145 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 5 monclient: authenticate success, global_id 94191 2024-09-18T10:17:36.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 10 monclient: _renew_subs 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c08f650 con 0x7f668c09e480 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66840064c0 con 0x7f668c09e480 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684006630 con 0x7f668c09e480 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6688d99700 4 set_mon_vals no callback set 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6672ffd700 10 monclient: dump: 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:36.147 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 10 monclient: _renew_subs 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.129+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c091300 con 0x7f668c09e480 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6691006700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6691006700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6691006700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c0a0ee0 con 0x7f668c09e480 2024-09-18T10:17:36.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6684012020 con 0x7f668c09e480 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6672ffd700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 0x7f6678037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(232..232 src has 176..232) v4 ==== 12953+0+0 (secure 0 0 0) 0x7f668404a780 con 0x7f668c09e480 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f668ad9d700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 0x7f6678037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.133+0000 7f668ad9d700 10 monclient: get_auth_request con 0x7f6678035a70 auth_method 0 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.137+0000 7f668ad9d700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 0x7f6678037f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f668c06fe50 tx=0x7f6680016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.141+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6684013a90 con 0x7f668c09e480 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.141+0000 7f6672ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:36.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.141+0000 7f6672ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:36.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.313+0000 7f6691006700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:36.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.313+0000 7f6691006700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:36.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.313+0000 7f6691006700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:36.321 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.317+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f668c063ed0 con 0x7f668c09e480 2024-09-18T10:17:36.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.905+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 removed item id -49 name 'h11' from crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6684013a90 con 0x7f668c09e480 2024-09-18T10:17:36.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.905+0000 7f6672ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:36.909 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:36.905+0000 7f6672ffd700 10 monclient: _finish_command 2 = system:0 removed item id -49 name 'h11' from crush map 2024-09-18T10:17:37.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.009+0000 7f6691006700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:37.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.009+0000 7f6691006700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:37.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.009+0000 7f6691006700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.009+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h11"} v 0) v1 -- 0x7f668c0008d0 con 0x7f668c09e480 2024-09-18T10:17:37.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6672ffd700 1 -- 172.21.15.150:0/1100697167 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h11"}]=0 device 'h11' does not appear in the crush map v233) v1 ==== 120+0+0 (secure 0 0 0) 0x7f6684013a90 con 0x7f668c09e480 2024-09-18T10:17:37.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6672ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h11"}] 2024-09-18T10:17:37.018 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6672ffd700 10 monclient: _finish_command 3 = system:0 device 'h11' does not appear in the crush map 2024-09-18T10:17:37.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 msgr2=0x7f6678037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:37.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 0x7f6678037f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f668c06fe50 tx=0x7f6680016000 comp rx=0 tx=0).stop 2024-09-18T10:17:37.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 10 monclient: shutdown 2024-09-18T10:17:37.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:37.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 msgr2=0x7f668c08de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f668400beb0 tx=0x7f66840096b0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 shutdown_connections 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6678035a70 0x7f6678037f20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f668c09e480 0x7f668c08de60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f668c09cf90 0x7f668c08d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 --2- 172.21.15.150:0/1100697167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f668c0990a0 0x7f668c08d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 >> 172.21.15.150:0/1100697167 conn(0x7f668c0a58a0 msgr2=0x7f668c0a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 shutdown_connections 2024-09-18T10:17:37.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.013+0000 7f6691006700 1 -- 172.21.15.150:0/1100697167 wait complete. 2024-09-18T10:17:37.023 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h11' does not appear in the crush map 2024-09-18T10:17:37.031 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:37.031 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h12 2024-09-18T10:17:37.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: get_monmap_and_config 2024-09-18T10:17:37.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: build_initial_monmap 2024-09-18T10:17:37.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: monmap: 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:37.133062+0000 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:37.133062+0000 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:37.135 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 Processor -- start 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 -- start start 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: init 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1b8009cf90 0x7f1b800a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-a con 0x7f1b8009cf90 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:37.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 0x7f1b800a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-c con 0x7f1b8009e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-b con 0x7f1b800990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b800a66c0 con 0x7f1b8009cf90 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8009e060 con 0x7f1b8009e480 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.143 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8009d750 con 0x7f1b800990a0 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b85278700 10 monclient(hunting): authenticate will time out at 1339.226740s 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7e7fc700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 msgr2=0x7f1b800a07d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7e7fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 0x7f1b800a07d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1b8009cf90 0x7f1b800a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1b8009cf90 0x7f1b800a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:49264/0 (socket says 172.21.15.150:49264) 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:47864/0 (socket says 172.21.15.150:47864) 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 -- 172.21.15.150:0/2226420580 learned_addr learned my addr 172.21.15.150:0/2226420580 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): get_auth_request con 0x7f1b8009cf90 auth_method 0 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:37.144 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): get_auth_request con 0x7f1b800990a0 auth_method 0 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:37.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient(hunting): handle_auth_done global_id 94194 payload 274 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 -- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 msgr2=0x7f1b800a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 -- 172.21.15.150:0/2226420580 >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 msgr2=0x7f1b800a07d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:37.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 --2- 172.21.15.150:0/2226420580 >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 0x7f1b800a07d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 monclient: found mon.noname-a 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7effd700 1 -- 172.21.15.150:0/2226420580 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8009d360 con 0x7f1b8009cf90 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.129+0000 7f1b7f7fe700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7effd700 1 -- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009cf90 msgr2=0x7f1b800a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7effd700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009cf90 0x7f1b800a0290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b800a1820 tx=0x7f1b74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c586c7e914167b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 1 -- 172.21.15.150:0/2226420580 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74009e70 con 0x7f1b8009cf90 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:37.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: dump: 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:37.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 5 monclient: authenticate success, global_id 94194 2024-09-18T10:17:37.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 1 -- 172.21.15.150:0/2226420580 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400ba70 con 0x7f1b8009cf90 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 1 -- 172.21.15.150:0/2226420580 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400be30 con 0x7f1b8009cf90 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 4 set_mon_vals no callback set 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:37.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b7dffb700 10 monclient: dump: 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:37.151 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 10 monclient: shutdown 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:37.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 -- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009cf90 msgr2=0x7f1b800a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009cf90 0x7f1b800a0290 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1b800a1820 tx=0x7f1b74009250 comp rx=0 tx=0).stop 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 -- 172.21.15.150:0/2226420580 shutdown_connections 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b800990a0 0x7f1b800a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 --2- 172.21.15.150:0/2226420580 >> v2:172.21.15.150:3300/0 conn(0x7f1b8009e480 0x7f1b800a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 --2- 172.21.15.150:0/2226420580 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009cf90 0x7f1b800a0290 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 -- 172.21.15.150:0/2226420580 >> 172.21.15.150:0/2226420580 conn(0x7f1b800a58a0 msgr2=0x7f1b800a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 -- 172.21.15.150:0/2226420580 shutdown_connections 2024-09-18T10:17:37.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.133+0000 7f1b85278700 1 -- 172.21.15.150:0/2226420580 wait complete. 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient: build_initial_monmap 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient: monmap: 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:37.154 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 Processor -- start 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 -- start start 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient: init 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 0x7f1b800980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-b con 0x7f1b800990a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 0x7f1b80137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-c con 0x7f1b8009cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 0x7f1b80137f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): picked mon.noname-a con 0x7f1b8009e480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8008e6a0 con 0x7f1b8009e480 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80090420 con 0x7f1b800990a0 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8008eda0 con 0x7f1b8009cf90 2024-09-18T10:17:37.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b85278700 10 monclient(hunting): authenticate will time out at 1339.234310s 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7effd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 msgr2=0x7f1b800980b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 0x7f1b800980b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 0x7f1b80137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 0x7f1b80137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 0x7f1b80137a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:47874/0 (socket says 172.21.15.150:47874) 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 1 -- 172.21.15.150:0/3061746439 learned_addr learned my addr 172.21.15.150:0/3061746439 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): get_auth_request con 0x7f1b8009cf90 auth_method 0 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:37.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): get_auth_request con 0x7f1b8009e480 auth_method 0 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient(hunting): handle_auth_done global_id 94197 payload 824 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 -- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 msgr2=0x7f1b80137a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 0x7f1b80137a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 -- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 msgr2=0x7f1b800980b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:37.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 0x7f1b800980b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.137+0000 7f1b7f7fe700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78009420 con 0x7f1b8009e480 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b7f7fe700 1 -- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 msgr2=0x7f1b80137f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b7f7fe700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 0x7f1b80137f80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800ea80 tx=0x7f1b780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78009670 con 0x7f1b8009e480 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: dump: 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:37.164 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7800ca60 con 0x7f1b8009e480 2024-09-18T10:17:37.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 5 monclient: authenticate success, global_id 94197 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 10 monclient: _renew_subs 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b740094f0 con 0x7f1b8009e480 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800ce20 con 0x7f1b8009e480 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b7d7fa700 4 set_mon_vals no callback set 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b66ffd700 10 monclient: dump: 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:37.166 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 10 monclient: _renew_subs 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.141+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b80090f70 con 0x7f1b8009e480 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b85278700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:37.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b85278700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b85278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b80063ed0 con 0x7f1b8009e480 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1b78013680 con 0x7f1b8009e480 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b66ffd700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 0x7f1b6c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(233..233 src has 176..233) v4 ==== 12740+0+0 (secure 0 0 0) 0x7f1b7804a2f0 con 0x7f1b8009e480 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b7effd700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 0x7f1b6c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.145+0000 7f1b7effd700 10 monclient: get_auth_request con 0x7f1b6c035a70 auth_method 0 2024-09-18T10:17:37.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.149+0000 7f1b7effd700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 0x7f1b6c037f20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1b800a1820 tx=0x7f1b74017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.153+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1b7801a720 con 0x7f1b8009e480 2024-09-18T10:17:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.153+0000 7f1b66ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:37.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.153+0000 7f1b66ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:37.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.325+0000 7f1b85278700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:37.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.325+0000 7f1b85278700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:37.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.325+0000 7f1b85278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:37.333 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.325+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f1b80067b10 con 0x7f1b8009e480 2024-09-18T10:17:37.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.913+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 removed item id -52 name 'h12' from crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1b7801a360 con 0x7f1b8009e480 2024-09-18T10:17:37.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.913+0000 7f1b66ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:37.919 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:37.913+0000 7f1b66ffd700 10 monclient: _finish_command 2 = system:0 removed item id -52 name 'h12' from crush map 2024-09-18T10:17:38.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.021+0000 7f1b85278700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:38.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.021+0000 7f1b85278700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:38.028 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.021+0000 7f1b85278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.021+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h12"} v 0) v1 -- 0x7f1b8008f1b0 con 0x7f1b8009e480 2024-09-18T10:17:38.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b66ffd700 1 -- 172.21.15.150:0/3061746439 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h12"}]=0 device 'h12' does not appear in the crush map v234) v1 ==== 120+0+0 (secure 0 0 0) 0x7f1b7801a540 con 0x7f1b8009e480 2024-09-18T10:17:38.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b66ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h12"}] 2024-09-18T10:17:38.029 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b66ffd700 10 monclient: _finish_command 3 = system:0 device 'h12' does not appear in the crush map 2024-09-18T10:17:38.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 msgr2=0x7f1b6c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:38.032 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 0x7f1b6c037f20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1b800a1820 tx=0x7f1b74017000 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 10 monclient: shutdown 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 msgr2=0x7f1b80137f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 0x7f1b80137f80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800ea80 tx=0x7f1b780091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 shutdown_connections 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1b6c035a70 0x7f1b6c037f20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1b8009e480 0x7f1b80137f80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1b8009cf90 0x7f1b80137a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 --2- 172.21.15.150:0/3061746439 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b800990a0 0x7f1b800980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 >> 172.21.15.150:0/3061746439 conn(0x7f1b800a58a0 msgr2=0x7f1b800a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 shutdown_connections 2024-09-18T10:17:38.033 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.025+0000 7f1b85278700 1 -- 172.21.15.150:0/3061746439 wait complete. 2024-09-18T10:17:38.034 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h12' does not appear in the crush map 2024-09-18T10:17:38.044 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:38.044 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h13 2024-09-18T10:17:38.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: get_monmap_and_config 2024-09-18T10:17:38.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: build_initial_monmap 2024-09-18T10:17:38.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: monmap: 2024-09-18T10:17:38.142 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:38.142 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:38.141031+0000 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:38.141031+0000 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:38.143 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:38.144 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 Processor -- start 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 -- start start 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: init 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:17:38.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 0x7f31380a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): picked mon.noname-c con 0x7f31380a4270 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): picked mon.noname-b con 0x7f313806e610 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3138132260 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313806afe0 con 0x7f3138132260 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380a2220 con 0x7f31380a4270 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380a2830 con 0x7f313806e610 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313f278700 10 monclient(hunting): authenticate will time out at 1340.234509s 2024-09-18T10:17:38.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313d014700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 msgr2=0x7f31380a1ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313d014700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 0x7f31380a1ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313d815700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f3138132260 legacy=0x7f31381326c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:47884/0 (socket says 172.21.15.150:47884) 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 -- 172.21.15.150:0/497541372 learned_addr learned my addr 172.21.15.150:0/497541372 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): get_auth_request con 0x7f313806e610 auth_method 0 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:38.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient(hunting): handle_auth_done global_id 74588 payload 274 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 -- 172.21.15.150:0/497541372 >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 msgr2=0x7f31380a1ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 --2- 172.21.15.150:0/497541372 >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 0x7f31380a1ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 -- 172.21.15.150:0/497541372 >> v1:172.21.15.150:6789/0 conn(0x7f3138132260 legacy=0x7f31381326c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient: _finish_hunting 0 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 monclient: found mon.noname-b 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:38.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.137+0000 7f313c813700 1 -- 172.21.15.150:0/497541372 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380a4640 con 0x7f313806e610 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313c813700 1 -- 172.21.15.150:0/497541372 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 msgr2=0x7f313806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313c813700 1 --2- 172.21.15.150:0/497541372 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f31340099b0 tx=0x7f3134009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40bde5d1e3baa59 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 1 -- 172.21.15.150:0/497541372 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313400bb50 con 0x7f313806e610 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: dump: 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:38.156 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: _finish_auth 0 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 5 monclient: authenticate success, global_id 74588 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:38.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 1 -- 172.21.15.150:0/497541372 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313400bcb0 con 0x7f313806e610 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 4 set_mon_vals no callback set 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 1 -- 172.21.15.150:0/497541372 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134011650 con 0x7f313806e610 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f312ffff700 10 monclient: dump: 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:38.158 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 10 monclient: shutdown 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 -- 172.21.15.150:0/497541372 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 msgr2=0x7f313806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 --2- 172.21.15.150:0/497541372 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f31340099b0 tx=0x7f3134009250 comp rx=0 tx=0).stop 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 -- 172.21.15.150:0/497541372 shutdown_connections 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 --2- 172.21.15.150:0/497541372 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f313806e610 0x7f313806e9e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 --2- 172.21.15.150:0/497541372 >> v2:172.21.15.150:3300/0 conn(0x7f31380a4270 0x7f31380a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 -- 172.21.15.150:0/497541372 >> 172.21.15.150:0/497541372 conn(0x7f31380716e0 msgr2=0x7f31380a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 -- 172.21.15.150:0/497541372 shutdown_connections 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.141+0000 7f313f278700 1 -- 172.21.15.150:0/497541372 wait complete. 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient: build_initial_monmap 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient: monmap: 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:38.160 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 Processor -- start 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 -- start start 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient: init 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): picked mon.noname-a con 0x7f313806e610 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 0x7f313811ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): picked mon.noname-b con 0x7f31380a4270 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3138132260 0x7f313811b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): picked mon.noname-c con 0x7f3138132260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313809e660 con 0x7f313806e610 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138142800 con 0x7f31380a4270 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313806f160 con 0x7f3138132260 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313f278700 10 monclient(hunting): authenticate will time out at 1340.241730s 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313c813700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 msgr2=0x7f313811ad00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313c813700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 0x7f313811ad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:38.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:49280/0 (socket says 172.21.15.150:49280) 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 -- 172.21.15.150:0/2543067834 learned_addr learned my addr 172.21.15.150:0/2543067834 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): get_auth_request con 0x7f313806e610 auth_method 0 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3138132260 0x7f313811b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): get_auth_request con 0x7f3138132260 auth_method 0 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:38.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d815700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient(hunting): handle_auth_done global_id 94200 payload 824 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 -- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3138132260 msgr2=0x7f313811b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3138132260 0x7f313811b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 -- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 msgr2=0x7f313811ad00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 0x7f313811ad00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient: _finish_hunting 0 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 monclient: found mon.noname-a 2024-09-18T10:17:38.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.145+0000 7f313d014700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31340094f0 con 0x7f313806e610 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313d014700 1 -- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 msgr2=0x7f31381311c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313d014700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f312800b810 tx=0x7f3128009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313d014700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f312800b810 tx=0x7f3128009220 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f3128006150 config(0 keys) v1 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128003e20 con 0x7f313806e610 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 10 monclient: dump: 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.165 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3128006150 con 0x7f313806e610 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 5 monclient: authenticate success, global_id 94200 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _renew_subs 2024-09-18T10:17:38.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313811cd30 con 0x7f313806e610 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f312f7fe700 4 set_mon_vals no callback set 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _renew_subs 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313811e8a0 con 0x7f313806e610 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.149+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:38.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128006510 con 0x7f313806e610 2024-09-18T10:17:38.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:38.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:38.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 10 monclient: dump: 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:38.169 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f312801b3b0 con 0x7f313806e610 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f312d7fa700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 0x7f311c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f313c813700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 0x7f311c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f313c813700 10 monclient: get_auth_request con 0x7f311c035ac0 auth_method 0 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.153+0000 7f313c813700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 0x7f311c037f70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f31340099b0 tx=0x7f3134016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:38.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.161+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(234..234 src has 176..234) v4 ==== 12527+0+0 (secure 0 0 0) 0x7f3128049840 con 0x7f313806e610 2024-09-18T10:17:38.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.773+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:38.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.773+0000 7f312d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:38.779 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.773+0000 7f312d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:38.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.945+0000 7f313f278700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:38.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.945+0000 7f313f278700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:38.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.945+0000 7f313f278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:38.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:38.945+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:39.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:39.925+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 removed item id -55 name 'h13' from crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:39.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:39.925+0000 7f312d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:39.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:39.925+0000 7f312d7fa700 10 monclient: _finish_command 2 = system:0 removed item id -55 name 'h13' from crush map 2024-09-18T10:17:40.042 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f313f278700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:40.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f313f278700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:40.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f313f278700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.043 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h13"} v 0) v1 -- 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f312d7fa700 1 -- 172.21.15.150:0/2543067834 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h13"}]=0 device 'h13' does not appear in the crush map v235) v1 ==== 120+0+0 (secure 0 0 0) 0x7f313811d4b0 con 0x7f313806e610 2024-09-18T10:17:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f312d7fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h13"}] 2024-09-18T10:17:40.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.037+0000 7f312d7fa700 10 monclient: _finish_command 3 = system:0 device 'h13' does not appear in the crush map 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 msgr2=0x7f311c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 0x7f311c037f70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f31340099b0 tx=0x7f3134016000 comp rx=0 tx=0).stop 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 10 monclient: shutdown 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 msgr2=0x7f31381311c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:40.047 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f312800b810 tx=0x7f3128009220 comp rx=0 tx=0).stop 2024-09-18T10:17:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 shutdown_connections 2024-09-18T10:17:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f311c035ac0 0x7f311c037f70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3138132260 0x7f313811b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31380a4270 0x7f313811ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.048 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 --2- 172.21.15.150:0/2543067834 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f313806e610 0x7f31381311c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 >> 172.21.15.150:0/2543067834 conn(0x7f31380716e0 msgr2=0x7f313809cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 shutdown_connections 2024-09-18T10:17:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.041+0000 7f313f278700 1 -- 172.21.15.150:0/2543067834 wait complete. 2024-09-18T10:17:40.049 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h13' does not appear in the crush map 2024-09-18T10:17:40.060 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:40.061 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h14 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: get_monmap_and_config 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: build_initial_monmap 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: monmap: 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:40.157470+0000 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:40.157470+0000 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:40.159 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:40.160 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:40.160 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:40.160 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:40.160 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 Processor -- start 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 -- start start 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: init 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f525c09e6d0 0x7f525c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): picked mon.noname-a con 0x7f525c09e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f525c0a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 0x7f525c13a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): picked mon.noname-c con 0x7f525c0a0bd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:40.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c09e220 con 0x7f525c0a07d0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c09ee30 con 0x7f525c09e6d0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c09eaa0 con 0x7f525c0a0bd0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5263118700 10 monclient(hunting): authenticate will time out at 1342.251022s 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f52616b5700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 msgr2=0x7f525c13a140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f52616b5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 0x7f525c13a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f525c09e6d0 0x7f525c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f525c09e6d0 0x7f525c0a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47272/0 (socket says 172.21.15.150:47272) 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 1 -- 172.21.15.150:0/268220001 learned_addr learned my addr 172.21.15.150:0/268220001 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): get_auth_request con 0x7f525c09e6d0 auth_method 0 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:40.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.153+0000 7f5260eb4700 10 monclient(hunting): handle_auth_done global_id 94203 payload 274 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 -- 172.21.15.150:0/268220001 >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 msgr2=0x7f525c13a140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 --2- 172.21.15.150:0/268220001 >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 0x7f525c13a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 -- 172.21.15.150:0/268220001 >> v1:172.21.15.142:6789/0 conn(0x7f525c0a07d0 legacy=0x7f525c136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 10 monclient: _finish_hunting 0 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 monclient: found mon.noname-a 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 -- 172.21.15.150:0/268220001 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c09d930 con 0x7f525c09e6d0 2024-09-18T10:17:40.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 -- 172.21.15.150:0/268220001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 msgr2=0x7f525c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5260eb4700 1 --2- 172.21.15.150:0/268220001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c0a0290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f525c0a6970 tx=0x7f5250009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4459f3564986a591 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 1 -- 172.21.15.150:0/268220001 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000ba70 con 0x7f525c09e6d0 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: dump: 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.167 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 5 monclient: authenticate success, global_id 94203 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 1 -- 172.21.15.150:0/268220001 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525000bbd0 con 0x7f525c09e6d0 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 4 set_mon_vals no callback set 2024-09-18T10:17:40.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 1 -- 172.21.15.150:0/268220001 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250011420 con 0x7f525c09e6d0 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f525b7fe700 10 monclient: dump: 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:40.169 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 10 monclient: shutdown 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 -- 172.21.15.150:0/268220001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 msgr2=0x7f525c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 --2- 172.21.15.150:0/268220001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c0a0290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f525c0a6970 tx=0x7f5250009250 comp rx=0 tx=0).stop 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 -- 172.21.15.150:0/268220001 shutdown_connections 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 --2- 172.21.15.150:0/268220001 >> v2:172.21.15.150:3300/0 conn(0x7f525c0a0bd0 0x7f525c13a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 --2- 172.21.15.150:0/268220001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c0a0290 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 -- 172.21.15.150:0/268220001 >> 172.21.15.150:0/268220001 conn(0x7f525c0a58a0 msgr2=0x7f525c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:40.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 -- 172.21.15.150:0/268220001 shutdown_connections 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.157+0000 7f5263118700 1 -- 172.21.15.150:0/268220001 wait complete. 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient: build_initial_monmap 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient: monmap: 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:40.171 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 Processor -- start 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 -- start start 2024-09-18T10:17:40.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient: init 2024-09-18T10:17:40.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:40.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:40.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): picked mon.noname-a con 0x7f525c09e6d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:40.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 0x7f525c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): picked mon.noname-b con 0x7f525c0a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f525c0a0bd0 0x7f525c08e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): picked mon.noname-c con 0x7f525c0a0bd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c090810 con 0x7f525c09e6d0 2024-09-18T10:17:40.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c091700 con 0x7f525c0a07d0 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c091cc0 con 0x7f525c0a0bd0 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f525bfff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 msgr2=0x7f525c142250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:40.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5263118700 10 monclient(hunting): authenticate will time out at 1342.257853s 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f525bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 0x7f525c142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47282/0 (socket says 172.21.15.150:47282) 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 -- 172.21.15.150:0/3642739202 learned_addr learned my addr 172.21.15.150:0/3642739202 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): get_auth_request con 0x7f525c09e6d0 auth_method 0 2024-09-18T10:17:40.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f525c0a0bd0 0x7f525c08e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): get_auth_request con 0x7f525c0a0bd0 auth_method 0 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:40.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:40.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:40.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:40.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:40.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:40.184 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f52616b5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient(hunting): handle_auth_done global_id 94206 payload 824 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 -- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f525c0a0bd0 msgr2=0x7f525c08e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f525c0a0bd0 0x7f525c08e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.185 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 -- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 msgr2=0x7f525c142250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:40.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 0x7f525c142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:40.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient: _finish_hunting 0 2024-09-18T10:17:40.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 monclient: found mon.noname-a 2024-09-18T10:17:40.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:40.186 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.161+0000 7f5260eb4700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500094f0 con 0x7f525c09e6d0 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5260eb4700 1 -- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 msgr2=0x7f525c141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5260eb4700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250006b60 con 0x7f525c09e6d0 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:40.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: dump: 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.188 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:40.189 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.189 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:40.189 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:40.189 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:40.189 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 5 monclient: authenticate success, global_id 94206 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _renew_subs 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c092920 con 0x7f525c09e6d0 2024-09-18T10:17:40.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250020070 con 0x7f525c09e6d0 2024-09-18T10:17:40.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:40.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525001b660 con 0x7f525c09e6d0 2024-09-18T10:17:40.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _renew_subs 2024-09-18T10:17:40.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c142930 con 0x7f525c09e6d0 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 10 monclient: dump: 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:40.192 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:40.193 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f525affd700 4 set_mon_vals no callback set 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5250022030 con 0x7f525c09e6d0 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:40.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c08fc40 con 0x7f525c09e6d0 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 0x7f5248037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(235..235 src has 176..235) v4 ==== 12314+0+0 (secure 0 0 0) 0x7f525004c100 con 0x7f525c09e6d0 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f525bfff700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 0x7f5248037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.165+0000 7f525bfff700 10 monclient: get_auth_request con 0x7f5248035a70 auth_method 0 2024-09-18T10:17:40.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.169+0000 7f525bfff700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 0x7f5248037f20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f524c007fd0 tx=0x7f524c007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:40.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.177+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f525001ba90 con 0x7f525c09e6d0 2024-09-18T10:17:40.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.177+0000 7f5258ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:40.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.177+0000 7f5258ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:40.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.345+0000 7f5263118700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:40.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.345+0000 7f5263118700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:40.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.345+0000 7f5263118700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:40.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.345+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f525c08fc40 con 0x7f525c09e6d0 2024-09-18T10:17:40.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.925+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 removed item id -58 name 'h14' from crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5250023e60 con 0x7f525c09e6d0 2024-09-18T10:17:40.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.925+0000 7f5258ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:40.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:40.925+0000 7f5258ff9700 10 monclient: _finish_command 2 = system:0 removed item id -58 name 'h14' from crush map 2024-09-18T10:17:41.044 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.037+0000 7f5263118700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.037+0000 7f5263118700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.037+0000 7f5263118700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:41.045 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.037+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h14"} v 0) v1 -- 0x7f525c08fc40 con 0x7f525c09e6d0 2024-09-18T10:17:41.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5258ff9700 1 -- 172.21.15.150:0/3642739202 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h14"}]=0 device 'h14' does not appear in the crush map v236) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5250023380 con 0x7f525c09e6d0 2024-09-18T10:17:41.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5258ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h14"}] 2024-09-18T10:17:41.046 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5258ff9700 10 monclient: _finish_command 3 = system:0 device 'h14' does not appear in the crush map 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 msgr2=0x7f5248037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 0x7f5248037f20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f524c007fd0 tx=0x7f524c007b00 comp rx=0 tx=0).stop 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 10 monclient: shutdown 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 msgr2=0x7f525c141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250016000 comp rx=0 tx=0).stop 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 shutdown_connections 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5248035a70 0x7f5248037f20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f525c0a0bd0 0x7f525c08e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f525c0a07d0 0x7f525c142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 --2- 172.21.15.150:0/3642739202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f525c09e6d0 0x7f525c141d10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 >> 172.21.15.150:0/3642739202 conn(0x7f525c0a58a0 msgr2=0x7f525c09adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 shutdown_connections 2024-09-18T10:17:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.041+0000 7f5263118700 1 -- 172.21.15.150:0/3642739202 wait complete. 2024-09-18T10:17:41.050 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h14' does not appear in the crush map 2024-09-18T10:17:41.059 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:41.059 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h15 2024-09-18T10:17:41.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: get_monmap_and_config 2024-09-18T10:17:41.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: build_initial_monmap 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: monmap: 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:41.157915+0000 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:41.157915+0000 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:41.160 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:41.161 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 Processor -- start 2024-09-18T10:17:41.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 -- start start 2024-09-18T10:17:41.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: init 2024-09-18T10:17:41.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:41.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f24c80a0120 0x7f24c809e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-a con 0x7f24c80a0120 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24c80a04f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24c81379b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c809dae0 con 0x7f24c81379b0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c80a1590 con 0x7f24c80a04f0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80a21d0 con 0x7f24c80a0120 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24d07f1700 10 monclient(hunting): authenticate will time out at 1343.251544s 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24cdd8c700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f24c80a04f0 legacy=0x7f24c80a0950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:41.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.153+0000 7f24ce58d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f24c80a0120 0x7f24c809e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ced8e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f24c81379b0 0x7f24c8139d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35214/0 (socket says 172.21.15.150:35214) 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): get_auth_request con 0x7f24c80a0120 auth_method 0 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ced8e700 1 -- 172.21.15.150:0/762267025 learned_addr learned my addr 172.21.15.150:0/762267025 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient(hunting): handle_auth_done global_id 94209 payload 274 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 -- 172.21.15.150:0/762267025 >> v1:172.21.15.150:6789/0 conn(0x7f24c80a04f0 legacy=0x7f24c80a0950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:41.164 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 -- 172.21.15.150:0/762267025 >> v1:172.21.15.142:6789/0 conn(0x7f24c81379b0 legacy=0x7f24c8139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:41.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:41.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 monclient: found mon.noname-a 2024-09-18T10:17:41.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:41.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:41.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 -- 172.21.15.150:0/762267025 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80a1da0 con 0x7f24c80a0120 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 -- 172.21.15.150:0/762267025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 msgr2=0x7f24c809e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24ce58d700 1 --2- 172.21.15.150:0/762267025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c809e7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0098e0 tx=0x7f24bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b61bdd56e772fc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 1 -- 172.21.15.150:0/762267025 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc00ba70 con 0x7f24c80a0120 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: dump: 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.167 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: _finish_auth 0 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24d07f1700 5 monclient: authenticate success, global_id 94209 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24d07f1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 1 -- 172.21.15.150:0/762267025 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc00bbd0 con 0x7f24c80a0120 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24d07f1700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:41.168 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24d07f1700 4 set_mon_vals no callback set 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 1 -- 172.21.15.150:0/762267025 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc011420 con 0x7f24c80a0120 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.157+0000 7f24cd58b700 10 monclient: dump: 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:41.169 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: shutdown 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- 172.21.15.150:0/762267025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 msgr2=0x7f24c809e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 --2- 172.21.15.150:0/762267025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c809e7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0098e0 tx=0x7f24bc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- 172.21.15.150:0/762267025 shutdown_connections 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 --2- 172.21.15.150:0/762267025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c809e7a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- 172.21.15.150:0/762267025 >> 172.21.15.150:0/762267025 conn(0x7f24c80a5830 msgr2=0x7f24c80a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- 172.21.15.150:0/762267025 shutdown_connections 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- 172.21.15.150:0/762267025 wait complete. 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: build_initial_monmap 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: monmap: 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:41.170 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 Processor -- start 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- start start 2024-09-18T10:17:41.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: init 2024-09-18T10:17:41.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:41.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:41.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c8142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:41.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-a con 0x7f24c80a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:41.174 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 0x7f24c8142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-b con 0x7f24c80a04f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): picked mon.noname-c con 0x7f24c81379b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c812f720 con 0x7f24c80a0120 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8130c70 con 0x7f24c80a04f0 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8130030 con 0x7f24c81379b0 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24d07f1700 10 monclient(hunting): authenticate will time out at 1343.258862s 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24cdd8c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 msgr2=0x7f24c8142590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24cdd8c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 0x7f24c8142590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:41.175 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c8142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46324/0 (socket says 172.21.15.150:46324) 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 -- 172.21.15.150:0/956274336 learned_addr learned my addr 172.21.15.150:0/956274336 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): get_auth_request con 0x7f24c81379b0 auth_method 0 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): get_auth_request con 0x7f24c80a0120 auth_method 0 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:41.176 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ce58d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient(hunting): handle_auth_done global_id 74597 payload 824 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 -- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 msgr2=0x7f24c8142590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 0x7f24c8142590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 -- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 msgr2=0x7f24c8142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c8142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient: _finish_hunting 0 2024-09-18T10:17:41.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 monclient: found mon.noname-c 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.161+0000 7f24ced8e700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc009420 con 0x7f24c81379b0 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.165+0000 7f24ced8e700 1 -- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 msgr2=0x7f24c812c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.165+0000 7f24ced8e700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24c000ccc0 tx=0x7f24c0005400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000ab40 con 0x7f24c81379b0 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: dump: 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:41.178 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c000acc0 con 0x7f24c81379b0 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 5 monclient: authenticate success, global_id 74597 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0007340 con 0x7f24c81379b0 2024-09-18T10:17:41.179 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _renew_subs 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c8130310 con 0x7f24c81379b0 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ccd8a700 4 set_mon_vals no callback set 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 10 monclient: dump: 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:41.180 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _renew_subs 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:41.181 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c8142c70 con 0x7f24c81379b0 2024-09-18T10:17:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:41.182 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f24c00076f0 con 0x7f24c81379b0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c8063ed0 con 0x7f24c81379b0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 0x7f24ac037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(236..236 src has 176..236) v4 ==== 12101+0+0 (secure 0 0 0) 0x7f24c004a4f0 con 0x7f24c81379b0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ce58d700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 0x7f24ac037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.169+0000 7f24ce58d700 10 monclient: get_auth_request con 0x7f24ac035aa0 auth_method 0 2024-09-18T10:17:41.183 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.173+0000 7f24ce58d700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 0x7f24ac037f50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0030c0 tx=0x7f24bc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.181+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f24c000ae30 con 0x7f24c81379b0 2024-09-18T10:17:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.181+0000 7f24ba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:41.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.181+0000 7f24ba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:41.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.353+0000 7f24d07f1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:41.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.353+0000 7f24d07f1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:41.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.353+0000 7f24d07f1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:41.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.353+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f24c8067b10 con 0x7f24c81379b0 2024-09-18T10:17:41.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.945+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 removed item id -61 name 'h15' from crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f24c0010af0 con 0x7f24c81379b0 2024-09-18T10:17:41.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.945+0000 7f24ba7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:41.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:41.945+0000 7f24ba7fc700 10 monclient: _finish_command 2 = system:0 removed item id -61 name 'h15' from crush map 2024-09-18T10:17:42.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.053+0000 7f24d07f1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:42.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.053+0000 7f24d07f1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:42.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.053+0000 7f24d07f1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:42.059 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.053+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h15"} v 0) v1 -- 0x7f24c812e0b0 con 0x7f24c81379b0 2024-09-18T10:17:42.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24ba7fc700 1 -- 172.21.15.150:0/956274336 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h15"}]=0 device 'h15' does not appear in the crush map v237) v1 ==== 120+0+0 (secure 0 0 0) 0x7f24c001a360 con 0x7f24c81379b0 2024-09-18T10:17:42.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24ba7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h15"}] 2024-09-18T10:17:42.061 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24ba7fc700 10 monclient: _finish_command 3 = system:0 device 'h15' does not appear in the crush map 2024-09-18T10:17:42.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 msgr2=0x7f24ac037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:42.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 0x7f24ac037f50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0030c0 tx=0x7f24bc016000 comp rx=0 tx=0).stop 2024-09-18T10:17:42.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 10 monclient: shutdown 2024-09-18T10:17:42.064 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 msgr2=0x7f24c812c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24c000ccc0 tx=0x7f24c0005400 comp rx=0 tx=0).stop 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 shutdown_connections 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f24ac035aa0 0x7f24ac037f50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24c81379b0 0x7f24c812c700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24c80a04f0 0x7f24c8142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 --2- 172.21.15.150:0/956274336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c80a0120 0x7f24c8142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 >> 172.21.15.150:0/956274336 conn(0x7f24c80a5830 msgr2=0x7f24c809abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 shutdown_connections 2024-09-18T10:17:42.065 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.057+0000 7f24d07f1700 1 -- 172.21.15.150:0/956274336 wait complete. 2024-09-18T10:17:42.066 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h15' does not appear in the crush map 2024-09-18T10:17:42.075 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:42.075 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h16 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.169+0000 7f7059b74700 10 monclient: get_monmap_and_config 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.169+0000 7f7059b74700 10 monclient: build_initial_monmap 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.169+0000 7f7059b74700 10 monclient: monmap: 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:42.177 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:42.175737+0000 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:42.175737+0000 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:42.178 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:42.179 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:42.179 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.188 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 Processor -- start 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 -- start start 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient: init 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f70540a0120 0x7f705409e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-a con 0x7f70540a0120 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 0x7f70540a0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-c con 0x7f70540a04f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:42.189 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-b con 0x7f70541369a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540a21d0 con 0x7f70540a0120 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f705409ddc0 con 0x7f70540a04f0 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705409dae0 con 0x7f70541369a0 2024-09-18T10:17:42.190 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7059b74700 10 monclient(hunting): authenticate will time out at 1344.269586s 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7052ffd700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 msgr2=0x7f70540a0950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7052ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 0x7f70540a0950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f70540a0120 0x7f705409e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46336/0 (socket says 172.21.15.150:46336) 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 1 -- 172.21.15.150:0/3239864025 learned_addr learned my addr 172.21.15.150:0/3239864025 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): get_auth_request con 0x7f70541369a0 auth_method 0 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:42.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): get_auth_request con 0x7f70540a0120 auth_method 0 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:42.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:42.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:42.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:42.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient(hunting): handle_auth_done global_id 94215 payload 274 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 -- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 msgr2=0x7f7054138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 -- 172.21.15.150:0/3239864025 >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 msgr2=0x7f70540a0950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 --2- 172.21.15.150:0/3239864025 >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 0x7f70540a0950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f70537fe700 1 -- 172.21.15.150:0/3239864025 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540a1590 con 0x7f70540a0120 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.173+0000 7f7053fff700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70537fe700 1 -- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 msgr2=0x7f705409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:42.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70537fe700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705409e7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70540a6850 tx=0x7f703c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b2bb254f6ea34d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 1 -- 172.21.15.150:0/3239864025 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c011840 con 0x7f70540a0120 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: dump: 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:42.195 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 5 monclient: authenticate success, global_id 94215 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 1 -- 172.21.15.150:0/3239864025 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c0119a0 con 0x7f70540a0120 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:42.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 4 set_mon_vals no callback set 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 1 -- 172.21.15.150:0/3239864025 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c011d40 con 0x7f70540a0120 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f70527fc700 10 monclient: dump: 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.197 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 10 monclient: shutdown 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 -- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 msgr2=0x7f705409e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705409e7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70540a6850 tx=0x7f703c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 -- 172.21.15.150:0/3239864025 shutdown_connections 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70541369a0 0x7f7054138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 --2- 172.21.15.150:0/3239864025 >> v2:172.21.15.150:3300/0 conn(0x7f70540a04f0 0x7f70540a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 --2- 172.21.15.150:0/3239864025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705409e7a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 -- 172.21.15.150:0/3239864025 >> 172.21.15.150:0/3239864025 conn(0x7f70540a5830 msgr2=0x7f70540a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:42.198 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 -- 172.21.15.150:0/3239864025 shutdown_connections 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 1 -- 172.21.15.150:0/3239864025 wait complete. 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 10 monclient: build_initial_monmap 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.177+0000 7f7059b74700 10 monclient: monmap: 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:42.199 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 Processor -- start 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 -- start start 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient: init 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-a con 0x7f70540a0120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 0x7f705412cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-c con 0x7f70540a04f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 0x7f705412d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): picked mon.noname-b con 0x7f70541369a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.200 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705409d720 con 0x7f70540a0120 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705409bf80 con 0x7f70541369a0 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7054130010 con 0x7f70540a04f0 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7059b74700 10 monclient(hunting): authenticate will time out at 1344.277076s 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7053fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 msgr2=0x7f705412d270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7053fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 0x7f705412d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 0x7f705412cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 0x7f705412cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46338/0 (socket says 172.21.15.150:46338) 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47326/0 (socket says 172.21.15.150:47326) 2024-09-18T10:17:42.201 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 1 -- 172.21.15.150:0/3126308636 learned_addr learned my addr 172.21.15.150:0/3126308636 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): get_auth_request con 0x7f70540a04f0 auth_method 0 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): get_auth_request con 0x7f70540a0120 auth_method 0 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:42.202 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f7052ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient(hunting): handle_auth_done global_id 94218 payload 824 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 -- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 msgr2=0x7f705412cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 0x7f705412cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 -- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 msgr2=0x7f705412d270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 0x7f705412d270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.203 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.181+0000 7f70537fe700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009ae0 con 0x7f70540a0120 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f70537fe700 1 -- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 msgr2=0x7f705412c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f70537fe700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7054068ec0 tx=0x7f703c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c00bd10 con 0x7f70540a0120 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: dump: 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:42.206 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 5 monclient: authenticate success, global_id 94218 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c00bed0 con 0x7f70540a0120 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _renew_subs 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c009420 con 0x7f70540a0120 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c024690 con 0x7f70540a0120 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 10 monclient: dump: 2024-09-18T10:17:42.208 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7051ffb700 4 set_mon_vals no callback set 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _renew_subs 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705412df90 con 0x7f70540a0120 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705409d890 con 0x7f70540a0120 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f703c01a420 con 0x7f70540a0120 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 0x7f7034037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(237..237 src has 176..237) v4 ==== 11888+0+0 (secure 0 0 0) 0x7f703c04b6f0 con 0x7f70540a0120 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.185+0000 7f7052ffd700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 0x7f7034037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.189+0000 7f7052ffd700 10 monclient: get_auth_request con 0x7f7034035aa0 auth_method 0 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.189+0000 7f7052ffd700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 0x7f7034037f50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7044006fd0 tx=0x7f7044009220 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.197+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f703c022de0 con 0x7f70540a0120 2024-09-18T10:17:42.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.201+0000 7f704b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:42.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.201+0000 7f704b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:42.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.373+0000 7f7059b74700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:42.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.373+0000 7f7059b74700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:42.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.373+0000 7f7059b74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:42.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.373+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f70540a1c90 con 0x7f70540a0120 2024-09-18T10:17:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.957+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 removed item id -64 name 'h16' from crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f703c015030 con 0x7f70540a0120 2024-09-18T10:17:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.957+0000 7f704b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:42.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:42.957+0000 7f704b7fe700 10 monclient: _finish_command 2 = system:0 removed item id -64 name 'h16' from crush map 2024-09-18T10:17:43.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.065+0000 7f7059b74700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:43.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.065+0000 7f7059b74700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:43.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.065+0000 7f7059b74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:43.073 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.065+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h16"} v 0) v1 -- 0x7f7054130530 con 0x7f70540a0120 2024-09-18T10:17:43.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f704b7fe700 1 -- 172.21.15.150:0/3126308636 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h16"}]=0 device 'h16' does not appear in the crush map v238) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7054130530 con 0x7f70540a0120 2024-09-18T10:17:43.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f704b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h16"}] 2024-09-18T10:17:43.074 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f704b7fe700 10 monclient: _finish_command 3 = system:0 device 'h16' does not appear in the crush map 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 msgr2=0x7f7034037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 0x7f7034037f50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7044006fd0 tx=0x7f7044009220 comp rx=0 tx=0).stop 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 10 monclient: shutdown 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 msgr2=0x7f705412c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7054068ec0 tx=0x7f703c017000 comp rx=0 tx=0).stop 2024-09-18T10:17:43.077 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 shutdown_connections 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7034035aa0 0x7f7034037f50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70541369a0 0x7f705412d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f70540a04f0 0x7f705412cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 --2- 172.21.15.150:0/3126308636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70540a0120 0x7f705412c7f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 >> 172.21.15.150:0/3126308636 conn(0x7f70540a5830 msgr2=0x7f705409ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 shutdown_connections 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.069+0000 7f7059b74700 1 -- 172.21.15.150:0/3126308636 wait complete. 2024-09-18T10:17:43.078 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h16' does not appear in the crush map 2024-09-18T10:17:43.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:43.087 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h17 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: get_monmap_and_config 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: build_initial_monmap 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: monmap: 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:43.185562+0000 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:43.185562+0000 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:43.187 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:43.188 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 Processor -- start 2024-09-18T10:17:43.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 -- start start 2024-09-18T10:17:43.191 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: init 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f1c0a07d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c0a0ba0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c09e220 con 0x7f8f1c0a07d0 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1c09ee30 con 0x7f8f1c0a0ba0 2024-09-18T10:17:43.192 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c09eaa0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f223c4700 10 monclient(hunting): authenticate will time out at 1345.279279s 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1b7fe700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f8f1c0a07d0 legacy=0x7f8f1c136a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46340/0 (socket says 172.21.15.150:46340) 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 1 -- 172.21.15.150:0/2622873405 learned_addr learned my addr 172.21.15.150:0/2622873405 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47336/0 (socket says 172.21.15.150:47336) 2024-09-18T10:17:43.193 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 10 monclient(hunting): get_auth_request con 0x7f8f1c09e6d0 auth_method 0 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 10 monclient(hunting): get_auth_request con 0x7f8f1c0a0ba0 auth_method 0 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:43.194 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.181+0000 7f8f20961700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:43.196 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_done global_id 74606 payload 274 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 -- 172.21.15.150:0/2622873405 >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 msgr2=0x7f8f1c13a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 --2- 172.21.15.150:0/2622873405 >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 -- 172.21.15.150:0/2622873405 >> v1:172.21.15.150:6789/0 conn(0x7f8f1c0a07d0 legacy=0x7f8f1c136a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 monclient: found mon.noname-b 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 -- 172.21.15.150:0/2622873405 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c09d930 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.197 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f20961700 1 --2- 172.21.15.150:0/2622873405 >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 -- 172.21.15.150:0/2622873405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 msgr2=0x7f8f1c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1bfff700 1 --2- 172.21.15.150:0/2622873405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8f100099b0 tx=0x7f8f10009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf8918ac857a54e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 1 -- 172.21.15.150:0/2622873405 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10009ea0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: dump: 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:43.207 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: _finish_auth 0 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f223c4700 5 monclient: authenticate success, global_id 74606 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f223c4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:43.208 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 1 -- 172.21.15.150:0/2622873405 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1000ba70 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f223c4700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f223c4700 4 set_mon_vals no callback set 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 1 -- 172.21.15.150:0/2622873405 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1000be30 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.185+0000 7f8f1affd700 10 monclient: dump: 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.209 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: shutdown 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- 172.21.15.150:0/2622873405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 msgr2=0x7f8f1c0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- 172.21.15.150:0/2622873405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8f100099b0 tx=0x7f8f10009250 comp rx=0 tx=0).stop 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- 172.21.15.150:0/2622873405 shutdown_connections 2024-09-18T10:17:43.210 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- 172.21.15.150:0/2622873405 >> v2:172.21.15.142:3300/0 conn(0x7f8f1c0a0ba0 0x7f8f1c13a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- 172.21.15.150:0/2622873405 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c0a0290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- 172.21.15.150:0/2622873405 >> 172.21.15.150:0/2622873405 conn(0x7f8f1c0a58a0 msgr2=0x7f8f1c0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- 172.21.15.150:0/2622873405 shutdown_connections 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- 172.21.15.150:0/2622873405 wait complete. 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: build_initial_monmap 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: monmap: 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:43.211 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 Processor -- start 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- start start 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: init 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-c con 0x7f8f1c09e6d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 0x7f8f1c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.212 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-b con 0x7f8f1c0a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 0x7f8f1c08deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): picked mon.noname-a con 0x7f8f1c0a0ba0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c09c0f0 con 0x7f8f1c0a0ba0 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c091b00 con 0x7f8f1c0a07d0 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c08feb0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f223c4700 10 monclient(hunting): authenticate will time out at 1345.286690s 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1b7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 msgr2=0x7f8f1c142250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:43.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 0x7f8f1c142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 0x7f8f1c08deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46354/0 (socket says 172.21.15.150:46354) 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 -- 172.21.15.150:0/436497662 learned_addr learned my addr 172.21.15.150:0/436497662 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): get_auth_request con 0x7f8f1c09e6d0 auth_method 0 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): get_auth_request con 0x7f8f1c0a0ba0 auth_method 0 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:43.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient(hunting): handle_auth_done global_id 74609 payload 824 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 -- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 msgr2=0x7f8f1c142250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 0x7f8f1c142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 -- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 msgr2=0x7f8f1c08deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 0x7f8f1c08deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 monclient: found mon.noname-c 2024-09-18T10:17:43.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f1bfff700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f14009a20 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.189+0000 7f8f20961700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 0x7f8f1c08deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f1bfff700 1 -- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 msgr2=0x7f8f1c141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f1bfff700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000b3c0 tx=0x7f8f10016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10006b60 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: dump: 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:43.216 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: _finish_auth 0 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 5 monclient: authenticate success, global_id 74609 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 10 monclient: _renew_subs 2024-09-18T10:17:43.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10020070 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f100094f0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10011dd0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f03fff700 10 monclient: dump: 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:43.218 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f1a7fc700 4 set_mon_vals no callback set 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 10 monclient: _renew_subs 2024-09-18T10:17:43.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.193+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c0927a0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f223c4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f223c4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f223c4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c09dbf0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8f10022030 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f03fff700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 0x7f8f08037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(238..238 src has 176..238) v4 ==== 11675+0+0 (secure 0 0 0) 0x7f8f1004bc40 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f1b7fe700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 0x7f8f08037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.197+0000 7f8f1b7fe700 10 monclient: get_auth_request con 0x7f8f08035a70 auth_method 0 2024-09-18T10:17:43.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.201+0000 7f8f1b7fe700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 0x7f8f08037f20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0039d0 tx=0x7f8f0c006e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:43.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.205+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8f10019750 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.205+0000 7f8f03fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:43.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.205+0000 7f8f03fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:43.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.373+0000 7f8f223c4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:43.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.373+0000 7f8f223c4700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:43.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.373+0000 7f8f223c4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:43.380 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.373+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f8f1c09dbf0 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.969+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 removed item id -67 name 'h17' from crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8f1002c380 con 0x7f8f1c09e6d0 2024-09-18T10:17:43.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.969+0000 7f8f03fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:43.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:43.969+0000 7f8f03fff700 10 monclient: _finish_command 2 = system:0 removed item id -67 name 'h17' from crush map 2024-09-18T10:17:44.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.077+0000 7f8f223c4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:44.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.077+0000 7f8f223c4700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:44.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.077+0000 7f8f223c4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.077+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h17"} v 0) v1 -- 0x7f8f1c09dbf0 con 0x7f8f1c09e6d0 2024-09-18T10:17:44.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f03fff700 1 -- 172.21.15.150:0/436497662 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h17"}]=0 device 'h17' does not appear in the crush map v239) v1 ==== 120+0+0 (secure 0 0 0) 0x7f8f1002c9d0 con 0x7f8f1c09e6d0 2024-09-18T10:17:44.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f03fff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h17"}] 2024-09-18T10:17:44.094 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f03fff700 10 monclient: _finish_command 3 = system:0 device 'h17' does not appear in the crush map 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 msgr2=0x7f8f08037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 0x7f8f08037f20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0039d0 tx=0x7f8f0c006e00 comp rx=0 tx=0).stop 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 10 monclient: shutdown 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 msgr2=0x7f8f1c141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000b3c0 tx=0x7f8f10016000 comp rx=0 tx=0).stop 2024-09-18T10:17:44.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 shutdown_connections 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8f08035a70 0x7f8f08037f20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f1c0a0ba0 0x7f8f1c08deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f1c0a07d0 0x7f8f1c142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 --2- 172.21.15.150:0/436497662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f1c09e6d0 0x7f8f1c141d10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 >> 172.21.15.150:0/436497662 conn(0x7f8f1c0a58a0 msgr2=0x7f8f1c09abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.089+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 shutdown_connections 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.093+0000 7f8f223c4700 1 -- 172.21.15.150:0/436497662 wait complete. 2024-09-18T10:17:44.098 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h17' does not appear in the crush map 2024-09-18T10:17:44.109 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:44.110 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h18 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.201+0000 7f93c1152700 10 monclient: get_monmap_and_config 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.201+0000 7f93c1152700 10 monclient: build_initial_monmap 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.201+0000 7f93c1152700 10 monclient: monmap: 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:44.207849+0000 2024-09-18T10:17:44.209 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:44.207849+0000 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:44.210 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 Processor -- start 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 -- start start 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient: init 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 0x7f93bc0a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-b con 0x7f93bc09cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93bc09e480 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:44.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 0x7f93bc139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-c con 0x7f93bc0990a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc09e060 con 0x7f93bc09e480 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc0a66c0 con 0x7f93bc0990a0 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc09d750 con 0x7f93bc09cf90 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93c1152700 10 monclient(hunting): authenticate will time out at 1346.301354s 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bb59e700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 msgr2=0x7f93bc139a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bb59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 0x7f93bc139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 0x7f93bc0a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93ba59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f93bc09e480 0x7f93bc0a07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35224/0 (socket says 172.21.15.150:35224) 2024-09-18T10:17:44.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93ba59c700 1 -- 172.21.15.150:0/4241880981 learned_addr learned my addr 172.21.15.150:0/4241880981 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:44.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): get_auth_request con 0x7f93bc09cf90 auth_method 0 2024-09-18T10:17:44.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:44.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient(hunting): handle_auth_done global_id 74612 payload 274 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 -- 172.21.15.150:0/4241880981 >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 msgr2=0x7f93bc139a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 --2- 172.21.15.150:0/4241880981 >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 0x7f93bc139a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 -- 172.21.15.150:0/4241880981 >> v1:172.21.15.142:6789/0 conn(0x7f93bc09e480 legacy=0x7f93bc0a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 monclient: found mon.noname-b 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.217 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.205+0000 7f93bad9d700 1 -- 172.21.15.150:0/4241880981 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc09d360 con 0x7f93bc09cf90 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93bad9d700 1 -- 172.21.15.150:0/4241880981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 msgr2=0x7f93bc0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93bad9d700 1 --2- 172.21.15.150:0/4241880981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 0x7f93bc0a0290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93a40098e0 tx=0x7f93a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e2e17cfcf83b954 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 1 -- 172.21.15.150:0/4241880981 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a400ba70 con 0x7f93bc09cf90 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: dump: 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.225 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:44.226 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 1 -- 172.21.15.150:0/4241880981 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a400bbf0 con 0x7f93bc09cf90 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 1 -- 172.21.15.150:0/4241880981 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a4011420 con 0x7f93bc09cf90 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93b9d9b700 10 monclient: dump: 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.227 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 5 monclient: authenticate success, global_id 74612 2024-09-18T10:17:44.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 4 set_mon_vals no callback set 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 10 monclient: shutdown 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 -- 172.21.15.150:0/4241880981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 msgr2=0x7f93bc0a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 --2- 172.21.15.150:0/4241880981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 0x7f93bc0a0290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93a40098e0 tx=0x7f93a40091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 -- 172.21.15.150:0/4241880981 shutdown_connections 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 --2- 172.21.15.150:0/4241880981 >> v2:172.21.15.150:3300/0 conn(0x7f93bc0990a0 0x7f93bc139a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 --2- 172.21.15.150:0/4241880981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09cf90 0x7f93bc0a0290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 -- 172.21.15.150:0/4241880981 >> 172.21.15.150:0/4241880981 conn(0x7f93bc0a58a0 msgr2=0x7f93bc0a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 -- 172.21.15.150:0/4241880981 shutdown_connections 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 -- 172.21.15.150:0/4241880981 wait complete. 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 10 monclient: build_initial_monmap 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 10 monclient: monmap: 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:44.230 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.209+0000 7f93c1152700 1 Processor -- start 2024-09-18T10:17:44.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 -- start start 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient: init 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93bc0990a0 0x7f93bc141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-a con 0x7f93bc0990a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 0x7f93bc142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-b con 0x7f93bc09cf90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:44.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): picked mon.noname-c con 0x7f93bc09e480 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc08f7a0 con 0x7f93bc0990a0 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc090700 con 0x7f93bc09cf90 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc090cc0 con 0x7f93bc09e480 2024-09-18T10:17:44.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93c1152700 10 monclient(hunting): authenticate will time out at 1346.308929s 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93ba59c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 msgr2=0x7f93bc142250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93ba59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 0x7f93bc142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93bc0990a0 0x7f93bc141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46362/0 (socket says 172.21.15.150:46362) 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 -- 172.21.15.150:0/2206343569 learned_addr learned my addr 172.21.15.150:0/2206343569 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:44.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): get_auth_request con 0x7f93bc09e480 auth_method 0 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): get_auth_request con 0x7f93bc0990a0 auth_method 0 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:44.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient(hunting): handle_auth_done global_id 74615 payload 824 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 -- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 msgr2=0x7f93bc142250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:44.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 0x7f93bc142250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 -- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93bc0990a0 msgr2=0x7f93bc141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93bc0990a0 0x7f93bc141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient: _finish_hunting 0 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 monclient: found mon.noname-c 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.213+0000 7f93bb59e700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4009420 con 0x7f93bc09e480 2024-09-18T10:17:44.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93bb59e700 1 -- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 msgr2=0x7f93bc08d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93bb59e700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f93b400b5b0 tx=0x7f93b40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4003e20 con 0x7f93bc09e480 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: dump: 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:44.238 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 5 monclient: authenticate success, global_id 74615 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _renew_subs 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4006150 con 0x7f93bc09e480 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93bc0918b0 con 0x7f93bc09e480 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4006510 con 0x7f93bc09e480 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 10 monclient: dump: 2024-09-18T10:17:44.240 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _renew_subs 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc142930 con 0x7f93bc09e480 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b8d99700 4 set_mon_vals no callback set 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f93b401a360 con 0x7f93bc09e480 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc0a0ee0 con 0x7f93bc09e480 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 0x7f939c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:44.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(239..239 src has 176..239) v4 ==== 11462+0+0 (secure 0 0 0) 0x7f93b4048f80 con 0x7f93bc09e480 2024-09-18T10:17:44.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93bad9d700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 0x7f939c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:44.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.217+0000 7f93bad9d700 10 monclient: get_auth_request con 0x7f939c035a70 auth_method 0 2024-09-18T10:17:44.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.221+0000 7f93bad9d700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 0x7f939c037f20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f93a40098e0 tx=0x7f93a4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:44.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.229+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f93b4017810 con 0x7f93bc09e480 2024-09-18T10:17:44.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.229+0000 7f93b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:44.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.229+0000 7f93b2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:44.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.401+0000 7f93c1152700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:44.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.401+0000 7f93c1152700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:44.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.401+0000 7f93c1152700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:44.405 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.401+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f93bc090aa0 con 0x7f93bc09e480 2024-09-18T10:17:44.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.981+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 removed item id -70 name 'h18' from crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7f93b40173f0 con 0x7f93bc09e480 2024-09-18T10:17:44.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.981+0000 7f93b2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:44.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:44.981+0000 7f93b2ffd700 10 monclient: _finish_command 2 = system:0 removed item id -70 name 'h18' from crush map 2024-09-18T10:17:45.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.089+0000 7f93c1152700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:45.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.089+0000 7f93c1152700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:45.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.089+0000 7f93c1152700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:45.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.089+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h18"} v 0) v1 -- 0x7f93bc063ed0 con 0x7f93bc09e480 2024-09-18T10:17:45.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93b2ffd700 1 -- 172.21.15.150:0/2206343569 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h18"}]=0 device 'h18' does not appear in the crush map v240) v1 ==== 120+0+0 (secure 0 0 0) 0x7f93b40173f0 con 0x7f93bc09e480 2024-09-18T10:17:45.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93b2ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h18"}] 2024-09-18T10:17:45.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93b2ffd700 10 monclient: _finish_command 3 = system:0 device 'h18' does not appear in the crush map 2024-09-18T10:17:45.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 msgr2=0x7f939c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 0x7f939c037f20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f93a40098e0 tx=0x7f93a4016000 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 10 monclient: shutdown 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 msgr2=0x7f93bc08d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f93b400b5b0 tx=0x7f93b40090f0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 shutdown_connections 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f939c035a70 0x7f939c037f20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93bc09e480 0x7f93bc08d1c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93bc09cf90 0x7f93bc142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 --2- 172.21.15.150:0/2206343569 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93bc0990a0 0x7f93bc141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 >> 172.21.15.150:0/2206343569 conn(0x7f93bc0a58a0 msgr2=0x7f93bc0a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:45.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 shutdown_connections 2024-09-18T10:17:45.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.093+0000 7f93c1152700 1 -- 172.21.15.150:0/2206343569 wait complete. 2024-09-18T10:17:45.102 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h18' does not appear in the crush map 2024-09-18T10:17:45.111 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:45.111 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h19 2024-09-18T10:17:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: get_monmap_and_config 2024-09-18T10:17:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: build_initial_monmap 2024-09-18T10:17:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: monmap: 2024-09-18T10:17:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:45.214 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:45.212880+0000 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:45.212880+0000 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:45.215 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 Processor -- start 2024-09-18T10:17:45.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 -- start start 2024-09-18T10:17:45.216 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: init 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d30071240 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d3009e1e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 0x7f9d301374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-b con 0x7f9d300994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d30098cf0 con 0x7f9d30071240 2024-09-18T10:17:45.219 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d30097890 con 0x7f9d3009e1e0 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3009e640 con 0x7f9d300994f0 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d2e7fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f9d3009e1e0 legacy=0x7f9d30097180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.209+0000 7f9d356d4700 10 monclient(hunting): authenticate will time out at 1347.307785s 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 0x7f9d301374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2effd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9d30071240 0x7f9d30096a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35228/0 (socket says 172.21.15.150:35228) 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2effd700 1 -- 172.21.15.150:0/2139917336 learned_addr learned my addr 172.21.15.150:0/2139917336 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): get_auth_request con 0x7f9d300994f0 auth_method 0 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:45.220 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient(hunting): handle_auth_done global_id 74618 payload 274 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 -- 172.21.15.150:0/2139917336 >> v1:172.21.15.150:6789/0 conn(0x7f9d3009e1e0 legacy=0x7f9d30097180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 -- 172.21.15.150:0/2139917336 >> v1:172.21.15.142:6789/0 conn(0x7f9d30071240 legacy=0x7f9d30096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:45.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 -- 172.21.15.150:0/2139917336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3009df80 con 0x7f9d300994f0 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 -- 172.21.15.150:0/2139917336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 msgr2=0x7f9d301374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2f7fe700 1 --2- 172.21.15.150:0/2139917336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 0x7f9d301374e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009e90 tx=0x7f9d28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9186bd0c03e8c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 1 -- 172.21.15.150:0/2139917336 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800e070 con 0x7f9d300994f0 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: dump: 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.229 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 1 -- 172.21.15.150:0/2139917336 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28012430 con 0x7f9d300994f0 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d356d4700 5 monclient: authenticate success, global_id 74618 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d356d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 1 -- 172.21.15.150:0/2139917336 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d280127f0 con 0x7f9d300994f0 2024-09-18T10:17:45.230 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d356d4700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d356d4700 4 set_mon_vals no callback set 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.213+0000 7f9d2dffb700 10 monclient: dump: 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:45.231 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: shutdown 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- 172.21.15.150:0/2139917336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 msgr2=0x7f9d301374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 --2- 172.21.15.150:0/2139917336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 0x7f9d301374e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009e90 tx=0x7f9d28009780 comp rx=0 tx=0).stop 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- 172.21.15.150:0/2139917336 shutdown_connections 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 --2- 172.21.15.150:0/2139917336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d300994f0 0x7f9d301374e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- 172.21.15.150:0/2139917336 >> 172.21.15.150:0/2139917336 conn(0x7f9d300a1cc0 msgr2=0x7f9d300a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- 172.21.15.150:0/2139917336 shutdown_connections 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- 172.21.15.150:0/2139917336 wait complete. 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: build_initial_monmap 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: monmap: 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:45.232 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 Processor -- start 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- start start 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: init 2024-09-18T10:17:45.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 0x7f9d30137340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-c con 0x7f9d30071240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-a con 0x7f9d300994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 0x7f9d3011b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:45.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient(hunting): picked mon.noname-b con 0x7f9d3009e1e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.217+0000 7f9d356d4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3011ef50 con 0x7f9d300994f0 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300980f0 con 0x7f9d3009e1e0 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2f7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 msgr2=0x7f9d3011b230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3009b180 con 0x7f9d30071240 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d356d4700 10 monclient(hunting): authenticate will time out at 1347.316105s 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2f7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 0x7f9d3011b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:45.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 0x7f9d30137340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47348/0 (socket says 172.21.15.150:47348) 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 -- 172.21.15.150:0/3972372146 learned_addr learned my addr 172.21.15.150:0/3972372146 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): get_auth_request con 0x7f9d300994f0 auth_method 0 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): get_auth_request con 0x7f9d30071240 auth_method 0 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:45.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient(hunting): handle_auth_done global_id 94230 payload 824 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 -- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 msgr2=0x7f9d30137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 0x7f9d30137340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 -- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 msgr2=0x7f9d3011b230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 0x7f9d3011b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:45.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 monclient: found mon.noname-a 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d28009a20 con 0x7f9d300994f0 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2effd700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 0x7f9d30137340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 -- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 msgr2=0x7f9d3011acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:45.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.221+0000 7f9d2e7fc700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d2400b7e0 tx=0x7f9d24009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:45.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24003e20 con 0x7f9d300994f0 2024-09-18T10:17:45.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:45.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: dump: 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:45.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d24006150 con 0x7f9d300994f0 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 5 monclient: authenticate success, global_id 94230 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d24006510 con 0x7f9d300994f0 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _renew_subs 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3011cc70 con 0x7f9d300994f0 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d2d7fa700 4 set_mon_vals no callback set 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 10 monclient: dump: 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:45.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _renew_subs 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d30136830 con 0x7f9d300994f0 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:45.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3006e3e0 con 0x7f9d300994f0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9d240103c0 con 0x7f9d300994f0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 0x7f9d1c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(240..240 src has 176..240) v4 ==== 11249+0+0 (secure 0 0 0) 0x7f9d240494e0 con 0x7f9d300994f0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d2effd700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 0x7f9d1c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.225+0000 7f9d2effd700 10 monclient: get_auth_request con 0x7f9d1c035a70 auth_method 0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.229+0000 7f9d2effd700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 0x7f9d1c037f20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9d20005220 tx=0x7f9d20007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:45.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.237+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9d2400e8a0 con 0x7f9d300994f0 2024-09-18T10:17:45.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.237+0000 7f9d16ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:45.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.237+0000 7f9d16ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:45.414 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.409+0000 7f9d356d4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:45.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.409+0000 7f9d356d4700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:45.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.409+0000 7f9d356d4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:45.415 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.409+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f9d30063ed0 con 0x7f9d300994f0 2024-09-18T10:17:45.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.989+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 removed item id -73 name 'h19' from crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9d24017030 con 0x7f9d300994f0 2024-09-18T10:17:45.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.989+0000 7f9d16ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:45.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:45.989+0000 7f9d16ffd700 10 monclient: _finish_command 2 = system:0 removed item id -73 name 'h19' from crush map 2024-09-18T10:17:46.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d356d4700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:46.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d356d4700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:46.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d356d4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h19"} v 0) v1 -- 0x7f9d300008d0 con 0x7f9d300994f0 2024-09-18T10:17:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d16ffd700 1 -- 172.21.15.150:0/3972372146 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h19"}]=0 device 'h19' does not appear in the crush map v241) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9d24017020 con 0x7f9d300994f0 2024-09-18T10:17:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d16ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h19"}] 2024-09-18T10:17:46.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.101+0000 7f9d16ffd700 10 monclient: _finish_command 3 = system:0 device 'h19' does not appear in the crush map 2024-09-18T10:17:46.112 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 msgr2=0x7f9d1c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 0x7f9d1c037f20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9d20005220 tx=0x7f9d20007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 10 monclient: shutdown 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 msgr2=0x7f9d3011acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9d2400b7e0 tx=0x7f9d24009220 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 shutdown_connections 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d1c035a70 0x7f9d1c037f20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d3009e1e0 0x7f9d3011b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d300994f0 0x7f9d3011acf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 --2- 172.21.15.150:0/3972372146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d30071240 0x7f9d30137340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 >> 172.21.15.150:0/3972372146 conn(0x7f9d300a1cc0 msgr2=0x7f9d3009d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:46.113 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 shutdown_connections 2024-09-18T10:17:46.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.105+0000 7f9d356d4700 1 -- 172.21.15.150:0/3972372146 wait complete. 2024-09-18T10:17:46.114 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h19' does not appear in the crush map 2024-09-18T10:17:46.126 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:46.127 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h20 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: get_monmap_and_config 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: build_initial_monmap 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: monmap: 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:46.226826+0000 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:46.226826+0000 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:46.228 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:46.229 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 1 Processor -- start 2024-09-18T10:17:46.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 1 -- start start 2024-09-18T10:17:46.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: init 2024-09-18T10:17:46.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:46.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-b con 0x7fd87809bfd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 0x7fd87809c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-c con 0x7fd87809c3a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd878136790 0x7fd878138b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-a con 0x7fd878136790 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.221+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd878069c50 con 0x7fd878136790 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd878098c60 con 0x7fd87809c3a0 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87809ba50 con 0x7fd87809bfd0 2024-09-18T10:17:46.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 10 monclient(hunting): authenticate will time out at 1348.320033s 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd876ffd700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 msgr2=0x7fd87809c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd876ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 0x7fd87809c790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd878136790 0x7fd878138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46402/0 (socket says 172.21.15.150:46402) 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 -- 172.21.15.150:0/2913359897 learned_addr learned my addr 172.21.15.150:0/2913359897 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): get_auth_request con 0x7fd87809bfd0 auth_method 0 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:46.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): get_auth_request con 0x7fd878136790 auth_method 0 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient(hunting): handle_auth_done global_id 74624 payload 274 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 -- 172.21.15.150:0/2913359897 >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 msgr2=0x7fd87809c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 --2- 172.21.15.150:0/2913359897 >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 0x7fd87809c790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 -- 172.21.15.150:0/2913359897 >> v2:172.21.15.142:3300/0 conn(0x7fd878136790 msgr2=0x7fd878138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 --2- 172.21.15.150:0/2913359897 >> v2:172.21.15.142:3300/0 conn(0x7fd878136790 0x7fd878138b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 monclient: found mon.noname-b 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 -- 172.21.15.150:0/2913359897 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87809dc70 con 0x7fd87809bfd0 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 -- 172.21.15.150:0/2913359897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 msgr2=0x7fd878099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8777fe700 1 --2- 172.21.15.150:0/2913359897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd8780723c0 tx=0x7fd868009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73afcf0967c57428 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 1 -- 172.21.15.150:0/2913359897 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868009e70 con 0x7fd87809bfd0 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: dump: 2024-09-18T10:17:46.238 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 5 monclient: authenticate success, global_id 74624 2024-09-18T10:17:46.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 1 -- 172.21.15.150:0/2913359897 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86800ba70 con 0x7fd87809bfd0 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd87e178700 4 set_mon_vals no callback set 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 1 -- 172.21.15.150:0/2913359897 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86800be30 con 0x7fd87809bfd0 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.225+0000 7fd8767fc700 10 monclient: dump: 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.240 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: shutdown 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- 172.21.15.150:0/2913359897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 msgr2=0x7fd878099360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- 172.21.15.150:0/2913359897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd8780723c0 tx=0x7fd868009250 comp rx=0 tx=0).stop 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- 172.21.15.150:0/2913359897 shutdown_connections 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- 172.21.15.150:0/2913359897 >> v2:172.21.15.142:3300/0 conn(0x7fd878136790 0x7fd878138b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- 172.21.15.150:0/2913359897 >> v2:172.21.15.150:3300/0 conn(0x7fd87809c3a0 0x7fd87809c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- 172.21.15.150:0/2913359897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809bfd0 0x7fd878099360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- 172.21.15.150:0/2913359897 >> 172.21.15.150:0/2913359897 conn(0x7fd8780a15e0 msgr2=0x7fd8780a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- 172.21.15.150:0/2913359897 shutdown_connections 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- 172.21.15.150:0/2913359897 wait complete. 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: build_initial_monmap 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: monmap: 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:46.242 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:46.243 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 Processor -- start 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- start start 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: init 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 0x7fd8781359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-b con 0x7fd87809bfd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 0x7fd8780940d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-c con 0x7fd87809c3a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 0x7fd87808dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): picked mon.noname-a con 0x7fd878136790 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:46.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87809ad50 con 0x7fd878136790 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87809fcc0 con 0x7fd87809bfd0 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780912f0 con 0x7fd87809c3a0 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd87e178700 10 monclient(hunting): authenticate will time out at 1348.326611s 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd8777fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 msgr2=0x7fd8781359b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd8777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 0x7fd8781359b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 0x7fd8780940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 0x7fd8780940d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46416/0 (socket says 172.21.15.150:46416) 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 1 -- 172.21.15.150:0/2333714717 learned_addr learned my addr 172.21.15.150:0/2333714717 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:46.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): get_auth_request con 0x7fd87809c3a0 auth_method 0 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 0x7fd87808dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): get_auth_request con 0x7fd878136790 auth_method 0 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd876ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:46.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient(hunting): handle_auth_done global_id 94236 payload 824 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 -- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 msgr2=0x7fd8780940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 0x7fd8780940d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 -- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 msgr2=0x7fd8781359b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 0x7fd8781359b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 monclient: found mon.noname-a 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.229+0000 7fd877fff700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680094f0 con 0x7fd878136790 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd877fff700 1 -- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 msgr2=0x7fd87808dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:46.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd877fff700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 0x7fd87808dc80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd870009e50 tx=0x7fd87000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87000cd60 con 0x7fd878136790 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: dump: 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:46.252 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 5 monclient: authenticate success, global_id 94236 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 10 monclient: _renew_subs 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87000f040 con 0x7fd878136790 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87808f7c0 con 0x7fd878136790 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700146d0 con 0x7fd878136790 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:46.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd86f7fe700 10 monclient: dump: 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 10 monclient: _renew_subs 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd878090390 con 0x7fd878136790 2024-09-18T10:17:46.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd875ffb700 4 set_mon_vals no callback set 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.233+0000 7fd87e178700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd87e178700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd87e178700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd878063ed0 con 0x7fd878136790 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd870014a40 con 0x7fd878136790 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd86f7fe700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 0x7fd858037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(241..241 src has 176..241) v4 ==== 11036+0+0 (secure 0 0 0) 0x7fd87004a8b0 con 0x7fd878136790 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd8777fe700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 0x7fd858037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd8777fe700 10 monclient: get_auth_request con 0x7fd858035a70 auth_method 0 2024-09-18T10:17:46.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.237+0000 7fd8777fe700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 0x7fd858037f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd8780723c0 tx=0x7fd868017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:46.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.245+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd870024530 con 0x7fd878136790 2024-09-18T10:17:46.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.245+0000 7fd86f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:46.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.245+0000 7fd86f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:46.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.417+0000 7fd87e178700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:46.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.417+0000 7fd87e178700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:46.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.417+0000 7fd87e178700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:46.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.417+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fd878067b10 con 0x7fd878136790 2024-09-18T10:17:47.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.997+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 removed item id -76 name 'h20' from crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd870018030 con 0x7fd878136790 2024-09-18T10:17:47.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.997+0000 7fd86f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:47.004 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:46.997+0000 7fd86f7fe700 10 monclient: _finish_command 2 = system:0 removed item id -76 name 'h20' from crush map 2024-09-18T10:17:47.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd87e178700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:47.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd87e178700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:47.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd87e178700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.114 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h20"} v 0) v1 -- 0x7fd878135ef0 con 0x7fd878136790 2024-09-18T10:17:47.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd86f7fe700 1 -- 172.21.15.150:0/2333714717 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h20"}]=0 device 'h20' does not appear in the crush map v242) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd870018020 con 0x7fd878136790 2024-09-18T10:17:47.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd86f7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h20"}] 2024-09-18T10:17:47.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.109+0000 7fd86f7fe700 10 monclient: _finish_command 3 = system:0 device 'h20' does not appear in the crush map 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 msgr2=0x7fd858037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 0x7fd858037f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd8780723c0 tx=0x7fd868017000 comp rx=0 tx=0).stop 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 10 monclient: shutdown 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 msgr2=0x7fd87808dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 0x7fd87808dc80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd870009e50 tx=0x7fd87000b040 comp rx=0 tx=0).stop 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 shutdown_connections 2024-09-18T10:17:47.118 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd858035a70 0x7fd858037f20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd878136790 0x7fd87808dc80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd87809c3a0 0x7fd8780940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 --2- 172.21.15.150:0/2333714717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd87809bfd0 0x7fd8781359b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 >> 172.21.15.150:0/2333714717 conn(0x7fd8780a15e0 msgr2=0x7fd87809df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 shutdown_connections 2024-09-18T10:17:47.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.113+0000 7fd87e178700 1 -- 172.21.15.150:0/2333714717 wait complete. 2024-09-18T10:17:47.120 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h20' does not appear in the crush map 2024-09-18T10:17:47.129 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:47.129 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h21 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 10 monclient: get_monmap_and_config 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 10 monclient: build_initial_monmap 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 10 monclient: monmap: 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:47.230748+0000 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:47.230748+0000 2024-09-18T10:17:47.232 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:47.233 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 1 Processor -- start 2024-09-18T10:17:47.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 1 -- start start 2024-09-18T10:17:47.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.225+0000 7fc21130c700 10 monclient: init 2024-09-18T10:17:47.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 0x7fc20c0a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-b con 0x7fc20c0a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc20c06e610 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc20c137590 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c06afe0 con 0x7fc20c137590 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20c0a2220 con 0x7fc20c06e610 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c0a2880 con 0x7fc20c0a4270 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc21130c700 10 monclient(hunting): authenticate will time out at 1349.324719s 2024-09-18T10:17:47.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc2027fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fc20c06e610 legacy=0x7fc20c06e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 0x7fc20c0a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc20c137590 0x7fc20c139970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35244/0 (socket says 172.21.15.150:35244) 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20b7fe700 1 -- 172.21.15.150:0/3406270409 learned_addr learned my addr 172.21.15.150:0/3406270409 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): get_auth_request con 0x7fc20c0a4270 auth_method 0 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient(hunting): handle_auth_done global_id 74630 payload 274 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 1 -- 172.21.15.150:0/3406270409 >> v1:172.21.15.150:6789/0 conn(0x7fc20c06e610 legacy=0x7fc20c06e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:47.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 1 -- 172.21.15.150:0/3406270409 >> v1:172.21.15.142:6789/0 conn(0x7fc20c137590 legacy=0x7fc20c139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:47.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:47.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 1 monclient: found mon.noname-b 2024-09-18T10:17:47.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:47.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:47.239 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.229+0000 7fc20affd700 1 -- 172.21.15.150:0/3406270409 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c0a4640 con 0x7fc20c0a4270 2024-09-18T10:17:47.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20affd700 1 -- 172.21.15.150:0/3406270409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 msgr2=0x7fc20c0a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:47.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20affd700 1 --2- 172.21.15.150:0/3406270409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 0x7fc20c0a1ce0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009910 tx=0x7fc1fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2727249c590e9e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:47.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 1 -- 172.21.15.150:0/3406270409 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00ba70 con 0x7fc20c0a4270 2024-09-18T10:17:47.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:47.241 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: dump: 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:47.242 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 5 monclient: authenticate success, global_id 74630 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 1 -- 172.21.15.150:0/3406270409 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc00bbd0 con 0x7fc20c0a4270 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 4 set_mon_vals no callback set 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 1 -- 172.21.15.150:0/3406270409 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc011420 con 0x7fc20c0a4270 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc20a7fc700 10 monclient: dump: 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:47.243 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 10 monclient: shutdown 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- 172.21.15.150:0/3406270409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 msgr2=0x7fc20c0a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 --2- 172.21.15.150:0/3406270409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 0x7fc20c0a1ce0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009910 tx=0x7fc1fc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- 172.21.15.150:0/3406270409 shutdown_connections 2024-09-18T10:17:47.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 --2- 172.21.15.150:0/3406270409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c0a4270 0x7fc20c0a1ce0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- 172.21.15.150:0/3406270409 >> 172.21.15.150:0/3406270409 conn(0x7fc20c0716e0 msgr2=0x7fc20c0a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- 172.21.15.150:0/3406270409 shutdown_connections 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- 172.21.15.150:0/3406270409 wait complete. 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 10 monclient: build_initial_monmap 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 10 monclient: monmap: 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:47.245 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:47.246 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 Processor -- start 2024-09-18T10:17:47.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 1 -- start start 2024-09-18T10:17:47.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.233+0000 7fc21130c700 10 monclient: init 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c06e610 0x7fc20c134330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-c con 0x7fc20c06e610 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-a con 0x7fc20c0a4270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:47.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 0x7fc20c11d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): picked mon.noname-b con 0x7fc20c137590 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c06f1d0 con 0x7fc20c0a4270 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c120720 con 0x7fc20c137590 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c142860 con 0x7fc20c06e610 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc21130c700 10 monclient(hunting): authenticate will time out at 1349.332836s 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20b7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 msgr2=0x7fc20c11d370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 0x7fc20c11d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:47.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c06e610 0x7fc20c134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47382/0 (socket says 172.21.15.150:47382) 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 -- 172.21.15.150:0/3024723097 learned_addr learned my addr 172.21.15.150:0/3024723097 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): get_auth_request con 0x7fc20c0a4270 auth_method 0 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): get_auth_request con 0x7fc20c06e610 auth_method 0 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:47.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc20affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient(hunting): handle_auth_done global_id 94239 payload 824 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 -- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c06e610 msgr2=0x7fc20c134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c06e610 0x7fc20c134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 -- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 msgr2=0x7fc20c11d370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 0x7fc20c11d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient: _finish_hunting 0 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 monclient: found mon.noname-a 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:47.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.237+0000 7fc2027fc700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc009420 con 0x7fc20c0a4270 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2027fc700 1 -- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 msgr2=0x7fc20c11ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2027fc700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400b7e0 tx=0x7fc1f4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4003e20 con 0x7fc20c0a4270 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: dump: 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.255 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 5 monclient: authenticate success, global_id 94239 2024-09-18T10:17:47.256 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _renew_subs 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4006150 con 0x7fc20c0a4270 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c11f5e0 con 0x7fc20c0a4270 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4006510 con 0x7fc20c0a4270 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 10 monclient: dump: 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:47.257 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _renew_subs 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc209ffb700 4 set_mon_vals no callback set 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c1350d0 con 0x7fc20c0a4270 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc1f40103c0 con 0x7fc20c0a4270 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 0x7fc1f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c11ea10 con 0x7fc20c0a4270 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(242..242 src has 176..242) v4 ==== 10823+0+0 (secure 0 0 0) 0x7fc1f4049310 con 0x7fc20c0a4270 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc20affd700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 0x7fc1f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.241+0000 7fc20affd700 10 monclient: get_auth_request con 0x7fc1f0035a70 auth_method 0 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.245+0000 7fc20affd700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 0x7fc1f0037f20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0093f0 tx=0x7fc1fc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:47.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.253+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc1f4017070 con 0x7fc20c0a4270 2024-09-18T10:17:47.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.253+0000 7fc2037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:47.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.253+0000 7fc2037fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:47.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.421+0000 7fc21130c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:47.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.421+0000 7fc21130c700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:47.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.421+0000 7fc21130c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:47.427 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:47.421+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fc20c063ed0 con 0x7fc20c0a4270 2024-09-18T10:17:48.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.001+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 removed item id -79 name 'h21' from crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc1f400e8a0 con 0x7fc20c0a4270 2024-09-18T10:17:48.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.001+0000 7fc2037fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:48.005 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.001+0000 7fc2037fe700 10 monclient: _finish_command 2 = system:0 removed item id -79 name 'h21' from crush map 2024-09-18T10:17:48.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc21130c700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:48.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc21130c700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:48.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc21130c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:48.115 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h21"} v 0) v1 -- 0x7fc20c0008d0 con 0x7fc20c0a4270 2024-09-18T10:17:48.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc2037fe700 1 -- 172.21.15.150:0/3024723097 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h21"}]=0 device 'h21' does not appear in the crush map v243) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc1f400e2c0 con 0x7fc20c0a4270 2024-09-18T10:17:48.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc2037fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h21"}] 2024-09-18T10:17:48.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.109+0000 7fc2037fe700 10 monclient: _finish_command 3 = system:0 device 'h21' does not appear in the crush map 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 msgr2=0x7fc1f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 0x7fc1f0037f20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0093f0 tx=0x7fc1fc016000 comp rx=0 tx=0).stop 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 10 monclient: shutdown 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 msgr2=0x7fc20c11ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400b7e0 tx=0x7fc1f4009220 comp rx=0 tx=0).stop 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 shutdown_connections 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc1f0035a70 0x7fc1f0037f20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.119 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc20c137590 0x7fc20c11d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc20c0a4270 0x7fc20c11ce30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 --2- 172.21.15.150:0/3024723097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc20c06e610 0x7fc20c134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 >> 172.21.15.150:0/3024723097 conn(0x7fc20c0716e0 msgr2=0x7fc20c09ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 shutdown_connections 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.113+0000 7fc21130c700 1 -- 172.21.15.150:0/3024723097 wait complete. 2024-09-18T10:17:48.120 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h21' does not appear in the crush map 2024-09-18T10:17:48.129 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:48.129 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h22 2024-09-18T10:17:48.224 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.217+0000 7fa52060d700 10 monclient: get_monmap_and_config 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.217+0000 7fa52060d700 10 monclient: build_initial_monmap 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.217+0000 7fa52060d700 10 monclient: monmap: 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:48.223336+0000 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:48.223336+0000 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:48.225 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:48.226 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:48.226 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 Processor -- start 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 -- start start 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient: init 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa51809acd0 0x7fa518096a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-c con 0x7fa51809acd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa51809bfd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa51809cab0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518095e30 con 0x7fa51809bfd0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5180958e0 con 0x7fa51809cab0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51809b0a0 con 0x7fa51809acd0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa52060d700 10 monclient(hunting): authenticate will time out at 1350.316879s 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51ebaa700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fa51809cab0 legacy=0x7fa518137790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51e3a9700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fa51809acd0 msgr2=0x7fa518096a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51e3a9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa51809acd0 0x7fa518096a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51dba8700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa51809bfd0 0x7fa51809c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:35252/0 (socket says 172.21.15.150:35252) 2024-09-18T10:17:48.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51dba8700 1 -- 172.21.15.150:0/3995480658 learned_addr learned my addr 172.21.15.150:0/3995480658 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.221+0000 7fa51dba8700 1 -- 172.21.15.150:0/3995480658 >> v1:172.21.15.142:6789/0 conn(0x7fa51809bfd0 legacy=0x7fa51809c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094353978 0 0) 0x7fa518095e30 con 0x7fa51809bfd0 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): my global_id is 94242 2024-09-18T10:17:48.229 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504003570 con 0x7fa51809bfd0 2024-09-18T10:17:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589825003 0 0) 0x7fa504003570 con 0x7fa51809bfd0 2024-09-18T10:17:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 >> v2:172.21.15.150:3300/0 conn(0x7fa51809acd0 msgr2=0x7fa518096a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 --2- 172.21.15.150:0/3995480658 >> v2:172.21.15.150:3300/0 conn(0x7fa51809acd0 0x7fa518096a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 >> v1:172.21.15.150:6789/0 conn(0x7fa51809cab0 legacy=0x7fa518137790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: _finish_hunting 0 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518095b50 con 0x7fa51809bfd0 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: _finish_auth 0 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa52060d700 5 monclient: authenticate success, global_id 94242 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fa5140017a0 con 0x7fa51809bfd0 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa52060d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:17:48.233 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: dump: 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:48.234 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 >> v1:172.21.15.142:6789/0 conn(0x7fa51809bfd0 legacy=0x7fa51809c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa504003a60 0x7fa504003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): picked mon.b con 0x7fa504003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa504004810 0x7fa504004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): picked mon.c con 0x7fa504004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 0x7fa50400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): picked mon.a con 0x7fa5040053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51809b0a0 con 0x7fa5040053c0 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51e3a9700 1 -- 172.21.15.150:0/3995480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa504003a60 msgr2=0x7fa504003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50400c380 con 0x7fa504003a60 2024-09-18T10:17:48.235 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51e3a9700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa504003a60 0x7fa504003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50400c540 con 0x7fa504004810 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51d3a7700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa52060d700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 0x7fa50400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa504004810 0x7fa504004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): get_auth_request con 0x7fa5040053c0 auth_method 0 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): get_auth_request con 0x7fa504004810 auth_method 0 2024-09-18T10:17:48.236 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_done global_id 94242 payload 293 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 -- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa504004810 msgr2=0x7fa504004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa504004810 0x7fa504004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.237 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 -- 172.21.15.150:0/3995480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa504003a60 msgr2=0x7fa504003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa504003a60 0x7fa504003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient: _finish_hunting 0 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 monclient: found mon.a 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 -- 172.21.15.150:0/3995480658 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa504003570 con 0x7fa5040053c0 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 10 monclient: _finish_auth 0 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:48.238 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.225+0000 7fa51ebaa700 1 -- 172.21.15.150:0/3995480658 reap_dead start 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 -- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 msgr2=0x7fa50400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 0x7fa50400baa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f90 tx=0x7fa5100095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4f11b038e9d5db server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510009900 con 0x7fa5040053c0 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: dump: 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:48.243 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51000be70 con 0x7fa5040053c0 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 4 set_mon_vals no callback set 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 1 -- 172.21.15.150:0/3995480658 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510015680 con 0x7fa5040053c0 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:48.244 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51d3a7700 10 monclient: dump: 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: shutdown 2024-09-18T10:17:48.245 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 msgr2=0x7fa50400baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 0x7fa50400baa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa510009f90 tx=0x7fa5100095d0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- 172.21.15.150:0/3995480658 shutdown_connections 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 --2- 172.21.15.150:0/3995480658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5040053c0 0x7fa50400baa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- 172.21.15.150:0/3995480658 >> 172.21.15.150:0/3995480658 conn(0x7fa5180a15e0 msgr2=0x7fa5180a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- 172.21.15.150:0/3995480658 shutdown_connections 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- 172.21.15.150:0/3995480658 wait complete. 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: build_initial_monmap 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: monmap: 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:48.246 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 Processor -- start 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- start start 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: init 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 0x7fa5181418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-b con 0x7fa5040053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:48.247 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa518094260 0x7fa518094630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-a con 0x7fa518094260 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): picked mon.noname-c con 0x7fa51808df40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518094b70 con 0x7fa518094260 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518098410 con 0x7fa5040053c0 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa518097ab0 con 0x7fa51808df40 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa52060d700 10 monclient(hunting): authenticate will time out at 1350.326952s 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51e3a9700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 msgr2=0x7fa5181418d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:48.248 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51e3a9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 0x7fa5181418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa518094260 0x7fa518094630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:46456/0 (socket says 172.21.15.150:46456) 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 -- 172.21.15.150:0/1862449832 learned_addr learned my addr 172.21.15.150:0/1862449832 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): get_auth_request con 0x7fa51808df40 auth_method 0 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): get_auth_request con 0x7fa518094260 auth_method 0 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:48.249 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51dba8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient(hunting): handle_auth_done global_id 74636 payload 824 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 -- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 msgr2=0x7fa5181418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 0x7fa5181418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 -- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa518094260 msgr2=0x7fa518094630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa518094260 0x7fa518094630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient: _finish_hunting 0 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 monclient: found mon.noname-c 2024-09-18T10:17:48.250 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.229+0000 7fa51ebaa700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518091480 con 0x7fa51808df40 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa51ebaa700 1 -- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 msgr2=0x7fa51808e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa51ebaa700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa510009fd0 tx=0x7fa510015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa510019030 con 0x7fa51808df40 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: dump: 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.251 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 5 monclient: authenticate success, global_id 74636 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 10 monclient: _renew_subs 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51000c900 con 0x7fa51808df40 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51808f950 con 0x7fa51808df40 2024-09-18T10:17:48.252 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50ffff700 4 set_mon_vals no callback set 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 10 monclient: _renew_subs 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51001c040 con 0x7fa51808df40 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181426a0 con 0x7fa51808df40 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.233+0000 7fa50e7fc700 10 monclient: dump: 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:48.253 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa52060d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa52060d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa52060d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa518063ed0 con 0x7fa51808df40 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa510012070 con 0x7fa51808df40 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa50e7fc700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 0x7fa4fc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(243..243 src has 176..243) v4 ==== 10610+0+0 (secure 0 0 0) 0x7fa51004af80 con 0x7fa51808df40 2024-09-18T10:17:48.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa51e3a9700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 0x7fa4fc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:48.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.237+0000 7fa51e3a9700 10 monclient: get_auth_request con 0x7fa4fc035a70 auth_method 0 2024-09-18T10:17:48.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.241+0000 7fa51e3a9700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 0x7fa4fc037f20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa5080039d0 tx=0x7fa508006e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:48.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.249+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa510020560 con 0x7fa51808df40 2024-09-18T10:17:48.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.249+0000 7fa50e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:48.255 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.249+0000 7fa50e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:48.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.421+0000 7fa52060d700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:48.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.421+0000 7fa52060d700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:48.425 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.421+0000 7fa52060d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:48.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:48.421+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7fa518067b10 con 0x7fa51808df40 2024-09-18T10:17:49.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.413+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 removed item id -82 name 'h22' from crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa510006170 con 0x7fa51808df40 2024-09-18T10:17:49.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.413+0000 7fa50e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:49.421 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.413+0000 7fa50e7fc700 10 monclient: _finish_command 2 = system:0 removed item id -82 name 'h22' from crush map 2024-09-18T10:17:49.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa52060d700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:49.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa52060d700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:49.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa52060d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h22"} v 0) v1 -- 0x7fa518068c60 con 0x7fa51808df40 2024-09-18T10:17:49.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa50e7fc700 1 -- 172.21.15.150:0/1862449832 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h22"}]=0 device 'h22' does not appear in the crush map v245) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa510006350 con 0x7fa51808df40 2024-09-18T10:17:49.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa50e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h22"}] 2024-09-18T10:17:49.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.525+0000 7fa50e7fc700 10 monclient: _finish_command 3 = system:0 device 'h22' does not appear in the crush map 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 msgr2=0x7fa4fc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 0x7fa4fc037f20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa5080039d0 tx=0x7fa508006e60 comp rx=0 tx=0).stop 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 10 monclient: shutdown 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 msgr2=0x7fa51808e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:49.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa510009fd0 tx=0x7fa510015000 comp rx=0 tx=0).stop 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 shutdown_connections 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa4fc035a70 0x7fa4fc037f20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa51808df40 0x7fa51808e370 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa518094260 0x7fa518094630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 --2- 172.21.15.150:0/1862449832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5040053c0 0x7fa5181418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 >> 172.21.15.150:0/1862449832 conn(0x7fa5180a15e0 msgr2=0x7fa51809f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 shutdown_connections 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.529+0000 7fa52060d700 1 -- 172.21.15.150:0/1862449832 wait complete. 2024-09-18T10:17:49.536 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h22' does not appear in the crush map 2024-09-18T10:17:49.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:49.546 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h23 2024-09-18T10:17:49.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: get_monmap_and_config 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: build_initial_monmap 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: monmap: 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:49.644442+0000 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:49.644442+0000 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:49.646 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:49.647 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:49.647 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:49.647 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:49.647 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 Processor -- start 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 -- start start 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: init 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T10:17:49.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce5009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-b con 0x7fce500a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce500a04f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce501379b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5009dae0 con 0x7fce500a04f0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce500a1590 con 0x7fce501379b0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500a21d0 con 0x7fce500a0120 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce57ba7700 10 monclient(hunting): authenticate will time out at 1351.738552s 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce56144700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fce501379b0 legacy=0x7fce50139d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce5009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:49.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55142700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fce500a04f0 0x7fce500a0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33960/0 (socket says 172.21.15.150:33960) 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55142700 1 -- 172.21.15.150:0/572561154 learned_addr learned my addr 172.21.15.150:0/572561154 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): get_auth_request con 0x7fce500a0120 auth_method 0 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient(hunting): handle_auth_done global_id 74639 payload 274 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 1 -- 172.21.15.150:0/572561154 >> v1:172.21.15.150:6789/0 conn(0x7fce501379b0 legacy=0x7fce50139d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 1 -- 172.21.15.150:0/572561154 >> v1:172.21.15.142:6789/0 conn(0x7fce500a04f0 legacy=0x7fce500a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient: _finish_hunting 0 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 1 monclient: found mon.noname-b 2024-09-18T10:17:49.651 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:49.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.641+0000 7fce55943700 1 -- 172.21.15.150:0/572561154 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500a1da0 con 0x7fce500a0120 2024-09-18T10:17:49.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce55943700 1 -- 172.21.15.150:0/572561154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 msgr2=0x7fce5009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:49.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce55943700 1 --2- 172.21.15.150:0/572561154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce5009e7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009910 tx=0x7fce4c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48182839b3925dab server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:49.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 1 -- 172.21.15.150:0/572561154 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00ba70 con 0x7fce500a0120 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: dump: 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:49.654 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: _finish_auth 0 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 5 monclient: authenticate success, global_id 74639 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 1 -- 172.21.15.150:0/572561154 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c00bbd0 con 0x7fce500a0120 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 4 set_mon_vals no callback set 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 1 -- 172.21.15.150:0/572561154 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011420 con 0x7fce500a0120 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce54941700 10 monclient: dump: 2024-09-18T10:17:49.655 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 10 monclient: shutdown 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 -- 172.21.15.150:0/572561154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 msgr2=0x7fce5009e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:49.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 --2- 172.21.15.150:0/572561154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce5009e7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009910 tx=0x7fce4c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 -- 172.21.15.150:0/572561154 shutdown_connections 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 --2- 172.21.15.150:0/572561154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce5009e7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 -- 172.21.15.150:0/572561154 >> 172.21.15.150:0/572561154 conn(0x7fce500a5830 msgr2=0x7fce500a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 -- 172.21.15.150:0/572561154 shutdown_connections 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.645+0000 7fce57ba7700 1 -- 172.21.15.150:0/572561154 wait complete. 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient: build_initial_monmap 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient: monmap: 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:49.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.658 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:49.658 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:49.658 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:49.658 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 Processor -- start 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 -- start start 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient: init 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-c con 0x7fce500a0120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 0x7fce50142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-b con 0x7fce500a04f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 0x7fce501428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): picked mon.noname-a con 0x7fce501379b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5012fc30 con 0x7fce501379b0 2024-09-18T10:17:49.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5012fe30 con 0x7fce500a04f0 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce50130e20 con 0x7fce500a0120 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce57ba7700 10 monclient(hunting): authenticate will time out at 1351.745838s 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55142700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 msgr2=0x7fce50142360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55142700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 0x7fce50142360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 0x7fce501428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44960/0 (socket says 172.21.15.150:44960) 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 -- 172.21.15.150:0/3052993435 learned_addr learned my addr 172.21.15.150:0/3052993435 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:49.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): get_auth_request con 0x7fce500a0120 auth_method 0 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): get_auth_request con 0x7fce501379b0 auth_method 0 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:49.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient(hunting): handle_auth_done global_id 74642 payload 824 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 -- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 msgr2=0x7fce50142360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 0x7fce50142360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 -- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 msgr2=0x7fce501428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 0x7fce501428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient: _finish_hunting 0 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 monclient: found mon.noname-c 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce55943700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c009420 con 0x7fce500a0120 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.649+0000 7fce56144700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 0x7fce501428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce55943700 1 -- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 msgr2=0x7fce50141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:49.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce55943700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0093f0 tx=0x7fce4c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011420 con 0x7fce500a0120 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: dump: 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:49.666 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 5 monclient: authenticate success, global_id 74642 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _renew_subs 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5009bfe0 con 0x7fce500a0120 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c01b680 con 0x7fce500a0120 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c01ba40 con 0x7fce500a0120 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:49.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 10 monclient: dump: 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce43fff700 4 set_mon_vals no callback set 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _renew_subs 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5012e4b0 con 0x7fce500a0120 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5012e6a0 con 0x7fce500a0120 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.653+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fce4c0193b0 con 0x7fce500a0120 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.657+0000 7fce41ffb700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 0x7fce34037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.657+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(245..245 src has 176..245) v4 ==== 10397+0+0 (secure 0 0 0) 0x7fce4c04b1c0 con 0x7fce500a0120 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.657+0000 7fce55142700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 0x7fce34037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:49.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.657+0000 7fce55142700 10 monclient: get_auth_request con 0x7fce34035a70 auth_method 0 2024-09-18T10:17:49.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.657+0000 7fce55142700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 0x7fce34037f20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fce44004860 tx=0x7fce44007b80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:49.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.665+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fce4c01bdb0 con 0x7fce500a0120 2024-09-18T10:17:49.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.665+0000 7fce41ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:49.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.665+0000 7fce41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:49.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.833+0000 7fce57ba7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:49.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.833+0000 7fce57ba7700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:49.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.833+0000 7fce57ba7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:49.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:49.833+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7fce5012e6a0 con 0x7fce500a0120 2024-09-18T10:17:50.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.421+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 removed item id -85 name 'h23' from crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7fce4c024350 con 0x7fce500a0120 2024-09-18T10:17:50.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.421+0000 7fce41ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:50.426 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.421+0000 7fce41ffb700 10 monclient: _finish_command 2 = system:0 removed item id -85 name 'h23' from crush map 2024-09-18T10:17:50.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.525+0000 7fce57ba7700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:50.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.525+0000 7fce57ba7700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:50.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.525+0000 7fce57ba7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.533 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.525+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h23"} v 0) v1 -- 0x7fce5012e6a0 con 0x7fce500a0120 2024-09-18T10:17:50.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.529+0000 7fce41ffb700 1 -- 172.21.15.150:0/3052993435 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h23"}]=0 device 'h23' does not appear in the crush map v246) v1 ==== 120+0+0 (secure 0 0 0) 0x7fce4c024530 con 0x7fce500a0120 2024-09-18T10:17:50.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.529+0000 7fce41ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h23"}] 2024-09-18T10:17:50.535 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.529+0000 7fce41ffb700 10 monclient: _finish_command 3 = system:0 device 'h23' does not appear in the crush map 2024-09-18T10:17:50.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 msgr2=0x7fce34037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:50.538 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 0x7fce34037f20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fce44004860 tx=0x7fce44007b80 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 10 monclient: shutdown 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 msgr2=0x7fce50141e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0093f0 tx=0x7fce4c016000 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 shutdown_connections 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce34035a70 0x7fce34037f20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce501379b0 0x7fce501428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce500a04f0 0x7fce50142360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 --2- 172.21.15.150:0/3052993435 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce500a0120 0x7fce50141e20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 >> 172.21.15.150:0/3052993435 conn(0x7fce500a5830 msgr2=0x7fce5009ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 shutdown_connections 2024-09-18T10:17:50.539 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.533+0000 7fce57ba7700 1 -- 172.21.15.150:0/3052993435 wait complete. 2024-09-18T10:17:50.540 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h23' does not appear in the crush map 2024-09-18T10:17:50.549 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:50.549 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h24 2024-09-18T10:17:50.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: get_monmap_and_config 2024-09-18T10:17:50.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: build_initial_monmap 2024-09-18T10:17:50.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: monmap: 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:50.649127+0000 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:50.649127+0000 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:50.651 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 Processor -- start 2024-09-18T10:17:50.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 -- start start 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: init 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-b con 0x7ff9ac0a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff9ac06e610 0x7ff9ac06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-a con 0x7ff9ac06e610 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 0x7ff9ac1326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-c con 0x7ff9ac132260 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac06afe0 con 0x7ff9ac06e610 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0a2220 con 0x7ff9ac132260 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0a2830 con 0x7ff9ac0a4270 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b2a4a700 10 monclient(hunting): authenticate will time out at 1352.742784s 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b0fe7700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 msgr2=0x7ff9ac1326c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9b0fe7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 0x7ff9ac1326c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff9ac06e610 0x7ff9ac06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44968/0 (socket says 172.21.15.150:44968) 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 1 -- 172.21.15.150:0/2769796983 learned_addr learned my addr 172.21.15.150:0/2769796983 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): get_auth_request con 0x7ff9ac0a4270 auth_method 0 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:50.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): get_auth_request con 0x7ff9ac06e610 auth_method 0 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:50.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_done global_id 94251 payload 274 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 1 -- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 msgr2=0x7ff9ac0a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 1 -- 172.21.15.150:0/2769796983 >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 msgr2=0x7ff9ac1326c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/2769796983 >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 0x7ff9ac1326c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.645+0000 7ff9a37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 1 -- 172.21.15.150:0/2769796983 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0a4640 con 0x7ff9ac06e610 2024-09-18T10:17:50.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9abfff700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:50.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 1 -- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac06e610 msgr2=0x7ff9ac06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:50.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac06e610 0x7ff9ac06e9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0a3510 tx=0x7ff9940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fa1a914de69440 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 1 -- 172.21.15.150:0/2769796983 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994011840 con 0x7ff9ac06e610 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: dump: 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:50.661 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 5 monclient: authenticate success, global_id 94251 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 1 -- 172.21.15.150:0/2769796983 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940119a0 con 0x7ff9ac06e610 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 4 set_mon_vals no callback set 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 1 -- 172.21.15.150:0/2769796983 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994011d40 con 0x7ff9ac06e610 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9ab7fe700 10 monclient: dump: 2024-09-18T10:17:50.662 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 10 monclient: shutdown 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac06e610 msgr2=0x7ff9ac06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac06e610 0x7ff9ac06e9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0a3510 tx=0x7ff9940091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/2769796983 shutdown_connections 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/2769796983 >> v2:172.21.15.150:3300/0 conn(0x7ff9ac132260 0x7ff9ac1326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac06e610 0x7ff9ac06e9e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/2769796983 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac0a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/2769796983 >> 172.21.15.150:0/2769796983 conn(0x7ff9ac0716e0 msgr2=0x7ff9ac0a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.649+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/2769796983 shutdown_connections 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/2769796983 wait complete. 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient: build_initial_monmap 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient: monmap: 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:50.664 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 Processor -- start 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 -- start start 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient: init 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 0x7ff9ac141790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-b con 0x7ff9ac06e610 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-c con 0x7ff9ac0a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac132260 0x7ff9ac142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9abfff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 msgr2=0x7ff9ac141790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): picked mon.noname-a con 0x7ff9ac132260 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac121560 con 0x7ff9ac132260 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac1212f0 con 0x7ff9ac06e610 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 0x7ff9ac141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac06e1c0 con 0x7ff9ac0a4270 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b2a4a700 10 monclient(hunting): authenticate will time out at 1352.750055s 2024-09-18T10:17:50.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac132260 0x7ff9ac142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44980/0 (socket says 172.21.15.150:44980) 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 -- 172.21.15.150:0/3194875862 learned_addr learned my addr 172.21.15.150:0/3194875862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): get_auth_request con 0x7ff9ac0a4270 auth_method 0 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): get_auth_request con 0x7ff9ac132260 auth_method 0 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:50.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9b0fe7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient(hunting): handle_auth_done global_id 74648 payload 824 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 -- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 msgr2=0x7ff9ac141790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 0x7ff9ac141790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 -- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac132260 msgr2=0x7ff9ac142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac132260 0x7ff9ac142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 monclient: found mon.noname-c 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.653+0000 7ff9a37fe700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0094f0 con 0x7ff9ac0a4270 2024-09-18T10:17:50.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a37fe700 1 -- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 msgr2=0x7ff9ac141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:50.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a37fe700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff994009f50 tx=0x7ff994017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:50.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400be60 con 0x7ff9ac0a4270 2024-09-18T10:17:50.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:50.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: dump: 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:50.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 5 monclient: authenticate success, global_id 74648 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff994024440 con 0x7ff9ac0a4270 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _renew_subs 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff994009420 con 0x7ff9ac0a4270 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994024800 con 0x7ff9ac0a4270 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 10 monclient: dump: 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:50.672 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9aaffd700 4 set_mon_vals no callback set 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _renew_subs 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac122420 con 0x7ff9ac0a4270 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:50.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff99402d430 con 0x7ff9ac0a4270 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.657+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac06ad00 con 0x7ff9ac0a4270 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.661+0000 7ff9a8ff9700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 0x7ff990037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.661+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(246..246 src has 176..246) v4 ==== 10184+0+0 (secure 0 0 0) 0x7ff99404b6e0 con 0x7ff9ac0a4270 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.661+0000 7ff9abfff700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 0x7ff990037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.661+0000 7ff9abfff700 10 monclient: get_auth_request con 0x7ff990035aa0 auth_method 0 2024-09-18T10:17:50.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.661+0000 7ff9abfff700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 0x7ff990037f50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff99c006fd0 tx=0x7ff99c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:50.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.673+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff994022da0 con 0x7ff9ac0a4270 2024-09-18T10:17:50.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.673+0000 7ff9a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:50.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.673+0000 7ff9a8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:50.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.845+0000 7ff9b2a4a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:50.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.845+0000 7ff9b2a4a700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:50.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.845+0000 7ff9b2a4a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:50.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:50.845+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7ff9ac063ed0 con 0x7ff9ac0a4270 2024-09-18T10:17:51.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.429+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 removed item id -88 name 'h24' from crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff994015030 con 0x7ff9ac0a4270 2024-09-18T10:17:51.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.429+0000 7ff9a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:51.435 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.429+0000 7ff9a8ff9700 10 monclient: _finish_command 2 = system:0 removed item id -88 name 'h24' from crush map 2024-09-18T10:17:51.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9b2a4a700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:51.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9b2a4a700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:51.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9b2a4a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.545 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h24"} v 0) v1 -- 0x7ff9ac1208f0 con 0x7ff9ac0a4270 2024-09-18T10:17:51.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9a8ff9700 1 -- 172.21.15.150:0/3194875862 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h24"}]=0 device 'h24' does not appear in the crush map v247) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff994015020 con 0x7ff9ac0a4270 2024-09-18T10:17:51.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9a8ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h24"}] 2024-09-18T10:17:51.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.541+0000 7ff9a8ff9700 10 monclient: _finish_command 3 = system:0 device 'h24' does not appear in the crush map 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 msgr2=0x7ff990037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 0x7ff990037f50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff99c006fd0 tx=0x7ff99c009250 comp rx=0 tx=0).stop 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 10 monclient: shutdown 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 msgr2=0x7ff9ac141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff994009f50 tx=0x7ff994017000 comp rx=0 tx=0).stop 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 shutdown_connections 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff990035aa0 0x7ff990037f50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9ac132260 0x7ff9ac142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9ac0a4270 0x7ff9ac141cd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 --2- 172.21.15.150:0/3194875862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9ac06e610 0x7ff9ac141790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 >> 172.21.15.150:0/3194875862 conn(0x7ff9ac0716e0 msgr2=0x7ff9ac09cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:51.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 shutdown_connections 2024-09-18T10:17:51.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.545+0000 7ff9b2a4a700 1 -- 172.21.15.150:0/3194875862 wait complete. 2024-09-18T10:17:51.554 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h24' does not appear in the crush map 2024-09-18T10:17:51.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:51.566 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h25 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: get_monmap_and_config 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: build_initial_monmap 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: monmap: 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:51.660681+0000 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:51.660681+0000 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.662 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:51.663 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:51.663 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:51.663 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:51.663 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:51.663 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 Processor -- start 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 -- start start 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: init 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:17:51.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf4409cf90 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 0x7faf440a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-c con 0x7faf4409e480 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-b con 0x7faf440990a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4409e060 con 0x7faf4409cf90 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf440a66c0 con 0x7faf4409e480 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4409d750 con 0x7faf440990a0 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf437fe700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7faf4409cf90 legacy=0x7faf440a0290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf49fe1700 10 monclient(hunting): authenticate will time out at 1353.754294s 2024-09-18T10:17:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf42ffd700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 msgr2=0x7faf440a09a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf42ffd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 0x7faf440a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:44992/0 (socket says 172.21.15.150:44992) 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 -- 172.21.15.150:0/1680598502 learned_addr learned my addr 172.21.15.150:0/1680598502 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): get_auth_request con 0x7faf440990a0 auth_method 0 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient(hunting): handle_auth_done global_id 74651 payload 274 2024-09-18T10:17:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 -- 172.21.15.150:0/1680598502 >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 msgr2=0x7faf440a09a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 --2- 172.21.15.150:0/1680598502 >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 0x7faf440a09a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 -- 172.21.15.150:0/1680598502 >> v1:172.21.15.150:6789/0 conn(0x7faf4409cf90 legacy=0x7faf440a0290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 monclient: found mon.noname-b 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.657+0000 7faf43fff700 1 -- 172.21.15.150:0/1680598502 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4409d360 con 0x7faf440990a0 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf43fff700 1 -- 172.21.15.150:0/1680598502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 msgr2=0x7faf44139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf43fff700 1 --2- 172.21.15.150:0/1680598502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009f20 tx=0x7faf3c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43f5d866732a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 1 -- 172.21.15.150:0/1680598502 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c00e070 con 0x7faf440990a0 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: dump: 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 1 -- 172.21.15.150:0/1680598502 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3c00bc90 con 0x7faf440990a0 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 1 -- 172.21.15.150:0/1680598502 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c012660 con 0x7faf440990a0 2024-09-18T10:17:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf427fc700 10 monclient: dump: 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:51.674 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 5 monclient: authenticate success, global_id 74651 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 4 set_mon_vals no callback set 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 10 monclient: shutdown 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 -- 172.21.15.150:0/1680598502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 msgr2=0x7faf44139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 --2- 172.21.15.150:0/1680598502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009f20 tx=0x7faf3c009840 comp rx=0 tx=0).stop 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 -- 172.21.15.150:0/1680598502 shutdown_connections 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 --2- 172.21.15.150:0/1680598502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf440990a0 0x7faf44139a30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 --2- 172.21.15.150:0/1680598502 >> v2:172.21.15.150:3300/0 conn(0x7faf4409e480 0x7faf440a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 -- 172.21.15.150:0/1680598502 >> 172.21.15.150:0/1680598502 conn(0x7faf440a58a0 msgr2=0x7faf440a5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 -- 172.21.15.150:0/1680598502 shutdown_connections 2024-09-18T10:17:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 1 -- 172.21.15.150:0/1680598502 wait complete. 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 10 monclient: build_initial_monmap 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.661+0000 7faf49fe1700 10 monclient: monmap: 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 Processor -- start 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 -- start start 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient: init 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 0x7faf4408ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-b con 0x7faf440990a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-c con 0x7faf4409cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf4409e480 0x7faf4408d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): picked mon.noname-a con 0x7faf4409e480 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf440907e0 con 0x7faf4409e480 2024-09-18T10:17:51.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4409be10 con 0x7faf440990a0 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf44090500 con 0x7faf4409cf90 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf49fe1700 10 monclient(hunting): authenticate will time out at 1353.761261s 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf437fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 msgr2=0x7faf4408ce00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf437fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 0x7faf4408ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:51.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:45004/0 (socket says 172.21.15.150:45004) 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 -- 172.21.15.150:0/2732762385 learned_addr learned my addr 172.21.15.150:0/2732762385 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): get_auth_request con 0x7faf4409cf90 auth_method 0 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf4409e480 0x7faf4408d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): get_auth_request con 0x7faf4409e480 auth_method 0 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:51.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient(hunting): handle_auth_done global_id 74654 payload 824 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 -- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 msgr2=0x7faf4408ce00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 0x7faf4408ce00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 -- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf4409e480 msgr2=0x7faf4408d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf4409e480 0x7faf4408d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 monclient: found mon.noname-c 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.665+0000 7faf42ffd700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3c009ae0 con 0x7faf4409cf90 2024-09-18T10:17:51.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf42ffd700 1 -- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 msgr2=0x7faf4408d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:51.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf42ffd700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faf3800b7e0 tx=0x7faf38009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38003e20 con 0x7faf4409cf90 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: dump: 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:51.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 5 monclient: authenticate success, global_id 74654 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf38006150 con 0x7faf4409cf90 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _renew_subs 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf44090220 con 0x7faf4409cf90 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38006510 con 0x7faf4409cf90 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:51.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 10 monclient: dump: 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf417fa700 4 set_mon_vals no callback set 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _renew_subs 2024-09-18T10:17:51.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf440983d0 con 0x7faf4409cf90 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf440a0ee0 con 0x7faf4409cf90 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7faf380103c0 con 0x7faf4409cf90 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 0x7faf30037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(247..247 src has 176..247) v4 ==== 9971+0+0 (secure 0 0 0) 0x7faf38049010 con 0x7faf4409cf90 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf437fe700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 0x7faf30037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.669+0000 7faf437fe700 10 monclient: get_auth_request con 0x7faf30035aa0 auth_method 0 2024-09-18T10:17:51.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.673+0000 7faf437fe700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 0x7faf30037f50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faf34006fd0 tx=0x7faf34006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:51.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.681+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7faf3800e8a0 con 0x7faf4409cf90 2024-09-18T10:17:51.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.681+0000 7faf2b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:51.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.681+0000 7faf2b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:51.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.849+0000 7faf49fe1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:51.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.849+0000 7faf49fe1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:51.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.849+0000 7faf49fe1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:51.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:51.849+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7faf44063ed0 con 0x7faf4409cf90 2024-09-18T10:17:52.441 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.437+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 removed item id -91 name 'h25' from crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7faf38017030 con 0x7faf4409cf90 2024-09-18T10:17:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.437+0000 7faf2b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:52.442 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.437+0000 7faf2b7fe700 10 monclient: _finish_command 2 = system:0 removed item id -91 name 'h25' from crush map 2024-09-18T10:17:52.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf49fe1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:52.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf49fe1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:52.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf49fe1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h25"} v 0) v1 -- 0x7faf440008d0 con 0x7faf4409cf90 2024-09-18T10:17:52.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf2b7fe700 1 -- 172.21.15.150:0/2732762385 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h25"}]=0 device 'h25' does not appear in the crush map v248) v1 ==== 120+0+0 (secure 0 0 0) 0x7faf38017020 con 0x7faf4409cf90 2024-09-18T10:17:52.552 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf2b7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h25"}] 2024-09-18T10:17:52.553 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.545+0000 7faf2b7fe700 10 monclient: _finish_command 3 = system:0 device 'h25' does not appear in the crush map 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 msgr2=0x7faf30037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 0x7faf30037f50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7faf34006fd0 tx=0x7faf34006e10 comp rx=0 tx=0).stop 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 10 monclient: shutdown 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 msgr2=0x7faf4408d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:52.555 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faf3800b7e0 tx=0x7faf38009220 comp rx=0 tx=0).stop 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 shutdown_connections 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7faf30035aa0 0x7faf30037f50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faf4409e480 0x7faf4408d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faf4409cf90 0x7faf4408d340 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 --2- 172.21.15.150:0/2732762385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf440990a0 0x7faf4408ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 >> 172.21.15.150:0/2732762385 conn(0x7faf440a58a0 msgr2=0x7faf4409b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 shutdown_connections 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.549+0000 7faf49fe1700 1 -- 172.21.15.150:0/2732762385 wait complete. 2024-09-18T10:17:52.556 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h25' does not appear in the crush map 2024-09-18T10:17:52.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:52.565 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h26 2024-09-18T10:17:52.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: get_monmap_and_config 2024-09-18T10:17:52.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: build_initial_monmap 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: monmap: 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:52.662207+0000 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:52.662207+0000 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:52.664 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:52.665 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 Processor -- start 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 -- start start 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: init 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc5809d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-b con 0x7fcc58096a70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc5806e990 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 0x7fcc58139640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-c con 0x7fcc58097550 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5809bc90 con 0x7fcc5806e990 2024-09-18T10:17:52.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5809c7c0 con 0x7fcc58097550 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5806eea0 con 0x7fcc58096a70 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc5759e700 10 monclient(hunting): authenticate will time out at 1354.755785s 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc56d9d700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 msgr2=0x7fcc58139640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.657+0000 7fcc56d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 0x7fcc58139640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc5809d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc55d9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcc5806e990 0x7fcc58096e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33966/0 (socket says 172.21.15.150:33966) 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc55d9b700 1 -- 172.21.15.150:0/2792469618 learned_addr learned my addr 172.21.15.150:0/2792469618 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): get_auth_request con 0x7fcc58096a70 auth_method 0 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:52.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient(hunting): handle_auth_done global_id 74657 payload 274 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 -- 172.21.15.150:0/2792469618 >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 msgr2=0x7fcc58139640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 --2- 172.21.15.150:0/2792469618 >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 0x7fcc58139640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 -- 172.21.15.150:0/2792469618 >> v1:172.21.15.142:6789/0 conn(0x7fcc5806e990 legacy=0x7fcc58096e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient: _finish_hunting 0 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 monclient: found mon.noname-b 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 -- 172.21.15.150:0/2792469618 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5809c4e0 con 0x7fcc58096a70 2024-09-18T10:17:52.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 -- 172.21.15.150:0/2792469618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 msgr2=0x7fcc5809d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:52.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5659c700 1 --2- 172.21.15.150:0/2792469618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc5809d4b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc400098e0 tx=0x7fcc400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f461512af0ae443d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 1 -- 172.21.15.150:0/2792469618 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000ba70 con 0x7fcc58096a70 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: dump: 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.672 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: _finish_auth 0 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5759e700 5 monclient: authenticate success, global_id 74657 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 1 -- 172.21.15.150:0/2792469618 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4000bbf0 con 0x7fcc58096a70 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 1 -- 172.21.15.150:0/2792469618 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc40011420 con 0x7fcc58096a70 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5759e700 4 set_mon_vals no callback set 2024-09-18T10:17:52.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.661+0000 7fcc5559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5559a700 10 monclient: dump: 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: shutdown 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- 172.21.15.150:0/2792469618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 msgr2=0x7fcc5809d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2792469618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc5809d4b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc400098e0 tx=0x7fcc400091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- 172.21.15.150:0/2792469618 shutdown_connections 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2792469618 >> v2:172.21.15.150:3300/0 conn(0x7fcc58097550 0x7fcc58139640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2792469618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc5809d4b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- 172.21.15.150:0/2792469618 >> 172.21.15.150:0/2792469618 conn(0x7fcc580a12a0 msgr2=0x7fcc580a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- 172.21.15.150:0/2792469618 shutdown_connections 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- 172.21.15.150:0/2792469618 wait complete. 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: build_initial_monmap 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: monmap: 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:52.675 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 Processor -- start 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- start start 2024-09-18T10:17:52.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: init 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-a con 0x7fcc5806e990 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc58134780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-c con 0x7fcc58096a70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 0x7fcc58134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): picked mon.noname-b con 0x7fcc58097550 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc58136250 con 0x7fcc5806e990 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:52.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc58133400 con 0x7fcc58097550 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580a02f0 con 0x7fcc58096a70 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5759e700 10 monclient(hunting): authenticate will time out at 1354.763210s 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc56d9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 msgr2=0x7fcc58134cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc56d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 0x7fcc58134cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc58134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:39352/0 (socket says 172.21.15.150:39352) 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 1 -- 172.21.15.150:0/2953506091 learned_addr learned my addr 172.21.15.150:0/2953506091 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): get_auth_request con 0x7fcc5806e990 auth_method 0 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:52.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): get_auth_request con 0x7fcc58096a70 auth_method 0 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc5659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.665+0000 7fcc55d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 10 monclient(hunting): handle_auth_done global_id 74660 payload 824 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 -- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 msgr2=0x7fcc58134cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:52.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 0x7fcc58134cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 -- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 msgr2=0x7fcc58134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 monclient: found mon.noname-c 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc40009420 con 0x7fcc58096a70 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5659c700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 -- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 msgr2=0x7fcc58134780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc55d9b700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc58134780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c560 tx=0x7fcc48007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48005d40 con 0x7fcc58096a70 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:52.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: dump: 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 5 monclient: authenticate success, global_id 74660 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 10 monclient: _renew_subs 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc581377d0 con 0x7fcc58096a70 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc480070d0 con 0x7fcc58096a70 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48007490 con 0x7fcc58096a70 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc4e7fc700 10 monclient: dump: 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:52.686 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc54d99700 4 set_mon_vals no callback set 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 10 monclient: _renew_subs 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.669+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc58137470 con 0x7fcc58096a70 2024-09-18T10:17:52.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcc4801b3b0 con 0x7fcc58096a70 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc4e7fc700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 0x7fcc38037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(248..248 src has 176..248) v4 ==== 9758+0+0 (secure 0 0 0) 0x7fcc48048b90 con 0x7fcc58096a70 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5804bbf0 con 0x7fcc58096a70 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5659c700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 0x7fcc38037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.673+0000 7fcc5659c700 10 monclient: get_auth_request con 0x7fcc38035a70 auth_method 0 2024-09-18T10:17:52.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.677+0000 7fcc5659c700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 0x7fcc38037f20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcc400093c0 tx=0x7fcc40016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:52.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.685+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcc48017070 con 0x7fcc58096a70 2024-09-18T10:17:52.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.685+0000 7fcc4e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:52.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.685+0000 7fcc4e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:52.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.861+0000 7fcc5759e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:52.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.861+0000 7fcc5759e700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:52.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.861+0000 7fcc5759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:52.867 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:52.861+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fcc58063ed0 con 0x7fcc58096a70 2024-09-18T10:17:53.587 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.581+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 removed item id -94 name 'h26' from crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7fcc4801a360 con 0x7fcc58096a70 2024-09-18T10:17:53.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.581+0000 7fcc4e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:53.588 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.581+0000 7fcc4e7fc700 10 monclient: _finish_command 2 = system:0 removed item id -94 name 'h26' from crush map 2024-09-18T10:17:53.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.697+0000 7fcc5759e700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:53.704 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.697+0000 7fcc5759e700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:53.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.697+0000 7fcc5759e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:53.705 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.697+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h26"} v 0) v1 -- 0x7fcc58067b10 con 0x7fcc58096a70 2024-09-18T10:17:53.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.701+0000 7fcc4e7fc700 1 -- 172.21.15.150:0/2953506091 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h26"}]=0 device 'h26' does not appear in the crush map v249) v1 ==== 120+0+0 (secure 0 0 0) 0x7fcc4800f260 con 0x7fcc58096a70 2024-09-18T10:17:53.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.701+0000 7fcc4e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h26"}] 2024-09-18T10:17:53.707 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.701+0000 7fcc4e7fc700 10 monclient: _finish_command 3 = system:0 device 'h26' does not appear in the crush map 2024-09-18T10:17:53.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 msgr2=0x7fcc38037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:53.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 0x7fcc38037f20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcc400093c0 tx=0x7fcc40016000 comp rx=0 tx=0).stop 2024-09-18T10:17:53.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 10 monclient: shutdown 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 msgr2=0x7fcc58134780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc58134780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c560 tx=0x7fcc48007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 shutdown_connections 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcc38035a70 0x7fcc38037f20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc58097550 0x7fcc58134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcc58096a70 0x7fcc58134780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 --2- 172.21.15.150:0/2953506091 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcc5806e990 0x7fcc58134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 >> 172.21.15.150:0/2953506091 conn(0x7fcc580a12a0 msgr2=0x7fcc58130de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 shutdown_connections 2024-09-18T10:17:53.711 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.705+0000 7fcc5759e700 1 -- 172.21.15.150:0/2953506091 wait complete. 2024-09-18T10:17:53.712 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h26' does not appear in the crush map 2024-09-18T10:17:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:53.720 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h27 2024-09-18T10:17:53.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: get_monmap_and_config 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: build_initial_monmap 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: monmap: 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:53.818357+0000 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:53.818357+0000 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.820 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:53.821 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:53.821 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:53.821 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:53.821 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:53.821 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 Processor -- start 2024-09-18T10:17:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 -- start start 2024-09-18T10:17:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: init 2024-09-18T10:17:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:53.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1440a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): picked mon.noname-b con 0x7fd1440a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 0x7fd14406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): picked mon.noname-c con 0x7fd14406e610 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd144132260 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14406afe0 con 0x7fd144132260 2024-09-18T10:17:53.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1440a2220 con 0x7fd14406e610 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440a2830 con 0x7fd1440a4270 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd149635700 10 monclient(hunting): authenticate will time out at 1355.911778s 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd1427fc700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 msgr2=0x7fd14406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.813+0000 7fd1427fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 0x7fd14406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd1437fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd144132260 0x7fd1441326c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33980/0 (socket says 172.21.15.150:33980) 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd1437fe700 1 -- 172.21.15.150:0/3245755409 learned_addr learned my addr 172.21.15.150:0/3245755409 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1440a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): get_auth_request con 0x7fd1440a4270 auth_method 0 2024-09-18T10:17:53.828 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient(hunting): handle_auth_done global_id 74663 payload 274 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 -- 172.21.15.150:0/3245755409 >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 msgr2=0x7fd14406e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3245755409 >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 0x7fd14406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 -- 172.21.15.150:0/3245755409 >> v1:172.21.15.142:6789/0 conn(0x7fd144132260 legacy=0x7fd1441326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 monclient: found mon.noname-b 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:53.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 -- 172.21.15.150:0/3245755409 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440a4640 con 0x7fd1440a4270 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 -- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 msgr2=0x7fd1440a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1440a1ce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd1380098e0 tx=0x7fd1380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60c6f2dbc801959 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 1 -- 172.21.15.150:0/3245755409 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13800ba70 con 0x7fd1440a4270 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: dump: 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:53.830 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd149635700 5 monclient: authenticate success, global_id 74663 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd149635700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 1 -- 172.21.15.150:0/3245755409 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13800bbf0 con 0x7fd1440a4270 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:53.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd149635700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd149635700 4 set_mon_vals no callback set 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 1 -- 172.21.15.150:0/3245755409 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138011420 con 0x7fd1440a4270 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.817+0000 7fd141ffb700 10 monclient: dump: 2024-09-18T10:17:53.832 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.833 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: shutdown 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 msgr2=0x7fd1440a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:53.834 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1440a1ce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd1380098e0 tx=0x7fd1380091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- 172.21.15.150:0/3245755409 shutdown_connections 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- 172.21.15.150:0/3245755409 >> v2:172.21.15.150:3300/0 conn(0x7fd14406e610 0x7fd14406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- 172.21.15.150:0/3245755409 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1440a1ce0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- 172.21.15.150:0/3245755409 >> 172.21.15.150:0/3245755409 conn(0x7fd1440716e0 msgr2=0x7fd1440a3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- 172.21.15.150:0/3245755409 shutdown_connections 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- 172.21.15.150:0/3245755409 wait complete. 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: build_initial_monmap 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: monmap: 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:53.835 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:53.836 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 Processor -- start 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- start start 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: init 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): picked mon.noname-a con 0x7fd14406e610 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): picked mon.noname-c con 0x7fd1440a4270 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 0x7fd14411ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): picked mon.noname-b con 0x7fd144132260 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14409e6a0 con 0x7fd14406e610 2024-09-18T10:17:53.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441426c0 con 0x7fd144132260 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd144122800 con 0x7fd1440a4270 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd149635700 10 monclient(hunting): authenticate will time out at 1355.918406s 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1437fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 msgr2=0x7fd14411ef70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1437fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 0x7fd14411ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:53.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:39366/0 (socket says 172.21.15.150:39366) 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 -- 172.21.15.150:0/3304707338 learned_addr learned my addr 172.21.15.150:0/3304707338 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): get_auth_request con 0x7fd14406e610 auth_method 0 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 10 monclient(hunting): get_auth_request con 0x7fd1440a4270 auth_method 0 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 msgr2=0x7fd1441313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 msgr2=0x7fd1441313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:53.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd1427fc700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient(hunting): handle_auth_done global_id 94263 payload 824 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 msgr2=0x7fd1441313c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 msgr2=0x7fd14411ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:53.845 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 0x7fd14411ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 monclient: found mon.noname-a 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.821+0000 7fd142ffd700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd138009420 con 0x7fd14406e610 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd142ffd700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 msgr2=0x7fd1441417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd142ffd700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd138002000 tx=0x7fd138016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13801b680 con 0x7fd14406e610 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: dump: 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:53.846 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: _finish_auth 0 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 5 monclient: authenticate success, global_id 94263 2024-09-18T10:17:53.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 10 monclient: _renew_subs 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd144120a60 con 0x7fd14406e610 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13801b800 con 0x7fd14406e610 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 10 monclient: _renew_subs 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd144121630 con 0x7fd14406e610 2024-09-18T10:17:53.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13801bbc0 con 0x7fd14406e610 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd12affd700 10 monclient: dump: 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:53.849 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.825+0000 7fd1417fa700 4 set_mon_vals no callback set 2024-09-18T10:17:53.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd149635700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd149635700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd149635700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd144121820 con 0x7fd14406e610 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd1380193b0 con 0x7fd14406e610 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd12affd700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 0x7fd130037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(249..249 src has 176..249) v4 ==== 9545+0+0 (secure 0 0 0) 0x7fd13804afa0 con 0x7fd14406e610 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd1427fc700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 0x7fd130037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd1427fc700 10 monclient: get_auth_request con 0x7fd130035a70 auth_method 0 2024-09-18T10:17:53.853 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.829+0000 7fd1427fc700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 0x7fd130037f20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd134004fd0 tx=0x7fd134006ef0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:53.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.841+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd138010380 con 0x7fd14406e610 2024-09-18T10:17:53.854 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.841+0000 7fd12affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:53.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:53.841+0000 7fd12affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:54.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.009+0000 7fd149635700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.009+0000 7fd149635700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.009+0000 7fd149635700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.009+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7fd144122970 con 0x7fd14406e610 2024-09-18T10:17:54.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.593+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 removed item id -97 name 'h27' from crush map v250) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd138024350 con 0x7fd14406e610 2024-09-18T10:17:54.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.593+0000 7fd12affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.593+0000 7fd12affd700 10 monclient: _finish_command 2 = system:0 removed item id -97 name 'h27' from crush map 2024-09-18T10:17:54.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.701+0000 7fd149635700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.701+0000 7fd149635700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.701+0000 7fd149635700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.709 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.701+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h27"} v 0) v1 -- 0x7fd144122970 con 0x7fd14406e610 2024-09-18T10:17:54.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd12affd700 1 -- 172.21.15.150:0/3304707338 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h27"}]=0 device 'h27' does not appear in the crush map v250) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd138024530 con 0x7fd14406e610 2024-09-18T10:17:54.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd12affd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h27"}] 2024-09-18T10:17:54.710 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd12affd700 10 monclient: _finish_command 3 = system:0 device 'h27' does not appear in the crush map 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 msgr2=0x7fd130037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 0x7fd130037f20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd134004fd0 tx=0x7fd134006ef0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 10 monclient: shutdown 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 msgr2=0x7fd1441417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd138002000 tx=0x7fd138016000 comp rx=0 tx=0).stop 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 shutdown_connections 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd130035a70 0x7fd130037f20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd144132260 0x7fd14411ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1440a4270 0x7fd1441313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 --2- 172.21.15.150:0/3304707338 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14406e610 0x7fd1441417e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 >> 172.21.15.150:0/3304707338 conn(0x7fd1440716e0 msgr2=0x7fd14409cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:54.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 shutdown_connections 2024-09-18T10:17:54.714 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.705+0000 7fd149635700 1 -- 172.21.15.150:0/3304707338 wait complete. 2024-09-18T10:17:54.714 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h27' does not appear in the crush map 2024-09-18T10:17:54.723 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:54.723 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h28 2024-09-18T10:17:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.813+0000 7ff707fff700 10 monclient: get_monmap_and_config 2024-09-18T10:17:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.813+0000 7ff707fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.813+0000 7ff707fff700 10 monclient: monmap: 2024-09-18T10:17:54.820 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:54.819052+0000 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:54.819052+0000 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:54.821 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 Processor -- start 2024-09-18T10:17:54.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 -- start start 2024-09-18T10:17:54.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient: init 2024-09-18T10:17:54.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff7080a0110 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff70809d160 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 0x7ff70813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff7080a0bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff708068c30 con 0x7ff7080a0110 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7080a1dc0 con 0x7ff70809d160 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70806a6c0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff707fff700 10 monclient(hunting): authenticate will time out at 1356.912886s 2024-09-18T10:17:54.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7067fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7ff70809d160 legacy=0x7ff7081375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff706ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff7080a0110 0x7ff7080a04e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33984/0 (socket says 172.21.15.150:33984) 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff706ffd700 1 -- 172.21.15.150:0/4103390618 learned_addr learned my addr 172.21.15.150:0/4103390618 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 1 --2- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 0x7ff70813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): get_auth_request con 0x7ff7080a0bf0 auth_method 0 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient(hunting): handle_auth_done global_id 74669 payload 274 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 1 -- 172.21.15.150:0/4103390618 >> v1:172.21.15.150:6789/0 conn(0x7ff70809d160 legacy=0x7ff7081375c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 1 -- 172.21.15.150:0/4103390618 >> v1:172.21.15.142:6789/0 conn(0x7ff7080a0110 legacy=0x7ff7080a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:54.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:54.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 1 monclient: found mon.noname-b 2024-09-18T10:17:54.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:54.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:54.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.817+0000 7ff7077fe700 1 -- 172.21.15.150:0/4103390618 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080a17b0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 1 -- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 msgr2=0x7ff70813acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 1 --2- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 0x7ff70813acf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff700009e90 tx=0x7ff700009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=453a692be6b4f6ae server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 1 -- 172.21.15.150:0/4103390618 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000e070 con 0x7ff7080a0bf0 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:54.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: dump: 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:54.828 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 5 monclient: authenticate success, global_id 74669 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 1 -- 172.21.15.150:0/4103390618 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff700012430 con 0x7ff7080a0bf0 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 4 set_mon_vals no callback set 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 1 -- 172.21.15.150:0/4103390618 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7000127d0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:54.829 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff705ffb700 10 monclient: dump: 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: shutdown 2024-09-18T10:17:54.830 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 msgr2=0x7ff70813acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 0x7ff70813acf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff700009e90 tx=0x7ff700009780 comp rx=0 tx=0).stop 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4103390618 shutdown_connections 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4103390618 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7080a0bf0 0x7ff70813acf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4103390618 >> 172.21.15.150:0/4103390618 conn(0x7ff7080a6170 msgr2=0x7ff7080a2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4103390618 shutdown_connections 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4103390618 wait complete. 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: build_initial_monmap 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: monmap: 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:54.831 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:54.832 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 Processor -- start 2024-09-18T10:17:54.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- start start 2024-09-18T10:17:54.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: init 2024-09-18T10:17:54.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff70809d160 0x7ff708141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff70809d160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 0x7ff708134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff7080a0110 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff7080a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff708142650 con 0x7ff7080a0bf0 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff708122a80 con 0x7ff7080a0110 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:54.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff708120ce0 con 0x7ff70809d160 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff707fff700 10 monclient(hunting): authenticate will time out at 1356.919447s 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7067fc700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 msgr2=0x7ff708134670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7067fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 0x7ff708134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff706ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff70809d160 0x7ff708141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:39374/0 (socket says 172.21.15.150:39374) 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 1 -- 172.21.15.150:0/4115182886 learned_addr learned my addr 172.21.15.150:0/4115182886 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff706ffd700 10 monclient(hunting): get_auth_request con 0x7ff70809d160 auth_method 0 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff706ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff706ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff706ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:54.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 10 monclient(hunting): get_auth_request con 0x7ff7080a0bf0 auth_method 0 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.821+0000 7ff7077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff706ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff706ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff706ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient(hunting): handle_auth_done global_id 94266 payload 824 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 -- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff70809d160 msgr2=0x7ff708141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff70809d160 0x7ff708141770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 -- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 msgr2=0x7ff708134670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 0x7ff708134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient: _finish_hunting 0 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 monclient: found mon.noname-a 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff700009a20 con 0x7ff7080a0bf0 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 -- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 msgr2=0x7ff70811f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff7077fe700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff700003000 tx=0x7ff700018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700026430 con 0x7ff7080a0bf0 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: dump: 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:54.839 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:54.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff700021070 con 0x7ff7080a0bf0 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700026840 con 0x7ff7080a0bf0 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff6eeffd700 10 monclient: dump: 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:54.841 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff704ff9700 4 set_mon_vals no callback set 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 5 monclient: authenticate success, global_id 94266 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 10 monclient: _renew_subs 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7081211f0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 10 monclient: _renew_subs 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.825+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7081227a0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff707fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff707fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff707fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:54.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff700023030 con 0x7ff7080a0bf0 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff6eeffd700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 0x7ff6f4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(250..250 src has 176..250) v4 ==== 9332+0+0 (secure 0 0 0) 0x7ff70004bcd0 con 0x7ff7080a0bf0 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff706ffd700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 0x7ff6f4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.829+0000 7ff706ffd700 10 monclient: get_auth_request con 0x7ff6f4035a70 auth_method 0 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.833+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:17:54.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.833+0000 7ff706ffd700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 0x7ff6f4037f20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8007880 tx=0x7ff6f8007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:54.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.841+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:54.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.841+0000 7ff6eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:54.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:54.841+0000 7ff6eeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:55.017 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.009+0000 7ff707fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.009+0000 7ff707fff700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.013+0000 7ff707fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.116 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.013+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:55.500 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:17:55.501 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:17:55.501 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:17:55.501 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:17:55.504 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:17:55.505 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:17:55.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: build_initial_monmap 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: monmap: 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:55.618398+0000 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:55.618398+0000 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.620 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:55.621 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:55.621 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:55.621 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:55.621 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:55.621 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 1 Processor -- start 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 1 -- start start 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: init 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 0x7fa05c09ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa05c0a0180 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa05c0a0550 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:55.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa05c0a1080 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.613+0000 7fa05b59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c09a110 con 0x7fa05c0a1080 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c09a3f0 con 0x7fa05c0a0550 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c09aa20 con 0x7fa05c0a0180 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 10 monclient(hunting): authenticate will time out at 1364.544036s 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa059d9b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa05c0a0550 0x7fa05c0a0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:39684/0 (socket says 172.21.15.142:39684) 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05ad9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa05c0a1080 0x7fa05c139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:39500/0 (socket says 172.21.15.142:39500) 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa059d9b700 1 -- 172.21.15.142:0/468172136 learned_addr learned my addr 172.21.15.142:0/468172136 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:55.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 0x7fa05c09ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 10 monclient(hunting): get_auth_request con 0x7fa05c0a0180 auth_method 0 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 msgr2=0x7fa05c09ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 msgr2=0x7fa05c09ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 0x7fa05c09ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:55.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 0x7fa05c09ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:55.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05ad9d700 1 -- 172.21.15.142:0/468172136 >> v1:172.21.15.142:6789/0 conn(0x7fa05c0a1080 legacy=0x7fa05c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa059d9b700 1 -- 172.21.15.142:0/468172136 >> v1:172.21.15.150:6789/0 conn(0x7fa05c0a0550 legacy=0x7fa05c0a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959832256 0 0) 0x7fa05c09a110 con 0x7fa05c0a1080 2024-09-18T10:17:55.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): my global_id is 94269 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa044003570 con 0x7fa05c0a1080 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170701282 0 0) 0x7fa044003570 con 0x7fa05c0a1080 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 msgr2=0x7fa05c09ae60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 --2- 172.21.15.142:0/468172136 >> v2:172.21.15.150:3300/0 conn(0x7fa05c0a0180 0x7fa05c09ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 >> v1:172.21.15.150:6789/0 conn(0x7fa05c0a0550 legacy=0x7fa05c0a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: _finish_hunting 0 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:17:55.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c099b50 con 0x7fa05c0a1080 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: _finish_auth 0 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fa054001be0 con 0x7fa05c0a1080 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 5 monclient: authenticate success, global_id 94269 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: dump: 2024-09-18T10:17:55.632 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 >> v1:172.21.15.142:6789/0 conn(0x7fa05c0a1080 legacy=0x7fa05c139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa044003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): picked mon.b con 0x7fa044003b80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa044004810 0x7fa044004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): picked mon.a con 0x7fa044004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa044005410 0x7fa04400bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): picked mon.c con 0x7fa044005410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c09aa20 con 0x7fa044004810 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04400c3e0 con 0x7fa044003b80 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04400c5a0 con 0x7fa044005410 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05959a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05b59e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa059d9b700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa044004810 0x7fa044004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05ad9d700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa044005410 0x7fa04400bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.617+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa044003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): get_auth_request con 0x7fa044003b80 auth_method 0 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:55.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): get_auth_request con 0x7fa044005410 auth_method 0 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): get_auth_request con 0x7fa044004810 auth_method 0 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient(hunting): handle_auth_done global_id 94269 payload 293 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa044005410 msgr2=0x7fa04400bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa044005410 0x7fa04400bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa044004810 msgr2=0x7fa044004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa044004810 0x7fa044004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 monclient: found mon.b 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044003570 con 0x7fa044003b80 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 10 monclient: _finish_auth 0 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa044005410 0x7fa04400bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05ad9d700 1 -- 172.21.15.142:0/468172136 reap_dead start 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 -- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 msgr2=0x7fa044003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05a59c700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa044003f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa05000bb40 tx=0x7fa050008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25aefe5bf0bf448c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050018070 con 0x7fa044003b80 2024-09-18T10:17:55.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: dump: 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:55.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05000d7e0 con 0x7fa044003b80 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 4 set_mon_vals no callback set 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 1 -- 172.21.15.142:0/468172136 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05000dba0 con 0x7fa044003b80 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05959a700 10 monclient: dump: 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 10 monclient: shutdown 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.621+0000 7fa05b59e700 1 -- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 msgr2=0x7fa044003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa044003f90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa05000bb40 tx=0x7fa050008040 comp rx=0 tx=0).stop 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- 172.21.15.142:0/468172136 shutdown_connections 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 --2- 172.21.15.142:0/468172136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa044003f90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- 172.21.15.142:0/468172136 >> 172.21.15.142:0/468172136 conn(0x7fa05c0a5790 msgr2=0x7fa05c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- 172.21.15.142:0/468172136 shutdown_connections 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- 172.21.15.142:0/468172136 wait complete. 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient: build_initial_monmap 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient: monmap: 2024-09-18T10:17:55.641 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:55.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 Processor -- start 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- start start 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient: init 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa044003b80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa05c141f10 0x7fa05c142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa05c141f10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa05c08cf60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:55.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c090af0 con 0x7fa05c08cf60 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c08f670 con 0x7fa044003b80 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0a4070 con 0x7fa05c141f10 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05b59e700 10 monclient(hunting): authenticate will time out at 1364.554771s 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa05c141f10 0x7fa05c142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50682/0 (socket says 172.21.15.142:50682) 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 -- 172.21.15.142:0/330296455 learned_addr learned my addr 172.21.15.142:0/330296455 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): get_auth_request con 0x7fa05c08cf60 auth_method 0 2024-09-18T10:17:55.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): get_auth_request con 0x7fa05c141f10 auth_method 0 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 10 monclient(hunting): get_auth_request con 0x7fa044003b80 auth_method 0 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa059d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 msgr2=0x7fa05c1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 msgr2=0x7fa05c1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient(hunting): handle_auth_done global_id 94272 payload 824 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05a59c700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa05c141f10 msgr2=0x7fa05c142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa05c141f10 0x7fa05c142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 msgr2=0x7fa05c1419d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:55.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 monclient: found mon.noname-a 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.625+0000 7fa05ad9d700 1 -- 172.21.15.142:0/330296455 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c142840 con 0x7fa05c08cf60 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05ad9d700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 msgr2=0x7fa05c08d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05ad9d700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa054002250 tx=0x7fa05400a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054019070 con 0x7fa05c08cf60 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: dump: 2024-09-18T10:17:55.652 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05400d040 con 0x7fa05c08cf60 2024-09-18T10:17:55.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 5 monclient: authenticate success, global_id 94272 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 monclient: _renew_subs 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c090810 con 0x7fa05c08cf60 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400c070 con 0x7fa05c08cf60 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa04a7fc700 10 monclient: dump: 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.654 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa058d99700 4 set_mon_vals no callback set 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 monclient: _renew_subs 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c098660 con 0x7fa05c08cf60 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.629+0000 7fa05b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa05b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c0918f0 con 0x7fa05c08cf60 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa054013b80 con 0x7fa05c08cf60 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa04a7fc700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 0x7fa03c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(250..250 src has 176..250) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa054049960 con 0x7fa05c08cf60 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa05a59c700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 0x7fa03c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.633+0000 7fa05a59c700 10 monclient: get_auth_request con 0x7fa03c035af0 auth_method 0 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.637+0000 7fa05a59c700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 0x7fa03c037fa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b5e0 tx=0x7fa05001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.637+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa05400ae10 con 0x7fa05c08cf60 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.637+0000 7fa04a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.637+0000 7fa04a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:55.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.709+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 removed item id -100 name 'h28' from crush map v251) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:55.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.709+0000 7ff6eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.713 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.709+0000 7ff6eeffd700 10 monclient: _finish_command 2 = system:0 removed item id -100 name 'h28' from crush map 2024-09-18T10:17:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff707fff700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff707fff700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff707fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h28"} v 0) v1 -- 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:55.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff6eeffd700 1 -- 172.21.15.150:0/4115182886 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h28"}]=0 device 'h28' does not appear in the crush map v251) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff708121900 con 0x7ff7080a0bf0 2024-09-18T10:17:55.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff6eeffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h28"}] 2024-09-18T10:17:55.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.817+0000 7ff6eeffd700 10 monclient: _finish_command 3 = system:0 device 'h28' does not appear in the crush map 2024-09-18T10:17:55.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 msgr2=0x7ff6f4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 0x7ff6f4037f20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8007880 tx=0x7ff6f8007f90 comp rx=0 tx=0).stop 2024-09-18T10:17:55.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 10 monclient: shutdown 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 msgr2=0x7ff70811f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff700003000 tx=0x7ff700018000 comp rx=0 tx=0).stop 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 shutdown_connections 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff6f4035a70 0x7ff6f4037f20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7080a0bf0 0x7ff70811f1d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7080a0110 0x7ff708134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 --2- 172.21.15.150:0/4115182886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff70809d160 0x7ff708141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 >> 172.21.15.150:0/4115182886 conn(0x7ff7080a6170 msgr2=0x7ff7080a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 shutdown_connections 2024-09-18T10:17:55.827 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.821+0000 7ff707fff700 1 -- 172.21.15.150:0/4115182886 wait complete. 2024-09-18T10:17:55.828 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h28' does not appear in the crush map 2024-09-18T10:17:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:55.836 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h29 2024-09-18T10:17:55.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa05b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:55.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa05b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:55.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa05b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa05c142840 con 0x7fa05c08cf60 2024-09-18T10:17:55.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa04a7fc700 1 -- 172.21.15.142:0/330296455 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa05c142840 con 0x7fa05c08cf60 2024-09-18T10:17:55.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa04a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:55.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.865+0000 7fa04a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:55.871 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 msgr2=0x7fa03c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 0x7fa03c037fa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa05000b5e0 tx=0x7fa05001a000 comp rx=0 tx=0).stop 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 10 monclient: shutdown 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 msgr2=0x7fa05c08d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa054002250 tx=0x7fa05400a370 comp rx=0 tx=0).stop 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 shutdown_connections 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa03c035af0 0x7fa03c037fa0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa05c08cf60 0x7fa05c08d3d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa05c141f10 0x7fa05c142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 --2- 172.21.15.142:0/330296455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa044003b80 0x7fa05c1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 >> 172.21.15.142:0/330296455 conn(0x7fa05c0a5790 msgr2=0x7fa05c09af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:55.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 shutdown_connections 2024-09-18T10:17:55.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:55.869+0000 7fa05b59e700 1 -- 172.21.15.142:0/330296455 wait complete. 2024-09-18T10:17:55.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: get_monmap_and_config 2024-09-18T10:17:55.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: build_initial_monmap 2024-09-18T10:17:55.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: monmap: 2024-09-18T10:17:55.939 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:55.938215+0000 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:55.938215+0000 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:55.940 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 Processor -- start 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 -- start start 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: init 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c200a6090 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:55.943 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 0x7f4c2009d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-a con 0x7f4c2009e460 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-b con 0x7f4c200a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200a6460 con 0x7f4c200a6090 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2009fd00 con 0x7f4c2009e460 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c200a6850 con 0x7f4c200a0a80 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:55.944 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.933+0000 7f4c25de1700 10 monclient(hunting): authenticate will time out at 1358.031759s 2024-09-18T10:17:55.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 0x7f4c2009d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 0x7f4c2009d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:39398/0 (socket says 172.21.15.150:39398) 2024-09-18T10:17:55.945 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:45098/0 (socket says 172.21.15.150:45098) 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4c200a6090 0x7f4c200a0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33990/0 (socket says 172.21.15.150:33990) 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 1 -- 172.21.15.150:0/3318748572 learned_addr learned my addr 172.21.15.150:0/3318748572 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): get_auth_request con 0x7f4c2009e460 auth_method 0 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): get_auth_request con 0x7f4c200a0a80 auth_method 0 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_done global_id 74678 payload 274 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3318748572 >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 msgr2=0x7f4c2009d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3318748572 >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 0x7f4c2009d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3318748572 >> v1:172.21.15.142:6789/0 conn(0x7f4c200a6090 legacy=0x7f4c200a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 monclient: found mon.noname-b 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:55.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3318748572 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2009d7a0 con 0x7f4c200a0a80 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3318748572 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 msgr2=0x7f4c20139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3318748572 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009e60 tx=0x7f4c10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4636054d70557f85 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 1 -- 172.21.15.150:0/3318748572 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000e070 con 0x7f4c200a0a80 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:55.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 10 monclient: dump: 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:55.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.937+0000 7f4c1e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 5 monclient: authenticate success, global_id 74678 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 1 -- 172.21.15.150:0/3318748572 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1000bd70 con 0x7f4c200a0a80 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 1 -- 172.21.15.150:0/3318748572 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10012660 con 0x7f4c200a0a80 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 4 set_mon_vals no callback set 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1e7fc700 10 monclient: dump: 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.956 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: shutdown 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- 172.21.15.150:0/3318748572 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 msgr2=0x7f4c20139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3318748572 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009e60 tx=0x7f4c10009780 comp rx=0 tx=0).stop 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- 172.21.15.150:0/3318748572 shutdown_connections 2024-09-18T10:17:55.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3318748572 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c200a0a80 0x7f4c20139b10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3318748572 >> v2:172.21.15.142:3300/0 conn(0x7f4c2009e460 0x7f4c2009d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- 172.21.15.150:0/3318748572 >> 172.21.15.150:0/3318748572 conn(0x7f4c20071830 msgr2=0x7f4c200a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- 172.21.15.150:0/3318748572 shutdown_connections 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- 172.21.15.150:0/3318748572 wait complete. 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: build_initial_monmap 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: monmap: 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:55.958 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 Processor -- start 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- start start 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: init 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c2009e460 0x7f4c2008cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-c con 0x7f4c2009e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-b con 0x7f4c200a0a80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 0x7f4c2008d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): picked mon.noname-a con 0x7f4c200a6090 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c20097780 con 0x7f4c200a6090 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2008ec90 con 0x7f4c200a0a80 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2009c190 con 0x7f4c2009e460 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c25de1700 10 monclient(hunting): authenticate will time out at 1358.038914s 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52540/0 (socket says 172.21.15.150:52540) 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 -- 172.21.15.150:0/3117914161 learned_addr learned my addr 172.21.15.150:0/3117914161 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 10 monclient(hunting): get_auth_request con 0x7f4c200a0a80 auth_method 0 2024-09-18T10:17:55.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 0x7f4c2008d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c2009e460 0x7f4c2008cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): get_auth_request con 0x7f4c200a6090 auth_method 0 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 msgr2=0x7f4c2008d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 msgr2=0x7f4c2008d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): get_auth_request con 0x7f4c2009e460 auth_method 0 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:55.961 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1effd700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient(hunting): handle_auth_done global_id 94278 payload 824 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c2009e460 msgr2=0x7f4c2008cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c2009e460 0x7f4c2008cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 msgr2=0x7f4c2008d3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:55.962 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 monclient: found mon.noname-a 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.941+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10009a20 con 0x7f4c200a6090 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c1ffff700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 msgr2=0x7f4c2008d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c1ffff700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 0x7f4c2008d920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c20068d70 tx=0x7f4c10018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100070d0 con 0x7f4c200a6090 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: dump: 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.963 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 5 monclient: authenticate success, global_id 94278 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c10021070 con 0x7f4c200a6090 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 monclient: _renew_subs 2024-09-18T10:17:55.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c200901d0 con 0x7f4c200a6090 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10026430 con 0x7f4c200a6090 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c177fe700 10 monclient: dump: 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:55.965 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 monclient: _renew_subs 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c1dffb700 4 set_mon_vals no callback set 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c200981e0 con 0x7f4c200a6090 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.945+0000 7f4c25de1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c25de1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2006d3c0 con 0x7f4c200a6090 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4c10023030 con 0x7f4c200a6090 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c177fe700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 0x7f4c00037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:55.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(251..251 src has 176..251) v4 ==== 9119+0+0 (secure 0 0 0) 0x7f4c1004b730 con 0x7f4c200a6090 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c1f7fe700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 0x7f4c00037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c1f7fe700 10 monclient: get_auth_request con 0x7f4c00035ac0 auth_method 0 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.949+0000 7f4c1f7fe700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 0x7f4c00037f70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4c18007450 tx=0x7f4c18007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.961+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4c1001c420 con 0x7f4c200a6090 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.961+0000 7f4c177fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:55.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:55.961+0000 7f4c177fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:56.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.129+0000 7f4c25de1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.129+0000 7f4c25de1700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.129+0000 7f4c25de1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.129+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f4c2009da80 con 0x7f4c200a6090 2024-09-18T10:17:56.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.721+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 removed item id -103 name 'h29' from crush map v252) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4c10011b80 con 0x7f4c200a6090 2024-09-18T10:17:56.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.721+0000 7f4c177fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.726 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.721+0000 7f4c177fe700 10 monclient: _finish_command 2 = system:0 removed item id -103 name 'h29' from crush map 2024-09-18T10:17:56.835 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c25de1700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c25de1700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c25de1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h29"} v 0) v1 -- 0x7f4c2009da80 con 0x7f4c200a6090 2024-09-18T10:17:56.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c177fe700 1 -- 172.21.15.150:0/3117914161 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h29"}]=0 device 'h29' does not appear in the crush map v252) v1 ==== 120+0+0 (secure 0 0 0) 0x7f4c10011d60 con 0x7f4c200a6090 2024-09-18T10:17:56.836 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c177fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h29"}] 2024-09-18T10:17:56.837 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.829+0000 7f4c177fe700 10 monclient: _finish_command 3 = system:0 device 'h29' does not appear in the crush map 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 msgr2=0x7f4c00037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 0x7f4c00037f70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4c18007450 tx=0x7f4c18007ed0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 10 monclient: shutdown 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 msgr2=0x7f4c2008d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 0x7f4c2008d920 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c20068d70 tx=0x7f4c10018000 comp rx=0 tx=0).stop 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 shutdown_connections 2024-09-18T10:17:56.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4c00035ac0 0x7f4c00037f70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c200a6090 0x7f4c2008d920 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c200a0a80 0x7f4c2008d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 --2- 172.21.15.150:0/3117914161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4c2009e460 0x7f4c2008cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 >> 172.21.15.150:0/3117914161 conn(0x7f4c20071830 msgr2=0x7f4c2009c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 shutdown_connections 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.833+0000 7f4c25de1700 1 -- 172.21.15.150:0/3117914161 wait complete. 2024-09-18T10:17:56.840 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h29' does not appear in the crush map 2024-09-18T10:17:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:56.849 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h30 2024-09-18T10:17:56.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.945+0000 7fb22c692700 10 monclient: get_monmap_and_config 2024-09-18T10:17:56.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.945+0000 7fb22c692700 10 monclient: build_initial_monmap 2024-09-18T10:17:56.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.945+0000 7fb22c692700 10 monclient: monmap: 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:56.951206+0000 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:56.951206+0000 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:56.953 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 Processor -- start 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 -- start start 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient: init 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T10:17:56.956 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb22409acd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb224096a90 0x7fb22409c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-a con 0x7fb224096a90 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-c con 0x7fb22409cc20 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2240958e0 con 0x7fb22409acd0 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22409d080 con 0x7fb224096a90 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22409b0a0 con 0x7fb22409cc20 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22c692700 10 monclient(hunting): authenticate will time out at 1359.044649s 2024-09-18T10:17:56.957 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22a42e700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb22409acd0 0x7fb22409bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38300/0 (socket says 172.21.15.150:38300) 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22a42e700 1 -- 172.21.15.150:0/971526763 learned_addr learned my addr 172.21.15.150:0/971526763 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 10 monclient(hunting): get_auth_request con 0x7fb22409cc20 auth_method 0 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 1 -- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 msgr2=0x7fb224137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 1 -- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 msgr2=0x7fb224137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 1 --2- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb22ac2f700 1 --2- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 --2- 172.21.15.150:0/971526763 >> v2:172.21.15.142:3300/0 conn(0x7fb224096a90 0x7fb22409c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): get_auth_request con 0x7fb224096a90 auth_method 0 2024-09-18T10:17:56.958 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient(hunting): handle_auth_done global_id 94281 payload 274 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 -- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 msgr2=0x7fb224137790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 --2- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 -- 172.21.15.150:0/971526763 >> v1:172.21.15.150:6789/0 conn(0x7fb22409acd0 legacy=0x7fb22409bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 monclient: found mon.noname-a 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:56.959 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.960 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.949+0000 7fb229c2d700 1 -- 172.21.15.150:0/971526763 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224095b50 con 0x7fb224096a90 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb229c2d700 1 -- 172.21.15.150:0/971526763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb224096a90 msgr2=0x7fb22409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb229c2d700 1 --2- 172.21.15.150:0/971526763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb224096a90 0x7fb22409c6e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb224095e60 tx=0x7fb2180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2ba84687bc6d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 1 -- 172.21.15.150:0/971526763 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21800b8c0 con 0x7fb224096a90 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: dump: 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:56.965 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: _finish_auth 0 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 1 -- 172.21.15.150:0/971526763 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21800ba20 con 0x7fb224096a90 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 5 monclient: authenticate success, global_id 94281 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 1 -- 172.21.15.150:0/971526763 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21800bdc0 con 0x7fb224096a90 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:56.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 4 set_mon_vals no callback set 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22942c700 10 monclient: dump: 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:56.967 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: shutdown 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- 172.21.15.150:0/971526763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb224096a90 msgr2=0x7fb22409c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 --2- 172.21.15.150:0/971526763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb224096a90 0x7fb22409c6e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb224095e60 tx=0x7fb2180091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- 172.21.15.150:0/971526763 shutdown_connections 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 --2- 172.21.15.150:0/971526763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb224096a90 0x7fb22409c6e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 --2- 172.21.15.150:0/971526763 >> v2:172.21.15.150:3300/0 conn(0x7fb22409cc20 0x7fb224137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- 172.21.15.150:0/971526763 >> 172.21.15.150:0/971526763 conn(0x7fb2240a15e0 msgr2=0x7fb2240a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- 172.21.15.150:0/971526763 shutdown_connections 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- 172.21.15.150:0/971526763 wait complete. 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: build_initial_monmap 2024-09-18T10:17:56.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: monmap: 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 Processor -- start 2024-09-18T10:17:56.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 -- start start 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: init 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.953+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-b con 0x7fb224096a90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 0x7fb224094380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-a con 0x7fb22409acd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 0x7fb22408bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): picked mon.noname-c con 0x7fb22409cc20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22408f680 con 0x7fb22409acd0 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22408fbb0 con 0x7fb224096a90 2024-09-18T10:17:56.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb224098410 con 0x7fb22409cc20 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22c692700 10 monclient(hunting): authenticate will time out at 1359.052108s 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52548/0 (socket says 172.21.15.150:52548) 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 -- 172.21.15.150:0/524159438 learned_addr learned my addr 172.21.15.150:0/524159438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 10 monclient(hunting): get_auth_request con 0x7fb224096a90 auth_method 0 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 0x7fb224094380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 0x7fb22408bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): get_auth_request con 0x7fb22409acd0 auth_method 0 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): get_auth_request con 0x7fb22409cc20 auth_method 0 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 msgr2=0x7fb22413f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 msgr2=0x7fb22413f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:56.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22a42e700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient(hunting): handle_auth_done global_id 94284 payload 824 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 msgr2=0x7fb22408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 0x7fb22408bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 msgr2=0x7fb22413f9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient: _finish_hunting 0 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 monclient: found mon.noname-a 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218009420 con 0x7fb22409acd0 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb22ac2f700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 0x7fb22408bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 msgr2=0x7fb224094380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:56.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.957+0000 7fb229c2d700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 0x7fb224094380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb218015000 tx=0x7fb218016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb218011bb0 con 0x7fb22409acd0 2024-09-18T10:17:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:56.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: dump: 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: _finish_auth 0 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb218020070 con 0x7fb22409acd0 2024-09-18T10:17:56.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 5 monclient: authenticate success, global_id 94284 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _renew_subs 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2240a01b0 con 0x7fb22409acd0 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21801b450 con 0x7fb22409acd0 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb217fff700 4 set_mon_vals no callback set 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 10 monclient: dump: 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.978 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _renew_subs 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22408e620 con 0x7fb22409acd0 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:56.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22408f960 con 0x7fb22409acd0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb218022030 con 0x7fb22409acd0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 0x7fb208037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(252..252 src has 176..252) v4 ==== 8906+0+0 (secure 0 0 0) 0x7fb21804b010 con 0x7fb22409acd0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22a42e700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 0x7fb208037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.961+0000 7fb22a42e700 10 monclient: get_auth_request con 0x7fb208035ac0 auth_method 0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.965+0000 7fb22a42e700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 0x7fb208037f70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb220007e60 tx=0x7fb220009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.973+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb218024c00 con 0x7fb22409acd0 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.973+0000 7fb2167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:56.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:56.973+0000 7fb2167fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:57.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.145+0000 7fb22c692700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.145+0000 7fb22c692700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.145+0000 7fb22c692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:57.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.145+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fb22408f960 con 0x7fb22409acd0 2024-09-18T10:17:57.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.733+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 removed item id -106 name 'h30' from crush map v253) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb2180243d0 con 0x7fb22409acd0 2024-09-18T10:17:57.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.733+0000 7fb2167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.740 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.733+0000 7fb2167fc700 10 monclient: _finish_command 2 = system:0 removed item id -106 name 'h30' from crush map 2024-09-18T10:17:57.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb22c692700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb22c692700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb22c692700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:57.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h30"} v 0) v1 -- 0x7fb22408f960 con 0x7fb22409acd0 2024-09-18T10:17:57.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb2167fc700 1 -- 172.21.15.150:0/524159438 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h30"}]=0 device 'h30' does not appear in the crush map v253) v1 ==== 120+0+0 (secure 0 0 0) 0x7fb218033030 con 0x7fb22409acd0 2024-09-18T10:17:57.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb2167fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h30"}] 2024-09-18T10:17:57.852 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.845+0000 7fb2167fc700 10 monclient: _finish_command 3 = system:0 device 'h30' does not appear in the crush map 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 msgr2=0x7fb208037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 0x7fb208037f70 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb220007e60 tx=0x7fb220009040 comp rx=0 tx=0).stop 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 10 monclient: shutdown 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 msgr2=0x7fb224094380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 0x7fb224094380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb218015000 tx=0x7fb218016000 comp rx=0 tx=0).stop 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 shutdown_connections 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb208035ac0 0x7fb208037f70 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb22409cc20 0x7fb22408bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb22409acd0 0x7fb224094380 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 --2- 172.21.15.150:0/524159438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb224096a90 0x7fb22413f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 >> 172.21.15.150:0/524159438 conn(0x7fb2240a15e0 msgr2=0x7fb224096f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:57.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 shutdown_connections 2024-09-18T10:17:57.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.849+0000 7fb22c692700 1 -- 172.21.15.150:0/524159438 wait complete. 2024-09-18T10:17:57.856 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h30' does not appear in the crush map 2024-09-18T10:17:57.864 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:57.864 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h31 2024-09-18T10:17:57.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 10 monclient: get_monmap_and_config 2024-09-18T10:17:57.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 10 monclient: build_initial_monmap 2024-09-18T10:17:57.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 10 monclient: monmap: 2024-09-18T10:17:57.963 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:57.963 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:57.961953+0000 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:57.961953+0000 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:57.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 1 Processor -- start 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 1 -- start start 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 10 monclient: init 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.957+0000 7fce4d180700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce4809c930 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce4809d410 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:57.966 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-c con 0x7fce48097730 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce48098f00 con 0x7fce4809d410 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce48096a70 con 0x7fce4809c930 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce4806f730 con 0x7fce48097730 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 10 monclient(hunting): authenticate will time out at 1360.056153s 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52562/0 (socket says 172.21.15.150:52562) 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 -- 172.21.15.150:0/117943312 learned_addr learned my addr 172.21.15.150:0/117943312 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce46d9d700 1 --1- 172.21.15.150:0/117943312 >> v1:172.21.15.150:6789/0 conn(0x7fce4809c930 0x7fce4809cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38316/0 (socket says 172.21.15.150:38316) 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 10 monclient(hunting): get_auth_request con 0x7fce48097730 auth_method 0 2024-09-18T10:17:57.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 -- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 msgr2=0x7fce4813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 -- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 msgr2=0x7fce4813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 --2- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:57.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4759e700 1 --2- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:57.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce46d9d700 1 -- 172.21.15.150:0/117943312 >> v1:172.21.15.150:6789/0 conn(0x7fce4809c930 legacy=0x7fce4809cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce3ffff700 1 -- 172.21.15.150:0/117943312 >> v1:172.21.15.142:6789/0 conn(0x7fce4809d410 legacy=0x7fce48137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730017877 0 0) 0x7fce48098f00 con 0x7fce4809d410 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient(hunting): my global_id is 94287 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce28003570 con 0x7fce4809d410 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938186102 0 0) 0x7fce28003570 con 0x7fce4809d410 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 msgr2=0x7fce4813aea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 --2- 172.21.15.150:0/117943312 >> v2:172.21.15.150:3300/0 conn(0x7fce48097730 0x7fce4813aea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 >> v1:172.21.15.150:6789/0 conn(0x7fce4809c930 legacy=0x7fce4809cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: _finish_hunting 0 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48097030 con 0x7fce4809d410 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: _finish_auth 0 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fce300017f0 con 0x7fce4809d410 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 5 monclient: authenticate success, global_id 94287 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4d180700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: dump: 2024-09-18T10:17:57.973 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.961+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 >> v1:172.21.15.142:6789/0 conn(0x7fce4809d410 legacy=0x7fce48137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:57.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce28003c50 0x7fce28004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): picked mon.a con 0x7fce28003c50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): picked mon.b con 0x7fce28004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce28005440 0x7fce2800bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): picked mon.c con 0x7fce28005440 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4806f730 con 0x7fce28003c50 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800c410 con 0x7fce28004810 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800c5d0 con 0x7fce28005440 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:57.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4d180700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): get_auth_request con 0x7fce28004810 auth_method 0 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce28003c50 0x7fce28004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): get_auth_request con 0x7fce28003c50 auth_method 0 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:57.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce28005440 0x7fce2800bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 10 monclient(hunting): get_auth_request con 0x7fce28005440 auth_method 0 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce46d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient(hunting): handle_auth_done global_id 94287 payload 293 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 -- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce28005440 msgr2=0x7fce2800bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce28005440 0x7fce2800bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 -- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce28003c50 msgr2=0x7fce28004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce28003c50 0x7fce28004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 monclient: found mon.b 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 -- 172.21.15.150:0/117943312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28003570 con 0x7fce28004810 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 10 monclient: _finish_auth 0 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce28005440 0x7fce2800bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:17:57.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4759e700 1 -- 172.21.15.150:0/117943312 reap_dead start 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 -- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 msgr2=0x7fce28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce3ffff700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce28004c80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fce30001520 tx=0x7fce30007a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a8aded75c2e6653 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3000cd90 con 0x7fce28004810 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:57.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: dump: 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4d180700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce300070d0 con 0x7fce28004810 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4d180700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4d180700 4 set_mon_vals no callback set 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 1 -- 172.21.15.150:0/117943312 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30007490 con 0x7fce28004810 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.965+0000 7fce4659c700 10 monclient: dump: 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.986 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: shutdown 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 msgr2=0x7fce28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce28004c80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fce30001520 tx=0x7fce30007a70 comp rx=0 tx=0).stop 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- 172.21.15.150:0/117943312 shutdown_connections 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 --2- 172.21.15.150:0/117943312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce28004c80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- 172.21.15.150:0/117943312 >> 172.21.15.150:0/117943312 conn(0x7fce48068360 msgr2=0x7fce480a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:57.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- 172.21.15.150:0/117943312 shutdown_connections 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- 172.21.15.150:0/117943312 wait complete. 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: build_initial_monmap 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: monmap: 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:57.988 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 Processor -- start 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- start start 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: init 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-b con 0x7fce28004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 0x7fce4813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-c con 0x7fce481218d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 0x7fce481415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): picked mon.noname-a con 0x7fce4813e9d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce48123040 con 0x7fce4813e9d0 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce48122a80 con 0x7fce28004810 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce481205a0 con 0x7fce481218d0 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4d180700 10 monclient(hunting): authenticate will time out at 1360.067058s 2024-09-18T10:17:57.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52584/0 (socket says 172.21.15.150:52584) 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 -- 172.21.15.150:0/1092863046 learned_addr learned my addr 172.21.15.150:0/1092863046 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 10 monclient(hunting): get_auth_request con 0x7fce28004810 auth_method 0 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 0x7fce481415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 0x7fce4813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): get_auth_request con 0x7fce4813e9d0 auth_method 0 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): get_auth_request con 0x7fce481218d0 auth_method 0 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 msgr2=0x7fce4813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 msgr2=0x7fce4813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce46d9d700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce4759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.969+0000 7fce3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 10 monclient(hunting): handle_auth_done global_id 74687 payload 824 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 msgr2=0x7fce4813e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 msgr2=0x7fce481415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 0x7fce481415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 monclient: found mon.noname-c 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce481227a0 con 0x7fce481218d0 2024-09-18T10:17:57.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4759e700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 0x7fce481415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 msgr2=0x7fce4813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3ffff700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 0x7fce4813d700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fce30014000 tx=0x7fce30015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30021070 con 0x7fce481218d0 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: dump: 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:57.994 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: _finish_auth 0 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 5 monclient: authenticate success, global_id 74687 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 10 monclient: _renew_subs 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30005e10 con 0x7fce481218d0 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce48122d60 con 0x7fce481218d0 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:57.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3001d420 con 0x7fce481218d0 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce3effd700 10 monclient: dump: 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:57.996 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce45d9b700 4 set_mon_vals no callback set 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 10 monclient: _renew_subs 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.973+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4813dde0 con 0x7fce481218d0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce4d180700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce4d180700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce4d180700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4809e140 con 0x7fce481218d0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fce3001d810 con 0x7fce481218d0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce3effd700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 0x7fce2c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(253..253 src has 176..253) v4 ==== 8693+0+0 (secure 0 0 0) 0x7fce3004a0a0 con 0x7fce481218d0 2024-09-18T10:17:57.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce46d9d700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 0x7fce2c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.977+0000 7fce46d9d700 10 monclient: get_auth_request con 0x7fce2c035af0 auth_method 0 2024-09-18T10:17:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.981+0000 7fce46d9d700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 0x7fce2c037fa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fce38007ab0 tx=0x7fce38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.989+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fce3007a670 con 0x7fce481218d0 2024-09-18T10:17:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.989+0000 7fce3effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:57.989+0000 7fce3effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:58.165 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.161+0000 7fce4d180700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.161+0000 7fce4d180700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.161+0000 7fce4d180700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:58.166 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.161+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fce48063ed0 con 0x7fce481218d0 2024-09-18T10:17:58.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.413+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 removed item id -109 name 'h31' from crush map v254) v1 ==== 121+0+0 (secure 0 0 0) 0x7fce30017020 con 0x7fce481218d0 2024-09-18T10:17:58.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.413+0000 7fce3effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.413+0000 7fce3effd700 10 monclient: _finish_command 2 = system:0 removed item id -109 name 'h31' from crush map 2024-09-18T10:17:58.525 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce4d180700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce4d180700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce4d180700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:58.526 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "h31"} v 0) v1 -- 0x7fce48063ed0 con 0x7fce481218d0 2024-09-18T10:17:58.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce3effd700 1 -- 172.21.15.150:0/1092863046 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h31"}]=0 device 'h31' does not appear in the crush map v254) v1 ==== 120+0+0 (secure 0 0 0) 0x7fce3001a7a0 con 0x7fce481218d0 2024-09-18T10:17:58.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce3effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h31"}] 2024-09-18T10:17:58.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.521+0000 7fce3effd700 10 monclient: _finish_command 3 = system:0 device 'h31' does not appear in the crush map 2024-09-18T10:17:58.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 msgr2=0x7fce2c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:58.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 0x7fce2c037fa0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fce38007ab0 tx=0x7fce38009040 comp rx=0 tx=0).stop 2024-09-18T10:17:58.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 10 monclient: shutdown 2024-09-18T10:17:58.530 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:58.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 msgr2=0x7fce4813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:58.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 0x7fce4813d700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fce30014000 tx=0x7fce30015000 comp rx=0 tx=0).stop 2024-09-18T10:17:58.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 shutdown_connections 2024-09-18T10:17:58.531 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce2c035af0 0x7fce2c037fa0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce4813e9d0 0x7fce481415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce481218d0 0x7fce4813d700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 --2- 172.21.15.150:0/1092863046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce28004810 0x7fce4813e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 >> 172.21.15.150:0/1092863046 conn(0x7fce48068360 msgr2=0x7fce4809a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 shutdown_connections 2024-09-18T10:17:58.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.525+0000 7fce4d180700 1 -- 172.21.15.150:0/1092863046 wait complete. 2024-09-18T10:17:58.533 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h31' does not appear in the crush map 2024-09-18T10:17:58.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in `seq 1 32` 2024-09-18T10:17:58.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm h32 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.633+0000 7f92fbf70700 10 monclient: get_monmap_and_config 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.633+0000 7f92fbf70700 10 monclient: build_initial_monmap 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.633+0000 7f92fbf70700 10 monclient: monmap: 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:58.641 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:58.639903+0000 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:58.639903+0000 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:58.642 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 Processor -- start 2024-09-18T10:17:58.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 -- start start 2024-09-18T10:17:58.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient: init 2024-09-18T10:17:58.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:58.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:17:58.646 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-c con 0x7f92f4071240 addr v2:172.21.15.150:3300/0 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f4096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-b con 0x7f92f409e1e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92f40994f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f4098cf0 con 0x7f92f40994f0 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f409e640 con 0x7f92f4071240 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f409df80 con 0x7f92f409e1e0 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.647 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fa50d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f92f40994f0 0x7f92f41354c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:38324/0 (socket says 172.21.15.150:38324) 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fa50d700 1 -- 172.21.15.150:0/2697937873 learned_addr learned my addr 172.21.15.150:0/2697937873 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92fbf70700 10 monclient(hunting): authenticate will time out at 1360.734809s 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 10 monclient(hunting): get_auth_request con 0x7f92f4071240 auth_method 0 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 1 -- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 msgr2=0x7f92f4096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 1 -- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 msgr2=0x7f92f4096a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 --2- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f4096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.648 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): get_auth_request con 0x7f92f409e1e0 auth_method 0 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient(hunting): handle_auth_done global_id 74690 payload 274 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 -- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 msgr2=0x7f92f4096a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 --2- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 -- 172.21.15.150:0/2697937873 >> v1:172.21.15.150:6789/0 conn(0x7f92f40994f0 legacy=0x7f92f41354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient: _finish_hunting 0 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 monclient: found mon.noname-b 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:58.649 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:17:58.650 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.637+0000 7f92f950b700 1 -- 172.21.15.150:0/2697937873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4098a10 con 0x7f92f409e1e0 2024-09-18T10:17:58.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f950b700 1 -- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 msgr2=0x7f92f4096fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:58.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f950b700 1 --2- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f4096fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f92e40098e0 tx=0x7f92e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aeedaae02f3a42b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:58.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 1 -- 172.21.15.150:0/2697937873 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4009e80 con 0x7f92f409e1e0 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: dump: 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:58.655 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: _finish_auth 0 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 1 -- 172.21.15.150:0/2697937873 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400ba70 con 0x7f92f409e1e0 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 1 -- 172.21.15.150:0/2697937873 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e400bdf0 con 0x7f92f409e1e0 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92f8d0a700 10 monclient: dump: 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.656 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 5 monclient: authenticate success, global_id 74690 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:58.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 4 set_mon_vals no callback set 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 10 monclient: shutdown 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 1 -- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 msgr2=0x7f92f4096fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.641+0000 7f92fbf70700 1 --2- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f4096fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f92e40098e0 tx=0x7f92e40091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- 172.21.15.150:0/2697937873 shutdown_connections 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 --2- 172.21.15.150:0/2697937873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f4096fb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 --2- 172.21.15.150:0/2697937873 >> v2:172.21.15.150:3300/0 conn(0x7f92f4071240 0x7f92f4096a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- 172.21.15.150:0/2697937873 >> 172.21.15.150:0/2697937873 conn(0x7f92f40a1cc0 msgr2=0x7f92f40a20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- 172.21.15.150:0/2697937873 shutdown_connections 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- 172.21.15.150:0/2697937873 wait complete. 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient: build_initial_monmap 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient: monmap: 2024-09-18T10:17:58.658 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 Processor -- start 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- start start 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient: init 2024-09-18T10:17:58.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 0x7f92f413e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-a con 0x7f92f4071240 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-b con 0x7f92f40994f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f41415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): picked mon.noname-c con 0x7f92f409e1e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4123070 con 0x7f92f4071240 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4122ab0 con 0x7f92f40994f0 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:58.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4120590 con 0x7f92f409e1e0 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fbf70700 10 monclient(hunting): authenticate will time out at 1360.742729s 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52596/0 (socket says 172.21.15.150:52596) 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 -- 172.21.15.150:0/3175460648 learned_addr learned my addr 172.21.15.150:0/3175460648 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 10 monclient(hunting): get_auth_request con 0x7f92f40994f0 auth_method 0 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f41415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 msgr2=0x7f92f413d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 msgr2=0x7f92f413d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:58.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): get_auth_request con 0x7f92f409e1e0 auth_method 0 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 0x7f92f413e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): get_auth_request con 0x7f92f4071240 auth_method 0 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f950b700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:58.662 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient(hunting): handle_auth_done global_id 94293 payload 824 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 msgr2=0x7f92f41415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f41415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 msgr2=0x7f92f413d730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient: _finish_hunting 0 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 monclient: found mon.noname-a 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:58.663 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92f9d0c700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4009420 con 0x7f92f4071240 2024-09-18T10:17:58.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.645+0000 7f92fa50d700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f41415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92f9d0c700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 msgr2=0x7f92f413e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92f9d0c700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 0x7f92f413e080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d890 tx=0x7f92f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0012800 con 0x7f92f4071240 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: dump: 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:58.666 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 5 monclient: authenticate success, global_id 94293 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f0012980 con 0x7f92f4071240 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0012d40 con 0x7f92f4071240 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _renew_subs 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:58.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f41227d0 con 0x7f92f4071240 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92ebfff700 4 set_mon_vals no callback set 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92e9ffb700 10 monclient: dump: 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:58.668 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _renew_subs 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f413de10 con 0x7f92f4071240 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.649+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f4122eb0 con 0x7f92f4071240 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f92f0014640 con 0x7f92f4071240 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92e9ffb700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 0x7f92d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(254..254 src has 176..254) v4 ==== 8480+0+0 (secure 0 0 0) 0x7f92f0049560 con 0x7f92f4071240 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92f950b700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 0x7f92d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:58.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92f950b700 10 monclient: get_auth_request con 0x7f92d8035ac0 auth_method 0 2024-09-18T10:17:58.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.653+0000 7f92f950b700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 0x7f92d8037f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f92f4065450 tx=0x7f92e4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.665+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f92f000fd30 con 0x7f92f4071240 2024-09-18T10:17:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.665+0000 7f92e9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:58.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.665+0000 7f92e9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:58.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.837+0000 7f92fbf70700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:58.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.837+0000 7f92fbf70700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:58.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.837+0000 7f92fbf70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:58.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:58.837+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f92f4063ed0 con 0x7f92f4071240 2024-09-18T10:17:58.887 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: get_monmap_and_config 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: build_initial_monmap 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: monmap: 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.002 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:17:59.000480+0000 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:17:59.000480+0000 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:59.003 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 Processor -- start 2024-09-18T10:17:59.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 -- start start 2024-09-18T10:17:59.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: init 2024-09-18T10:17:59.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:59.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:17:59.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ef80a2700 addr v1:172.21.15.150:6789/0 2024-09-18T10:17:59.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ef8097ad0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7ef809e290 0x7f7ef81374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-a con 0x7f7ef809e290 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef809b910 con 0x7f7ef8097ad0 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef80a2ad0 con 0x7f7ef80a2700 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef8098730 con 0x7f7ef809e290 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7efcb87700 10 monclient(hunting): authenticate will time out at 1367.927860s 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7eeffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7ef8097ad0 0x7f7ef8097160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:39508/0 (socket says 172.21.15.142:39508) 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7ef7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7ef809e290 0x7f7ef81374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:58.997+0000 7f7eeffff700 1 -- 172.21.15.142:0/804059329 learned_addr learned my addr 172.21.15.142:0/804059329 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): get_auth_request con 0x7f7ef809e290 auth_method 0 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_done global_id 94296 payload 274 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 -- 172.21.15.142:0/804059329 >> v1:172.21.15.150:6789/0 conn(0x7f7ef80a2700 legacy=0x7f7ef8096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 -- 172.21.15.142:0/804059329 >> v1:172.21.15.142:6789/0 conn(0x7f7ef8097ad0 legacy=0x7f7ef8097160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 monclient: found mon.noname-a 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:59.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 -- 172.21.15.142:0/804059329 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8097870 con 0x7f7ef809e290 2024-09-18T10:17:59.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 -- 172.21.15.142:0/804059329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef809e290 msgr2=0x7f7ef81374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:59.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef7fff700 1 --2- 172.21.15.142:0/804059329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef809e290 0x7f7ef81374c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009e60 tx=0x7f7ef0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46900b4c1b94782 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:59.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 1 -- 172.21.15.142:0/804059329 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000e070 con 0x7f7ef809e290 2024-09-18T10:17:59.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: dump: 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:17:59.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 1 -- 172.21.15.142:0/804059329 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef0012430 con 0x7f7ef809e290 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 5 monclient: authenticate success, global_id 94296 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 1 -- 172.21.15.142:0/804059329 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00127d0 con 0x7f7ef809e290 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 4 set_mon_vals no callback set 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7ef6ffd700 10 monclient: dump: 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.012 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 10 monclient: shutdown 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 1 -- 172.21.15.142:0/804059329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef809e290 msgr2=0x7f7ef81374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.001+0000 7f7efcb87700 1 --2- 172.21.15.142:0/804059329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef809e290 0x7f7ef81374c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009e60 tx=0x7f7ef0009780 comp rx=0 tx=0).stop 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- 172.21.15.142:0/804059329 shutdown_connections 2024-09-18T10:17:59.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 --2- 172.21.15.142:0/804059329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef809e290 0x7f7ef81374c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- 172.21.15.142:0/804059329 >> 172.21.15.142:0/804059329 conn(0x7f7ef8071700 msgr2=0x7f7ef80a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- 172.21.15.142:0/804059329 shutdown_connections 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- 172.21.15.142:0/804059329 wait complete. 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient: build_initial_monmap 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient: monmap: 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:59.014 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:59.015 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:59.015 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 Processor -- start 2024-09-18T10:17:59.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- start start 2024-09-18T10:17:59.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient: init 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-a con 0x7f7ef8097ad0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-b con 0x7f7ef809e290 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): picked mon.noname-c con 0x7f7ef80a2700 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:59.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8122900 con 0x7f7ef8097ad0 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8120a80 con 0x7f7ef809e290 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8097ec0 con 0x7f7ef80a2700 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:59.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7efcb87700 10 monclient(hunting): authenticate will time out at 1367.934190s 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:36426/0 (socket says 172.21.15.142:36426) 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 -- 172.21.15.142:0/1663210437 learned_addr learned my addr 172.21.15.142:0/1663210437 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): get_auth_request con 0x7f7ef80a2700 auth_method 0 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50700/0 (socket says 172.21.15.142:50700) 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): get_auth_request con 0x7f7ef8097ad0 auth_method 0 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 10 monclient(hunting): get_auth_request con 0x7f7ef809e290 auth_method 0 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 msgr2=0x7f7ef811f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:17:59.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 msgr2=0x7f7ef811f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7eeffff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient(hunting): handle_auth_done global_id 74696 payload 824 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 msgr2=0x7f7ef811f4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 msgr2=0x7f7ef811efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 monclient: found mon.noname-c 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef7fff700 1 -- 172.21.15.142:0/1663210437 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef0009a20 con 0x7f7ef80a2700 2024-09-18T10:17:59.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.005+0000 7f7ef77fe700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7ef7fff700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 msgr2=0x7f7ef811fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7ef7fff700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000b6c0 tx=0x7f7ef0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0026430 con 0x7f7ef80a2700 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: dump: 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.024 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 5 monclient: authenticate success, global_id 74696 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef0021070 con 0x7f7ef80a2700 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _renew_subs 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0026820 con 0x7f7ef80a2700 2024-09-18T10:17:59.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef806d210 con 0x7f7ef80a2700 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7ef67fc700 4 set_mon_vals no callback set 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 10 monclient: dump: 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:17:59.026 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _renew_subs 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef8122520 con 0x7f7ef80a2700 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7ef0023030 con 0x7f7ef80a2700 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 0x7f7ed8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(254..254 src has 176..254) v4 ==== 8480+0+0 (secure 0 0 0) 0x7f7ef004bb20 con 0x7f7ef80a2700 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef8072a80 con 0x7f7ef80a2700 2024-09-18T10:17:59.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.009+0000 7f7ef77fe700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 0x7f7ed8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.013+0000 7f7ef77fe700 10 monclient: get_auth_request con 0x7f7ed8035ac0 auth_method 0 2024-09-18T10:17:59.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.013+0000 7f7ef77fe700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 0x7f7ed8037f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8002e00 tx=0x7f7ee8007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:59.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.017+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7ef001d780 con 0x7f7ef80a2700 2024-09-18T10:17:59.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.017+0000 7f7eef7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:17:59.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.017+0000 7f7eef7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:17:59.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7efcb87700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:17:59.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7efcb87700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:59.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7efcb87700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7ef8122cb0 con 0x7f7ef80a2700 2024-09-18T10:17:59.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7eef7fe700 1 -- 172.21.15.142:0/1663210437 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f7ef00247f0 con 0x7f7ef80a2700 2024-09-18T10:17:59.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7eef7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:17:59.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.241+0000 7f7eef7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:17:59.250 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":94,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:17:59.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 msgr2=0x7f7ed8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 0x7f7ed8037f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8002e00 tx=0x7f7ee8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 10 monclient: shutdown 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 msgr2=0x7f7ef811fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000b6c0 tx=0x7f7ef0018000 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 shutdown_connections 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7ed8035ac0 0x7f7ed8037f70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ef80a2700 0x7f7ef811fa20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef809e290 0x7f7ef811f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 --2- 172.21.15.142:0/1663210437 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ef8097ad0 0x7f7ef811efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 >> 172.21.15.142:0/1663210437 conn(0x7f7ef8071700 msgr2=0x7f7ef80a0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:59.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 shutdown_connections 2024-09-18T10:17:59.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:17:59.245+0000 7f7efcb87700 1 -- 172.21.15.142:0/1663210437 wait complete. 2024-09-18T10:17:59.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.741+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 removed item id -112 name 'h32' from crush map v255) v1 ==== 121+0+0 (secure 0 0 0) 0x7f92f0011e60 con 0x7f92f4071240 2024-09-18T10:17:59.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.741+0000 7f92e9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:59.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.741+0000 7f92e9ffb700 10 monclient: _finish_command 2 = system:0 removed item id -112 name 'h32' from crush map 2024-09-18T10:17:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92fbf70700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92fbf70700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92fbf70700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:17:59.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd crush rm", "name": "h32"} v 0) v1 -- 0x7f92f40008d0 con 0x7f92f4071240 2024-09-18T10:17:59.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92e9ffb700 1 -- 172.21.15.150:0/3175460648 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "h32"}]=0 device 'h32' does not appear in the crush map v255) v1 ==== 120+0+0 (secure 0 0 0) 0x7f92f000f730 con 0x7f92f4071240 2024-09-18T10:17:59.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92e9ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "h32"}] 2024-09-18T10:17:59.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.853+0000 7f92e9ffb700 10 monclient: _finish_command 3 = system:0 device 'h32' does not appear in the crush map 2024-09-18T10:17:59.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 msgr2=0x7f92d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 0x7f92d8037f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f92f4065450 tx=0x7f92e4016000 comp rx=0 tx=0).stop 2024-09-18T10:17:59.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 10 monclient: shutdown 2024-09-18T10:17:59.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:59.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 msgr2=0x7f92f413e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 0x7f92f413e080 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d890 tx=0x7f92f0008040 comp rx=0 tx=0).stop 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 shutdown_connections 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f92d8035ac0 0x7f92d8037f70 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f92f409e1e0 0x7f92f41415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f40994f0 0x7f92f413d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 --2- 172.21.15.150:0/3175460648 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f92f4071240 0x7f92f413e080 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 >> 172.21.15.150:0/3175460648 conn(0x7f92f40a1cc0 msgr2=0x7f92f409c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 shutdown_connections 2024-09-18T10:17:59.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.857+0000 7f92fbf70700 1 -- 172.21.15.150:0/3175460648 wait complete. 2024-09-18T10:17:59.866 INFO:tasks.workunit.client.0.smithi150.stderr:device 'h32' does not appear in the crush map 2024-09-18T10:17:59.878 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush rm r1 2024-09-18T10:17:59.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 10 monclient: get_monmap_and_config 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 10 monclient: build_initial_monmap 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 10 monclient: monmap: 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:17:59.971209+0000 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:17:59.971209+0000 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:17:59.973 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:17:59.974 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:17:59.974 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 1 Processor -- start 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 1 -- start start 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.965+0000 7f26e6fb5700 10 monclient: init 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f26e009e6d0 0x7f26e00a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-a con 0x7f26e009e6d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:17:59.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26e00a07d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-b con 0x7f26e00a0bd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e009e220 con 0x7f26e00a07d0 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e009ee30 con 0x7f26e009e6d0 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e009eaa0 con 0x7f26e00a0bd0 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e6fb5700 10 monclient(hunting): authenticate will time out at 1362.064401s 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f26e009e6d0 0x7f26e00a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26dffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f26e00a07d0 0x7f26e0136a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:37326/0 (socket says 172.21.15.150:37326) 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26dffff700 1 -- 172.21.15.150:0/2302596573 learned_addr learned my addr 172.21.15.150:0/2302596573 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f26e009e6d0 0x7f26e00a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:54124/0 (socket says 172.21.15.150:54124) 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.150:45954/0 (socket says 172.21.15.150:45954) 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): get_auth_request con 0x7f26e009e6d0 auth_method 0 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.978 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): get_auth_request con 0x7f26e00a0bd0 auth_method 0 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient(hunting): handle_auth_done global_id 94302 payload 274 2024-09-18T10:17:59.979 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 -- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 msgr2=0x7f26e013a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 -- 172.21.15.150:0/2302596573 >> v1:172.21.15.142:6789/0 conn(0x7f26e00a07d0 legacy=0x7f26e0136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient: _finish_hunting 0 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 monclient: found mon.noname-a 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e4d51700 1 -- 172.21.15.150:0/2302596573 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e009d930 con 0x7f26e009e6d0 2024-09-18T10:17:59.980 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.969+0000 7f26e5552700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e4d51700 1 -- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 msgr2=0x7f26e00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e4d51700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e00a0290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f26d80098e0 tx=0x7f26d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83307891b9eb6db server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 1 -- 172.21.15.150:0/2302596573 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800b850 con 0x7f26e009e6d0 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:17:59.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: dump: 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: _finish_auth 0 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 5 monclient: authenticate success, global_id 94302 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 1 -- 172.21.15.150:0/2302596573 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d800b9b0 con 0x7f26e009e6d0 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: get_monmap_and_config success 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 4 set_mon_vals no callback set 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 1 -- 172.21.15.150:0/2302596573 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d800bd50 con 0x7f26e009e6d0 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26df7fe700 10 monclient: dump: 2024-09-18T10:17:59.987 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: shutdown 2024-09-18T10:17:59.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 msgr2=0x7f26e00a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e00a0290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f26d80098e0 tx=0x7f26d80091a0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- 172.21.15.150:0/2302596573 shutdown_connections 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e013a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/2302596573 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e00a0290 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- 172.21.15.150:0/2302596573 >> 172.21.15.150:0/2302596573 conn(0x7f26e00a58a0 msgr2=0x7f26e00a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- 172.21.15.150:0/2302596573 shutdown_connections 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- 172.21.15.150:0/2302596573 wait complete. 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: build_initial_monmap 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: monmap: 2024-09-18T10:17:59.989 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 Processor -- start 2024-09-18T10:17:59.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- start start 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: init 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: _reopen_session rank -1 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e0141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-a con 0x7f26e009e6d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-b con 0x7f26e00a07d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): picked mon.noname-c con 0x7f26e00a0bd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.991 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0092840 con 0x7f26e009e6d0 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00994e0 con 0x7f26e00a07d0 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e009c230 con 0x7f26e00a0bd0 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): _renew_subs 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26e6fb5700 10 monclient(hunting): authenticate will time out at 1362.071640s 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49182/0 (socket says 172.21.15.150:49182) 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 1 -- 172.21.15.150:0/343658335 learned_addr learned my addr 172.21.15.150:0/343658335 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 10 monclient(hunting): get_auth_request con 0x7f26e00a07d0 auth_method 0 2024-09-18T10:17:59.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.973+0000 7f26dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e0141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): get_auth_request con 0x7f26e00a0bd0 auth_method 0 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): get_auth_request con 0x7f26e009e6d0 auth_method 0 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:17:59.993 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dffff700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 msgr2=0x7f26e0142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dffff700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 msgr2=0x7f26e0142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dffff700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dffff700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:17:59.994 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient(hunting): handle_auth_done global_id 74702 payload 824 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 msgr2=0x7f26e0142020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 msgr2=0x7f26e0141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e0141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient: _finish_hunting 0 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 monclient: found mon.noname-c 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d0009420 con 0x7f26e00a0bd0 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e4d51700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e0141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 msgr2=0x7f26e0142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:17:59.995 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e5552700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26d000ea50 tx=0x7f26d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0009670 con 0x7f26e00a0bd0 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: dump: 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.996 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: _finish_auth 0 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 5 monclient: authenticate success, global_id 74702 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d000ca60 con 0x7f26e00a0bd0 2024-09-18T10:17:59.997 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 10 monclient: _renew_subs 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d000ce20 con 0x7f26e00a0bd0 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d8009420 con 0x7f26e00a0bd0 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26deffd700 4 set_mon_vals no callback set 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26dcff9700 10 monclient: dump: 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:17:59.998 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 10 monclient: _renew_subs 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:17:59.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.977+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e00912e0 con 0x7f26e00a0bd0 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e6fb5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e6fb5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e6fb5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e5552700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26d000ea50 tx=0x7f26d00091a0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f26d0049210 osd_map(255..255 src has 176..255) v4 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f26d0013680 con 0x7f26e00a0bd0 2024-09-18T10:18:00.001 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26dcff9700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 0x7f26c4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:00.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e4d51700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 0x7f26c4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:00.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.981+0000 7f26e4d51700 10 monclient: get_auth_request con 0x7f26c4035ac0 auth_method 0 2024-09-18T10:18:00.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.985+0000 7f26e4d51700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 0x7f26c4037f70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f26d80030c0 tx=0x7f26d8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:00.002 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:17:59.993+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(255..255 src has 176..255) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f26d0049210 con 0x7f26e00a0bd0 2024-09-18T10:18:00.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.809+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:00.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.809+0000 7f26dcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:00.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.809+0000 7f26dcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:00.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.977+0000 7f26e6fb5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:00.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.977+0000 7f26e6fb5700 10 monclient: _send_command 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:00.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.977+0000 7f26e6fb5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:00.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:00.977+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:02.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.033+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 removed item id -16 name 'r1' from crush map v256) v1 ==== 118+0+0 (secure 0 0 0) 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:02.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.033+0000 7f26dcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:02.039 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.033+0000 7f26dcff9700 10 monclient: _finish_command 2 = system:0 removed item id -16 name 'r1' from crush map 2024-09-18T10:18:02.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.141+0000 7f26e6fb5700 10 start_mon_command cmd=[{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:02.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.141+0000 7f26e6fb5700 10 monclient: _send_command 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:02.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.141+0000 7f26e6fb5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.141+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush rm", "name": "r1"} v 0) v1 -- 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:02.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26dcff9700 1 -- 172.21.15.150:0/343658335 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush rm", "name": "r1"}]=0 device 'r1' does not appear in the crush map v256) v1 ==== 118+0+0 (secure 0 0 0) 0x7f26e0091a60 con 0x7f26e00a0bd0 2024-09-18T10:18:02.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26dcff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush rm", "name": "r1"}] 2024-09-18T10:18:02.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26dcff9700 10 monclient: _finish_command 3 = system:0 device 'r1' does not appear in the crush map 2024-09-18T10:18:02.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 msgr2=0x7f26c4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 0x7f26c4037f70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f26d80030c0 tx=0x7f26d8016000 comp rx=0 tx=0).stop 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 10 monclient: shutdown 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 msgr2=0x7f26e0142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f26d000ea50 tx=0x7f26d00091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 shutdown_connections 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f26c4035ac0 0x7f26c4037f70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26e00a0bd0 0x7f26e0142560 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26e00a07d0 0x7f26e0142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 --2- 172.21.15.150:0/343658335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f26e009e6d0 0x7f26e0141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 >> 172.21.15.150:0/343658335 conn(0x7f26e00a58a0 msgr2=0x7f26e009ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 shutdown_connections 2024-09-18T10:18:02.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.145+0000 7f26e6fb5700 1 -- 172.21.15.150:0/343658335 wait complete. 2024-09-18T10:18:02.154 INFO:tasks.workunit.client.0.smithi150.stderr:device 'r1' does not appear in the crush map 2024-09-18T10:18:02.163 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd crush weight-set rm-compat 2024-09-18T10:18:02.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 10 monclient: get_monmap_and_config 2024-09-18T10:18:02.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 10 monclient: monmap: 2024-09-18T10:18:02.259 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:02.257942+0000 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:02.257942+0000 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:02.260 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:02.261 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:02.261 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:02.261 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:02.261 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 1 Processor -- start 2024-09-18T10:18:02.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 1 -- start start 2024-09-18T10:18:02.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.253+0000 7f0b963bc700 10 monclient: init 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 0x7f0b9009ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-b con 0x7f0b90071a40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b9006e7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b9009f2a0 0x7f0b9006cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-a con 0x7f0b9009f2a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b90071e10 con 0x7f0b9006e7f0 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b90096c70 con 0x7f0b9009f2a0 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b90130470 con 0x7f0b90071a40 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b963bc700 10 monclient(hunting): authenticate will time out at 1364.352721s 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b87fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b9006e7f0 0x7f0b9006c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44400/0 (socket says 172.21.15.150:44400) 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b87fff700 1 -- 172.21.15.150:0/2849882635 learned_addr learned my addr 172.21.15.150:0/2849882635 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 0x7f0b9009ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 1 --2- 172.21.15.150:0/2849882635 >> v2:172.21.15.142:3300/0 conn(0x7f0b9009f2a0 0x7f0b9006cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): get_auth_request con 0x7f0b90071a40 auth_method 0 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): get_auth_request con 0x7f0b9009f2a0 auth_method 0 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b94959700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_done global_id 74705 payload 274 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 -- 172.21.15.150:0/2849882635 >> v2:172.21.15.142:3300/0 conn(0x7f0b9009f2a0 msgr2=0x7f0b9006cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/2849882635 >> v2:172.21.15.142:3300/0 conn(0x7f0b9009f2a0 0x7f0b9006cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 -- 172.21.15.150:0/2849882635 >> v1:172.21.15.150:6789/0 conn(0x7f0b9006e7f0 legacy=0x7f0b9006c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 monclient: found mon.noname-b 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.257+0000 7f0b8ffff700 1 -- 172.21.15.150:0/2849882635 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90096f30 con 0x7f0b90071a40 2024-09-18T10:18:02.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8ffff700 1 -- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 msgr2=0x7f0b9009ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 0x7f0b9009ed60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=234a8bd9d91d60de server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 1 -- 172.21.15.150:0/2849882635 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000b850 con 0x7f0b90071a40 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: dump: 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.279 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.280 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 5 monclient: authenticate success, global_id 74705 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 1 -- 172.21.15.150:0/2849882635 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8000b9b0 con 0x7f0b90071a40 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 4 set_mon_vals no callback set 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 1 -- 172.21.15.150:0/2849882635 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000bd50 con 0x7f0b90071a40 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b8f7fe700 10 monclient: dump: 2024-09-18T10:18:02.281 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 10 monclient: shutdown 2024-09-18T10:18:02.282 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 -- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 msgr2=0x7f0b9009ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 --2- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 0x7f0b9009ed60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 -- 172.21.15.150:0/2849882635 shutdown_connections 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 --2- 172.21.15.150:0/2849882635 >> v2:172.21.15.142:3300/0 conn(0x7f0b9009f2a0 0x7f0b9006cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 --2- 172.21.15.150:0/2849882635 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b90071a40 0x7f0b9009ed60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 -- 172.21.15.150:0/2849882635 >> 172.21.15.150:0/2849882635 conn(0x7f0b900a41f0 msgr2=0x7f0b900710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 -- 172.21.15.150:0/2849882635 shutdown_connections 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 1 -- 172.21.15.150:0/2849882635 wait complete. 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.283 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.261+0000 7f0b963bc700 10 monclient: monmap: 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:02.284 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 Processor -- start 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 -- start start 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient: init 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 0x7f0b9012f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-c con 0x7f0b9006e7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 0x7f0b9013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-a con 0x7f0b90071a40 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.285 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): picked mon.noname-b con 0x7f0b9009f2a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9013fe70 con 0x7f0b90071a40 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9011b810 con 0x7f0b9009f2a0 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9011d650 con 0x7f0b9006e7f0 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b963bc700 10 monclient(hunting): authenticate will time out at 1364.361468s 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49194/0 (socket says 172.21.15.150:49194) 2024-09-18T10:18:02.286 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 -- 172.21.15.150:0/795404998 learned_addr learned my addr 172.21.15.150:0/795404998 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 10 monclient(hunting): get_auth_request con 0x7f0b9009f2a0 auth_method 0 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 msgr2=0x7f0b9013f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 msgr2=0x7f0b9013f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 0x7f0b9012f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 0x7f0b9013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.287 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b94959700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): get_auth_request con 0x7f0b9006e7f0 auth_method 0 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): get_auth_request con 0x7f0b90071a40 auth_method 0 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.288 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient(hunting): handle_auth_done global_id 74708 payload 824 2024-09-18T10:18:02.289 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 msgr2=0x7f0b9013f930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 msgr2=0x7f0b9013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 0x7f0b9013f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 monclient: found mon.noname-c 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b8ffff700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b80009420 con 0x7f0b9006e7f0 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.265+0000 7f0b87fff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 0x7f0b9013f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8ffff700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 msgr2=0x7f0b9012f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8ffff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 0x7f0b9012f210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0b90069910 tx=0x7f0b80016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80011bb0 con 0x7f0b9006e7f0 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.290 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: dump: 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.291 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80020070 con 0x7f0b9006e7f0 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8001b450 con 0x7f0b9006e7f0 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 10 monclient: dump: 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.292 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 5 monclient: authenticate success, global_id 74708 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _renew_subs 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9011c5c0 con 0x7f0b9006e7f0 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8effd700 4 set_mon_vals no callback set 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _renew_subs 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9011d900 con 0x7f0b9006e7f0 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.293 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b90097210 con 0x7f0b9006e7f0 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b80022030 con 0x7f0b9006e7f0 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.269+0000 7f0b8cff9700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 0x7f0b70037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.273+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(256..256 src has 176..256) v4 ==== 8057+0+0 (secure 0 0 0) 0x7f0b8004ac70 con 0x7f0b9006e7f0 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.273+0000 7f0b87fff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 0x7f0b70037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.294 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.273+0000 7f0b87fff700 10 monclient: get_auth_request con 0x7f0b70035ac0 auth_method 0 2024-09-18T10:18:02.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.273+0000 7f0b87fff700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 0x7f0b70037f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0b78005220 tx=0x7f0b78007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.281+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0b80023380 con 0x7f0b9006e7f0 2024-09-18T10:18:02.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.281+0000 7f0b8cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.295 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.281+0000 7f0b8cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:02.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:02.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: monmap: 2024-09-18T10:18:02.389 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.389 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:02.387843+0000 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:02.387843+0000 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:02.390 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 Processor -- start 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 -- start start 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: init 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15bc0676a0 0x7f15bc09a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f15bc0676a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:02.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15bc09c7a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f15bc09d2b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15bc098e60 con 0x7f15bc09c7a0 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15bc0998e0 con 0x7f15bc0676a0 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc067a70 con 0x7f15bc09d2b0 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bbfff700 10 monclient(hunting): authenticate will time out at 1371.313377s 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15bc0676a0 0x7f15bc09a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15bc0676a0 0x7f15bc09a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51842/0 (socket says 172.21.15.142:51842) 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 -- 172.21.15.142:0/292305043 learned_addr learned my addr 172.21.15.142:0/292305043 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:02.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): get_auth_request con 0x7f15bc0676a0 auth_method 0 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 10 monclient(hunting): get_auth_request con 0x7f15bc09d2b0 auth_method 0 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient(hunting): handle_auth_done global_id 94314 payload 274 2024-09-18T10:18:02.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 -- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 msgr2=0x7f15bc1375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc1375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 -- 172.21.15.142:0/292305043 >> v1:172.21.15.150:6789/0 conn(0x7f15bc09c7a0 legacy=0x7f15bc09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 monclient: found mon.noname-a 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15baffd700 1 -- 172.21.15.142:0/292305043 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc099600 con 0x7f15bc0676a0 2024-09-18T10:18:02.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.385+0000 7f15bb7fe700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc1375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15baffd700 1 -- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc0676a0 msgr2=0x7f15bc09a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15baffd700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc0676a0 0x7f15bc09a2c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009f20 tx=0x7f15b4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1203e301340e13 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 1 -- 172.21.15.142:0/292305043 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b400e070 con 0x7f15bc0676a0 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: dump: 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.402 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 5 monclient: authenticate success, global_id 94314 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 1 -- 172.21.15.142:0/292305043 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b400bc90 con 0x7f15bc0676a0 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 1 -- 172.21.15.142:0/292305043 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b40092e0 con 0x7f15bc0676a0 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15b9ffb700 10 monclient: dump: 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 4 set_mon_vals no callback set 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 10 monclient: shutdown 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc0676a0 msgr2=0x7f15bc09a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc0676a0 0x7f15bc09a2c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009f20 tx=0x7f15b4009840 comp rx=0 tx=0).stop 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- 172.21.15.142:0/292305043 shutdown_connections 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc1375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 --2- 172.21.15.142:0/292305043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc0676a0 0x7f15bc09a2c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- 172.21.15.142:0/292305043 >> 172.21.15.142:0/292305043 conn(0x7f15bc071ae0 msgr2=0x7f15bc0a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- 172.21.15.142:0/292305043 shutdown_connections 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- 172.21.15.142:0/292305043 wait complete. 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 10 monclient: monmap: 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 Processor -- start 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 1 -- start start 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.389+0000 7f15bbfff700 10 monclient: init 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:02.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f15bc0676a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc09c7a0 0x7f15bc13b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f15bc09c7a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f15bc09d2b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc120ad0 con 0x7f15bc09c7a0 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc1207f0 con 0x7f15bc0676a0 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15bc120510 con 0x7f15bc09d2b0 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bbfff700 10 monclient(hunting): authenticate will time out at 1371.320736s 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50852/0 (socket says 172.21.15.142:50852) 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42452/0 (socket says 172.21.15.142:42452) 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 -- 172.21.15.142:0/3667876512 learned_addr learned my addr 172.21.15.142:0/3667876512 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 10 monclient(hunting): get_auth_request con 0x7f15bc0676a0 auth_method 0 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): get_auth_request con 0x7f15bc09d2b0 auth_method 0 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc09c7a0 0x7f15bc13b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 msgr2=0x7f15bc13bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 msgr2=0x7f15bc13bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): get_auth_request con 0x7f15bc09c7a0 auth_method 0 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15baffd700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient(hunting): handle_auth_done global_id 74714 payload 824 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 msgr2=0x7f15bc13bdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc09c7a0 msgr2=0x7f15bc13b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc09c7a0 0x7f15bc13b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 monclient: found mon.noname-c 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 -- 172.21.15.142:0/3667876512 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4009ae0 con 0x7f15bc09d2b0 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 msgr2=0x7f15bc13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.393+0000 7f15bb7fe700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009c60 tx=0x7f15a4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a4015070 con 0x7f15bc09d2b0 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: dump: 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.411 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 5 monclient: authenticate success, global_id 74714 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15a4012540 con 0x7f15bc09d2b0 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _renew_subs 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15a40128e0 con 0x7f15bc09d2b0 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15bc11e280 con 0x7f15bc09d2b0 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b97fa700 4 set_mon_vals no callback set 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 10 monclient: dump: 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _renew_subs 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15bc13c9e0 con 0x7f15bc09d2b0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15bc072a80 con 0x7f15bc09d2b0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f15a4012c90 con 0x7f15bc09d2b0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 0x7f159c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(256..256 src has 176..256) v4 ==== 8057+0+0 (secure 0 0 0) 0x7f15a404ca10 con 0x7f15bc09d2b0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15baffd700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 0x7f159c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.397+0000 7f15baffd700 10 monclient: get_auth_request con 0x7f159c035b10 auth_method 0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.401+0000 7f15baffd700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 0x7f159c037fc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009a60 tx=0x7f15b4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.405+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f15a40223b0 con 0x7f15bc09d2b0 2024-09-18T10:18:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.405+0000 7f15b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.405+0000 7f15b2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:02.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.453+0000 7f0b963bc700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:02.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.453+0000 7f0b963bc700 10 monclient: _send_command 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:02.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.453+0000 7f0b963bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.461 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:02.453+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0b90097210 con 0x7f0b9006e7f0 2024-09-18T10:18:02.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15bbfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:02.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15bbfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15bbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f15bc068360 con 0x7f15bc09d2b0 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15b2ffd700 1 -- 172.21.15.142:0/3667876512 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f15a40172f0 con 0x7f15bc09d2b0 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15b2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.629+0000 7f15b2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:02.637 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:02.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 msgr2=0x7f159c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 0x7f159c037fc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009a60 tx=0x7f15b4017000 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 10 monclient: shutdown 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 msgr2=0x7f15bc13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009c60 tx=0x7f15a4008040 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 shutdown_connections 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f159c035b10 0x7f159c037fc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15bc09d2b0 0x7f15bc13f330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15bc09c7a0 0x7f15bc13b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 --2- 172.21.15.142:0/3667876512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0676a0 0x7f15bc13bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 >> 172.21.15.142:0/3667876512 conn(0x7f15bc071ae0 msgr2=0x7f15bc09eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 shutdown_connections 2024-09-18T10:18:02.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.633+0000 7f15bbfff700 1 -- 172.21.15.142:0/3667876512 wait complete. 2024-09-18T10:18:02.651 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:18:02.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:18:02.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: monmap: 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:02.753341+0000 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:02.753341+0000 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.755 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:02.756 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:02.756 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:02.756 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:02.756 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:02.756 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 Processor -- start 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 -- start start 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: init 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-a con 0x7f3d480999e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-c con 0x7f3d4809c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 0x7f3d48138910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-b con 0x7f3d480a0c00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4809a000 con 0x7f3d480999e0 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d480a20c0 con 0x7f3d4809c410 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d480a1c80 con 0x7f3d480a0c00 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d47fff700 10 monclient(hunting): authenticate will time out at 1371.679337s 2024-09-18T10:18:02.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 0x7f3d48138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51856/0 (socket says 172.21.15.142:51856) 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 1 -- 172.21.15.142:0/4123738889 learned_addr learned my addr 172.21.15.142:0/4123738889 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request con 0x7f3d480999e0 auth_method 0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): get_auth_request con 0x7f3d480a0c00 auth_method 0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d467fc700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d467fc700 10 monclient(hunting): get_auth_request con 0x7f3d4809c410 auth_method 0 2024-09-18T10:18:02.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.749+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d467fc700 1 -- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 msgr2=0x7f3d480a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d467fc700 1 -- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 msgr2=0x7f3d480a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d467fc700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d467fc700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:02.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 10 monclient(hunting): handle_auth_done global_id 74717 payload 274 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 -- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 msgr2=0x7f3d480a06c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 -- 172.21.15.142:0/4123738889 >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 msgr2=0x7f3d480a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 monclient: found mon.noname-b 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 -- 172.21.15.142:0/4123738889 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4809a580 con 0x7f3d480a0c00 2024-09-18T10:18:02.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d46ffd700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:02.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 -- 172.21.15.142:0/4123738889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 msgr2=0x7f3d48138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d477fe700 1 --2- 172.21.15.142:0/4123738889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 0x7f3d48138910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d4806fbb0 tx=0x7f3d40009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b54095a471515e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 1 -- 172.21.15.142:0/4123738889 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000e070 con 0x7f3d480a0c00 2024-09-18T10:18:02.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: dump: 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 1 -- 172.21.15.142:0/4123738889 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4000bd30 con 0x7f3d480a0c00 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d47fff700 5 monclient: authenticate success, global_id 74717 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d47fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d47fff700 4 set_mon_vals no callback set 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 1 -- 172.21.15.142:0/4123738889 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000bea0 con 0x7f3d480a0c00 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.753+0000 7f3d45ffb700 10 monclient: dump: 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.769 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: shutdown 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- 172.21.15.142:0/4123738889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 msgr2=0x7f3d48138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- 172.21.15.142:0/4123738889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 0x7f3d48138910 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d4806fbb0 tx=0x7f3d40009780 comp rx=0 tx=0).stop 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- 172.21.15.142:0/4123738889 shutdown_connections 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- 172.21.15.142:0/4123738889 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480a0c00 0x7f3d48138910 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.150:3300/0 conn(0x7f3d4809c410 0x7f3d480a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- 172.21.15.142:0/4123738889 >> v2:172.21.15.142:3300/0 conn(0x7f3d480999e0 0x7f3d480a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- 172.21.15.142:0/4123738889 >> 172.21.15.142:0/4123738889 conn(0x7f3d480a5790 msgr2=0x7f3d480a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- 172.21.15.142:0/4123738889 shutdown_connections 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- 172.21.15.142:0/4123738889 wait complete. 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: monmap: 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.771 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 Processor -- start 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- start start 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: init 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480999e0 0x7f3d48098900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-c con 0x7f3d480999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 0x7f3d48098e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-b con 0x7f3d4809c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:02.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): picked mon.noname-a con 0x7f3d480a0c00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4809f170 con 0x7f3d480a0c00 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812bf60 con 0x7f3d4809c410 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812bc80 con 0x7f3d480999e0 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.757+0000 7f3d47fff700 10 monclient(hunting): authenticate will time out at 1371.687986s 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480999e0 0x7f3d48098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480999e0 0x7f3d48098900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42474/0 (socket says 172.21.15.142:42474) 2024-09-18T10:18:02.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 1 -- 172.21.15.142:0/2087971025 learned_addr learned my addr 172.21.15.142:0/2087971025 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request con 0x7f3d480999e0 auth_method 0 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51872/0 (socket says 172.21.15.142:51872) 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): get_auth_request con 0x7f3d480a0c00 auth_method 0 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 0x7f3d48098e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 10 monclient(hunting): get_auth_request con 0x7f3d4809c410 auth_method 0 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 msgr2=0x7f3d48098e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 msgr2=0x7f3d48098e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 0x7f3d48098e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d467fc700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 0x7f3d48098e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient(hunting): handle_auth_done global_id 94323 payload 824 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480999e0 msgr2=0x7f3d48098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d480999e0 0x7f3d48098900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 msgr2=0x7f3d48098e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d4809c410 0x7f3d48098e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 monclient: found mon.noname-a 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380094f0 con 0x7f3d480a0c00 2024-09-18T10:18:02.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 msgr2=0x7f3d4806daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d477fe700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d40016000 tx=0x7f3d40019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d400263f0 con 0x7f3d480a0c00 2024-09-18T10:18:02.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: dump: 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 5 monclient: authenticate success, global_id 94323 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40026550 con 0x7f3d480a0c00 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40026910 con 0x7f3d480a0c00 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 10 monclient: _renew_subs 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d40009a20 con 0x7f3d480a0c00 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d457fa700 4 set_mon_vals no callback set 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d2effd700 10 monclient: dump: 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.781 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 10 monclient: _renew_subs 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.761+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4812c2f0 con 0x7f3d480a0c00 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d47fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d47fff700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 0x7f3d4806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d480647f0 con 0x7f3d4812cf10 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 0x7f3d4806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient: get_auth_request con 0x7f3d4812cf10 auth_method 0 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3d4001b3f0 con 0x7f3d480a0c00 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_done global_id 94326 payload 824 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d4806f4a0 con 0x7f3d4812cf10 2024-09-18T10:18:02.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d34035ac0 0x7f3d34037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(256..256 src has 176..256) v4 ==== 8057+0+0 (secure 0 0 0) 0x7f3d4004b130 con 0x7f3d480a0c00 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d467fc700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d34035ac0 0x7f3d34037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d467fc700 10 monclient: get_auth_request con 0x7f3d34035ac0 auth_method 0 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 msgr2=0x7f3d4806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d46ffd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 0x7f3d4806dfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38014440 con 0x7f3d4812cf10 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.765+0000 7f3d2effd700 10 monclient: dump: 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.784 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3d380145a0 con 0x7f3d4812cf10 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 msgr2=0x7f3d4806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 0x7f3d4806dfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d38009250 comp rx=0 tx=0).stop 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 msgr2=0x7f3d4806dfe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d2effd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4812cf10 0x7f3d4806dfe0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d38009250 comp rx=0 tx=0).stop 2024-09-18T10:18:02.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.769+0000 7f3d467fc700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d34035ac0 0x7f3d34037f70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c006f30 tx=0x7f3d3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d47fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d47fff700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 0x7f3d4806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d47fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4812d2e0 con 0x7f3d4806e520 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 0x7f3d4806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient: get_auth_request con 0x7f3d4806e520 auth_method 0 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 10 monclient(hunting): handle_auth_done global_id 94329 payload 824 2024-09-18T10:18:02.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.777+0000 7f3d46ffd700 1 -- 172.21.15.142:0/2087971025 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d4806ee60 con 0x7f3d4806e520 2024-09-18T10:18:02.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d46ffd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 msgr2=0x7f3d4806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d46ffd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 0x7f3d4806e920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d3801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38024040 con 0x7f3d4806e520 2024-09-18T10:18:02.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 10 monclient: dump: 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f3d380133a0 con 0x7f3d4806e520 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 msgr2=0x7f3d4806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 0x7f3d4806e920 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d3801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 msgr2=0x7f3d4806e920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d2effd700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d4806e520 0x7f3d4806e920 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d38006e80 tx=0x7f3d3801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:02.790 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:02.791 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:02.792 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:02.793 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:02.794 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:02.795 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.796 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:02.797 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:02.798 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:02.799 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:02.800 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:02.801 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:02.802 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:02.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d34035ac0 msgr2=0x7f3d34037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d34035ac0 0x7f3d34037f70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c006f30 tx=0x7f3d3c008040 comp rx=0 tx=0).stop 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 10 monclient: shutdown 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d477fe700 1 -- 172.21.15.142:0/2087971025 reap_dead start 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 msgr2=0x7f3d4806daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3d40016000 tx=0x7f3d40019000 comp rx=0 tx=0).stop 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 shutdown_connections 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 --2- 172.21.15.142:0/2087971025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d480a0c00 0x7f3d4806daa0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 >> 172.21.15.142:0/2087971025 conn(0x7f3d480a5790 msgr2=0x7f3d48134cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 shutdown_connections 2024-09-18T10:18:02.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.781+0000 7f3d47fff700 1 -- 172.21.15.142:0/2087971025 wait complete. 2024-09-18T10:18:02.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:02.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:18:02.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:18:02.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: get_monmap_and_config 2024-09-18T10:18:02.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: monmap: 2024-09-18T10:18:02.914 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:02.912793+0000 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:02.912793+0000 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:02.915 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 Processor -- start 2024-09-18T10:18:02.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 -- start start 2024-09-18T10:18:02.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: init 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60f80a5f90 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 0x7f60f80a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-b con 0x7f60f80990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f60f809e360 0x7f60f8139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-a con 0x7f60f809e360 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f809d980 con 0x7f60f80a5f90 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f809d780 con 0x7f60f809e360 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f809fc70 con 0x7f60f80990b0 2024-09-18T10:18:02.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f61000bc700 10 monclient(hunting): authenticate will time out at 1371.839013s 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 0x7f60f80a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fde58700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f60f80a5f90 0x7f60f80a0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49010/0 (socket says 172.21.15.142:49010) 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fde58700 1 -- 172.21.15.142:0/2495657160 learned_addr learned my addr 172.21.15.142:0/2495657160 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f60f809e360 0x7f60f8139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): get_auth_request con 0x7f60f80990b0 auth_method 0 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): get_auth_request con 0x7f60f809e360 auth_method 0 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient(hunting): handle_auth_done global_id 94332 payload 274 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 -- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 msgr2=0x7f60f80a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 0x7f60f80a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 -- 172.21.15.142:0/2495657160 >> v1:172.21.15.142:6789/0 conn(0x7f60f80a5f90 legacy=0x7f60f80a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 monclient: found mon.noname-a 2024-09-18T10:18:02.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fe659700 1 -- 172.21.15.142:0/2495657160 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f80a6360 con 0x7f60f809e360 2024-09-18T10:18:02.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.909+0000 7f60fd657700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 0x7f60f80a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:02.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fe659700 1 -- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f809e360 msgr2=0x7f60f8139a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fe659700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f809e360 0x7f60f8139a10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f60f00098e0 tx=0x7f60f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb8bfe0779b1d59 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 1 -- 172.21.15.142:0/2495657160 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f000b850 con 0x7f60f809e360 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: dump: 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.924 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: _finish_auth 0 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 1 -- 172.21.15.142:0/2495657160 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f000b9b0 con 0x7f60f809e360 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f61000bc700 5 monclient: authenticate success, global_id 94332 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f61000bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f61000bc700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f61000bc700 4 set_mon_vals no callback set 2024-09-18T10:18:02.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 1 -- 172.21.15.142:0/2495657160 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f000bd50 con 0x7f60f809e360 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.913+0000 7f60fce56700 10 monclient: dump: 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:02.926 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 10 monclient: shutdown 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f809e360 msgr2=0x7f60f8139a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f809e360 0x7f60f8139a10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f60f00098e0 tx=0x7f60f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- 172.21.15.142:0/2495657160 shutdown_connections 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f80990b0 0x7f60f80a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 --2- 172.21.15.142:0/2495657160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f809e360 0x7f60f8139a10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- 172.21.15.142:0/2495657160 >> 172.21.15.142:0/2495657160 conn(0x7f60f8071680 msgr2=0x7f60f80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- 172.21.15.142:0/2495657160 shutdown_connections 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- 172.21.15.142:0/2495657160 wait complete. 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 10 monclient: build_initial_monmap 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 10 monclient: monmap: 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:02.927 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:02.928 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:02.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 Processor -- start 2024-09-18T10:18:02.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 1 -- start start 2024-09-18T10:18:02.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.917+0000 7f61000bc700 10 monclient: init 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60f808cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-b con 0x7f60f80990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60f808d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-c con 0x7f60f809e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): picked mon.noname-a con 0x7f60f80a5f90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8090720 con 0x7f60f80a5f90 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f809bef0 con 0x7f60f80990b0 2024-09-18T10:18:02.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f808dff0 con 0x7f60f809e360 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f61000bc700 10 monclient(hunting): authenticate will time out at 1371.848696s 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60f808d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60f808d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42492/0 (socket says 172.21.15.142:42492) 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 1 -- 172.21.15.142:0/2439674047 learned_addr learned my addr 172.21.15.142:0/2439674047 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): get_auth_request con 0x7f60f809e360 auth_method 0 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60f808cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:02.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 10 monclient(hunting): get_auth_request con 0x7f60f80990b0 auth_method 0 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): get_auth_request con 0x7f60f80a5f90 auth_method 0 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 msgr2=0x7f60f808cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:02.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 msgr2=0x7f60f808cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60f808cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60f808cb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient(hunting): handle_auth_done global_id 94335 payload 824 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 msgr2=0x7f60f808d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60f808d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 msgr2=0x7f60f808cb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60f808cb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient: _finish_hunting 0 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 monclient: found mon.noname-a 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:02.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec009a20 con 0x7f60f80a5f90 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60f808d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 msgr2=0x7f60f808d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 msgr2=0x7f60f808d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f60f0015000 tx=0x7f60f0016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:02.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:02.921+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.045+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v257) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0b800243d0 con 0x7f0b9006e7f0 2024-09-18T10:18:03.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.045+0000 7f0b8cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:03.049 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.045+0000 7f0b8cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient: get_auth_request con 0x7f60f80a5f90 auth_method 0 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 0 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 msgr2=0x7f60f808d600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60f80a5f90 0x7f60f808d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:03.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60dc005500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f60f809e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60dc005eb0 0x7f60dc008330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f60dc005eb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60dc00aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f60f80990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc00b870 con 0x7f60dc005eb0 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc00b9b0 con 0x7f60f80990b0 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc00bb70 con 0x7f60f809e360 2024-09-18T10:18:03.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60e9ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60dc005500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): get_auth_request con 0x7f60f809e360 auth_method 0 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60dc005eb0 0x7f60dc008330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): get_auth_request con 0x7f60dc005eb0 auth_method 0 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60dc00aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): get_auth_request con 0x7f60f80990b0 auth_method 0 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fe659700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient(hunting): handle_auth_done global_id 94335 payload 293 2024-09-18T10:18:03.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 msgr2=0x7f60dc00aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f80990b0 0x7f60dc00aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60dc005eb0 msgr2=0x7f60dc008330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60dc005eb0 0x7f60dc008330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient: _finish_hunting 0 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 monclient: found mon.noname-c 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc00bdf0 con 0x7f60f809e360 2024-09-18T10:18:03.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.125+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f60dc005eb0 0x7f60dc008330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fde58700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 msgr2=0x7f60dc005500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fde58700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60dc005500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60f4007da0 tx=0x7f60f4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f400ce70 con 0x7f60f809e360 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: dump: 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.139 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f4004c00 con 0x7f60f809e360 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 5 monclient: authenticate success, global_id 94335 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 monclient: _renew_subs 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f0009420 con 0x7f60f809e360 2024-09-18T10:18:03.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4007170 con 0x7f60f809e360 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 10 monclient: dump: 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.141 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 monclient: _renew_subs 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60ebfff700 4 set_mon_vals no callback set 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f8098330 con 0x7f60f809e360 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 0x7f60f808cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f80905e0 con 0x7f60f808f200 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 0x7f60f808cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient: get_auth_request con 0x7f60f808f200 auth_method 0 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f60f4007560 con 0x7f60f809e360 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 10 monclient(hunting): handle_auth_done global_id 74729 payload 824 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60fd657700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60f808e550 con 0x7f60f808f200 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.129+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f60f80a5f90 0x7f60dc0400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(257..257 src has 176..257) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f60f40479f0 con 0x7f60f809e360 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f60f80a5f90 0x7f60dc0400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60fe659700 10 monclient: get_auth_request con 0x7f60f80a5f90 auth_method 0 2024-09-18T10:18:03.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60fd657700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 msgr2=0x7f60f808cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 0x7f60f808cb80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec012ee0 con 0x7f60f808f200 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 10 monclient: dump: 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60fe659700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f60f80a5f90 0x7f60dc0400e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f60f00098e0 tx=0x7f60f001e070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f60ec00f040 con 0x7f60f808f200 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 msgr2=0x7f60f808cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 0x7f60f808cb80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec00b040 comp rx=0 tx=0).stop 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 msgr2=0x7f60f808cb80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:03.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.133+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f808f200 0x7f60f808cb80 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec00b040 comp rx=0 tx=0).stop 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f61000bc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f61000bc700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 0x7f60f808d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f61000bc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f8064950 con 0x7f60f8090e20 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 0x7f60f808d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient: get_auth_request con 0x7f60f8090e20 auth_method 0 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 10 monclient(hunting): handle_auth_done global_id 74732 payload 824 2024-09-18T10:18:03.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.145+0000 7f60fd657700 1 -- 172.21.15.142:0/2439674047 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f60f80696a0 con 0x7f60f8090e20 2024-09-18T10:18:03.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60fd657700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 msgr2=0x7f60f808d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60fd657700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 0x7f60f808d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec016950 con 0x7f60f8090e20 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 10 monclient: dump: 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.154 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f60ec016ab0 con 0x7f60f8090e20 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:03.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 msgr2=0x7f60f808d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 0x7f60f808d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec01d000 comp rx=0 tx=0).stop 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 msgr2=0x7f60f808d0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60e9ffb700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f8090e20 0x7f60f808d0c0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f60f808e360 tx=0x7f60ec01d000 comp rx=0 tx=0).stop 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:03.156 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:03.157 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:03.158 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:03.159 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:03.160 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:03.161 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:03.163 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:03.164 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:03.166 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.167 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:18:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:03.169 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f60f80a5f90 msgr2=0x7f60dc0400e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f60f80a5f90 0x7f60dc0400e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f60f00098e0 tx=0x7f60f001e070 comp rx=0 tx=0).stop 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 10 monclient: shutdown 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 msgr2=0x7f60dc005500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 --2- 172.21.15.142:0/2439674047 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60f809e360 0x7f60dc005500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60f4007da0 tx=0x7f60f4009040 comp rx=0 tx=0).stop 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 reap_dead start 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f60fe659700 1 -- 172.21.15.142:0/2439674047 reap_dead start 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 shutdown_connections 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 >> 172.21.15.142:0/2439674047 conn(0x7f60f8071680 msgr2=0x7f60f809b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 shutdown_connections 2024-09-18T10:18:03.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.149+0000 7f61000bc700 1 -- 172.21.15.142:0/2439674047 wait complete. 2024-09-18T10:18:03.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.157+0000 7f0b963bc700 10 start_mon_command cmd=[{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:03.171 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.157+0000 7f0b963bc700 10 monclient: _send_command 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:03.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.157+0000 7f0b963bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.172 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.157+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd crush weight-set rm-compat"} v 0) v1 -- 0x7f0b90097210 con 0x7f0b9006e7f0 2024-09-18T10:18:03.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:03.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:18:03.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.281+0000 7fd7d8b6b700 10 monclient: get_monmap_and_config 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.281+0000 7fd7d8b6b700 10 monclient: build_initial_monmap 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.281+0000 7fd7d8b6b700 10 monclient: monmap: 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:03.287350+0000 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:03.287350+0000 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:03.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:03.290 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:03.290 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:03.290 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:03.290 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 Processor -- start 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 -- start start 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient: init 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-c con 0x7fd7d409abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7d409b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-b con 0x7fd7d409c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d409dc90 con 0x7fd7d409b740 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d409b500 con 0x7fd7d409abd0 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d409afa0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d8b6b700 10 monclient(hunting): authenticate will time out at 1372.213133s 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42534/0 (socket says 172.21.15.142:42534) 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2897232356 learned_addr learned my addr 172.21.15.142:0/2897232356 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): get_auth_request con 0x7fd7d409c5e0 auth_method 0 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request con 0x7fd7d409abd0 auth_method 0 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 msgr2=0x7fd7d40960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 msgr2=0x7fd7d40960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_done global_id 74735 payload 274 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 msgr2=0x7fd7d40960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 --2- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2897232356 >> v1:172.21.15.150:6789/0 conn(0x7fd7d409b740 legacy=0x7fd7d409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 monclient: found mon.noname-b 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.285+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2897232356 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d406f810 con 0x7fd7d409c5e0 2024-09-18T10:18:03.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2897232356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 msgr2=0x7fd7d4137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d3fff700 1 --2- 172.21.15.142:0/2897232356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009f20 tx=0x7fd7cc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d14eda9fdaf1aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 1 -- 172.21.15.142:0/2897232356 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc00e070 con 0x7fd7d409c5e0 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: dump: 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.300 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 5 monclient: authenticate success, global_id 74735 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 1 -- 172.21.15.142:0/2897232356 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc00bc90 con 0x7fd7d409c5e0 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 1 -- 172.21.15.142:0/2897232356 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0092e0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 4 set_mon_vals no callback set 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d2ffd700 10 monclient: dump: 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 10 monclient: shutdown 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2897232356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 msgr2=0x7fd7d4137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2897232356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009f20 tx=0x7fd7cc009840 comp rx=0 tx=0).stop 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2897232356 shutdown_connections 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2897232356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d4137690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2897232356 >> v2:172.21.15.150:3300/0 conn(0x7fd7d409abd0 0x7fd7d40960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2897232356 >> 172.21.15.142:0/2897232356 conn(0x7fd7d4070ef0 msgr2=0x7fd7d40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2897232356 shutdown_connections 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2897232356 wait complete. 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 10 monclient: build_initial_monmap 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 10 monclient: monmap: 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:03.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 Processor -- start 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 1 -- start start 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.289+0000 7fd7d8b6b700 10 monclient: init 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 0x7fd7d4141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-b con 0x7fd7d409abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 0x7fd7d4141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-a con 0x7fd7d409b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:03.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d41421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): picked mon.noname-c con 0x7fd7d409c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d40901a0 con 0x7fd7d409b740 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d409d240 con 0x7fd7d409abd0 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d406eaf0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient(hunting): authenticate will time out at 1372.220550s 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d41421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 0x7fd7d4141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d41421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42544/0 (socket says 172.21.15.142:42544) 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 learned_addr learned my addr 172.21.15.142:0/2686031109 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:03.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): get_auth_request con 0x7fd7d409c5e0 auth_method 0 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 10 monclient(hunting): get_auth_request con 0x7fd7d409b740 auth_method 0 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 0x7fd7d4141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 msgr2=0x7fd7d4141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 msgr2=0x7fd7d4141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 0x7fd7d4141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request con 0x7fd7d409abd0 auth_method 0 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7caffd700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 0x7fd7d4141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 msgr2=0x7fd7d4141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 msgr2=0x7fd7d4141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 0x7fd7d4141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient(hunting): handle_auth_done global_id 74738 payload 824 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 0x7fd7d4141760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 msgr2=0x7fd7d4141760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:03.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d409abd0 0x7fd7d4141760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 msgr2=0x7fd7d4141ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7d409b740 0x7fd7d4141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 monclient: found mon.noname-c 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc009ae0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 msgr2=0x7fd7d41421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d3fff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d41421e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b6c0 tx=0x7fd7cc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc012d10 con 0x7fd7d409c5e0 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 10 monclient: dump: 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 10 monclient: _finish_auth 0 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 5 monclient: authenticate success, global_id 74738 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc012e70 con 0x7fd7d409c5e0 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient: _renew_subs 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d40923b0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:03.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.293+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc01e640 con 0x7fd7d409c5e0 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 10 monclient: dump: 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d27fc700 4 set_mon_vals no callback set 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 10 monclient: _renew_subs 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d408f5d0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4091770 con 0x7fd7d4089ff0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd7cc01ea70 con 0x7fd7d409c5e0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d4089ff0 auth_method 0 2024-09-18T10:18:03.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd7b4035ac0 0x7fd7b4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(257..257 src has 176..257) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd7cc04aaf0 con 0x7fd7d409c5e0 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7caffd700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd7b4035ac0 0x7fd7b4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.297+0000 7fd7caffd700 10 monclient: get_auth_request con 0x7fd7b4035ac0 auth_method 0 2024-09-18T10:18:03.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.301+0000 7fd7caffd700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd7b4035ac0 0x7fd7b4037f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007cb0 tx=0x7fd7bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.413+0000 7f0b8cff9700 1 -- 172.21.15.150:0/795404998 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd crush weight-set rm-compat"}]=0 v258) v1 ==== 78+0+0 (secure 0 0 0) 0x7f0b80024a20 con 0x7f0b9006e7f0 2024-09-18T10:18:03.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.413+0000 7f0b8cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd crush weight-set rm-compat"}] 2024-09-18T10:18:03.419 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.413+0000 7f0b8cff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:03.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 msgr2=0x7f0b70037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 0x7f0b70037f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0b78005220 tx=0x7f0b78007f90 comp rx=0 tx=0).stop 2024-09-18T10:18:03.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 10 monclient: shutdown 2024-09-18T10:18:03.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:03.423 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 msgr2=0x7f0b9012f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 0x7f0b9012f210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0b90069910 tx=0x7f0b80016000 comp rx=0 tx=0).stop 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 shutdown_connections 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b70035ac0 0x7f0b70037f70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b9009f2a0 0x7f0b9013f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b90071a40 0x7f0b9013f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 --2- 172.21.15.150:0/795404998 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b9006e7f0 0x7f0b9012f210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 >> 172.21.15.150:0/795404998 conn(0x7f0b900a41f0 msgr2=0x7f0b900987f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 shutdown_connections 2024-09-18T10:18:03.424 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.417+0000 7f0b963bc700 1 -- 172.21.15.150:0/795404998 wait complete. 2024-09-18T10:18:03.437 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo OK 2024-09-18T10:18:03.437 INFO:tasks.workunit.client.0.smithi150.stdout:OK 2024-09-18T10:18:03.438 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:18:03.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:18:03.497 INFO:tasks.workunit:Running workunits matching mon/osd.sh on client.0... 2024-09-18T10:18:03.498 INFO:tasks.workunit:Running workunit mon/osd.sh... 2024-09-18T10:18:03.499 DEBUG:teuthology.orchestra.run.smithi150:workunit test mon/osd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/osd.sh 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d4089ff0 auth_method 0 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.501+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:18:03.508 INFO:tasks.workunit.client.0.smithi150.stderr:+ set -e 2024-09-18T10:18:03.508 INFO:tasks.workunit.client.0.smithi150.stderr:+ uuidgen 2024-09-18T10:18:03.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ ua=912b7d9d-08f4-4308-9252-03c9211c6cd6 2024-09-18T10:18:03.510 INFO:tasks.workunit.client.0.smithi150.stderr:+ uuidgen 2024-09-18T10:18:03.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ ub=d10f425b-f28f-4608-ad77-c468ac23312b 2024-09-18T10:18:03.511 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd create 912b7d9d-08f4-4308-9252-03c9211c6cd6 2024-09-18T10:18:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.605+0000 7fa207b63700 10 monclient: get_monmap_and_config 2024-09-18T10:18:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.605+0000 7fa207b63700 10 monclient: build_initial_monmap 2024-09-18T10:18:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.605+0000 7fa207b63700 10 monclient: monmap: 2024-09-18T10:18:03.613 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:03.611924+0000 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:03.611924+0000 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:03.614 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 Processor -- start 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 -- start start 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient: init 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa20009cf10 0x7fa2000a0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-a con 0x7fa20009cf10 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:03.617 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2000a0750 0x7fa2000a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-b con 0x7fa2000a0750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-c con 0x7fa200136990 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20009e2e0 con 0x7fa20009cf10 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20009dc20 con 0x7fa200136990 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20009e6e0 con 0x7fa2000a0750 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa207b63700 10 monclient(hunting): authenticate will time out at 1365.705371s 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.618 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49196/0 (socket says 172.21.15.150:49196) 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 -- 172.21.15.150:0/536498646 learned_addr learned my addr 172.21.15.150:0/536498646 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 10 monclient(hunting): get_auth_request con 0x7fa200136990 auth_method 0 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 --2- 172.21.15.150:0/536498646 >> v2:172.21.15.142:3300/0 conn(0x7fa20009cf10 0x7fa2000a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): get_auth_request con 0x7fa20009cf10 auth_method 0 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2000a0750 0x7fa2000a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): get_auth_request con 0x7fa2000a0750 auth_method 0 2024-09-18T10:18:03.619 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 -- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 msgr2=0x7fa200138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 -- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 msgr2=0x7fa200138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 --2- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa206100700 1 --2- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.620 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient(hunting): handle_auth_done global_id 94338 payload 274 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 -- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2000a0750 msgr2=0x7fa2000a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2000a0750 0x7fa2000a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 -- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 msgr2=0x7fa200138d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 --2- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 monclient: found mon.noname-a 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:03.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.609+0000 7fa2058ff700 1 -- 172.21.15.150:0/536498646 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000a1060 con 0x7fa20009cf10 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2058ff700 1 -- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa20009cf10 msgr2=0x7fa2000a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2058ff700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa20009cf10 0x7fa2000a0210 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009e60 tx=0x7fa1f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13076ffdce75ae98 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 1 -- 172.21.15.150:0/536498646 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000e070 con 0x7fa20009cf10 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: dump: 2024-09-18T10:18:03.628 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: _finish_auth 0 2024-09-18T10:18:03.629 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 5 monclient: authenticate success, global_id 94338 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 1 -- 172.21.15.150:0/536498646 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f000bd30 con 0x7fa20009cf10 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 4 set_mon_vals no callback set 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 1 -- 172.21.15.150:0/536498646 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000bea0 con 0x7fa20009cf10 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa2048fd700 10 monclient: dump: 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.630 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 10 monclient: shutdown 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 -- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa20009cf10 msgr2=0x7fa2000a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa20009cf10 0x7fa2000a0210 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa1f0009e60 tx=0x7fa1f0009780 comp rx=0 tx=0).stop 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 -- 172.21.15.150:0/536498646 shutdown_connections 2024-09-18T10:18:03.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 --2- 172.21.15.150:0/536498646 >> v2:172.21.15.150:3300/0 conn(0x7fa200136990 0x7fa200138d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa2000a0750 0x7fa2000a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 --2- 172.21.15.150:0/536498646 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa20009cf10 0x7fa2000a0210 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 -- 172.21.15.150:0/536498646 >> 172.21.15.150:0/536498646 conn(0x7fa2000a5820 msgr2=0x7fa2000a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 -- 172.21.15.150:0/536498646 shutdown_connections 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 1 -- 172.21.15.150:0/536498646 wait complete. 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 10 monclient: build_initial_monmap 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.613+0000 7fa207b63700 10 monclient: monmap: 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:03.632 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 Processor -- start 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 -- start start 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient: init 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 0x7fa20012c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-c con 0x7fa20009cf10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 0x7fa20012c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-a con 0x7fa2000a0750 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): picked mon.noname-b con 0x7fa200136990 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20012fa90 con 0x7fa2000a0750 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20012dcf0 con 0x7fa200136990 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20012e410 con 0x7fa20009cf10 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa207b63700 10 monclient(hunting): authenticate will time out at 1365.713502s 2024-09-18T10:18:03.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49210/0 (socket says 172.21.15.150:49210) 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 -- 172.21.15.150:0/1052272283 learned_addr learned my addr 172.21.15.150:0/1052272283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 10 monclient(hunting): get_auth_request con 0x7fa200136990 auth_method 0 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 0x7fa20012c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 0x7fa20012c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 msgr2=0x7fa20012cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): get_auth_request con 0x7fa20009cf10 auth_method 0 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 msgr2=0x7fa20012cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): get_auth_request con 0x7fa2000a0750 auth_method 0 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa206100700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:03.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient(hunting): handle_auth_done global_id 74744 payload 824 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 msgr2=0x7fa20012cec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 msgr2=0x7fa20012c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 0x7fa20012c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 monclient: found mon.noname-c 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2058ff700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0009a20 con 0x7fa20009cf10 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.617+0000 7fa2050fe700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 0x7fa20012c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:03.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa2058ff700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 msgr2=0x7fa20012c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa2058ff700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 0x7fa20012c440 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa20006aac0 tx=0x7fa1f0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0012e30 con 0x7fa20009cf10 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: dump: 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.638 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0027470 con 0x7fa20009cf10 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0027830 con 0x7fa20009cf10 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 10 monclient: dump: 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:03.639 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 5 monclient: authenticate success, global_id 74744 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _renew_subs 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f77fe700 4 set_mon_vals no callback set 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.640 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20009bb90 con 0x7fa20009cf10 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _renew_subs 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa200130940 con 0x7fa20009cf10 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20012fdc0 con 0x7fa20009cf10 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.621+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa1f0024070 con 0x7fa20009cf10 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.625+0000 7fa1f5ffb700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 0x7fa1ec037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.625+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(258..258 src has 176..258) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa1f004b440 con 0x7fa20009cf10 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.625+0000 7fa2050fe700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 0x7fa1ec037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.625+0000 7fa2050fe700 10 monclient: get_auth_request con 0x7fa1ec035ac0 auth_method 0 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.625+0000 7fa2050fe700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 0x7fa1ec037f70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc011480 tx=0x7fa1fc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.637+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa1f001c5d0 con 0x7fa20009cf10 2024-09-18T10:18:03.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.637+0000 7fa1f5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:03.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.637+0000 7fa1f5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:03.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.817+0000 7fa207b63700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:03.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.817+0000 7fa207b63700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:03.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.817+0000 7fa207b63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:03.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:03.817+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fa20006e350 con 0x7fa20009cf10 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d4089ff0 auth_method 0 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:03.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:03.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:03.905+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:18:04.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.421+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v259) v1 ==== 106+0+3 (secure 0 0 0) 0x7fa1f002c020 con 0x7fa20009cf10 2024-09-18T10:18:04.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.421+0000 7fa1f5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.429 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.421+0000 7fa1f5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa207b63700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa207b63700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa207b63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:04.546 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fa200063ed0 con 0x7fa20009cf10 2024-09-18T10:18:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa1f5ffb700 1 -- 172.21.15.150:0/1052272283 <== mon.2 v2:172.21.15.142:3301/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v259) v1 ==== 106+0+3 (secure 0 0 0) 0x7fa1f002c020 con 0x7fa20009cf10 2024-09-18T10:18:04.547 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa1f5ffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.548 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.541+0000 7fa1f5ffb700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:04.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 msgr2=0x7fa1ec037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 0x7fa1ec037f70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc011480 tx=0x7fa1fc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.550 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 10 monclient: shutdown 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 msgr2=0x7fa20012c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 0x7fa20012c440 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa20006aac0 tx=0x7fa1f0019000 comp rx=0 tx=0).stop 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 shutdown_connections 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa1ec035ac0 0x7fa1ec037f70 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa200136990 0x7fa20012cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa2000a0750 0x7fa20012c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 --2- 172.21.15.150:0/1052272283 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa20009cf10 0x7fa20012c440 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 >> 172.21.15.150:0/1052272283 conn(0x7fa2000a5820 msgr2=0x7fa2000a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 shutdown_connections 2024-09-18T10:18:04.551 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.545+0000 7fa207b63700 1 -- 172.21.15.150:0/1052272283 wait complete. 2024-09-18T10:18:04.561 INFO:tasks.workunit.client.0.smithi150.stderr:+ na=13 2024-09-18T10:18:04.561 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd create 912b7d9d-08f4-4308-9252-03c9211c6cd6 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 10 monclient: get_monmap_and_config 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 10 monclient: build_initial_monmap 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 10 monclient: monmap: 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:04.658573+0000 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:04.658573+0000 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:04.660 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:04.661 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.664 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 1 Processor -- start 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 1 -- start start 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.653+0000 7fe305d52700 10 monclient: init 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe30009f330 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000a1220 0x7fe30009f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-b con 0x7fe3000a1220 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe30013b9e0 0x7fe30013ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-a con 0x7fe30013b9e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30009dc20 con 0x7fe30009f330 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30009d6f0 con 0x7fe30013b9e0 2024-09-18T10:18:04.665 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000a15f0 con 0x7fe3000a1220 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe305d52700 10 monclient(hunting): authenticate will time out at 1366.752340s 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe30013b9e0 0x7fe30013ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000a1220 0x7fe30009f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe30013b9e0 0x7fe30013ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:54198/0 (socket says 172.21.15.150:54198) 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): get_auth_request con 0x7fe3000a1220 auth_method 0 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 -- 172.21.15.150:0/3959892357 learned_addr learned my addr 172.21.15.150:0/3959892357 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2ff7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe30009f330 0x7fe30009ebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:37332/0 (socket says 172.21.15.150:37332) 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): get_auth_request con 0x7fe30013b9e0 auth_method 0 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient(hunting): handle_auth_done global_id 94344 payload 274 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 -- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000a1220 msgr2=0x7fe30009f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:04.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 --2- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000a1220 0x7fe30009f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 -- 172.21.15.150:0/3959892357 >> v1:172.21.15.142:6789/0 conn(0x7fe30009f330 legacy=0x7fe30009ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 monclient: found mon.noname-a 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 -- 172.21.15.150:0/3959892357 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000a1d70 con 0x7fe30013b9e0 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 -- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30013b9e0 msgr2=0x7fe30013ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.657+0000 7fe2fffff700 1 --2- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30013b9e0 0x7fe30013ddc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80099b0 tx=0x7fe2f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9141c5a1a52725b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 1 -- 172.21.15.150:0/3959892357 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8009ea0 con 0x7fe30013b9e0 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: dump: 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:04.673 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 5 monclient: authenticate success, global_id 94344 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:04.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 1 -- 172.21.15.150:0/3959892357 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f800ba70 con 0x7fe30013b9e0 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 4 set_mon_vals no callback set 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 1 -- 172.21.15.150:0/3959892357 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800be10 con 0x7fe30013b9e0 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fe7fc700 10 monclient: dump: 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:04.675 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: shutdown 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30013b9e0 msgr2=0x7fe30013ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30013b9e0 0x7fe30013ddc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80099b0 tx=0x7fe2f8009250 comp rx=0 tx=0).stop 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- 172.21.15.150:0/3959892357 shutdown_connections 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000a1220 0x7fe30009f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- 172.21.15.150:0/3959892357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30013b9e0 0x7fe30013ddc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- 172.21.15.150:0/3959892357 >> 172.21.15.150:0/3959892357 conn(0x7fe3000a5820 msgr2=0x7fe3000a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:04.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- 172.21.15.150:0/3959892357 shutdown_connections 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- 172.21.15.150:0/3959892357 wait complete. 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: build_initial_monmap 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: monmap: 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:04.677 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 Processor -- start 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- start start 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: init 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009f330 0x7fe30013a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-c con 0x7fe30009f330 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 0x7fe30013af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-a con 0x7fe3000a1220 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): picked mon.noname-b con 0x7fe30013b9e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006abe0 con 0x7fe3000a1220 2024-09-18T10:18:04.678 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006faa0 con 0x7fe30013b9e0 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000992d0 con 0x7fe30009f330 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe305d52700 10 monclient(hunting): authenticate will time out at 1366.759186s 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49226/0 (socket says 172.21.15.150:49226) 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 -- 172.21.15.150:0/2628073228 learned_addr learned my addr 172.21.15.150:0/2628073228 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 10 monclient(hunting): get_auth_request con 0x7fe30013b9e0 auth_method 0 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:04.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 0x7fe30013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): get_auth_request con 0x7fe3000a1220 auth_method 0 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 msgr2=0x7fe30013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 msgr2=0x7fe30013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009f330 0x7fe30013a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2fffff700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): get_auth_request con 0x7fe30009f330 auth_method 0 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:04.680 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2feffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.661+0000 7fe2ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 10 monclient(hunting): handle_auth_done global_id 94347 payload 824 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009f330 msgr2=0x7fe30013a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009f330 0x7fe30013a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 msgr2=0x7fe30013b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 monclient: found mon.noname-a 2024-09-18T10:18:04.681 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f80094f0 con 0x7fe3000a1220 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 msgr2=0x7fe30013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2feffd700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 0x7fe30013af30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009ed0 tx=0x7fe2e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800ce30 con 0x7fe3000a1220 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: dump: 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.682 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 5 monclient: authenticate success, global_id 94347 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e800f040 con 0x7fe3000a1220 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 10 monclient: _renew_subs 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3000a3f20 con 0x7fe3000a1220 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:04.683 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80146d0 con 0x7fe3000a1220 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2f77fe700 10 monclient: dump: 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:04.684 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe2fdffb700 4 set_mon_vals no callback set 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 10 monclient: _renew_subs 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30006bab0 con 0x7fe3000a1220 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.665+0000 7fe305d52700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe305d52700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe305d52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300063ed0 con 0x7fe3000a1220 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe2e8014a40 con 0x7fe3000a1220 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe2f77fe700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(259..259 src has 176..259) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe2e8049cf0 con 0x7fe3000a1220 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe2ff7fe700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.669+0000 7fe2ff7fe700 10 monclient: get_auth_request con 0x7fe2e4035ac0 auth_method 0 2024-09-18T10:18:04.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.673+0000 7fe2ff7fe700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009ad0 tx=0x7fe2f0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:04.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.681+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe2e8024500 con 0x7fe3000a1220 2024-09-18T10:18:04.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.681+0000 7fe2f77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:04.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.681+0000 7fe2f77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d4089ff0 auth_method 0 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:04.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:04.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:04.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:04.709+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:18:04.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.857+0000 7fe305d52700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.857+0000 7fe305d52700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.857+0000 7fe305d52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.857+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fe30006bcd0 con 0x7fe3000a1220 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.861+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v259) v1 ==== 106+0+3 (secure 0 0 0) 0x7fe2e8018020 con 0x7fe3000a1220 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.861+0000 7fe2f77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.861+0000 7fe2f77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:04.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe305d52700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe305d52700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe305d52700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:04.971 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fe300098150 con 0x7fe3000a1220 2024-09-18T10:18:04.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe2f77fe700 1 -- 172.21.15.150:0/2628073228 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v259) v1 ==== 106+0+3 (secure 0 0 0) 0x7fe2e8007e60 con 0x7fe3000a1220 2024-09-18T10:18:04.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe2f77fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:04.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.965+0000 7fe2f77fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:04.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 msgr2=0x7fe2e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009ad0 tx=0x7fe2f0009060 comp rx=0 tx=0).stop 2024-09-18T10:18:04.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 10 monclient: shutdown 2024-09-18T10:18:04.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 msgr2=0x7fe30013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 0x7fe30013af30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009ed0 tx=0x7fe2e800b040 comp rx=0 tx=0).stop 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 shutdown_connections 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30013b9e0 0x7fe30013b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe3000a1220 0x7fe30013af30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 --2- 172.21.15.150:0/2628073228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009f330 0x7fe30013a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 >> 172.21.15.150:0/2628073228 conn(0x7fe3000a5820 msgr2=0x7fe30009b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 shutdown_connections 2024-09-18T10:18:04.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:04.969+0000 7fe305d52700 1 -- 172.21.15.150:0/2628073228 wait complete. 2024-09-18T10:18:04.984 INFO:tasks.workunit.client.0.smithi150.stderr:+ test 13 -eq 13 2024-09-18T10:18:04.984 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd create d10f425b-f28f-4608-ad77-c468ac23312b 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: get_monmap_and_config 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: build_initial_monmap 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: monmap: 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:05.080460+0000 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:05.080460+0000 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:05.082 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:05.083 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:05.083 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:05.083 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 Processor -- start 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 -- start start 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: init 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00f40a28b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-c con 0x7f00f40a07b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 0x7f00f4139c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f00f40a0bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f40a1940 con 0x7f00f40a28b0 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f40a3070 con 0x7f00f40a07b0 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f40a2c80 con 0x7f00f40a0bb0 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f359e700 10 monclient(hunting): authenticate will time out at 1367.173974s 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49240/0 (socket says 172.21.15.150:49240) 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 -- 172.21.15.150:0/2218304768 learned_addr learned my addr 172.21.15.150:0/2218304768 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:05.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 10 monclient(hunting): get_auth_request con 0x7f00f40a07b0 auth_method 0 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 --2- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 0x7f00f4139c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 -- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 msgr2=0x7f00f41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 -- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 msgr2=0x7f00f41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): get_auth_request con 0x7f00f40a0bb0 auth_method 0 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.087 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_done global_id 74753 payload 274 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 -- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 msgr2=0x7f00f41376e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 --2- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 -- 172.21.15.150:0/2218304768 >> v1:172.21.15.142:6789/0 conn(0x7f00f40a28b0 legacy=0x7f00f40a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 monclient: found mon.noname-b 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:05.088 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.077+0000 7f00f2d9d700 1 -- 172.21.15.150:0/2218304768 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f409d3b0 con 0x7f00f40a0bb0 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f2d9d700 1 -- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 msgr2=0x7f00f4139c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f2d9d700 1 --2- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 0x7f00f4139c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009e60 tx=0x7f00ec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44079da98fe29be0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 1 -- 172.21.15.150:0/2218304768 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec00e070 con 0x7f00f40a0bb0 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: dump: 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.090 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: _finish_auth 0 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 1 -- 172.21.15.150:0/2218304768 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec00be40 con 0x7f00f40a0bb0 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 5 monclient: authenticate success, global_id 74753 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 1 -- 172.21.15.150:0/2218304768 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec012660 con 0x7f00f40a0bb0 2024-09-18T10:18:05.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 4 set_mon_vals no callback set 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f159a700 10 monclient: dump: 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.092 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 10 monclient: shutdown 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 -- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 msgr2=0x7f00f4139c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 --2- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 0x7f00f4139c30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009e60 tx=0x7f00ec009780 comp rx=0 tx=0).stop 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 -- 172.21.15.150:0/2218304768 shutdown_connections 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 --2- 172.21.15.150:0/2218304768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a0bb0 0x7f00f4139c30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 --2- 172.21.15.150:0/2218304768 >> v2:172.21.15.150:3300/0 conn(0x7f00f40a07b0 0x7f00f41376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 -- 172.21.15.150:0/2218304768 >> 172.21.15.150:0/2218304768 conn(0x7f00f40a6100 msgr2=0x7f00f40a21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 -- 172.21.15.150:0/2218304768 shutdown_connections 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.081+0000 7f00f359e700 1 -- 172.21.15.150:0/2218304768 wait complete. 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient: build_initial_monmap 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient: monmap: 2024-09-18T10:18:05.093 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:05.094 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 Processor -- start 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 -- start start 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient: init 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:05.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 0x7f00f4123260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-a con 0x7f00f40a07b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f00f40a0bb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 0x7f00f411e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): picked mon.noname-c con 0x7f00f40a28b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f41237a0 con 0x7f00f40a07b0 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f41222b0 con 0x7f00f40a0bb0 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f4120510 con 0x7f00f40a28b0 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f359e700 10 monclient(hunting): authenticate will time out at 1367.182676s 2024-09-18T10:18:05.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49246/0 (socket says 172.21.15.150:49246) 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 -- 172.21.15.150:0/945339044 learned_addr learned my addr 172.21.15.150:0/945339044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 10 monclient(hunting): get_auth_request con 0x7f00f40a0bb0 auth_method 0 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 msgr2=0x7f00f41320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 msgr2=0x7f00f41320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 0x7f00f4123260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 0x7f00f411e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): get_auth_request con 0x7f00f40a07b0 auth_method 0 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): get_auth_request con 0x7f00f40a28b0 auth_method 0 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient(hunting): handle_auth_done global_id 94350 payload 824 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 msgr2=0x7f00f411e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 0x7f00f411e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 msgr2=0x7f00f41320c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 monclient: found mon.noname-a 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f259c700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec009a20 con 0x7f00f40a07b0 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.085+0000 7f00f2d9d700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 0x7f00f411e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f259c700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 msgr2=0x7f00f4123260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:05.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f259c700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 0x7f00f4123260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c4e0 tx=0x7f00e8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80117a0 con 0x7f00f40a07b0 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: dump: 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.103 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 5 monclient: authenticate success, global_id 94350 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 10 monclient: _renew_subs 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e8011920 con 0x7f00f40a07b0 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f4121fd0 con 0x7f00f40a07b0 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8011ce0 con 0x7f00f40a07b0 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:05.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f0d99700 4 set_mon_vals no callback set 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00e27fc700 10 monclient: dump: 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 10 monclient: _renew_subs 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.089+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f4132600 con 0x7f00f40a07b0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f4063ed0 con 0x7f00f40a07b0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f00e8012630 con 0x7f00f40a07b0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00e27fc700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 0x7f00d4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(259..259 src has 176..259) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f00e8048c20 con 0x7f00f40a07b0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 0x7f00d4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f1d9b700 10 monclient: get_auth_request con 0x7f00d4035ac0 auth_method 0 2024-09-18T10:18:05.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.093+0000 7f00f1d9b700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 0x7f00d4037f70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009240 tx=0x7f00e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.105+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f00e8002090 con 0x7f00f40a07b0 2024-09-18T10:18:05.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.105+0000 7f00e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.105+0000 7f00e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:05.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.269+0000 7f00f359e700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.269+0000 7f00f359e700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.269+0000 7f00f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.269+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"} v 0) v1 -- 0x7f00f409c620 con 0x7f00f40a07b0 2024-09-18T10:18:05.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.617+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}]=0 v260) v1 ==== 106+0+3 (secure 0 0 0) 0x7f00e80102e0 con 0x7f00f40a07b0 2024-09-18T10:18:05.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.617+0000 7f00e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.622 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.617+0000 7f00e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:05.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00f359e700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.727 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00f359e700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00f359e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:05.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"} v 0) v1 -- 0x7f00f404bbf0 con 0x7f00f40a07b0 2024-09-18T10:18:05.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00e27fc700 1 -- 172.21.15.150:0/945339044 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}]=0 v260) v1 ==== 106+0+3 (secure 0 0 0) 0x7f00e80104c0 con 0x7f00f40a07b0 2024-09-18T10:18:05.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00e27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:05.728 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.721+0000 7f00e27fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:05.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 msgr2=0x7f00d4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 0x7f00d4037f70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009240 tx=0x7f00e4008040 comp rx=0 tx=0).stop 2024-09-18T10:18:05.733 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 10 monclient: shutdown 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 msgr2=0x7f00f4123260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 0x7f00f4123260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c4e0 tx=0x7f00e8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 shutdown_connections 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00d4035ac0 0x7f00d4037f70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f00f40a28b0 0x7f00f411e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00f40a0bb0 0x7f00f41320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 --2- 172.21.15.150:0/945339044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f00f40a07b0 0x7f00f4123260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.734 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 >> 172.21.15.150:0/945339044 conn(0x7f00f40a6100 msgr2=0x7f00f40a4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:05.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 shutdown_connections 2024-09-18T10:18:05.735 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.725+0000 7f00f359e700 1 -- 172.21.15.150:0/945339044 wait complete. 2024-09-18T10:18:05.740 INFO:tasks.workunit.client.0.smithi150.stderr:+ nb=15 2024-09-18T10:18:05.740 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd create d10f425b-f28f-4608-ad77-c468ac23312b 2024-09-18T10:18:05.838 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: get_monmap_and_config 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: build_initial_monmap 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: monmap: 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:05.837545+0000 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:05.837545+0000 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:05.839 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:05.840 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:05.840 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:05.840 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 Processor -- start 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 -- start start 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: init 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2c409d000 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 0x7fd2c40a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c409fae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:05.842 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2c4137a20 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409e060 con 0x7fd2c409d000 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c409d3d0 con 0x7fd2c4137a20 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c409ea90 con 0x7fd2c409fae0 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c9dca700 10 monclient(hunting): authenticate will time out at 1367.931021s 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c3fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd2c4137a20 0x7fd2c4139e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44408/0 (socket says 172.21.15.150:44408) 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c3fff700 1 -- 172.21.15.150:0/3958573102 learned_addr learned my addr 172.21.15.150:0/3958573102 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 1 --2- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 0x7fd2c40a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): get_auth_request con 0x7fd2c409fae0 auth_method 0 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.843 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_done global_id 74759 payload 274 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/3958573102 >> v1:172.21.15.150:6789/0 conn(0x7fd2c4137a20 legacy=0x7fd2c4139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/3958573102 >> v1:172.21.15.142:6789/0 conn(0x7fd2c409d000 legacy=0x7fd2c40a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 1 monclient: found mon.noname-b 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:05.844 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.833+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/3958573102 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40a1610 con 0x7fd2c409fae0 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 msgr2=0x7fd2c40a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c2ffd700 1 --2- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 0x7fd2c40a08a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009940 tx=0x7fd2b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238fe0e067aa4886 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 1 -- 172.21.15.150:0/3958573102 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b400ba30 con 0x7fd2c409fae0 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: dump: 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.847 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 1 -- 172.21.15.150:0/3958573102 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b400bb90 con 0x7fd2c409fae0 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 1 -- 172.21.15.150:0/3958573102 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011420 con 0x7fd2c409fae0 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 5 monclient: authenticate success, global_id 74759 2024-09-18T10:18:05.848 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 4 set_mon_vals no callback set 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c27fc700 10 monclient: dump: 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.849 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 10 monclient: shutdown 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 1 -- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 msgr2=0x7fd2c40a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 0x7fd2c40a08a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009940 tx=0x7fd2b40091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 1 -- 172.21.15.150:0/3958573102 shutdown_connections 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/3958573102 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409fae0 0x7fd2c40a08a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.837+0000 7fd2c9dca700 1 -- 172.21.15.150:0/3958573102 >> 172.21.15.150:0/3958573102 conn(0x7fd2c40a58a0 msgr2=0x7fd2c40a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- 172.21.15.150:0/3958573102 shutdown_connections 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- 172.21.15.150:0/3958573102 wait complete. 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient: build_initial_monmap 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient: monmap: 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:05.850 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 Processor -- start 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- start start 2024-09-18T10:18:05.851 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient: init 2024-09-18T10:18:05.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:05.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:05.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409d000 0x7fd2c4141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-c con 0x7fd2c409d000 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-b con 0x7fd2c409fae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c4137a20 0x7fd2c4142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): picked mon.noname-a con 0x7fd2c4137a20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c409bf90 con 0x7fd2c4137a20 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c412fce0 con 0x7fd2c409fae0 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c4130e50 con 0x7fd2c409d000 2024-09-18T10:18:05.856 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c9dca700 10 monclient(hunting): authenticate will time out at 1367.938997s 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49254/0 (socket says 172.21.15.150:49254) 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/315424217 learned_addr learned my addr 172.21.15.150:0/315424217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): get_auth_request con 0x7fd2c409fae0 auth_method 0 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409d000 0x7fd2c4141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c4137a20 0x7fd2c4142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): get_auth_request con 0x7fd2c4137a20 auth_method 0 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): get_auth_request con 0x7fd2c409d000 auth_method 0 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.858 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient(hunting): handle_auth_done global_id 104098 payload 824 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409d000 msgr2=0x7fd2c4141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409d000 0x7fd2c4141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c4137a20 msgr2=0x7fd2c4142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c4137a20 0x7fd2c4142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 monclient: found mon.noname-b 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.841+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b4009420 con 0x7fd2c409fae0 2024-09-18T10:18:05.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c2ffd700 1 -- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 msgr2=0x7fd2c41423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c2ffd700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40030c0 tx=0x7fd2b4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011e50 con 0x7fd2c409fae0 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: dump: 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.860 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 5 monclient: authenticate success, global_id 104098 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4020070 con 0x7fd2c409fae0 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 10 monclient: _renew_subs 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c412fa00 con 0x7fd2c409fae0 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b401c6a0 con 0x7fd2c409fae0 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:05.861 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2ab7fe700 10 monclient: dump: 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c1ffb700 4 set_mon_vals no callback set 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 10 monclient: _renew_subs 2024-09-18T10:18:05.862 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.845+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c412eaa0 con 0x7fd2c409fae0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c9dca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c9dca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c9dca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd2b4022030 con 0x7fd2c409fae0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c412eaa0 con 0x7fd2c409fae0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2ab7fe700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 0x7fd2b0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(260..260 src has 176..260) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd2b404adf0 con 0x7fd2c409fae0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c37fe700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 0x7fd2b0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:05.863 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.849+0000 7fd2c37fe700 10 monclient: get_auth_request con 0x7fd2b0035ac0 auth_method 0 2024-09-18T10:18:05.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.853+0000 7fd2c37fe700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 0x7fd2b0037f70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009a60 tx=0x7fd2b8009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:05.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.857+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd2b404e030 con 0x7fd2c409fae0 2024-09-18T10:18:05.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.857+0000 7fd2ab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:05.864 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:05.857+0000 7fd2ab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:06.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.017+0000 7fd2c9dca700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.017+0000 7fd2c9dca700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.017+0000 7fd2c9dca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.017+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"} v 0) v1 -- 0x7fd2c412eaa0 con 0x7fd2c409fae0 2024-09-18T10:18:06.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.021+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}]=0 v260) v1 ==== 106+0+3 (secure 0 0 0) 0x7fd2b4010980 con 0x7fd2c409fae0 2024-09-18T10:18:06.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.021+0000 7fd2ab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.021+0000 7fd2ab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:06.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.137+0000 7fd2c9dca700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.137+0000 7fd2c9dca700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.137+0000 7fd2c9dca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.137+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"} v 0) v1 -- 0x7fd2c412eaa0 con 0x7fd2c409fae0 2024-09-18T10:18:06.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2ab7fe700 1 -- 172.21.15.150:0/315424217 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}]=0 v260) v1 ==== 106+0+3 (secure 0 0 0) 0x7fd2b401a3d0 con 0x7fd2c409fae0 2024-09-18T10:18:06.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2ab7fe700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "d10f425b-f28f-4608-ad77-c468ac23312b"}] 2024-09-18T10:18:06.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2ab7fe700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:06.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 msgr2=0x7fd2b0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 0x7fd2b0037f70 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009a60 tx=0x7fd2b8009060 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 10 monclient: shutdown 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 msgr2=0x7fd2c41423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.141+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40030c0 tx=0x7fd2b4016000 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 shutdown_connections 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd2b0035ac0 0x7fd2b0037f70 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2c4137a20 0x7fd2c4142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2c409fae0 0x7fd2c41423d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 --2- 172.21.15.150:0/315424217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2c409d000 0x7fd2c4141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 >> 172.21.15.150:0/315424217 conn(0x7fd2c40a58a0 msgr2=0x7fd2c40a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 shutdown_connections 2024-09-18T10:18:06.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.145+0000 7fd2c9dca700 1 -- 172.21.15.150:0/315424217 wait complete. 2024-09-18T10:18:06.159 INFO:tasks.workunit.client.0.smithi150.stderr:+ test 15 -eq 15 2024-09-18T10:18:06.159 INFO:tasks.workunit.client.0.smithi150.stderr:+ test 15 -ne 13 2024-09-18T10:18:06.160 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm 13 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: get_monmap_and_config 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: monmap: 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:06.252724+0000 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:06.252724+0000 2024-09-18T10:18:06.254 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:06.255 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 Processor -- start 2024-09-18T10:18:06.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 -- start start 2024-09-18T10:18:06.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: init 2024-09-18T10:18:06.257 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24880a6090 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f248809e460 0x7f248809d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-c con 0x7f248809e460 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24880a0a80 0x7f2488139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-b con 0x7f24880a0a80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880a6460 con 0x7f24880a6090 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248809fd00 con 0x7f248809e460 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24880a6850 con 0x7f24880a0a80 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.258 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f248d4cd700 10 monclient(hunting): authenticate will time out at 1368.346387s 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f248809e460 0x7f248809d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f248809e460 0x7f248809d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49258/0 (socket says 172.21.15.150:49258) 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 -- 172.21.15.150:0/1920271279 learned_addr learned my addr 172.21.15.150:0/1920271279 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): get_auth_request con 0x7f248809e460 auth_method 0 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24880a0a80 0x7f2488139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): get_auth_request con 0x7f24880a0a80 auth_method 0 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.259 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient(hunting): handle_auth_done global_id 104101 payload 274 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 -- 172.21.15.150:0/1920271279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24880a0a80 msgr2=0x7f2488139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24880a0a80 0x7f2488139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 -- 172.21.15.150:0/1920271279 >> v1:172.21.15.142:6789/0 conn(0x7f24880a6090 legacy=0x7f24880a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 monclient: found mon.noname-c 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.260 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.249+0000 7f24867fc700 1 -- 172.21.15.150:0/1920271279 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248809d7a0 con 0x7f248809e460 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f24867fc700 1 -- 172.21.15.150:0/1920271279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f248809e460 msgr2=0x7f248809d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f24867fc700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f248809e460 0x7f248809d0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f24780099b0 tx=0x7f2478009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2612d8682630706a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 1 -- 172.21.15.150:0/1920271279 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478009ea0 con 0x7f248809e460 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: dump: 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.264 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 5 monclient: authenticate success, global_id 104101 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 1 -- 172.21.15.150:0/1920271279 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247800ba70 con 0x7f248809e460 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 4 set_mon_vals no callback set 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 1 -- 172.21.15.150:0/1920271279 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247800be30 con 0x7f248809e460 2024-09-18T10:18:06.265 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f2485ffb700 10 monclient: dump: 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 10 monclient: shutdown 2024-09-18T10:18:06.266 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 -- 172.21.15.150:0/1920271279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f248809e460 msgr2=0x7f248809d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f248809e460 0x7f248809d0f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f24780099b0 tx=0x7f2478009250 comp rx=0 tx=0).stop 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 -- 172.21.15.150:0/1920271279 shutdown_connections 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24880a0a80 0x7f2488139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 --2- 172.21.15.150:0/1920271279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f248809e460 0x7f248809d0f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 -- 172.21.15.150:0/1920271279 >> 172.21.15.150:0/1920271279 conn(0x7f2488071830 msgr2=0x7f24880a5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 -- 172.21.15.150:0/1920271279 shutdown_connections 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 1 -- 172.21.15.150:0/1920271279 wait complete. 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.253+0000 7f248d4cd700 10 monclient: monmap: 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:06.267 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 Processor -- start 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 -- start start 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient: init 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 0x7f2488091730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.268 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-c con 0x7f248809e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24880a0a80 0x7f2488097290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-a con 0x7f24880a0a80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): picked mon.noname-b con 0x7f24880a6090 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880984d0 con 0x7f24880a0a80 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24880a4b50 con 0x7f24880a6090 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248808e970 con 0x7f248809e460 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f248d4cd700 10 monclient(hunting): authenticate will time out at 1368.352961s 2024-09-18T10:18:06.269 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49270/0 (socket says 172.21.15.150:49270) 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 -- 172.21.15.150:0/2573027761 learned_addr learned my addr 172.21.15.150:0/2573027761 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): get_auth_request con 0x7f24880a6090 auth_method 0 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24867fc700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24880a0a80 0x7f2488097290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f2486ffd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 0x7f2488091730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24867fc700 10 monclient(hunting): get_auth_request con 0x7f24880a0a80 auth_method 0 2024-09-18T10:18:06.270 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24867fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient(hunting): handle_auth_done global_id 104104 payload 824 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 -- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 msgr2=0x7f2488091730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 0x7f2488091730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 -- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24880a0a80 msgr2=0x7f2488097290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24880a0a80 0x7f2488097290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 monclient: found mon.noname-b 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780094f0 con 0x7f24880a6090 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f2486ffd700 10 monclient: get_auth_request con 0x7f248809e460 auth_method 0 2024-09-18T10:18:06.271 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f2486ffd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 0x7f2488091730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:18:06.273 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 -- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 msgr2=0x7f24880977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f24877fe700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2480009ec0 tx=0x7f248000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.257+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248000ce30 con 0x7f24880a6090 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: dump: 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.274 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 5 monclient: authenticate success, global_id 104104 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248000f040 con 0x7f24880a6090 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _renew_subs 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248808ee80 con 0x7f24880a6090 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24800146d0 con 0x7f24880a6090 2024-09-18T10:18:06.275 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f24857fa700 4 set_mon_vals no callback set 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 10 monclient: dump: 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.276 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _renew_subs 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248808cec0 con 0x7f24880a6090 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2480014a80 con 0x7f24880a6090 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2488063ed0 con 0x7f24880a6090 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 0x7f2474037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(260..260 src has 176..260) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2480049a00 con 0x7f24880a6090 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f2486ffd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 0x7f2474037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.261+0000 7f2486ffd700 10 monclient: get_auth_request con 0x7f2474035ac0 auth_method 0 2024-09-18T10:18:06.277 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.265+0000 7f2486ffd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 0x7f2474037f70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f247c0027e0 tx=0x7f247c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.269+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f24800112c0 con 0x7f24880a6090 2024-09-18T10:18:06.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.269+0000 7f246effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.278 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.269+0000 7f246effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:06.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.293+0000 7fd7ca7fc700 10 monclient: tick 2024-09-18T10:18:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.293+0000 7fd7ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.293+0000 7fd7ca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d4089ff0 auth_method 0 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_done global_id 104107 payload 824 2024-09-18T10:18:06.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7d409aa30 con 0x7fd7d4089ff0 2024-09-18T10:18:06.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.313+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4017990 tx=0x7fd7c4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4007ab0 con 0x7fd7d4089ff0 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 10 monclient: dump: 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd7c401c070 con 0x7fd7d4089ff0 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:06.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4017990 tx=0x7fd7c4008040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 msgr2=0x7fd7d408a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:06.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.317+0000 7fd7cbfff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d4089ff0 0x7fd7d408a3c0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4017990 tx=0x7fd7c4008040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d8b6b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:06.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 0x7fd7d408dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d8b6b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d40648a0 con 0x7fd7d408a900 2024-09-18T10:18:06.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 0x7fd7d408dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient: get_auth_request con 0x7fd7d408a900 auth_method 0 2024-09-18T10:18:06.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 10 monclient(hunting): handle_auth_done global_id 104110 payload 824 2024-09-18T10:18:06.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd7d4072a80 con 0x7fd7d408a900 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 msgr2=0x7fd7d408dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.333+0000 7fd7d37fe700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 0x7fd7d408dbc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4018010 tx=0x7fd7c4022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4022640 con 0x7fd7d408a900 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 10 monclient: dump: 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fd7c402e470 con 0x7fd7d408a900 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 msgr2=0x7fd7d408dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 0x7fd7d408dbc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4018010 tx=0x7fd7c4022000 comp rx=0 tx=0).stop 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 msgr2=0x7fd7d408dbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7cbfff700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7d408a900 0x7fd7d408dbc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.343 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:06.344 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:06.345 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:06.346 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:06.347 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:06.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:06.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:06.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:06.352 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:06.353 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:06.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:06.355 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd7b4035ac0 msgr2=0x7fd7b4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd7b4035ac0 0x7fd7b4037f70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007cb0 tx=0x7fd7bc009040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 10 monclient: shutdown 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 msgr2=0x7fd7d41421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 --2- 172.21.15.142:0/2686031109 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd7d409c5e0 0x7fd7d41421e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00b6c0 tx=0x7fd7cc017000 comp rx=0 tx=0).stop 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 reap_dead start 2024-09-18T10:18:06.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d3fff700 1 -- 172.21.15.142:0/2686031109 reap_dead start 2024-09-18T10:18:06.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 shutdown_connections 2024-09-18T10:18:06.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 >> 172.21.15.142:0/2686031109 conn(0x7fd7d4070ef0 msgr2=0x7fd7d409de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 shutdown_connections 2024-09-18T10:18:06.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.337+0000 7fd7d8b6b700 1 -- 172.21.15.142:0/2686031109 wait complete. 2024-09-18T10:18:06.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:06.358 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:18:06.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:18:06.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.445+0000 7f248d4cd700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.445+0000 7f248d4cd700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.445+0000 7f248d4cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.451 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.445+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["13"]} v 0) v1 -- 0x7f248809da80 con 0x7f24880a6090 2024-09-18T10:18:06.452 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.445+0000 7f2439e88700 -1 mon.a@0(leader).osd e260 definitely_dead 0 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.469+0000 7fb0d4890700 10 monclient: get_monmap_and_config 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.469+0000 7fb0d4890700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.469+0000 7fb0d4890700 10 monclient: monmap: 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:06.475611+0000 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:06.475611+0000 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 Processor -- start 2024-09-18T10:18:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 -- start start 2024-09-18T10:18:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient: init 2024-09-18T10:18:06.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0d0098900 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 0x7fb0d0133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-a con 0x7fb0d012edf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d0135f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-b con 0x7fb0d012f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d0098450 con 0x7fb0d0098900 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d009e310 con 0x7fb0d012edf0 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d009e570 con 0x7fb0d012f230 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0d4890700 10 monclient(hunting): authenticate will time out at 1375.402833s 2024-09-18T10:18:06.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 0x7fb0d0133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 0x7fb0d0133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51962/0 (socket says 172.21.15.142:51962) 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 1 -- 172.21.15.142:0/2144396294 learned_addr learned my addr 172.21.15.142:0/2144396294 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): get_auth_request con 0x7fb0d012edf0 auth_method 0 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 --2- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d0135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): get_auth_request con 0x7fb0d012f230 auth_method 0 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_done global_id 74768 payload 274 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 -- 172.21.15.142:0/2144396294 >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 msgr2=0x7fb0d0133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 --2- 172.21.15.142:0/2144396294 >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 0x7fb0d0133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 -- 172.21.15.142:0/2144396294 >> v1:172.21.15.150:6789/0 conn(0x7fb0d0098900 legacy=0x7fb0d012e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 monclient: found mon.noname-b 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.473+0000 7fb0cf59e700 1 -- 172.21.15.142:0/2144396294 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0098100 con 0x7fb0d012f230 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0cf59e700 1 -- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 msgr2=0x7fb0d0135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0cf59e700 1 --2- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d0135f60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009e60 tx=0x7fb0c8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=386f75a8596df965 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 1 -- 172.21.15.142:0/2144396294 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800e070 con 0x7fb0d012f230 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: dump: 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 5 monclient: authenticate success, global_id 74768 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 1 -- 172.21.15.142:0/2144396294 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c800bd70 con 0x7fb0d012f230 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 4 set_mon_vals no callback set 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 1 -- 172.21.15.142:0/2144396294 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8012680 con 0x7fb0d012f230 2024-09-18T10:18:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0ce59c700 10 monclient: dump: 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 10 monclient: shutdown 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 -- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 msgr2=0x7fb0d0135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d0135f60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009e60 tx=0x7fb0c8009780 comp rx=0 tx=0).stop 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 -- 172.21.15.142:0/2144396294 shutdown_connections 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2144396294 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d0135f60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2144396294 >> v2:172.21.15.142:3300/0 conn(0x7fb0d012edf0 0x7fb0d0133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 -- 172.21.15.142:0/2144396294 >> 172.21.15.142:0/2144396294 conn(0x7fb0d0071970 msgr2=0x7fb0d00a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 -- 172.21.15.142:0/2144396294 shutdown_connections 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.477+0000 7fb0d4890700 1 -- 172.21.15.142:0/2144396294 wait complete. 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient: monmap: 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:06.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 Processor -- start 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 -- start start 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient: init 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:06.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0098900 0x7fb0d0119a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-b con 0x7fb0d0098900 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-a con 0x7fb0d012edf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d011a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): picked mon.noname-c con 0x7fb0d012f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d012d5a0 con 0x7fb0d012edf0 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d011bcd0 con 0x7fb0d0098900 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d006f430 con 0x7fb0d012f230 2024-09-18T10:18:06.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0d4890700 10 monclient(hunting): authenticate will time out at 1375.409850s 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d011a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51970/0 (socket says 172.21.15.142:51970) 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 -- 172.21.15.142:0/2856253452 learned_addr learned my addr 172.21.15.142:0/2856253452 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): get_auth_request con 0x7fb0d012edf0 auth_method 0 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): get_auth_request con 0x7fb0d012f230 auth_method 0 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0098900 0x7fb0d0119a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): get_auth_request con 0x7fb0d0098900 auth_method 0 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0cf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0ced9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient(hunting): handle_auth_done global_id 94362 payload 824 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 -- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 msgr2=0x7fb0d011a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d011a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 -- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0098900 msgr2=0x7fb0d0119a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0098900 0x7fb0d0119a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 monclient: found mon.noname-a 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.481+0000 7fb0c659c700 1 -- 172.21.15.142:0/2856253452 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c8009a20 con 0x7fb0d012edf0 2024-09-18T10:18:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c659c700 1 -- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 msgr2=0x7fb0d0119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c659c700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800dfa0 tx=0x7fb0b800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8012040 con 0x7fb0d012edf0 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: dump: 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 5 monclient: authenticate success, global_id 94362 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b8010d50 con 0x7fb0d012edf0 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _renew_subs 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8018660 con 0x7fb0d012edf0 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d00a1030 con 0x7fb0d012edf0 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0cdd9b700 4 set_mon_vals no callback set 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 10 monclient: dump: 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _renew_subs 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d011b540 con 0x7fb0d012edf0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb0b80189d0 con 0x7fb0d012edf0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d006da50 con 0x7fb0d012edf0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 0x7fb0b0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(260..260 src has 176..260) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb0b804b180 con 0x7fb0d012edf0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0ced9d700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 0x7fb0b0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.485+0000 7fb0ced9d700 10 monclient: get_auth_request con 0x7fb0b0035ac0 auth_method 0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.489+0000 7fb0ced9d700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 0x7fb0b0037f70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0003a30 tx=0x7fb0c0007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.493+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb0b8020ba0 con 0x7fb0d012edf0 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.493+0000 7fb0c77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.493+0000 7fb0c77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:06.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.673+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["13"]}]=0 removed osd.13 v261) v1 ==== 83+0+0 (secure 0 0 0) 0x7f2480018080 con 0x7f24880a6090 2024-09-18T10:18:06.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.673+0000 7f246effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.679 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.673+0000 7f246effd700 10 monclient: _finish_command 2 = system:0 removed osd.13 2024-09-18T10:18:06.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.717+0000 7fb0d4890700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:18:06.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.717+0000 7fb0d4890700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:18:06.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.717+0000 7fb0d4890700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:06.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.721+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb0d0072a80 con 0x7fb0d012edf0 2024-09-18T10:18:06.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.721+0000 7fb0c77fe700 1 -- 172.21.15.142:0/2856253452 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb0b8020370 con 0x7fb0d012edf0 2024-09-18T10:18:06.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.721+0000 7fb0c77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:18:06.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.721+0000 7fb0c77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:06.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=10,kv=7,logm=64} crc {auth=1880922580,config=3140514009,health=2517688005,kv=2693826500,logm=3150691219}) 2024-09-18T10:18:06.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=8,logm=63} crc {auth=1168884804,config=3140514009,health=2040191363,kv=4261215878,logm=2394654271}) 2024-09-18T10:18:06.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=10,kv=7,logm=64} crc {auth=1880922580,config=3140514009,health=2517688005,kv=2693826500,logm=3150691219}) 2024-09-18T10:18:06.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=11,kv=7,logm=64} crc {auth=3940148845,config=3140514009,health=28157231,kv=1523155648,logm=1880483493}) 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188061359}) 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238065879}) 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188061359}) 2024-09-18T10:18:06.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.725+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746116639}) 2024-09-18T10:18:06.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 msgr2=0x7fb0b0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 0x7fb0b0037f70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0003a30 tx=0x7fb0c0007ea0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 10 monclient: shutdown 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 msgr2=0x7fb0d0119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800dfa0 tx=0x7fb0b800f040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 shutdown_connections 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb0b0035ac0 0x7fb0b0037f70 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb0d012f230 0x7fb0d011a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb0d012edf0 0x7fb0d0119f80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 --2- 172.21.15.142:0/2856253452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0098900 0x7fb0d0119a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 >> 172.21.15.142:0/2856253452 conn(0x7fb0d0071970 msgr2=0x7fb0d006abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 shutdown_connections 2024-09-18T10:18:06.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:06.729+0000 7fb0d4890700 1 -- 172.21.15.142:0/2856253452 wait complete. 2024-09-18T10:18:06.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476341747}) 2024-09-18T10:18:06.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635074496}) 2024-09-18T10:18:06.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476341747}) 2024-09-18T10:18:06.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.729+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2060509839}) 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298229933}) 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2618514488}) 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298229933}) 2024-09-18T10:18:06.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108685374}) 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265532068}) 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999845513}) 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265532068}) 2024-09-18T10:18:06.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.733+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284344985}) 2024-09-18T10:18:06.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064681263}) 2024-09-18T10:18:06.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649719105}) 2024-09-18T10:18:06.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064681263}) 2024-09-18T10:18:06.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278293829}) 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150603109}) 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201934629}) 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150603109}) 2024-09-18T10:18:06.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.737+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1272799136}) 2024-09-18T10:18:06.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2978590354}) 2024-09-18T10:18:06.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625391196}) 2024-09-18T10:18:06.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2978590354}) 2024-09-18T10:18:06.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231652860}) 2024-09-18T10:18:06.747 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428111540}) 2024-09-18T10:18:06.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467637494}) 2024-09-18T10:18:06.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428111540}) 2024-09-18T10:18:06.748 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1809633839}) 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166192250}) 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710293162}) 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3166192250}) 2024-09-18T10:18:06.749 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.741+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=100428367}) 2024-09-18T10:18:06.753 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.753 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160020115}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339822471}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160020115}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045418283}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600316097}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915571856}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600316097}) 2024-09-18T10:18:06.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124620078}) 2024-09-18T10:18:06.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323410265}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4184832688}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.745+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323410265}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133423734}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198323656}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225925249}) 2024-09-18T10:18:06.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198323656}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090927911}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493889820}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1720206825}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493889820}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588130786}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750018980}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238562990}) 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750018980}) 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.749+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853183360}) 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999110651}) 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446003040}) 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999110651}) 2024-09-18T10:18:06.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.753+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368007897}) 2024-09-18T10:18:06.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f248d4cd700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f248d4cd700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f248d4cd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.795 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["13"]} v 0) v1 -- 0x7f24880982b0 con 0x7f24880a6090 2024-09-18T10:18:06.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.789+0000 7f2439e88700 -1 mon.a@0(leader).osd e261 definitely_dead 0 2024-09-18T10:18:06.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f246effd700 1 -- 172.21.15.150:0/2573027761 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["13"]}]=0 osd.13 does not exist. v261) v1 ==== 92+0+0 (secure 0 0 0) 0x7f248001a8d0 con 0x7f24880a6090 2024-09-18T10:18:06.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f246effd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:06.797 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.789+0000 7f246effd700 10 monclient: _finish_command 3 = system:0 osd.13 does not exist. 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 msgr2=0x7f2474037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 0x7f2474037f70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f247c0027e0 tx=0x7f247c009040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 10 monclient: shutdown 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 msgr2=0x7f24880977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2480009ec0 tx=0x7f248000b040 comp rx=0 tx=0).stop 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 shutdown_connections 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2474035ac0 0x7f2474037f70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24880a6090 0x7f24880977d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24880a0a80 0x7f2488097290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.800 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 --2- 172.21.15.150:0/2573027761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f248809e460 0x7f2488091730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 >> 172.21.15.150:0/2573027761 conn(0x7f2488071830 msgr2=0x7f24880a3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 shutdown_connections 2024-09-18T10:18:06.801 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.793+0000 7f248d4cd700 1 -- 172.21.15.150:0/2573027761 wait complete. 2024-09-18T10:18:06.801 INFO:tasks.workunit.client.0.smithi150.stderr:osd.13 does not exist. 2024-09-18T10:18:06.809 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm 13 2024-09-18T10:18:06.907 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: get_monmap_and_config 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: monmap: 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:06.906362+0000 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:06.906362+0000 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:06.908 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:06.909 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:06.909 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:06.909 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.910 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 Processor -- start 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 -- start start 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: init 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 0x7fbdbc06e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-b con 0x7fbdbc132260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbdbc0699a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdbc132630 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc06b9f0 con 0x7fbdbc132630 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc06ada0 con 0x7fbdbc0699a0 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc06b080 con 0x7fbdbc132260 2024-09-18T10:18:06.911 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.901+0000 7fbdc4281700 10 monclient(hunting): authenticate will time out at 1368.999842s 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc181c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbdbc0699a0 0x7fbdbc0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44414/0 (socket says 172.21.15.150:44414) 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc181c700 1 -- 172.21.15.150:0/1302794563 learned_addr learned my addr 172.21.15.150:0/1302794563 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 --2- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 0x7fbdbc06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): get_auth_request con 0x7fbdbc132260 auth_method 0 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient(hunting): handle_auth_done global_id 74774 payload 274 2024-09-18T10:18:06.912 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 -- 172.21.15.150:0/1302794563 >> v1:172.21.15.150:6789/0 conn(0x7fbdbc0699a0 legacy=0x7fbdbc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 -- 172.21.15.150:0/1302794563 >> v1:172.21.15.142:6789/0 conn(0x7fbdbc132630 legacy=0x7fbdbc0a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 monclient: found mon.noname-b 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:06.913 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 -- 172.21.15.150:0/1302794563 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc1330f0 con 0x7fbdbc132260 2024-09-18T10:18:06.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 -- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 msgr2=0x7fbdbc06e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.914 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.905+0000 7fbdc201d700 1 --2- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 0x7fbdbc06e680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc06f610 tx=0x7fbdac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42241745968f343 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 1 -- 172.21.15.150:0/1302794563 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac00ba70 con 0x7fbdbc132260 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: dump: 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.915 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 5 monclient: authenticate success, global_id 74774 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 1 -- 172.21.15.150:0/1302794563 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac00bbd0 con 0x7fbdbc132260 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 4 set_mon_vals no callback set 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 1 -- 172.21.15.150:0/1302794563 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac011420 con 0x7fbdbc132260 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc101b700 10 monclient: dump: 2024-09-18T10:18:06.916 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: shutdown 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 msgr2=0x7fbdbc06e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.917 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 0x7fbdbc06e680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc06f610 tx=0x7fbdac0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- 172.21.15.150:0/1302794563 shutdown_connections 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1302794563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132260 0x7fbdbc06e680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- 172.21.15.150:0/1302794563 >> 172.21.15.150:0/1302794563 conn(0x7fbdbc071cc0 msgr2=0x7fbdbc0a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- 172.21.15.150:0/1302794563 shutdown_connections 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- 172.21.15.150:0/1302794563 wait complete. 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: build_initial_monmap 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: monmap: 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.918 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:06.919 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:06.919 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 Processor -- start 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- start start 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: init 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbdbc0699a0 0x7fbdbc141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-a con 0x7fbdbc0699a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-b con 0x7fbdbc132260 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132630 0x7fbdbc11b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.923 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): picked mon.noname-c con 0x7fbdbc132630 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc06f1c0 con 0x7fbdbc0699a0 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc0a0710 con 0x7fbdbc132260 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1427d0 con 0x7fbdbc132630 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc4281700 10 monclient(hunting): authenticate will time out at 1369.006918s 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49282/0 (socket says 172.21.15.150:49282) 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 -- 172.21.15.150:0/1529732097 learned_addr learned my addr 172.21.15.150:0/1529732097 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:06.924 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): get_auth_request con 0x7fbdbc132260 auth_method 0 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbdbc0699a0 0x7fbdbc141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132630 0x7fbdbc11b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): get_auth_request con 0x7fbdbc0699a0 auth_method 0 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.925 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): get_auth_request con 0x7fbdbc132630 auth_method 0 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc201d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc281e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient(hunting): handle_auth_done global_id 104116 payload 824 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 -- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132630 msgr2=0x7fbdbc11b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132630 0x7fbdbc11b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.926 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 -- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbdbc0699a0 msgr2=0x7fbdbc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbdbc0699a0 0x7fbdbc141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 monclient: found mon.noname-b 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.909+0000 7fbdc181c700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdac009420 con 0x7fbdbc132260 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc181c700 1 -- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 msgr2=0x7fbdbc1314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc181c700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800b5e0 tx=0x7fbdb8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8011840 con 0x7fbdbc132260 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: dump: 2024-09-18T10:18:06.927 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:06.928 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb80119a0 con 0x7fbdbc132260 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 5 monclient: authenticate success, global_id 104116 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 10 monclient: _renew_subs 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc11e600 con 0x7fbdbc132260 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8011d60 con 0x7fbdbc132260 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb27fc700 10 monclient: dump: 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.929 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdb3fff700 4 set_mon_vals no callback set 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 10 monclient: _renew_subs 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.913+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc11d710 con 0x7fbdbc132260 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc4281700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc4281700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc4281700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:06.930 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc063ed0 con 0x7fbdbc132260 2024-09-18T10:18:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbdb8010590 con 0x7fbdbc132260 2024-09-18T10:18:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdb27fc700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 0x7fbda0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(261..261 src has 176..261) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fbdb8048ef0 con 0x7fbdbc132260 2024-09-18T10:18:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc201d700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 0x7fbda0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:06.931 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.917+0000 7fbdc201d700 10 monclient: get_auth_request con 0x7fbda0035ac0 auth_method 0 2024-09-18T10:18:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.921+0000 7fbdc201d700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 0x7fbda0037f70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0093f0 tx=0x7fbdac016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.925+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fbdb800fac0 con 0x7fbdbc132260 2024-09-18T10:18:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.925+0000 7fbdb27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:06.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:06.925+0000 7fbdb27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:06.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056137066}) 2024-09-18T10:18:06.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3798798407}) 2024-09-18T10:18:06.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056137066}) 2024-09-18T10:18:06.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.913+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223339313}) 2024-09-18T10:18:06.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358887455}) 2024-09-18T10:18:06.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586620072}) 2024-09-18T10:18:06.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:06.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358887455}) 2024-09-18T10:18:06.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:06.985+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007651743}) 2024-09-18T10:18:07.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.097+0000 7fbdc4281700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.097+0000 7fbdc4281700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.097+0000 7fbdc4281700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.097+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["13"]} v 0) v1 -- 0x7fbdbc11ea80 con 0x7fbdbc132260 2024-09-18T10:18:07.103 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.104 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308638716}) 2024-09-18T10:18:07.104 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145618829}) 2024-09-18T10:18:07.104 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.104 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308638716}) 2024-09-18T10:18:07.104 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.097+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973602136}) 2024-09-18T10:18:07.107 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.101+0000 7f2439e88700 -1 mon.a@0(leader).osd e261 definitely_dead 0 2024-09-18T10:18:07.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.101+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["13"]}]=0 osd.13 does not exist. v261) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbdb8015030 con 0x7fbdbc132260 2024-09-18T10:18:07.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.101+0000 7fbdb27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.101+0000 7fbdb27fc700 10 monclient: _finish_command 2 = system:0 osd.13 does not exist. 2024-09-18T10:18:07.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.205+0000 7fbdc4281700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.205+0000 7fbdc4281700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.205+0000 7fbdc4281700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.213 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.205+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["13"]} v 0) v1 -- 0x7fbdbc0008d0 con 0x7fbdbc132260 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258862413}) 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252021779}) 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258862413}) 2024-09-18T10:18:07.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.209+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118038204}) 2024-09-18T10:18:07.217 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.213+0000 7f2439e88700 -1 mon.a@0(leader).osd e261 definitely_dead 0 2024-09-18T10:18:07.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdb27fc700 1 -- 172.21.15.150:0/1529732097 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["13"]}]=0 osd.13 does not exist. v261) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbdb8015030 con 0x7fbdbc132260 2024-09-18T10:18:07.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdb27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["13"]}] 2024-09-18T10:18:07.218 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdb27fc700 10 monclient: _finish_command 3 = system:0 osd.13 does not exist. 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 msgr2=0x7fbda0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 0x7fbda0037f70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0093f0 tx=0x7fbdac016000 comp rx=0 tx=0).stop 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 10 monclient: shutdown 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 msgr2=0x7fbdbc1314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800b5e0 tx=0x7fbdb8009120 comp rx=0 tx=0).stop 2024-09-18T10:18:07.221 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 shutdown_connections 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbda0035ac0 0x7fbda0037f70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbdbc132630 0x7fbdbc11b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdbc132260 0x7fbdbc1314e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 --2- 172.21.15.150:0/1529732097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbdbc0699a0 0x7fbdbc141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 >> 172.21.15.150:0/1529732097 conn(0x7fbdbc071cc0 msgr2=0x7fbdbc06d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.213+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 shutdown_connections 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.217+0000 7fbdc4281700 1 -- 172.21.15.150:0/1529732097 wait complete. 2024-09-18T10:18:07.222 INFO:tasks.workunit.client.0.smithi150.stderr:osd.13 does not exist. 2024-09-18T10:18:07.231 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm 15 2024-09-18T10:18:07.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439952811}) 2024-09-18T10:18:07.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1674390686}) 2024-09-18T10:18:07.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439952811}) 2024-09-18T10:18:07.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.305+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3391547447}) 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: get_monmap_and_config 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: monmap: 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:07.329703+0000 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:07.329703+0000 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.331 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:07.332 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 Processor -- start 2024-09-18T10:18:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 -- start start 2024-09-18T10:18:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: init 2024-09-18T10:18:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.334 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe56809d1f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 0x7fe5680a0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-b con 0x7fe56809e840 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5681379a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56809ddd0 con 0x7fe5681379a0 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56809e400 con 0x7fe56809d1f0 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680a66e0 con 0x7fe56809e840 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56fbb1700 10 monclient(hunting): authenticate will time out at 1369.423351s 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d94d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe56809d1f0 0x7fe5680a0110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44422/0 (socket says 172.21.15.150:44422) 2024-09-18T10:18:07.335 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d94d700 1 -- 172.21.15.150:0/3606912118 learned_addr learned my addr 172.21.15.150:0/3606912118 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d14c700 1 --2- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 0x7fe5680a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d14c700 10 monclient(hunting): get_auth_request con 0x7fe56809e840 auth_method 0 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d14c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d14c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.325+0000 7fe56d14c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient(hunting): handle_auth_done global_id 74780 payload 274 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 -- 172.21.15.150:0/3606912118 >> v1:172.21.15.150:6789/0 conn(0x7fe56809d1f0 legacy=0x7fe5680a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 -- 172.21.15.150:0/3606912118 >> v1:172.21.15.142:6789/0 conn(0x7fe5681379a0 legacy=0x7fe568139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.336 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 monclient: found mon.noname-b 2024-09-18T10:18:07.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:07.337 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 -- 172.21.15.150:0/3606912118 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56809ecc0 con 0x7fe56809e840 2024-09-18T10:18:07.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 -- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 msgr2=0x7fe5680a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56d14c700 1 --2- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 0x7fe5680a0820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe558009940 tx=0x7fe5580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795c9419f48de23f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 1 -- 172.21.15.150:0/3606912118 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800ba30 con 0x7fe56809e840 2024-09-18T10:18:07.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.338 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: dump: 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.339 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56fbb1700 5 monclient: authenticate success, global_id 74780 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56fbb1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 1 -- 172.21.15.150:0/3606912118 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55800bb90 con 0x7fe56809e840 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56fbb1700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56fbb1700 4 set_mon_vals no callback set 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 1 -- 172.21.15.150:0/3606912118 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558011420 con 0x7fe56809e840 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.329+0000 7fe56c94b700 10 monclient: dump: 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.340 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: shutdown 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 msgr2=0x7fe5680a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 0x7fe5680a0820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe558009940 tx=0x7fe5580091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- 172.21.15.150:0/3606912118 shutdown_connections 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/3606912118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809e840 0x7fe5680a0820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- 172.21.15.150:0/3606912118 >> 172.21.15.150:0/3606912118 conn(0x7fe5680683a0 msgr2=0x7fe5680a5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- 172.21.15.150:0/3606912118 shutdown_connections 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- 172.21.15.150:0/3606912118 wait complete. 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: monmap: 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.342 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.343 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:07.343 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:07.343 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:07.343 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 Processor -- start 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- start start 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: init 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809d1f0 0x7fe56812c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-c con 0x7fe56809d1f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-b con 0x7fe56809e840 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 0x7fe56812cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): picked mon.noname-a con 0x7fe5681379a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56809be00 con 0x7fe5681379a0 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56812fd60 con 0x7fe56809e840 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56812fa80 con 0x7fe56809d1f0 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56fbb1700 10 monclient(hunting): authenticate will time out at 1369.430463s 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49284/0 (socket says 172.21.15.150:49284) 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 -- 172.21.15.150:0/60426806 learned_addr learned my addr 172.21.15.150:0/60426806 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): get_auth_request con 0x7fe56809e840 auth_method 0 2024-09-18T10:18:07.349 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 0x7fe56812cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809d1f0 0x7fe56812c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): get_auth_request con 0x7fe5681379a0 auth_method 0 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): get_auth_request con 0x7fe56809d1f0 auth_method 0 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.350 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d94d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient(hunting): handle_auth_done global_id 104119 payload 824 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 -- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809d1f0 msgr2=0x7fe56812c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809d1f0 0x7fe56812c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 -- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 msgr2=0x7fe56812cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 0x7fe56812cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.351 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 monclient: found mon.noname-b 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56d14c700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558009420 con 0x7fe56809e840 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.333+0000 7fe56e14e700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 0x7fe56812cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56d14c700 1 -- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 msgr2=0x7fe56812ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56d14c700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe558009fd0 tx=0x7fe558016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe558011e30 con 0x7fe56809e840 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: dump: 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.352 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 5 monclient: authenticate success, global_id 104119 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe558020070 con 0x7fe56809e840 2024-09-18T10:18:07.353 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55801b6a0 con 0x7fe56809e840 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55ffff700 4 set_mon_vals no callback set 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe55dffb700 10 monclient: dump: 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.354 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _renew_subs 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56812e840 con 0x7fe56809e840 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _renew_subs 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56812f400 con 0x7fe56809e840 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.337+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56812fef0 con 0x7fe56809e840 2024-09-18T10:18:07.355 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe558022030 con 0x7fe56809e840 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe55dffb700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 0x7fe54c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(261..261 src has 176..261) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe55804aac0 con 0x7fe56809e840 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe56d94d700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 0x7fe54c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe56d94d700 10 monclient: get_auth_request con 0x7fe54c035b10 auth_method 0 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.341+0000 7fe56d94d700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 0x7fe54c037fc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe564006df0 tx=0x7fe5640053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.349+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe558010380 con 0x7fe56809e840 2024-09-18T10:18:07.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.349+0000 7fe55dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.349+0000 7fe55dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100344418}) 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414272361}) 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100344418}) 2024-09-18T10:18:07.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.377+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169321570}) 2024-09-18T10:18:07.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.521+0000 7fe56fbb1700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.521+0000 7fe56fbb1700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.521+0000 7fe56fbb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.528 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.521+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["15"]} v 0) v1 -- 0x7fe56809ecc0 con 0x7fe56809e840 2024-09-18T10:18:07.529 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.525+0000 7f2439e88700 -1 mon.a@0(leader).osd e261 definitely_dead 0 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683156674}) 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825473745}) 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683156674}) 2024-09-18T10:18:07.645 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.637+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565315847}) 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013566850}) 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725791212}) 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013566850}) 2024-09-18T10:18:07.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.669+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948080069}) 2024-09-18T10:18:07.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969213940}) 2024-09-18T10:18:07.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250266768}) 2024-09-18T10:18:07.686 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.686 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969213940}) 2024-09-18T10:18:07.686 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.677+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669549373}) 2024-09-18T10:18:07.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.693+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.693+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157596075}) 2024-09-18T10:18:07.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.697+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522767972}) 2024-09-18T10:18:07.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.697+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.697+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157596075}) 2024-09-18T10:18:07.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.697+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973588697}) 2024-09-18T10:18:07.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.697+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["15"]}]=0 removed osd.15 v262) v1 ==== 83+0+0 (secure 0 0 0) 0x7fe56809ecc0 con 0x7fe56809e840 2024-09-18T10:18:07.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.697+0000 7fe55dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.703 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.697+0000 7fe55dffb700 10 monclient: _finish_command 2 = system:0 removed osd.15 2024-09-18T10:18:07.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:18:07.756 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:07.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.809+0000 7fe56fbb1700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.809+0000 7fe56fbb1700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.815 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.809+0000 7fe56fbb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.809+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["15"]} v 0) v1 -- 0x7fe56809ecc0 con 0x7fe56809e840 2024-09-18T10:18:07.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598132821}) 2024-09-18T10:18:07.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788447965}) 2024-09-18T10:18:07.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598132821}) 2024-09-18T10:18:07.818 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.809+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=743930566}) 2024-09-18T10:18:07.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe55dffb700 1 -- 172.21.15.150:0/60426806 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["15"]}]=0 osd.15 does not exist. v262) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe56809ecc0 con 0x7fe56809e840 2024-09-18T10:18:07.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe55dffb700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["15"]}] 2024-09-18T10:18:07.819 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe55dffb700 10 monclient: _finish_command 3 = system:0 osd.15 does not exist. 2024-09-18T10:18:07.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.813+0000 7f2439e88700 -1 mon.a@0(leader).osd e262 definitely_dead 0 2024-09-18T10:18:07.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 msgr2=0x7fe54c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.821 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 0x7fe54c037fc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe564006df0 tx=0x7fe5640053c0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 10 monclient: shutdown 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 msgr2=0x7fe56812ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe558009fd0 tx=0x7fe558016000 comp rx=0 tx=0).stop 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 shutdown_connections 2024-09-18T10:18:07.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe54c035b10 0x7fe54c037fc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe5681379a0 0x7fe56812cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe56809e840 0x7fe56812ca00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 --2- 172.21.15.150:0/60426806 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe56809d1f0 0x7fe56812c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 >> 172.21.15.150:0/60426806 conn(0x7fe5680683a0 msgr2=0x7fe5680a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:07.823 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 shutdown_connections 2024-09-18T10:18:07.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.813+0000 7fe56fbb1700 1 -- 172.21.15.150:0/60426806 wait complete. 2024-09-18T10:18:07.824 INFO:tasks.workunit.client.0.smithi150.stderr:osd.15 does not exist. 2024-09-18T10:18:07.831 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd rm 1000 2024-09-18T10:18:07.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.861+0000 7f911e24a700 10 monclient: get_monmap_and_config 2024-09-18T10:18:07.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.861+0000 7f911e24a700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient: monmap: 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:07.867811+0000 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:07.867811+0000 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:07.870 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 Processor -- start 2024-09-18T10:18:07.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 -- start start 2024-09-18T10:18:07.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient: init 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264866970}) 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759999159}) 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264866970}) 2024-09-18T10:18:07.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308870356}) 2024-09-18T10:18:07.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T10:18:07.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f911809c690 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:07.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f911809ca60 0x7f911809ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-a con 0x7f911809ca60 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 0x7f9118135d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-c con 0x7f9118133930 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9118097d70 con 0x7f911809c690 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9118097370 con 0x7f911809ca60 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911d248700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f911809c690 0x7f9118096c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:42812/0 (socket says 172.21.15.142:42812) 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f911809ca60 0x7f911809ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911d248700 1 -- 172.21.15.142:0/1863797963 learned_addr learned my addr 172.21.15.142:0/1863797963 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9118097a90 con 0x7f9118133930 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911e24a700 10 monclient(hunting): authenticate will time out at 1376.795676s 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): get_auth_request con 0x7f911809ca60 auth_method 0 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911da49700 1 --2- 172.21.15.142:0/1863797963 >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 0x7f9118135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911da49700 10 monclient(hunting): get_auth_request con 0x7f9118133930 auth_method 0 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911da49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911da49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.865+0000 7f911da49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 10 monclient(hunting): handle_auth_done global_id 94371 payload 274 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 -- 172.21.15.142:0/1863797963 >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 msgr2=0x7f9118135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 --2- 172.21.15.142:0/1863797963 >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 0x7f9118135d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 -- 172.21.15.142:0/1863797963 >> v1:172.21.15.150:6789/0 conn(0x7f911809c690 legacy=0x7f9118096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 monclient: found mon.noname-a 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 -- 172.21.15.142:0/1863797963 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911809d3a0 con 0x7f911809ca60 2024-09-18T10:18:07.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911da49700 1 --2- 172.21.15.142:0/1863797963 >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 0x7f9118135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:18:07.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 -- 172.21.15.142:0/1863797963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809ca60 msgr2=0x7f911809ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911ca47700 1 --2- 172.21.15.142:0/1863797963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809ca60 0x7f911809ce60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f911809e610 tx=0x7f91080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830865b4ac7b15a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 1 -- 172.21.15.142:0/1863797963 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910800b8c0 con 0x7f911809ca60 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: dump: 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.885 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 1 -- 172.21.15.142:0/1863797963 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910800ba20 con 0x7f911809ca60 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911e24a700 5 monclient: authenticate success, global_id 94371 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911e24a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911e24a700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 1 -- 172.21.15.142:0/1863797963 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910800bdc0 con 0x7f911809ca60 2024-09-18T10:18:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f911e24a700 4 set_mon_vals no callback set 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.869+0000 7f910ffff700 10 monclient: dump: 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.887 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: shutdown 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- 172.21.15.142:0/1863797963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809ca60 msgr2=0x7f911809ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- 172.21.15.142:0/1863797963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809ca60 0x7f911809ce60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f911809e610 tx=0x7f91080091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- 172.21.15.142:0/1863797963 shutdown_connections 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- 172.21.15.142:0/1863797963 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809ca60 0x7f911809ce60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- 172.21.15.142:0/1863797963 >> v2:172.21.15.150:3300/0 conn(0x7f9118133930 0x7f9118135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- 172.21.15.142:0/1863797963 >> 172.21.15.142:0/1863797963 conn(0x7f9118071700 msgr2=0x7f91180a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- 172.21.15.142:0/1863797963 shutdown_connections 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- 172.21.15.142:0/1863797963 wait complete. 2024-09-18T10:18:07.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: monmap: 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 Processor -- start 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- start start 2024-09-18T10:18:07.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: init 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-a con 0x7f911809c690 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 0x7f911812fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-c con 0x7f911809ca60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9118133930 0x7f9118118300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): picked mon.noname-b con 0x7f9118133930 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911811b200 con 0x7f911809c690 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9118097570 con 0x7f9118133930 2024-09-18T10:18:07.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911809b2e0 con 0x7f911809ca60 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911e24a700 10 monclient(hunting): authenticate will time out at 1376.802994s 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 0x7f911812fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:51976/0 (socket says 172.21.15.142:51976) 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 -- 172.21.15.142:0/3780832028 learned_addr learned my addr 172.21.15.142:0/3780832028 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): get_auth_request con 0x7f911809c690 auth_method 0 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): get_auth_request con 0x7f911809ca60 auth_method 0 2024-09-18T10:18:07.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9118133930 0x7f9118118300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): get_auth_request con 0x7f9118133930 auth_method 0 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911da49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient(hunting): handle_auth_done global_id 94374 payload 824 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 -- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 msgr2=0x7f911812fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 0x7f911812fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 -- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9118133930 msgr2=0x7f9118118300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9118133930 0x7f9118118300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 monclient: found mon.noname-a 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911d248700 1 -- 172.21.15.142:0/3780832028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9108009420 con 0x7f911809c690 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.873+0000 7f911ca47700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 0x7f911812fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911d248700 1 -- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 msgr2=0x7f911806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911d248700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f911400ea30 tx=0x7f911400c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400c440 con 0x7f911809c690 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: dump: 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.894 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9114005d40 con 0x7f911809c690 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 5 monclient: authenticate success, global_id 94374 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _renew_subs 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9118119560 con 0x7f911809c690 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114007340 con 0x7f911809c690 2024-09-18T10:18:07.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 10 monclient: dump: 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:07.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910f7fe700 4 set_mon_vals no callback set 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _renew_subs 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911813d6d0 con 0x7f911809c690 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9114004f20 con 0x7f911809c690 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911811acd0 con 0x7f911809c690 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 0x7f90fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.877+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f911404b280 con 0x7f911809c690 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.881+0000 7f911ca47700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 0x7f90fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.881+0000 7f911ca47700 10 monclient: get_auth_request con 0x7f90fc035ac0 auth_method 0 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.881+0000 7f911ca47700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 0x7f90fc037f70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f91080030c0 tx=0x7f9108016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.885+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f91140051d0 con 0x7f911809c690 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.885+0000 7f910d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:07.885+0000 7f910d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.921+0000 7f2752b2f700 10 monclient: get_monmap_and_config 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient: monmap: 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:07.928061+0000 2024-09-18T10:18:07.929 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:07.928061+0000 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:07.930 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 Processor -- start 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 -- start start 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient: init 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f274c09c2e0 0x7f274c0956d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-a con 0x7f274c09c2e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f274c098a50 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:07.932 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f274c099560 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0984f0 con 0x7f274c099560 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c09af60 con 0x7f274c098a50 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c09b520 con 0x7f274c09c2e0 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.933 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f2752b2f700 10 monclient(hunting): authenticate will time out at 1370.021546s 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f274bfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f274c098a50 0x7f274c098e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44438/0 (socket says 172.21.15.150:44438) 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f274bfff700 1 -- 172.21.15.150:0/978758496 learned_addr learned my addr 172.21.15.150:0/978758496 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 1 --2- 172.21.15.150:0/978758496 >> v2:172.21.15.142:3300/0 conn(0x7f274c09c2e0 0x7f274c0956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): get_auth_request con 0x7f274c09c2e0 auth_method 0 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient(hunting): handle_auth_done global_id 94377 payload 274 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 1 -- 172.21.15.150:0/978758496 >> v1:172.21.15.150:6789/0 conn(0x7f274c098a50 legacy=0x7f274c098e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.934 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 1 -- 172.21.15.150:0/978758496 >> v1:172.21.15.142:6789/0 conn(0x7f274c099560 legacy=0x7f274c136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:07.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 1 monclient: found mon.noname-a 2024-09-18T10:18:07.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:07.935 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.925+0000 7f27508cb700 1 -- 172.21.15.150:0/978758496 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c09c6b0 con 0x7f274c09c2e0 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f27508cb700 1 -- 172.21.15.150:0/978758496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c09c2e0 msgr2=0x7f274c0956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f27508cb700 1 --2- 172.21.15.150:0/978758496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c09c2e0 0x7f274c0956d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27400098e0 tx=0x7f27400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428229e0d50f4736 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 1 -- 172.21.15.150:0/978758496 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274000ba70 con 0x7f274c09c2e0 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: dump: 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.937 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 1 -- 172.21.15.150:0/978758496 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274000bbd0 con 0x7f274c09c2e0 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 5 monclient: authenticate success, global_id 94377 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 4 set_mon_vals no callback set 2024-09-18T10:18:07.938 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 1 -- 172.21.15.150:0/978758496 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2740011420 con 0x7f274c09c2e0 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f274b7fe700 10 monclient: dump: 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.939 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 10 monclient: shutdown 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- 172.21.15.150:0/978758496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c09c2e0 msgr2=0x7f274c0956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 --2- 172.21.15.150:0/978758496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c09c2e0 0x7f274c0956d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f27400098e0 tx=0x7f27400091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- 172.21.15.150:0/978758496 shutdown_connections 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 --2- 172.21.15.150:0/978758496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c09c2e0 0x7f274c0956d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- 172.21.15.150:0/978758496 >> 172.21.15.150:0/978758496 conn(0x7f274c0a1710 msgr2=0x7f274c0a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- 172.21.15.150:0/978758496 shutdown_connections 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- 172.21.15.150:0/978758496 wait complete. 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 10 monclient: build_initial_monmap 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 10 monclient: monmap: 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:07.940 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 Processor -- start 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 1 -- start start 2024-09-18T10:18:07.941 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.929+0000 7f2752b2f700 10 monclient: init 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f274c098a50 0x7f274c141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-c con 0x7f274c098a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c099560 0x7f274c141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.946 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-a con 0x7f274c099560 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): picked mon.noname-b con 0x7f274c09c2e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c08ff90 con 0x7f274c099560 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0977c0 con 0x7f274c09c2e0 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0951f0 con 0x7f274c098a50 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f2752b2f700 10 monclient(hunting): authenticate will time out at 1370.028516s 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c099560 0x7f274c141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.947 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f274c098a50 0x7f274c141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49298/0 (socket says 172.21.15.150:49298) 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 -- 172.21.15.150:0/927646126 learned_addr learned my addr 172.21.15.150:0/927646126 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): get_auth_request con 0x7f274c09c2e0 auth_method 0 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): get_auth_request con 0x7f274c098a50 auth_method 0 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): get_auth_request con 0x7f274c099560 auth_method 0 2024-09-18T10:18:07.948 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27508cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f274bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient(hunting): handle_auth_done global_id 104128 payload 824 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 -- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f274c098a50 msgr2=0x7f274c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.949 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f274c098a50 0x7f274c141a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 -- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c099560 msgr2=0x7f274c141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c099560 0x7f274c141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 monclient: found mon.noname-b 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2740009420 con 0x7f274c09c2e0 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 -- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 msgr2=0x7f274c08df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.933+0000 7f27510cc700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f274400c930 tx=0x7f27440026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2744012840 con 0x7f274c09c2e0 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:07.950 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: dump: 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:07.951 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27440129c0 con 0x7f274c09c2e0 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 5 monclient: authenticate success, global_id 104128 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _renew_subs 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c091540 con 0x7f274c09c2e0 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2744012d80 con 0x7f274c09c2e0 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 10 monclient: dump: 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.952 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f274a7fc700 4 set_mon_vals no callback set 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _renew_subs 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c0927e0 con 0x7f274c09c2e0 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:07.953 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c063ed0 con 0x7f274c09c2e0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2744011590 con 0x7f274c09c2e0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 0x7f2730037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f27440492a0 con 0x7f274c09c2e0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f27508cb700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 0x7f2730037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.937+0000 7f27508cb700 10 monclient: get_auth_request con 0x7f2730035ac0 auth_method 0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.941+0000 7f27508cb700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 0x7f2730037f70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f27400093f0 tx=0x7f2740016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.945+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2744022ae0 con 0x7f274c09c2e0 2024-09-18T10:18:07.954 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.945+0000 7f2748ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:07.955 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:07.945+0000 7f2748ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:08.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.109+0000 7f911e24a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:08.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.109+0000 7f911e24a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.109+0000 7f911e24a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.109+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f911813d950 con 0x7f911809c690 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.109+0000 7f910d7fa700 1 -- 172.21.15.142:0/3780832028 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f911400f360 con 0x7f911809c690 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f910d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f910d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:08.117 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 msgr2=0x7f90fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 0x7f90fc037f70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f91080030c0 tx=0x7f9108016000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 10 monclient: shutdown 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 msgr2=0x7f911806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f911400ea30 tx=0x7f911400c3b0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 shutdown_connections 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f90fc035ac0 0x7f90fc037f70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9118133930 0x7f9118118300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f911809ca60 0x7f911812fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 --2- 172.21.15.142:0/3780832028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f911809c690 0x7f911806dff0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 >> 172.21.15.142:0/3780832028 conn(0x7f9118071700 msgr2=0x7f911809fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 shutdown_connections 2024-09-18T10:18:08.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.113+0000 7f911e24a700 1 -- 172.21.15.142:0/3780832028 wait complete. 2024-09-18T10:18:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.117+0000 7f2752b2f700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.117+0000 7f2752b2f700 10 monclient: _send_command 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.117+0000 7f2752b2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.117+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f274c099cc0 con 0x7f274c09c2e0 2024-09-18T10:18:08.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.121+0000 7f2439e88700 -1 mon.a@0(leader).osd e262 definitely_dead 0 2024-09-18T10:18:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.121+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v262) v1 ==== 96+0+0 (secure 0 0 0) 0x7f2744022300 con 0x7f274c09c2e0 2024-09-18T10:18:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.121+0000 7f2748ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.126 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.121+0000 7f2748ff9700 10 monclient: _finish_command 2 = system:0 osd.1000 does not exist. 2024-09-18T10:18:08.131 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:18:08.131 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:18:08.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:18:08.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:18:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2752b2f700 10 start_mon_command cmd=[{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2752b2f700 10 monclient: _send_command 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2752b2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.227 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd rm", "ids": ["1000"]} v 0) v1 -- 0x7f274c0906a0 con 0x7f274c09c2e0 2024-09-18T10:18:08.228 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.221+0000 7f2439e88700 -1 mon.a@0(leader).osd e262 definitely_dead 0 2024-09-18T10:18:08.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2748ff9700 1 -- 172.21.15.150:0/927646126 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "osd rm", "ids": ["1000"]}]=0 osd.1000 does not exist. v262) v1 ==== 96+0+0 (secure 0 0 0) 0x7f27440224e0 con 0x7f274c09c2e0 2024-09-18T10:18:08.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2748ff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd rm", "ids": ["1000"]}] 2024-09-18T10:18:08.228 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.221+0000 7f2748ff9700 10 monclient: _finish_command 3 = system:0 osd.1000 does not exist. 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 msgr2=0x7f2730037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 0x7f2730037f70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f27400093f0 tx=0x7f2740016000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 10 monclient: shutdown 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 msgr2=0x7f274c08df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.231 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f274400c930 tx=0x7f27440026e0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 shutdown_connections 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2730035ac0 0x7f2730037f70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f274c09c2e0 0x7f274c08df00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f274c099560 0x7f274c141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 --2- 172.21.15.150:0/927646126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f274c098a50 0x7f274c141a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 >> 172.21.15.150:0/927646126 conn(0x7f274c0a1710 msgr2=0x7f274c095e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 shutdown_connections 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.225+0000 7f2752b2f700 1 -- 172.21.15.150:0/927646126 wait complete. 2024-09-18T10:18:08.232 INFO:tasks.workunit.client.0.smithi150.stderr:osd.1000 does not exist. 2024-09-18T10:18:08.240 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph osd create 912b7d9d-08f4-4308-9252-03c9211c6cd6 2024-09-18T10:18:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: get_monmap_and_config 2024-09-18T10:18:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: monmap: 2024-09-18T10:18:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:08.239906+0000 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:08.239906+0000 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:08.242 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 Processor -- start 2024-09-18T10:18:08.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 -- start start 2024-09-18T10:18:08.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: init 2024-09-18T10:18:08.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fae4c09eee0 0x7fae4c1325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-a con 0x7fae4c09eee0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-b con 0x7fae4c132b00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae4c09f7f0 0x7fae4c09ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-c con 0x7fae4c09f7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0a0660 con 0x7fae4c09eee0 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0a1c40 con 0x7fae4c09f7f0 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0a1600 con 0x7fae4c132b00 2024-09-18T10:18:08.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae52bc0700 10 monclient(hunting): authenticate will time out at 1377.165660s 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae4c09f7f0 0x7fae4c09ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fae4c09f7f0 0x7fae4c09ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50986/0 (socket says 172.21.15.142:50986) 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42564/0 (socket says 172.21.15.142:42564) 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 -- 172.21.15.142:0/401509888 learned_addr learned my addr 172.21.15.142:0/401509888 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): get_auth_request con 0x7fae4c09f7f0 auth_method 0 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): get_auth_request con 0x7fae4c132b00 auth_method 0 2024-09-18T10:18:08.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae51bbe700 1 --2- 172.21.15.142:0/401509888 >> v2:172.21.15.142:3300/0 conn(0x7fae4c09eee0 0x7fae4c1325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae51bbe700 10 monclient(hunting): get_auth_request con 0x7fae4c09eee0 auth_method 0 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae51bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae51bbe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae51bbe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient(hunting): handle_auth_done global_id 104131 payload 274 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 -- 172.21.15.142:0/401509888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 msgr2=0x7fae4c09f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 -- 172.21.15.142:0/401509888 >> v2:172.21.15.142:3300/0 conn(0x7fae4c09eee0 msgr2=0x7fae4c1325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 --2- 172.21.15.142:0/401509888 >> v2:172.21.15.142:3300/0 conn(0x7fae4c09eee0 0x7fae4c1325c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 monclient: found mon.noname-c 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae523bf700 1 -- 172.21.15.142:0/401509888 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0a1f20 con 0x7fae4c09f7f0 2024-09-18T10:18:08.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.237+0000 7fae513bd700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae523bf700 1 -- 172.21.15.142:0/401509888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 msgr2=0x7fae4c09ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae523bf700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c09ae60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0a35b0 tx=0x7fae44009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3babc718a68a18bc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 1 -- 172.21.15.142:0/401509888 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4400e070 con 0x7fae4c09f7f0 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: dump: 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 5 monclient: authenticate success, global_id 104131 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 1 -- 172.21.15.142:0/401509888 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4400bc30 con 0x7fae4c09f7f0 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 4 set_mon_vals no callback set 2024-09-18T10:18:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 1 -- 172.21.15.142:0/401509888 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440092e0 con 0x7fae4c09f7f0 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae50bbc700 10 monclient: dump: 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.259 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: shutdown 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- 172.21.15.142:0/401509888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 msgr2=0x7fae4c09ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c09ae60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0a35b0 tx=0x7fae44009840 comp rx=0 tx=0).stop 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- 172.21.15.142:0/401509888 shutdown_connections 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c09ae60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 --2- 172.21.15.142:0/401509888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c132b00 0x7fae4c09f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 --2- 172.21.15.142:0/401509888 >> v2:172.21.15.142:3300/0 conn(0x7fae4c09eee0 0x7fae4c1325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- 172.21.15.142:0/401509888 >> 172.21.15.142:0/401509888 conn(0x7fae4c0a5700 msgr2=0x7fae4c0a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- 172.21.15.142:0/401509888 shutdown_connections 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- 172.21.15.142:0/401509888 wait complete. 2024-09-18T10:18:08.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: monmap: 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 Processor -- start 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 -- start start 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: init 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c09eee0 0x7fae4c12ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.241+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-c con 0x7fae4c09eee0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c06d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-b con 0x7fae4c09f7f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c132b00 0x7fae4c06dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): picked mon.noname-a con 0x7fae4c132b00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0a11c0 con 0x7fae4c132b00 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c06f320 con 0x7fae4c09f7f0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c142500 con 0x7fae4c09eee0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient(hunting): authenticate will time out at 1377.172159s 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c09eee0 0x7fae4c12ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c132b00 0x7fae4c06dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c09eee0 0x7fae4c12ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42566/0 (socket says 172.21.15.142:42566) 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c06d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 1 -- 172.21.15.142:0/1569120108 learned_addr learned my addr 172.21.15.142:0/1569120108 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): get_auth_request con 0x7fae4c09eee0 auth_method 0 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): get_auth_request con 0x7fae4c09f7f0 auth_method 0 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): get_auth_request con 0x7fae4c132b00 auth_method 0 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae523bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient(hunting): handle_auth_done global_id 104134 payload 824 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c09eee0 msgr2=0x7fae4c12ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae4c09eee0 0x7fae4c12ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c132b00 msgr2=0x7fae4c06dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c132b00 0x7fae4c06dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 monclient: found mon.noname-b 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c009420 con 0x7fae4c09f7f0 2024-09-18T10:18:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 msgr2=0x7fae4c06d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae513bd700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c06d6d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00ec80 tx=0x7fae3c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c009700 con 0x7fae4c09f7f0 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: dump: 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.266 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 5 monclient: authenticate success, global_id 104134 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient: _renew_subs 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c00ca60 con 0x7fae4c09f7f0 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae44009ae0 con 0x7fae4c09f7f0 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c00ce20 con 0x7fae4c09f7f0 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae41ffb700 10 monclient: dump: 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.267 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae43fff700 4 set_mon_vals no callback set 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient: _renew_subs 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.245+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c06f830 con 0x7fae4c09f7f0 2024-09-18T10:18:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae52bc0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae52bc0700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 0x7fae4c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0647a0 con 0x7fae4c135920 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 0x7fae4c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient: get_auth_request con 0x7fae4c135920 auth_method 0 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fae3c013680 con 0x7fae4c09f7f0 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae41ffb700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae30035ac0 0x7fae30037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 10 monclient(hunting): handle_auth_done global_id 94389 payload 824 2024-09-18T10:18:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fae3c049210 con 0x7fae4c09f7f0 2024-09-18T10:18:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae51bbe700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae4c11c180 con 0x7fae4c135920 2024-09-18T10:18:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae523bf700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae30035ac0 0x7fae30037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.249+0000 7fae523bf700 10 monclient: get_auth_request con 0x7fae30035ac0 auth_method 0 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae51bbe700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 msgr2=0x7fae4c06e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae51bbe700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 0x7fae4c06e150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800ece0 con 0x7fae4c135920 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 10 monclient: dump: 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fae4800ee40 con 0x7fae4c135920 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 msgr2=0x7fae4c06e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 0x7fae4c06e150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae48008040 comp rx=0 tx=0).stop 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 msgr2=0x7fae4c06e150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae41ffb700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c135920 0x7fae4c06e150 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae48008040 comp rx=0 tx=0).stop 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.253+0000 7fae523bf700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae30035ac0 0x7fae30037f70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fae44009fd0 tx=0x7fae44018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae52bc0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae52bc0700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 0x7fae4c06e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae52bc0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c072a80 con 0x7fae4c12f3c0 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 0x7fae4c06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient: get_auth_request con 0x7fae4c12f3c0 auth_method 0 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 10 monclient(hunting): handle_auth_done global_id 94392 payload 824 2024-09-18T10:18:08.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 1 -- 172.21.15.142:0/1569120108 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fae4c09c630 con 0x7fae4c12f3c0 2024-09-18T10:18:08.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 msgr2=0x7fae4c06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae51bbe700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 0x7fae4c06e690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae4801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4802b070 con 0x7fae4c12f3c0 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.265+0000 7fae41ffb700 10 monclient: dump: 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.276 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fae48023040 con 0x7fae4c12f3c0 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 msgr2=0x7fae4c06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 0x7fae4c06e690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae4801d000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 msgr2=0x7fae4c06e690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae41ffb700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae4c12f3c0 0x7fae4c06e690 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae48010b80 tx=0x7fae4801d000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 6, 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.288 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:08.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.290 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:08.290 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae30035ac0 msgr2=0x7fae30037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae30035ac0 0x7fae30037f70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fae44009fd0 tx=0x7fae44018000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 10 monclient: shutdown 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 msgr2=0x7fae4c06d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae523bf700 1 -- 172.21.15.142:0/1569120108 reap_dead start 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c06d6d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fae3c00ec80 tx=0x7fae3c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 shutdown_connections 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 --2- 172.21.15.142:0/1569120108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae4c09f7f0 0x7fae4c06d6d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 >> 172.21.15.142:0/1569120108 conn(0x7fae4c0a5700 msgr2=0x7fae4c0a3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 shutdown_connections 2024-09-18T10:18:08.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.269+0000 7fae52bc0700 1 -- 172.21.15.142:0/1569120108 wait complete. 2024-09-18T10:18:08.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:08.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:18:08.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:18:08.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950952559}) 2024-09-18T10:18:08.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192396601}) 2024-09-18T10:18:08.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950952559}) 2024-09-18T10:18:08.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957091621}) 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980557905}) 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3702789254}) 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980557905}) 2024-09-18T10:18:08.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.321+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254829973}) 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036359611}) 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967900096}) 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036359611}) 2024-09-18T10:18:08.331 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288481264}) 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553552259}) 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002757583}) 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553552259}) 2024-09-18T10:18:08.333 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.325+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276732013}) 2024-09-18T10:18:08.334 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.334 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2059862926}) 2024-09-18T10:18:08.334 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3688895228}) 2024-09-18T10:18:08.334 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.334 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2059862926}) 2024-09-18T10:18:08.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355698926}) 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929417537}) 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186254577}) 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929417537}) 2024-09-18T10:18:08.336 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.329+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1593143030}) 2024-09-18T10:18:08.337 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.337 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824083253}) 2024-09-18T10:18:08.337 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561651313}) 2024-09-18T10:18:08.337 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.338 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824083253}) 2024-09-18T10:18:08.338 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394279529}) 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707937809}) 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148844174}) 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707937809}) 2024-09-18T10:18:08.339 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204937095}) 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: get_monmap_and_config 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: monmap: 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.340 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:08.338296+0000 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:08.338296+0000 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 1 Processor -- start 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 1 -- start start 2024-09-18T10:18:08.341 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: init 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772343814}) 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140431538}) 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772343814}) 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.333+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679504798}) 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498873711}) 2024-09-18T10:18:08.344 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=424855575}) 2024-09-18T10:18:08.345 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.345 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498873711}) 2024-09-18T10:18:08.345 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251377615}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630543892}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002239243}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630543892}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.337+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876439144}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187950775}) 2024-09-18T10:18:08.352 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152395255}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187950775}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055732427}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139978430}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396773543}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139978430}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.341+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150051042}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443095002}) 2024-09-18T10:18:08.353 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2360481479}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443095002}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292632710}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863766756}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616110137}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863766756}) 2024-09-18T10:18:08.354 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.345+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709089093}) 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.333+0000 7fdeb7861700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdeb009cf90 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb00a09a0 0x7fdeb0137a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-b con 0x7fdeb00a09a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.356 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdeb00a0de0 0x7fdeb0139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-c con 0x7fdeb00a0de0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb009dda0 con 0x7fdeb009cf90 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb00a2220 con 0x7fdeb00a0de0 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb009e340 con 0x7fdeb00a09a0 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient(hunting): authenticate will time out at 1370.432332s 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb55fd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdeb009cf90 0x7fdeb00a0290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:37396/0 (socket says 172.21.15.150:37396) 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdeb00a0de0 0x7fdeb0139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb55fd700 1 -- 172.21.15.150:0/417053438 learned_addr learned my addr 172.21.15.150:0/417053438 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:18:08.357 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb00a09a0 0x7fdeb0137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request con 0x7fdeb00a0de0 auth_method 0 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): get_auth_request con 0x7fdeb00a09a0 auth_method 0 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_done global_id 104137 payload 274 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 -- 172.21.15.150:0/417053438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb00a09a0 msgr2=0x7fdeb0137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb00a09a0 0x7fdeb0137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 -- 172.21.15.150:0/417053438 >> v1:172.21.15.142:6789/0 conn(0x7fdeb009cf90 legacy=0x7fdeb00a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 monclient: found mon.noname-c 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 -- 172.21.15.150:0/417053438 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb009d360 con 0x7fdeb00a0de0 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 -- 172.21.15.150:0/417053438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a0de0 msgr2=0x7fdeb0139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb5dfe700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a0de0 0x7fdeb0139f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009e60 tx=0x7fdea8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c41be74af76f966 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.359 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 1 -- 172.21.15.150:0/417053438 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800e070 con 0x7fdeb00a0de0 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: dump: 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.360 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 5 monclient: authenticate success, global_id 104137 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 1 -- 172.21.15.150:0/417053438 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea800bd70 con 0x7fdeb00a0de0 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 4 set_mon_vals no callback set 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 1 -- 172.21.15.150:0/417053438 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8012660 con 0x7fdeb00a0de0 2024-09-18T10:18:08.361 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdea7fff700 10 monclient: dump: 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 10 monclient: shutdown 2024-09-18T10:18:08.362 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 -- 172.21.15.150:0/417053438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a0de0 msgr2=0x7fdeb0139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.337+0000 7fdeb7861700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a0de0 0x7fdeb0139f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009e60 tx=0x7fdea8009780 comp rx=0 tx=0).stop 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- 172.21.15.150:0/417053438 shutdown_connections 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a0de0 0x7fdeb0139f70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 --2- 172.21.15.150:0/417053438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb00a09a0 0x7fdeb0137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- 172.21.15.150:0/417053438 >> 172.21.15.150:0/417053438 conn(0x7fdeb00a58a0 msgr2=0x7fdeb00a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- 172.21.15.150:0/417053438 shutdown_connections 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- 172.21.15.150:0/417053438 wait complete. 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient: monmap: 2024-09-18T10:18:08.363 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 Processor -- start 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- start start 2024-09-18T10:18:08.364 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient: init 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fdeb008dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-c con 0x7fdeb009cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-b con 0x7fdeb00a09a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fdeb008e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): picked mon.noname-a con 0x7fdeb00a0de0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb008fce0 con 0x7fdeb00a0de0 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb009c060 con 0x7fdeb00a09a0 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.365 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0091220 con 0x7fdeb009cf90 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb7861700 10 monclient(hunting): authenticate will time out at 1370.437496s 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49310/0 (socket says 172.21.15.150:49310) 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 learned_addr learned my addr 172.21.15.150:0/3418983841 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): get_auth_request con 0x7fdeb00a09a0 auth_method 0 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fdeb008dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.366 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): get_auth_request con 0x7fdeb009cf90 auth_method 0 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb5dfe700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fdeb008e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request con 0x7fdeb00a0de0 auth_method 0 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient(hunting): handle_auth_done global_id 104140 payload 824 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 msgr2=0x7fdeb008dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.367 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fdeb008dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 msgr2=0x7fdeb008e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fdeb008e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 monclient: found mon.noname-b 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.341+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8009a20 con 0x7fdeb00a09a0 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 msgr2=0x7fdeb008e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00ecb0 tx=0x7fdeac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac00cb30 con 0x7fdeb00a09a0 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.368 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: dump: 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.369 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac00cc90 con 0x7fdeb00a09a0 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 5 monclient: authenticate success, global_id 104140 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _renew_subs 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb0099340 con 0x7fdeb00a09a0 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeac013470 con 0x7fdeb00a09a0 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 10 monclient: dump: 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:08.370 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _renew_subs 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb0090330 con 0x7fdeb00a09a0 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea77fe700 4 set_mon_vals no callback set 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.371 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb0063ed0 con 0x7fdeb00a09a0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdeac0138e0 con 0x7fdeb00a09a0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 0x7fde94037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdeac049340 con 0x7fdeb00a09a0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 0x7fde94037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.345+0000 7fdeb55fd700 10 monclient: get_auth_request con 0x7fde94035ac0 auth_method 0 2024-09-18T10:18:08.372 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.349+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 0x7fde94037f70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009ad0 tx=0x7fdea0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.349+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdeac0236b0 con 0x7fdeb00a09a0 2024-09-18T10:18:08.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.349+0000 7fdea57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.373 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.349+0000 7fdea57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:08.377 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.349+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.377 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.349+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087182872}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.349+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719672611}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087182872}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522159644}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696767025}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692893399}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696767025}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861813347}) 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.378 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969295141}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61694647}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969295141}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.353+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291848790}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142280119}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878204411}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142280119}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3615765495}) 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.379 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286934381}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198522170}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286934381}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993756086}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=2564184525,mdsmap=1531151339,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2907646644,mon_config_key=620546692,monmap=3505902565,osd_metadata=361655855,osd_pg_creating=1011622056,osd_snap=454132730,osdmap=834748089}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=12} crc {logm=3067565221,mdsmap=1531151339,mgr=1117889293,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1779682553,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=1011622056,osd_snap=454132730,osdmap=1754296959}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=10} crc {logm=2564184525,mdsmap=1531151339,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2907646644,mon_config_key=620546692,monmap=3505902565,osd_metadata=361655855,osd_pg_creating=1011622056,osd_snap=454132730,osdmap=834748089}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.357+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=11} crc {logm=882948330,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1911734341,mon_config_key=789588371,monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=1011622056,osd_snap=3823785925,osdmap=1457726879}) 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.380 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1094364938}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2713555164}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1094364938}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.361+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=837886100}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=255599539}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=34} crc {osdmap=3695989758}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=255599539}) 2024-09-18T10:18:08.381 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:08.369+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=482425016}) 2024-09-18T10:18:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: monmap: 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:08.404919+0000 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:08.404919+0000 2024-09-18T10:18:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:08.408 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 Processor -- start 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 -- start start 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: init 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62fc0999e0 0x7f62fc0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f62fc0999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62fc09c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62fc137910 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc0a20c0 con 0x7f62fc09c410 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc0a1c80 con 0x7f62fc137910 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc09a000 con 0x7f62fc0999e0 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fbfff700 10 monclient(hunting): authenticate will time out at 1377.330786s 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fa7fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f62fc09c410 0x7f62fc0a06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49016/0 (socket says 172.21.15.142:49016) 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62fa7fc700 1 -- 172.21.15.142:0/1852893162 learned_addr learned my addr 172.21.15.142:0/1852893162 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 1 --2- 172.21.15.142:0/1852893162 >> v2:172.21.15.150:3300/0 conn(0x7f62fc0999e0 0x7f62fc0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): get_auth_request con 0x7f62fc0999e0 auth_method 0 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient(hunting): handle_auth_done global_id 104143 payload 274 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 1 -- 172.21.15.142:0/1852893162 >> v1:172.21.15.150:6789/0 conn(0x7f62fc137910 legacy=0x7f62fc139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 1 -- 172.21.15.142:0/1852893162 >> v1:172.21.15.142:6789/0 conn(0x7f62fc09c410 legacy=0x7f62fc0a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 1 monclient: found mon.noname-c 2024-09-18T10:18:08.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.401+0000 7f62faffd700 1 -- 172.21.15.142:0/1852893162 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc09a580 con 0x7f62fc0999e0 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62faffd700 1 -- 172.21.15.142:0/1852893162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc0999e0 msgr2=0x7f62fc0a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62faffd700 1 --2- 172.21.15.142:0/1852893162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc0999e0 0x7f62fc0a0180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0098e0 tx=0x7f62ec0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c23e55148b916d29 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 1 -- 172.21.15.142:0/1852893162 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec00ba70 con 0x7f62fc0999e0 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: dump: 2024-09-18T10:18:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 1 -- 172.21.15.142:0/1852893162 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec00bbd0 con 0x7f62fc0999e0 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 5 monclient: authenticate success, global_id 104143 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 1 -- 172.21.15.142:0/1852893162 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec011420 con 0x7f62fc0999e0 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 4 set_mon_vals no callback set 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62f9ffb700 10 monclient: dump: 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 10 monclient: shutdown 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 -- 172.21.15.142:0/1852893162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc0999e0 msgr2=0x7f62fc0a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 --2- 172.21.15.142:0/1852893162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc0999e0 0x7f62fc0a0180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0098e0 tx=0x7f62ec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 -- 172.21.15.142:0/1852893162 shutdown_connections 2024-09-18T10:18:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 --2- 172.21.15.142:0/1852893162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc0999e0 0x7f62fc0a0180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 -- 172.21.15.142:0/1852893162 >> 172.21.15.142:0/1852893162 conn(0x7f62fc0a5790 msgr2=0x7f62fc0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.405+0000 7f62fbfff700 1 -- 172.21.15.142:0/1852893162 shutdown_connections 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 -- 172.21.15.142:0/1852893162 wait complete. 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient: monmap: 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 Processor -- start 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 -- start start 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient: init 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f62fc0999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc09c410 0x7f62fc141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f62fc09c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc137910 0x7f62fc1424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f62fc137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc12f8f0 con 0x7f62fc0999e0 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc12ff20 con 0x7f62fc09c410 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc09f280 con 0x7f62fc137910 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fbfff700 10 monclient(hunting): authenticate will time out at 1377.338032s 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52012/0 (socket says 172.21.15.142:52012) 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc137910 0x7f62fc1424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 -- 172.21.15.142:0/2479856691 learned_addr learned my addr 172.21.15.142:0/2479856691 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): get_auth_request con 0x7f62fc0999e0 auth_method 0 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): get_auth_request con 0x7f62fc137910 auth_method 0 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc09c410 0x7f62fc141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): get_auth_request con 0x7f62fc09c410 auth_method 0 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient(hunting): handle_auth_done global_id 94398 payload 824 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc137910 msgr2=0x7f62fc1424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc137910 0x7f62fc1424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc09c410 msgr2=0x7f62fc141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62fc09c410 0x7f62fc141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 monclient: found mon.noname-a 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62faffd700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec009420 con 0x7f62fc0999e0 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.409+0000 7f62fb7fe700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc137910 0x7f62fc1424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62faffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 msgr2=0x7f62fc130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62faffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0a1e10 tx=0x7f62ec016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01a460 con 0x7f62fc0999e0 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:08.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: dump: 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 5 monclient: authenticate success, global_id 94398 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec01a5c0 con 0x7f62fc0999e0 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01a940 con 0x7f62fc0999e0 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 10 monclient: dump: 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 monclient: _renew_subs 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc12de90 con 0x7f62fc0999e0 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f97fa700 4 set_mon_vals no callback set 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 monclient: _renew_subs 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc142a30 con 0x7f62fc0999e0 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f62ec01ad30 con 0x7f62fc0999e0 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 0x7f62fc06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc12e680 con 0x7f62fc06dad0 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 0x7f62fc06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient: get_auth_request con 0x7f62fc06dad0 auth_method 0 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62dc035ac0 0x7f62dc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f62ec04ae50 con 0x7f62fc0999e0 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_done global_id 74807 payload 824 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fa7fc700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62fc142c20 con 0x7f62fc06dad0 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fb7fe700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62dc035ac0 0x7f62dc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.413+0000 7f62fb7fe700 10 monclient: get_auth_request con 0x7f62dc035ac0 auth_method 0 2024-09-18T10:18:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62fa7fc700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 msgr2=0x7f62fc06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62fa7fc700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 0x7f62fc06dea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f62e400eac0 tx=0x7f62e4007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400cc40 con 0x7f62fc06dad0 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 10 monclient: dump: 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f62e400cda0 con 0x7f62fc06dad0 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62fb7fe700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62dc035ac0 0x7f62dc037f70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 msgr2=0x7f62fc06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 0x7f62fc06dea0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f62e400eac0 tx=0x7f62e4007f90 comp rx=0 tx=0).stop 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 msgr2=0x7f62fc06dea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.417+0000 7f62f2ffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06dad0 0x7f62fc06dea0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fbfff700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 0x7f62fc12c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0647f0 con 0x7f62fc06e3e0 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 0x7f62fc12c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient: get_auth_request con 0x7f62fc06e3e0 auth_method 0 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 10 monclient(hunting): handle_auth_done global_id 74810 payload 824 2024-09-18T10:18:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.433+0000 7f62fa7fc700 1 -- 172.21.15.142:0/2479856691 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f62fc142a30 con 0x7f62fc06e3e0 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fa7fc700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 msgr2=0x7f62fc12c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fa7fc700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 0x7f62fc12c3f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007360 tx=0x7f62e4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e4029400 con 0x7f62fc06e3e0 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 10 monclient: dump: 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.443 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f62e400ca30 con 0x7f62fc06e3e0 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 msgr2=0x7f62fc12c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 0x7f62fc12c3f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007360 tx=0x7f62e4016000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 msgr2=0x7f62fc12c3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62f2ffd700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62fc06e3e0 0x7f62fc12c3f0 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007360 tx=0x7f62e4016000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:08.456 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:08.457 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:08.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62dc035ac0 msgr2=0x7f62dc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62dc035ac0 0x7f62dc037f70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4002620 comp rx=0 tx=0).stop 2024-09-18T10:18:08.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 10 monclient: shutdown 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 msgr2=0x7f62fc130a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0a1e10 tx=0x7f62ec016000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fb7fe700 1 -- 172.21.15.142:0/2479856691 reap_dead start 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 shutdown_connections 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 --2- 172.21.15.142:0/2479856691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62fc0999e0 0x7f62fc130a80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 >> 172.21.15.142:0/2479856691 conn(0x7f62fc0a5790 msgr2=0x7f62fc09de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 shutdown_connections 2024-09-18T10:18:08.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.437+0000 7f62fbfff700 1 -- 172.21.15.142:0/2479856691 wait complete. 2024-09-18T10:18:08.460 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:08.460 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:18:08.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:18:08.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.525+0000 7fdeb7861700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:08.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.525+0000 7fdeb7861700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:08.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.525+0000 7fdeb7861700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.532 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.525+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fdeb009e890 con 0x7fdeb00a09a0 2024-09-18T10:18:08.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: monmap: 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:08.613158+0000 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:08.613158+0000 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.615 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:08.616 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:08.616 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:08.616 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:08.616 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:08.616 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 Processor -- start 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 -- start start 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: init 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fcc0999e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 0x7f1fcc0a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1fcc09c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fcc137910 0x7f1fcc139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc137910 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc0a20c0 con 0x7f1fcc0999e0 2024-09-18T10:18:08.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc09a000 con 0x7f1fcc137910 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc0a1c80 con 0x7f1fcc09c410 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcbfff700 10 monclient(hunting): authenticate will time out at 1377.538960s 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fcc137910 0x7f1fcc139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fcc137910 0x7f1fcc139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52022/0 (socket says 172.21.15.142:52022) 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/2033178835 learned_addr learned my addr 172.21.15.142:0/2033178835 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request con 0x7f1fcc137910 auth_method 0 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_done global_id 94401 payload 274 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/2033178835 >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 msgr2=0x7f1fcc0a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/2033178835 >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 0x7f1fcc0a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/2033178835 >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 0x7f1fcc0a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/2033178835 >> v1:172.21.15.150:6789/0 conn(0x7f1fcc0999e0 legacy=0x7f1fcc0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 monclient: found mon.noname-a 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/2033178835 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc09a580 con 0x7f1fcc137910 2024-09-18T10:18:08.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fca7fc700 10 monclient: get_auth_request con 0x7f1fcc09c410 auth_method 0 2024-09-18T10:18:08.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.609+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/2033178835 >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 0x7f1fcc0a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/2033178835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc137910 msgr2=0x7f1fcc139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/2033178835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc137910 0x7f1fcc139cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06fbb0 tx=0x7f1fc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3286364b4c22a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 1 -- 172.21.15.142:0/2033178835 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400e070 con 0x7f1fcc137910 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: dump: 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.624 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 5 monclient: authenticate success, global_id 94401 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 1 -- 172.21.15.142:0/2033178835 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc400bc60 con 0x7f1fcc137910 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:08.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 4 set_mon_vals no callback set 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 1 -- 172.21.15.142:0/2033178835 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4012430 con 0x7f1fcc137910 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fc9ffb700 10 monclient: dump: 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.626 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 10 monclient: shutdown 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 -- 172.21.15.142:0/2033178835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc137910 msgr2=0x7f1fcc139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/2033178835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc137910 0x7f1fcc139cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06fbb0 tx=0x7f1fc4009840 comp rx=0 tx=0).stop 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 -- 172.21.15.142:0/2033178835 shutdown_connections 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/2033178835 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc137910 0x7f1fcc139cf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/2033178835 >> v2:172.21.15.150:3300/0 conn(0x7f1fcc09c410 0x7f1fcc0a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 -- 172.21.15.142:0/2033178835 >> 172.21.15.142:0/2033178835 conn(0x7f1fcc0a5790 msgr2=0x7f1fcc0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.613+0000 7f1fcbfff700 1 -- 172.21.15.142:0/2033178835 shutdown_connections 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 -- 172.21.15.142:0/2033178835 wait complete. 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient: monmap: 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 Processor -- start 2024-09-18T10:18:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 -- start start 2024-09-18T10:18:08.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient: init 2024-09-18T10:18:08.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:08.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fcc095f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f1fcc0999e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f1fcc09c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fcc06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f1fcc137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc095410 con 0x7f1fcc09c410 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc06f2c0 con 0x7f1fcc0999e0 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0994c0 con 0x7f1fcc137910 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcbfff700 10 monclient(hunting): authenticate will time out at 1377.547512s 2024-09-18T10:18:08.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52030/0 (socket says 172.21.15.142:52030) 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 learned_addr learned my addr 172.21.15.142:0/357911150 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): get_auth_request con 0x7f1fcc09c410 auth_method 0 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fcc06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request con 0x7f1fcc137910 auth_method 0 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fcc095f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request con 0x7f1fcc0999e0 auth_method 0 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.617+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_done global_id 94404 payload 824 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 msgr2=0x7f1fcc06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fcc06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 msgr2=0x7f1fcc095f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fcc095f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 monclient: found mon.noname-a 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009ae0 con 0x7f1fcc09c410 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 msgr2=0x7f1fcc094ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c830 tx=0x7f1fbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc011810 con 0x7f1fcc09c410 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: dump: 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 5 monclient: authenticate success, global_id 94404 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fbc011970 con 0x7f1fcc09c410 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 10 monclient: _renew_subs 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc011d30 con 0x7f1fcc09c410 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc136b50 con 0x7f1fcc09c410 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fc97fa700 4 set_mon_vals no callback set 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: dump: 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.643 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 10 monclient: _renew_subs 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc06f820 con 0x7f1fcc09c410 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 msgr2=0x7f1fcc094ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:08.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 msgr2=0x7f1fcc094ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c830 tx=0x7f1fbc008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00c830 tx=0x7f1fbc008040 comp rx=0 tx=0).stop 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 msgr2=0x7f1fcc094ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fcc09c410 0x7f1fcc094ed0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fb8005880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): picked mon.c con 0x7f1fcc137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fb80081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): picked mon.b con 0x7f1fcc0999e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8008b70 0x7f1fb800aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): picked mon.a con 0x7f1fb8008b70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb800baa0 con 0x7f1fb8008b70 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb800bbe0 con 0x7f1fcc0999e0 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb800bd20 con 0x7f1fcc137910 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fb2ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fb8005880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fb80081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request con 0x7f1fcc137910 auth_method 0 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8008b70 0x7f1fb800aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcaffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient(hunting): get_auth_request con 0x7f1fcc0999e0 auth_method 0 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fca7fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request con 0x7f1fb8008b70 auth_method 0 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.621+0000 7f1fcb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:08.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient(hunting): handle_auth_done global_id 94404 payload 293 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 msgr2=0x7f1fb8005880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc137910 0x7f1fb8005880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8008b70 msgr2=0x7f1fb800aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8008b70 0x7f1fb800aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 monclient: found mon.b 2024-09-18T10:18:08.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f1fb800c010 con 0x7f1fcc0999e0 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8008b70 0x7f1fb800aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 0x7f1fcc095f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc0647f0 con 0x7f1fcc094ed0 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 msgr2=0x7f1fb80081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fca7fc700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fb80081f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc006fd0 tx=0x7f1fbc01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 0x7f1fcc095f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient: get_auth_request con 0x7f1fcc094ed0 auth_method 0 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc029070 con 0x7f1fcc0999e0 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 10 monclient: dump: 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fbc019a80 con 0x7f1fcc0999e0 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fb2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fc8ff9700 4 set_mon_vals no callback set 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_done global_id 104152 payload 824 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.625+0000 7f1fcaffd700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fcc0952a0 con 0x7f1fcc094ed0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1fbc019bf0 con 0x7f1fcc0999e0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fb803e130 0x7f1fb80405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 4 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1fbc04ae30 con 0x7f1fcc0999e0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fb803e130 0x7f1fb80405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fcb7fe700 10 monclient: get_auth_request con 0x7f1fb803e130 auth_method 0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fcaffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 msgr2=0x7f1fcc095f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 0x7f1fcc095f40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0011800 tx=0x7f1fc0009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00158a0 con 0x7f1fcc094ed0 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 10 monclient: dump: 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1fc0015a00 con 0x7f1fcc094ed0 2024-09-18T10:18:08.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 msgr2=0x7f1fcc095f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 0x7f1fcc095f40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0011800 tx=0x7f1fc0009040 comp rx=0 tx=0).stop 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 msgr2=0x7f1fcc095f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.629+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc094ed0 0x7f1fcc095f40 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0011800 tx=0x7f1fc0009040 comp rx=0 tx=0).stop 2024-09-18T10:18:08.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.633+0000 7f1fcb7fe700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fb803e130 0x7f1fb80405e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06fbb0 tx=0x7f1fc4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 0x7f1fcc06d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc072a80 con 0x7f1fcc06d140 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 0x7f1fcc06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient: get_auth_request con 0x7f1fcc06d140 auth_method 0 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 10 monclient(hunting): handle_auth_done global_id 104155 payload 824 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.641+0000 7f1fcaffd700 1 -- 172.21.15.142:0/357911150 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1fcc067800 con 0x7f1fcc06d140 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcaffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 msgr2=0x7f1fcc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcaffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 0x7f1fcc06d540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007d40 tx=0x7f1fc001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc002c070 con 0x7f1fcc06d140 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 10 monclient: dump: 2024-09-18T10:18:08.657 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f1fc0024040 con 0x7f1fcc06d140 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:08.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 msgr2=0x7f1fcc06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 0x7f1fcc06d540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0007d40 tx=0x7f1fc001e000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 msgr2=0x7f1fcc06d540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fb2ffd700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc06d140 0x7f1fcc06d540 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:08.659 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 96, 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:18:08.660 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.661 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:08.663 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:08.664 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:08.665 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:08.666 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:08.667 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.671 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:08.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:08.673 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:08.673 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fb803e130 msgr2=0x7f1fb80405e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fb803e130 0x7f1fb80405e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc06fbb0 tx=0x7f1fc4017000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 10 monclient: shutdown 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 msgr2=0x7f1fb80081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 --2- 172.21.15.142:0/357911150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fcc0999e0 0x7f1fb80081f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc006fd0 tx=0x7f1fbc01a000 comp rx=0 tx=0).stop 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/357911150 reap_dead start 2024-09-18T10:18:08.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcb7fe700 1 -- 172.21.15.142:0/357911150 reap_dead start 2024-09-18T10:18:08.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 shutdown_connections 2024-09-18T10:18:08.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 >> 172.21.15.142:0/357911150 conn(0x7f1fcc0a5790 msgr2=0x7f1fcc096590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 shutdown_connections 2024-09-18T10:18:08.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.645+0000 7f1fcbfff700 1 -- 172.21.15.142:0/357911150 wait complete. 2024-09-18T10:18:08.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:08.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:18:08.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:18:08.675 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:18:08.676 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:18:08.676 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:18:08.676 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 msgr2=0x7fdeb008e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 msgr2=0x7fdeb008e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00ecb0 tx=0x7fdeac0091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00ecb0 tx=0x7fdeac0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.684 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 msgr2=0x7fdeb008e150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdeb00a09a0 0x7fdeb008e150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 0x7fde94040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): picked mon.b con 0x7fde9403ea60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fde94043760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): picked mon.a con 0x7fdeb00a0de0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 msgr2=0x7fde94040e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fde940461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb4dfc700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 0x7fde94040e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): picked mon.c con 0x7fdeb009cf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.685 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde94046b20 con 0x7fdeb00a0de0 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde94046c60 con 0x7fde9403ea60 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde94046df0 con 0x7fdeb009cf90 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdea57fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fde94043760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fde940461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request con 0x7fdeb00a0de0 auth_method 0 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): get_auth_request con 0x7fdeb009cf90 auth_method 0 2024-09-18T10:18:08.686 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient(hunting): handle_auth_done global_id 104140 payload 293 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 msgr2=0x7fde94040e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 0x7fde94040e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 msgr2=0x7fde94043760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.687 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fde94043760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 monclient: found mon.c 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fde94046fe0 con 0x7fdeb009cf90 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient: _send_command 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fdea000ec40 con 0x7fdeb009cf90 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.688 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb55fd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.689 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.677+0000 7fdeb5dfe700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fde94043760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdeb55fd700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 msgr2=0x7fde940461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdeb55fd700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fde940461a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdea000edd0 tx=0x7fdea0015480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0015840 con 0x7fdeb009cf90 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 10 monclient: dump: 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:08.690 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00159a0 con 0x7fdeb009cf90 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea77fe700 4 set_mon_vals no callback set 2024-09-18T10:18:08.691 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:08.681+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdea0015b10 con 0x7fdeb009cf90 2024-09-18T10:18:08.692 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:18:08.692 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:18:08.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: get_monmap_and_config 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: monmap: 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:08.798331+0000 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:08.798331+0000 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.800 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:08.801 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:08.801 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:08.801 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:08.801 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:08.801 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 Processor -- start 2024-09-18T10:18:08.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 -- start start 2024-09-18T10:18:08.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: init 2024-09-18T10:18:08.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c0a0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-b con 0x7fd39c0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd39c09c3f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd39c137890 0x7fd39c139c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-a con 0x7fd39c137890 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39c09a5f0 con 0x7fd39c09c3f0 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c09b300 con 0x7fd39c137890 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c09a310 con 0x7fd39c0999e0 2024-09-18T10:18:08.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.793+0000 7fd3a19ef700 10 monclient(hunting): authenticate will time out at 1377.723811s 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd39c137890 0x7fd39c139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd393fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd39c09c3f0 0x7fd39c0a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49020/0 (socket says 172.21.15.142:49020) 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c0a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd393fff700 1 -- 172.21.15.142:0/3763652769 learned_addr learned my addr 172.21.15.142:0/3763652769 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): get_auth_request con 0x7fd39c137890 auth_method 0 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): get_auth_request con 0x7fd39c0999e0 auth_method 0 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient(hunting): handle_auth_done global_id 94407 payload 274 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 -- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 msgr2=0x7fd39c0a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 --2- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c0a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 -- 172.21.15.142:0/3763652769 >> v1:172.21.15.142:6789/0 conn(0x7fd39c09c3f0 legacy=0x7fd39c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 monclient: found mon.noname-a 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:08.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 -- 172.21.15.142:0/3763652769 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c09a8d0 con 0x7fd39c137890 2024-09-18T10:18:08.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 -- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c137890 msgr2=0x7fd39c139c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a11ee700 1 --2- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c137890 0x7fd39c139c70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd394009e60 tx=0x7fd394009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f14d1452321d96 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 1 -- 172.21.15.142:0/3763652769 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39400e070 con 0x7fd39c137890 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: dump: 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.809 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a19ef700 5 monclient: authenticate success, global_id 94407 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a19ef700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 1 -- 172.21.15.142:0/3763652769 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39400bd70 con 0x7fd39c137890 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 1 -- 172.21.15.142:0/3763652769 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394012660 con 0x7fd39c137890 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a19ef700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3a19ef700 4 set_mon_vals no callback set 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.797+0000 7fd3937fe700 10 monclient: dump: 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 10 monclient: shutdown 2024-09-18T10:18:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c137890 msgr2=0x7fd39c139c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c137890 0x7fd39c139c70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd394009e60 tx=0x7fd394009780 comp rx=0 tx=0).stop 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- 172.21.15.142:0/3763652769 shutdown_connections 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c137890 0x7fd39c139c70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/3763652769 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c0a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- 172.21.15.142:0/3763652769 >> 172.21.15.142:0/3763652769 conn(0x7fd39c0a5710 msgr2=0x7fd39c0a5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- 172.21.15.142:0/3763652769 shutdown_connections 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- 172.21.15.142:0/3763652769 wait complete. 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 10 monclient: build_initial_monmap 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 10 monclient: monmap: 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 Processor -- start 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 1 -- start start 2024-09-18T10:18:08.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.801+0000 7fd3a19ef700 10 monclient: init 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-c con 0x7fd39c0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 0x7fd39c12b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-a con 0x7fd39c09c3f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:08.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 0x7fd39c12b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): picked mon.noname-b con 0x7fd39c137890 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c09f100 con 0x7fd39c09c3f0 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c12ea10 con 0x7fd39c137890 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c12e730 con 0x7fd39c0999e0 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient(hunting): authenticate will time out at 1377.732195s 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 0x7fd39c12b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42640/0 (socket says 172.21.15.142:42640) 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a11ee700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 msgr2=0x7fd39c12b980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 -- 172.21.15.142:0/1801342403 learned_addr learned my addr 172.21.15.142:0/1801342403 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): get_auth_request con 0x7fd39c0999e0 auth_method 0 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): get_auth_request con 0x7fd39c09c3f0 auth_method 0 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a11ee700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 0x7fd39c12b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient(hunting): handle_auth_done global_id 74819 payload 824 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 -- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 msgr2=0x7fd39c12b980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 0x7fd39c12b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 -- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 msgr2=0x7fd39c12b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 0x7fd39c12b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient: _finish_hunting 0 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 monclient: found mon.noname-c 2024-09-18T10:18:08.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 -- 172.21.15.142:0/1801342403 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394009a20 con 0x7fd39c0999e0 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd393fff700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 0x7fd39c12b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 -- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 msgr2=0x7fd39c12af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a09ed700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd39800e930 tx=0x7fd3980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398013890 con 0x7fd39c0999e0 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: dump: 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.821 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3980139f0 con 0x7fd39c0999e0 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 5 monclient: authenticate success, global_id 74819 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _renew_subs 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c12e450 con 0x7fd39c0999e0 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398013db0 con 0x7fd39c0999e0 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd390ff9700 10 monclient: dump: 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:08.823 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd392ffd700 4 set_mon_vals no callback set 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _renew_subs 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.805+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c1365e0 con 0x7fd39c0999e0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd3a19ef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd3a19ef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd3a19ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c12eba0 con 0x7fd39c0999e0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd39800c670 con 0x7fd39c0999e0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd390ff9700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 0x7fd388037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd3980490d0 con 0x7fd39c0999e0 2024-09-18T10:18:08.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd393fff700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 0x7fd388037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:08.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.809+0000 7fd393fff700 10 monclient: get_auth_request con 0x7fd388035a70 auth_method 0 2024-09-18T10:18:08.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.813+0000 7fd393fff700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 0x7fd388037f20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005220 tx=0x7fd38c007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:08.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.817+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd398079670 con 0x7fd39c0999e0 2024-09-18T10:18:08.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.817+0000 7fd390ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:08.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:08.817+0000 7fd390ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:09.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:09.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:09.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:09.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd39c072a80 con 0x7fd39c0999e0 2024-09-18T10:18:09.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd390ff9700 1 -- 172.21.15.142:0/1801342403 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fd398023ad0 con 0x7fd39c0999e0 2024-09-18T10:18:09.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd390ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:09.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd390ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:09.047 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:09.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 msgr2=0x7fd388037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.041+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 0x7fd388037f20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005220 tx=0x7fd38c007f90 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 10 monclient: shutdown 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 msgr2=0x7fd39c12af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd39800e930 tx=0x7fd3980091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 shutdown_connections 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd388035a70 0x7fd388037f20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c137890 0x7fd39c12b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd39c09c3f0 0x7fd39c12b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 --2- 172.21.15.142:0/1801342403 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd39c0999e0 0x7fd39c12af00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 >> 172.21.15.142:0/1801342403 conn(0x7fd39c0a5710 msgr2=0x7fd39c137090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:09.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 shutdown_connections 2024-09-18T10:18:09.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:09.045+0000 7fd3a19ef700 1 -- 172.21.15.142:0/1801342403 wait complete. 2024-09-18T10:18:11.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:11.344+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:11.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:11.344+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:11.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:11.344+0000 7fde9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:12.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 10 monclient: get_monmap_and_config 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 10 monclient: build_initial_monmap 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 10 monclient: monmap: 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:12.170169+0000 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:12.170169+0000 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.172 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:12.173 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:12.173 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 1 Processor -- start 2024-09-18T10:18:12.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 1 -- start start 2024-09-18T10:18:12.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.165+0000 7f1fde71f700 10 monclient: init 2024-09-18T10:18:12.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fd8132030 0x7f1fd8132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-a con 0x7f1fd8132030 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 0x7f1fd809f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-c con 0x7f1fd809ece0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 0x7f1fd809b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-b con 0x7f1fd809ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd809f960 con 0x7f1fd8132030 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd80a1a90 con 0x7f1fd809ece0 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8132940 con 0x7f1fd809ae60 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fde71f700 10 monclient(hunting): authenticate will time out at 1381.096371s 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdcf1c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 msgr2=0x7f1fd809f0e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdcf1c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 0x7f1fd809f0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fd8132030 0x7f1fd8132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fd8132030 0x7f1fd8132400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60024/0 (socket says 172.21.15.142:60024) 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 -- 172.21.15.142:0/260061309 learned_addr learned my addr 172.21.15.142:0/260061309 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): get_auth_request con 0x7f1fd8132030 auth_method 0 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 0x7f1fd809b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:12.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): get_auth_request con 0x7f1fd809ae60 auth_method 0 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_done global_id 94413 payload 274 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 -- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 msgr2=0x7f1fd809b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 0x7f1fd809b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 -- 172.21.15.142:0/260061309 >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 msgr2=0x7f1fd809f0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 --2- 172.21.15.142:0/260061309 >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 0x7f1fd809f0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 monclient: found mon.noname-a 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 -- 172.21.15.142:0/260061309 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80a17b0 con 0x7f1fd8132030 2024-09-18T10:18:12.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fddf1e700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 0x7f1fd809b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:12.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 -- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8132030 msgr2=0x7f1fd8132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:12.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.169+0000 7f1fdd71d700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8132030 0x7f1fd8132400 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80649a0 tx=0x7f1fcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9adb920322a0fa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 1 -- 172.21.15.142:0/260061309 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc009e70 con 0x7f1fd8132030 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: dump: 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:12.187 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: _finish_auth 0 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 5 monclient: authenticate success, global_id 94413 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 1 -- 172.21.15.142:0/260061309 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc00ba70 con 0x7f1fd8132030 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 4 set_mon_vals no callback set 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 1 -- 172.21.15.142:0/260061309 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc00be30 con 0x7f1fd8132030 2024-09-18T10:18:12.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fcbfff700 10 monclient: dump: 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: shutdown 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8132030 msgr2=0x7f1fd8132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8132030 0x7f1fd8132400 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80649a0 tx=0x7f1fcc009250 comp rx=0 tx=0).stop 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- 172.21.15.142:0/260061309 shutdown_connections 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd809ae60 0x7f1fd809b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- 172.21.15.142:0/260061309 >> v2:172.21.15.150:3300/0 conn(0x7f1fd809ece0 0x7f1fd809f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- 172.21.15.142:0/260061309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd8132030 0x7f1fd8132400 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- 172.21.15.142:0/260061309 >> 172.21.15.142:0/260061309 conn(0x7f1fd80a5700 msgr2=0x7f1fd80a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- 172.21.15.142:0/260061309 shutdown_connections 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- 172.21.15.142:0/260061309 wait complete. 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: build_initial_monmap 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: monmap: 2024-09-18T10:18:12.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 Processor -- start 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- start start 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: init 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:12.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd809ae60 0x7f1fd806ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-a con 0x7f1fd809ae60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 0x7f1fd8131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-b con 0x7f1fd809ece0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): picked mon.noname-c con 0x7f1fd8132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd809f6e0 con 0x7f1fd809ae60 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80a4820 con 0x7f1fd809ece0 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd806f6e0 con 0x7f1fd8132030 2024-09-18T10:18:12.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fde71f700 10 monclient(hunting): authenticate will time out at 1381.102985s 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdcf1c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 msgr2=0x7f1fd8131320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdcf1c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 0x7f1fd8131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd809ae60 0x7f1fd806ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59560/0 (socket says 172.21.15.142:59560) 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 -- 172.21.15.142:0/3578129641 learned_addr learned my addr 172.21.15.142:0/3578129641 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): get_auth_request con 0x7f1fd8132030 auth_method 0 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): get_auth_request con 0x7f1fd809ae60 auth_method 0 2024-09-18T10:18:12.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fdd71d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient(hunting): handle_auth_done global_id 74825 payload 824 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 -- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 msgr2=0x7f1fd8131320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 0x7f1fd8131320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 -- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd809ae60 msgr2=0x7f1fd806ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd809ae60 0x7f1fd806ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient: _finish_hunting 0 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 monclient: found mon.noname-c 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.173+0000 7f1fddf1e700 1 -- 172.21.15.142:0/3578129641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0009a20 con 0x7f1fd8132030 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fddf1e700 1 -- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 msgr2=0x7f1fd811bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fddf1e700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009f20 tx=0x7f1fd000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0009c80 con 0x7f1fd8132030 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: dump: 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:12.195 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: _finish_auth 0 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd000f040 con 0x7f1fd8132030 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 5 monclient: authenticate success, global_id 74825 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 10 monclient: _renew_subs 2024-09-18T10:18:12.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc0094f0 con 0x7f1fd8132030 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0014480 con 0x7f1fd8132030 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 10 monclient: dump: 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:12.197 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 10 monclient: _renew_subs 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fcb7fe700 4 set_mon_vals no callback set 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd811d550 con 0x7f1fd8132030 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1fd0003020 con 0x7f1fd8132030 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 0x7f1fbc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.177+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1fd0049900 con 0x7f1fd8132030 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fde71f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fde71f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fde71f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:12.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fdd71d700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 0x7f1fbc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:12.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd80a5470 con 0x7f1fd8132030 2024-09-18T10:18:12.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fdd71d700 10 monclient: get_auth_request con 0x7f1fbc035a70 auth_method 0 2024-09-18T10:18:12.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.181+0000 7f1fdd71d700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 0x7f1fbc037f20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80649a0 tx=0x7f1fcc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:12.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.185+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1fd80a5470 con 0x7f1fd8132030 2024-09-18T10:18:12.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.185+0000 7f1fc97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:12.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.185+0000 7f1fc97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:12.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:12.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:12.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:12.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fde71f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fde71f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fde71f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1fd80a5470 con 0x7f1fd8132030 2024-09-18T10:18:12.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fc97fa700 1 -- 172.21.15.142:0/3578129641 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f1fd80a5470 con 0x7f1fd8132030 2024-09-18T10:18:12.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fc97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:12.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.413+0000 7f1fc97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:12.420 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 msgr2=0x7f1fbc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 0x7f1fbc037f20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1fd80649a0 tx=0x7f1fcc017000 comp rx=0 tx=0).stop 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 10 monclient: shutdown 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 msgr2=0x7f1fd811bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009f20 tx=0x7f1fd000b040 comp rx=0 tx=0).stop 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 shutdown_connections 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1fbc035a70 0x7f1fbc037f20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fd8132030 0x7f1fd811bd60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fd809ece0 0x7f1fd8131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 --2- 172.21.15.142:0/3578129641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fd809ae60 0x7f1fd806ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:12.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 >> 172.21.15.142:0/3578129641 conn(0x7f1fd80a5700 msgr2=0x7f1fd81351b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:12.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 shutdown_connections 2024-09-18T10:18:12.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:12.417+0000 7f1fde71f700 1 -- 172.21.15.142:0/3578129641 wait complete. 2024-09-18T10:18:13.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:13.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:13.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:13.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:14.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:14.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:14.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:14.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:15.347 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:15.344+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:15.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:15.344+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:15.434 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:15.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: get_monmap_and_config 2024-09-18T10:18:15.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: build_initial_monmap 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: monmap: 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:15.545604+0000 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:15.545604+0000 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:15.548 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:15.549 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:15.549 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 Processor -- start 2024-09-18T10:18:15.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 -- start start 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: init 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0180a3ba0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 0x7fd01806ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd01806ebc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd018134fe0 0x7fd0181373c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd018134fe0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01806bbc0 con 0x7fd0180a3ba0 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01806cf20 con 0x7fd018134fe0 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:15.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01806d120 con 0x7fd01806ebc0 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01759e700 10 monclient(hunting): authenticate will time out at 1384.471380s 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd01659c700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fd0180a3ba0 legacy=0x7fd018130190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd018134fe0 0x7fd0181373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 0x7fd01806ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd018134fe0 0x7fd0181373c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60046/0 (socket says 172.21.15.142:60046) 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 0x7fd01806ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:59564/0 (socket says 172.21.15.142:59564) 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 1 -- 172.21.15.142:0/2901072877 learned_addr learned my addr 172.21.15.142:0/2901072877 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): get_auth_request con 0x7fd018134fe0 auth_method 0 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:15.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 10 monclient(hunting): get_auth_request con 0x7fd01806ebc0 auth_method 0 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd015d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.541+0000 7fd016d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:15.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 10 monclient(hunting): handle_auth_done global_id 94419 payload 274 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 -- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 msgr2=0x7fd01806ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 --2- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 0x7fd01806ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 -- 172.21.15.142:0/2901072877 >> v1:172.21.15.150:6789/0 conn(0x7fd0180a3ba0 legacy=0x7fd018130190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 monclient: found mon.noname-a 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 -- 172.21.15.142:0/2901072877 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01806d690 con 0x7fd018134fe0 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 -- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd018134fe0 msgr2=0x7fd0181373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:15.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd016d9d700 1 --2- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd018134fe0 0x7fd0181373c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0100099e0 tx=0x7fd010009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6320ef1881e05482 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 1 -- 172.21.15.142:0/2901072877 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01000ba30 con 0x7fd018134fe0 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: dump: 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:15.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: _finish_auth 0 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 5 monclient: authenticate success, global_id 94419 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 1 -- 172.21.15.142:0/2901072877 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01000bb90 con 0x7fd018134fe0 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 1 -- 172.21.15.142:0/2901072877 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd010011420 con 0x7fd018134fe0 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 4 set_mon_vals no callback set 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01559a700 10 monclient: dump: 2024-09-18T10:18:15.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 10 monclient: shutdown 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 1 -- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd018134fe0 msgr2=0x7fd0181373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:15.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.545+0000 7fd01759e700 1 --2- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd018134fe0 0x7fd0181373c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0100099e0 tx=0x7fd010009250 comp rx=0 tx=0).stop 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- 172.21.15.142:0/2901072877 shutdown_connections 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 --2- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd018134fe0 0x7fd0181373c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 --2- 172.21.15.142:0/2901072877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd01806ebc0 0x7fd01806ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- 172.21.15.142:0/2901072877 >> 172.21.15.142:0/2901072877 conn(0x7fd018071680 msgr2=0x7fd0180a3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- 172.21.15.142:0/2901072877 shutdown_connections 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- 172.21.15.142:0/2901072877 wait complete. 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient: build_initial_monmap 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient: monmap: 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:15.568 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 Processor -- start 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- start start 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient: init 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd01806ebc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0180a3ba0 0x7fd0180699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd0180a3ba0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 0x7fd018069ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd018134fe0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0181318a0 con 0x7fd01806ebc0 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd018131af0 con 0x7fd018134fe0 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0180a0870 con 0x7fd0180a3ba0 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01759e700 10 monclient(hunting): authenticate will time out at 1384.478460s 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd016d9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 msgr2=0x7fd018069ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd016d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 0x7fd018069ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:15.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0180a3ba0 0x7fd0180699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60050/0 (socket says 172.21.15.142:60050) 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 -- 172.21.15.142:0/3966771336 learned_addr learned my addr 172.21.15.142:0/3966771336 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): get_auth_request con 0x7fd01806ebc0 auth_method 0 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): get_auth_request con 0x7fd0180a3ba0 auth_method 0 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:15.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd015d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient(hunting): handle_auth_done global_id 94422 payload 824 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 -- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0180a3ba0 msgr2=0x7fd0180699b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0180a3ba0 0x7fd0180699b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 -- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 msgr2=0x7fd018069ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 0x7fd018069ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 monclient: found mon.noname-a 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.549+0000 7fd01659c700 1 -- 172.21.15.142:0/3966771336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0100094f0 con 0x7fd01806ebc0 2024-09-18T10:18:15.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01659c700 1 -- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 msgr2=0x7fd018099930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01659c700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd00000b840 tx=0x7fd0000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000003dd0 con 0x7fd01806ebc0 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: dump: 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.573 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 5 monclient: authenticate success, global_id 94422 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd000006150 con 0x7fd01806ebc0 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 10 monclient: _renew_subs 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01806f6b0 con 0x7fd01806ebc0 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000006510 con 0x7fd01806ebc0 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:15.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd014d99700 4 set_mon_vals no callback set 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 10 monclient: dump: 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:15.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 10 monclient: _renew_subs 2024-09-18T10:18:15.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01812f800 con 0x7fd01806ebc0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.553+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd00001b3b0 con 0x7fd01806ebc0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd00e7fc700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 0x7fcff8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd01759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd0000483f0 con 0x7fd01806ebc0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd01759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd01759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd018131c80 con 0x7fd01806ebc0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd015d9b700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 0x7fcff8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:15.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd015d9b700 10 monclient: get_auth_request con 0x7fcff8035a70 auth_method 0 2024-09-18T10:18:15.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.557+0000 7fd015d9b700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 0x7fcff8037f20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd008009fd0 tx=0x7fd00800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:15.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.561+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd000017070 con 0x7fd01806ebc0 2024-09-18T10:18:15.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.561+0000 7fd00e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:15.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.561+0000 7fd00e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:15.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.785+0000 7fd01759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:15.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.785+0000 7fd01759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.785+0000 7fd01759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.785+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd018131650 con 0x7fd01806ebc0 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.785+0000 7fd00e7fc700 1 -- 172.21.15.142:0/3966771336 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd00001a520 con 0x7fd01806ebc0 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd00e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd00e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:15.793 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 msgr2=0x7fcff8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 0x7fcff8037f20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd008009fd0 tx=0x7fd00800b040 comp rx=0 tx=0).stop 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 10 monclient: shutdown 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 msgr2=0x7fd018099930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd00000b840 tx=0x7fd0000091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 shutdown_connections 2024-09-18T10:18:15.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcff8035a70 0x7fcff8037f20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd018134fe0 0x7fd018069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0180a3ba0 0x7fd0180699b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 --2- 172.21.15.142:0/3966771336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd01806ebc0 0x7fd018099930 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 >> 172.21.15.142:0/3966771336 conn(0x7fd018071680 msgr2=0x7fd0180a1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 shutdown_connections 2024-09-18T10:18:15.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:15.789+0000 7fd01759e700 1 -- 172.21.15.142:0/3966771336 wait complete. 2024-09-18T10:18:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:16.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:16.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:16.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:17.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:17.344+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:17.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:17.344+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:18.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:18.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:18.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:18.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:18.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:18.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 10 monclient: get_monmap_and_config 2024-09-18T10:18:18.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 10 monclient: build_initial_monmap 2024-09-18T10:18:18.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 10 monclient: monmap: 2024-09-18T10:18:18.920 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:18.920 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:18.918482+0000 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:18.918482+0000 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:18.921 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:18.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 1 Processor -- start 2024-09-18T10:18:18.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 1 -- start start 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.913+0000 7f6ff1468700 10 monclient: init 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-c con 0x7f6fec09aae0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fec0967a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-a con 0x7f6fec09c4f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec09aeb0 con 0x7f6fec0967a0 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec06f6d0 con 0x7f6fec09c4f0 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:18.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fec0955a0 con 0x7f6fec09aae0 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff1468700 10 monclient(hunting): authenticate will time out at 1387.844520s 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6febfff700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 msgr2=0x7f6fec096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6febfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6fe3fff700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f6fec0967a0 legacy=0x7f6fec09bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60058/0 (socket says 172.21.15.142:60058) 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 learned_addr learned my addr 172.21.15.142:0/1973609622 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fec09c4f0 auth_method 0 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:18.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:18:18.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:18.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:18.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:18.917+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6fe3fff700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.150:6789/0 conn(0x7f6fec0967a0 legacy=0x7f6fec09bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 msgr2=0x7f6fec096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fec09c4f0 auth_method 0 2024-09-18T10:18:19.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:19.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.121+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:18:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:19.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:19.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:19.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 msgr2=0x7f6fec096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6fe3fff700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.150:6789/0 conn(0x7f6fec0967a0 legacy=0x7f6fec09bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fec09c4f0 auth_method 0 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:18:19.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:19.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:19.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:19.525+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:18:20.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 msgr2=0x7f6fec096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:20.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6fe3fff700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.150:6789/0 conn(0x7f6fec0967a0 legacy=0x7f6fec09bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:20.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:18:20.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:20.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fec09c4f0 auth_method 0 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:20.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:20.329+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:18:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:20.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:20.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:20.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:21.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:21.344+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:21.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:21.344+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:21.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:21.344+0000 7fde9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:21.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): tick 2024-09-18T10:18:21.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 monclient(hunting): continuing hunt 2024-09-18T10:18:21.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:18:21.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 msgr2=0x7f6fec096260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:21.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.150:3300/0 conn(0x7f6fec09aae0 0x7f6fec096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:21.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 -- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 msgr2=0x7f6fec1375a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:21.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 --2- 172.21.15.142:0/1973609622 >> v2:172.21.15.142:3300/0 conn(0x7f6fec09c4f0 0x7f6fec1375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:21.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.150:6789/0 conn(0x7f6fec0967a0 legacy=0x7f6fec09bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:21.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient: _add_conns ranks=[2,0] 2024-09-18T10:18:21.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): picked mon.noname-b con 0x7f6fcc0036e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fcc006190 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 -- 172.21.15.142:0/1973609622 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec095b60 con 0x7f6fcc006190 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 1 -- 172.21.15.142:0/1973609622 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec06f6d0 con 0x7f6fcc0036e0 2024-09-18T10:18:21.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fea7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 10 monclient(hunting): get_auth_request con 0x7f6fcc0036e0 auth_method 0 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:21.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:21.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:21.917+0000 7f6fe3fff700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.142:6789/0 conn(0x7f6fcc006190 legacy=0x7f6fcc008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 10 monclient(hunting): get_auth_request con 0x7f6fcc0036e0 auth_method 0 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:22.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:22.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.121+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:18:22.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:22.344+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:22.348 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:22.344+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 10 monclient(hunting): get_auth_request con 0x7f6fcc0036e0 auth_method 0 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:18:22.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:22.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:22.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.525+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:18:22.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.745+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498801077 0 0) 0x7f6fec095b60 con 0x7f6fcc006190 2024-09-18T10:18:22.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.745+0000 7f6feb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.745+0000 7f6feb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:22.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.745+0000 7f6feb7fe700 10 monclient(hunting): my global_id is 94425 2024-09-18T10:18:22.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.745+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd8003570 con 0x7f6fcc006190 2024-09-18T10:18:22.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3219356925 0 0) 0x7f6fd8003570 con 0x7f6fcc006190 2024-09-18T10:18:22.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 msgr2=0x7f6fcc005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fcc0036e0 0x7f6fcc005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec09aeb0 con 0x7f6fcc006190 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f6fd4004200 con 0x7f6fcc006190 2024-09-18T10:18:22.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff1468700 5 monclient: authenticate success, global_id 94425 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff1468700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: dump: 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:22.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 >> v1:172.21.15.142:6789/0 conn(0x7f6fcc006190 legacy=0x7f6fcc008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:22.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 1 -- 172.21.15.142:0/1973609622 reap_dead start 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fd8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): picked mon.c con 0x7f6fd8003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): picked mon.b con 0x7f6fd8004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fd800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): picked mon.a con 0x7f6fd8005230 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec06f6d0 con 0x7f6fd8005230 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd800c200 con 0x7f6fd8004810 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd800c3c0 con 0x7f6fd8003a60 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6feb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff1468700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6fe3fff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 msgr2=0x7f6fd8004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6fe3fff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fd8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fd8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): get_auth_request con 0x7f6fd8003a60 auth_method 0 2024-09-18T10:18:22.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fd800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fd8005230 auth_method 0 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:22.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient(hunting): handle_auth_done global_id 94425 payload 293 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 msgr2=0x7f6fd8004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fd8004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 msgr2=0x7f6fd800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fd800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 monclient: found mon.c 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:22.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8003570 con 0x7f6fd8003a60 2024-09-18T10:18:22.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 10 monclient: _finish_auth 0 2024-09-18T10:18:22.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.749+0000 7f6febfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6febfff700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 msgr2=0x7f6fd8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6febfff700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fd8003e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009b70 tx=0x7f6fdc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f132528078e41cd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc016650 con 0x7f6fd8003a60 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:22.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: dump: 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6ff1468700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc0167d0 con 0x7f6fd8003a60 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 1 -- 172.21.15.142:0/1973609622 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc016b90 con 0x7f6fd8003a60 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6ff1468700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6ff1468700 4 set_mon_vals no callback set 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.753+0000 7f6feb7fe700 10 monclient: dump: 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:22.771 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: shutdown 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:22.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 msgr2=0x7f6fd8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fd8003e50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009b70 tx=0x7f6fdc00b040 comp rx=0 tx=0).stop 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- 172.21.15.142:0/1973609622 shutdown_connections 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fd800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fd8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- 172.21.15.142:0/1973609622 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fd8003e50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- 172.21.15.142:0/1973609622 >> 172.21.15.142:0/1973609622 conn(0x7f6fec0a1fe0 msgr2=0x7f6fec070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- 172.21.15.142:0/1973609622 shutdown_connections 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- 172.21.15.142:0/1973609622 wait complete. 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: build_initial_monmap 2024-09-18T10:18:22.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: monmap: 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 Processor -- start 2024-09-18T10:18:22.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- start start 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: init 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd8003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fec12d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-b con 0x7f6fd8004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): picked mon.noname-a con 0x7f6fd8005230 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec127d80 con 0x7f6fd8005230 2024-09-18T10:18:22.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec06f940 con 0x7f6fd8004810 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec099c30 con 0x7f6fd8003a60 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff1468700 10 monclient(hunting): authenticate will time out at 1391.687824s 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6fe3fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 msgr2=0x7f6fec12d890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6febfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.757+0000 7f6ff0c67700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fec12d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:22.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49218/0 (socket says 172.21.15.142:49218) 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 1 -- 172.21.15.142:0/673547285 learned_addr learned my addr 172.21.15.142:0/673547285 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57602/0 (socket says 172.21.15.142:57602) 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request con 0x7f6fd8005230 auth_method 0 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): get_auth_request con 0x7f6fd8003a60 auth_method 0 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:22.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient(hunting): handle_auth_done global_id 74834 payload 824 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 -- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 msgr2=0x7f6fec12d890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fec12d890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 -- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 msgr2=0x7f6fec06d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:22.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 monclient: found mon.noname-c 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 -- 172.21.15.142:0/673547285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec126b40 con 0x7f6fd8003a60 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff0c67700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 -- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 msgr2=0x7f6fec13cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6febfff700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc01e000 tx=0x7f6fdc01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc016b50 con 0x7f6fd8003a60 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:22.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: dump: 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:22.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 5 monclient: authenticate success, global_id 74834 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc01f680 con 0x7f6fd8003a60 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc01fa00 con 0x7f6fd8003a60 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 10 monclient: _renew_subs 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec127aa0 con 0x7f6fd8003a60 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6feaffd700 4 set_mon_vals no callback set 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6fe8ff9700 10 monclient: dump: 2024-09-18T10:18:22.781 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 10 monclient: _renew_subs 2024-09-18T10:18:22.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.761+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec13d040 con 0x7f6fd8003a60 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6fdc01c070 con 0x7f6fd8003a60 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6ff1468700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6ff1468700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6ff1468700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec1281f0 con 0x7f6fd8003a60 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6fe8ff9700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 0x7f6fcc0387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6fdc04e750 con 0x7f6fd8003a60 2024-09-18T10:18:22.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6fe3fff700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 0x7f6fcc0387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:22.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.765+0000 7f6fe3fff700 10 monclient: get_auth_request con 0x7f6fcc036340 auth_method 0 2024-09-18T10:18:22.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.769+0000 7f6fe3fff700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 0x7f6fcc0387f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4000c00 tx=0x7f6fd4004290 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:22.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.769+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6fdc028450 con 0x7f6fd8003a60 2024-09-18T10:18:22.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.769+0000 7f6fe8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:22.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.769+0000 7f6fe8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:22.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.989+0000 7f6ff1468700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:22.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.989+0000 7f6ff1468700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:22.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.989+0000 7f6ff1468700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:22.997 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.989+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6fec072a80 con 0x7f6fd8003a60 2024-09-18T10:18:22.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6fe8ff9700 1 -- 172.21.15.142:0/673547285 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f6fdc02d070 con 0x7f6fd8003a60 2024-09-18T10:18:22.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6fe8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:22.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6fe8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:22.998 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:23.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 msgr2=0x7f6fcc0387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 0x7f6fcc0387f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4000c00 tx=0x7f6fd4004290 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 10 monclient: shutdown 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 msgr2=0x7f6fec13cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc01e000 tx=0x7f6fdc01f000 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 shutdown_connections 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fcc036340 0x7f6fcc0387f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fd8005230 0x7f6fec06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fd8004810 0x7f6fec12d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 --2- 172.21.15.142:0/673547285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fd8003a60 0x7f6fec13cb00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 >> 172.21.15.142:0/673547285 conn(0x7f6fec0a1fe0 msgr2=0x7f6fec09d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 shutdown_connections 2024-09-18T10:18:23.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:22.993+0000 7f6ff1468700 1 -- 172.21.15.142:0/673547285 wait complete. 2024-09-18T10:18:23.012 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:18:23.012 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:18:23.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:18:23.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: build_initial_monmap 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: monmap: 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:23.120118+0000 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:23.120118+0000 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:23.122 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:23.123 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:23.123 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:23.123 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 Processor -- start 2024-09-18T10:18:23.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 -- start start 2024-09-18T10:18:23.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: init 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa63c098140 0x7fa63c0719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa63c098140 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa63c09b5b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 0x7fa63c136610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa63c099060 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63c097d80 con 0x7fa63c09b5b0 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63c09ab60 con 0x7fa63c098140 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c09a880 con 0x7fa63c099060 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63b59e700 10 monclient(hunting): authenticate will time out at 1392.045842s 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa63c098140 0x7fa63c0719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 0x7fa63c136610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa639d9b700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fa63c09b5b0 legacy=0x7fa63c098950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 0x7fa63c136610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57612/0 (socket says 172.21.15.142:57612) 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa63c098140 0x7fa63c0719a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49228/0 (socket says 172.21.15.142:49228) 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 1 -- 172.21.15.142:0/2235158121 learned_addr learned my addr 172.21.15.142:0/2235158121 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): get_auth_request con 0x7fa63c099060 auth_method 0 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): get_auth_request con 0x7fa63c098140 auth_method 0 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient(hunting): handle_auth_done global_id 94431 payload 274 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 -- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 msgr2=0x7fa63c136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 --2- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 0x7fa63c136610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 -- 172.21.15.142:0/2235158121 >> v1:172.21.15.150:6789/0 conn(0x7fa63c09b5b0 legacy=0x7fa63c098950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:23.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:23.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 monclient: found mon.noname-a 2024-09-18T10:18:23.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:23.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.117+0000 7fa63a59c700 1 -- 172.21.15.142:0/2235158121 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c09ae40 con 0x7fa63c098140 2024-09-18T10:18:23.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63a59c700 1 -- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c098140 msgr2=0x7fa63c0719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63a59c700 1 --2- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c098140 0x7fa63c0719a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098e0 tx=0x7fa62c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db43c3860ea78822 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 1 -- 172.21.15.142:0/2235158121 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00b850 con 0x7fa63c098140 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: dump: 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.136 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: _finish_auth 0 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 5 monclient: authenticate success, global_id 94431 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 1 -- 172.21.15.142:0/2235158121 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62c00b9b0 con 0x7fa63c098140 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 1 -- 172.21.15.142:0/2235158121 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00bd50 con 0x7fa63c098140 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 4 set_mon_vals no callback set 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63959a700 10 monclient: dump: 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: shutdown 2024-09-18T10:18:23.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c098140 msgr2=0x7fa63c0719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c098140 0x7fa63c0719a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0098e0 tx=0x7fa62c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- 172.21.15.142:0/2235158121 shutdown_connections 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c099060 0x7fa63c136610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- 172.21.15.142:0/2235158121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c098140 0x7fa63c0719a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- 172.21.15.142:0/2235158121 >> 172.21.15.142:0/2235158121 conn(0x7fa63c0a1ec0 msgr2=0x7fa63c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- 172.21.15.142:0/2235158121 shutdown_connections 2024-09-18T10:18:23.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- 172.21.15.142:0/2235158121 wait complete. 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: build_initial_monmap 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: monmap: 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:23.140 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:23.141 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 Processor -- start 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- start start 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: init 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:18:23.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c098140 0x7fa63c141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa63c098140 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63c099060 0x7fa63c141e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa63c099060 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 0x7fa63c06a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa63c09b5b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:23.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c09fb10 con 0x7fa63c09b5b0 2024-09-18T10:18:23.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0919d0 con 0x7fa63c099060 2024-09-18T10:18:23.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c090e00 con 0x7fa63c098140 2024-09-18T10:18:23.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:23.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63b59e700 10 monclient(hunting): authenticate will time out at 1392.051846s 2024-09-18T10:18:23.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.121+0000 7fa63a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c098140 0x7fa63c141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa639d9b700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63c099060 msgr2=0x7fa63c141e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:23.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa639d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63c099060 0x7fa63c141e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 0x7fa63c06a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c098140 0x7fa63c141920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57626/0 (socket says 172.21.15.142:57626) 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 1 -- 172.21.15.142:0/1950417885 learned_addr learned my addr 172.21.15.142:0/1950417885 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): get_auth_request con 0x7fa63c098140 auth_method 0 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): get_auth_request con 0x7fa63c09b5b0 auth_method 0 2024-09-18T10:18:23.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient(hunting): handle_auth_done global_id 94434 payload 824 2024-09-18T10:18:23.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c098140 msgr2=0x7fa63c141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa63c098140 0x7fa63c141920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63c099060 msgr2=0x7fa63c141e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63c099060 0x7fa63c141e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 monclient: found mon.noname-a 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c009420 con 0x7fa63c09b5b0 2024-09-18T10:18:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 msgr2=0x7fa63c06a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63ad9d700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 0x7fa63c06a3e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa634009e50 tx=0x7fa63400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634009c30 con 0x7fa63c09b5b0 2024-09-18T10:18:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:23.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: dump: 2024-09-18T10:18:23.152 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.152 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.152 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.152 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.153 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:23.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:23.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 5 monclient: authenticate success, global_id 94434 2024-09-18T10:18:23.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63400f040 con 0x7fa63c09b5b0 2024-09-18T10:18:23.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 monclient: _renew_subs 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c091130 con 0x7fa63c09b5b0 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63400cda0 con 0x7fa63c09b5b0 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa6327fc700 10 monclient: dump: 2024-09-18T10:18:23.156 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.156 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.156 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.156 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.157 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 monclient: _renew_subs 2024-09-18T10:18:23.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c0922c0 con 0x7fa63c09b5b0 2024-09-18T10:18:23.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa638d99700 4 set_mon_vals no callback set 2024-09-18T10:18:23.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:23.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 0x7fa63c06acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.125+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0648a0 con 0x7fa63c06a920 2024-09-18T10:18:23.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 0x7fa63c06acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient: get_auth_request con 0x7fa63c06a920 auth_method 0 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 10 monclient(hunting): handle_auth_done global_id 94437 payload 824 2024-09-18T10:18:23.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa63c072a80 con 0x7fa63c06a920 2024-09-18T10:18:23.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa634014650 con 0x7fa63c09b5b0 2024-09-18T10:18:23.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa61c035a70 0x7fa61c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa634049650 con 0x7fa63c09b5b0 2024-09-18T10:18:23.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa639d9b700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa61c035a70 0x7fa61c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa639d9b700 10 monclient: get_auth_request con 0x7fa61c035a70 auth_method 0 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 msgr2=0x7fa63c06acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa63a59c700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 0x7fa63c06acf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0030c0 tx=0x7fa62c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c021070 con 0x7fa63c06a920 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 10 monclient: dump: 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.167 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa62c01c450 con 0x7fa63c06a920 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 msgr2=0x7fa63c06acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 0x7fa63c06acf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0030c0 tx=0x7fa62c016000 comp rx=0 tx=0).stop 2024-09-18T10:18:23.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 msgr2=0x7fa63c06acf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.129+0000 7fa6327fc700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06a920 0x7fa63c06acf0 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0030c0 tx=0x7fa62c016000 comp rx=0 tx=0).stop 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.133+0000 7fa639d9b700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa61c035a70 0x7fa61c037f20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa624006fd0 tx=0x7fa624006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63b59e700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 0x7fa63c087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0912c0 con 0x7fa63c06b230 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 0x7fa63c087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient: get_auth_request con 0x7fa63c06b230 auth_method 0 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 10 monclient(hunting): handle_auth_done global_id 94440 payload 824 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.141+0000 7fa63a59c700 1 -- 172.21.15.142:0/1950417885 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa63c0678a0 con 0x7fa63c06b230 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63a59c700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 msgr2=0x7fa63c087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63a59c700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 0x7fa63c087fe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00a010 tx=0x7fa62c01e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62c01cea0 con 0x7fa63c06b230 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 10 monclient: dump: 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.170 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fa62c031410 con 0x7fa63c06b230 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 msgr2=0x7fa63c087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 0x7fa63c087fe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00a010 tx=0x7fa62c01e070 comp rx=0 tx=0).stop 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 msgr2=0x7fa63c087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:23.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa6327fc700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c06b230 0x7fa63c087fe0 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00a010 tx=0x7fa62c01e070 comp rx=0 tx=0).stop 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 100, 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.172 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.173 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:23.175 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.176 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:23.177 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.178 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.179 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.180 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:23.181 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.182 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.183 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:23.184 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.185 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.185 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.185 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:23.185 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:23.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa61c035a70 msgr2=0x7fa61c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa61c035a70 0x7fa61c037f20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa624006fd0 tx=0x7fa624006e10 comp rx=0 tx=0).stop 2024-09-18T10:18:23.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 10 monclient: shutdown 2024-09-18T10:18:23.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63ad9d700 1 -- 172.21.15.142:0/1950417885 reap_dead start 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 msgr2=0x7fa63c06a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 0x7fa63c06a3e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa634009e50 tx=0x7fa63400b040 comp rx=0 tx=0).stop 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 shutdown_connections 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 --2- 172.21.15.142:0/1950417885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63c09b5b0 0x7fa63c06a3e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 >> 172.21.15.142:0/1950417885 conn(0x7fa63c0a1ec0 msgr2=0x7fa63c0954f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 shutdown_connections 2024-09-18T10:18:23.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.145+0000 7fa63b59e700 1 -- 172.21.15.142:0/1950417885 wait complete. 2024-09-18T10:18:23.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:18:23.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:18:23.188 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: get_monmap_and_config 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: build_initial_monmap 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: monmap: 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:23.300169+0000 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:23.300169+0000 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:23.302 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:23.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 Processor -- start 2024-09-18T10:18:23.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 -- start start 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: init 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fddec06f3b0 0x7fddec0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-a con 0x7fddec06f3b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddec098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 0x7fddec136d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-c con 0x7fddec099320 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec09c330 con 0x7fddec098f00 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec09b030 con 0x7fddec06f3b0 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddec09c530 con 0x7fddec099320 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1919700 10 monclient(hunting): authenticate will time out at 1392.226487s 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddebfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fddec098f00 0x7fddec133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:45202/0 (socket says 172.21.15.142:45202) 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1118700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 msgr2=0x7fddec136d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fddec06f3b0 0x7fddec0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddebfff700 1 -- 172.21.15.142:0/563865846 learned_addr learned my addr 172.21.15.142:0/563865846 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf1118700 1 --2- 172.21.15.142:0/563865846 >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 0x7fddec136d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): get_auth_request con 0x7fddec06f3b0 auth_method 0 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient(hunting): handle_auth_done global_id 94443 payload 274 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 -- 172.21.15.142:0/563865846 >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 msgr2=0x7fddec136d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 --2- 172.21.15.142:0/563865846 >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 0x7fddec136d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 -- 172.21.15.142:0/563865846 >> v1:172.21.15.142:6789/0 conn(0x7fddec098f00 legacy=0x7fddec133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient: _finish_hunting 0 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 monclient: found mon.noname-a 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.297+0000 7fddf0917700 1 -- 172.21.15.142:0/563865846 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec09c0f0 con 0x7fddec06f3b0 2024-09-18T10:18:23.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf0917700 1 -- 172.21.15.142:0/563865846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec06f3b0 msgr2=0x7fddec0989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf0917700 1 --2- 172.21.15.142:0/563865846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec06f3b0 0x7fddec0989c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fddec09a7f0 tx=0x7fdddc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7979106790657943 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 1 -- 172.21.15.142:0/563865846 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc00b9a0 con 0x7fddec06f3b0 2024-09-18T10:18:23.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: dump: 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 5 monclient: authenticate success, global_id 94443 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 1 -- 172.21.15.142:0/563865846 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc00bb20 con 0x7fddec06f3b0 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 4 set_mon_vals no callback set 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 1 -- 172.21.15.142:0/563865846 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc011420 con 0x7fddec06f3b0 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddeb7fe700 10 monclient: dump: 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.314 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 10 monclient: shutdown 2024-09-18T10:18:23.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 -- 172.21.15.142:0/563865846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec06f3b0 msgr2=0x7fddec0989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 --2- 172.21.15.142:0/563865846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec06f3b0 0x7fddec0989c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fddec09a7f0 tx=0x7fdddc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 -- 172.21.15.142:0/563865846 shutdown_connections 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 --2- 172.21.15.142:0/563865846 >> v2:172.21.15.150:3300/0 conn(0x7fddec099320 0x7fddec136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 --2- 172.21.15.142:0/563865846 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec06f3b0 0x7fddec0989c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 -- 172.21.15.142:0/563865846 >> 172.21.15.142:0/563865846 conn(0x7fddec070ef0 msgr2=0x7fddec0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 -- 172.21.15.142:0/563865846 shutdown_connections 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 1 -- 172.21.15.142:0/563865846 wait complete. 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.301+0000 7fddf1919700 10 monclient: build_initial_monmap 2024-09-18T10:18:23.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient: monmap: 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 Processor -- start 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 -- start start 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient: init 2024-09-18T10:18:23.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddec06f3b0 0x7fddec08ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-b con 0x7fddec06f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec098f00 0x7fddec08b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-c con 0x7fddec098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 0x7fddec08b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): picked mon.noname-a con 0x7fddec099320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec08bda0 con 0x7fddec099320 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec06fa20 con 0x7fddec06f3b0 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec097280 con 0x7fddec098f00 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1919700 10 monclient(hunting): authenticate will time out at 1392.233467s 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 0x7fddec08b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf0917700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddec06f3b0 msgr2=0x7fddec08ac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 0x7fddec08b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49272/0 (socket says 172.21.15.142:49272) 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 learned_addr learned my addr 172.21.15.142:0/1568509910 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): get_auth_request con 0x7fddec099320 auth_method 0 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf0917700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddec06f3b0 0x7fddec08ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec098f00 0x7fddec08b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): get_auth_request con 0x7fddec098f00 auth_method 0 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient(hunting): handle_auth_done global_id 94446 payload 824 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec098f00 msgr2=0x7fddec08b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec098f00 0x7fddec08b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddec06f3b0 msgr2=0x7fddec08ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:23.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddec06f3b0 0x7fddec08ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient: _finish_hunting 0 2024-09-18T10:18:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 monclient: found mon.noname-a 2024-09-18T10:18:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.305+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009420 con 0x7fddec099320 2024-09-18T10:18:23.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 msgr2=0x7fddec08b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1118700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 0x7fddec08b680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdde400bf80 tx=0x7fdde4009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde400ce50 con 0x7fddec099320 2024-09-18T10:18:23.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: dump: 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 5 monclient: authenticate success, global_id 94446 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4016430 con 0x7fddec099320 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 monclient: _renew_subs 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec091550 con 0x7fddec099320 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4016620 con 0x7fddec099320 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 10 monclient: dump: 2024-09-18T10:18:23.326 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddeaffd700 4 set_mon_vals no callback set 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 monclient: _renew_subs 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:23.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddec092820 con 0x7fddec099320 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 0x7fddec141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec0648a0 con 0x7fddec09c990 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 0x7fddec141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient: get_auth_request con 0x7fddec09c990 auth_method 0 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdde4012020 con 0x7fddec099320 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 10 monclient(hunting): handle_auth_done global_id 74846 payload 824 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddf0917700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddec072a80 con 0x7fddec09c990 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fddd8035a70 0x7fddd8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(262..262 src has 176..262) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdde4049300 con 0x7fddec099320 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddebfff700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fddd8035a70 0x7fddd8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.309+0000 7fddebfff700 10 monclient: get_auth_request con 0x7fddd8035a70 auth_method 0 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fddf0917700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 msgr2=0x7fddec141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fddf0917700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 0x7fddec141760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fddec098600 tx=0x7fdddc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc00bea0 con 0x7fddec09c990 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:23.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 10 monclient: dump: 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdddc00b9a0 con 0x7fddec09c990 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 msgr2=0x7fddec141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 0x7fddec141760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fddec098600 tx=0x7fdddc016000 comp rx=0 tx=0).stop 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 msgr2=0x7fddec141760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fdde8ff9700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec09c990 0x7fddec141760 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fddec098600 tx=0x7fdddc016000 comp rx=0 tx=0).stop 2024-09-18T10:18:23.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.313+0000 7fddebfff700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fddd8035a70 0x7fddd8037f20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fddec09ce70 tx=0x7fdde0007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf1919700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:23.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf1919700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 0x7fddec142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:23.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf1919700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec1425b0 con 0x7fddec141ca0 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 0x7fddec142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient: get_auth_request con 0x7fddec141ca0 auth_method 0 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 10 monclient(hunting): handle_auth_done global_id 74849 payload 824 2024-09-18T10:18:23.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 1 -- 172.21.15.142:0/1568509910 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fddec0678a0 con 0x7fddec141ca0 2024-09-18T10:18:23.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 msgr2=0x7fddec142070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.325+0000 7fddf0917700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 0x7fddec142070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fddec072200 tx=0x7fdddc01f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc01a650 con 0x7fddec141ca0 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 10 monclient: dump: 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:23.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fdddc01a7b0 con 0x7fddec141ca0 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 msgr2=0x7fddec142070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 0x7fddec142070 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fddec072200 tx=0x7fdddc01f040 comp rx=0 tx=0).stop 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 msgr2=0x7fddec142070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fdde8ff9700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fddec141ca0 0x7fddec142070 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fddec072200 tx=0x7fdddc01f040 comp rx=0 tx=0).stop 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 100, 2024-09-18T10:18:23.338 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.339 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:23.340 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:23.341 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:23.342 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.343 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.346 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:23.348 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:23.349 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:23.350 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:23.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.352 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.353 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:23.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:23.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:23.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:23.355 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:23.355 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:23.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fddd8035a70 msgr2=0x7fddd8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fddd8035a70 0x7fddd8037f20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fddec09ce70 tx=0x7fdde0007f90 comp rx=0 tx=0).stop 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 10 monclient: shutdown 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 msgr2=0x7fddec08b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 --2- 172.21.15.142:0/1568509910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fddec099320 0x7fddec08b680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdde400bf80 tx=0x7fdde4009730 comp rx=0 tx=0).stop 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 reap_dead start 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1118700 1 -- 172.21.15.142:0/1568509910 reap_dead start 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 shutdown_connections 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 >> 172.21.15.142:0/1568509910 conn(0x7fddec070ef0 msgr2=0x7fddec0a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 shutdown_connections 2024-09-18T10:18:23.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:23.329+0000 7fddf1919700 1 -- 172.21.15.142:0/1568509910 wait complete. 2024-09-18T10:18:23.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:18:23.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:18:23.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.340+0000 7fde9bfff700 10 monclient: tick 2024-09-18T10:18:23.358 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.340+0000 7fde9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:23.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.732+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7fdea00283d0 con 0x7fdeb009cf90 2024-09-18T10:18:23.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.732+0000 7fdea57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:23.739 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.732+0000 7fdea57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdeb7861700 10 start_mon_command cmd=[{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdeb7861700 10 monclient: _send_command 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdeb7861700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:23.855 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"} v 0) v1 -- 0x7fdeb00008d0 con 0x7fdeb009cf90 2024-09-18T10:18:23.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdea57fa700 1 -- 172.21.15.150:0/3418983841 <== mon.2 v2:172.21.15.142:3301/0 5 ==== mon_command_ack([{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}]=0 v263) v1 ==== 106+0+3 (secure 0 0 0) 0x7fdea0028600 con 0x7fdeb009cf90 2024-09-18T10:18:23.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdea57fa700 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd create", "uuid": "912b7d9d-08f4-4308-9252-03c9211c6cd6"}] 2024-09-18T10:18:23.857 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.848+0000 7fdea57fa700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:23.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 msgr2=0x7fde94037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 0x7fde94037f70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009ad0 tx=0x7fdea0009060 comp rx=0 tx=0).stop 2024-09-18T10:18:23.859 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 10 monclient: shutdown 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 msgr2=0x7fde940461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fde940461a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdea000edd0 tx=0x7fdea0015480 comp rx=0 tx=0).stop 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 shutdown_connections 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdeb009cf90 0x7fde940461a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdeb00a0de0 0x7fde94043760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde9403ea60 0x7fde94040e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 --2- 172.21.15.150:0/3418983841 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fde94035ac0 0x7fde94037f70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 >> 172.21.15.150:0/3418983841 conn(0x7fdeb00a58a0 msgr2=0x7fdeb009b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 shutdown_connections 2024-09-18T10:18:23.860 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:23.852+0000 7fdeb7861700 1 -- 172.21.15.150:0/3418983841 wait complete. 2024-09-18T10:18:23.869 INFO:tasks.workunit.client.0.smithi150.stderr:+ na2=13 2024-09-18T10:18:23.869 INFO:tasks.workunit.client.0.smithi150.stdout:OK 2024-09-18T10:18:23.870 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo OK 2024-09-18T10:18:23.870 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:18:23.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:18:23.884 INFO:tasks.workunit:Running workunits matching mon/caps.sh on client.0... 2024-09-18T10:18:23.886 INFO:tasks.workunit:Running workunit mon/caps.sh... 2024-09-18T10:18:23.886 DEBUG:teuthology.orchestra.run.smithi150:workunit test mon/caps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh 2024-09-18T10:18:23.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ tmp=/tmp/cephtest-mon-caps-madness 2024-09-18T10:18:23.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ exit_on_error=1 2024-09-18T10:18:23.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ ! -z '' ]] 2024-09-18T10:18:23.896 INFO:tasks.workunit.client.0.smithi150.stderr:++ uname 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ '[' Linux = FreeBSD ']' 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ ETIMEDOUT=110 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 0 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring' 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:18:23.897 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T10:18:23.898 INFO:tasks.workunit.client.0.smithi150.stdout:ceph auth get-or-create client.bazar > /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T10:18:23.898 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph auth get-or-create client.bazar '>' /tmp/cephtest-mon-caps-madness.bazar.keyring 2024-09-18T10:18:24.424 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 13 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status' 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T10:18:24.425 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T10:18:24.426 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bazar.keyring --user bazar quorum_status 2024-09-18T10:18:43.358 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:18:43.359 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:18:43.359 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:18:43.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:18:43.362 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:18:43.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:18:43.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: get_monmap_and_config 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: build_initial_monmap 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: monmap: 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:43.473583+0000 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:43.473583+0000 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 Processor -- start 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 -- start start 2024-09-18T10:18:43.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: init 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5fc09d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5fc0954b0 0x7fd5fc0990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-c con 0x7fd5fc0954b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc134670 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5fc069c70 con 0x7fd5fc09d870 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5fc06f9d0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc0958b0 con 0x7fd5fc134670 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd603d50700 10 monclient(hunting): authenticate will time out at 1412.399058s 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33142/0 (socket says 172.21.15.142:33142) 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 -- 172.21.15.142:0/909606083 learned_addr learned my addr 172.21.15.142:0/909606083 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): get_auth_request con 0x7fd5fc134670 auth_method 0 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6012eb700 1 --2- 172.21.15.142:0/909606083 >> v2:172.21.15.150:3300/0 conn(0x7fd5fc0954b0 0x7fd5fc0990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6012eb700 10 monclient(hunting): get_auth_request con 0x7fd5fc0954b0 auth_method 0 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6012eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6012eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6012eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient(hunting): handle_auth_done global_id 74867 payload 274 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 -- 172.21.15.142:0/909606083 >> v2:172.21.15.150:3300/0 conn(0x7fd5fc0954b0 msgr2=0x7fd5fc0990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 --2- 172.21.15.142:0/909606083 >> v2:172.21.15.150:3300/0 conn(0x7fd5fc0954b0 0x7fd5fc0990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 -- 172.21.15.142:0/909606083 >> v1:172.21.15.150:6789/0 conn(0x7fd5fc09d870 legacy=0x7fd5fc0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient: _finish_hunting 0 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 monclient: found mon.noname-b 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:43.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.468+0000 7fd6022ed700 1 -- 172.21.15.142:0/909606083 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc09dc40 con 0x7fd5fc134670 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd6022ed700 1 -- 172.21.15.142:0/909606083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 msgr2=0x7fd5fc136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd6022ed700 1 --2- 172.21.15.142:0/909606083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009f20 tx=0x7fd5f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73762306d41473b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 1 -- 172.21.15.142:0/909606083 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400e070 con 0x7fd5fc134670 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: dump: 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: _finish_auth 0 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 5 monclient: authenticate success, global_id 74867 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 1 -- 172.21.15.142:0/909606083 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f400bc90 con 0x7fd5fc134670 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 1 -- 172.21.15.142:0/909606083 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f40092e0 con 0x7fd5fc134670 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd600aea700 10 monclient: dump: 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 4 set_mon_vals no callback set 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 10 monclient: shutdown 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 -- 172.21.15.142:0/909606083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 msgr2=0x7fd5fc136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 --2- 172.21.15.142:0/909606083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009f20 tx=0x7fd5f4009840 comp rx=0 tx=0).stop 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 -- 172.21.15.142:0/909606083 shutdown_connections 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 --2- 172.21.15.142:0/909606083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc136a50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 --2- 172.21.15.142:0/909606083 >> v2:172.21.15.150:3300/0 conn(0x7fd5fc0954b0 0x7fd5fc0990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 -- 172.21.15.142:0/909606083 >> 172.21.15.142:0/909606083 conn(0x7fd5fc0a14e0 msgr2=0x7fd5fc0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 -- 172.21.15.142:0/909606083 shutdown_connections 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.472+0000 7fd603d50700 1 -- 172.21.15.142:0/909606083 wait complete. 2024-09-18T10:18:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient: build_initial_monmap 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient: monmap: 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 Processor -- start 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 -- start start 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient: init 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-a con 0x7fd5fc0954b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-b con 0x7fd5fc09d870 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc06e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): picked mon.noname-c con 0x7fd5fc134670 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc097b80 con 0x7fd5fc0954b0 2024-09-18T10:18:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc131890 con 0x7fd5fc09d870 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc129870 con 0x7fd5fc134670 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd603d50700 10 monclient(hunting): authenticate will time out at 1412.406936s 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36930/0 (socket says 172.21.15.142:36930) 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 -- 172.21.15.142:0/319891688 learned_addr learned my addr 172.21.15.142:0/319891688 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): get_auth_request con 0x7fd5fc0954b0 auth_method 0 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): get_auth_request con 0x7fd5fc134670 auth_method 0 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 10 monclient(hunting): get_auth_request con 0x7fd5fc09d870 auth_method 0 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient(hunting): handle_auth_done global_id 94476 payload 824 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 msgr2=0x7fd5fc06e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc06e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 msgr2=0x7fd5fc06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 msgr2=0x7fd5fc06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 msgr2=0x7fd5fc06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6012eb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient: _finish_hunting 0 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 monclient: found mon.noname-a 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd601aec700 1 -- 172.21.15.142:0/319891688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f4009ae0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.476+0000 7fd6022ed700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc06e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd601aec700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 msgr2=0x7fd5fc09c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd601aec700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c5d0 tx=0x7fd5ec007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0117a0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-18T10:18:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 5 monclient: authenticate success, global_id 94476 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec011900 con 0x7fd5fc0954b0 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _renew_subs 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc09d5d0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec011cc0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f37fe700 4 set_mon_vals no callback set 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 10 monclient: dump: 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _renew_subs 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc12f6a0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc0696a0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.480+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd5ec012630 con 0x7fd5fc0954b0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.484+0000 7fd5f1ffb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 0x7fd5e0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.484+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd5ec048cf0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.484+0000 7fd6012eb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 0x7fd5e0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.484+0000 7fd6012eb700 10 monclient: get_auth_request con 0x7fd5e0035ac0 auth_method 0 2024-09-18T10:18:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.484+0000 7fd6012eb700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 0x7fd5e0037f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006fd0 tx=0x7fd5f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.492+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd5ec0108c0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.492+0000 7fd5f1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.492+0000 7fd5f1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd5fc1293b0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd5f1ffb700 1 -- 172.21.15.142:0/319891688 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fd5ec0102e0 con 0x7fd5fc0954b0 2024-09-18T10:18:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd5f1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd5f1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:43.723 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 msgr2=0x7fd5e0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 0x7fd5e0037f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006fd0 tx=0x7fd5f8008040 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 10 monclient: shutdown 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 msgr2=0x7fd5fc09c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c5d0 tx=0x7fd5ec007f60 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 shutdown_connections 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd5e0035ac0 0x7fd5e0037f70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5fc134670 0x7fd5fc06e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5fc09d870 0x7fd5fc06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 --2- 172.21.15.142:0/319891688 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5fc0954b0 0x7fd5fc09c760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.716+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 >> 172.21.15.142:0/319891688 conn(0x7fd5fc0a14e0 msgr2=0x7fd5fc133cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.720+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 shutdown_connections 2024-09-18T10:18:43.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:43.720+0000 7fd603d50700 1 -- 172.21.15.142:0/319891688 wait complete. 2024-09-18T10:18:46.738 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:46.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:46.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:46.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: monmap: 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:46.849714+0000 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:46.849714+0000 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:46.852 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 Processor -- start 2024-09-18T10:18:46.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 -- start start 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: init 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce5409d920 0x7fce5406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-c con 0x7fce5409d920 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce540953c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fce54095800 0x7fce540973c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-a con 0x7fce54095800 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5409b0b0 con 0x7fce540953c0 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5406fba0 con 0x7fce54095800 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:46.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5409a7e0 con 0x7fce5409d920 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce53fff700 10 monclient(hunting): authenticate will time out at 1415.775527s 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fce54095800 0x7fce540973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fce54095800 0x7fce540973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36942/0 (socket says 172.21.15.142:36942) 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 1 -- 172.21.15.142:0/3404281529 learned_addr learned my addr 172.21.15.142:0/3404281529 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): get_auth_request con 0x7fce54095800 auth_method 0 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:46.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce52ffd700 1 --2- 172.21.15.142:0/3404281529 >> v2:172.21.15.150:3300/0 conn(0x7fce5409d920 0x7fce5406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.844+0000 7fce537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce52ffd700 10 monclient(hunting): get_auth_request con 0x7fce5409d920 auth_method 0 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 10 monclient(hunting): handle_auth_done global_id 94482 payload 274 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 -- 172.21.15.142:0/3404281529 >> v2:172.21.15.150:3300/0 conn(0x7fce5409d920 msgr2=0x7fce5406f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 --2- 172.21.15.142:0/3404281529 >> v2:172.21.15.150:3300/0 conn(0x7fce5409d920 0x7fce5406f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 -- 172.21.15.142:0/3404281529 >> v1:172.21.15.150:6789/0 conn(0x7fce540953c0 legacy=0x7fce54096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 monclient: found mon.noname-a 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:46.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 -- 172.21.15.142:0/3404281529 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54097900 con 0x7fce54095800 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 -- 172.21.15.142:0/3404281529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 msgr2=0x7fce540973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce537fe700 1 --2- 172.21.15.142:0/3404281529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce540973c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce54067d40 tx=0x7fce4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c610067773fc53d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 1 -- 172.21.15.142:0/3404281529 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c00ba70 con 0x7fce54095800 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: dump: 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:46.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 5 monclient: authenticate success, global_id 94482 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:46.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 1 -- 172.21.15.142:0/3404281529 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4c00bbd0 con 0x7fce54095800 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 4 set_mon_vals no callback set 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 1 -- 172.21.15.142:0/3404281529 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c010470 con 0x7fce54095800 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce51ffb700 10 monclient: dump: 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:46.866 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 10 monclient: shutdown 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 -- 172.21.15.142:0/3404281529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 msgr2=0x7fce540973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 --2- 172.21.15.142:0/3404281529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce540973c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce54067d40 tx=0x7fce4c009250 comp rx=0 tx=0).stop 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 -- 172.21.15.142:0/3404281529 shutdown_connections 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 --2- 172.21.15.142:0/3404281529 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce540973c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 --2- 172.21.15.142:0/3404281529 >> v2:172.21.15.150:3300/0 conn(0x7fce5409d920 0x7fce5406f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 -- 172.21.15.142:0/3404281529 >> 172.21.15.142:0/3404281529 conn(0x7fce540a1fe0 msgr2=0x7fce54070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:46.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 -- 172.21.15.142:0/3404281529 shutdown_connections 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.848+0000 7fce53fff700 1 -- 172.21.15.142:0/3404281529 wait complete. 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient: monmap: 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:46.868 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 Processor -- start 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 -- start start 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient: init 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce540953c0 0x7fce5412d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-c con 0x7fce540953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-a con 0x7fce54095800 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): picked mon.noname-b con 0x7fce5409d920 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:46.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce54126890 con 0x7fce54095800 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce54128150 con 0x7fce5409d920 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce540a0760 con 0x7fce540953c0 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce53fff700 10 monclient(hunting): authenticate will time out at 1415.782176s 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36946/0 (socket says 172.21.15.142:36946) 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 -- 172.21.15.142:0/3211307299 learned_addr learned my addr 172.21.15.142:0/3211307299 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): get_auth_request con 0x7fce54095800 auth_method 0 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:46.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 10 monclient(hunting): get_auth_request con 0x7fce5409d920 auth_method 0 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce540953c0 0x7fce5412d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 msgr2=0x7fce5413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 msgr2=0x7fce5413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): get_auth_request con 0x7fce540953c0 auth_method 0 2024-09-18T10:18:46.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce537fe700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient(hunting): handle_auth_done global_id 94485 payload 824 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce540953c0 msgr2=0x7fce5412d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:46.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce540953c0 0x7fce5412d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 msgr2=0x7fce5413d460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 monclient: found mon.noname-a 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.852+0000 7fce527fc700 1 -- 172.21.15.142:0/3211307299 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4c0094f0 con 0x7fce54095800 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce527fc700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 msgr2=0x7fce5413cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce527fc700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c950 tx=0x7fce44002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44012840 con 0x7fce54095800 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:46.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: dump: 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: _finish_auth 0 2024-09-18T10:18:46.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 5 monclient: authenticate success, global_id 94485 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 10 monclient: _renew_subs 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce440129c0 con 0x7fce54095800 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5406e9e0 con 0x7fce54095800 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44012d80 con 0x7fce54095800 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce3affd700 10 monclient: dump: 2024-09-18T10:18:46.875 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 10 monclient: _renew_subs 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce54127b90 con 0x7fce54095800 2024-09-18T10:18:46.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.856+0000 7fce517fa700 4 set_mon_vals no callback set 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce53fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce53fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce54072a80 con 0x7fce54095800 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fce4400a5b0 con 0x7fce54095800 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce3affd700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 0x7fce40037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fce440493c0 con 0x7fce54095800 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce52ffd700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 0x7fce40037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.860+0000 7fce52ffd700 10 monclient: get_auth_request con 0x7fce40035ac0 auth_method 0 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.864+0000 7fce52ffd700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 0x7fce40037f70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fce48006fd0 tx=0x7fce48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.868+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fce44022d70 con 0x7fce54095800 2024-09-18T10:18:46.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.868+0000 7fce3affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:46.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:46.868+0000 7fce3affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:47.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce53fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:47.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce53fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce53fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce540678a0 con 0x7fce54095800 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce3affd700 1 -- 172.21.15.142:0/3211307299 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fce44016020 con 0x7fce54095800 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce3affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.092+0000 7fce3affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:47.098 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 msgr2=0x7fce40037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 0x7fce40037f70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fce48006fd0 tx=0x7fce48008040 comp rx=0 tx=0).stop 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 10 monclient: shutdown 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 msgr2=0x7fce5413cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:47.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c950 tx=0x7fce44002760 comp rx=0 tx=0).stop 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 shutdown_connections 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce40035ac0 0x7fce40037f70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5409d920 0x7fce5413d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce54095800 0x7fce5413cf20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 --2- 172.21.15.142:0/3211307299 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce540953c0 0x7fce5412d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 >> 172.21.15.142:0/3211307299 conn(0x7fce540a1fe0 msgr2=0x7fce5409de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 shutdown_connections 2024-09-18T10:18:47.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:47.096+0000 7fce53fff700 1 -- 172.21.15.142:0/3211307299 wait complete. 2024-09-18T10:18:50.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:50.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: monmap: 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:50.224012+0000 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:50.224012+0000 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:50.226 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:50.227 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:50.227 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 Processor -- start 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 -- start start 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: init 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:18:50.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fb8099a60 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f9fb809c500 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9fb809b080 0x7f9fb81399a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f9fb809b080 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb809a6a0 con 0x7f9fb8099a60 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb809acd0 con 0x7f9fb809b080 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb80a15e0 con 0x7f9fb809c500 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb7fff700 10 monclient(hunting): authenticate will time out at 1419.149972s 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9fb809b080 0x7f9fb81399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:55774/0 (socket says 172.21.15.142:55774) 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9fb809b080 0x7f9fb81399a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36948/0 (socket says 172.21.15.142:36948) 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1377712868 learned_addr learned my addr 172.21.15.142:0/1377712868 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 10 monclient(hunting): get_auth_request con 0x7f9fb809c500 auth_method 0 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): get_auth_request con 0x7f9fb809b080 auth_method 0 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 msgr2=0x7f9fb80a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:50.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 msgr2=0x7f9fb80a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 --2- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb67fc700 1 --2- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_done global_id 94494 payload 274 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 -- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 msgr2=0x7f9fb80a0910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 --2- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 -- 172.21.15.142:0/1377712868 >> v1:172.21.15.150:6789/0 conn(0x7f9fb8099a60 legacy=0x7f9fb80a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 monclient: found mon.noname-a 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:50.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.220+0000 7f9fb77fe700 1 -- 172.21.15.142:0/1377712868 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8099e30 con 0x7f9fb809b080 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb77fe700 1 -- 172.21.15.142:0/1377712868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809b080 msgr2=0x7f9fb81399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb77fe700 1 --2- 172.21.15.142:0/1377712868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809b080 0x7f9fb81399a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9fb809ba10 tx=0x7f9fb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba7d9dee3037c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 1 -- 172.21.15.142:0/1377712868 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb000b890 con 0x7f9fb809b080 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: dump: 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.237 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 5 monclient: authenticate success, global_id 94494 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 1 -- 172.21.15.142:0/1377712868 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb000b9f0 con 0x7f9fb809b080 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 4 set_mon_vals no callback set 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 1 -- 172.21.15.142:0/1377712868 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb000bd90 con 0x7f9fb809b080 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb5ffb700 10 monclient: dump: 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.239 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 10 monclient: shutdown 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1377712868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809b080 msgr2=0x7f9fb81399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1377712868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809b080 0x7f9fb81399a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9fb809ba10 tx=0x7f9fb00091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1377712868 shutdown_connections 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1377712868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809b080 0x7f9fb81399a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1377712868 >> v2:172.21.15.150:3300/0 conn(0x7f9fb809c500 0x7f9fb80a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1377712868 >> 172.21.15.142:0/1377712868 conn(0x7f9fb80a5810 msgr2=0x7f9fb80a5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1377712868 shutdown_connections 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1377712868 wait complete. 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.224+0000 7f9fb7fff700 10 monclient: monmap: 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 Processor -- start 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 -- start start 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient: init 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f9fb8099a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f9fb809b080 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 0x7f9fb808e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f9fb809c500 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb808f6e0 con 0x7f9fb809c500 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb808fb80 con 0x7f9fb8099a60 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb8091130 con 0x7f9fb809b080 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb7fff700 10 monclient(hunting): authenticate will time out at 1419.157531s 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 0x7f9fb808e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33186/0 (socket says 172.21.15.142:33186) 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1635905715 learned_addr learned my addr 172.21.15.142:0/1635905715 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): get_auth_request con 0x7f9fb809b080 auth_method 0 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): get_auth_request con 0x7f9fb809c500 auth_method 0 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 10 monclient(hunting): get_auth_request con 0x7f9fb8099a60 auth_method 0 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 msgr2=0x7f9fb808db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 msgr2=0x7f9fb808db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb6ffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient(hunting): handle_auth_done global_id 74876 payload 824 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 msgr2=0x7f9fb808db70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 msgr2=0x7f9fb808e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 0x7f9fb808e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 monclient: found mon.noname-c 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1635905715 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb0009420 con 0x7f9fb809b080 2024-09-18T10:18:50.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.228+0000 7f9fb77fe700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 0x7f9fb808e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb67fc700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 msgr2=0x7f9fb808e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb67fc700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8007e80 tx=0x7f9fa80092f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa800dd70 con 0x7f9fb809b080 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: dump: 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.248 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa800f040 con 0x7f9fb809b080 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 5 monclient: authenticate success, global_id 74876 2024-09-18T10:18:50.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _renew_subs 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb8091410 con 0x7f9fb809b080 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8013640 con 0x7f9fb809b080 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 10 monclient: dump: 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.250 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb4ff9700 4 set_mon_vals no callback set 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _renew_subs 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb8090240 con 0x7f9fb809b080 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9fa8003020 con 0x7f9fb809b080 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb8091740 con 0x7f9fb809b080 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 0x7f9f98037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.232+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9fa8048b40 con 0x7f9fb809b080 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.236+0000 7f9fb6ffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 0x7f9f98037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.236+0000 7f9fb6ffd700 10 monclient: get_auth_request con 0x7f9f98035ac0 auth_method 0 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.236+0000 7f9fb6ffd700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 0x7f9f98037f70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0007cb0 tx=0x7f9fa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.244+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9fa8002880 con 0x7f9fb809b080 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.244+0000 7f9faeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.244+0000 7f9faeffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9fb7fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9fb7fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9fb7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9fb8072a80 con 0x7f9fb809b080 2024-09-18T10:18:50.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9faeffd700 1 -- 172.21.15.142:0/1635905715 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f9fa8017070 con 0x7f9fb809b080 2024-09-18T10:18:50.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9faeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:50.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.476+0000 7f9faeffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:50.484 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 msgr2=0x7f9f98037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 0x7f9f98037f70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0007cb0 tx=0x7f9fa0009040 comp rx=0 tx=0).stop 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 10 monclient: shutdown 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 msgr2=0x7f9fb808e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8007e80 tx=0x7f9fa80092f0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 shutdown_connections 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f98035ac0 0x7f9f98037f70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9fb809c500 0x7f9fb808e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9fb809b080 0x7f9fb808e0b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 --2- 172.21.15.142:0/1635905715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fb8099a60 0x7f9fb808db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 >> 172.21.15.142:0/1635905715 conn(0x7f9fb80a5810 msgr2=0x7f9fb809e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 shutdown_connections 2024-09-18T10:18:50.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.480+0000 7f9fb7fff700 1 -- 172.21.15.142:0/1635905715 wait complete. 2024-09-18T10:18:50.503 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:18:50.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:18:50.503 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 10 monclient: get_monmap_and_config 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 10 monclient: monmap: 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:50.613152+0000 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:50.613152+0000 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:50.615 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 1 Processor -- start 2024-09-18T10:18:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 1 -- start start 2024-09-18T10:18:50.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.608+0000 7f4e1147f700 10 monclient: init 2024-09-18T10:18:50.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:18:50.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e0c09c690 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:50.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c09ca60 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e0c133930 0x7f4e0c135d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c133930 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c09df40 con 0x7f4e0c09c690 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0c098df0 con 0x7f4e0c133930 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0c097890 con 0x7f4e0c09ca60 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient(hunting): authenticate will time out at 1419.540369s 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e0c133930 0x7f4e0c135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4e0c133930 0x7f4e0c135d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36976/0 (socket says 172.21.15.142:36976) 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 -- 172.21.15.142:0/1638967449 learned_addr learned my addr 172.21.15.142:0/1638967449 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 10 monclient(hunting): get_auth_request con 0x7f4e0c09ca60 auth_method 0 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): get_auth_request con 0x7f4e0c133930 auth_method 0 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 1 -- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 msgr2=0x7f4e0c09cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 1 -- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 msgr2=0x7f4e0c09cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_done global_id 94500 payload 274 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 -- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 msgr2=0x7f4e0c09cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 --2- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 -- 172.21.15.142:0/1638967449 >> v1:172.21.15.150:6789/0 conn(0x7f4e0c09c690 legacy=0x7f4e0c0986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 monclient: found mon.noname-a 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 -- 172.21.15.142:0/1638967449 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c09d3e0 con 0x7f4e0c133930 2024-09-18T10:18:50.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 -- 172.21.15.142:0/1638967449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c133930 msgr2=0x7f4e0c135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e10c7e700 1 --2- 172.21.15.142:0/1638967449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c133930 0x7f4e0c135d10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c09dc10 tx=0x7f4e04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2853d17a90357c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 1 -- 172.21.15.142:0/1638967449 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400e070 con 0x7f4e0c133930 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: dump: 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 5 monclient: authenticate success, global_id 94500 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 1 -- 172.21.15.142:0/1638967449 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0400bc60 con 0x7f4e0c133930 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e1147f700 4 set_mon_vals no callback set 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 1 -- 172.21.15.142:0/1638967449 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e040092e0 con 0x7f4e0c133930 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.612+0000 7f4e0affd700 10 monclient: dump: 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.631 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: shutdown 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- 172.21.15.142:0/1638967449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c133930 msgr2=0x7f4e0c135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- 172.21.15.142:0/1638967449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c133930 0x7f4e0c135d10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c09dc10 tx=0x7f4e04009840 comp rx=0 tx=0).stop 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- 172.21.15.142:0/1638967449 shutdown_connections 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- 172.21.15.142:0/1638967449 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c133930 0x7f4e0c135d10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- 172.21.15.142:0/1638967449 >> v2:172.21.15.150:3300/0 conn(0x7f4e0c09ca60 0x7f4e0c09cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- 172.21.15.142:0/1638967449 >> 172.21.15.142:0/1638967449 conn(0x7f4e0c0714d0 msgr2=0x7f4e0c0a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- 172.21.15.142:0/1638967449 shutdown_connections 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- 172.21.15.142:0/1638967449 wait complete. 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: monmap: 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 Processor -- start 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- start start 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: init 2024-09-18T10:18:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c09c690 0x7f4e0c13f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c09c690 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 0x7f4e0c130990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c09ca60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c133930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c096c50 con 0x7f4e0c09c690 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c11dd90 con 0x7f4e0c09ca60 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c0a1e40 con 0x7f4e0c133930 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e1147f700 10 monclient(hunting): authenticate will time out at 1419.547158s 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c09c690 0x7f4e0c13f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33188/0 (socket says 172.21.15.142:33188) 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 0x7f4e0c130990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 learned_addr learned my addr 172.21.15.142:0/2222080487 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): get_auth_request con 0x7f4e0c133930 auth_method 0 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c09c690 0x7f4e0c13f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36982/0 (socket says 172.21.15.142:36982) 2024-09-18T10:18:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): get_auth_request con 0x7f4e0c09c690 auth_method 0 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 10 monclient(hunting): get_auth_request con 0x7f4e0c09ca60 auth_method 0 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 msgr2=0x7f4e0c130990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 msgr2=0x7f4e0c130990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 0x7f4e0c130990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 0x7f4e0c130990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient(hunting): handle_auth_done global_id 74879 payload 824 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 msgr2=0x7f4e0c130990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e0c09ca60 0x7f4e0c130990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c09c690 msgr2=0x7f4e0c13f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c09c690 0x7f4e0c13f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 monclient: found mon.noname-c 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.616+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04009ae0 con 0x7f4e0c133930 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 msgr2=0x7f4e0c11a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e10c7e700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e04003400 tx=0x7f4e04017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04012b60 con 0x7f4e0c133930 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: dump: 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04012cc0 con 0x7f4e0c133930 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 5 monclient: authenticate success, global_id 74879 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 monclient: _renew_subs 2024-09-18T10:18:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c11cb50 con 0x7f4e0c133930 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0401d670 con 0x7f4e0c133930 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4df3fff700 10 monclient: dump: 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e0a7fc700 4 set_mon_vals no callback set 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 monclient: _renew_subs 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c11bfa0 con 0x7f4e0c133930 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 0x7f4e0c11ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.620+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c064680 con 0x7f4e0c130ed0 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 0x7f4e0c11ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient: get_auth_request con 0x7f4e0c130ed0 auth_method 0 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4e0401daa0 con 0x7f4e0c133930 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_done global_id 94506 payload 824 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e0c11e100 con 0x7f4e0c130ed0 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4df8035ac0 0x7f4df8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4e0404a610 con 0x7f4e0c133930 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4df8035ac0 0x7f4df8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0b7fe700 10 monclient: get_auth_request con 0x7f4df8035ac0 auth_method 0 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 msgr2=0x7f4e0c11ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0bfff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 0x7f4e0c11ac40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00f7b0 tx=0x7f4dfc007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc014890 con 0x7f4e0c130ed0 2024-09-18T10:18:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 10 monclient: dump: 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4dfc0149f0 con 0x7f4e0c130ed0 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 msgr2=0x7f4e0c11ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 0x7f4e0c11ac40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00f7b0 tx=0x7f4dfc007ea0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 msgr2=0x7f4e0c11ac40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4df3fff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c130ed0 0x7f4e0c11ac40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.624+0000 7f4e0b7fe700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4df8035ac0 0x7f4df8037f70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000a210 tx=0x7f4e00008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e1147f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e1147f700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 0x7f4e0c06ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e1147f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c11b580 con 0x7f4e0c11b180 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 0x7f4e0c06ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient: get_auth_request con 0x7f4e0c11b180 auth_method 0 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 10 monclient(hunting): handle_auth_done global_id 94509 payload 824 2024-09-18T10:18:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.636+0000 7f4e0bfff700 1 -- 172.21.15.142:0/2222080487 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e0c1312a0 con 0x7f4e0c11b180 2024-09-18T10:18:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e0bfff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 msgr2=0x7f4e0c06ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e0bfff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 0x7f4e0c06ab40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00f7b0 tx=0x7f4dfc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc013aa0 con 0x7f4e0c11b180 2024-09-18T10:18:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 10 monclient: dump: 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f4dfc024070 con 0x7f4e0c11b180 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 msgr2=0x7f4e0c06ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 0x7f4e0c06ab40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00f7b0 tx=0x7f4dfc019000 comp rx=0 tx=0).stop 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 msgr2=0x7f4e0c06ab40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4df3fff700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e0c11b180 0x7f4e0c06ab40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.652 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:50.653 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:50.654 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:50.660 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:50.667 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:50.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4df8035ac0 msgr2=0x7f4df8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4df8035ac0 0x7f4df8037f70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4e0000a210 tx=0x7f4e00008040 comp rx=0 tx=0).stop 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 10 monclient: shutdown 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 msgr2=0x7f4e0c11a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e10c7e700 1 -- 172.21.15.142:0/2222080487 reap_dead start 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e04003400 tx=0x7f4e04017000 comp rx=0 tx=0).stop 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 shutdown_connections 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 --2- 172.21.15.142:0/2222080487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4e0c133930 0x7f4e0c11a700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 >> 172.21.15.142:0/2222080487 conn(0x7f4e0c0714d0 msgr2=0x7f4e0c0a05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 shutdown_connections 2024-09-18T10:18:50.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.640+0000 7f4e1147f700 1 -- 172.21.15.142:0/2222080487 wait complete. 2024-09-18T10:18:50.670 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:50.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:18:50.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: get_monmap_and_config 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: monmap: 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:50.782212+0000 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:50.782212+0000 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 Processor -- start 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 -- start start 2024-09-18T10:18:50.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: init 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f016006f3b0 0x7f01600989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): picked mon.noname-c con 0x7f016006f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 0x7f0160099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): picked mon.noname-a con 0x7f0160098f00 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:50.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): picked mon.noname-b con 0x7f0160133660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0160099860 con 0x7f0160098f00 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016009b030 con 0x7f016006f3b0 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016009c330 con 0x7f0160133660 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0165615700 10 monclient(hunting): authenticate will time out at 1419.707663s 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33200/0 (socket says 172.21.15.142:33200) 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 1 -- 172.21.15.142:0/681370685 learned_addr learned my addr 172.21.15.142:0/681370685 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 10 monclient(hunting): get_auth_request con 0x7f0160133660 auth_method 0 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.776+0000 7f0164e14700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 0x7f0160099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015ffff700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.150:3300/0 conn(0x7f016006f3b0 0x7f01600989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 10 monclient(hunting): get_auth_request con 0x7f0160098f00 auth_method 0 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015ffff700 10 monclient(hunting): get_auth_request con 0x7f016006f3b0 auth_method 0 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient(hunting): handle_auth_done global_id 74882 payload 274 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 -- 172.21.15.142:0/681370685 >> v2:172.21.15.150:3300/0 conn(0x7f016006f3b0 msgr2=0x7f01600989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.150:3300/0 conn(0x7f016006f3b0 0x7f01600989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 -- 172.21.15.142:0/681370685 >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 msgr2=0x7f0160099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 0x7f0160099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 monclient: found mon.noname-b 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 -- 172.21.15.142:0/681370685 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016009c530 con 0x7f0160133660 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015f7fe700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 0x7f0160099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 -- 172.21.15.142:0/681370685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 msgr2=0x7f0160135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0164e14700 1 --2- 172.21.15.142:0/681370685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0160076de0 tx=0x7f015000ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4a371eb1ab9b51 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 1 -- 172.21.15.142:0/681370685 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150010070 con 0x7f0160133660 2024-09-18T10:18:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: dump: 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 1 -- 172.21.15.142:0/681370685 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015000de90 con 0x7f0160133660 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 5 monclient: authenticate success, global_id 74882 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 1 -- 172.21.15.142:0/681370685 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150009610 con 0x7f0160133660 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 4 set_mon_vals no callback set 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f015effd700 10 monclient: dump: 2024-09-18T10:18:50.799 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 10 monclient: shutdown 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 -- 172.21.15.142:0/681370685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 msgr2=0x7f0160135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 --2- 172.21.15.142:0/681370685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0160076de0 tx=0x7f015000ba50 comp rx=0 tx=0).stop 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 -- 172.21.15.142:0/681370685 shutdown_connections 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 --2- 172.21.15.142:0/681370685 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160133660 0x7f0160135a40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.142:3300/0 conn(0x7f0160098f00 0x7f0160099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 --2- 172.21.15.142:0/681370685 >> v2:172.21.15.150:3300/0 conn(0x7f016006f3b0 0x7f01600989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 -- 172.21.15.142:0/681370685 >> 172.21.15.142:0/681370685 conn(0x7f0160070ef0 msgr2=0x7f01600712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 -- 172.21.15.142:0/681370685 shutdown_connections 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.780+0000 7f0165615700 1 -- 172.21.15.142:0/681370685 wait complete. 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient: monmap: 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 Processor -- start 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 -- start start 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient: init 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 0x7f016013ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): picked mon.noname-b con 0x7f016006f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 0x7f0160091630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): picked mon.noname-c con 0x7f0160098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0160133660 0x7f0160091b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): picked mon.noname-a con 0x7f0160133660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016006fa90 con 0x7f0160133660 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160132710 con 0x7f016006f3b0 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016006ec00 con 0x7f0160098f00 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0165615700 10 monclient(hunting): authenticate will time out at 1419.714286s 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 0x7f0160091630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0160133660 0x7f0160091b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 0x7f0160091630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33202/0 (socket says 172.21.15.142:33202) 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 -- 172.21.15.142:0/3569154223 learned_addr learned my addr 172.21.15.142:0/3569154223 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): get_auth_request con 0x7f0160098f00 auth_method 0 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): get_auth_request con 0x7f0160133660 auth_method 0 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 0x7f016013ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 10 monclient(hunting): get_auth_request con 0x7f016006f3b0 auth_method 0 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 msgr2=0x7f016013ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 msgr2=0x7f016013ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 0x7f016013ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient(hunting): handle_auth_done global_id 74885 payload 824 2024-09-18T10:18:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 0x7f016013ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 msgr2=0x7f016013ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f016006f3b0 0x7f016013ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0160133660 msgr2=0x7f0160091b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0160133660 0x7f0160091b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f015f7fe700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015000bcf0 con 0x7f0160098f00 2024-09-18T10:18:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.784+0000 7f0164e14700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0160133660 0x7f0160091b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015f7fe700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 msgr2=0x7f0160091630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015f7fe700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 0x7f0160091630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f014800e8d0 tx=0x7f0148008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148013620 con 0x7f0160098f00 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: dump: 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0148013780 con 0x7f0160098f00 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 5 monclient: authenticate success, global_id 74885 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 monclient: _renew_subs 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01600887f0 con 0x7f0160098f00 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148013b20 con 0x7f0160098f00 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 10 monclient: dump: 2024-09-18T10:18:50.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015e7fc700 4 set_mon_vals no callback set 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 monclient: _renew_subs 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016013dc70 con 0x7f0160098f00 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 0x7f016006c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01600648a0 con 0x7f016006c470 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 0x7f016006c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient: get_auth_request con 0x7f016006c470 auth_method 0 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 10 monclient(hunting): handle_auth_done global_id 74888 payload 824 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0160088f40 con 0x7f016006c470 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.788+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f014800c3a0 con 0x7f0160098f00 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0144035b10 0x7f0144037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0148048c90 con 0x7f0160098f00 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0164e14700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0144035b10 0x7f0144037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0164e14700 10 monclient: get_auth_request con 0x7f0144035b10 auth_method 0 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 msgr2=0x7f016006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 0x7f016006c840 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f01580071b0 tx=0x7f0158008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158014880 con 0x7f016006c470 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 10 monclient: dump: 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f01580149e0 con 0x7f016006c470 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 msgr2=0x7f016006c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 0x7f016006c840 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f01580071b0 tx=0x7f0158008040 comp rx=0 tx=0).stop 2024-09-18T10:18:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 msgr2=0x7f016006c840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0157fff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006c470 0x7f016006c840 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f01580071b0 tx=0x7f0158008040 comp rx=0 tx=0).stop 2024-09-18T10:18:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.792+0000 7f0164e14700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0144035b10 0x7f0144037fc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f016009a830 tx=0x7f015001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f0165615700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f0165615700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 0x7f0160093460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f0165615700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016006e0f0 con 0x7f016006cd80 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 0x7f0160093460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient: get_auth_request con 0x7f016006cd80 auth_method 0 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 10 monclient(hunting): handle_auth_done global_id 74891 payload 824 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.804+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0160072a80 con 0x7f016006cd80 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f015ffff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 msgr2=0x7f0160093460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f015ffff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 0x7f0160093460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0158019000 tx=0x7f015801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158020830 con 0x7f016006cd80 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 10 monclient: dump: 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f0158020990 con 0x7f016006cd80 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 msgr2=0x7f0160093460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 0x7f0160093460 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0158019000 tx=0x7f015801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 msgr2=0x7f0160093460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0157fff700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f016006cd80 0x7f0160093460 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0158019000 tx=0x7f015801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:50.820 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.821 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.822 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:50.823 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:50.824 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:50.825 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.826 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:50.828 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.830 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.831 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:50.832 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:50.833 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:50.833 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:50.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:50.833 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:50.833 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:50.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0144035b10 msgr2=0x7f0144037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0144035b10 0x7f0144037fc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f016009a830 tx=0x7f015001a000 comp rx=0 tx=0).stop 2024-09-18T10:18:50.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 10 monclient: shutdown 2024-09-18T10:18:50.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 msgr2=0x7f0160091630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 --2- 172.21.15.142:0/3569154223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0160098f00 0x7f0160091630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f014800e8d0 tx=0x7f0148008040 comp rx=0 tx=0).stop 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0164e14700 1 -- 172.21.15.142:0/3569154223 reap_dead start 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0164e14700 1 -- 172.21.15.142:0/3569154223 reap_dead start 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 shutdown_connections 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 >> 172.21.15.142:0/3569154223 conn(0x7f0160070ef0 msgr2=0x7f01600920f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 shutdown_connections 2024-09-18T10:18:50.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.808+0000 7f0165615700 1 -- 172.21.15.142:0/3569154223 wait complete. 2024-09-18T10:18:50.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:50.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:18:50.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:18:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:18:50.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: monmap: 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:50.948895+0000 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:50.948895+0000 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:50.951 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 Processor -- start 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 -- start start 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: init 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd1009c760 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fdd1009cb30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 0x7fdd10135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fdd10133870 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd10098b00 con 0x7fdd1009c760 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd10098d20 con 0x7fdd10133870 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1009a100 con 0x7fdd1009cb30 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0f59e700 10 monclient(hunting): authenticate will time out at 1419.875038s 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 0x7fdd10135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33224/0 (socket says 172.21.15.142:33224) 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 0x7fdd10135c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37036/0 (socket says 172.21.15.142:37036) 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 -- 172.21.15.142:0/989509960 learned_addr learned my addr 172.21.15.142:0/989509960 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): get_auth_request con 0x7fdd1009cb30 auth_method 0 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): get_auth_request con 0x7fdd10133870 auth_method 0 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_done global_id 74894 payload 274 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 -- 172.21.15.142:0/989509960 >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 msgr2=0x7fdd10135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 --2- 172.21.15.142:0/989509960 >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 0x7fdd10135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 -- 172.21.15.142:0/989509960 >> v1:172.21.15.150:6789/0 conn(0x7fdd1009c760 legacy=0x7fdd1009b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 monclient: found mon.noname-b 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.944+0000 7fdd05d9b700 1 -- 172.21.15.142:0/989509960 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1009a3e0 con 0x7fdd1009cb30 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd05d9b700 1 -- 172.21.15.142:0/989509960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 msgr2=0x7fdd1009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd05d9b700 1 --2- 172.21.15.142:0/989509960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80099b0 tx=0x7fdcf8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbfed3a6bdfb77e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 1 -- 172.21.15.142:0/989509960 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8009ea0 con 0x7fdd1009cb30 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: dump: 2024-09-18T10:18:50.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 5 monclient: authenticate success, global_id 74894 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:50.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 1 -- 172.21.15.142:0/989509960 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf800ba70 con 0x7fdd1009cb30 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 4 set_mon_vals no callback set 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 1 -- 172.21.15.142:0/989509960 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf800be30 con 0x7fdd1009cb30 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0dd9b700 10 monclient: dump: 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.965 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 10 monclient: shutdown 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 -- 172.21.15.142:0/989509960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 msgr2=0x7fdd1009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/989509960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdcf80099b0 tx=0x7fdcf8009250 comp rx=0 tx=0).stop 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 -- 172.21.15.142:0/989509960 shutdown_connections 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/989509960 >> v2:172.21.15.142:3300/0 conn(0x7fdd10133870 0x7fdd10135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/989509960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd1009cb30 0x7fdd1009cf60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 -- 172.21.15.142:0/989509960 >> 172.21.15.142:0/989509960 conn(0x7fdd10070ef0 msgr2=0x7fdd100712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 -- 172.21.15.142:0/989509960 shutdown_connections 2024-09-18T10:18:50.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.948+0000 7fdd0f59e700 1 -- 172.21.15.142:0/989509960 wait complete. 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient: build_initial_monmap 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient: monmap: 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 Processor -- start 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 -- start start 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient: init 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 0x7fdd1006d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fdd1009c760 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 0x7fdd1006de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fdd1009cb30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 0x7fdd1006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fdd10133870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1012b120 con 0x7fdd1009cb30 2024-09-18T10:18:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1006f810 con 0x7fdd1009c760 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100977e0 con 0x7fdd10133870 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0f59e700 10 monclient(hunting): authenticate will time out at 1419.882815s 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 0x7fdd1006de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 0x7fdd1006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 0x7fdd1006de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:37046/0 (socket says 172.21.15.142:37046) 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 1 -- 172.21.15.142:0/3402521726 learned_addr learned my addr 172.21.15.142:0/3402521726 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): get_auth_request con 0x7fdd1009cb30 auth_method 0 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): get_auth_request con 0x7fdd10133870 auth_method 0 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 0x7fdd1006d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request con 0x7fdd1009c760 auth_method 0 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:50.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 msgr2=0x7fdd1006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 msgr2=0x7fdd1006d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 0x7fdd1006d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 0x7fdd1006d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient(hunting): handle_auth_done global_id 74897 payload 824 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 msgr2=0x7fdd1006d940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1009c760 0x7fdd1006d940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 msgr2=0x7fdd1006de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 0x7fdd1006de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:50.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80094f0 con 0x7fdd10133870 2024-09-18T10:18:50.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.952+0000 7fdd05d9b700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdd1009cb30 0x7fdd1006de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:50.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 msgr2=0x7fdd1006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:50.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0ed9d700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 0x7fdd1006e3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009eb0 tx=0x7fdd0800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:50.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800ce30 con 0x7fdd10133870 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: dump: 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.975 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 5 monclient: authenticate success, global_id 74897 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0800f040 con 0x7fdd10133870 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 monclient: _renew_subs 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1012a9c0 con 0x7fdd10133870 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd080146d0 con 0x7fdd10133870 2024-09-18T10:18:50.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0cd99700 4 set_mon_vals no callback set 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd06ffd700 10 monclient: dump: 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:50.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 monclient: _renew_subs 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1012b4f0 con 0x7fdd10133870 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.956+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1012c580 con 0x7fdd1006e900 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 10 monclient: get_auth_request con 0x7fdd1006e900 auth_method 0 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:50.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdd08014a40 con 0x7fdd10133870 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd06ffd700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcf0035ac0 0x7fdcf0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdd08049c10 con 0x7fdd10133870 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd05d9b700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcf0035ac0 0x7fdcf0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.960+0000 7fdd05d9b700 10 monclient: get_auth_request con 0x7fdcf0035ac0 auth_method 0 2024-09-18T10:18:50.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:50.964+0000 7fdd05d9b700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcf0035ac0 0x7fdcf0037f70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdd1012ad50 tx=0x7fdcf8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 10 monclient: get_auth_request con 0x7fdd1006e900 auth_method 0 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:18:51.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:18:51.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.164+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:18:51.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:18:51.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:18:51.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:51.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph auth del client.bazar 2024-09-18T10:18:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient: get_auth_request con 0x7fdd1006e900 auth_method 0 2024-09-18T10:18:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:18:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_done global_id 114103 payload 824 2024-09-18T10:18:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd10072a80 con 0x7fdd1006e900 2024-09-18T10:18:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.568+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdd00012490 tx=0x7fdd00009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00016860 con 0x7fdd1006e900 2024-09-18T10:18:51.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:51.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 10 monclient: dump: 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdd000169c0 con 0x7fdd1006e900 2024-09-18T10:18:51.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:51.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdd00012490 tx=0x7fdd00009040 comp rx=0 tx=0).stop 2024-09-18T10:18:51.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 msgr2=0x7fdd10131a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:51.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.572+0000 7fdd06ffd700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd1006e900 0x7fdd10131a80 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdd00012490 tx=0x7fdd00009040 comp rx=0 tx=0).stop 2024-09-18T10:18:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 0x7fdd101323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100648a0 con 0x7fdd10131fc0 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 0x7fdd101323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient: get_auth_request con 0x7fdd10131fc0 auth_method 0 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.584+0000 7fdd0e59c700 10 monclient(hunting): handle_auth_done global_id 114106 payload 824 2024-09-18T10:18:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdd100a02f0 con 0x7fdd10131fc0 2024-09-18T10:18:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd0e59c700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 msgr2=0x7fdd101323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd0e59c700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 0x7fdd101323c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd0001b000 tx=0x7fdd0001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00006e50 con 0x7fdd10131fc0 2024-09-18T10:18:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 10 monclient: dump: 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.598 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fdd00016650 con 0x7fdd10131fc0 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 msgr2=0x7fdd101323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 0x7fdd101323c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd0001b000 tx=0x7fdd0001e000 comp rx=0 tx=0).stop 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 msgr2=0x7fdd101323c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:51.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.588+0000 7fdd06ffd700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10131fc0 0x7fdd101323c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:51.612 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.613 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.614 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:51.615 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:51.615 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:51.615 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:51.615 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:51.615 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.616 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:51.617 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:51.617 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:51.617 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.617 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:51.617 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:51.618 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:51.618 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:51.618 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:51.618 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:51.618 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:51.619 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:51.619 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:51.619 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:51.619 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:51.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcf0035ac0 msgr2=0x7fdcf0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcf0035ac0 0x7fdcf0037f70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdd1012ad50 tx=0x7fdcf8016000 comp rx=0 tx=0).stop 2024-09-18T10:18:51.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 10 monclient: shutdown 2024-09-18T10:18:51.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 msgr2=0x7fdd1006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 --2- 172.21.15.142:0/3402521726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdd10133870 0x7fdd1006e3c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009eb0 tx=0x7fdd0800b040 comp rx=0 tx=0).stop 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 reap_dead start 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0ed9d700 1 -- 172.21.15.142:0/3402521726 reap_dead start 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 shutdown_connections 2024-09-18T10:18:51.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 >> 172.21.15.142:0/3402521726 conn(0x7fdd10070ef0 msgr2=0x7fdd1009f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:51.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 shutdown_connections 2024-09-18T10:18:51.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.592+0000 7fdd0f59e700 1 -- 172.21.15.142:0/3402521726 wait complete. 2024-09-18T10:18:51.624 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:51.625 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:18:51.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.636+0000 7fa96e8f2700 10 monclient: get_monmap_and_config 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.636+0000 7fa96e8f2700 10 monclient: build_initial_monmap 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.636+0000 7fa96e8f2700 10 monclient: monmap: 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.645 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:18:51.643423+0000 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:18:51.643423+0000 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:51.646 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:51.647 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.652 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 Processor -- start 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 -- start start 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient: init 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809f8b0 0x7fa96809a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-b con 0x7fa96809f8b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:51.653 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa96809df10 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa96809e9f0 0x7fa9681396d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-c con 0x7fa96809e9f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96806a810 con 0x7fa96809df10 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa968068200 con 0x7fa96809e9f0 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96809a6d0 con 0x7fa96809f8b0 2024-09-18T10:18:51.654 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96e8f2700 10 monclient(hunting): authenticate will time out at 1413.738268s 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa96809e9f0 0x7fa9681396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa96809e9f0 0x7fa9681396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34384/0 (socket says 172.21.15.150:34384) 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 -- 172.21.15.150:0/3049404139 learned_addr learned my addr 172.21.15.150:0/3049404139 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): get_auth_request con 0x7fa96809e9f0 auth_method 0 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.655 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809f8b0 0x7fa96809a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): get_auth_request con 0x7fa96809f8b0 auth_method 0 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.656 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_done global_id 114109 payload 274 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 -- 172.21.15.150:0/3049404139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809f8b0 msgr2=0x7fa96809a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809f8b0 0x7fa96809a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 -- 172.21.15.150:0/3049404139 >> v1:172.21.15.142:6789/0 conn(0x7fa96809df10 legacy=0x7fa96809e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient: _finish_hunting 0 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 monclient: found mon.noname-c 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.640+0000 7fa96ce8f700 1 -- 172.21.15.150:0/3049404139 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680687c0 con 0x7fa96809e9f0 2024-09-18T10:18:51.657 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96ce8f700 1 -- 172.21.15.150:0/3049404139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809e9f0 msgr2=0x7fa9681396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96ce8f700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809e9f0 0x7fa9681396d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa960009f20 tx=0x7fa960009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6094e8e89e3bf805 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 1 -- 172.21.15.150:0/3049404139 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96000e070 con 0x7fa96809e9f0 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: dump: 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.658 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 1 -- 172.21.15.150:0/3049404139 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96000bbe0 con 0x7fa96809e9f0 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 5 monclient: authenticate success, global_id 114109 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 1 -- 172.21.15.150:0/3049404139 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9600092e0 con 0x7fa96809e9f0 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 4 set_mon_vals no callback set 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.659 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa966ffd700 10 monclient: dump: 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.660 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 10 monclient: shutdown 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 -- 172.21.15.150:0/3049404139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809e9f0 msgr2=0x7fa9681396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809e9f0 0x7fa9681396d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa960009f20 tx=0x7fa960009840 comp rx=0 tx=0).stop 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 -- 172.21.15.150:0/3049404139 shutdown_connections 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809e9f0 0x7fa9681396d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/3049404139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809f8b0 0x7fa96809a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 -- 172.21.15.150:0/3049404139 >> 172.21.15.150:0/3049404139 conn(0x7fa9680a2f60 msgr2=0x7fa96809f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 -- 172.21.15.150:0/3049404139 shutdown_connections 2024-09-18T10:18:51.661 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 1 -- 172.21.15.150:0/3049404139 wait complete. 2024-09-18T10:18:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 10 monclient: build_initial_monmap 2024-09-18T10:18:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.644+0000 7fa96e8f2700 10 monclient: monmap: 2024-09-18T10:18:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:18:51.666 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 Processor -- start 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 -- start start 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient: init 2024-09-18T10:18:51.667 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809df10 0x7fa968094750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-c con 0x7fa96809df10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96809e9f0 0x7fa96808cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-a con 0x7fa96809e9f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): picked mon.noname-b con 0x7fa96809f8b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96808f440 con 0x7fa96809e9f0 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96809c340 con 0x7fa96809f8b0 2024-09-18T10:18:51.668 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa968091230 con 0x7fa96809df10 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96e8f2700 10 monclient(hunting): authenticate will time out at 1413.745920s 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34386/0 (socket says 172.21.15.150:34386) 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 -- 172.21.15.150:0/2997693187 learned_addr learned my addr 172.21.15.150:0/2997693187 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): get_auth_request con 0x7fa96809f8b0 auth_method 0 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809df10 0x7fa968094750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): get_auth_request con 0x7fa96809df10 auth_method 0 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.669 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96809e9f0 0x7fa96808cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): get_auth_request con 0x7fa96809e9f0 auth_method 0 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa967fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.670 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa9677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient(hunting): handle_auth_done global_id 114112 payload 824 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 -- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809df10 msgr2=0x7fa968094750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809df10 0x7fa968094750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 -- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96809e9f0 msgr2=0x7fa96808cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96809e9f0 0x7fa96808cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient: _finish_hunting 0 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 monclient: found mon.noname-b 2024-09-18T10:18:51.671 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.648+0000 7fa96ce8f700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960009ae0 con 0x7fa96809f8b0 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96ce8f700 1 -- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 msgr2=0x7fa96808d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96ce8f700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa9600098c0 tx=0x7fa960017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9600070e0 con 0x7fa96809f8b0 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: dump: 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:51.672 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: _finish_auth 0 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 5 monclient: authenticate success, global_id 114112 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa960012a60 con 0x7fa96809f8b0 2024-09-18T10:18:51.673 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960012e00 con 0x7fa96809f8b0 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _renew_subs 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96808ef50 con 0x7fa96809f8b0 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa9667fc700 4 set_mon_vals no callback set 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 10 monclient: dump: 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.674 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _renew_subs 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96808f610 con 0x7fa96809f8b0 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa96001d630 con 0x7fa96809f8b0 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96806d480 con 0x7fa96809f8b0 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 0x7fa954037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.675 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.652+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa96004a4d0 con 0x7fa96809f8b0 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.656+0000 7fa967fff700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 0x7fa954037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.656+0000 7fa967fff700 10 monclient: get_auth_request con 0x7fa954035ac0 auth_method 0 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.656+0000 7fa967fff700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 0x7fa954037f70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa958009db0 tx=0x7fa9580091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.660+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa960007b00 con 0x7fa96809f8b0 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.660+0000 7fa94ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.676 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.660+0000 7fa94ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 10 monclient: get_monmap_and_config 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 10 monclient: build_initial_monmap 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 10 monclient: monmap: 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:51.738101+0000 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:51.738101+0000 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:51.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 1 Processor -- start 2024-09-18T10:18:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 1 -- start start 2024-09-18T10:18:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.732+0000 7f2249eba700 10 monclient: init 2024-09-18T10:18:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-18T10:18:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f224409b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-b con 0x7f224409aea0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2244097660 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2244098140 0x7f2244135880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-c con 0x7f2244098140 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22440954b0 con 0x7f2244097660 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224409a450 con 0x7f2244098140 2024-09-18T10:18:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2244098f00 con 0x7f224409aea0 2024-09-18T10:18:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient(hunting): authenticate will time out at 1420.664183s 2024-09-18T10:18:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2243fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2244097660 0x7f2244097a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37454/0 (socket says 172.21.15.142:37454) 2024-09-18T10:18:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2243fff700 1 -- 172.21.15.142:0/3029534966 learned_addr learned my addr 172.21.15.142:0/3029534966 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 --2- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f224409b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): get_auth_request con 0x7f224409aea0 auth_method 0 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 1 --2- 172.21.15.142:0/3029534966 >> v2:172.21.15.150:3300/0 conn(0x7f2244098140 0x7f2244135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): get_auth_request con 0x7f2244098140 auth_method 0 2024-09-18T10:18:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient(hunting): handle_auth_done global_id 74909 payload 274 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 -- 172.21.15.142:0/3029534966 >> v2:172.21.15.150:3300/0 conn(0x7f2244098140 msgr2=0x7f2244135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 --2- 172.21.15.142:0/3029534966 >> v2:172.21.15.150:3300/0 conn(0x7f2244098140 0x7f2244135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 -- 172.21.15.142:0/3029534966 >> v1:172.21.15.142:6789/0 conn(0x7f2244097660 legacy=0x7f2244097a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient: _finish_hunting 0 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 monclient: found mon.noname-b 2024-09-18T10:18:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 -- 172.21.15.142:0/3029534966 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2244099e60 con 0x7f224409aea0 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 -- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 msgr2=0x7f224409b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2248eb8700 1 --2- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f224409b270 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f22380098e0 tx=0x7f22380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ff916bc14d182c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 1 -- 172.21.15.142:0/3029534966 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800b850 con 0x7f224409aea0 2024-09-18T10:18:51.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: dump: 2024-09-18T10:18:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 1 -- 172.21.15.142:0/3029534966 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223800b9b0 con 0x7f224409aea0 2024-09-18T10:18:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 5 monclient: authenticate success, global_id 74909 2024-09-18T10:18:51.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:51.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f2249eba700 4 set_mon_vals no callback set 2024-09-18T10:18:51.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 1 -- 172.21.15.142:0/3029534966 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800bd50 con 0x7f224409aea0 2024-09-18T10:18:51.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:51.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.736+0000 7f22437fe700 10 monclient: dump: 2024-09-18T10:18:51.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.763 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: shutdown 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 msgr2=0x7f224409b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f224409b270 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f22380098e0 tx=0x7f22380091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- 172.21.15.142:0/3029534966 shutdown_connections 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- 172.21.15.142:0/3029534966 >> v2:172.21.15.150:3300/0 conn(0x7f2244098140 0x7f2244135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- 172.21.15.142:0/3029534966 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f224409b270 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- 172.21.15.142:0/3029534966 >> 172.21.15.142:0/3029534966 conn(0x7f2244071ae0 msgr2=0x7f22440a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- 172.21.15.142:0/3029534966 shutdown_connections 2024-09-18T10:18:51.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- 172.21.15.142:0/3029534966 wait complete. 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: build_initial_monmap 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: monmap: 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:51.766 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:51.767 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 Processor -- start 2024-09-18T10:18:51.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- start start 2024-09-18T10:18:51.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: init 2024-09-18T10:18:51.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:51.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:51.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 0x7f224408c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-b con 0x7f2244097660 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:51.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2244098140 0x7f2244093fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-a con 0x7f2244098140 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:51.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): picked mon.noname-c con 0x7f224409aea0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224408d650 con 0x7f2244098140 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224408b9e0 con 0x7f2244097660 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2244098140 0x7f2244093fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:51.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22440a11e0 con 0x7f224409aea0 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33262/0 (socket says 172.21.15.142:33262) 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 1 -- 172.21.15.142:0/1758215882 learned_addr learned my addr 172.21.15.142:0/1758215882 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 0x7f224408c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2249eba700 10 monclient(hunting): authenticate will time out at 1420.671384s 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): get_auth_request con 0x7f224409aea0 auth_method 0 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): get_auth_request con 0x7f2244098140 auth_method 0 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): get_auth_request con 0x7f2244097660 auth_method 0 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f22496b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2248eb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.740+0000 7f2243fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 10 monclient(hunting): handle_auth_done global_id 74912 payload 824 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 -- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 msgr2=0x7f224408c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 0x7f224408c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 -- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2244098140 msgr2=0x7f2244093fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2244098140 0x7f2244093fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 10 monclient: _finish_hunting 0 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 monclient: found mon.noname-c 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:51.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 -- 172.21.15.142:0/1758215882 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009420 con 0x7f224409aea0 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2248eb8700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 0x7f224408c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 -- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 msgr2=0x7f2244089920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f22496b9700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f223c009ef0 tx=0x7f223c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00cd80 con 0x7f224409aea0 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: dump: 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.782 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 5 monclient: authenticate success, global_id 74912 2024-09-18T10:18:51.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 10 monclient: _renew_subs 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224406f870 con 0x7f224409aea0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 10 monclient: _renew_subs 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224408c030 con 0x7f224409aea0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c00f040 con 0x7f224409aea0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00cee0 con 0x7f224409aea0 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:51.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2240ff9700 10 monclient: dump: 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:51.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.744+0000 7f2242ffd700 4 set_mon_vals no callback set 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2249eba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f22496b9700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f223c009ef0 tx=0x7f223c00b040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f223c049b00 osd_map(264..264 src has 176..264) v4 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2249eba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2249eba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f223c0148e0 con 0x7f224409aea0 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22240050b0 con 0x7f224409aea0 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2240ff9700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 0x7f2228037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:51.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2248eb8700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 0x7f2228037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:51.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.748+0000 7f2248eb8700 10 monclient: get_auth_request con 0x7f2228035ac0 auth_method 0 2024-09-18T10:18:51.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.752+0000 7f2248eb8700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 0x7f2228037f70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f22380030c0 tx=0x7f2238016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:51.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:51.756+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f223c049b00 con 0x7f224409aea0 2024-09-18T10:18:51.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.888+0000 7fa96e8f2700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:51.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.892+0000 7fa96e8f2700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:51.897 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.892+0000 7fa96e8f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:51.898 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:51.892+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7fa968063ed0 con 0x7fa96809f8b0 2024-09-18T10:18:52.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.128+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v16) v1 ==== 82+0+0 (secure 0 0 0) 0x7fa960007320 con 0x7fa96809f8b0 2024-09-18T10:18:52.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.128+0000 7fa94ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:52.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.128+0000 7fa94ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:52.297 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.288+0000 7fa96e8f2700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:52.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.288+0000 7fa96e8f2700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:52.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.288+0000 7fa96e8f2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:52.298 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.288+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bazar"} v 0) v1 -- 0x7fa968068aa0 con 0x7fa96809f8b0 2024-09-18T10:18:52.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.576+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f223c024530 con 0x7f224409aea0 2024-09-18T10:18:52.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.576+0000 7f2240ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:52.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.576+0000 7f2240ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:52.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.808+0000 7f2249eba700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:18:52.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.808+0000 7f2249eba700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:18:52.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.808+0000 7f2249eba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:52.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.808+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f2224005b70 con 0x7f224409aea0 2024-09-18T10:18:52.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa94ffff700 1 -- 172.21.15.150:0/2997693187 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bazar"}]=0 v16) v1 ==== 82+0+0 (secure 0 0 0) 0x7fa960007500 con 0x7fa96809f8b0 2024-09-18T10:18:52.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa94ffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bazar"}] 2024-09-18T10:18:52.816 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa94ffff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:18:52.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.812+0000 7f2240ff9700 1 -- 172.21.15.142:0/1758215882 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f223c018070 con 0x7f224409aea0 2024-09-18T10:18:52.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.812+0000 7f2240ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:18:52.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.812+0000 7f2240ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:52.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 msgr2=0x7fa954037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:52.824 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 0x7fa954037f70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa958009db0 tx=0x7fa9580091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 10 monclient: shutdown 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 msgr2=0x7fa96808d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa9600098c0 tx=0x7fa960017000 comp rx=0 tx=0).stop 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 shutdown_connections 2024-09-18T10:18:52.825 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa954035ac0 0x7fa954037f70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96809f8b0 0x7fa96808d440 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa96809e9f0 0x7fa96808cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 --2- 172.21.15.150:0/2997693187 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa96809df10 0x7fa968094750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 >> 172.21.15.150:0/2997693187 conn(0x7fa9680a2f60 msgr2=0x7fa968095990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 shutdown_connections 2024-09-18T10:18:52.826 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:18:52.812+0000 7fa96e8f2700 1 -- 172.21.15.150:0/2997693187 wait complete. 2024-09-18T10:18:52.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 msgr2=0x7f2228037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:52.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 0x7f2228037f70 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f22380030c0 tx=0x7f2238016000 comp rx=0 tx=0).stop 2024-09-18T10:18:52.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 10 monclient: shutdown 2024-09-18T10:18:52.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 msgr2=0x7f2244089920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f223c009ef0 tx=0x7f223c00b040 comp rx=0 tx=0).stop 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 shutdown_connections 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2228035ac0 0x7f2228037f70 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f224409aea0 0x7f2244089920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2244098140 0x7f2244093fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 --2- 172.21.15.142:0/1758215882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2244097660 0x7f224408c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 >> 172.21.15.142:0/1758215882 conn(0x7f2244071ae0 msgr2=0x7f22440a0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 shutdown_connections 2024-09-18T10:18:52.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:52.816+0000 7f2249eba700 1 -- 172.21.15.142:0/1758215882 wait complete. 2024-09-18T10:18:52.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:18:52.837 INFO:tasks.workunit.client.0.smithi150.stderr:+ c=''\''allow command "auth ls", allow command quorum_status'\''' 2024-09-18T10:18:52.837 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 0 2024-09-18T10:18:52.837 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph auth get-or-create client.foo mon '\''allow command "auth ls", allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.foo.keyring' 2024-09-18T10:18:52.837 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:18:52.837 INFO:tasks.workunit.client.0.smithi150.stdout:ceph auth get-or-create client.foo mon 'allow command "auth ls", allow command quorum_status' > /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T10:18:52.838 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T10:18:52.838 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph auth get-or-create client.foo mon ''\''allow' command '"auth' 'ls",' allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.foo.keyring 2024-09-18T10:18:52.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=7,logm=59} crc {auth=4067369411,config=3140514009,health=1744959055,kv=4095092511,logm=1091261505}) 2024-09-18T10:18:52.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=8,logm=59} crc {auth=4067369411,config=3140514009,health=2508420555,kv=4261215878,logm=3359320700}) 2024-09-18T10:18:52.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=7,logm=59} crc {auth=4067369411,config=3140514009,health=1744959055,kv=4095092511,logm=1091261505}) 2024-09-18T10:18:52.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=3205878231,config=3140514009,health=1744959055,kv=4261215878,logm=654123484}) 2024-09-18T10:18:52.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356612823}) 2024-09-18T10:18:52.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145704742}) 2024-09-18T10:18:52.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356612823}) 2024-09-18T10:18:52.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.892+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418655687}) 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645185004}) 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3660863334}) 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645185004}) 2024-09-18T10:18:52.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.896+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737096628}) 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112991436}) 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953777577}) 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112991436}) 2024-09-18T10:18:52.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=487762878}) 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915259503}) 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1023594017}) 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915259503}) 2024-09-18T10:18:52.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344972333}) 2024-09-18T10:18:52.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610443094}) 2024-09-18T10:18:52.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945049209}) 2024-09-18T10:18:52.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610443094}) 2024-09-18T10:18:52.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.900+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981248710}) 2024-09-18T10:18:52.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691092932}) 2024-09-18T10:18:52.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167280215}) 2024-09-18T10:18:52.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:52.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691092932}) 2024-09-18T10:18:52.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:52.928+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253442378}) 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021653322}) 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983805813}) 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021653322}) 2024-09-18T10:18:53.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.040+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959669920}) 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203483177}) 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226088651}) 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203483177}) 2024-09-18T10:18:53.058 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.052+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914824417}) 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237234192}) 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248220065}) 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237234192}) 2024-09-18T10:18:53.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.056+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290159187}) 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904536751}) 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=725670816}) 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904536751}) 2024-09-18T10:18:53.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.192+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884121835}) 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952759528}) 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273588939}) 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952759528}) 2024-09-18T10:18:53.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.200+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628215631}) 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041660206}) 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738776444}) 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041660206}) 2024-09-18T10:18:53.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.208+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456526087}) 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604518503}) 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744029843}) 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604518503}) 2024-09-18T10:18:53.342 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.336+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464447047}) 2024-09-18T10:18:53.368 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.368 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974973179}) 2024-09-18T10:18:53.368 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096719545}) 2024-09-18T10:18:53.368 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.369 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974973179}) 2024-09-18T10:18:53.369 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.360+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572301041}) 2024-09-18T10:18:53.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:18:53.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:18:53.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:53.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 0 2024-09-18T10:18:53.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-18T10:18:53.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:18:53.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:18:53.386 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:18:53.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:18:53.565 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.566 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377343524}) 2024-09-18T10:18:53.566 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089347686}) 2024-09-18T10:18:53.566 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.566 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=377343524}) 2024-09-18T10:18:53.566 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2636784684}) 2024-09-18T10:18:53.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239563091}) 2024-09-18T10:18:53.567 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423192190}) 2024-09-18T10:18:53.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239563091}) 2024-09-18T10:18:53.568 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.560+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2324391518}) 2024-09-18T10:18:53.569 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.569 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030070907}) 2024-09-18T10:18:53.569 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3719694994}) 2024-09-18T10:18:53.569 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030070907}) 2024-09-18T10:18:53.570 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=267824517}) 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397883263}) 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3738293023}) 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397883263}) 2024-09-18T10:18:53.571 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202399464}) 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472016069}) 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3066122819}) 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472016069}) 2024-09-18T10:18:53.573 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.564+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243633483}) 2024-09-18T10:18:53.574 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.574 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050643714}) 2024-09-18T10:18:53.575 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1696409450}) 2024-09-18T10:18:53.575 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.575 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050643714}) 2024-09-18T10:18:53.575 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434438729}) 2024-09-18T10:18:53.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857954923}) 2024-09-18T10:18:53.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060337282}) 2024-09-18T10:18:53.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857954923}) 2024-09-18T10:18:53.577 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.568+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027779077}) 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865075854}) 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198009805}) 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865075854}) 2024-09-18T10:18:53.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372152609}) 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959036047}) 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454888427}) 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959036047}) 2024-09-18T10:18:53.580 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.572+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789988850}) 2024-09-18T10:18:53.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376119394}) 2024-09-18T10:18:53.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779012798}) 2024-09-18T10:18:53.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376119394}) 2024-09-18T10:18:53.582 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709555665}) 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056874173}) 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=600322554}) 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056874173}) 2024-09-18T10:18:53.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900346239}) 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747577374}) 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344733926}) 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747577374}) 2024-09-18T10:18:53.584 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815893196}) 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.576+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938037145}) 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394357577}) 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938037145}) 2024-09-18T10:18:53.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3912016551}) 2024-09-18T10:18:53.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161761065}) 2024-09-18T10:18:53.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763723902}) 2024-09-18T10:18:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161761065}) 2024-09-18T10:18:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415053866}) 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959689029}) 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867712075}) 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959689029}) 2024-09-18T10:18:53.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2763425889}) 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794886151}) 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986626817}) 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794886151}) 2024-09-18T10:18:53.589 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.580+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357812856}) 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539413735}) 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020944243}) 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539413735}) 2024-09-18T10:18:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874946026}) 2024-09-18T10:18:53.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813087702}) 2024-09-18T10:18:53.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596407940}) 2024-09-18T10:18:53.591 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813087702}) 2024-09-18T10:18:53.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164128167}) 2024-09-18T10:18:53.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010985410}) 2024-09-18T10:18:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588458622}) 2024-09-18T10:18:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010985410}) 2024-09-18T10:18:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.584+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517151394}) 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603525330}) 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214445485}) 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603525330}) 2024-09-18T10:18:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739910444}) 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472943466}) 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454079133}) 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472943466}) 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553494228}) 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518849025}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088924503}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518849025}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.588+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531185417}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233530461}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996693538}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233530461}) 2024-09-18T10:18:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1901167218}) 2024-09-18T10:18:53.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194123836}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890926599}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2194123836}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608031784}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465493421}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177041080}) 2024-09-18T10:18:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465493421}) 2024-09-18T10:18:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.592+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931081344}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702358021}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023220015}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702358021}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367976390}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376044163}) 2024-09-18T10:18:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785174262}) 2024-09-18T10:18:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376044163}) 2024-09-18T10:18:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498620468}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822245258}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468149532}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822245258}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1709132048}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289806461}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2657337845}) 2024-09-18T10:18:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289806461}) 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.596+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057189294}) 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068383060}) 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252172826}) 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068383060}) 2024-09-18T10:18:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659095001}) 2024-09-18T10:18:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862836132}) 2024-09-18T10:18:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360474128}) 2024-09-18T10:18:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862836132}) 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940136569}) 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008352266}) 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074499134}) 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008352266}) 2024-09-18T10:18:53.609 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.600+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2593218037}) 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267966785}) 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764459356}) 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267966785}) 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1811055279}) 2024-09-18T10:18:53.611 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.612 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049797067}) 2024-09-18T10:18:53.612 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294557207}) 2024-09-18T10:18:53.612 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.612 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049797067}) 2024-09-18T10:18:53.612 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105605347}) 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212097384}) 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403429826}) 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212097384}) 2024-09-18T10:18:53.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.604+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178655169}) 2024-09-18T10:18:53.614 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.614 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341752973}) 2024-09-18T10:18:53.614 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781940226}) 2024-09-18T10:18:53.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341752973}) 2024-09-18T10:18:53.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761644927}) 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050778014}) 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091293587}) 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050778014}) 2024-09-18T10:18:53.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269475629}) 2024-09-18T10:18:53.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.608+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136566384}) 2024-09-18T10:18:53.618 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053034003}) 2024-09-18T10:18:53.618 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.618 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136566384}) 2024-09-18T10:18:53.618 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2207625219}) 2024-09-18T10:18:53.620 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.621 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {logm=2228024617,mdsmap=3140514010,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1868973795,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=3541866103}) 2024-09-18T10:18:53.621 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {logm=2177712532,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=4259934215,mon_config_key=840343407,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1563569046}) 2024-09-18T10:18:53.621 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.621 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {logm=2228024617,mdsmap=3140514010,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1868973795,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=3541866103}) 2024-09-18T10:18:53.621 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.612+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=8,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=57} crc {logm=815857630,mdsmap=3140514009,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1581226467,mon_config_key=620546692,monmap=3505902565,osd_metadata=2166527243,osd_pg_creating=69081760,osd_snap=454132730,osdmap=636811917}) 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=3791912870}) 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=88} crc {osdmap=2135772306}) 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=3791912870}) 2024-09-18T10:18:53.628 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:18:53.620+0000 7f2439e88700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=95} crc {osdmap=759660120}) 2024-09-18T10:18:53.834 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:18:53.835 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:53.894 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:18:53.894 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:18:53.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:53.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 0 2024-09-18T10:18:53.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls' 2024-09-18T10:18:53.895 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:18:53.895 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T10:18:53.896 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T10:18:53.896 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth ls 2024-09-18T10:18:53.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:53.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: monmap: 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:53.945919+0000 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:53.945919+0000 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:53.948 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 Processor -- start 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 -- start start 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: init 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f230c068bd0 0x7f230c096070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f230c068bd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f230c0965b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f230c09c570 0x7f230c137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f230c09c570 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c095970 con 0x7f230c0965b0 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230c068fa0 con 0x7f230c068bd0 2024-09-18T10:18:53.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230c095690 con 0x7f230c09c570 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230bfff700 10 monclient(hunting): authenticate will time out at 1422.871397s 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f230c068bd0 0x7f230c096070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f230c068bd0 0x7f230c096070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42910/0 (socket says 172.21.15.142:42910) 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 1 -- 172.21.15.142:0/118935202 learned_addr learned my addr 172.21.15.142:0/118935202 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): get_auth_request con 0x7f230c068bd0 auth_method 0 2024-09-18T10:18:53.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230b7fe700 1 --2- 172.21.15.142:0/118935202 >> v2:172.21.15.150:3300/0 conn(0x7f230c09c570 0x7f230c137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230b7fe700 10 monclient(hunting): get_auth_request con 0x7f230c09c570 auth_method 0 2024-09-18T10:18:53.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.940+0000 7f230b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 10 monclient(hunting): handle_auth_done global_id 94539 payload 274 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 -- 172.21.15.142:0/118935202 >> v2:172.21.15.150:3300/0 conn(0x7f230c09c570 msgr2=0x7f230c137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 --2- 172.21.15.142:0/118935202 >> v2:172.21.15.150:3300/0 conn(0x7f230c09c570 0x7f230c137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 -- 172.21.15.142:0/118935202 >> v1:172.21.15.150:6789/0 conn(0x7f230c0965b0 legacy=0x7f230c09be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 monclient: found mon.noname-a 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 -- 172.21.15.142:0/118935202 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c06f6f0 con 0x7f230c068bd0 2024-09-18T10:18:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 -- 172.21.15.142:0/118935202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 msgr2=0x7f230c096070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230affd700 1 --2- 172.21.15.142:0/118935202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c096070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f230c0a1440 tx=0x7f22fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0700cc016ed55cb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 1 -- 172.21.15.142:0/118935202 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc009e70 con 0x7f230c068bd0 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: dump: 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:53.962 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:53.962 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:53.962 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 1 -- 172.21.15.142:0/118935202 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc00ba70 con 0x7f230c068bd0 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 5 monclient: authenticate success, global_id 94539 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 1 -- 172.21.15.142:0/118935202 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc00be30 con 0x7f230c068bd0 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 4 set_mon_vals no callback set 2024-09-18T10:18:53.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230a7fc700 10 monclient: dump: 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:53.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 10 monclient: shutdown 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 1 -- 172.21.15.142:0/118935202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 msgr2=0x7f230c096070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.944+0000 7f230bfff700 1 --2- 172.21.15.142:0/118935202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c096070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f230c0a1440 tx=0x7f22fc009250 comp rx=0 tx=0).stop 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- 172.21.15.142:0/118935202 shutdown_connections 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 --2- 172.21.15.142:0/118935202 >> v2:172.21.15.150:3300/0 conn(0x7f230c09c570 0x7f230c137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 --2- 172.21.15.142:0/118935202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c096070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- 172.21.15.142:0/118935202 >> 172.21.15.142:0/118935202 conn(0x7f230c0a1470 msgr2=0x7f230c0a1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- 172.21.15.142:0/118935202 shutdown_connections 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- 172.21.15.142:0/118935202 wait complete. 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient: build_initial_monmap 2024-09-18T10:18:53.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient: monmap: 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 Processor -- start 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- start start 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient: init 2024-09-18T10:18:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f230c068bd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 0x7f230c141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f230c0965b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f230c09c570 0x7f230c098a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f230c09c570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c06ed90 con 0x7f230c068bd0 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c091390 con 0x7f230c0965b0 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c091670 con 0x7f230c09c570 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230bfff700 10 monclient(hunting): authenticate will time out at 1422.878332s 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42912/0 (socket says 172.21.15.142:42912) 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 -- 172.21.15.142:0/3923653344 learned_addr learned my addr 172.21.15.142:0/3923653344 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): get_auth_request con 0x7f230c068bd0 auth_method 0 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230b7fe700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f230c09c570 0x7f230c098a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f23027fc700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 0x7f230c141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230b7fe700 10 monclient(hunting): get_auth_request con 0x7f230c09c570 auth_method 0 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient(hunting): handle_auth_done global_id 94542 payload 824 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 -- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f230c09c570 msgr2=0x7f230c098a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f230c09c570 0x7f230c098a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 -- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 msgr2=0x7f230c141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 0x7f230c141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 monclient: found mon.noname-a 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f230affd700 1 -- 172.21.15.142:0/3923653344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0094f0 con 0x7f230c068bd0 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f23027fc700 10 monclient: get_auth_request con 0x7f230c0965b0 auth_method 0 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.948+0000 7f23027fc700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 0x7f230c141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230affd700 1 -- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 msgr2=0x7f230c141890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230affd700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f230c067b40 tx=0x7f22fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0119d0 con 0x7f230c068bd0 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: dump: 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:53.971 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 5 monclient: authenticate success, global_id 94542 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc020070 con 0x7f230c068bd0 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _renew_subs 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c09a4b0 con 0x7f230c068bd0 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc011dc0 con 0x7f230c068bd0 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 10 monclient: dump: 2024-09-18T10:18:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f2309ffb700 4 set_mon_vals no callback set 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _renew_subs 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c0905d0 con 0x7f230c068bd0 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f22fc022030 con 0x7f230c068bd0 2024-09-18T10:18:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:53.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:53.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:53.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c091830 con 0x7f230c068bd0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.952+0000 7f23037fe700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 0x7f22ec037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.956+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f22fc04abd0 con 0x7f230c068bd0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.956+0000 7f23027fc700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 0x7f22ec037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.956+0000 7f23027fc700 10 monclient: get_auth_request con 0x7f22ec035ac0 auth_method 0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.956+0000 7f23027fc700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 0x7f22ec037f70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f22f4002880 tx=0x7f22f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.964+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f22fc01ac00 con 0x7f230c068bd0 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.964+0000 7f23037fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:53.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:53.964+0000 7f23037fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:54.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:54.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:54.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f230c091830 con 0x7f230c068bd0 2024-09-18T10:18:54.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f23037fe700 1 -- 172.21.15.142:0/3923653344 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f22fc023e60 con 0x7f230c068bd0 2024-09-18T10:18:54.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f23037fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:54.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f23037fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:54.199 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:54.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 msgr2=0x7f22ec037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 0x7f22ec037f70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f22f4002880 tx=0x7f22f4009040 comp rx=0 tx=0).stop 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 10 monclient: shutdown 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 msgr2=0x7f230c141890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f230c067b40 tx=0x7f22fc016000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 shutdown_connections 2024-09-18T10:18:54.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22ec035ac0 0x7f22ec037f70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f230c09c570 0x7f230c098a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f230c0965b0 0x7f230c141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 --2- 172.21.15.142:0/3923653344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f230c068bd0 0x7f230c141890 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 >> 172.21.15.142:0/3923653344 conn(0x7f230c0a1470 msgr2=0x7f230c09bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 shutdown_connections 2024-09-18T10:18:54.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.192+0000 7f230bfff700 1 -- 172.21.15.142:0/3923653344 wait complete. 2024-09-18T10:18:54.212 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:18:54.212 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:18:54.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:18:54.212 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:18:54.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.312+0000 7f4d61c0c700 10 monclient: get_monmap_and_config 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.312+0000 7f4d61c0c700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.312+0000 7f4d61c0c700 10 monclient: monmap: 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:54.319130+0000 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:54.319130+0000 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:54.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 Processor -- start 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 -- start start 2024-09-18T10:18:54.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient: init 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 0x7f4d5c0986e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-c con 0x7f4d5c09c690 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4d5c09ca60 0x7f4d5c09cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-a con 0x7f4d5c09ca60 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c12e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-b con 0x7f4d5c12e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5c09df40 con 0x7f4d5c09ca60 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5c097890 con 0x7f4d5c09c690 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c09d3e0 con 0x7f4d5c12e460 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d61c0c700 10 monclient(hunting): authenticate will time out at 1423.246268s 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4d5c09ca60 0x7f4d5c09cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c12e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 0x7f4d5c0986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 0x7f4d5c0986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50750/0 (socket says 172.21.15.142:50750) 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 1 -- 172.21.15.142:0/1753882499 learned_addr learned my addr 172.21.15.142:0/1753882499 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): get_auth_request con 0x7f4d5c12e460 auth_method 0 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request con 0x7f4d5c09c690 auth_method 0 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): get_auth_request con 0x7f4d5c09ca60 auth_method 0 2024-09-18T10:18:54.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_done global_id 94551 payload 274 2024-09-18T10:18:54.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 msgr2=0x7f4d5c12e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c12e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1753882499 >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 msgr2=0x7f4d5c0986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1753882499 >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 0x7f4d5c0986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 monclient: found mon.noname-a 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.316+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1753882499 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c06f180 con 0x7f4d5c09ca60 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 msgr2=0x7f4d5c09cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c09cea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c06a550 tx=0x7f4d5000a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b975ffacfe0420b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 1 -- 172.21.15.142:0/1753882499 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000aea0 con 0x7f4d5c09ca60 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: dump: 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.334 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 5 monclient: authenticate success, global_id 94551 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 1 -- 172.21.15.142:0/1753882499 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5000cb80 con 0x7f4d5c09ca60 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:54.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 4 set_mon_vals no callback set 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 1 -- 172.21.15.142:0/1753882499 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50018400 con 0x7f4d5c09ca60 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d5b7fe700 10 monclient: dump: 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.336 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 10 monclient: shutdown 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 msgr2=0x7f4d5c09cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c09cea0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c06a550 tx=0x7f4d5000a330 comp rx=0 tx=0).stop 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1753882499 shutdown_connections 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c12e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1753882499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c09cea0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1753882499 >> v2:172.21.15.150:3300/0 conn(0x7f4d5c09c690 0x7f4d5c0986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1753882499 >> 172.21.15.142:0/1753882499 conn(0x7f4d5c0714d0 msgr2=0x7f4d5c0a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1753882499 shutdown_connections 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.320+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1753882499 wait complete. 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient: monmap: 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.338 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 Processor -- start 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 -- start start 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient: init 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d5c09c690 0x7f4d5c1417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-b con 0x7f4d5c09c690 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-a con 0x7f4d5c09ca60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c119bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): picked mon.noname-c con 0x7f4d5c12e460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c121a40 con 0x7f4d5c09ca60 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c121760 con 0x7f4d5c09c690 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c121d20 con 0x7f4d5c12e460 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d61c0c700 10 monclient(hunting): authenticate will time out at 1423.254838s 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c141d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42934/0 (socket says 172.21.15.142:42934) 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1145184471 learned_addr learned my addr 172.21.15.142:0/1145184471 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): get_auth_request con 0x7f4d5c09ca60 auth_method 0 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c119bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): get_auth_request con 0x7f4d5c12e460 auth_method 0 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d60c0a700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d5c09c690 0x7f4d5c1417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d6140b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request con 0x7f4d5c09c690 auth_method 0 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d60c0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d60c0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient(hunting): handle_auth_done global_id 94554 payload 824 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 msgr2=0x7f4d5c119bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4d5c12e460 0x7f4d5c119bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d5c09c690 msgr2=0x7f4d5c1417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d5c09c690 0x7f4d5c1417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 monclient: found mon.noname-a 2024-09-18T10:18:54.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.324+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5000a5d0 con 0x7f4d5c09ca60 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d5bfff700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 msgr2=0x7f4d5c141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d5bfff700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c141d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000c4d0 tx=0x7f4d5001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5001c040 con 0x7f4d5c09ca60 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: dump: 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.346 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 5 monclient: authenticate success, global_id 94554 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 monclient: _renew_subs 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50006b10 con 0x7f4d5c09ca60 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5c142240 con 0x7f4d5c09ca60 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50010830 con 0x7f4d5c09ca60 2024-09-18T10:18:54.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 monclient: _renew_subs 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5c120a10 con 0x7f4d5c09ca60 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d5affd700 4 set_mon_vals no callback set 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 10 monclient: dump: 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 0x7f4d5c029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c1218d0 con 0x7f4d5c11a8e0 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 0x7f4d5c029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient: get_auth_request con 0x7f4d5c11a8e0 auth_method 0 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_done global_id 94557 payload 824 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d60c0a700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d5c064680 con 0x7f4d5c11a8e0 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.328+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4d50010c20 con 0x7f4d5c09ca60 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d48035b10 0x7f4d48037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4d5004a0e0 con 0x7f4d5c09ca60 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d6140b700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d48035b10 0x7f4d48037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d6140b700 10 monclient: get_auth_request con 0x7f4d48035b10 auth_method 0 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d60c0a700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 msgr2=0x7f4d5c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d60c0a700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 0x7f4d5c029d00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c011810 tx=0x7f4d4c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c0159b0 con 0x7f4d5c11a8e0 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 10 monclient: dump: 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.351 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4d4c015b10 con 0x7f4d5c11a8e0 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 msgr2=0x7f4d5c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 0x7f4d5c029d00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c011810 tx=0x7f4d4c008040 comp rx=0 tx=0).stop 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 msgr2=0x7f4d5c029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d58ff9700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11a8e0 0x7f4d5c029d00 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c011810 tx=0x7f4d4c008040 comp rx=0 tx=0).stop 2024-09-18T10:18:54.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.332+0000 7f4d6140b700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d48035b10 0x7f4d48037fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400f4d0 tx=0x7f4d5400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d61c0c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:54.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 0x7f4d5c02a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d61c0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c06d2b0 con 0x7f4d5c11acb0 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 0x7f4d5c02a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient: get_auth_request con 0x7f4d5c11acb0 auth_method 0 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 10 monclient(hunting): handle_auth_done global_id 94560 payload 824 2024-09-18T10:18:54.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.340+0000 7f4d60c0a700 1 -- 172.21.15.142:0/1145184471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4d5c072a80 con 0x7f4d5c11acb0 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d60c0a700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 msgr2=0x7f4d5c02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d60c0a700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 0x7f4d5c02a240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c011810 tx=0x7f4d4c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c021040 con 0x7f4d5c11acb0 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 10 monclient: dump: 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.356 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f4d4c02a410 con 0x7f4d5c11acb0 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 msgr2=0x7f4d5c02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 0x7f4d5c02a240 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c011810 tx=0x7f4d4c01a000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 msgr2=0x7f4d5c02a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d58ff9700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c11acb0 0x7f4d5c02a240 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:18:54.358 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:54.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:54.360 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:54.361 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:54.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.363 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.364 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.368 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.369 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.370 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:54.370 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:54.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d48035b10 msgr2=0x7f4d48037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4d48035b10 0x7f4d48037fc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400f4d0 tx=0x7f4d5400b040 comp rx=0 tx=0).stop 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 10 monclient: shutdown 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 msgr2=0x7f4d5c141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 --2- 172.21.15.142:0/1145184471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4d5c09ca60 0x7f4d5c141d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000c4d0 tx=0x7f4d5001a000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d6140b700 1 -- 172.21.15.142:0/1145184471 reap_dead start 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d6140b700 1 -- 172.21.15.142:0/1145184471 reap_dead start 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 shutdown_connections 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 >> 172.21.15.142:0/1145184471 conn(0x7f4d5c0714d0 msgr2=0x7f4d5c0a0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 shutdown_connections 2024-09-18T10:18:54.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.344+0000 7f4d61c0c700 1 -- 172.21.15.142:0/1145184471 wait complete. 2024-09-18T10:18:54.373 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:54.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:18:54.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:18:54.385 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 13 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export' 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T10:18:54.386 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth export 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 10 monclient: get_monmap_and_config 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 10 monclient: monmap: 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:54.482484+0000 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:54.482484+0000 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:54.485 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 1 Processor -- start 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 1 -- start start 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.476+0000 7ffa1a434700 10 monclient: init 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:18:54.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa140a2e80 0x7ffa14070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-c con 0x7ffa140a2e80 addr v2:172.21.15.150:3300/0 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-b con 0x7ffa1409e050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa14071970 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1409a560 con 0x7ffa14071970 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1409e7d0 con 0x7ffa140a2e80 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14071d90 con 0x7ffa1409e050 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa1a434700 10 monclient(hunting): authenticate will time out at 1423.409146s 2024-09-18T10:18:54.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa140a2e80 0x7ffa14070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa140a2e80 0x7ffa14070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50760/0 (socket says 172.21.15.142:50760) 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 -- 172.21.15.142:0/331207761 learned_addr learned my addr 172.21.15.142:0/331207761 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): get_auth_request con 0x7ffa140a2e80 auth_method 0 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa189d1700 1 --1- 172.21.15.142:0/331207761 >> v1:172.21.15.150:6789/0 conn(0x7ffa14071970 0x7ffa14138780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:50486/0 (socket says 172.21.15.142:50486) 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): get_auth_request con 0x7ffa1409e050 auth_method 0 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient(hunting): handle_auth_done global_id 114151 payload 274 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 -- 172.21.15.142:0/331207761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 msgr2=0x7ffa14071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 --2- 172.21.15.142:0/331207761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 -- 172.21.15.142:0/331207761 >> v1:172.21.15.150:6789/0 conn(0x7ffa14071970 legacy=0x7ffa14138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 monclient: found mon.noname-c 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:18:54.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.480+0000 7ffa13fff700 1 -- 172.21.15.142:0/331207761 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa140a3250 con 0x7ffa140a2e80 2024-09-18T10:18:54.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa13fff700 1 -- 172.21.15.142:0/331207761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa140a2e80 msgr2=0x7ffa14070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa13fff700 1 --2- 172.21.15.142:0/331207761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa140a2e80 0x7ffa14070ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009f20 tx=0x7ffa04009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74d7007dcb9e7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 1 -- 172.21.15.142:0/331207761 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0400e070 con 0x7ffa140a2e80 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: dump: 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.497 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 1 -- 172.21.15.142:0/331207761 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0400bbe0 con 0x7ffa140a2e80 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 5 monclient: authenticate success, global_id 114151 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 1 -- 172.21.15.142:0/331207761 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa040092e0 con 0x7ffa140a2e80 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 4 set_mon_vals no callback set 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa12ffd700 10 monclient: dump: 2024-09-18T10:18:54.498 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: shutdown 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- 172.21.15.142:0/331207761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa140a2e80 msgr2=0x7ffa14070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 --2- 172.21.15.142:0/331207761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa140a2e80 0x7ffa14070ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009f20 tx=0x7ffa04009840 comp rx=0 tx=0).stop 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- 172.21.15.142:0/331207761 shutdown_connections 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 --2- 172.21.15.142:0/331207761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 --2- 172.21.15.142:0/331207761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa140a2e80 0x7ffa14070ef0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- 172.21.15.142:0/331207761 >> 172.21.15.142:0/331207761 conn(0x7ffa1406a4e0 msgr2=0x7ffa140a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- 172.21.15.142:0/331207761 shutdown_connections 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- 172.21.15.142:0/331207761 wait complete. 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: monmap: 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:54.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 Processor -- start 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 -- start start 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: init 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 0x7ffa141419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.484+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-b con 0x7ffa14071970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-c con 0x7ffa1409e050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa140a2e80 0x7ffa1408d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): picked mon.noname-a con 0x7ffa140a2e80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14090ba0 con 0x7ffa140a2e80 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1409e550 con 0x7ffa14071970 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140a1fe0 con 0x7ffa1409e050 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa1a434700 10 monclient(hunting): authenticate will time out at 1423.416111s 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa140a2e80 0x7ffa1408d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 0x7ffa141419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 0x7ffa141419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:50772/0 (socket says 172.21.15.142:50772) 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43768/0 (socket says 172.21.15.142:43768) 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 -- 172.21.15.142:0/875845496 learned_addr learned my addr 172.21.15.142:0/875845496 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): get_auth_request con 0x7ffa14071970 auth_method 0 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): get_auth_request con 0x7ffa1409e050 auth_method 0 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): get_auth_request con 0x7ffa140a2e80 auth_method 0 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa189d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient(hunting): handle_auth_done global_id 114154 payload 824 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 msgr2=0x7ffa14141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1409e050 0x7ffa14141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa140a2e80 msgr2=0x7ffa1408d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa140a2e80 0x7ffa1408d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 monclient: found mon.noname-b 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04009ae0 con 0x7ffa14071970 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 msgr2=0x7ffa141419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.488+0000 7ffa13fff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 0x7ffa141419e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400b610 tx=0x7ffa04017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04012850 con 0x7ffa14071970 2024-09-18T10:18:54.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: dump: 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa040129b0 con 0x7ffa14071970 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04012d70 con 0x7ffa14071970 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 5 monclient: authenticate success, global_id 114154 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 monclient: _renew_subs 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1408ed90 con 0x7ffa14071970 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa127fc700 4 set_mon_vals no callback set 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 10 monclient: dump: 2024-09-18T10:18:54.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 monclient: _renew_subs 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1408f960 con 0x7ffa14071970 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 0x7ffa1408e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14064a50 con 0x7ffa14142460 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 0x7ffa1408e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient: get_auth_request con 0x7ffa14142460 auth_method 0 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.492+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffa0401d630 con 0x7ffa14071970 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa137fe700 10 monclient(hunting): handle_auth_done global_id 74948 payload 824 2024-09-18T10:18:54.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa137fe700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa140696a0 con 0x7ffa14142460 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa00035b10 0x7ffa00037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ffa0404a370 con 0x7ffa14071970 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa189d1700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa00035b10 0x7ffa00037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa189d1700 10 monclient: get_auth_request con 0x7ffa00035b10 auth_method 0 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa137fe700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 msgr2=0x7ffa1408e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa137fe700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 0x7ffa1408e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa08011c20 tx=0x7ffa080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800fe80 con 0x7ffa14142460 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 10 monclient: dump: 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.513 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ffa08014440 con 0x7ffa14142460 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 msgr2=0x7ffa1408e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 0x7ffa1408e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa08011c20 tx=0x7ffa080091a0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 msgr2=0x7ffa1408e2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ff9fbfff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa14142460 0x7ffa1408e2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.496+0000 7ffa189d1700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa00035b10 0x7ffa00037fc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006fd0 tx=0x7ffa0c006f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa1a434700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa1a434700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 0x7ffa14029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa1a434700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa14090de0 con 0x7ffa1408e830 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 0x7ffa14029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient: get_auth_request con 0x7ffa1408e830 auth_method 0 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 10 monclient(hunting): handle_auth_done global_id 74951 payload 824 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.508+0000 7ffa137fe700 1 -- 172.21.15.142:0/875845496 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa1406d320 con 0x7ffa1408e830 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa137fe700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 msgr2=0x7ffa14029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa137fe700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 0x7ffa14029d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffa08011c20 tx=0x7ffa0801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa080273f0 con 0x7ffa1408e830 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 10 monclient: dump: 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.519 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7ffa08023040 con 0x7ffa1408e830 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 msgr2=0x7ffa14029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 0x7ffa14029d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffa08011c20 tx=0x7ffa0801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 msgr2=0x7ffa14029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ff9fbfff700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa1408e830 0x7ffa14029d00 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffa08011c20 tx=0x7ffa0801c000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:54.521 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.522 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:54.524 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.525 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.526 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.527 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.530 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.531 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:18:54.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.533 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:54.534 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:54.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa00035b10 msgr2=0x7ffa00037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa00035b10 0x7ffa00037fc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c006fd0 tx=0x7ffa0c006f60 comp rx=0 tx=0).stop 2024-09-18T10:18:54.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 10 monclient: shutdown 2024-09-18T10:18:54.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 msgr2=0x7ffa141419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 --2- 172.21.15.142:0/875845496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa14071970 0x7ffa141419e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400b610 tx=0x7ffa04017000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa189d1700 1 -- 172.21.15.142:0/875845496 reap_dead start 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa189d1700 1 -- 172.21.15.142:0/875845496 reap_dead start 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 shutdown_connections 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 >> 172.21.15.142:0/875845496 conn(0x7ffa1406a4e0 msgr2=0x7ffa140984b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 shutdown_connections 2024-09-18T10:18:54.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.512+0000 7ffa1a434700 1 -- 172.21.15.142:0/875845496 wait complete. 2024-09-18T10:18:54.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:54.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:18:54.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:18:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: get_monmap_and_config 2024-09-18T10:18:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: monmap: 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:54.644857+0000 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:54.644857+0000 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:54.647 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 Processor -- start 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 -- start start 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: init 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae9c0a00e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fae9c09cdf0 0x7fae9c09d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-a con 0x7fae9c09cdf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae9c09d720 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c069dc0 con 0x7fae9c0a00e0 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c069ae0 con 0x7fae9c09d720 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06ad00 con 0x7fae9c09cdf0 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7faea0d64700 10 monclient(hunting): authenticate will time out at 1423.570370s 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fae9c0a00e0 0x7fae9c06b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55994/0 (socket says 172.21.15.142:55994) 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fae9c09cdf0 0x7fae9c09d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9b7fe700 1 -- 172.21.15.142:0/4117651198 learned_addr learned my addr 172.21.15.142:0/4117651198 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): get_auth_request con 0x7fae9c09cdf0 auth_method 0 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient(hunting): handle_auth_done global_id 94572 payload 274 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 1 -- 172.21.15.142:0/4117651198 >> v1:172.21.15.150:6789/0 conn(0x7fae9c09d720 legacy=0x7fae9c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 1 -- 172.21.15.142:0/4117651198 >> v1:172.21.15.142:6789/0 conn(0x7fae9c0a00e0 legacy=0x7fae9c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 1 monclient: found mon.noname-a 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:18:54.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.640+0000 7fae9affd700 1 -- 172.21.15.142:0/4117651198 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c13cab0 con 0x7fae9c09cdf0 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9affd700 1 -- 172.21.15.142:0/4117651198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09cdf0 msgr2=0x7fae9c09d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9affd700 1 --2- 172.21.15.142:0/4117651198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09cdf0 0x7fae9c09d1e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae94009e60 tx=0x7fae94009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd763212bfe9790 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 1 -- 172.21.15.142:0/4117651198 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9400e070 con 0x7fae9c09cdf0 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: dump: 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.656 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 5 monclient: authenticate success, global_id 94572 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 1 -- 172.21.15.142:0/4117651198 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94012430 con 0x7fae9c09cdf0 2024-09-18T10:18:54.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:54.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 1 -- 172.21.15.142:0/4117651198 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940127d0 con 0x7fae9c09cdf0 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 4 set_mon_vals no callback set 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7fae9a7fc700 10 monclient: dump: 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.661 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 10 monclient: shutdown 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 -- 172.21.15.142:0/4117651198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09cdf0 msgr2=0x7fae9c09d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 --2- 172.21.15.142:0/4117651198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09cdf0 0x7fae9c09d1e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fae94009e60 tx=0x7fae94009780 comp rx=0 tx=0).stop 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 -- 172.21.15.142:0/4117651198 shutdown_connections 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 --2- 172.21.15.142:0/4117651198 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09cdf0 0x7fae9c09d1e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 -- 172.21.15.142:0/4117651198 >> 172.21.15.142:0/4117651198 conn(0x7fae9c096a50 msgr2=0x7fae9c071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 -- 172.21.15.142:0/4117651198 shutdown_connections 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 1 -- 172.21.15.142:0/4117651198 wait complete. 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.644+0000 7faea0d64700 10 monclient: monmap: 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:54.663 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 Processor -- start 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 -- start start 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient: init 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-c con 0x7fae9c09cdf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 0x7fae9c134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-a con 0x7fae9c09d720 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c0a00e0 0x7fae9c134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): picked mon.noname-b con 0x7fae9c0a00e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c06b7f0 con 0x7fae9c09d720 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c13c360 con 0x7fae9c0a00e0 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c136b40 con 0x7fae9c09cdf0 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7faea0d64700 10 monclient(hunting): authenticate will time out at 1423.576983s 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 0x7fae9c134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43802/0 (socket says 172.21.15.142:43802) 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 0x7fae9c134890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42980/0 (socket says 172.21.15.142:42980) 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 -- 172.21.15.142:0/132863531 learned_addr learned my addr 172.21.15.142:0/132863531 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): get_auth_request con 0x7fae9c09cdf0 auth_method 0 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): get_auth_request con 0x7fae9c09d720 auth_method 0 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c0a00e0 0x7fae9c134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): get_auth_request con 0x7fae9c0a00e0 auth_method 0 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient(hunting): handle_auth_done global_id 74954 payload 824 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c0a00e0 msgr2=0x7fae9c134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c0a00e0 0x7fae9c134dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 msgr2=0x7fae9c134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 0x7fae9c134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9b7fe700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94009a20 con 0x7fae9c09cdf0 2024-09-18T10:18:54.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.648+0000 7fae9affd700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fae9c09d720 0x7fae9c134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9b7fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 msgr2=0x7fae9c134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9b7fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c5e0 tx=0x7fae90004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90005d40 con 0x7fae9c09cdf0 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:54.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: dump: 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 5 monclient: authenticate success, global_id 74954 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae900070d0 con 0x7fae9c09cdf0 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 monclient: _renew_subs 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c136860 con 0x7fae9c09cdf0 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90007490 con 0x7fae9c09cdf0 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 10 monclient: dump: 2024-09-18T10:18:54.676 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae99ffb700 4 set_mon_vals no callback set 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 monclient: _renew_subs 2024-09-18T10:18:54.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c13a7c0 con 0x7fae9c09cdf0 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 0x7fae9c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c064810 con 0x7fae9c06dc90 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 0x7fae9c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient: get_auth_request con 0x7fae9c06dc90 auth_method 0 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 10 monclient(hunting): handle_auth_done global_id 114160 payload 824 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9affd700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae9c136fb0 con 0x7fae9c06dc90 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fae9001b3b0 con 0x7fae9c09cdf0 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae88035ac0 0x7fae88037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fae90048430 con 0x7fae9c09cdf0 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9bfff700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae88035ac0 0x7fae88037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.652+0000 7fae9bfff700 10 monclient: get_auth_request con 0x7fae88035ac0 auth_method 0 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae9affd700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 msgr2=0x7fae9c06e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae9affd700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 0x7fae9c06e060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae94009e60 tx=0x7fae94018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94022070 con 0x7fae9c06dc90 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 10 monclient: dump: 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.680 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fae9401d9f0 con 0x7fae9c06dc90 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 msgr2=0x7fae9c06e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 0x7fae9c06e060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae94009e60 tx=0x7fae94018000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 msgr2=0x7fae9c06e060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae837fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06dc90 0x7fae9c06e060 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae94009e60 tx=0x7fae94018000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.656+0000 7fae9bfff700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae88035ac0 0x7fae88037f70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fae9c09f160 tx=0x7fae8c007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7faea0d64700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7faea0d64700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 0x7fae9c14bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7faea0d64700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c135510 con 0x7fae9c06e5a0 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 0x7fae9c14bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient: get_auth_request con 0x7fae9c06e5a0 auth_method 0 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 10 monclient(hunting): handle_auth_done global_id 114163 payload 824 2024-09-18T10:18:54.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.664+0000 7fae9affd700 1 -- 172.21.15.142:0/132863531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fae9c136d60 con 0x7fae9c06e5a0 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae9affd700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 msgr2=0x7fae9c14bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae9affd700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 0x7fae9c14bce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae9c137370 tx=0x7fae94022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94031830 con 0x7fae9c06e5a0 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 10 monclient: dump: 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.684 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fae9401c410 con 0x7fae9c06e5a0 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 msgr2=0x7fae9c14bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 0x7fae9c14bce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae9c137370 tx=0x7fae94022000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 msgr2=0x7fae9c14bce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae837fe700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae9c06e5a0 0x7fae9c14bce0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fae9c137370 tx=0x7fae94022000 comp rx=0 tx=0).stop 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:54.686 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 102, 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:18:54.687 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:18:54.688 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.689 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:18:54.690 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:18:54.691 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.694 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:18:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:18:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:18:54.696 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:18:54.697 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:18:54.698 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:18:54.699 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.699 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:18:54.699 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:18:54.699 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.700 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:18:54.701 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:18:54.702 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:18:54.703 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae88035ac0 msgr2=0x7fae88037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fae88035ac0 0x7fae88037f70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fae9c09f160 tx=0x7fae8c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 10 monclient: shutdown 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 msgr2=0x7fae9c134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7fae9bfff700 1 -- 172.21.15.142:0/132863531 reap_dead start 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c5e0 tx=0x7fae90004a80 comp rx=0 tx=0).stop 2024-09-18T10:18:54.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 shutdown_connections 2024-09-18T10:18:54.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 --2- 172.21.15.142:0/132863531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fae9c09cdf0 0x7fae9c134350 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 >> 172.21.15.142:0/132863531 conn(0x7fae9c096a50 msgr2=0x7fae9c0a1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 shutdown_connections 2024-09-18T10:18:54.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.668+0000 7faea0d64700 1 -- 172.21.15.142:0/132863531 wait complete. 2024-09-18T10:18:54.706 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:18:54.706 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:18:54.706 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:18:54.706 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:18:54.706 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:18:54.707 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:18:54.707 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:18:54.725 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:18:54.725 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:18:54.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 10 monclient: get_monmap_and_config 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 10 monclient: monmap: 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:54.878227+0000 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:54.878227+0000 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:54.881 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:54.881 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:54.881 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:54.881 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:54.881 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 1 Processor -- start 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 1 -- start start 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.872+0000 7f8e37fff700 10 monclient: init 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T10:18:54.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e3809b050 addr v1:172.21.15.150:6789/0 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 0x7f8e380953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8e38098080 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8e38133780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3809a570 con 0x7f8e3809b050 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3806f900 con 0x7f8e38098080 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3809b420 con 0x7f8e38133780 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e37fff700 10 monclient(hunting): authenticate will time out at 1423.804267s 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e367fc700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 msgr2=0x7f8e380953c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:18:54.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e367fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 0x7f8e380953c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43804/0 (socket says 172.21.15.142:43804) 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 -- 172.21.15.142:0/1226574589 learned_addr learned my addr 172.21.15.142:0/1226574589 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): get_auth_request con 0x7f8e38133780 auth_method 0 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient(hunting): handle_auth_done global_id 74957 payload 274 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 -- 172.21.15.142:0/1226574589 >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 msgr2=0x7f8e380953c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 --2- 172.21.15.142:0/1226574589 >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 0x7f8e380953c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 -- 172.21.15.142:0/1226574589 >> v1:172.21.15.150:6789/0 conn(0x7f8e3809b050 legacy=0x7f8e38097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 monclient: found mon.noname-b 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 -- 172.21.15.142:0/1226574589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3809a290 con 0x7f8e38133780 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 -- 172.21.15.142:0/1226574589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 msgr2=0x7f8e38135b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e377fe700 1 --2- 172.21.15.142:0/1226574589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806fc80 tx=0x7f8e30009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9210406c89ac863b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e35ffb700 1 -- 172.21.15.142:0/1226574589 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000e070 con 0x7f8e38133780 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.876+0000 7f8e35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: dump: 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.888 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 5 monclient: authenticate success, global_id 74957 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:54.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 1 -- 172.21.15.142:0/1226574589 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3000bc60 con 0x7f8e38133780 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 1 -- 172.21.15.142:0/1226574589 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30007400 con 0x7f8e38133780 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e35ffb700 10 monclient: dump: 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.890 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 4 set_mon_vals no callback set 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 10 monclient: shutdown 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- 172.21.15.142:0/1226574589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 msgr2=0x7f8e38135b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 --2- 172.21.15.142:0/1226574589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806fc80 tx=0x7f8e30009840 comp rx=0 tx=0).stop 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- 172.21.15.142:0/1226574589 shutdown_connections 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 --2- 172.21.15.142:0/1226574589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38133780 0x7f8e38135b60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 --2- 172.21.15.142:0/1226574589 >> v2:172.21.15.142:3300/0 conn(0x7f8e38098080 0x7f8e380953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- 172.21.15.142:0/1226574589 >> 172.21.15.142:0/1226574589 conn(0x7f8e380a1fe0 msgr2=0x7f8e38070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- 172.21.15.142:0/1226574589 shutdown_connections 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- 172.21.15.142:0/1226574589 wait complete. 2024-09-18T10:18:54.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 10 monclient: build_initial_monmap 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 10 monclient: monmap: 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:54.892 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 Processor -- start 2024-09-18T10:18:54.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 1 -- start start 2024-09-18T10:18:54.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.880+0000 7f8e37fff700 10 monclient: init 2024-09-18T10:18:54.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:54.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8e38098080 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 0x7f8e3806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8e3809b050 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e38133780 0x7f8e380946f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8e38133780 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3808eea0 con 0x7f8e3809b050 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3808ec30 con 0x7f8e38133780 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3808ceb0 con 0x7f8e38098080 2024-09-18T10:18:54.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e367fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 msgr2=0x7f8e3806e3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e37fff700 10 monclient(hunting): authenticate will time out at 1423.812554s 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e367fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 0x7f8e3806e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43820/0 (socket says 172.21.15.142:43820) 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 -- 172.21.15.142:0/4164209062 learned_addr learned my addr 172.21.15.142:0/4164209062 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): get_auth_request con 0x7f8e38098080 auth_method 0 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e38133780 0x7f8e380946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): get_auth_request con 0x7f8e38133780 auth_method 0 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient(hunting): handle_auth_done global_id 74960 payload 824 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 -- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e38133780 msgr2=0x7f8e380946f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e38133780 0x7f8e380946f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 -- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 msgr2=0x7f8e3806e3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 0x7f8e3806e3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:54.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 monclient: found mon.noname-c 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 -- 172.21.15.142:0/4164209062 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30009ae0 con 0x7f8e38098080 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 -- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 msgr2=0x7f8e3808c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.884+0000 7f8e36ffd700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800c5d0 tx=0x7f8e28007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e280117a0 con 0x7f8e38098080 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: dump: 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: _finish_auth 0 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 5 monclient: authenticate success, global_id 74960 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e28011900 con 0x7f8e38098080 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:54.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e28011cc0 con 0x7f8e38098080 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _renew_subs 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3808f180 con 0x7f8e38098080 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 10 monclient: dump: 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:54.903 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e357fa700 4 set_mon_vals no callback set 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _renew_subs 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e38094c30 con 0x7f8e38098080 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3806eea0 con 0x7f8e38098080 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8e28012630 con 0x7f8e38098080 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 0x7f8e24037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:54.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f8e28048cd0 con 0x7f8e38098080 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e367fc700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 0x7f8e24037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.888+0000 7f8e367fc700 10 monclient: get_auth_request con 0x7f8e24035a70 auth_method 0 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.892+0000 7f8e367fc700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 0x7f8e24037f20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806eb20 tx=0x7f8e2c006dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.896+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f8e2801c460 con 0x7f8e38098080 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.896+0000 7f8e1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:54.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:54.896+0000 7f8e1effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:55.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e37fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e37fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e37fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8e38072a80 con 0x7f8e38098080 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e1effd700 1 -- 172.21.15.142:0/4164209062 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f8e2801bed0 con 0x7f8e38098080 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:55.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.120+0000 7f8e1effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:55.128 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 msgr2=0x7f8e24037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 0x7f8e24037f20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8e3806eb20 tx=0x7f8e2c006dc0 comp rx=0 tx=0).stop 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 10 monclient: shutdown 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 msgr2=0x7f8e3808c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800c5d0 tx=0x7f8e28007f60 comp rx=0 tx=0).stop 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 shutdown_connections 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8e24035a70 0x7f8e24037f20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e38133780 0x7f8e380946f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:55.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8e3809b050 0x7f8e3806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:55.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 --2- 172.21.15.142:0/4164209062 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8e38098080 0x7f8e3808c8c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:55.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 >> 172.21.15.142:0/4164209062 conn(0x7f8e380a1fe0 msgr2=0x7f8e38095930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:55.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 shutdown_connections 2024-09-18T10:18:55.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:55.124+0000 7f8e37fff700 1 -- 172.21.15.142:0/4164209062 wait complete. 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 13 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar' 2024-09-18T10:18:55.541 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:18:55.542 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T10:18:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T10:18:55.542 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo auth del client.bazar 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 13 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump' 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T10:18:56.492 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T10:18:56.493 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo osd dump 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 110 2024-09-18T10:18:56.769 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump' 2024-09-18T10:18:56.770 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=110 2024-09-18T10:18:56.770 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T10:18:56.770 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T10:18:56.770 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo pg dump 2024-09-18T10:18:58.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: get_monmap_and_config 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: build_initial_monmap 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: monmap: 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:18:58.253205+0000 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:18:58.253205+0000 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.255 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:18:58.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:18:58.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:18:58.256 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:18:58.256 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:18:58.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 Processor -- start 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 -- start start 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: init 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-b con 0x7f15e8132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 0x7f15e809f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-a con 0x7f15e809ece0 addr v2:172.21.15.142:3300/0 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15e809ae60 addr v1:172.21.15.142:6789/0 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e8132940 con 0x7f15e809ae60 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e809f960 con 0x7f15e809ece0 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80a1a90 con 0x7f15e8132030 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15eca10700 10 monclient(hunting): authenticate will time out at 1427.179166s 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e759e700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f15e809ae60 legacy=0x7f15e809b300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e659c700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 msgr2=0x7f15e809f0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:18:58.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e659c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 0x7f15e809f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43836/0 (socket says 172.21.15.142:43836) 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 1 -- 172.21.15.142:0/125791844 learned_addr learned my addr 172.21.15.142:0/125791844 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): get_auth_request con 0x7f15e8132030 auth_method 0 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.248+0000 7f15e6d9d700 10 monclient(hunting): handle_auth_done global_id 74972 payload 274 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 -- 172.21.15.142:0/125791844 >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 msgr2=0x7f15e809f0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:58.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/125791844 >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 0x7f15e809f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 -- 172.21.15.142:0/125791844 >> v1:172.21.15.142:6789/0 conn(0x7f15e809ae60 legacy=0x7f15e809b300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 monclient: found mon.noname-b 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 -- 172.21.15.142:0/125791844 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80a17b0 con 0x7f15e8132030 2024-09-18T10:18:58.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 -- 172.21.15.142:0/125791844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 msgr2=0x7f15e8132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:58.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/125791844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f15e00099b0 tx=0x7f15e0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ca5bcaed78ca54 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:58.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 1 -- 172.21.15.142:0/125791844 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e000bb50 con 0x7f15e8132030 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: dump: 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:58.267 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: _finish_auth 0 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 5 monclient: authenticate success, global_id 74972 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 1 -- 172.21.15.142:0/125791844 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e000bcb0 con 0x7f15e8132030 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 10 monclient: get_monmap_and_config success 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 4 set_mon_vals no callback set 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 1 -- 172.21.15.142:0/125791844 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0011670 con 0x7f15e8132030 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:58.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15e5d9b700 10 monclient: dump: 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 10 monclient: shutdown 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:58.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 -- 172.21.15.142:0/125791844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 msgr2=0x7f15e8132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 --2- 172.21.15.142:0/125791844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f15e00099b0 tx=0x7f15e0009250 comp rx=0 tx=0).stop 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 -- 172.21.15.142:0/125791844 shutdown_connections 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 --2- 172.21.15.142:0/125791844 >> v2:172.21.15.142:3300/0 conn(0x7f15e809ece0 0x7f15e809f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 --2- 172.21.15.142:0/125791844 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e8132030 0x7f15e8132400 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 -- 172.21.15.142:0/125791844 >> 172.21.15.142:0/125791844 conn(0x7f15e80a5700 msgr2=0x7f15e80a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 -- 172.21.15.142:0/125791844 shutdown_connections 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.252+0000 7f15eca10700 1 -- 172.21.15.142:0/125791844 wait complete. 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient: build_initial_monmap 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient: monmap: 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:18:58.270 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 Processor -- start 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 -- start start 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient: init 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient: _reopen_session rank -1 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 0x7f15e806ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-b con 0x7f15e809ae60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:18:58.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-c con 0x7f15e809ece0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 0x7f15e8131240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): picked mon.noname-a con 0x7f15e8132030 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e806f600 con 0x7f15e8132030 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e809f6e0 con 0x7f15e809ae60 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): start opening mon connection 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80a4960 con 0x7f15e809ece0 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): _renew_subs 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15eca10700 10 monclient(hunting): authenticate will time out at 1427.185259s 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e759e700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 msgr2=0x7f15e8131240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:18:58.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 0x7f15e8131240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43844/0 (socket says 172.21.15.142:43844) 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 -- 172.21.15.142:0/127822610 learned_addr learned my addr 172.21.15.142:0/127822610 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): get_auth_request con 0x7f15e809ece0 auth_method 0 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 0x7f15e806ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient(hunting): handle_auth_done global_id 74975 payload 824 2024-09-18T10:18:58.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 -- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 msgr2=0x7f15e806ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 0x7f15e806ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 -- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 msgr2=0x7f15e8131240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 0x7f15e8131240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient: _finish_hunting 0 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 monclient: found mon.noname-c 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e659c700 1 -- 172.21.15.142:0/127822610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e00094f0 con 0x7f15e809ece0 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e6d9d700 10 monclient: get_auth_request con 0x7f15e809ae60 auth_method 0 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.256+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 0x7f15e806ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15e659c700 1 -- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 msgr2=0x7f15e811c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15e659c700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15d000bbb0 tx=0x7f15d00097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:58.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d000cbb0 con 0x7f15e809ece0 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: dump: 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:58.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: _finish_auth 0 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d000cd10 con 0x7f15e809ece0 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 5 monclient: authenticate success, global_id 74975 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _renew_subs 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e811e640 con 0x7f15e809ece0 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0016430 con 0x7f15e809ece0 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 10 monclient: dump: 2024-09-18T10:18:58.276 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15e559a700 4 set_mon_vals no callback set 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _renew_subs 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e8131920 con 0x7f15e809ece0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f15d0012020 con 0x7f15e809ece0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 0x7f15cc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f15d0048e40 con 0x7f15e809ece0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e8072a80 con 0x7f15e809ece0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 0x7f15cc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.260+0000 7f15e6d9d700 10 monclient: get_auth_request con 0x7f15cc035a70 auth_method 0 2024-09-18T10:18:58.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.264+0000 7f15e6d9d700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 0x7f15cc037f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f15e00099b0 tx=0x7f15e0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:18:58.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.268+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f15d0006150 con 0x7f15e809ece0 2024-09-18T10:18:58.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.268+0000 7f15deffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:18:58.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.268+0000 7f15deffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:18:58.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.492+0000 7f15eca10700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:18:58.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.492+0000 7f15eca10700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:58.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.492+0000 7f15eca10700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:18:58.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.492+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f15e811dfd0 con 0x7f15e809ece0 2024-09-18T10:18:58.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:58.496+0000 7f15e659c700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15d000bbb0 tx=0x7f15d00097b0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f15d0024070 mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 2024-09-18T10:18:59.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.324+0000 7f15deffd700 1 -- 172.21.15.142:0/127822610 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f15d0024070 con 0x7f15e809ece0 2024-09-18T10:18:59.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.324+0000 7f15deffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:18:59.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.324+0000 7f15deffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:18:59.330 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:18:59.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 msgr2=0x7f15cc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:59.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 0x7f15cc037f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f15e00099b0 tx=0x7f15e0016000 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 10 monclient: shutdown 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 msgr2=0x7f15e811c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f15d000bbb0 tx=0x7f15d00097b0 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 shutdown_connections 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15cc035a70 0x7f15cc037f20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15e8132030 0x7f15e8131240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15e809ece0 0x7f15e811c240 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 --2- 172.21.15.142:0/127822610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e809ae60 0x7f15e806ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 >> 172.21.15.142:0/127822610 conn(0x7f15e80a5700 msgr2=0x7f15e80a4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:18:59.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 shutdown_connections 2024-09-18T10:18:59.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:18:59.328+0000 7f15eca10700 1 -- 172.21.15.142:0/127822610 wait complete. 2024-09-18T10:19:02.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:02.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: get_monmap_and_config 2024-09-18T10:19:02.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: build_initial_monmap 2024-09-18T10:19:02.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: monmap: 2024-09-18T10:19:02.463 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:02.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:02.461592+0000 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:02.461592+0000 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:02.464 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 Processor -- start 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 -- start start 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: init 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 0x7f515812e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-c con 0x7f515809b0b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-b con 0x7f515809b6f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f515812f120 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:02.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515809b480 con 0x7f515812f120 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515806f940 con 0x7f515809b0b0 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5158098cd0 con 0x7f515809b6f0 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515fac6700 10 monclient(hunting): authenticate will time out at 1431.387390s 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515e063700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f515812f120 legacy=0x7f515809cc30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53366/0 (socket says 172.21.15.142:53366) 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 1 -- 172.21.15.142:0/3912208285 learned_addr learned my addr 172.21.15.142:0/3912208285 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 10 monclient(hunting): get_auth_request con 0x7f515809b6f0 auth_method 0 2024-09-18T10:19:02.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d862700 1 --2- 172.21.15.142:0/3912208285 >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 0x7f515812e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d061700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d862700 10 monclient(hunting): get_auth_request con 0x7f515809b0b0 auth_method 0 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d862700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d862700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.456+0000 7f515d862700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:02.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient(hunting): handle_auth_done global_id 74978 payload 274 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 -- 172.21.15.142:0/3912208285 >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 msgr2=0x7f515812e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 --2- 172.21.15.142:0/3912208285 >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 0x7f515812e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 -- 172.21.15.142:0/3912208285 >> v1:172.21.15.142:6789/0 conn(0x7f515812f120 legacy=0x7f515809cc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient: _finish_hunting 0 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 monclient: found mon.noname-b 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 -- 172.21.15.142:0/3912208285 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158099110 con 0x7f515809b6f0 2024-09-18T10:19:02.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d862700 1 --2- 172.21.15.142:0/3912208285 >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 0x7f515812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:02.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 -- 172.21.15.142:0/3912208285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 msgr2=0x7f515812ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:02.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515d061700 1 --2- 172.21.15.142:0/3912208285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51480099b0 tx=0x7f5148009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9292ba4473bd4731 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:02.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 1 -- 172.21.15.142:0/3912208285 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148009ea0 con 0x7f515809b6f0 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: dump: 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:02.475 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: _finish_auth 0 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515fac6700 5 monclient: authenticate success, global_id 74978 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 1 -- 172.21.15.142:0/3912208285 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514800ba70 con 0x7f515809b6f0 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515fac6700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 1 -- 172.21.15.142:0/3912208285 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514800be30 con 0x7f515809b6f0 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515fac6700 4 set_mon_vals no callback set 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.460+0000 7f515c860700 10 monclient: dump: 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:02.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: shutdown 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- 172.21.15.142:0/3912208285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 msgr2=0x7f515812ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- 172.21.15.142:0/3912208285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f51480099b0 tx=0x7f5148009250 comp rx=0 tx=0).stop 2024-09-18T10:19:02.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- 172.21.15.142:0/3912208285 shutdown_connections 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- 172.21.15.142:0/3912208285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f515812ebe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- 172.21.15.142:0/3912208285 >> v2:172.21.15.150:3300/0 conn(0x7f515809b0b0 0x7f515812e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- 172.21.15.142:0/3912208285 >> 172.21.15.142:0/3912208285 conn(0x7f51580a14e0 msgr2=0x7f51580a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- 172.21.15.142:0/3912208285 shutdown_connections 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- 172.21.15.142:0/3912208285 wait complete. 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: build_initial_monmap 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: monmap: 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:02.478 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 Processor -- start 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- start start 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: init 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f515809b0b0 0x7f5158141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-b con 0x7f515809b0b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-c con 0x7f515809b6f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:02.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 0x7f515812b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): picked mon.noname-a con 0x7f515812f120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580888a0 con 0x7f515812f120 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51580882e0 con 0x7f515809b0b0 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515812cf20 con 0x7f515809b6f0 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515fac6700 10 monclient(hunting): authenticate will time out at 1431.395067s 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515e063700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 msgr2=0x7f515812b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515e063700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 0x7f515812b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:02.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53376/0 (socket says 172.21.15.142:53376) 2024-09-18T10:19:02.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 1 -- 172.21.15.142:0/3418403604 learned_addr learned my addr 172.21.15.142:0/3418403604 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:02.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): get_auth_request con 0x7f515809b6f0 auth_method 0 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f515809b0b0 0x7f5158141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): get_auth_request con 0x7f515809b0b0 auth_method 0 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:02.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d862700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 10 monclient(hunting): handle_auth_done global_id 74981 payload 824 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.464+0000 7f515d061700 1 -- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f515809b0b0 msgr2=0x7f5158141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f515809b0b0 0x7f5158141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 -- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 msgr2=0x7f515812b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 0x7f515812b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 10 monclient: _finish_hunting 0 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 monclient: found mon.noname-c 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 -- 172.21.15.142:0/3418403604 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515400ab30 con 0x7f515809b6f0 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 -- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 msgr2=0x7f5158141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515d061700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f51580a2530 tx=0x7f5148016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148006b60 con 0x7f515809b6f0 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: dump: 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.486 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 5 monclient: authenticate success, global_id 74981 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148020070 con 0x7f515809b6f0 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148011e10 con 0x7f515809b6f0 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:02.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514f7fe700 4 set_mon_vals no callback set 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f514dffb700 10 monclient: dump: 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:02.488 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 10 monclient: _renew_subs 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51480094f0 con 0x7f515809b6f0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 10 monclient: _renew_subs 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.468+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51580885c0 con 0x7f515809b6f0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515fac6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515fac6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515fac6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51580696a0 con 0x7f515809b6f0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5148022030 con 0x7f515809b6f0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f514dffb700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 0x7f513c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f514804ac20 con 0x7f515809b6f0 2024-09-18T10:19:02.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515d862700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 0x7f513c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.472+0000 7f515d862700 10 monclient: get_auth_request con 0x7f513c035a70 auth_method 0 2024-09-18T10:19:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.476+0000 7f515d862700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 0x7f513c037f20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5154008650 tx=0x7f515400a890 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.480+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f5148019750 con 0x7f515809b6f0 2024-09-18T10:19:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.480+0000 7f514dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:02.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.480+0000 7f514dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:02.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f515fac6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:02.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f515fac6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:02.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f515fac6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:02.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f51580696a0 con 0x7f515809b6f0 2024-09-18T10:19:02.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f514dffb700 1 -- 172.21.15.142:0/3418403604 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f514802c7a0 con 0x7f515809b6f0 2024-09-18T10:19:02.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f514dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:02.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.712+0000 7f514dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:02.720 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:02.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 msgr2=0x7f513c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:02.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 0x7f513c037f20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5154008650 tx=0x7f515400a890 comp rx=0 tx=0).stop 2024-09-18T10:19:02.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 10 monclient: shutdown 2024-09-18T10:19:02.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 msgr2=0x7f5158141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f51580a2530 tx=0x7f5148016000 comp rx=0 tx=0).stop 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 shutdown_connections 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f513c035a70 0x7f513c037f20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f515812f120 0x7f515812b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f515809b6f0 0x7f5158141fa0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 --2- 172.21.15.142:0/3418403604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f515809b0b0 0x7f5158141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 >> 172.21.15.142:0/3418403604 conn(0x7f51580a14e0 msgr2=0x7f5158097d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 shutdown_connections 2024-09-18T10:19:02.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:02.716+0000 7f515fac6700 1 -- 172.21.15.142:0/3418403604 wait complete. 2024-09-18T10:19:05.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.836+0000 7f26b74db700 10 monclient: get_monmap_and_config 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.836+0000 7f26b74db700 10 monclient: build_initial_monmap 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.836+0000 7f26b74db700 10 monclient: monmap: 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:05.843017+0000 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:05.843017+0000 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:05.845 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:05.846 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:05.846 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:05.846 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:05.846 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:05.846 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 Processor -- start 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 -- start start 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient: init 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26b009c700 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-c con 0x7f26b0097b10 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-b con 0x7f26b0137750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:05.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b00970a0 con 0x7f26b009c700 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5277700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f26b009c700 legacy=0x7f26b009cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b0097f10 con 0x7f26b0097b10 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b009e240 con 0x7f26b0137750 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b74db700 10 monclient(hunting): authenticate will time out at 1434.769047s 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53386/0 (socket says 172.21.15.142:53386) 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 learned_addr learned my addr 172.21.15.142:0/3385094968 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 10 monclient(hunting): get_auth_request con 0x7f26b0137750 auth_method 0 2024-09-18T10:19:05.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f26b0097b10 auth_method 0 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:05.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:05.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:05.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:05.840+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:06.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.142:6789/0 conn(0x7f26b009c700 legacy=0x7f26b009cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:06.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 10 monclient(hunting): get_auth_request con 0x7f26b0137750 auth_method 0 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f26b0097b10 auth_method 0 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:06.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:06.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.044+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:06.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.142:6789/0 conn(0x7f26b009c700 legacy=0x7f26b009cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:06.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:06.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 10 monclient(hunting): get_auth_request con 0x7f26b0137750 auth_method 0 2024-09-18T10:19:06.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f26b0097b10 auth_method 0 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:19:06.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:06.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:06.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:06.448+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:19:07.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.142:6789/0 conn(0x7f26b009c700 legacy=0x7f26b009cad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 10 monclient(hunting): get_auth_request con 0x7f26b0137750 auth_method 0 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f26b0097b10 auth_method 0 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:19:07.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:07.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:07.252+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): tick 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 monclient(hunting): continuing hunt 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 msgr2=0x7f26b0139b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f26b0137750 0x7f26b0139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 msgr2=0x7f26b009d1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:08.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.150:3300/0 conn(0x7f26b0097b10 0x7f26b009d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.142:6789/0 conn(0x7f26b009c700 legacy=0x7f26b009cad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient: _add_conns ranks=[1,3] 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26940036e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 0x7f26940098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f2694007400 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 -- 172.21.15.142:0/3385094968 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b0096dc0 con 0x7f26940036e0 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 1 -- 172.21.15.142:0/3385094968 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b0097f10 con 0x7f2694007400 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26a6ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 msgr2=0x7f26940098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:08.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 0x7f26940098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:08.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:08.844+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.150:6789/0 conn(0x7f26940036e0 legacy=0x7f2694005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.048+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 msgr2=0x7f26940098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.048+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 0x7f26940098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:09.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465375846 0 0) 0x7f26b0096dc0 con 0x7f26940036e0 2024-09-18T10:19:09.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 10 monclient(hunting): my global_id is 114190 2024-09-18T10:19:09.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f269c003570 con 0x7f26940036e0 2024-09-18T10:19:09.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521066105 0 0) 0x7f269c003570 con 0x7f26940036e0 2024-09-18T10:19:09.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 msgr2=0x7f26940098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:09.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 --2- 172.21.15.142:0/3385094968 >> v2:172.21.15.142:3300/0 conn(0x7f2694007400 0x7f26940098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.100+0000 7f26a7fff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b00970a0 con 0x7f26940036e0 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f26ac004350 con 0x7f26940036e0 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b74db700 5 monclient: authenticate success, global_id 114190 2024-09-18T10:19:09.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b74db700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: dump: 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.116 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 >> v1:172.21.15.150:6789/0 conn(0x7f26940036e0 legacy=0x7f2694005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 reap_dead start 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 0x7f269c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): picked mon.b con 0x7f269c003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): picked mon.c con 0x7f269c004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 0x7f269c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): picked mon.a con 0x7f269c0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b0097f10 con 0x7f269c0053c0 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c00c320 con 0x7f269c003a60 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f269c00c4e0 con 0x7f269c004810 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26a7fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b74db700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5a78700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 msgr2=0x7f269c00ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5a78700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 0x7f269c00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 0x7f269c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f269c004810 auth_method 0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): get_auth_request con 0x7f269c003a60 auth_method 0 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 msgr2=0x7f269c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 msgr2=0x7f269c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:09.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient(hunting): handle_auth_done global_id 114190 payload 293 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 msgr2=0x7f269c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 msgr2=0x7f269c00ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 0x7f269c00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 monclient: found mon.b 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c003570 con 0x7f269c003a60 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.104+0000 7f26b5277700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b5277700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 msgr2=0x7f269c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b5277700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 0x7f269c003e50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac007bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6140199b69a363cd server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac007150 con 0x7f269c003a60 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: dump: 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.122 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac0072b0 con 0x7f269c003a60 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 1 -- 172.21.15.142:0/3385094968 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac007670 con 0x7f269c003a60 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26a7fff700 10 monclient: dump: 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.123 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:09.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 4 set_mon_vals no callback set 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 10 monclient: shutdown 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 -- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 msgr2=0x7f269c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 0x7f269c003e50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f26ac000c00 tx=0x7f26ac007bb0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 -- 172.21.15.142:0/3385094968 shutdown_connections 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c0053c0 0x7f269c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c004810 0x7f269c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 --2- 172.21.15.142:0/3385094968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c003a60 0x7f269c003e50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 -- 172.21.15.142:0/3385094968 >> 172.21.15.142:0/3385094968 conn(0x7f26b0071920 msgr2=0x7f26b00a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 -- 172.21.15.142:0/3385094968 shutdown_connections 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 1 -- 172.21.15.142:0/3385094968 wait complete. 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 10 monclient: build_initial_monmap 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.108+0000 7f26b74db700 10 monclient: monmap: 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:09.132 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 Processor -- start 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 -- start start 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient: init 2024-09-18T10:19:09.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 0x7f26b0119710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-a con 0x7f269c003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-b con 0x7f269c004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c0053c0 0x7f26b011a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): picked mon.noname-c con 0x7f269c0053c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b011d010 con 0x7f269c003a60 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b009e470 con 0x7f269c004810 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b009b5c0 con 0x7f269c0053c0 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b74db700 10 monclient(hunting): authenticate will time out at 1438.041775s 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5277700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 msgr2=0x7f26b0119710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5277700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 0x7f26b0119710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c0053c0 0x7f26b011a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:36844/0 (socket says 172.21.15.142:36844) 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 -- 172.21.15.142:0/3476251535 learned_addr learned my addr 172.21.15.142:0/3476251535 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): get_auth_request con 0x7f269c0053c0 auth_method 0 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): get_auth_request con 0x7f269c004810 auth_method 0 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b5a78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient(hunting): handle_auth_done global_id 114193 payload 824 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 -- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c0053c0 msgr2=0x7f26b011a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c0053c0 0x7f26b011a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 -- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 msgr2=0x7f26b0119710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 0x7f26b0119710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 monclient: found mon.noname-b 2024-09-18T10:19:09.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.112+0000 7f26b4a76700 1 -- 172.21.15.142:0/3476251535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b011cd30 con 0x7f269c004810 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b4a76700 1 -- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 msgr2=0x7f26b0119c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b4a76700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26a0016990 tx=0x7f26a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0016e80 con 0x7f269c004810 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: dump: 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.138 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 5 monclient: authenticate success, global_id 114193 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a001b070 con 0x7f269c004810 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _renew_subs 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b011b200 con 0x7f269c004810 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0014ca0 con 0x7f269c004810 2024-09-18T10:19:09.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a57fa700 10 monclient: dump: 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.141 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26a77fe700 4 set_mon_vals no callback set 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _renew_subs 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b011bdd0 con 0x7f269c004810 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.116+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b011c050 con 0x7f269c004810 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.120+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f26a001d030 con 0x7f269c004810 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.120+0000 7f26a57fa700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 0x7f26940386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.120+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f26a0049bd0 con 0x7f269c004810 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.120+0000 7f26b5277700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 0x7f26940386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.120+0000 7f26b5277700 10 monclient: get_auth_request con 0x7f2694036240 auth_method 0 2024-09-18T10:19:09.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.124+0000 7f26b5277700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 0x7f26940386f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f26ac0038a0 tx=0x7f26ac014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.132+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f26b011c050 con 0x7f269c004810 2024-09-18T10:19:09.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.132+0000 7f26a57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.132+0000 7f26a57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f26b011c050 con 0x7f269c004810 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26a57fa700 1 -- 172.21.15.142:0/3476251535 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f26b011c050 con 0x7f269c004810 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26a57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26a57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:09.366 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":106,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 msgr2=0x7f26940386f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 0x7f26940386f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f26ac0038a0 tx=0x7f26ac014000 comp rx=0 tx=0).stop 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 10 monclient: shutdown 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 msgr2=0x7f26b0119c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f26a0016990 tx=0x7f26a0008040 comp rx=0 tx=0).stop 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 shutdown_connections 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2694036240 0x7f26940386f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f269c0053c0 0x7f26b011a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f269c004810 0x7f26b0119c50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 --2- 172.21.15.142:0/3476251535 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f269c003a60 0x7f26b0119710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 >> 172.21.15.142:0/3476251535 conn(0x7f26b0071920 msgr2=0x7f26b009fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 shutdown_connections 2024-09-18T10:19:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.360+0000 7f26b74db700 1 -- 172.21.15.142:0/3476251535 wait complete. 2024-09-18T10:19:09.381 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:19:09.381 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:19:09.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: get_monmap_and_config 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: build_initial_monmap 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: monmap: 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:09.484339+0000 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:09.484339+0000 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.486 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:09.487 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:09.487 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:09.487 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:09.487 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:09.487 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 Processor -- start 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 -- start start 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: init 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73e406f3b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 0x7f73e4134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-a con 0x7f73e40990d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f73e40994f0 0x7f73e4136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-c con 0x7f73e40994f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e409c330 con 0x7f73e406f3b0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e409b380 con 0x7f73e40990d0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e409b030 con 0x7f73e40994f0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e359e700 10 monclient(hunting): authenticate will time out at 1438.410064s 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e1d9b700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 msgr2=0x7f73e4134670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e1d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 0x7f73e4134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f73e40994f0 0x7f73e4136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f73e40994f0 0x7f73e4136bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:36848/0 (socket says 172.21.15.142:36848) 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2038990210 learned_addr learned my addr 172.21.15.142:0/2038990210 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:09.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): get_auth_request con 0x7f73e40994f0 auth_method 0 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient(hunting): handle_auth_done global_id 114196 payload 274 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2038990210 >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 msgr2=0x7f73e4134670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 --2- 172.21.15.142:0/2038990210 >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 0x7f73e4134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2038990210 >> v1:172.21.15.150:6789/0 conn(0x7f73e406f3b0 legacy=0x7f73e40989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 monclient: found mon.noname-c 2024-09-18T10:19:09.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:19:09.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.480+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2038990210 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e409c530 con 0x7f73e40994f0 2024-09-18T10:19:09.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2038990210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40994f0 msgr2=0x7f73e4136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e2d9d700 1 --2- 172.21.15.142:0/2038990210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40994f0 0x7f73e4136bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73e409a7f0 tx=0x7f73dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b46861f910518338 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 1 -- 172.21.15.142:0/2038990210 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc00e070 con 0x7f73e40994f0 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: dump: 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 5 monclient: authenticate success, global_id 114196 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 1 -- 172.21.15.142:0/2038990210 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc00bc60 con 0x7f73e40994f0 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 4 set_mon_vals no callback set 2024-09-18T10:19:09.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 1 -- 172.21.15.142:0/2038990210 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc012430 con 0x7f73e40994f0 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e159a700 10 monclient: dump: 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 10 monclient: shutdown 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 -- 172.21.15.142:0/2038990210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40994f0 msgr2=0x7f73e4136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 --2- 172.21.15.142:0/2038990210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40994f0 0x7f73e4136bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73e409a7f0 tx=0x7f73dc009840 comp rx=0 tx=0).stop 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 -- 172.21.15.142:0/2038990210 shutdown_connections 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 --2- 172.21.15.142:0/2038990210 >> v2:172.21.15.142:3300/0 conn(0x7f73e40990d0 0x7f73e4134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 --2- 172.21.15.142:0/2038990210 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40994f0 0x7f73e4136bc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 -- 172.21.15.142:0/2038990210 >> 172.21.15.142:0/2038990210 conn(0x7f73e4070ef0 msgr2=0x7f73e40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 -- 172.21.15.142:0/2038990210 shutdown_connections 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 1 -- 172.21.15.142:0/2038990210 wait complete. 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 10 monclient: build_initial_monmap 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.484+0000 7f73e359e700 10 monclient: monmap: 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 Processor -- start 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 -- start start 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient: init 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:19:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 0x7f73e408a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-c con 0x7f73e406f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40990d0 0x7f73e408adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-b con 0x7f73e40990d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e40994f0 0x7f73e408b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): picked mon.noname-a con 0x7f73e40994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4091850 con 0x7f73e40994f0 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e406fa20 con 0x7f73e40990d0 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4097370 con 0x7f73e406f3b0 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e359e700 10 monclient(hunting): authenticate will time out at 1438.417063s 2024-09-18T10:19:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e2d9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e40994f0 msgr2=0x7f73e408b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e40994f0 0x7f73e408b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 0x7f73e408a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 0x7f73e408a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53450/0 (socket says 172.21.15.142:53450) 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 -- 172.21.15.142:0/2197982245 learned_addr learned my addr 172.21.15.142:0/2197982245 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): get_auth_request con 0x7f73e406f3b0 auth_method 0 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40990d0 0x7f73e408adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): get_auth_request con 0x7f73e40990d0 auth_method 0 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient(hunting): handle_auth_done global_id 74987 payload 824 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40990d0 msgr2=0x7f73e408adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73e40990d0 0x7f73e408adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e40994f0 msgr2=0x7f73e408b330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73e40994f0 0x7f73e408b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 monclient: found mon.noname-c 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.488+0000 7f73e259c700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc009ae0 con 0x7f73e406f3b0 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e259c700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 msgr2=0x7f73e408a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e259c700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 0x7f73e408a8b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c600 tx=0x7f73d4007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d40117a0 con 0x7f73e406f3b0 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: dump: 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 5 monclient: authenticate success, global_id 74987 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d4011900 con 0x7f73e406f3b0 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4011cc0 con 0x7f73e406f3b0 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 10 monclient: dump: 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e0d99700 4 set_mon_vals no callback set 2024-09-18T10:19:09.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 monclient: _renew_subs 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e408be60 con 0x7f73e406f3b0 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 monclient: _renew_subs 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e408b9e0 con 0x7f73e406f3b0 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 0x7f73e4141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40648a0 con 0x7f73e409c990 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 0x7f73e4141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient: get_auth_request con 0x7f73e409c990 auth_method 0 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_done global_id 74990 payload 824 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f73d4012630 con 0x7f73e406f3b0 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e1d9b700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73e4091320 con 0x7f73e409c990 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f73c8035a70 0x7f73c8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f73d4048c80 con 0x7f73e406f3b0 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e2d9d700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f73c8035a70 0x7f73c8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.492+0000 7f73e2d9d700 10 monclient: get_auth_request con 0x7f73c8035a70 auth_method 0 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73e1d9b700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 msgr2=0x7f73e4141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73e1d9b700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 0x7f73e4141f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009650 tx=0x7f73cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc012940 con 0x7f73e409c990 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 10 monclient: dump: 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f73cc012aa0 con 0x7f73e409c990 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 msgr2=0x7f73e4141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 0x7f73e4141f30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009650 tx=0x7f73cc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 msgr2=0x7f73e4141f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:09.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73da7fc700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e409c990 0x7f73e4141f30 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009650 tx=0x7f73cc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.496+0000 7f73e2d9d700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f73c8035a70 0x7f73c8037f20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73e409a7f0 tx=0x7f73dc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e359e700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 0x7f73e402a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e359e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4091600 con 0x7f73e4029d00 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 0x7f73e402a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient: get_auth_request con 0x7f73e4029d00 auth_method 0 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 10 monclient(hunting): handle_auth_done global_id 74993 payload 824 2024-09-18T10:19:09.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.508+0000 7f73e1d9b700 1 -- 172.21.15.142:0/2197982245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f73e406dad0 con 0x7f73e4029d00 2024-09-18T10:19:09.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e1d9b700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 msgr2=0x7f73e402a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e1d9b700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 0x7f73e402a100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0093d0 tx=0x7f73cc015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0219e0 con 0x7f73e4029d00 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 10 monclient: dump: 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f73cc021b40 con 0x7f73e4029d00 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 msgr2=0x7f73e402a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 0x7f73e402a100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0093d0 tx=0x7f73cc015000 comp rx=0 tx=0).stop 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 msgr2=0x7f73e402a100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73da7fc700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e4029d00 0x7f73e402a100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:09.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 106, 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.522 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.523 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:09.524 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.525 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:09.526 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.528 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.529 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.533 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:09.534 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f73c8035a70 msgr2=0x7f73c8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f73c8035a70 0x7f73c8037f20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73e409a7f0 tx=0x7f73dc017000 comp rx=0 tx=0).stop 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 10 monclient: shutdown 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 msgr2=0x7f73e408a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 --2- 172.21.15.142:0/2197982245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73e406f3b0 0x7f73e408a8b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c600 tx=0x7f73d4007f60 comp rx=0 tx=0).stop 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2197982245 reap_dead start 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e2d9d700 1 -- 172.21.15.142:0/2197982245 reap_dead start 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 shutdown_connections 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 >> 172.21.15.142:0/2197982245 conn(0x7f73e4070ef0 msgr2=0x7f73e4097580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 shutdown_connections 2024-09-18T10:19:09.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.512+0000 7f73e359e700 1 -- 172.21.15.142:0/2197982245 wait complete. 2024-09-18T10:19:09.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:19:09.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:19:09.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:19:09.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: get_monmap_and_config 2024-09-18T10:19:09.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: build_initial_monmap 2024-09-18T10:19:09.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: monmap: 2024-09-18T10:19:09.646 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:09.644659+0000 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:09.644659+0000 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:09.647 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 Processor -- start 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 -- start start 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: init 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:19:09.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc67009b0f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc67009b4c0 0x7fc67009cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-c con 0x7fc67009b4c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-b con 0x7fc6700947b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc670098b00 con 0x7fc67009b0f0 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67009b920 con 0x7fc67009b4c0 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc670098d20 con 0x7fc6700947b0 2024-09-18T10:19:09.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc676b0d700 10 monclient(hunting): authenticate will time out at 1438.570624s 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc675b0b700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fc67009b0f0 legacy=0x7fc6700940a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53480/0 (socket says 172.21.15.142:53480) 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 -- 172.21.15.142:0/2510416384 learned_addr learned my addr 172.21.15.142:0/2510416384 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): get_auth_request con 0x7fc6700947b0 auth_method 0 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 1 --2- 172.21.15.142:0/2510416384 >> v2:172.21.15.150:3300/0 conn(0x7fc67009b4c0 0x7fc67009cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): get_auth_request con 0x7fc67009b4c0 auth_method 0 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67530a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient(hunting): handle_auth_done global_id 74996 payload 274 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 -- 172.21.15.142:0/2510416384 >> v2:172.21.15.150:3300/0 conn(0x7fc67009b4c0 msgr2=0x7fc67009cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 --2- 172.21.15.142:0/2510416384 >> v2:172.21.15.150:3300/0 conn(0x7fc67009b4c0 0x7fc67009cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 -- 172.21.15.142:0/2510416384 >> v1:172.21.15.142:6789/0 conn(0x7fc67009b0f0 legacy=0x7fc6700940a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 monclient: found mon.noname-b 2024-09-18T10:19:09.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.640+0000 7fc67630c700 1 -- 172.21.15.142:0/2510416384 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc67009a100 con 0x7fc6700947b0 2024-09-18T10:19:09.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc67630c700 1 -- 172.21.15.142:0/2510416384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 msgr2=0x7fc670136990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc67630c700 1 --2- 172.21.15.142:0/2510416384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc668009f20 tx=0x7fc668009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd2f87ee827a3aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 1 -- 172.21.15.142:0/2510416384 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66800e070 con 0x7fc6700947b0 2024-09-18T10:19:09.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: dump: 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 5 monclient: authenticate success, global_id 74996 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 1 -- 172.21.15.142:0/2510416384 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66800bbe0 con 0x7fc6700947b0 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 1 -- 172.21.15.142:0/2510416384 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680092e0 con 0x7fc6700947b0 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 4 set_mon_vals no callback set 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc674b09700 10 monclient: dump: 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.660 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 10 monclient: shutdown 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 -- 172.21.15.142:0/2510416384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 msgr2=0x7fc670136990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 --2- 172.21.15.142:0/2510416384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc668009f20 tx=0x7fc668009840 comp rx=0 tx=0).stop 2024-09-18T10:19:09.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 -- 172.21.15.142:0/2510416384 shutdown_connections 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 --2- 172.21.15.142:0/2510416384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6700947b0 0x7fc670136990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 --2- 172.21.15.142:0/2510416384 >> v2:172.21.15.150:3300/0 conn(0x7fc67009b4c0 0x7fc67009cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 -- 172.21.15.142:0/2510416384 >> 172.21.15.142:0/2510416384 conn(0x7fc670070ef0 msgr2=0x7fc6700712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 -- 172.21.15.142:0/2510416384 shutdown_connections 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.644+0000 7fc676b0d700 1 -- 172.21.15.142:0/2510416384 wait complete. 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient: build_initial_monmap 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient: monmap: 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:09.662 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 Processor -- start 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 -- start start 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient: init 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6700947b0 0x7fc670142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-b con 0x7fc6700947b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67009b0f0 0x7fc6701418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-a con 0x7fc67009b0f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 0x7fc6701457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): picked mon.noname-c con 0x7fc67009b4c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700a0590 con 0x7fc67009b0f0 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670126c50 con 0x7fc6700947b0 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67006e9c0 con 0x7fc67009b4c0 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc676b0d700 10 monclient(hunting): authenticate will time out at 1438.577107s 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67530a700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67009b0f0 msgr2=0x7fc6701418d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67530a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67009b0f0 0x7fc6701418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:09.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 0x7fc6701457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 0x7fc6701457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53494/0 (socket says 172.21.15.142:53494) 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 learned_addr learned my addr 172.21.15.142:0/392354245 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): get_auth_request con 0x7fc67009b4c0 auth_method 0 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6700947b0 0x7fc670142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): get_auth_request con 0x7fc6700947b0 auth_method 0 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient(hunting): handle_auth_done global_id 74999 payload 824 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6700947b0 msgr2=0x7fc670142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6700947b0 0x7fc670142290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67009b0f0 msgr2=0x7fc6701418d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc67009b0f0 0x7fc6701418d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient: _finish_hunting 0 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 monclient: found mon.noname-c 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:09.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.648+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668009ae0 con 0x7fc67009b4c0 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 msgr2=0x7fc6701457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc67630c700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 0x7fc6701457a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc6680098c0 tx=0x7fc668017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668012a60 con 0x7fc67009b4c0 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: dump: 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.667 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 5 monclient: authenticate success, global_id 74999 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668012bc0 con 0x7fc67009b4c0 2024-09-18T10:19:09.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66801d420 con 0x7fc67009b4c0 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 monclient: _renew_subs 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc67006d600 con 0x7fc67009b4c0 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc6677fe700 4 set_mon_vals no callback set 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 10 monclient: dump: 2024-09-18T10:19:09.669 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 monclient: _renew_subs 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:09.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6701265b0 con 0x7fc67009b4c0 2024-09-18T10:19:09.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 0x7fc670145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc670126b10 con 0x7fc670142d10 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc66801d850 con 0x7fc67009b4c0 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc654035ac0 0x7fc654037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 0x7fc670145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient: get_auth_request con 0x7fc670142d10 auth_method 0 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc66804a480 con 0x7fc67009b4c0 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc67530a700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc654035ac0 0x7fc654037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc67530a700 10 monclient: get_auth_request con 0x7fc654035ac0 auth_method 0 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 10 monclient(hunting): handle_auth_done global_id 114208 payload 824 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.652+0000 7fc675b0b700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc670072a80 con 0x7fc670142d10 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc675b0b700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 msgr2=0x7fc670145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc675b0b700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 0x7fc670145ce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc660009650 tx=0x7fc6600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc67530a700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc654035ac0 0x7fc654037f70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc67006e190 tx=0x7fc66c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660012940 con 0x7fc670142d10 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 10 monclient: dump: 2024-09-18T10:19:09.675 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc660012aa0 con 0x7fc670142d10 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:09.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 msgr2=0x7fc670145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 0x7fc670145ce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc660009650 tx=0x7fc6600091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 msgr2=0x7fc670145ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.656+0000 7fc665ffb700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670142d10 0x7fc670145ce0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc676b0d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc676b0d700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 0x7fc670146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc676b0d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:09.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6700648a0 con 0x7fc670141e10 2024-09-18T10:19:09.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 0x7fc670146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:09.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient: get_auth_request con 0x7fc670141e10 auth_method 0 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 10 monclient(hunting): handle_auth_done global_id 114211 payload 824 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.668+0000 7fc675b0b700 1 -- 172.21.15.142:0/392354245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc6700678a0 con 0x7fc670141e10 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc675b0b700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 msgr2=0x7fc670146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc675b0b700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 0x7fc670146220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6600093d0 tx=0x7fc660015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66001cde0 con 0x7fc670141e10 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:09.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 10 monclient: dump: 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fc660012730 con 0x7fc670141e10 2024-09-18T10:19:09.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 msgr2=0x7fc670146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 0x7fc670146220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6600093d0 tx=0x7fc660015000 comp rx=0 tx=0).stop 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 msgr2=0x7fc670146220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc665ffb700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc670141e10 0x7fc670146220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:09.681 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 106, 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:09.682 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.683 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.684 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:09.685 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:09.686 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:09.688 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:09.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.690 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.691 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.693 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:09.694 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc654035ac0 msgr2=0x7fc654037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc654035ac0 0x7fc654037f70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc67006e190 tx=0x7fc66c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 10 monclient: shutdown 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 msgr2=0x7fc6701457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:09.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 --2- 172.21.15.142:0/392354245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc67009b4c0 0x7fc6701457a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc6680098c0 tx=0x7fc668017000 comp rx=0 tx=0).stop 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 reap_dead start 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc67630c700 1 -- 172.21.15.142:0/392354245 reap_dead start 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 shutdown_connections 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 >> 172.21.15.142:0/392354245 conn(0x7fc670070ef0 msgr2=0x7fc67009f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 shutdown_connections 2024-09-18T10:19:09.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:09.672+0000 7fc676b0d700 1 -- 172.21.15.142:0/392354245 wait complete. 2024-09-18T10:19:09.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:19:09.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:19:29.698 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:19:29.698 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:19:29.699 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:19:29.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:19:29.702 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:19:29.702 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:19:29.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.796+0000 7f1e8ffff700 10 monclient: get_monmap_and_config 2024-09-18T10:19:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.796+0000 7f1e8ffff700 10 monclient: build_initial_monmap 2024-09-18T10:19:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.796+0000 7f1e8ffff700 10 monclient: monmap: 2024-09-18T10:19:29.806 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:29.803545+0000 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:29.803545+0000 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 Processor -- start 2024-09-18T10:19:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 -- start start 2024-09-18T10:19:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient: init 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e9006eec0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f1e9006f810 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e9009ce00 0x7f1e901395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f1e9009ce00 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e90069090 con 0x7f1e9006eec0 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e90098f60 con 0x7f1e9006f810 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9009d200 con 0x7f1e9009ce00 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8ffff700 10 monclient(hunting): authenticate will time out at 1458.729691s 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:50766/0 (socket says 172.21.15.142:50766) 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 -- 172.21.15.142:0/1957080248 learned_addr learned my addr 172.21.15.142:0/1957080248 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 10 monclient(hunting): get_auth_request con 0x7f1e9006f810 auth_method 0 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 -- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 msgr2=0x7f1e9009c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 -- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 msgr2=0x7f1e9009c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 --2- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 --2- 172.21.15.142:0/1957080248 >> v2:172.21.15.150:3300/0 conn(0x7f1e9009ce00 0x7f1e901395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e867fc700 1 --2- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): get_auth_request con 0x7f1e9009ce00 auth_method 0 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient(hunting): handle_auth_done global_id 114214 payload 274 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 msgr2=0x7f1e9009c8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 --2- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/1957080248 >> v1:172.21.15.150:6789/0 conn(0x7f1e9006eec0 legacy=0x7f1e9009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:19:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.800+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/1957080248 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9006f290 con 0x7f1e9009ce00 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/1957080248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9009ce00 msgr2=0x7f1e901395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8f7fe700 1 --2- 172.21.15.142:0/1957080248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9009ce00 0x7f1e901395c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e90068da0 tx=0x7f1e88009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3fe9a3590855de8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 1 -- 172.21.15.142:0/1957080248 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8800e070 con 0x7f1e9009ce00 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: dump: 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 1 -- 172.21.15.142:0/1957080248 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8800be40 con 0x7f1e9009ce00 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 1 -- 172.21.15.142:0/1957080248 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e88012660 con 0x7f1e9009ce00 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:29.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8e7fc700 10 monclient: dump: 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 5 monclient: authenticate success, global_id 114214 2024-09-18T10:19:29.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 4 set_mon_vals no callback set 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 10 monclient: shutdown 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 -- 172.21.15.142:0/1957080248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9009ce00 msgr2=0x7f1e901395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/1957080248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9009ce00 0x7f1e901395c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1e90068da0 tx=0x7f1e88009780 comp rx=0 tx=0).stop 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 -- 172.21.15.142:0/1957080248 shutdown_connections 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/1957080248 >> v2:172.21.15.142:3300/0 conn(0x7f1e9006f810 0x7f1e9009c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/1957080248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9009ce00 0x7f1e901395c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 -- 172.21.15.142:0/1957080248 >> 172.21.15.142:0/1957080248 conn(0x7f1e900a13f0 msgr2=0x7f1e900a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 -- 172.21.15.142:0/1957080248 shutdown_connections 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 -- 172.21.15.142:0/1957080248 wait complete. 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 10 monclient: build_initial_monmap 2024-09-18T10:19:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 10 monclient: monmap: 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.804+0000 7f1e8ffff700 1 Processor -- start 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 -- start start 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient: init 2024-09-18T10:19:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f1e9006eec0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9006f810 0x7f1e90141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f1e9006f810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f1e9009ce00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90093500 con 0x7f1e9009ce00 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9009d450 con 0x7f1e9006f810 2024-09-18T10:19:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9009fdc0 con 0x7f1e9006eec0 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8ffff700 10 monclient(hunting): authenticate will time out at 1458.736784s 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:56634/0 (socket says 172.21.15.142:56634) 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 -- 172.21.15.142:0/3850890546 learned_addr learned my addr 172.21.15.142:0/3850890546 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 10 monclient(hunting): get_auth_request con 0x7f1e9009ce00 auth_method 0 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): get_auth_request con 0x7f1e9006eec0 auth_method 0 2024-09-18T10:19:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9006f810 0x7f1e90141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 msgr2=0x7f1e9008f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 msgr2=0x7f1e9008f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): get_auth_request con 0x7f1e9006f810 auth_method 0 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8f7fe700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e867fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient(hunting): handle_auth_done global_id 75002 payload 824 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9006f810 msgr2=0x7f1e90141d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9006f810 0x7f1e90141d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 msgr2=0x7f1e9008f8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient: _finish_hunting 0 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 monclient: found mon.noname-c 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.808+0000 7f1e8effd700 1 -- 172.21.15.142:0/3850890546 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88009a20 con 0x7f1e9006eec0 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8effd700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 msgr2=0x7f1e90141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8effd700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c600 tx=0x7f1e80007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800117a0 con 0x7f1e9006eec0 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: dump: 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: _finish_auth 0 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e80011900 con 0x7f1e9006eec0 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 5 monclient: authenticate success, global_id 75002 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _renew_subs 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e90091900 con 0x7f1e9006eec0 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e80011cc0 con 0x7f1e9006eec0 2024-09-18T10:19:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 10 monclient: dump: 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8dffb700 4 set_mon_vals no callback set 2024-09-18T10:19:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _renew_subs 2024-09-18T10:19:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e90092f60 con 0x7f1e9006eec0 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1e80012630 con 0x7f1e9006eec0 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e90072a80 con 0x7f1e9006eec0 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 0x7f1e70037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1e80048c70 con 0x7f1e9006eec0 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e867fc700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 0x7f1e70037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.812+0000 7f1e867fc700 10 monclient: get_auth_request con 0x7f1e70035af0 auth_method 0 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.816+0000 7f1e867fc700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 0x7f1e70037fa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.820+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f1e80011e20 con 0x7f1e9006eec0 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.820+0000 7f1e877fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:29.820+0000 7f1e877fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e8ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e8ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e8ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:30.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1e90093220 con 0x7f1e9006eec0 2024-09-18T10:19:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e877fe700 1 -- 172.21.15.142:0/3850890546 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f1e800102e0 con 0x7f1e9006eec0 2024-09-18T10:19:30.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e877fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:30.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.043+0000 7f1e877fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:30.054 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":106,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:30.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 msgr2=0x7f1e70037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:30.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 0x7f1e70037fa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1e78006fd0 tx=0x7f1e78008040 comp rx=0 tx=0).stop 2024-09-18T10:19:30.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 10 monclient: shutdown 2024-09-18T10:19:30.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:30.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 msgr2=0x7f1e90141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c600 tx=0x7f1e80007f60 comp rx=0 tx=0).stop 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 shutdown_connections 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e70035af0 0x7f1e70037fa0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e9009ce00 0x7f1e9008f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e9006f810 0x7f1e90141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 --2- 172.21.15.142:0/3850890546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e9006eec0 0x7f1e90141810 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 >> 172.21.15.142:0/3850890546 conn(0x7f1e900a13f0 msgr2=0x7f1e90099920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:30.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 shutdown_connections 2024-09-18T10:19:30.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:30.047+0000 7f1e8ffff700 1 -- 172.21.15.142:0/3850890546 wait complete. 2024-09-18T10:19:33.068 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.171+0000 7ff88d05e700 10 monclient: get_monmap_and_config 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.171+0000 7ff88d05e700 10 monclient: build_initial_monmap 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.171+0000 7ff88d05e700 10 monclient: monmap: 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:33.180 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:33.178705+0000 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:33.178705+0000 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:33.181 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 Processor -- start 2024-09-18T10:19:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 -- start start 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient: init 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-b con 0x7ff8880a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff88812f810 0x7ff88809f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-c con 0x7ff88812f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8880a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88809d990 con 0x7ff8880a02d0 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88809dc70 con 0x7ff88812f810 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8880a20c0 con 0x7ff8880a1cf0 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff88d05e700 10 monclient(hunting): authenticate will time out at 1462.105565s 2024-09-18T10:19:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35314/0 (socket says 172.21.15.142:35314) 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 -- 172.21.15.142:0/707891289 learned_addr learned my addr 172.21.15.142:0/707891289 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): get_auth_request con 0x7ff8880a1cf0 auth_method 0 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:33.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:33.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 1 --2- 172.21.15.142:0/707891289 >> v2:172.21.15.150:3300/0 conn(0x7ff88812f810 0x7ff88809f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): get_auth_request con 0x7ff88812f810 auth_method 0 2024-09-18T10:19:33.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:33.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:33.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff87ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient(hunting): handle_auth_done global_id 75005 payload 274 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 -- 172.21.15.142:0/707891289 >> v2:172.21.15.150:3300/0 conn(0x7ff88812f810 msgr2=0x7ff88809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:33.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 --2- 172.21.15.142:0/707891289 >> v2:172.21.15.150:3300/0 conn(0x7ff88812f810 0x7ff88809f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 -- 172.21.15.142:0/707891289 >> v1:172.21.15.150:6789/0 conn(0x7ff8880a02d0 legacy=0x7ff8881320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient: _finish_hunting 0 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 monclient: found mon.noname-b 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.175+0000 7ff887fff700 1 -- 172.21.15.142:0/707891289 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88809f9c0 con 0x7ff8880a1cf0 2024-09-18T10:19:33.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff887fff700 1 -- 172.21.15.142:0/707891289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 msgr2=0x7ff88812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff887fff700 1 --2- 172.21.15.142:0/707891289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff8780099b0 tx=0x7ff878009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8fecb3adba9c199 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 1 -- 172.21.15.142:0/707891289 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878009ea0 con 0x7ff8880a1cf0 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: dump: 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.197 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: _finish_auth 0 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 1 -- 172.21.15.142:0/707891289 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87800ba70 con 0x7ff8880a1cf0 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 5 monclient: authenticate success, global_id 75005 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:33.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 4 set_mon_vals no callback set 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 1 -- 172.21.15.142:0/707891289 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800be10 con 0x7ff8880a1cf0 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff8877fe700 10 monclient: dump: 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.199 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 10 monclient: shutdown 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 -- 172.21.15.142:0/707891289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 msgr2=0x7ff88812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 --2- 172.21.15.142:0/707891289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff8780099b0 tx=0x7ff878009250 comp rx=0 tx=0).stop 2024-09-18T10:19:33.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 -- 172.21.15.142:0/707891289 shutdown_connections 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 --2- 172.21.15.142:0/707891289 >> v2:172.21.15.150:3300/0 conn(0x7ff88812f810 0x7ff88809f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 --2- 172.21.15.142:0/707891289 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a1cf0 0x7ff88812f030 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 -- 172.21.15.142:0/707891289 >> 172.21.15.142:0/707891289 conn(0x7ff888071820 msgr2=0x7ff8880a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 -- 172.21.15.142:0/707891289 shutdown_connections 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 1 -- 172.21.15.142:0/707891289 wait complete. 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 10 monclient: build_initial_monmap 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.179+0000 7ff88d05e700 10 monclient: monmap: 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:33.201 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 Processor -- start 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 -- start start 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient: init 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-c con 0x7ff8880a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-a con 0x7ff8880a1cf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88812f810 0x7ff88809bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): picked mon.noname-b con 0x7ff88812f810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88806cf60 con 0x7ff8880a1cf0 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88812ef50 con 0x7ff88812f810 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88806cc80 con 0x7ff8880a02d0 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88d05e700 10 monclient(hunting): authenticate will time out at 1462.114333s 2024-09-18T10:19:33.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35326/0 (socket says 172.21.15.142:35326) 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 -- 172.21.15.142:0/1987341223 learned_addr learned my addr 172.21.15.142:0/1987341223 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): get_auth_request con 0x7ff8880a02d0 auth_method 0 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 10 monclient(hunting): get_auth_request con 0x7ff8880a1cf0 auth_method 0 2024-09-18T10:19:33.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88812f810 0x7ff88809bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): get_auth_request con 0x7ff88812f810 auth_method 0 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 msgr2=0x7ff88809b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:19:33.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 msgr2=0x7ff88809b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff87ffff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff88c85d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient(hunting): handle_auth_done global_id 75008 payload 824 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88812f810 msgr2=0x7ff88809bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88812f810 0x7ff88809bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 msgr2=0x7ff88809b6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient: _finish_hunting 0 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 monclient: found mon.noname-c 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.183+0000 7ff887fff700 1 -- 172.21.15.142:0/1987341223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8780094f0 con 0x7ff8880a02d0 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff887fff700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 msgr2=0x7ff88809b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff887fff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8880a2250 tx=0x7ff878016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878006bc0 con 0x7ff8880a02d0 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: dump: 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:33.207 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: _finish_auth 0 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87801b470 con 0x7ff8880a02d0 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 5 monclient: authenticate success, global_id 75008 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:33.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87801b850 con 0x7ff8880a02d0 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _renew_subs 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88806bb50 con 0x7ff8880a02d0 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff8867fc700 4 set_mon_vals no callback set 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 10 monclient: dump: 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:33.209 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:33.210 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _renew_subs 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88806af00 con 0x7ff8880a02d0 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:33.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:33.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88806d1a0 con 0x7ff8880a02d0 2024-09-18T10:19:33.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.187+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff87801bc40 con 0x7ff8880a02d0 2024-09-18T10:19:33.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.191+0000 7ff884ff9700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 0x7ff868037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:33.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.191+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff87804aae0 con 0x7ff8880a02d0 2024-09-18T10:19:33.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.191+0000 7ff87ffff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 0x7ff868037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:33.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.191+0000 7ff87ffff700 10 monclient: get_auth_request con 0x7ff868035ac0 auth_method 0 2024-09-18T10:19:33.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.191+0000 7ff87ffff700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 0x7ff868037f70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff88812fc20 tx=0x7ff870008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:33.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.199+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff8780193b0 con 0x7ff8880a02d0 2024-09-18T10:19:33.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.199+0000 7ff884ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:33.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.199+0000 7ff884ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:33.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff88d05e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:33.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff88d05e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:33.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff88d05e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:33.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff888072a80 con 0x7ff8880a02d0 2024-09-18T10:19:33.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff884ff9700 1 -- 172.21.15.142:0/1987341223 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ff87802d350 con 0x7ff8880a02d0 2024-09-18T10:19:33.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff884ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:33.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.427+0000 7ff884ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:33.436 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":106,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 msgr2=0x7ff868037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 0x7ff868037f70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff88812fc20 tx=0x7ff870008040 comp rx=0 tx=0).stop 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 10 monclient: shutdown 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 msgr2=0x7ff88809b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8880a2250 tx=0x7ff878016000 comp rx=0 tx=0).stop 2024-09-18T10:19:33.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 shutdown_connections 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff868035ac0 0x7ff868037f70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88812f810 0x7ff88809bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8880a1cf0 0x7ff88809b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 --2- 172.21.15.142:0/1987341223 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8880a02d0 0x7ff88809b160 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 >> 172.21.15.142:0/1987341223 conn(0x7ff888071820 msgr2=0x7ff88809e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 shutdown_connections 2024-09-18T10:19:33.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:33.431+0000 7ff88d05e700 1 -- 172.21.15.142:0/1987341223 wait complete. 2024-09-18T10:19:36.450 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:36.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.555+0000 7f7f857a8700 10 monclient: get_monmap_and_config 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.555+0000 7f7f857a8700 10 monclient: build_initial_monmap 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.555+0000 7f7f857a8700 10 monclient: monmap: 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:36.563110+0000 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:36.563110+0000 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:36.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:36.566 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:36.566 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:36.566 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.555+0000 7f7f857a8700 1 Processor -- start 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 1 -- start start 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient: init 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f80132030 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f80132400 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7f80138a80 0x7f7f8013ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-a con 0x7f7f80138a80 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f800a17b0 con 0x7f7f80132400 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f80132b90 con 0x7f7f80132030 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f800a1a90 con 0x7f7f80138a80 2024-09-18T10:19:36.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f857a8700 10 monclient(hunting): authenticate will time out at 1465.488636s 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f7f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7f80132400 0x7f7f80137360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55782/0 (socket says 172.21.15.142:55782) 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/3941337658 learned_addr learned my addr 172.21.15.142:0/3941337658 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 1 --2- 172.21.15.142:0/3941337658 >> v2:172.21.15.142:3300/0 conn(0x7f7f80138a80 0x7f7f8013ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): get_auth_request con 0x7f7f80138a80 auth_method 0 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient(hunting): handle_auth_done global_id 124098 payload 274 2024-09-18T10:19:36.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 1 -- 172.21.15.142:0/3941337658 >> v1:172.21.15.150:6789/0 conn(0x7f7f80132030 legacy=0x7f7f8009ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 1 -- 172.21.15.142:0/3941337658 >> v1:172.21.15.142:6789/0 conn(0x7f7f80132400 legacy=0x7f7f80137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient: _finish_hunting 0 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 1 monclient: found mon.noname-a 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:36.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.559+0000 7f7f84fa7700 1 -- 172.21.15.142:0/3941337658 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f800a00f0 con 0x7f7f80138a80 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f84fa7700 1 -- 172.21.15.142:0/3941337658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80138a80 msgr2=0x7f7f8013ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f84fa7700 1 --2- 172.21.15.142:0/3941337658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80138a80 0x7f7f8013ae60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009e60 tx=0x7f7f78009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cce22eeefadb68 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 1 -- 172.21.15.142:0/3941337658 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7800e070 con 0x7f7f80138a80 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:36.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: dump: 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: _finish_auth 0 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 1 -- 172.21.15.142:0/3941337658 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f78012430 con 0x7f7f80138a80 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 5 monclient: authenticate success, global_id 124098 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 1 -- 172.21.15.142:0/3941337658 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f780127d0 con 0x7f7f80138a80 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 4 set_mon_vals no callback set 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f7effd700 10 monclient: dump: 2024-09-18T10:19:36.575 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 10 monclient: shutdown 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:36.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- 172.21.15.142:0/3941337658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80138a80 msgr2=0x7f7f8013ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 --2- 172.21.15.142:0/3941337658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80138a80 0x7f7f8013ae60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009e60 tx=0x7f7f78009780 comp rx=0 tx=0).stop 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- 172.21.15.142:0/3941337658 shutdown_connections 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 --2- 172.21.15.142:0/3941337658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80138a80 0x7f7f8013ae60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- 172.21.15.142:0/3941337658 >> 172.21.15.142:0/3941337658 conn(0x7f7f800a5700 msgr2=0x7f7f800a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- 172.21.15.142:0/3941337658 shutdown_connections 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- 172.21.15.142:0/3941337658 wait complete. 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 10 monclient: build_initial_monmap 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 10 monclient: monmap: 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:36.577 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 Processor -- start 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 1 -- start start 2024-09-18T10:19:36.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.563+0000 7f7f857a8700 10 monclient: init 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-a con 0x7f7f80132030 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-c con 0x7f7f80132400 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:36.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 0x7f7f8006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): picked mon.noname-b con 0x7f7f80138a80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8009f6e0 con 0x7f7f80132030 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f800a4890 con 0x7f7f80138a80 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8006e1f0 con 0x7f7f80132400 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f857a8700 10 monclient(hunting): authenticate will time out at 1465.496774s 2024-09-18T10:19:36.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35346/0 (socket says 172.21.15.142:35346) 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/4096679616 learned_addr learned my addr 172.21.15.142:0/4096679616 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): get_auth_request con 0x7f7f80132400 auth_method 0 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): get_auth_request con 0x7f7f80132030 auth_method 0 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 0x7f7f8006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 10 monclient(hunting): get_auth_request con 0x7f7f80138a80 auth_method 0 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 msgr2=0x7f7f8006c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 msgr2=0x7f7f8006c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:19:36.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7ffff700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient(hunting): handle_auth_done global_id 75011 payload 824 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 msgr2=0x7f7f8006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 0x7f7f8006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 msgr2=0x7f7f8006c160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/4096679616 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78009a20 con 0x7f7f80132400 2024-09-18T10:19:36.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f84fa7700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 0x7f7f8006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.567+0000 7f7f7f7fe700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 msgr2=0x7f7f8006c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f7f7fe700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c5e0 tx=0x7f7f70004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70005d40 con 0x7f7f80132400 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: dump: 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.587 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: _finish_auth 0 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 5 monclient: authenticate success, global_id 75011 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f700070d0 con 0x7f7f80132400 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _renew_subs 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70007490 con 0x7f7f80132400 2024-09-18T10:19:36.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8006dd40 con 0x7f7f80132400 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f7e7fc700 4 set_mon_vals no callback set 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 10 monclient: dump: 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.589 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _renew_subs 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f80130ef0 con 0x7f7f80132400 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7f7001b3b0 con 0x7f7f80132400 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 0x7f7f6c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f80072a80 con 0x7f7f80132400 2024-09-18T10:19:36.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7f70048530 con 0x7f7f80132400 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f7ffff700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 0x7f7f6c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.571+0000 7f7f7ffff700 10 monclient: get_auth_request con 0x7f7f6c035ac0 auth_method 0 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.575+0000 7f7f7ffff700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 0x7f7f6c037f70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009300 tx=0x7f7f74009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.579+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7f7000f820 con 0x7f7f80132400 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.579+0000 7f7f67fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:36.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.579+0000 7f7f67fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:36.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f857a8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:36.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f857a8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:36.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f857a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7f800a0310 con 0x7f7f80132400 2024-09-18T10:19:36.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f67fff700 1 -- 172.21.15.142:0/4096679616 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f7f70017030 con 0x7f7f80132400 2024-09-18T10:19:36.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f67fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:36.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.799+0000 7f7f67fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:36.809 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 msgr2=0x7f7f6c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 0x7f7f6c037f70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f7f74009300 tx=0x7f7f74009060 comp rx=0 tx=0).stop 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 10 monclient: shutdown 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 msgr2=0x7f7f8006c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c5e0 tx=0x7f7f70004a80 comp rx=0 tx=0).stop 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 shutdown_connections 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7f6c035ac0 0x7f7f6c037f70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7f80138a80 0x7f7f8006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7f80132400 0x7f7f8006c6a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 --2- 172.21.15.142:0/4096679616 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f80132030 0x7f7f8006c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 >> 172.21.15.142:0/4096679616 conn(0x7f7f800a5700 msgr2=0x7f7f8009e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 shutdown_connections 2024-09-18T10:19:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.803+0000 7f7f857a8700 1 -- 172.21.15.142:0/4096679616 wait complete. 2024-09-18T10:19:36.828 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:19:36.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:19:36.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:19:36.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: get_monmap_and_config 2024-09-18T10:19:36.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: build_initial_monmap 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: monmap: 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:36.929376+0000 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:36.929376+0000 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:36.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 Processor -- start 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 -- start start 2024-09-18T10:19:36.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: init 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58ec09d870 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58ec0954b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f58ec135680 0x7f58ec137a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-c con 0x7f58ec135680 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec069c70 con 0x7f58ec09d870 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0958b0 con 0x7f58ec0954b0 2024-09-18T10:19:36.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec06f9d0 con 0x7f58ec135680 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f46cf700 10 monclient(hunting): authenticate will time out at 1465.855073s 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f246b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f58ec09d870 0x7f58ec0989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55792/0 (socket says 172.21.15.142:55792) 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f246b700 1 -- 172.21.15.142:0/784401182 learned_addr learned my addr 172.21.15.142:0/784401182 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 1 --2- 172.21.15.142:0/784401182 >> v2:172.21.15.150:3300/0 conn(0x7f58ec135680 0x7f58ec137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): get_auth_request con 0x7f58ec135680 auth_method 0 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.923+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f246b700 1 -- 172.21.15.142:0/784401182 >> v1:172.21.15.142:6789/0 conn(0x7f58ec09d870 legacy=0x7f58ec0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1469700 1 -- 172.21.15.142:0/784401182 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190850594 0 0) 0x7f58ec069c70 con 0x7f58ec09d870 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1469700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1469700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1469700 10 monclient(hunting): my global_id is 124104 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1469700 1 -- 172.21.15.142:0/784401182 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003570 con 0x7f58ec09d870 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f1c6a700 1 -- 172.21.15.142:0/784401182 >> v1:172.21.15.150:6789/0 conn(0x7f58ec0954b0 legacy=0x7f58ec0990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_done global_id 114229 payload 274 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 1 -- 172.21.15.142:0/784401182 >> v1:172.21.15.150:6789/0 conn(0x7f58ec0954b0 legacy=0x7f58ec0990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 1 -- 172.21.15.142:0/784401182 >> v1:172.21.15.142:6789/0 conn(0x7f58ec09d870 legacy=0x7f58ec0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 10 monclient: _finish_hunting 0 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 1 monclient: found mon.noname-c 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.927+0000 7f58f2c6c700 1 -- 172.21.15.142:0/784401182 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec09dc40 con 0x7f58ec135680 2024-09-18T10:19:36.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f2c6c700 1 -- 172.21.15.142:0/784401182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec135680 msgr2=0x7f58ec137a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f2c6c700 1 --2- 172.21.15.142:0/784401182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec135680 0x7f58ec137a60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009f60 tx=0x7f58e4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9484e2daee82cd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 1 -- 172.21.15.142:0/784401182 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e400e070 con 0x7f58ec135680 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: dump: 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.941 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: _finish_auth 0 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f46cf700 5 monclient: authenticate success, global_id 114229 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f46cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 1 -- 172.21.15.142:0/784401182 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e400be40 con 0x7f58ec135680 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f46cf700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f46cf700 4 set_mon_vals no callback set 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 1 -- 172.21.15.142:0/784401182 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4012660 con 0x7f58ec135680 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.931+0000 7f58f1469700 10 monclient: dump: 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.943 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.944 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.944 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.944 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: shutdown 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- 172.21.15.142:0/784401182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec135680 msgr2=0x7f58ec137a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 --2- 172.21.15.142:0/784401182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec135680 0x7f58ec137a60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009f60 tx=0x7f58e4009780 comp rx=0 tx=0).stop 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- 172.21.15.142:0/784401182 shutdown_connections 2024-09-18T10:19:36.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 --2- 172.21.15.142:0/784401182 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec135680 0x7f58ec137a60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- 172.21.15.142:0/784401182 >> 172.21.15.142:0/784401182 conn(0x7f58ec0a14e0 msgr2=0x7f58ec0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- 172.21.15.142:0/784401182 shutdown_connections 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- 172.21.15.142:0/784401182 wait complete. 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: build_initial_monmap 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: monmap: 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:36.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 Processor -- start 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- start start 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: init 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:19:36.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec0954b0 0x7f58ec12b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-b con 0x7f58ec0954b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 0x7f58ec06bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-c con 0x7f58ec09d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec135680 0x7f58ec06c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): picked mon.noname-a con 0x7f58ec135680 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec06e400 con 0x7f58ec135680 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec130780 con 0x7f58ec0954b0 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec12b8e0 con 0x7f58ec09d870 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f46cf700 10 monclient(hunting): authenticate will time out at 1465.866974s 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 0x7f58ec06bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec135680 0x7f58ec06c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 0x7f58ec06bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35354/0 (socket says 172.21.15.142:35354) 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 -- 172.21.15.142:0/2361522441 learned_addr learned my addr 172.21.15.142:0/2361522441 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): get_auth_request con 0x7f58ec135680 auth_method 0 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): get_auth_request con 0x7f58ec09d870 auth_method 0 2024-09-18T10:19:36.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec0954b0 0x7f58ec12b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): get_auth_request con 0x7f58ec0954b0 auth_method 0 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient(hunting): handle_auth_done global_id 75014 payload 824 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec0954b0 msgr2=0x7f58ec12b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58ec0954b0 0x7f58ec12b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec135680 msgr2=0x7f58ec06c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec135680 0x7f58ec06c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient: _finish_hunting 0 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 monclient: found mon.noname-c 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f1c6a700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4009a20 con 0x7f58ec09d870 2024-09-18T10:19:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.935+0000 7f58f2c6c700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec135680 0x7f58ec06c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f1c6a700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 msgr2=0x7f58ec06bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f1c6a700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 0x7f58ec06bb50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00c7f0 tx=0x7f58dc007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc0074f0 con 0x7f58ec09d870 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: dump: 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 5 monclient: authenticate success, global_id 75014 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc007670 con 0x7f58ec09d870 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc005a60 con 0x7f58ec09d870 2024-09-18T10:19:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 10 monclient: _renew_subs 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec097b80 con 0x7f58ec09d870 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e3fff700 4 set_mon_vals no callback set 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58e27fc700 10 monclient: dump: 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 10 monclient: _renew_subs 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.939+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec06def0 con 0x7f58ec09d870 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f46cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f46cf700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 0x7f58ec1422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec06d840 con 0x7f58ec06c5d0 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f58dc012400 con 0x7f58ec09d870 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 0x7f58ec1422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient: get_auth_request con 0x7f58ec06c5d0 auth_method 0 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58d0035ac0 0x7f58d0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f58dc048150 con 0x7f58ec09d870 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 10 monclient(hunting): handle_auth_done global_id 124110 payload 824 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58ec06c9a0 con 0x7f58ec06c5d0 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f2c6c700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58d0035ac0 0x7f58d0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f2c6c700 10 monclient: get_auth_request con 0x7f58d0035ac0 auth_method 0 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 msgr2=0x7f58ec1422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58f246b700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 0x7f58ec1422e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e80117a0 con 0x7f58ec06c5d0 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.943+0000 7f58e27fc700 10 monclient: dump: 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58f2c6c700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58d0035ac0 0x7f58d0037f70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009f60 tx=0x7f58e4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f58e8011900 con 0x7f58ec06c5d0 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 msgr2=0x7f58ec1422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 0x7f58ec1422e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8007bb0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 msgr2=0x7f58ec1422e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:36.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.947+0000 7f58e27fc700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec06c5d0 0x7f58ec1422e0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8007bb0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f46cf700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f46cf700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 0x7f58ec02a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f46cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec064950 con 0x7f58ec029d00 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 0x7f58ec02a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient: get_auth_request con 0x7f58ec029d00 auth_method 0 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:36.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 10 monclient(hunting): handle_auth_done global_id 124113 payload 824 2024-09-18T10:19:36.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.955+0000 7f58f246b700 1 -- 172.21.15.142:0/2361522441 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f58ec06dca0 con 0x7f58ec029d00 2024-09-18T10:19:36.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f246b700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 msgr2=0x7f58ec02a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:36.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f246b700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 0x7f58ec02a100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e801c400 con 0x7f58ec029d00 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 10 monclient: dump: 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:36.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f58e801c560 con 0x7f58ec029d00 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 msgr2=0x7f58ec02a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 0x7f58ec02a100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8018000 comp rx=0 tx=0).stop 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 msgr2=0x7f58ec02a100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58e27fc700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58ec029d00 0x7f58ec02a100 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f58e80014d0 tx=0x7f58e8018000 comp rx=0 tx=0).stop 2024-09-18T10:19:36.968 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:36.969 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:36.970 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:36.971 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:36.972 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:36.973 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.974 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:36.975 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:36.976 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:36.977 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:36.978 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:36.979 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:36.980 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:36.981 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58d0035ac0 msgr2=0x7f58d0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f58d0035ac0 0x7f58d0037f70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009f60 tx=0x7f58e4017000 comp rx=0 tx=0).stop 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 10 monclient: shutdown 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 msgr2=0x7f58ec06bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 --2- 172.21.15.142:0/2361522441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f58ec09d870 0x7f58ec06bb50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f58dc00c7f0 tx=0x7f58dc007dc0 comp rx=0 tx=0).stop 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f2c6c700 1 -- 172.21.15.142:0/2361522441 reap_dead start 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f2c6c700 1 -- 172.21.15.142:0/2361522441 reap_dead start 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 shutdown_connections 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 >> 172.21.15.142:0/2361522441 conn(0x7f58ec0a14e0 msgr2=0x7f58ec134be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 shutdown_connections 2024-09-18T10:19:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:36.959+0000 7f58f46cf700 1 -- 172.21.15.142:0/2361522441 wait complete. 2024-09-18T10:19:36.984 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:36.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:19:36.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:19:37.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 10 monclient: get_monmap_and_config 2024-09-18T10:19:37.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 10 monclient: monmap: 2024-09-18T10:19:37.100 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:37.098772+0000 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:37.098772+0000 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:37.101 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 1 Processor -- start 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 1 -- start start 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.091+0000 7f1e824e7700 10 monclient: init 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1e7c09abd0 0x7f1e7c0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c09abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e7c09b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c09c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c09dc90 con 0x7f1e7c09b740 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c09b500 con 0x7f1e7c09abd0 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c09afa0 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient(hunting): authenticate will time out at 1466.024357s 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35376/0 (socket says 172.21.15.142:35376) 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 1 -- 172.21.15.142:0/1590885458 learned_addr learned my addr 172.21.15.142:0/1590885458 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): get_auth_request con 0x7f1e7c09c5e0 auth_method 0 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 --2- 172.21.15.142:0/1590885458 >> v2:172.21.15.142:3300/0 conn(0x7f1e7c09abd0 0x7f1e7c0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): get_auth_request con 0x7f1e7c09abd0 auth_method 0 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient(hunting): handle_auth_done global_id 124116 payload 274 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 -- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 msgr2=0x7f1e7c137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 -- 172.21.15.142:0/1590885458 >> v1:172.21.15.142:6789/0 conn(0x7f1e7c09b740 legacy=0x7f1e7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 monclient: found mon.noname-a 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 -- 172.21.15.142:0/1590885458 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c06f810 con 0x7f1e7c09abd0 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e81ce6700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 -- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09abd0 msgr2=0x7f1e7c0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e814e5700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09abd0 0x7f1e7c0960e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e700098e0 tx=0x7f1e700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1a2a2eed28fd56 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 1 -- 172.21.15.142:0/1590885458 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000b850 con 0x7f1e7c09abd0 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:37.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: dump: 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 5 monclient: authenticate success, global_id 124116 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 1 -- 172.21.15.142:0/1590885458 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7000b9b0 con 0x7f1e7c09abd0 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e824e7700 4 set_mon_vals no callback set 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 1 -- 172.21.15.142:0/1590885458 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000bd50 con 0x7f1e7c09abd0 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.095+0000 7f1e6ffff700 10 monclient: dump: 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.112 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: shutdown 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09abd0 msgr2=0x7f1e7c0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09abd0 0x7f1e7c0960e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e700098e0 tx=0x7f1e700091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- 172.21.15.142:0/1590885458 shutdown_connections 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09c5e0 0x7f1e7c137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- 172.21.15.142:0/1590885458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09abd0 0x7f1e7c0960e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- 172.21.15.142:0/1590885458 >> 172.21.15.142:0/1590885458 conn(0x7f1e7c070ef0 msgr2=0x7f1e7c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- 172.21.15.142:0/1590885458 shutdown_connections 2024-09-18T10:19:37.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- 172.21.15.142:0/1590885458 wait complete. 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: monmap: 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:37.119 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 Processor -- start 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- start start 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: init 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09abd0 0x7f1e7c13f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-c con 0x7f1e7c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e7c09b740 0x7f1e7c13fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-b con 0x7f1e7c09b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 0x7f1e7c08bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): picked mon.noname-a con 0x7f1e7c09c5e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:37.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c09d240 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c09ffe0 con 0x7f1e7c09b740 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c08f7b0 con 0x7f1e7c09abd0 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.099+0000 7f1e824e7700 10 monclient(hunting): authenticate will time out at 1466.031943s 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09abd0 0x7f1e7c13f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 0x7f1e7c08bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09abd0 0x7f1e7c13f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35388/0 (socket says 172.21.15.142:35388) 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 1 -- 172.21.15.142:0/3197621394 learned_addr learned my addr 172.21.15.142:0/3197621394 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): get_auth_request con 0x7f1e7c09abd0 auth_method 0 2024-09-18T10:19:37.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): get_auth_request con 0x7f1e7c09c5e0 auth_method 0 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e80ce4700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e7c09b740 0x7f1e7c13fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e80ce4700 10 monclient(hunting): get_auth_request con 0x7f1e7c09b740 auth_method 0 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e80ce4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e80ce4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e80ce4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient(hunting): handle_auth_done global_id 124119 payload 824 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09abd0 msgr2=0x7f1e7c13f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c09abd0 0x7f1e7c13f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e7c09b740 msgr2=0x7f1e7c13fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e7c09b740 0x7f1e7c13fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 monclient: found mon.noname-a 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009ae0 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 msgr2=0x7f1e7c08bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e81ce6700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 0x7f1e7c08bc30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009f10 tx=0x7f1e7400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74018070 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: dump: 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.124 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 5 monclient: authenticate success, global_id 124119 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7400f040 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 10 monclient: _renew_subs 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e70009420 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740092e0 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6f7fe700 4 set_mon_vals no callback set 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e6d7fa700 10 monclient: dump: 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 10 monclient: _renew_subs 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.103+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c08e300 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e824e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e824e7700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 0x7f1e7c08c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c064880 con 0x7f1e7c140360 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 0x7f1e7c08c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient: get_auth_request con 0x7f1e7c140360 auth_method 0 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 10 monclient(hunting): handle_auth_done global_id 75023 payload 824 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e7c072a80 con 0x7f1e7c140360 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1e74003020 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e60035ac0 0x7f1e60037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1e74048e80 con 0x7f1e7c09c5e0 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e80ce4700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e60035ac0 0x7f1e60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e80ce4700 10 monclient: get_auth_request con 0x7f1e60035ac0 auth_method 0 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 msgr2=0x7f1e7c08c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e814e5700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 0x7f1e7c08c170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1e700093f0 tx=0x7f1e70016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70021070 con 0x7f1e7c140360 2024-09-18T10:19:37.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 10 monclient: dump: 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1e7001c450 con 0x7f1e7c140360 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 msgr2=0x7f1e7c08c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 0x7f1e7c08c170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1e700093f0 tx=0x7f1e70016000 comp rx=0 tx=0).stop 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 msgr2=0x7f1e7c08c170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.107+0000 7f1e6d7fa700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c140360 0x7f1e7c08c170 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.111+0000 7f1e80ce4700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e60035ac0 0x7f1e60037f70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1e78003870 tx=0x7f1e78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e824e7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e824e7700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 0x7f1e7c08cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e824e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c08cff0 con 0x7f1e7c08c6b0 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 0x7f1e7c08cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient: get_auth_request con 0x7f1e7c08c6b0 auth_method 0 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 10 monclient(hunting): handle_auth_done global_id 75026 payload 824 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.119+0000 7f1e814e5700 1 -- 172.21.15.142:0/3197621394 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1e7c0008d0 con 0x7f1e7c08c6b0 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e814e5700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 msgr2=0x7f1e7c08cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e814e5700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 0x7f1e7c08cab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000a010 tx=0x7f1e7001e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70033670 con 0x7f1e7c08c6b0 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 10 monclient: dump: 2024-09-18T10:19:37.133 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f1e70011cd0 con 0x7f1e7c08c6b0 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 msgr2=0x7f1e7c08cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 0x7f1e7c08cab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000a010 tx=0x7f1e7001e070 comp rx=0 tx=0).stop 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 msgr2=0x7f1e7c08cab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e6d7fa700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1e7c08c6b0 0x7f1e7c08cab0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000a010 tx=0x7f1e7001e070 comp rx=0 tx=0).stop 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:37.135 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.136 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.137 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:37.138 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.139 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:37.141 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.142 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:37.144 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:37.145 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.146 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.147 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.148 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.148 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:37.148 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e60035ac0 msgr2=0x7f1e60037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1e60035ac0 0x7f1e60037f70 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1e78003870 tx=0x7f1e78009040 comp rx=0 tx=0).stop 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 10 monclient: shutdown 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 msgr2=0x7f1e7c08bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 --2- 172.21.15.142:0/3197621394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1e7c09c5e0 0x7f1e7c08bc30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009f10 tx=0x7f1e7400b040 comp rx=0 tx=0).stop 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 reap_dead start 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e81ce6700 1 -- 172.21.15.142:0/3197621394 reap_dead start 2024-09-18T10:19:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 shutdown_connections 2024-09-18T10:19:37.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 >> 172.21.15.142:0/3197621394 conn(0x7f1e7c070ef0 msgr2=0x7f1e7c09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 shutdown_connections 2024-09-18T10:19:37.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.123+0000 7f1e824e7700 1 -- 172.21.15.142:0/3197621394 wait complete. 2024-09-18T10:19:37.151 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:37.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:19:37.152 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:19:37.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.251+0000 7fa634bd5700 10 monclient: get_monmap_and_config 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.251+0000 7fa634bd5700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.251+0000 7fa634bd5700 10 monclient: monmap: 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:37.259083+0000 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:37.259083+0000 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:37.261 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:37.262 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:37.262 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 Processor -- start 2024-09-18T10:19:37.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 -- start start 2024-09-18T10:19:37.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient: init 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa630099cc0 0x7fa63009cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-c con 0x7fa630099cc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa63009c7d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-b con 0x7fa630134750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63009a8b0 con 0x7fa63009c7d0 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa63009aee0 con 0x7fa630099cc0 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63009a090 con 0x7fa630134750 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa634bd5700 10 monclient(hunting): authenticate will time out at 1466.186168s 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35424/0 (socket says 172.21.15.142:35424) 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 -- 172.21.15.142:0/551073603 learned_addr learned my addr 172.21.15.142:0/551073603 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): get_auth_request con 0x7fa630134750 auth_method 0 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62e59c700 1 --2- 172.21.15.142:0/551073603 >> v2:172.21.15.150:3300/0 conn(0x7fa630099cc0 0x7fa63009cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62e59c700 10 monclient(hunting): get_auth_request con 0x7fa630099cc0 auth_method 0 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_done global_id 75029 payload 274 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 -- 172.21.15.142:0/551073603 >> v2:172.21.15.150:3300/0 conn(0x7fa630099cc0 msgr2=0x7fa63009cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 --2- 172.21.15.142:0/551073603 >> v2:172.21.15.150:3300/0 conn(0x7fa630099cc0 0x7fa63009cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 -- 172.21.15.142:0/551073603 >> v1:172.21.15.150:6789/0 conn(0x7fa63009c7d0 legacy=0x7fa63009d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 monclient: found mon.noname-b 2024-09-18T10:19:37.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:37.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.255+0000 7fa62ed9d700 1 -- 172.21.15.142:0/551073603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63009b580 con 0x7fa630134750 2024-09-18T10:19:37.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62ed9d700 1 -- 172.21.15.142:0/551073603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 msgr2=0x7fa630136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62ed9d700 1 --2- 172.21.15.142:0/551073603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6280099b0 tx=0x7fa628009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba80004369256a81 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 1 -- 172.21.15.142:0/551073603 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62800ba70 con 0x7fa630134750 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: dump: 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.272 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 1 -- 172.21.15.142:0/551073603 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62800bbd0 con 0x7fa630134750 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 5 monclient: authenticate success, global_id 75029 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 1 -- 172.21.15.142:0/551073603 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628011420 con 0x7fa630134750 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 4 set_mon_vals no callback set 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa62d59a700 10 monclient: dump: 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 10 monclient: shutdown 2024-09-18T10:19:37.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 -- 172.21.15.142:0/551073603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 msgr2=0x7fa630136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 --2- 172.21.15.142:0/551073603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6280099b0 tx=0x7fa628009250 comp rx=0 tx=0).stop 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 -- 172.21.15.142:0/551073603 shutdown_connections 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 --2- 172.21.15.142:0/551073603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630134750 0x7fa630136b30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 --2- 172.21.15.142:0/551073603 >> v2:172.21.15.150:3300/0 conn(0x7fa630099cc0 0x7fa63009cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 -- 172.21.15.142:0/551073603 >> 172.21.15.142:0/551073603 conn(0x7fa6300a25d0 msgr2=0x7fa6300a29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 -- 172.21.15.142:0/551073603 shutdown_connections 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 1 -- 172.21.15.142:0/551073603 wait complete. 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.259+0000 7fa634bd5700 10 monclient: monmap: 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 Processor -- start 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 -- start start 2024-09-18T10:19:37.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient: init 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 0x7fa630098e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-c con 0x7fa630099cc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63009c7d0 0x7fa630070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-a con 0x7fa63009c7d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa630134750 0x7fa630071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): picked mon.noname-b con 0x7fa630134750 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63006f180 con 0x7fa63009c7d0 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63006cc60 con 0x7fa630134750 2024-09-18T10:19:37.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa630133900 con 0x7fa630099cc0 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa634bd5700 10 monclient(hunting): authenticate will time out at 1466.192707s 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 0x7fa630098e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63009c7d0 0x7fa630070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 0x7fa630098e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35440/0 (socket says 172.21.15.142:35440) 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 -- 172.21.15.142:0/1223993135 learned_addr learned my addr 172.21.15.142:0/1223993135 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): get_auth_request con 0x7fa630099cc0 auth_method 0 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): get_auth_request con 0x7fa63009c7d0 auth_method 0 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa630134750 0x7fa630071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): get_auth_request con 0x7fa630134750 auth_method 0 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient(hunting): handle_auth_done global_id 75032 payload 824 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa630134750 msgr2=0x7fa630071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa630134750 0x7fa630071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63009c7d0 msgr2=0x7fa630070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa63009c7d0 0x7fa630070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 monclient: found mon.noname-c 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:37.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6280094f0 con 0x7fa630099cc0 2024-09-18T10:19:37.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 msgr2=0x7fa630098e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa62e59c700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 0x7fa630098e30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa620009fa0 tx=0x7fa620009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.263+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62000ed50 con 0x7fa630099cc0 2024-09-18T10:19:37.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: dump: 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa620014420 con 0x7fa630099cc0 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 5 monclient: authenticate success, global_id 75032 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 monclient: _renew_subs 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63006ee30 con 0x7fa630099cc0 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6200147e0 con 0x7fa630099cc0 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 10 monclient: dump: 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.287 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62cd99700 4 set_mon_vals no callback set 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 monclient: _renew_subs 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63006cff0 con 0x7fa630099cc0 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 0x7fa630071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa630064a50 con 0x7fa63006f7a0 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 0x7fa630071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient: get_auth_request con 0x7fa63006f7a0 auth_method 0 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa620014b50 con 0x7fa630099cc0 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa610035af0 0x7fa610037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa620049630 con 0x7fa630099cc0 2024-09-18T10:19:37.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_done global_id 114244 payload 824 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62dd9b700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa63006ea90 con 0x7fa63006f7a0 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62ed9d700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa610035af0 0x7fa610037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.267+0000 7fa62ed9d700 10 monclient: get_auth_request con 0x7fa610035af0 auth_method 0 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa62dd9b700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 msgr2=0x7fa630071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa62dd9b700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 0x7fa630071970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624014890 con 0x7fa63006f7a0 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 10 monclient: dump: 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa62ed9d700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa610035af0 0x7fa610037fa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa6280099b0 tx=0x7fa628016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa6240149f0 con 0x7fa63006f7a0 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 msgr2=0x7fa630071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 0x7fa630071970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624007ea0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 msgr2=0x7fa630071970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:37.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.271+0000 7fa61e7fc700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa63006f7a0 0x7fa630071970 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624007ea0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa634bd5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:37.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa634bd5700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 0x7fa630094af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa634bd5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6300696a0 con 0x7fa6300946f0 2024-09-18T10:19:37.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 0x7fa630094af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient: get_auth_request con 0x7fa6300946f0 auth_method 0 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 10 monclient(hunting): handle_auth_done global_id 114247 payload 824 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 1 -- 172.21.15.142:0/1223993135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa630067a10 con 0x7fa6300946f0 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 msgr2=0x7fa630094af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.283+0000 7fa62dd9b700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 0x7fa630094af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624006820 con 0x7fa6300946f0 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 10 monclient: dump: 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.295 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.296 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.296 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fa624024070 con 0x7fa6300946f0 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 msgr2=0x7fa630094af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 0x7fa630094af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624019000 comp rx=0 tx=0).stop 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 msgr2=0x7fa630094af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa61e7fc700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6300946f0 0x7fa630094af0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa62400f780 tx=0x7fa624019000 comp rx=0 tx=0).stop 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:37.297 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:37.298 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.299 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:37.300 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:37.301 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:37.302 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:37.303 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:37.304 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:37.306 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:37.307 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.308 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:37.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:37.310 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:37.310 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa610035af0 msgr2=0x7fa610037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa610035af0 0x7fa610037fa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa6280099b0 tx=0x7fa628016000 comp rx=0 tx=0).stop 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 10 monclient: shutdown 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 msgr2=0x7fa630098e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 --2- 172.21.15.142:0/1223993135 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa630099cc0 0x7fa630098e30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa620009fa0 tx=0x7fa620009880 comp rx=0 tx=0).stop 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa62ed9d700 1 -- 172.21.15.142:0/1223993135 reap_dead start 2024-09-18T10:19:37.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa62ed9d700 1 -- 172.21.15.142:0/1223993135 reap_dead start 2024-09-18T10:19:37.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 shutdown_connections 2024-09-18T10:19:37.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 >> 172.21.15.142:0/1223993135 conn(0x7fa6300a25d0 msgr2=0x7fa6300933a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 shutdown_connections 2024-09-18T10:19:37.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.287+0000 7fa634bd5700 1 -- 172.21.15.142:0/1223993135 wait complete. 2024-09-18T10:19:37.312 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:37.312 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:19:37.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:19:37.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.415+0000 7f56cf09b700 10 monclient: get_monmap_and_config 2024-09-18T10:19:37.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.415+0000 7f56cf09b700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.415+0000 7f56cf09b700 10 monclient: monmap: 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:37.422813+0000 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:37.422813+0000 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.425 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:37.426 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:37.426 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:37.426 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:37.426 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:37.426 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.415+0000 7f56cf09b700 1 Processor -- start 2024-09-18T10:19:37.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 -- start start 2024-09-18T10:19:37.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient: init 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-b con 0x7f56c809aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f56c80967a0 0x7f56c809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-a con 0x7f56c80967a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56c809c320 0x7f56c809c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-c con 0x7f56c809c320 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c8095880 con 0x7f56c80967a0 2024-09-18T10:19:37.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c809aeb0 con 0x7f56c809c320 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c806f6d0 con 0x7f56c809aae0 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient(hunting): authenticate will time out at 1466.348515s 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f56c80967a0 0x7f56c809bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f56c80967a0 0x7f56c809bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43696/0 (socket says 172.21.15.142:43696) 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 -- 172.21.15.142:0/1970263343 learned_addr learned my addr 172.21.15.142:0/1970263343 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): get_auth_request con 0x7f56c80967a0 auth_method 0 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce099700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce89a700 1 --2- 172.21.15.142:0/1970263343 >> v2:172.21.15.150:3300/0 conn(0x7f56c809c320 0x7f56c809c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce099700 10 monclient(hunting): get_auth_request con 0x7f56c809aae0 auth_method 0 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce099700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce099700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce099700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce89a700 10 monclient(hunting): get_auth_request con 0x7f56c809c320 auth_method 0 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce89a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce89a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56ce89a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient(hunting): handle_auth_done global_id 124125 payload 274 2024-09-18T10:19:37.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 -- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 msgr2=0x7f56c8096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 -- 172.21.15.142:0/1970263343 >> v2:172.21.15.150:3300/0 conn(0x7f56c809c320 msgr2=0x7f56c809c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 --2- 172.21.15.142:0/1970263343 >> v2:172.21.15.150:3300/0 conn(0x7f56c809c320 0x7f56c809c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 monclient: found mon.noname-a 2024-09-18T10:19:37.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 -- 172.21.15.142:0/1970263343 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c80955a0 con 0x7f56c80967a0 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 -- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c80967a0 msgr2=0x7f56c809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd898700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c80967a0 0x7f56c809bde0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56c8095210 tx=0x7f56c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594ce26a2817547c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 1 -- 172.21.15.142:0/1970263343 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c400ba70 con 0x7f56c80967a0 2024-09-18T10:19:37.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: dump: 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.440 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.441 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 1 -- 172.21.15.142:0/1970263343 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c400bbd0 con 0x7f56c80967a0 2024-09-18T10:19:37.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 5 monclient: authenticate success, global_id 124125 2024-09-18T10:19:37.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 1 -- 172.21.15.142:0/1970263343 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4011420 con 0x7f56c80967a0 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cf09b700 4 set_mon_vals no callback set 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.419+0000 7f56cd097700 10 monclient: dump: 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.443 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 10 monclient: shutdown 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c80967a0 msgr2=0x7f56c809bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c80967a0 0x7f56c809bde0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f56c8095210 tx=0x7f56c4009250 comp rx=0 tx=0).stop 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- 172.21.15.142:0/1970263343 shutdown_connections 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1970263343 >> v2:172.21.15.150:3300/0 conn(0x7f56c809c320 0x7f56c809c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c80967a0 0x7f56c809bde0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1970263343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- 172.21.15.142:0/1970263343 >> 172.21.15.142:0/1970263343 conn(0x7f56c80a1fe0 msgr2=0x7f56c8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- 172.21.15.142:0/1970263343 shutdown_connections 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- 172.21.15.142:0/1970263343 wait complete. 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 10 monclient: build_initial_monmap 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 10 monclient: monmap: 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:37.445 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:37.446 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 Processor -- start 2024-09-18T10:19:37.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 1 -- start start 2024-09-18T10:19:37.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.423+0000 7f56cf09b700 10 monclient: init 2024-09-18T10:19:37.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:37.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c80967a0 0x7f56c81355b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-b con 0x7f56c80967a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8135af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-c con 0x7f56c809aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): picked mon.noname-a con 0x7f56c809c320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c808b300 con 0x7f56c809c320 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c806f8d0 con 0x7f56c80967a0 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80a09d0 con 0x7f56c809aae0 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:37.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient(hunting): authenticate will time out at 1466.356373s 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8135af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43700/0 (socket says 172.21.15.142:43700) 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 -- 172.21.15.142:0/1485646291 learned_addr learned my addr 172.21.15.142:0/1485646291 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): get_auth_request con 0x7f56c809c320 auth_method 0 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8135af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35468/0 (socket says 172.21.15.142:35468) 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): get_auth_request con 0x7f56c809aae0 auth_method 0 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c80967a0 0x7f56c81355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): get_auth_request con 0x7f56c80967a0 auth_method 0 2024-09-18T10:19:37.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:37.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:37.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cd898700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:37.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce099700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient(hunting): handle_auth_done global_id 124128 payload 824 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 -- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 msgr2=0x7f56c8135af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8135af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 -- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c80967a0 msgr2=0x7f56c81355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c80967a0 0x7f56c81355b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient: _finish_hunting 0 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 monclient: found mon.noname-a 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 -- 172.21.15.142:0/1485646291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c40094f0 con 0x7f56c809c320 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 -- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 msgr2=0x7f56c8136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:37.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ce89a700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009fa0 tx=0x7f56c0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c000edd0 con 0x7f56c809c320 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: dump: 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.456 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 5 monclient: authenticate success, global_id 124128 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c0014420 con 0x7f56c809c320 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c00147e0 con 0x7f56c809c320 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56ba7fc700 10 monclient: dump: 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:37.457 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient: _renew_subs 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cc896700 4 set_mon_vals no callback set 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c808bee0 con 0x7f56c809c320 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.427+0000 7f56cf09b700 10 monclient: _renew_subs 2024-09-18T10:19:37.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c8094d40 con 0x7f56c809c320 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f56c0014b50 con 0x7f56c809c320 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c8094f60 con 0x7f56c809c320 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56ba7fc700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 0x7f56ac037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f56c00495e0 con 0x7f56c809c320 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56ce099700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 0x7f56ac037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.431+0000 7f56ce099700 10 monclient: get_auth_request con 0x7f56ac035af0 auth_method 0 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.435+0000 7f56ce099700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 0x7f56ac037fa0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006fd0 tx=0x7f56bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.443+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f56c0014e00 con 0x7f56c809c320 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.443+0000 7f56ba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:37.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.443+0000 7f56ba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:37.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.667+0000 7f56cf09b700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:19:37.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.667+0000 7f56cf09b700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:19:37.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.667+0000 7f56cf09b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:37.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.667+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f56c8072a80 con 0x7f56c809c320 2024-09-18T10:19:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f56ba7fc700 1 -- 172.21.15.142:0/1485646291 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f56c001bb00 con 0x7f56c809c320 2024-09-18T10:19:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f56ba7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:19:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f56ba7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:37.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=2550322299,config=3140514009,health=3670073431,kv=4261215878,logm=910144169}) 2024-09-18T10:19:37.678 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=1753511519,config=3140514009,health=528802886,kv=4261215878,logm=3075301541}) 2024-09-18T10:19:37.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=2550322299,config=3140514009,health=3670073431,kv=4261215878,logm=910144169}) 2024-09-18T10:19:37.679 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=1,health=10,kv=8,logm=58} crc {auth=3153290628,config=1228700967,health=2486391017,kv=4261215878,logm=2671809358}) 2024-09-18T10:19:37.681 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.681 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586077242}) 2024-09-18T10:19:37.681 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.671+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553912363}) 2024-09-18T10:19:37.681 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.681 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586077242}) 2024-09-18T10:19:37.682 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4154282711}) 2024-09-18T10:19:37.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 msgr2=0x7f56ac037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 0x7f56ac037fa0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006fd0 tx=0x7f56bc008040 comp rx=0 tx=0).stop 2024-09-18T10:19:37.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 10 monclient: shutdown 2024-09-18T10:19:37.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 msgr2=0x7f56c8136030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009fa0 tx=0x7f56c0009900 comp rx=0 tx=0).stop 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 shutdown_connections 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56ac035af0 0x7f56ac037fa0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56c809c320 0x7f56c8136030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56c809aae0 0x7f56c8135af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 --2- 172.21.15.142:0/1485646291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56c80967a0 0x7f56c81355b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 >> 172.21.15.142:0/1485646291 conn(0x7f56c80a1fe0 msgr2=0x7f56c8099b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 shutdown_connections 2024-09-18T10:19:37.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f56cf09b700 1 -- 172.21.15.142:0/1485646291 wait complete. 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532524277}) 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239318653}) 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532524277}) 2024-09-18T10:19:37.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298978165}) 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039954941}) 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487841220}) 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1039954941}) 2024-09-18T10:19:37.689 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4115094795}) 2024-09-18T10:19:37.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977509255}) 2024-09-18T10:19:37.691 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954624636}) 2024-09-18T10:19:37.691 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.691 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977509255}) 2024-09-18T10:19:37.691 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801227209}) 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176252317}) 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129673361}) 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176252317}) 2024-09-18T10:19:37.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059707296}) 2024-09-18T10:19:37.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670051865}) 2024-09-18T10:19:37.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830925380}) 2024-09-18T10:19:37.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670051865}) 2024-09-18T10:19:37.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178072422}) 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843605094}) 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047682686}) 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843605094}) 2024-09-18T10:19:37.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127221813}) 2024-09-18T10:19:37.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:19:37.696 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910301113}) 2024-09-18T10:19:37.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412390351}) 2024-09-18T10:19:37.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910301113}) 2024-09-18T10:19:37.697 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1143805206}) 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380688464}) 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520666198}) 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380688464}) 2024-09-18T10:19:37.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624953681}) 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734301227}) 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=586674340}) 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734301227}) 2024-09-18T10:19:37.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=440189811}) 2024-09-18T10:19:37.700 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658125513}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4237692898}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658125513}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982323405}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401292031}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1944088975}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401292031}) 2024-09-18T10:19:37.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051517044}) 2024-09-18T10:19:37.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261953522}) 2024-09-18T10:19:37.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042382169}) 2024-09-18T10:19:37.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261953522}) 2024-09-18T10:19:37.711 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1697553235}) 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244724142}) 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211968819}) 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1244724142}) 2024-09-18T10:19:37.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.703+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43146383}) 2024-09-18T10:19:37.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991267222}) 2024-09-18T10:19:37.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041409602}) 2024-09-18T10:19:37.714 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.714 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991267222}) 2024-09-18T10:19:37.714 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601833664}) 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794995100}) 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949738424}) 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794995100}) 2024-09-18T10:19:37.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888095471}) 2024-09-18T10:19:37.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097922489}) 2024-09-18T10:19:37.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741384610}) 2024-09-18T10:19:37.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097922489}) 2024-09-18T10:19:37.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766129910}) 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400657236}) 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681366855}) 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400657236}) 2024-09-18T10:19:37.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569670166}) 2024-09-18T10:19:37.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149645341}) 2024-09-18T10:19:37.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3977721104}) 2024-09-18T10:19:37.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149645341}) 2024-09-18T10:19:37.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=741749565}) 2024-09-18T10:19:37.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889447751}) 2024-09-18T10:19:37.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3444165238}) 2024-09-18T10:19:37.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889447751}) 2024-09-18T10:19:37.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.711+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577449066}) 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925569071}) 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534092346}) 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925569071}) 2024-09-18T10:19:37.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296494820}) 2024-09-18T10:19:37.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652818443}) 2024-09-18T10:19:37.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214187041}) 2024-09-18T10:19:37.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652818443}) 2024-09-18T10:19:37.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226623406}) 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153328503}) 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659494727}) 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153328503}) 2024-09-18T10:19:37.725 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.715+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673580009}) 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550390548}) 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2102329020}) 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550390548}) 2024-09-18T10:19:37.726 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882521197}) 2024-09-18T10:19:37.727 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.727 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511599386}) 2024-09-18T10:19:37.727 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405021939}) 2024-09-18T10:19:37.727 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.727 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2511599386}) 2024-09-18T10:19:37.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450598091}) 2024-09-18T10:19:37.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456691193}) 2024-09-18T10:19:37.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3701644200}) 2024-09-18T10:19:37.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456691193}) 2024-09-18T10:19:37.729 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.719+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3039666913}) 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393937545}) 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3576184879}) 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393937545}) 2024-09-18T10:19:37.730 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568070060}) 2024-09-18T10:19:37.731 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.731 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217428991}) 2024-09-18T10:19:37.731 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260122988}) 2024-09-18T10:19:37.731 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3217428991}) 2024-09-18T10:19:37.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=989922565}) 2024-09-18T10:19:37.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.732 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582251971}) 2024-09-18T10:19:37.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4139298387}) 2024-09-18T10:19:37.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582251971}) 2024-09-18T10:19:37.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.723+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1113144613}) 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910060375}) 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042206194}) 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910060375}) 2024-09-18T10:19:37.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=896003562}) 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754153780}) 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763292985}) 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754153780}) 2024-09-18T10:19:37.735 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389740239}) 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291759687}) 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=27837640}) 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291759687}) 2024-09-18T10:19:37.737 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.727+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984176951}) 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072294705}) 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339387024}) 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2072294705}) 2024-09-18T10:19:37.738 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794327496}) 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792860523}) 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3907305074}) 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792860523}) 2024-09-18T10:19:37.739 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261623170}) 2024-09-18T10:19:37.740 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.741 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693248486}) 2024-09-18T10:19:37.741 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225224654}) 2024-09-18T10:19:37.741 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.741 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693248486}) 2024-09-18T10:19:37.741 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.731+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531606514}) 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945008680}) 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369182729}) 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945008680}) 2024-09-18T10:19:37.742 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2279006527}) 2024-09-18T10:19:37.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506292165}) 2024-09-18T10:19:37.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1357016721}) 2024-09-18T10:19:37.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.744 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506292165}) 2024-09-18T10:19:37.744 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656340432}) 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705196254}) 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661598277}) 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705196254}) 2024-09-18T10:19:37.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.735+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392601821}) 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738527948}) 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758170247}) 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738527948}) 2024-09-18T10:19:37.746 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:37.739+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010400536}) 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696598722}) 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581179890}) 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696598722}) 2024-09-18T10:19:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.027+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554158142}) 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230016497}) 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1710126873}) 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230016497}) 2024-09-18T10:19:38.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.139+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1987164937}) 2024-09-18T10:19:38.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448437860}) 2024-09-18T10:19:38.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=751055507}) 2024-09-18T10:19:38.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448437860}) 2024-09-18T10:19:38.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.155+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158380237}) 2024-09-18T10:19:38.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729799292}) 2024-09-18T10:19:38.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528002503}) 2024-09-18T10:19:38.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729799292}) 2024-09-18T10:19:38.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008413634}) 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198890234}) 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351608886}) 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198890234}) 2024-09-18T10:19:38.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2274439976}) 2024-09-18T10:19:38.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368984235}) 2024-09-18T10:19:38.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956499809}) 2024-09-18T10:19:38.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368984235}) 2024-09-18T10:19:38.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.167+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234310816}) 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283665045}) 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3962901434}) 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283665045}) 2024-09-18T10:19:38.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206168122}) 2024-09-18T10:19:38.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1189558495}) 2024-09-18T10:19:38.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1693518741}) 2024-09-18T10:19:38.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1189558495}) 2024-09-18T10:19:38.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1966400995}) 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262949059}) 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323429082}) 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262949059}) 2024-09-18T10:19:38.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.171+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428872368}) 2024-09-18T10:19:38.682 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.682 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079422275}) 2024-09-18T10:19:38.683 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363095935}) 2024-09-18T10:19:38.683 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.683 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079422275}) 2024-09-18T10:19:38.683 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195066303}) 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561432966}) 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463178701}) 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561432966}) 2024-09-18T10:19:38.685 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.675+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789060193}) 2024-09-18T10:19:38.687 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.687 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831595682}) 2024-09-18T10:19:38.687 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567644080}) 2024-09-18T10:19:38.687 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.688 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831595682}) 2024-09-18T10:19:38.688 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.679+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964043378}) 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330313501}) 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73235506}) 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330313501}) 2024-09-18T10:19:38.690 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3357952961}) 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020721649}) 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058188575}) 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020721649}) 2024-09-18T10:19:38.692 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.683+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924732063}) 2024-09-18T10:19:38.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141836946}) 2024-09-18T10:19:38.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203337466}) 2024-09-18T10:19:38.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4141836946}) 2024-09-18T10:19:38.695 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653773307}) 2024-09-18T10:19:38.696 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:19:38.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:38.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365576014}) 2024-09-18T10:19:38.698 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841250398}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365576014}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.687+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418019416}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753927661}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307087188}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753927661}) 2024-09-18T10:19:38.699 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2588641840}) 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581245071}) 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155999691}) 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581245071}) 2024-09-18T10:19:38.701 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.691+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1723365675}) 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=2279616157,mdsmap=1531151339,mgr=4188356461}) 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2025052992,mdsmap=1531151339,mgr=2286626049}) 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=2279616157,mdsmap=1531151339,mgr=4188356461}) 2024-09-18T10:19:38.702 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.695+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1474425337,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgr=2252546380,mgr_command_descs=47663135,mgrstat=3926230508,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4293609621}) 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=1176933844,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3926230508,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=796330106}) 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgr=2252546380,mgr_command_descs=47663135,mgrstat=3926230508,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4293609621}) 2024-09-18T10:19:38.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.699+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=1176933844,mgr_metadata=2738435100,mgrstat=3926230508,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=3201831156}) 2024-09-18T10:19:38.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=82} crc {osdmap=1559891101}) 2024-09-18T10:19:38.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=76} crc {osdmap=965241190}) 2024-09-18T10:19:38.716 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:19:38.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=82} crc {osdmap=1559891101}) 2024-09-18T10:19:38.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:19:38.707+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=79} crc {osdmap=3697199511}) 2024-09-18T10:19:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:19:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: build_initial_monmap 2024-09-18T10:19:38.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: monmap: 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:38.800865+0000 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:38.800865+0000 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:38.803 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 Processor -- start 2024-09-18T10:19:38.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 -- start start 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: init 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac1c0676a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac1c098650 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 0x7fac1c13a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fac1c098a90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c09e0c0 con 0x7fac1c098650 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1c132850 con 0x7fac1c0676a0 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c096e40 con 0x7fac1c098a90 2024-09-18T10:19:38.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1bfff700 10 monclient(hunting): authenticate will time out at 1467.727250s 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac13fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fac1c098650 0x7fac1c137170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55804/0 (socket says 172.21.15.142:55804) 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac13fff700 1 -- 172.21.15.142:0/2669343470 learned_addr learned my addr 172.21.15.142:0/2669343470 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 1 --2- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 0x7fac1c13a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): get_auth_request con 0x7fac1c098a90 auth_method 0 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.795+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_done global_id 75041 payload 274 2024-09-18T10:19:38.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 -- 172.21.15.142:0/2669343470 >> v1:172.21.15.150:6789/0 conn(0x7fac1c0676a0 legacy=0x7fac1c132140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 -- 172.21.15.142:0/2669343470 >> v1:172.21.15.142:6789/0 conn(0x7fac1c098650 legacy=0x7fac1c137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:38.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 -- 172.21.15.142:0/2669343470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c067a70 con 0x7fac1c098a90 2024-09-18T10:19:38.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 -- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 msgr2=0x7fac1c13a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1b7fe700 1 --2- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 0x7fac1c13a8a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fac14009e90 tx=0x7fac14009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d73976979e4f3b8c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 1 -- 172.21.15.142:0/2669343470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1400e070 con 0x7fac1c098a90 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: dump: 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:38.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 1 -- 172.21.15.142:0/2669343470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14012430 con 0x7fac1c098a90 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1bfff700 5 monclient: authenticate success, global_id 75041 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 1 -- 172.21.15.142:0/2669343470 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac140127d0 con 0x7fac1c098a90 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1bfff700 4 set_mon_vals no callback set 2024-09-18T10:19:38.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.799+0000 7fac1a7fc700 10 monclient: dump: 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:38.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: shutdown 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 msgr2=0x7fac1c13a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 --2- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 0x7fac1c13a8a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fac14009e90 tx=0x7fac14009780 comp rx=0 tx=0).stop 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- 172.21.15.142:0/2669343470 shutdown_connections 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 --2- 172.21.15.142:0/2669343470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c098a90 0x7fac1c13a8a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- 172.21.15.142:0/2669343470 >> 172.21.15.142:0/2669343470 conn(0x7fac1c071ae0 msgr2=0x7fac1c0a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- 172.21.15.142:0/2669343470 shutdown_connections 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- 172.21.15.142:0/2669343470 wait complete. 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: build_initial_monmap 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: monmap: 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:38.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 Processor -- start 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- start start 2024-09-18T10:19:38.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: init 2024-09-18T10:19:38.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:38.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 0x7fac1c12c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fac1c0676a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 0x7fac1c12ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fac1c098650 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 0x7fac1c12e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fac1c098a90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1189f0 con 0x7fac1c098a90 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c12c650 con 0x7fac1c098650 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c1184e0 con 0x7fac1c0676a0 2024-09-18T10:19:38.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1bfff700 10 monclient(hunting): authenticate will time out at 1467.734596s 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 0x7fac1c12c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 0x7fac1c12e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 0x7fac1c12c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35500/0 (socket says 172.21.15.142:35500) 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 1 -- 172.21.15.142:0/402325420 learned_addr learned my addr 172.21.15.142:0/402325420 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): get_auth_request con 0x7fac1c0676a0 auth_method 0 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): get_auth_request con 0x7fac1c098a90 auth_method 0 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:38.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 0x7fac1c12ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 10 monclient(hunting): get_auth_request con 0x7fac1c098650 auth_method 0 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:38.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient(hunting): handle_auth_done global_id 124131 payload 824 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 -- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 msgr2=0x7fac1c12c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 0x7fac1c12c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 -- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 msgr2=0x7fac1c12ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 0x7fac1c12ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac1b7fe700 1 -- 172.21.15.142:0/402325420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14009a20 con 0x7fac1c098a90 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.803+0000 7fac13fff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 0x7fac1c12ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1b7fe700 1 -- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 msgr2=0x7fac1c12e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1b7fe700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 0x7fac1c12e410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac14009fd0 tx=0x7fac14018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14026430 con 0x7fac1c098a90 2024-09-18T10:19:38.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: dump: 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:38.824 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: _finish_auth 0 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac14021070 con 0x7fac1c098a90 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 5 monclient: authenticate success, global_id 124131 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac14026840 con 0x7fac1c098a90 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac12ffd700 10 monclient: dump: 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:38.825 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _renew_subs 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac19ffb700 4 set_mon_vals no callback set 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c09e250 con 0x7fac1c098a90 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _renew_subs 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c12b5d0 con 0x7fac1c098a90 2024-09-18T10:19:38.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.807+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c072a80 con 0x7fac1c098a90 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fac14023030 con 0x7fac1c098a90 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac12ffd700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 0x7fabfc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fac1404b790 con 0x7fac1c098a90 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac1affd700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 0x7fabfc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:38.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac1affd700 10 monclient: get_auth_request con 0x7fabfc035ac0 auth_method 0 2024-09-18T10:19:38.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.811+0000 7fac1affd700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 0x7fabfc037f70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fac0c007fd0 tx=0x7fac0c007f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:38.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.819+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fac1401d420 con 0x7fac1c098a90 2024-09-18T10:19:38.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.819+0000 7fac12ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:38.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:38.819+0000 7fac12ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:39.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fac1c072a80 con 0x7fac1c098a90 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac12ffd700 1 -- 172.21.15.142:0/402325420 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fac1402e390 con 0x7fac1c098a90 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac12ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:39.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac12ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:39.055 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 msgr2=0x7fabfc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 0x7fabfc037f70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fac0c007fd0 tx=0x7fac0c007f60 comp rx=0 tx=0).stop 2024-09-18T10:19:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 10 monclient: shutdown 2024-09-18T10:19:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 msgr2=0x7fac1c12e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 0x7fac1c12e410 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac14009fd0 tx=0x7fac14018000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 shutdown_connections 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fabfc035ac0 0x7fabfc037f70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac1c098a90 0x7fac1c12e410 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c098650 0x7fac1c12ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 --2- 172.21.15.142:0/402325420 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac1c0676a0 0x7fac1c12c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.047+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 >> 172.21.15.142:0/402325420 conn(0x7fac1c071ae0 msgr2=0x7fac1c12fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.051+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 shutdown_connections 2024-09-18T10:19:39.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.051+0000 7fac1bfff700 1 -- 172.21.15.142:0/402325420 wait complete. 2024-09-18T10:19:39.072 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:19:39.072 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:19:39.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:19:39.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:19:39.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 10 monclient: get_monmap_and_config 2024-09-18T10:19:39.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 10 monclient: monmap: 2024-09-18T10:19:39.223 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:39.221728+0000 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:39.221728+0000 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:39.224 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 1 Processor -- start 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 1 -- start start 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.215+0000 7fbb0df68700 10 monclient: init 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:19:39.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb0809cdd0 0x7fbb0809e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-a con 0x7fbb0809cdd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-b con 0x7fbb080a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb080a0bf0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0809d1a0 con 0x7fbb080a0bf0 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0809e980 con 0x7fbb0809cdd0 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0809d780 con 0x7fbb080a02e0 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient(hunting): authenticate will time out at 1468.148526s 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb0809cdd0 0x7fbb0809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35510/0 (socket says 172.21.15.142:35510) 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 -- 172.21.15.142:0/1313441945 learned_addr learned my addr 172.21.15.142:0/1313441945 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): get_auth_request con 0x7fbb080a02e0 auth_method 0 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): get_auth_request con 0x7fbb0809cdd0 auth_method 0 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_done global_id 75047 payload 274 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 -- 172.21.15.142:0/1313441945 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809cdd0 msgr2=0x7fbb0809e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 --2- 172.21.15.142:0/1313441945 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809cdd0 0x7fbb0809e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 -- 172.21.15.142:0/1313441945 >> v1:172.21.15.150:6789/0 conn(0x7fbb080a0bf0 legacy=0x7fbb08138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 monclient: found mon.noname-b 2024-09-18T10:19:39.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 -- 172.21.15.142:0/1313441945 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0809dd40 con 0x7fbb080a02e0 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 -- 172.21.15.142:0/1313441945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 msgr2=0x7fbb080a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb06ffd700 1 --2- 172.21.15.142:0/1313441945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00099b0 tx=0x7fbaf0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd96e41ba45b65d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 1 -- 172.21.15.142:0/1313441945 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0009ea0 con 0x7fbb080a02e0 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: dump: 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.235 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 5 monclient: authenticate success, global_id 75047 2024-09-18T10:19:39.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 1 -- 172.21.15.142:0/1313441945 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf000ba70 con 0x7fbb080a02e0 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb0df68700 4 set_mon_vals no callback set 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 1 -- 172.21.15.142:0/1313441945 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf000be30 con 0x7fbb080a02e0 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.219+0000 7fbb067fc700 10 monclient: dump: 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.237 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 10 monclient: shutdown 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- 172.21.15.142:0/1313441945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 msgr2=0x7fbb080a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 --2- 172.21.15.142:0/1313441945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00099b0 tx=0x7fbaf0009250 comp rx=0 tx=0).stop 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- 172.21.15.142:0/1313441945 shutdown_connections 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 --2- 172.21.15.142:0/1313441945 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a02e0 0x7fbb080a06b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 --2- 172.21.15.142:0/1313441945 >> v2:172.21.15.142:3300/0 conn(0x7fbb0809cdd0 0x7fbb0809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- 172.21.15.142:0/1313441945 >> 172.21.15.142:0/1313441945 conn(0x7fbb08071680 msgr2=0x7fbb080a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- 172.21.15.142:0/1313441945 shutdown_connections 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- 172.21.15.142:0/1313441945 wait complete. 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 10 monclient: monmap: 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:39.239 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 Processor -- start 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 1 -- start start 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 10 monclient: init 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.223+0000 7fbb0df68700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb0809cdd0 0x7fbb0808cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-b con 0x7fbb0809cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 0x7fbb0808d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-a con 0x7fbb080a02e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a0bf0 0x7fbb0808da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): picked mon.noname-c con 0x7fbb080a0bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0809c070 con 0x7fbb080a02e0 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080905f0 con 0x7fbb0809cdd0 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0808e7e0 con 0x7fbb080a0bf0 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient(hunting): authenticate will time out at 1468.156320s 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 0x7fbb0808d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 0x7fbb0808d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43732/0 (socket says 172.21.15.142:43732) 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 -- 172.21.15.142:0/3751565591 learned_addr learned my addr 172.21.15.142:0/3751565591 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): get_auth_request con 0x7fbb080a02e0 auth_method 0 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a0bf0 0x7fbb0808da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): get_auth_request con 0x7fbb080a0bf0 auth_method 0 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb0809cdd0 0x7fbb0808cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): get_auth_request con 0x7fbb0809cdd0 auth_method 0 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient(hunting): handle_auth_done global_id 124137 payload 824 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a0bf0 msgr2=0x7fbb0808da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a0bf0 0x7fbb0808da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb0809cdd0 msgr2=0x7fbb0808cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb0809cdd0 0x7fbb0808cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 monclient: found mon.noname-a 2024-09-18T10:19:39.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00094f0 con 0x7fbb080a02e0 2024-09-18T10:19:39.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb07fff700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb080a0bf0 0x7fbb0808da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:39.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 msgr2=0x7fbb0808d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb06ffd700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 0x7fbb0808d4f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb0809d350 tx=0x7fbaf0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0006b60 con 0x7fbb080a02e0 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: dump: 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.248 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 5 monclient: authenticate success, global_id 124137 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf0020070 con 0x7fbb080a02e0 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient: _renew_subs 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb08090310 con 0x7fbb080a02e0 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0011cd0 con 0x7fbb080a02e0 2024-09-18T10:19:39.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbaff7fe700 10 monclient: dump: 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb05ffb700 4 set_mon_vals no callback set 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient: _renew_subs 2024-09-18T10:19:39.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.227+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb08091530 con 0x7fbb080a02e0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbaf0022030 con 0x7fbb080a02e0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb0df68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb0df68700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 0x7fbb0806cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08090a70 con 0x7fbb0808df70 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbaff7fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbae8035ac0 0x7fbae8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 0x7fbb0806cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient: get_auth_request con 0x7fbb0808df70 auth_method 0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fbaf004aae0 con 0x7fbb080a02e0 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb07fff700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbae8035ac0 0x7fbae8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb07fff700 10 monclient: get_auth_request con 0x7fbae8035ac0 auth_method 0 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 10 monclient(hunting): handle_auth_done global_id 124140 payload 824 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.231+0000 7fbb077fe700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb08090000 con 0x7fbb0808df70 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbb077fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 msgr2=0x7fbb0806cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbb077fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 0x7fbb0806cbc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8011c20 tx=0x7fbaf80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf800fe40 con 0x7fbb0808df70 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 10 monclient: dump: 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbb07fff700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbae8035ac0 0x7fbae8037f70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbb00007fd0 tx=0x7fbb00002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbaf8014440 con 0x7fbb0808df70 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 msgr2=0x7fbb0806cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 0x7fbb0806cbc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8011c20 tx=0x7fbaf80091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 msgr2=0x7fbb0806cbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.235+0000 7fbaff7fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0808df70 0x7fbb0806cbc0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8011c20 tx=0x7fbaf80091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb0df68700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb0df68700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 0x7fbb0806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb0df68700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb080648c0 con 0x7fbb0806d100 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb077fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 0x7fbb0806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb077fe700 10 monclient: get_auth_request con 0x7fbb0806d100 auth_method 0 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.243+0000 7fbb077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 10 monclient(hunting): handle_auth_done global_id 124143 payload 824 2024-09-18T10:19:39.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 1 -- 172.21.15.142:0/3751565591 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb080696a0 con 0x7fbb0806d100 2024-09-18T10:19:39.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 msgr2=0x7fbb0806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbb077fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 0x7fbb0806d500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009940 tx=0x7fbaf801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf802b070 con 0x7fbb0806d100 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 10 monclient: dump: 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.257 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fbaf8023040 con 0x7fbb0806d100 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 msgr2=0x7fbb0806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 0x7fbb0806d500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009940 tx=0x7fbaf801c000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 msgr2=0x7fbb0806d500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.247+0000 7fbaff7fe700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb0806d100 0x7fbb0806d500 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009940 tx=0x7fbaf801c000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.258 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.259 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:39.260 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:39.261 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:39.262 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.263 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.264 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:39.265 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.266 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.267 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.268 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.269 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:39.270 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:39.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbae8035ac0 msgr2=0x7fbae8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbae8035ac0 0x7fbae8037f70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbb00007fd0 tx=0x7fbb00002620 comp rx=0 tx=0).stop 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 10 monclient: shutdown 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 msgr2=0x7fbb0808d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 --2- 172.21.15.142:0/3751565591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb080a02e0 0x7fbb0808d4f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb0809d350 tx=0x7fbaf0016000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb07fff700 1 -- 172.21.15.142:0/3751565591 reap_dead start 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb07fff700 1 -- 172.21.15.142:0/3751565591 reap_dead start 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 shutdown_connections 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 >> 172.21.15.142:0/3751565591 conn(0x7fbb08071680 msgr2=0x7fbb0809eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 shutdown_connections 2024-09-18T10:19:39.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.251+0000 7fbb0df68700 1 -- 172.21.15.142:0/3751565591 wait complete. 2024-09-18T10:19:39.273 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:39.273 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:19:39.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:19:39.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: get_monmap_and_config 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: monmap: 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:39.385150+0000 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:39.385150+0000 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:39.388 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 Processor -- start 2024-09-18T10:19:39.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 -- start start 2024-09-18T10:19:39.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: init 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e2c09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e2c09b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c138170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c09dc90 con 0x7f7e2c09b740 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c09afa0 con 0x7f7e2c09abd0 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c09b500 con 0x7f7e2c135d90 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3282b700 10 monclient(hunting): authenticate will time out at 1468.310879s 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e31028700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7e2c09b740 0x7f7e2c134670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:55818/0 (socket says 172.21.15.142:55818) 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e31028700 1 -- 172.21.15.142:0/4043713088 learned_addr learned my addr 172.21.15.142:0/4043713088 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 1 --2- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): get_auth_request con 0x7f7e2c135d90 auth_method 0 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient(hunting): handle_auth_done global_id 75053 payload 274 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 1 -- 172.21.15.142:0/4043713088 >> v1:172.21.15.150:6789/0 conn(0x7f7e2c09abd0 legacy=0x7f7e2c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 1 -- 172.21.15.142:0/4043713088 >> v1:172.21.15.142:6789/0 conn(0x7f7e2c09b740 legacy=0x7f7e2c134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 1 monclient: found mon.noname-b 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.379+0000 7f7e3202a700 1 -- 172.21.15.142:0/4043713088 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c06f810 con 0x7f7e2c135d90 2024-09-18T10:19:39.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3202a700 1 -- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 msgr2=0x7f7e2c138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3202a700 1 --2- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c138170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009910 tx=0x7f7e240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02a00886a6dc971 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 1 -- 172.21.15.142:0/4043713088 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2400ba70 con 0x7f7e2c135d90 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: dump: 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.397 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 1 -- 172.21.15.142:0/4043713088 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2400bbd0 con 0x7f7e2c135d90 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 5 monclient: authenticate success, global_id 75053 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 1 -- 172.21.15.142:0/4043713088 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24011420 con 0x7f7e2c135d90 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 4 set_mon_vals no callback set 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e30827700 10 monclient: dump: 2024-09-18T10:19:39.398 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 10 monclient: shutdown 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 -- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 msgr2=0x7f7e2c138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 --2- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c138170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009910 tx=0x7f7e240091a0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 -- 172.21.15.142:0/4043713088 shutdown_connections 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 --2- 172.21.15.142:0/4043713088 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c138170 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 -- 172.21.15.142:0/4043713088 >> 172.21.15.142:0/4043713088 conn(0x7f7e2c070ef0 msgr2=0x7f7e2c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 -- 172.21.15.142:0/4043713088 shutdown_connections 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.383+0000 7f7e3282b700 1 -- 172.21.15.142:0/4043713088 wait complete. 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient: monmap: 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.400 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 Processor -- start 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 -- start start 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient: init 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e2c09abd0 0x7f7e2c08ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 0x7f7e2c088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c09b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:39.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c0894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c09f4f0 con 0x7f7e2c09b740 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c08c260 con 0x7f7e2c09abd0 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c08c790 con 0x7f7e2c135d90 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3282b700 10 monclient(hunting): authenticate will time out at 1468.317941s 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 0x7f7e2c088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 0x7f7e2c088fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43760/0 (socket says 172.21.15.142:43760) 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 -- 172.21.15.142:0/2981636991 learned_addr learned my addr 172.21.15.142:0/2981636991 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): get_auth_request con 0x7f7e2c09b740 auth_method 0 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c0894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): get_auth_request con 0x7f7e2c135d90 auth_method 0 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e2c09abd0 0x7f7e2c08ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 10 monclient(hunting): get_auth_request con 0x7f7e2c09abd0 auth_method 0 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e3202a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient(hunting): handle_auth_done global_id 124146 payload 824 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 msgr2=0x7f7e2c0894f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c135d90 0x7f7e2c0894f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e2c09abd0 msgr2=0x7f7e2c08ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e2c09abd0 0x7f7e2c08ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 monclient: found mon.noname-a 2024-09-18T10:19:39.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31028700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e24009420 con 0x7f7e2c09b740 2024-09-18T10:19:39.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.387+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e2c09abd0 0x7f7e2c08ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:19:39.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31028700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 msgr2=0x7f7e2c088fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31028700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 0x7f7e2c088fb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00cca0 tx=0x7f7e1c005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00abd0 con 0x7f7e2c09b740 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: dump: 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 5 monclient: authenticate success, global_id 124146 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c00ad50 con 0x7f7e2c09b740 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c007340 con 0x7f7e2c09b740 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 monclient: _renew_subs 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c08b270 con 0x7f7e2c09b740 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e2bfff700 4 set_mon_vals no callback set 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:39.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 10 monclient: dump: 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.411 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 monclient: _renew_subs 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c08ce90 con 0x7f7e2c09b740 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 0x7f7e2c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0648a0 con 0x7f7e2c089a30 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 0x7f7e2c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient: get_auth_request con 0x7f7e2c089a30 auth_method 0 2024-09-18T10:19:39.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 10 monclient(hunting): handle_auth_done global_id 75059 payload 824 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e31829700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e2c09d2d0 con 0x7f7e2c089a30 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.391+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7e1c007730 con 0x7f7e2c09b740 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e18035ac0 0x7f7e18037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7e1c049520 con 0x7f7e2c09b740 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e3202a700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e18035ac0 0x7f7e18037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e3202a700 10 monclient: get_auth_request con 0x7f7e18035ac0 auth_method 0 2024-09-18T10:19:39.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e31829700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 msgr2=0x7f7e2c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 0x7f7e2c06c370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000e710 tx=0x7f7e20009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20013860 con 0x7f7e2c089a30 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 10 monclient: dump: 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.414 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7e200139c0 con 0x7f7e2c089a30 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 msgr2=0x7f7e2c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 0x7f7e2c06c370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000e710 tx=0x7f7e20009040 comp rx=0 tx=0).stop 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 msgr2=0x7f7e2c06c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e29ffb700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c089a30 0x7f7e2c06c370 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.395+0000 7f7e3202a700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e18035ac0 0x7f7e18037f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7e240030c0 tx=0x7f7e24016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e3282b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e3282b700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 0x7f7e2c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e3282b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c089e00 con 0x7f7e2c06c8b0 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 0x7f7e2c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient: get_auth_request con 0x7f7e2c06c8b0 auth_method 0 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 10 monclient(hunting): handle_auth_done global_id 75062 payload 824 2024-09-18T10:19:39.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.407+0000 7f7e31829700 1 -- 172.21.15.142:0/2981636991 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7e2c072a80 con 0x7f7e2c06c8b0 2024-09-18T10:19:39.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e31829700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 msgr2=0x7f7e2c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e31829700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 0x7f7e2c06ccb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e20018000 tx=0x7f7e20019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20021840 con 0x7f7e2c06c8b0 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 10 monclient: dump: 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.419 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.421 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f7e200135f0 con 0x7f7e2c06c8b0 2024-09-18T10:19:39.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:39.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:39.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 msgr2=0x7f7e2c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 0x7f7e2c06ccb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e20018000 tx=0x7f7e20019000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 msgr2=0x7f7e2c06ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e29ffb700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e2c06c8b0 0x7f7e2c06ccb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:39.422 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.423 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.424 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.425 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.426 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.427 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.428 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:39.430 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.431 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.432 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.433 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:19:39.434 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.435 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:39.436 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.436 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.436 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:39.436 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e18035ac0 msgr2=0x7f7e18037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7e18035ac0 0x7f7e18037f70 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7e240030c0 tx=0x7f7e24016000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 10 monclient: shutdown 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 msgr2=0x7f7e2c088fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 --2- 172.21.15.142:0/2981636991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e2c09b740 0x7f7e2c088fb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00cca0 tx=0x7f7e1c005480 comp rx=0 tx=0).stop 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3202a700 1 -- 172.21.15.142:0/2981636991 reap_dead start 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3202a700 1 -- 172.21.15.142:0/2981636991 reap_dead start 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 shutdown_connections 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 >> 172.21.15.142:0/2981636991 conn(0x7f7e2c070ef0 msgr2=0x7f7e2c09de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 shutdown_connections 2024-09-18T10:19:39.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.411+0000 7f7e3282b700 1 -- 172.21.15.142:0/2981636991 wait complete. 2024-09-18T10:19:39.438 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:39.438 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:19:39.439 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:19:39.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: get_monmap_and_config 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: monmap: 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:39.549149+0000 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:39.549149+0000 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:39.551 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:39.552 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:39.552 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:39.552 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 Processor -- start 2024-09-18T10:19:39.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 -- start start 2024-09-18T10:19:39.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: init 2024-09-18T10:19:39.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb48c09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb48c09b740 0x7fb48c134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): picked mon.noname-c con 0x7fb48c09b740 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): picked mon.noname-b con 0x7fb48c134bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c09dc90 con 0x7fb48c09abd0 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48c09b500 con 0x7fb48c09b740 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c0967f0 con 0x7fb48c134bb0 2024-09-18T10:19:39.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb490938700 10 monclient(hunting): authenticate will time out at 1468.474669s 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35560/0 (socket says 172.21.15.142:35560) 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 -- 172.21.15.142:0/911192578 learned_addr learned my addr 172.21.15.142:0/911192578 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): get_auth_request con 0x7fb48c134bb0 auth_method 0 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 1 --2- 172.21.15.142:0/911192578 >> v2:172.21.15.150:3300/0 conn(0x7fb48c09b740 0x7fb48c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): get_auth_request con 0x7fb48c09b740 auth_method 0 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient(hunting): handle_auth_done global_id 75065 payload 274 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 -- 172.21.15.142:0/911192578 >> v2:172.21.15.150:3300/0 conn(0x7fb48c09b740 msgr2=0x7fb48c134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 --2- 172.21.15.142:0/911192578 >> v2:172.21.15.150:3300/0 conn(0x7fb48c09b740 0x7fb48c134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 -- 172.21.15.142:0/911192578 >> v1:172.21.15.150:6789/0 conn(0x7fb48c09abd0 legacy=0x7fb48c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 monclient: found mon.noname-b 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.543+0000 7fb48bfff700 1 -- 172.21.15.142:0/911192578 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c09afa0 con 0x7fb48c134bb0 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48bfff700 1 -- 172.21.15.142:0/911192578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 msgr2=0x7fb48c136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48bfff700 1 --2- 172.21.15.142:0/911192578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=123f55afaf27e799 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 1 -- 172.21.15.142:0/911192578 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484009ea0 con 0x7fb48c134bb0 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: dump: 2024-09-18T10:19:39.562 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 5 monclient: authenticate success, global_id 75065 2024-09-18T10:19:39.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 1 -- 172.21.15.142:0/911192578 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48400ba70 con 0x7fb48c134bb0 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 4 set_mon_vals no callback set 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 1 -- 172.21.15.142:0/911192578 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400be10 con 0x7fb48c134bb0 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb48a7fc700 10 monclient: dump: 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.564 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 10 monclient: shutdown 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 -- 172.21.15.142:0/911192578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 msgr2=0x7fb48c136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 --2- 172.21.15.142:0/911192578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484009250 comp rx=0 tx=0).stop 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 -- 172.21.15.142:0/911192578 shutdown_connections 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 --2- 172.21.15.142:0/911192578 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c134bb0 0x7fb48c136f90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 --2- 172.21.15.142:0/911192578 >> v2:172.21.15.150:3300/0 conn(0x7fb48c09b740 0x7fb48c134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 -- 172.21.15.142:0/911192578 >> 172.21.15.142:0/911192578 conn(0x7fb48c070ef0 msgr2=0x7fb48c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 -- 172.21.15.142:0/911192578 shutdown_connections 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 1 -- 172.21.15.142:0/911192578 wait complete. 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.547+0000 7fb490938700 10 monclient: monmap: 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:39.566 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 Processor -- start 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 -- start start 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient: init 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 0x7fb48c08b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): picked mon.noname-c con 0x7fb48c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb48c09b740 0x7fb48c088fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): picked mon.noname-a con 0x7fb48c09b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb48c134bb0 0x7fb48c0894f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): picked mon.noname-b con 0x7fb48c134bb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:39.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c09f510 con 0x7fb48c09b740 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c08c380 con 0x7fb48c134bb0 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c09d1c0 con 0x7fb48c09abd0 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb490938700 10 monclient(hunting): authenticate will time out at 1468.480811s 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 0x7fb48c08b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb48c09b740 0x7fb48c088fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 0x7fb48c08b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35566/0 (socket says 172.21.15.142:35566) 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 -- 172.21.15.142:0/1939045537 learned_addr learned my addr 172.21.15.142:0/1939045537 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): get_auth_request con 0x7fb48c09abd0 auth_method 0 2024-09-18T10:19:39.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): get_auth_request con 0x7fb48c09b740 auth_method 0 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb48c134bb0 0x7fb48c0894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 10 monclient(hunting): get_auth_request con 0x7fb48c134bb0 auth_method 0 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient(hunting): handle_auth_done global_id 75068 payload 824 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb48c134bb0 msgr2=0x7fb48c0894f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb48c134bb0 0x7fb48c0894f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb48c09b740 msgr2=0x7fb48c088fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb48c09b740 0x7fb48c088fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 monclient: found mon.noname-c 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4840094f0 con 0x7fb48c09abd0 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48affd700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb48c09b740 0x7fb48c088fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48bfff700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb48c134bb0 0x7fb48c0894f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.551+0000 7fb48b7fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 msgr2=0x7fb48c08b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48b7fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 0x7fb48c08b910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c5d0 tx=0x7fb480007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4800117a0 con 0x7fb48c09abd0 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: dump: 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.571 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 5 monclient: authenticate success, global_id 75068 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 monclient: _renew_subs 2024-09-18T10:19:39.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c08af90 con 0x7fb48c09abd0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 monclient: _renew_subs 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c0945d0 con 0x7fb48c09abd0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480011900 con 0x7fb48c09abd0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480011cc0 con 0x7fb48c09abd0 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4897fa700 4 set_mon_vals no callback set 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 10 monclient: dump: 2024-09-18T10:19:39.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 0x7fb46c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c007cd0 con 0x7fb46c002860 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb480012630 con 0x7fb48c09abd0 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 0x7fb46c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient: get_auth_request con 0x7fb46c002860 auth_method 0 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb478035ac0 0x7fb478037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb480048cc0 con 0x7fb48c09abd0 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48bfff700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb478035ac0 0x7fb478037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48bfff700 10 monclient: get_auth_request con 0x7fb478035ac0 auth_method 0 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 10 monclient(hunting): handle_auth_done global_id 114271 payload 824 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.555+0000 7fb48affd700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb46c007e90 con 0x7fb46c002860 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb48affd700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 msgr2=0x7fb46c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb48affd700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 0x7fb46c004cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0076f0 tx=0x7fb47c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb48bfff700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb478035ac0 0x7fb478037f70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c017960 con 0x7fb46c002860 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 10 monclient: dump: 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.579 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fb47c00f040 con 0x7fb46c002860 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 msgr2=0x7fb46c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 0x7fb46c004cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb47c0076f0 tx=0x7fb47c00b040 comp rx=0 tx=0).stop 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 msgr2=0x7fb46c004cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.559+0000 7fb4737fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c002860 0x7fb46c004cf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb490938700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb490938700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 0x7fb46c007140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb490938700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0077a0 con 0x7fb46c006cc0 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 0x7fb46c007140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient: get_auth_request con 0x7fb46c006cc0 auth_method 0 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 10 monclient(hunting): handle_auth_done global_id 114274 payload 824 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.571+0000 7fb48affd700 1 -- 172.21.15.142:0/1939045537 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb46c007960 con 0x7fb46c006cc0 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb48affd700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 msgr2=0x7fb46c007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb48affd700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 0x7fb46c007140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb47c01c000 tx=0x7fb47c01d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47c015c40 con 0x7fb46c006cc0 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 10 monclient: dump: 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.583 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fb47c017750 con 0x7fb46c006cc0 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 msgr2=0x7fb46c007140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 0x7fb46c007140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb47c01c000 tx=0x7fb47c01d000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 110, 2024-09-18T10:19:39.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.586 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.587 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:39.588 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:39.589 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:39.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:39.592 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.596 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:39.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:39.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:39.599 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:39.599 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 msgr2=0x7fb46c007140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb4737fe700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb46c006cc0 0x7fb46c007140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb478035ac0 msgr2=0x7fb478037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb478035ac0 0x7fb478037f70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484016000 comp rx=0 tx=0).stop 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 10 monclient: shutdown 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 msgr2=0x7fb48c08b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 --2- 172.21.15.142:0/1939045537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb48c09abd0 0x7fb48c08b910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb48000c5d0 tx=0x7fb480007f60 comp rx=0 tx=0).stop 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb48bfff700 1 -- 172.21.15.142:0/1939045537 reap_dead start 2024-09-18T10:19:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb48bfff700 1 -- 172.21.15.142:0/1939045537 reap_dead start 2024-09-18T10:19:39.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 shutdown_connections 2024-09-18T10:19:39.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 >> 172.21.15.142:0/1939045537 conn(0x7fb48c070ef0 msgr2=0x7fb48c09de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 shutdown_connections 2024-09-18T10:19:39.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.575+0000 7fb490938700 1 -- 172.21.15.142:0/1939045537 wait complete. 2024-09-18T10:19:39.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:19:39.602 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:19:39.602 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:19:39.602 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:19:39.602 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:19:39.603 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:19:39.603 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:19:39.622 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:19:39.622 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:19:39.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 10 monclient: get_monmap_and_config 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 10 monclient: monmap: 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:39.730176+0000 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:39.730176+0000 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.732 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:39.733 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 1 Processor -- start 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 1 -- start start 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.723+0000 7f84095c9700 10 monclient: init 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 0x7f840409a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-a con 0x7f840409bd80 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f840409a740 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 0x7f8404139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-b con 0x7f84041376a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8404068ff0 con 0x7f840409a740 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8404099b00 con 0x7f840409bd80 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840409e5f0 con 0x7f84041376a0 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84095c9700 10 monclient(hunting): authenticate will time out at 1468.656623s 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84027fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f840409a740 legacy=0x7f840409c150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 0x7f840409a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 0x7f8404139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 0x7f840409a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43770/0 (socket says 172.21.15.142:43770) 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 1 -- 172.21.15.142:0/3458263981 learned_addr learned my addr 172.21.15.142:0/3458263981 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): get_auth_request con 0x7f840409bd80 auth_method 0 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): get_auth_request con 0x7f84041376a0 auth_method 0 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f8402ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient(hunting): handle_auth_done global_id 75071 payload 274 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 -- 172.21.15.142:0/3458263981 >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 msgr2=0x7f840409a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 --2- 172.21.15.142:0/3458263981 >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 0x7f840409a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 -- 172.21.15.142:0/3458263981 >> v1:172.21.15.150:6789/0 conn(0x7f840409a740 legacy=0x7f840409c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 monclient: found mon.noname-b 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 -- 172.21.15.142:0/3458263981 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840409ea80 con 0x7f84041376a0 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 -- 172.21.15.142:0/3458263981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 msgr2=0x7f8404139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.727+0000 7f84037fe700 1 --2- 172.21.15.142:0/3458263981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 0x7f8404139a80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009e60 tx=0x7f83fc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e59e90a883fc808 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 1 -- 172.21.15.142:0/3458263981 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00e070 con 0x7f84041376a0 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: dump: 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.744 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 5 monclient: authenticate success, global_id 75071 2024-09-18T10:19:39.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 1 -- 172.21.15.142:0/3458263981 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc00bd70 con 0x7f84041376a0 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 4 set_mon_vals no callback set 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 1 -- 172.21.15.142:0/3458263981 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc012680 con 0x7f84041376a0 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f8401ffb700 10 monclient: dump: 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.746 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 10 monclient: shutdown 2024-09-18T10:19:39.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- 172.21.15.142:0/3458263981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 msgr2=0x7f8404139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 --2- 172.21.15.142:0/3458263981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 0x7f8404139a80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009e60 tx=0x7f83fc009780 comp rx=0 tx=0).stop 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- 172.21.15.142:0/3458263981 shutdown_connections 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 --2- 172.21.15.142:0/3458263981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84041376a0 0x7f8404139a80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 --2- 172.21.15.142:0/3458263981 >> v2:172.21.15.142:3300/0 conn(0x7f840409bd80 0x7f840409a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- 172.21.15.142:0/3458263981 >> 172.21.15.142:0/3458263981 conn(0x7f84040a24f0 msgr2=0x7f84040a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- 172.21.15.142:0/3458263981 shutdown_connections 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- 172.21.15.142:0/3458263981 wait complete. 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 10 monclient: build_initial_monmap 2024-09-18T10:19:39.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 10 monclient: monmap: 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:39.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 Processor -- start 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 1 -- start start 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.731+0000 7f84095c9700 10 monclient: init 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:19:39.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 0x7f840406f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-b con 0x7f840409a740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-c con 0x7f840409bd80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 0x7f8404134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): picked mon.noname-a con 0x7f84041376a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8404098930 con 0x7f84041376a0 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840409fc00 con 0x7f840409a740 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8404134f00 con 0x7f840409bd80 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:39.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84095c9700 10 monclient(hunting): authenticate will time out at 1468.664845s 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f8402ffd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 msgr2=0x7f840406f790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f8402ffd700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 0x7f840406f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 0x7f8404134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35588/0 (socket says 172.21.15.142:35588) 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 -- 172.21.15.142:0/568875793 learned_addr learned my addr 172.21.15.142:0/568875793 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): get_auth_request con 0x7f840409bd80 auth_method 0 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): get_auth_request con 0x7f84041376a0 auth_method 0 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:39.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient(hunting): handle_auth_done global_id 75074 payload 824 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 -- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 msgr2=0x7f840406f790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 0x7f840406f790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 -- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 msgr2=0x7f8404134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 0x7f8404134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient: _finish_hunting 0 2024-09-18T10:19:39.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 monclient: found mon.noname-c 2024-09-18T10:19:39.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:39.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84027fc700 1 -- 172.21.15.142:0/568875793 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc009a20 con 0x7f840409bd80 2024-09-18T10:19:39.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.735+0000 7f84037fe700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 0x7f8404134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:39.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84027fc700 1 -- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 msgr2=0x7f8404133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:39.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84027fc700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b7e0 tx=0x7f83f4009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4003e20 con 0x7f840409bd80 2024-09-18T10:19:39.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: dump: 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.757 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: _finish_auth 0 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f4006150 con 0x7f840409bd80 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 5 monclient: authenticate success, global_id 75074 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _renew_subs 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8404098f90 con 0x7f840409bd80 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f4006510 con 0x7f840409bd80 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 10 monclient: dump: 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:39.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84017fa700 4 set_mon_vals no callback set 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _renew_subs 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8404136f30 con 0x7f840409bd80 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f83f40103c0 con 0x7f840409bd80 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8404136790 con 0x7f840409bd80 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 0x7f83e4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:39.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f83f40487e0 con 0x7f840409bd80 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f8402ffd700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 0x7f83e4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.739+0000 7f8402ffd700 10 monclient: get_auth_request con 0x7f83e4035a70 auth_method 0 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.743+0000 7f8402ffd700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 0x7f83e4037f20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f83ec006fd0 tx=0x7f83ec009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.747+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f83f400ed30 con 0x7f840409bd80 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.747+0000 7f83faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:39.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.747+0000 7f83faffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:39.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.971+0000 7f84095c9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:39.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.971+0000 7f84095c9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:39.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.971+0000 7f84095c9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:39.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.971+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f84040696a0 con 0x7f840409bd80 2024-09-18T10:19:39.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.971+0000 7f83faffd700 1 -- 172.21.15.142:0/568875793 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f83f4017030 con 0x7f840409bd80 2024-09-18T10:19:39.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.975+0000 7f83faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:39.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.975+0000 7f83faffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:39.982 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 msgr2=0x7f83e4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 0x7f83e4037f20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f83ec006fd0 tx=0x7f83ec009250 comp rx=0 tx=0).stop 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 10 monclient: shutdown 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 msgr2=0x7f8404133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83f400b7e0 tx=0x7f83f4009220 comp rx=0 tx=0).stop 2024-09-18T10:19:39.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 shutdown_connections 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f83e4035a70 0x7f83e4037f20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84041376a0 0x7f8404134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f840409bd80 0x7f8404133cf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 --2- 172.21.15.142:0/568875793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f840409a740 0x7f840406f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 >> 172.21.15.142:0/568875793 conn(0x7f84040a24f0 msgr2=0x7f84041329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 shutdown_connections 2024-09-18T10:19:39.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:39.979+0000 7f84095c9700 1 -- 172.21.15.142:0/568875793 wait complete. 2024-09-18T10:19:43.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:43.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: get_monmap_and_config 2024-09-18T10:19:43.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: build_initial_monmap 2024-09-18T10:19:43.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: monmap: 2024-09-18T10:19:43.115 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:43.115 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:43.113675+0000 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:43.113675+0000 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:43.116 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 1 Processor -- start 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 1 -- start start 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: init 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 0x7f3ac809af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-c con 0x7f3ac80953c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ac8097970 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-b con 0x7f3ac809cdf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.107+0000 7f3ac759e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac809a570 con 0x7f3ac8097970 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:43.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac809b490 con 0x7f3ac80953c0 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8095790 con 0x7f3ac809cdf0 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac759e700 10 monclient(hunting): authenticate will time out at 1472.040027s 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac659c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 msgr2=0x7f3ac809af50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac5d9b700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f3ac8097970 legacy=0x7f3ac8097d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac659c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 0x7f3ac809af50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:56986/0 (socket says 172.21.15.142:56986) 2024-09-18T10:19:43.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 -- 172.21.15.142:0/871576781 learned_addr learned my addr 172.21.15.142:0/871576781 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): get_auth_request con 0x7f3ac809cdf0 auth_method 0 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient(hunting): handle_auth_done global_id 75077 payload 274 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 -- 172.21.15.142:0/871576781 >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 msgr2=0x7f3ac809af50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 --2- 172.21.15.142:0/871576781 >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 0x7f3ac809af50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 -- 172.21.15.142:0/871576781 >> v1:172.21.15.150:6789/0 conn(0x7f3ac8097970 legacy=0x7f3ac8097d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:43.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:19:43.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 monclient: found mon.noname-b 2024-09-18T10:19:43.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:43.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:43.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 -- 172.21.15.142:0/871576781 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac806f900 con 0x7f3ac809cdf0 2024-09-18T10:19:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 -- 172.21.15.142:0/871576781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 msgr2=0x7f3ac81368a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.111+0000 7f3ac6d9d700 1 --2- 172.21.15.142:0/871576781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009f20 tx=0x7f3ac0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f699d5b180d568e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:43.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 1 -- 172.21.15.142:0/871576781 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000e070 con 0x7f3ac809cdf0 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: dump: 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:43.126 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: _finish_auth 0 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 5 monclient: authenticate success, global_id 75077 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 1 -- 172.21.15.142:0/871576781 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac000bc90 con 0x7f3ac809cdf0 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 4 set_mon_vals no callback set 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 1 -- 172.21.15.142:0/871576781 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0012660 con 0x7f3ac809cdf0 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac559a700 10 monclient: dump: 2024-09-18T10:19:43.127 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: shutdown 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- 172.21.15.142:0/871576781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 msgr2=0x7f3ac81368a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:43.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- 172.21.15.142:0/871576781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009f20 tx=0x7f3ac0009840 comp rx=0 tx=0).stop 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- 172.21.15.142:0/871576781 shutdown_connections 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- 172.21.15.142:0/871576781 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac809cdf0 0x7f3ac81368a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- 172.21.15.142:0/871576781 >> v2:172.21.15.150:3300/0 conn(0x7f3ac80953c0 0x7f3ac809af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- 172.21.15.142:0/871576781 >> 172.21.15.142:0/871576781 conn(0x7f3ac80a1fe0 msgr2=0x7f3ac8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- 172.21.15.142:0/871576781 shutdown_connections 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- 172.21.15.142:0/871576781 wait complete. 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: build_initial_monmap 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: monmap: 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:43.129 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 Processor -- start 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- start start 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: init 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-c con 0x7f3ac80953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 0x7f3ac8141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-a con 0x7f3ac8097970 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 0x7f3ac806a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): picked mon.noname-b con 0x7f3ac809cdf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80916d0 con 0x7f3ac8097970 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8090570 con 0x7f3ac809cdf0 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80913f0 con 0x7f3ac80953c0 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac759e700 10 monclient(hunting): authenticate will time out at 1472.046492s 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:43.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 0x7f3ac8141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac6d9d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 msgr2=0x7f3ac806a320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac6d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 0x7f3ac806a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57000/0 (socket says 172.21.15.142:57000) 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): get_auth_request con 0x7f3ac8097970 auth_method 0 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 1 -- 172.21.15.142:0/3441011251 learned_addr learned my addr 172.21.15.142:0/3441011251 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): get_auth_request con 0x7f3ac80953c0 auth_method 0 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:43.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient(hunting): handle_auth_done global_id 124158 payload 824 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 -- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 msgr2=0x7f3ac8141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 -- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 msgr2=0x7f3ac806a320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 0x7f3ac806a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 monclient: found mon.noname-a 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:43.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:43.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac5d9b700 1 -- 172.21.15.142:0/3441011251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009ae0 con 0x7f3ac8097970 2024-09-18T10:19:43.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.115+0000 7f3ac659c700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac5d9b700 1 -- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 msgr2=0x7f3ac8141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac5d9b700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 0x7f3ac8141d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00b590 tx=0x7f3abc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc003dd0 con 0x7f3ac8097970 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: dump: 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.136 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc006150 con 0x7f3ac8097970 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 5 monclient: authenticate success, global_id 124158 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _renew_subs 2024-09-18T10:19:43.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc006510 con 0x7f3ac8097970 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac4d99700 4 set_mon_vals no callback set 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac80919b0 con 0x7f3ac8097970 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ab67fc700 10 monclient: dump: 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:43.138 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _renew_subs 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac8094f70 con 0x7f3ac8097970 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.119+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac8091b40 con 0x7f3ac8097970 2024-09-18T10:19:43.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3abc01b3b0 con 0x7f3ac8097970 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ab67fc700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 0x7f3aa8037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3abc048590 con 0x7f3ac8097970 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ac659c700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 0x7f3aa8037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ac659c700 10 monclient: get_auth_request con 0x7f3aa8035aa0 auth_method 0 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.123+0000 7f3ac659c700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 0x7f3aa8037f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8003fd0 tx=0x7f3ab8009220 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:43.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.135+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3abc019870 con 0x7f3ac8097970 2024-09-18T10:19:43.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.135+0000 7f3ab67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:43.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.135+0000 7f3ab67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:43.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ac759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:43.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ac759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:43.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ac759e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:43.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3ac8072a80 con 0x7f3ac8097970 2024-09-18T10:19:43.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ab67fc700 1 -- 172.21.15.142:0/3441011251 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f3abc0192c0 con 0x7f3ac8097970 2024-09-18T10:19:43.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ab67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:43.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.347+0000 7f3ab67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:43.357 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:43.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 msgr2=0x7f3aa8037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:43.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 0x7f3aa8037f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8003fd0 tx=0x7f3ab8009220 comp rx=0 tx=0).stop 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 10 monclient: shutdown 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 msgr2=0x7f3ac8141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 0x7f3ac8141d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00b590 tx=0x7f3abc009120 comp rx=0 tx=0).stop 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 shutdown_connections 2024-09-18T10:19:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3aa8035aa0 0x7f3aa8037f50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ac809cdf0 0x7f3ac806a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ac8097970 0x7f3ac8141d50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 --2- 172.21.15.142:0/3441011251 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ac80953c0 0x7f3ac8141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 >> 172.21.15.142:0/3441011251 conn(0x7f3ac80a1fe0 msgr2=0x7f3ac806e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 shutdown_connections 2024-09-18T10:19:43.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:43.351+0000 7f3ac759e700 1 -- 172.21.15.142:0/3441011251 wait complete. 2024-09-18T10:19:46.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:46.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: get_monmap_and_config 2024-09-18T10:19:46.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: build_initial_monmap 2024-09-18T10:19:46.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: monmap: 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:46.480737+0000 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:46.480737+0000 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:46.483 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 Processor -- start 2024-09-18T10:19:46.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 -- start start 2024-09-18T10:19:46.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: init 2024-09-18T10:19:46.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb4b40a5f90 0x7fb4b40a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-a con 0x7fb4b40a5f90 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4b40990b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 0x7fb4b4139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b409e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b409d980 con 0x7fb4b40990b0 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b409d780 con 0x7fb4b40a5f90 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:46.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b409fc70 con 0x7fb4b409e360 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4bbb7d700 10 monclient(hunting): authenticate will time out at 1475.406818s 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb4b40a5f90 0x7fb4b40a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9118700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb4b40990b0 0x7fb4b40a07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:32794/0 (socket says 172.21.15.142:32794) 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9118700 1 -- 172.21.15.142:0/2064160081 learned_addr learned my addr 172.21.15.142:0/2064160081 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): get_auth_request con 0x7fb4b40a5f90 auth_method 0 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 0x7fb4b4139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:46.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 10 monclient(hunting): get_auth_request con 0x7fb4b409e360 auth_method 0 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient(hunting): handle_auth_done global_id 124161 payload 274 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 -- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 msgr2=0x7fb4b4139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 0x7fb4b4139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 -- 172.21.15.142:0/2064160081 >> v1:172.21.15.142:6789/0 conn(0x7fb4b40990b0 legacy=0x7fb4b40a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient: _finish_hunting 0 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 monclient: found mon.noname-a 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4b9919700 1 -- 172.21.15.142:0/2064160081 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40a6360 con 0x7fb4b40a5f90 2024-09-18T10:19:46.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.479+0000 7fb4ba11a700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 0x7fb4b4139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b9919700 1 -- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b40a5f90 msgr2=0x7fb4b40a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b9919700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b40a5f90 0x7fb4b40a0270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4099720 tx=0x7fb4b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723150bcbe93cb03 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 1 -- 172.21.15.142:0/2064160081 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b000e070 con 0x7fb4b40a5f90 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: dump: 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:46.499 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: _finish_auth 0 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 1 -- 172.21.15.142:0/2064160081 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b000be40 con 0x7fb4b40a5f90 2024-09-18T10:19:46.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 5 monclient: authenticate success, global_id 124161 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 4 set_mon_vals no callback set 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 1 -- 172.21.15.142:0/2064160081 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0012660 con 0x7fb4b40a5f90 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4b8917700 10 monclient: dump: 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 10 monclient: shutdown 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b40a5f90 msgr2=0x7fb4b40a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b40a5f90 0x7fb4b40a0270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4099720 tx=0x7fb4b0009780 comp rx=0 tx=0).stop 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2064160081 shutdown_connections 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b409e360 0x7fb4b4139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2064160081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b40a5f90 0x7fb4b40a0270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2064160081 >> 172.21.15.142:0/2064160081 conn(0x7fb4b4071680 msgr2=0x7fb4b40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2064160081 shutdown_connections 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2064160081 wait complete. 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 10 monclient: build_initial_monmap 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 10 monclient: monmap: 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:46.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.483+0000 7fb4bbb7d700 1 Processor -- start 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 -- start start 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient: init 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 0x7fb4b4141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b40990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b409e360 0x7fb4b41421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-a con 0x7fb4b409e360 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): picked mon.noname-c con 0x7fb4b40a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:46.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b408f390 con 0x7fb4b409e360 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b409f810 con 0x7fb4b40990b0 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40a4490 con 0x7fb4b40a5f90 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 10 monclient(hunting): authenticate will time out at 1475.412582s 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9919700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 msgr2=0x7fb4b4141c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9919700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 0x7fb4b4141c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57020/0 (socket says 172.21.15.142:57020) 2024-09-18T10:19:46.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 -- 172.21.15.142:0/2861624488 learned_addr learned my addr 172.21.15.142:0/2861624488 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): get_auth_request con 0x7fb4b40a5f90 auth_method 0 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b409e360 0x7fb4b41421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): get_auth_request con 0x7fb4b409e360 auth_method 0 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:46.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4b9118700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient(hunting): handle_auth_done global_id 75086 payload 824 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 -- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 msgr2=0x7fb4b4141c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 0x7fb4b4141c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 -- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b409e360 msgr2=0x7fb4b41421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b409e360 0x7fb4b41421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient: _finish_hunting 0 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 monclient: found mon.noname-c 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 -- 172.21.15.142:0/2861624488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0009a20 con 0x7fb4b40a5f90 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 -- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 msgr2=0x7fb4b408d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4ba11a700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00e870 tx=0x7fb4ac008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:46.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac013620 con 0x7fb4b40a5f90 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 10 monclient: dump: 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:46.508 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac013780 con 0x7fb4b40a5f90 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.487+0000 7fb4bbb7d700 5 monclient: authenticate success, global_id 75086 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _renew_subs 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac013b20 con 0x7fb4b40a5f90 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b4090940 con 0x7fb4b40a5f90 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4abfff700 4 set_mon_vals no callback set 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 10 monclient: dump: 2024-09-18T10:19:46.509 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _renew_subs 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b408f9e0 con 0x7fb4b40a5f90 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:46.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b40696a0 con 0x7fb4b40a5f90 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb4ac00c380 con 0x7fb4b40a5f90 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 0x7fb498037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb4ac048c40 con 0x7fb4b40a5f90 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4b9919700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 0x7fb498037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.491+0000 7fb4b9919700 10 monclient: get_auth_request con 0x7fb498035aa0 auth_method 0 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.495+0000 7fb4b9919700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 0x7fb498037f50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009fd0 tx=0x7fb4b0019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:46.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.499+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb4ac012370 con 0x7fb4b40a5f90 2024-09-18T10:19:46.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.499+0000 7fb4a9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:46.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.499+0000 7fb4a9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4bbb7d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4bbb7d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4bbb7d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb4b40a1060 con 0x7fb4b40a5f90 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4a9ffb700 1 -- 172.21.15.142:0/2861624488 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fb4ac022300 con 0x7fb4b40a5f90 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4a9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:46.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.723+0000 7fb4a9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:46.729 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:46.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 msgr2=0x7fb498037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 0x7fb498037f50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009fd0 tx=0x7fb4b0019000 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 10 monclient: shutdown 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 msgr2=0x7fb4b408d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00e870 tx=0x7fb4ac008040 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 shutdown_connections 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb498035aa0 0x7fb498037f50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb4b40a5f90 0x7fb4b408d540 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb4b409e360 0x7fb4b41421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 --2- 172.21.15.142:0/2861624488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4b40990b0 0x7fb4b4141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:46.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 >> 172.21.15.142:0/2861624488 conn(0x7fb4b4071680 msgr2=0x7fb4b40a2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:46.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 shutdown_connections 2024-09-18T10:19:46.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:46.727+0000 7fb4bbb7d700 1 -- 172.21.15.142:0/2861624488 wait complete. 2024-09-18T10:19:49.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: get_monmap_and_config 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: build_initial_monmap 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: monmap: 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:49.852247+0000 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:49.852247+0000 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:49.855 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 Processor -- start 2024-09-18T10:19:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 -- start start 2024-09-18T10:19:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: init 2024-09-18T10:19:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8c809a7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-b con 0x7fa8c80993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-c con 0x7fa8c813bb30 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80997e0 con 0x7fa8c809a7f0 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c809abc0 con 0x7fa8c813bb30 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c809d950 con 0x7fa8c80993f0 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cef63700 10 monclient(hunting): authenticate will time out at 1478.777871s 2024-09-18T10:19:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cdf61700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fa8c809a7f0 legacy=0x7fa8c8098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8ce762700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 msgr2=0x7fa8c813df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8ce762700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57026/0 (socket says 172.21.15.142:57026) 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 learned_addr learned my addr 172.21.15.142:0/613873608 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8c80993f0 auth_method 0 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:49.851+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:50.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 msgr2=0x7fa8c813df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:50.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cdf61700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.150:6789/0 conn(0x7fa8c809a7f0 legacy=0x7fa8c8098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:50.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:50.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:50.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8c80993f0 auth_method 0 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:50.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cdf61700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.150:6789/0 conn(0x7fa8c809a7f0 legacy=0x7fa8c8098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 msgr2=0x7fa8c813df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8c80993f0 auth_method 0 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:19:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:50.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:19:51.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cdf61700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.150:6789/0 conn(0x7fa8c809a7f0 legacy=0x7fa8c8098ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:51.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 msgr2=0x7fa8c813df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8c80993f0 auth_method 0 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:51.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:51.263+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): tick 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 msgr2=0x7fa8c813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8c80993f0 0x7fa8c813b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 msgr2=0x7fa8c813df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.150:3300/0 conn(0x7fa8c813bb30 0x7fa8c813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.150:6789/0 conn(0x7fa8c809a7f0 legacy=0x7fa8c8098ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient: _add_conns ranks=[0,3] 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b40036f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:52.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): picked mon.noname-a con 0x7fa8b4007430 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 -- 172.21.15.142:0/613873608 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c809db90 con 0x7fa8b40036f0 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 1 -- 172.21.15.142:0/613873608 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c809abc0 con 0x7fa8b4007430 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8bf7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8b4007430 auth_method 0 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:52.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:52.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:52.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:52.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.851+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:52.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:52.855+0000 7fa8cdf61700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.142:6789/0 conn(0x7fa8b40036f0 legacy=0x7fa8b4005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:53.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:53.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8b4007430 auth_method 0 2024-09-18T10:19:53.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:53.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.055+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8b4007430 auth_method 0 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:53.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:19:53.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:19:53.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:19:53.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:53.459+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:19:54.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338575736 0 0) 0x7fa8c809db90 con 0x7fa8b40036f0 2024-09-18T10:19:54.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 10 monclient(hunting): my global_id is 124167 2024-09-18T10:19:54.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003570 con 0x7fa8b40036f0 2024-09-18T10:19:54.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457918741 0 0) 0x7fa8b0003570 con 0x7fa8b40036f0 2024-09-18T10:19:54.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 msgr2=0x7fa8b40098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:54.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 --2- 172.21.15.142:0/613873608 >> v2:172.21.15.142:3300/0 conn(0x7fa8b4007430 0x7fa8b40098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.207+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80997e0 con 0x7fa8b40036f0 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cef63700 5 monclient: authenticate success, global_id 124167 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cef63700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fa8b80043c0 con 0x7fa8b40036f0 2024-09-18T10:19:54.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: dump: 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.217 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 >> v1:172.21.15.142:6789/0 conn(0x7fa8b40036f0 legacy=0x7fa8b4005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 reap_dead start 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8b0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): picked mon.b con 0x7fa8b0003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): picked mon.a con 0x7fa8b0004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8b000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): picked mon.c con 0x7fa8b0005230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c809abc0 con 0x7fa8b0004810 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cdf61700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 msgr2=0x7fa8b0003e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c200 con 0x7fa8b0003a60 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cdf61700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8b0003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c3c0 con 0x7fa8b0005230 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ccf5f700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cef63700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:54.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8b000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8b0004810 auth_method 0 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): get_auth_request con 0x7fa8b0005230 auth_method 0 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient(hunting): handle_auth_done global_id 124167 payload 293 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 msgr2=0x7fa8b0003e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8b0003e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 msgr2=0x7fa8b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 monclient: found mon.c 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0003570 con 0x7fa8b0005230 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8ce762700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.211+0000 7fa8cd760700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.215+0000 7fa8ce762700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 msgr2=0x7fa8b000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.215+0000 7fa8ce762700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8b000ba60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c809a6f0 tx=0x7fa8c00094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e58a86bc3d06469 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00097e0 con 0x7fa8b0005230 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: dump: 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.224 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0003d10 con 0x7fa8b0005230 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 1 -- 172.21.15.142:0/613873608 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c000bbd0 con 0x7fa8b0005230 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 4 set_mon_vals no callback set 2024-09-18T10:19:54.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8ccf5f700 10 monclient: dump: 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 10 monclient: shutdown 2024-09-18T10:19:54.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 msgr2=0x7fa8b000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8b000ba60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa8c809a6f0 tx=0x7fa8c00094b0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- 172.21.15.142:0/613873608 shutdown_connections 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8b000ba60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 --2- 172.21.15.142:0/613873608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8b0003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- 172.21.15.142:0/613873608 >> 172.21.15.142:0/613873608 conn(0x7fa8c80a1fe0 msgr2=0x7fa8c8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- 172.21.15.142:0/613873608 shutdown_connections 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- 172.21.15.142:0/613873608 wait complete. 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 10 monclient: build_initial_monmap 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 10 monclient: monmap: 2024-09-18T10:19:54.227 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:54.228 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 Processor -- start 2024-09-18T10:19:54.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 1 -- start start 2024-09-18T10:19:54.232 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.219+0000 7fa8cef63700 10 monclient: init 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8c806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-b con 0x7fa8b0003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-a con 0x7fa8b0004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8c806c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): picked mon.noname-c con 0x7fa8b0005230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c806f970 con 0x7fa8b0004810 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80a00f0 con 0x7fa8b0003a60 2024-09-18T10:19:54.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c813b710 con 0x7fa8b0005230 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient(hunting): authenticate will time out at 1483.148262s 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cdf61700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 msgr2=0x7fa8c806bf10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cdf61700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8c806bf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8c806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38936/0 (socket says 172.21.15.142:38936) 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 1 -- 172.21.15.142:0/3274977505 learned_addr learned my addr 172.21.15.142:0/3274977505 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): get_auth_request con 0x7fa8b0004810 auth_method 0 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): get_auth_request con 0x7fa8b0005230 auth_method 0 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient(hunting): handle_auth_done global_id 75089 payload 824 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 -- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 msgr2=0x7fa8c806bf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:54.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8c806bf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 -- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 msgr2=0x7fa8c806c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 monclient: found mon.noname-c 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 -- 172.21.15.142:0/3274977505 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c806d9d0 con 0x7fa8b0005230 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cd760700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 -- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 msgr2=0x7fa8c806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8ce762700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8c806c990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c550 tx=0x7fa8c0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0006640 con 0x7fa8b0005230 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: dump: 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.237 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0006800 con 0x7fa8b0005230 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 5 monclient: authenticate success, global_id 75089 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient: _renew_subs 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8126710 con 0x7fa8b0005230 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0017450 con 0x7fa8b0005230 2024-09-18T10:19:54.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bdffb700 10 monclient: dump: 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8cef63700 10 monclient: _renew_subs 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c806e5a0 con 0x7fa8b0005230 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.223+0000 7fa8bffff700 4 set_mon_vals no callback set 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa8c000c040 con 0x7fa8b0005230 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c8072a80 con 0x7fa8b0005230 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8bdffb700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 0x7fa8b4038760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa8c0049ed0 con 0x7fa8b0005230 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cdf61700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 0x7fa8b4038760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.227+0000 7fa8cdf61700 10 monclient: get_auth_request con 0x7fa8b40362b0 auth_method 0 2024-09-18T10:19:54.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.231+0000 7fa8cdf61700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 0x7fa8b4038760 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8003a40 tx=0x7fa8b8003b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.235+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa8c002ab80 con 0x7fa8b0005230 2024-09-18T10:19:54.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.235+0000 7fa8bdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.235+0000 7fa8bdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:54.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8cef63700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:19:54.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8cef63700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:54.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8cef63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8c80981d0 con 0x7fa8b0005230 2024-09-18T10:19:54.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8bdffb700 1 -- 172.21.15.142:0/3274977505 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fa8c002a3a0 con 0x7fa8b0005230 2024-09-18T10:19:54.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8bdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:19:54.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.463+0000 7fa8bdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:54.468 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 msgr2=0x7fa8b4038760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 0x7fa8b4038760 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8003a40 tx=0x7fa8b8003b70 comp rx=0 tx=0).stop 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 10 monclient: shutdown 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 msgr2=0x7fa8c806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8c806c990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c550 tx=0x7fa8c0017000 comp rx=0 tx=0).stop 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 shutdown_connections 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa8b40362b0 0x7fa8b4038760 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b0005230 0x7fa8c806c990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b0004810 0x7fa8c806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 --2- 172.21.15.142:0/3274977505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8b0003a60 0x7fa8c806bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 >> 172.21.15.142:0/3274977505 conn(0x7fa8c80a1fe0 msgr2=0x7fa8c809bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 shutdown_connections 2024-09-18T10:19:54.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.467+0000 7fa8cef63700 1 -- 172.21.15.142:0/3274977505 wait complete. 2024-09-18T10:19:54.487 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:19:54.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:19:54.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: build_initial_monmap 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: monmap: 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:54.592184+0000 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:54.592184+0000 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.594 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:54.595 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:54.595 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 Processor -- start 2024-09-18T10:19:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 -- start start 2024-09-18T10:19:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: init 2024-09-18T10:19:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbc8c09d710 0x7fbc8c0988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fbc8c09d710 addr v2:172.21.15.142:3300/0 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc8c0953c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 0x7fbc8c136520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fbc8c095f30 addr v2:172.21.15.150:3300/0 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8c09dae0 con 0x7fbc8c0953c0 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c09b880 con 0x7fbc8c09d710 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c09c320 con 0x7fbc8c095f30 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8bfff700 10 monclient(hunting): authenticate will time out at 1483.517872s 2024-09-18T10:19:54.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbc8c09d710 0x7fbc8c0988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc827fc700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fbc8c0953c0 legacy=0x7fbc8c095820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8b7fe700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 msgr2=0x7fbc8c136520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbc8c09d710 0x7fbc8c0988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38950/0 (socket says 172.21.15.142:38950) 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 -- 172.21.15.142:0/1291892070 learned_addr learned my addr 172.21.15.142:0/1291892070 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): get_auth_request con 0x7fbc8c09d710 auth_method 0 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8b7fe700 1 --2- 172.21.15.142:0/1291892070 >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 0x7fbc8c136520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient(hunting): handle_auth_done global_id 124173 payload 274 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 -- 172.21.15.142:0/1291892070 >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 msgr2=0x7fbc8c136520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 --2- 172.21.15.142:0/1291892070 >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 0x7fbc8c136520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 -- 172.21.15.142:0/1291892070 >> v1:172.21.15.150:6789/0 conn(0x7fbc8c0953c0 legacy=0x7fbc8c095820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 monclient: found mon.noname-a 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.591+0000 7fbc8affd700 1 -- 172.21.15.142:0/1291892070 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c097700 con 0x7fbc8c09d710 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8affd700 1 -- 172.21.15.142:0/1291892070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c09d710 msgr2=0x7fbc8c0988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8affd700 1 --2- 172.21.15.142:0/1291892070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c09d710 0x7fbc8c0988f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c097630 tx=0x7fbc7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b1d98a31bb2ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 1 -- 172.21.15.142:0/1291892070 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00ba70 con 0x7fbc8c09d710 2024-09-18T10:19:54.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: dump: 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 5 monclient: authenticate success, global_id 124173 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 1 -- 172.21.15.142:0/1291892070 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c00bbd0 con 0x7fbc8c09d710 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 4 set_mon_vals no callback set 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 1 -- 172.21.15.142:0/1291892070 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c011420 con 0x7fbc8c09d710 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8a7fc700 10 monclient: dump: 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.605 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: shutdown 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- 172.21.15.142:0/1291892070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c09d710 msgr2=0x7fbc8c0988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/1291892070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c09d710 0x7fbc8c0988f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c097630 tx=0x7fbc7c009250 comp rx=0 tx=0).stop 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- 172.21.15.142:0/1291892070 shutdown_connections 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/1291892070 >> v2:172.21.15.150:3300/0 conn(0x7fbc8c095f30 0x7fbc8c136520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/1291892070 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c09d710 0x7fbc8c0988f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- 172.21.15.142:0/1291892070 >> 172.21.15.142:0/1291892070 conn(0x7fbc8c071ae0 msgr2=0x7fbc8c0a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- 172.21.15.142:0/1291892070 shutdown_connections 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- 172.21.15.142:0/1291892070 wait complete. 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: build_initial_monmap 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: monmap: 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:54.607 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:54.608 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 Processor -- start 2024-09-18T10:19:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- start start 2024-09-18T10:19:54.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: init 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc8c0953c0 0x7fbc8c141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fbc8c0953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 0x7fbc8c141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fbc8c095f30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8c09d710 0x7fbc8c08deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fbc8c09d710 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0913e0 con 0x7fbc8c095f30 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0919a0 con 0x7fbc8c09d710 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c092670 con 0x7fbc8c0953c0 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.595+0000 7fbc8bfff700 10 monclient(hunting): authenticate will time out at 1483.523924s 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8b7fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8c09d710 msgr2=0x7fbc8c08deb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc8c0953c0 0x7fbc8c141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 0x7fbc8c141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8b7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8c09d710 0x7fbc8c08deb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 0x7fbc8c141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38954/0 (socket says 172.21.15.142:38954) 2024-09-18T10:19:54.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 -- 172.21.15.142:0/3278654466 learned_addr learned my addr 172.21.15.142:0/3278654466 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): get_auth_request con 0x7fbc8c0953c0 auth_method 0 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): get_auth_request con 0x7fbc8c095f30 auth_method 0 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient(hunting): handle_auth_done global_id 124176 payload 824 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc8c0953c0 msgr2=0x7fbc8c141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc8c0953c0 0x7fbc8c141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8c09d710 msgr2=0x7fbc8c08deb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc8c09d710 0x7fbc8c08deb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 monclient: found mon.noname-a 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:19:54.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0094f0 con 0x7fbc8c095f30 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 msgr2=0x7fbc8c141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc827fc700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 0x7fbc8c141d70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400b7e0 tx=0x7fbc74009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74003e20 con 0x7fbc8c095f30 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.618 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 5 monclient: authenticate success, global_id 124176 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74006150 con 0x7fbc8c095f30 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 10 monclient: _renew_subs 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c090480 con 0x7fbc8c095f30 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74006510 con 0x7fbc8c095f30 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc89ffb700 4 set_mon_vals no callback set 2024-09-18T10:19:54.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 10 monclient: _renew_subs 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.599+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c1422b0 con 0x7fbc8c095f30 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 0x7fbc8c08e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c092830 con 0x7fbc8c08e3f0 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 0x7fbc8c08e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient: get_auth_request con 0x7fbc8c08e3f0 auth_method 0 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbc740103c0 con 0x7fbc8c095f30 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 10 monclient(hunting): handle_auth_done global_id 124179 payload 824 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc8c095100 con 0x7fbc8c08e3f0 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbc6c035a70 0x7fbc6c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fbc74048750 con 0x7fbc8c095f30 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8b7fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbc6c035a70 0x7fbc6c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8b7fe700 10 monclient: get_auth_request con 0x7fbc6c035a70 auth_method 0 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 msgr2=0x7fbc8c08e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc8affd700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 0x7fbc8c08e7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c06a310 tx=0x7fbc7c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c021070 con 0x7fbc8c08e3f0 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:54.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbc7c01c660 con 0x7fbc8c08e3f0 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:54.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 msgr2=0x7fbc8c08e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 0x7fbc8c08e7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c06a310 tx=0x7fbc7c016000 comp rx=0 tx=0).stop 2024-09-18T10:19:54.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.603+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 msgr2=0x7fbc8c08e7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:54.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.607+0000 7fbc837fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08e3f0 0x7fbc8c08e7c0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c06a310 tx=0x7fbc7c016000 comp rx=0 tx=0).stop 2024-09-18T10:19:54.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.607+0000 7fbc8b7fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbc6c035a70 0x7fbc6c037f20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbc84003d00 tx=0x7fbc84009310 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 0x7fbc8c089ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c064830 con 0x7fbc8c08ed00 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 0x7fbc8c089ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient: get_auth_request con 0x7fbc8c08ed00 auth_method 0 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 10 monclient(hunting): handle_auth_done global_id 124182 payload 824 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.615+0000 7fbc8affd700 1 -- 172.21.15.142:0/3278654466 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbc8c091550 con 0x7fbc8c08ed00 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8affd700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 msgr2=0x7fbc8c089ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8affd700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 0x7fbc8c089ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c01d000 tx=0x7fbc7c01e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c025e90 con 0x7fbc8c08ed00 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 10 monclient: dump: 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.627 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7fbc7c01b440 con 0x7fbc8c08ed00 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 msgr2=0x7fbc8c089ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 0x7fbc8c089ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c01d000 tx=0x7fbc7c01e070 comp rx=0 tx=0).stop 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 msgr2=0x7fbc8c089ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:54.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc837fe700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c08ed00 0x7fbc8c089ff0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c01d000 tx=0x7fbc7c01e070 comp rx=0 tx=0).stop 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 114, 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:19:54.629 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.630 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.631 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:54.632 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.633 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:54.634 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.635 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.636 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.637 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.638 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.639 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.640 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:19:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.641 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:54.641 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbc6c035a70 msgr2=0x7fbc6c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbc6c035a70 0x7fbc6c037f20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbc84003d00 tx=0x7fbc84009310 comp rx=0 tx=0).stop 2024-09-18T10:19:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 10 monclient: shutdown 2024-09-18T10:19:54.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 msgr2=0x7fbc8c141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 --2- 172.21.15.142:0/3278654466 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc8c095f30 0x7fbc8c141d70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400b7e0 tx=0x7fbc74009220 comp rx=0 tx=0).stop 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8b7fe700 1 -- 172.21.15.142:0/3278654466 reap_dead start 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8b7fe700 1 -- 172.21.15.142:0/3278654466 reap_dead start 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 shutdown_connections 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 >> 172.21.15.142:0/3278654466 conn(0x7fbc8c071ae0 msgr2=0x7fbc8c06ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 shutdown_connections 2024-09-18T10:19:54.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.619+0000 7fbc8bfff700 1 -- 172.21.15.142:0/3278654466 wait complete. 2024-09-18T10:19:54.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:19:54.644 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:19:54.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:19:54.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: get_monmap_and_config 2024-09-18T10:19:54.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: build_initial_monmap 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: monmap: 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:19:54.748491+0000 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:19:54.748491+0000 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:19:54.751 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:19:54.752 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 Processor -- start 2024-09-18T10:19:54.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 -- start start 2024-09-18T10:19:54.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: init 2024-09-18T10:19:54.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d5809abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d5809b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-b con 0x7f7d58135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5809dc90 con 0x7f7d5809abd0 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5809afa0 con 0x7f7d5809b740 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5809b500 con 0x7f7d58135d90 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5e05f700 10 monclient(hunting): authenticate will time out at 1483.673988s 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35296/0 (socket says 172.21.15.142:35296) 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 -- 172.21.15.142:0/3948844603 learned_addr learned my addr 172.21.15.142:0/3948844603 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): get_auth_request con 0x7f7d58135d90 auth_method 0 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d05d700 1 --1- 172.21.15.142:0/3948844603 >> v1:172.21.15.142:6789/0 conn(0x7f7d5809abd0 0x7f7d580960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51824/0 (socket says 172.21.15.142:51824) 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5c85c700 1 -- 172.21.15.142:0/3948844603 >> v1:172.21.15.150:6789/0 conn(0x7f7d5809b740 legacy=0x7f7d58134670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_done global_id 75095 payload 274 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 -- 172.21.15.142:0/3948844603 >> v1:172.21.15.150:6789/0 conn(0x7f7d5809b740 legacy=0x7f7d58134670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 -- 172.21.15.142:0/3948844603 >> v1:172.21.15.142:6789/0 conn(0x7f7d5809abd0 legacy=0x7f7d580960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 monclient: found mon.noname-b 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.747+0000 7f7d5d85e700 1 -- 172.21.15.142:0/3948844603 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5806f810 con 0x7f7d58135d90 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 -- 172.21.15.142:0/3948844603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 msgr2=0x7f7d58138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 --2- 172.21.15.142:0/3948844603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009f50 tx=0x7f7d50009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93057ba34ddcca1d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 1 -- 172.21.15.142:0/3948844603 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5000e070 con 0x7f7d58135d90 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: dump: 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 5 monclient: authenticate success, global_id 75095 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 1 -- 172.21.15.142:0/3948844603 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5000bcc0 con 0x7f7d58135d90 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: get_monmap_and_config success 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 4 set_mon_vals no callback set 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 1 -- 172.21.15.142:0/3948844603 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d50012680 con 0x7f7d58135d90 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d4ffff700 10 monclient: dump: 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.760 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: shutdown 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- 172.21.15.142:0/3948844603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 msgr2=0x7f7d58138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/3948844603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009f50 tx=0x7f7d50009840 comp rx=0 tx=0).stop 2024-09-18T10:19:54.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- 172.21.15.142:0/3948844603 shutdown_connections 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/3948844603 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d58138170 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- 172.21.15.142:0/3948844603 >> 172.21.15.142:0/3948844603 conn(0x7f7d58070ef0 msgr2=0x7f7d580712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- 172.21.15.142:0/3948844603 shutdown_connections 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- 172.21.15.142:0/3948844603 wait complete. 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: build_initial_monmap 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: monmap: 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:19:54.762 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:19:54.763 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:19:54.763 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 Processor -- start 2024-09-18T10:19:54.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- start start 2024-09-18T10:19:54.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: init 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: _reopen_session rank -1 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5809abd0 0x7f7d5808dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-a con 0x7f7d5809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d5809b740 0x7f7d5808ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-b con 0x7f7d5809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): picked mon.noname-c con 0x7f7d58135d90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5809f510 con 0x7f7d5809abd0 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5808c7a0 con 0x7f7d5809b740 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5808cef0 con 0x7f7d58135d90 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): _renew_subs 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5e05f700 10 monclient(hunting): authenticate will time out at 1483.679042s 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5c85c700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d5809b740 msgr2=0x7f7d5808ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5809abd0 0x7f7d5808dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5c85c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d5809b740 0x7f7d5808ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35310/0 (socket says 172.21.15.142:35310) 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 learned_addr learned my addr 172.21.15.142:0/1128527972 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): get_auth_request con 0x7f7d58135d90 auth_method 0 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): get_auth_request con 0x7f7d5809abd0 auth_method 0 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient(hunting): handle_auth_done global_id 75098 payload 824 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d5809b740 msgr2=0x7f7d5808ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d5809b740 0x7f7d5808ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5809abd0 msgr2=0x7f7d5808dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5809abd0 0x7f7d5808dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient: _finish_hunting 0 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 monclient: found mon.noname-c 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d50009ae0 con 0x7f7d58135d90 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.751+0000 7f7d5d05d700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d5809abd0 0x7f7d5808dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.755+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 msgr2=0x7f7d5808b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.755+0000 7f7d5d85e700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7d5809b150 tx=0x7f7d50017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.755+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d50012e50 con 0x7f7d58135d90 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: dump: 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5001e400 con 0x7f7d58135d90 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5001e7c0 con 0x7f7d58135d90 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 10 monclient: dump: 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.773 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 5 monclient: authenticate success, global_id 75098 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 monclient: _renew_subs 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5808e2d0 con 0x7f7d58135d90 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4f7fe700 4 set_mon_vals no callback set 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 monclient: _renew_subs 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5808d370 con 0x7f7d58135d90 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 0x7f7d58087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d580648a0 con 0x7f7d5808b6e0 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7d50019030 con 0x7f7d58135d90 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 0x7f7d58087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient: get_auth_request con 0x7f7d5808b6e0 auth_method 0 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d3c035ac0 0x7f7d3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7d5004ad60 con 0x7f7d58135d90 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_done global_id 75101 payload 824 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5d05d700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d5808ed90 con 0x7f7d5808b6e0 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5c85c700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d3c035ac0 0x7f7d3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.759+0000 7f7d5c85c700 10 monclient: get_auth_request con 0x7f7d3c035ac0 auth_method 0 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d5d05d700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 msgr2=0x7f7d58087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d5d05d700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 0x7f7d58087fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005210 tx=0x7f7d54007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400fb70 con 0x7f7d5808b6e0 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 10 monclient: dump: 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d5c85c700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d3c035ac0 0x7f7d3c037f70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7d48006fd0 tx=0x7f7d48006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7d5400fcd0 con 0x7f7d5808b6e0 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 msgr2=0x7f7d58087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 0x7f7d58087fe0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005210 tx=0x7f7d54007ed0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 msgr2=0x7f7d58087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:54.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.763+0000 7f7d4d7fa700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d5808b6e0 0x7f7d58087fe0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.771+0000 7f7d5e05f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.771+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 0x7f7d58088920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5e05f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5808e460 con 0x7f7d58088520 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 0x7f7d58088920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient: get_auth_request con 0x7f7d58088520 auth_method 0 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:19:54.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 10 monclient(hunting): handle_auth_done global_id 75104 payload 824 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 1 -- 172.21.15.142:0/1128527972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7d58072a80 con 0x7f7d58088520 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 msgr2=0x7f7d58088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:19:54.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d5d05d700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 0x7f7d58088920 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005210 tx=0x7f7d54016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:19:54.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5401ca50 con 0x7f7d58088520 2024-09-18T10:19:54.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d4d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:19:54.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d4d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.775+0000 7f7d4d7fa700 10 monclient: dump: 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f7d5400f960 con 0x7f7d58088520 2024-09-18T10:19:54.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 msgr2=0x7f7d58088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 0x7f7d58088920 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005210 tx=0x7f7d54016000 comp rx=0 tx=0).stop 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 msgr2=0x7f7d58088920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d4d7fa700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58088520 0x7f7d58088920 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:54.783 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 114, 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.784 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.785 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:19:54.786 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:19:54.787 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:19:54.788 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.789 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.790 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:19:54.791 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:19:54.792 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:19:54.793 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.794 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:19:54.795 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:19:54.796 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d3c035ac0 msgr2=0x7f7d3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d3c035ac0 0x7f7d3c037f70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7d48006fd0 tx=0x7f7d48006e10 comp rx=0 tx=0).stop 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 10 monclient: shutdown 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5d85e700 1 -- 172.21.15.142:0/1128527972 reap_dead start 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 msgr2=0x7f7d5808b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7d5809b150 tx=0x7f7d50017000 comp rx=0 tx=0).stop 2024-09-18T10:19:54.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 shutdown_connections 2024-09-18T10:19:54.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 --2- 172.21.15.142:0/1128527972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d58135d90 0x7f7d5808b1a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:19:54.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 >> 172.21.15.142:0/1128527972 conn(0x7f7d58070ef0 msgr2=0x7f7d5809de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:19:54.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 shutdown_connections 2024-09-18T10:19:54.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:19:54.779+0000 7f7d5e05f700 1 -- 172.21.15.142:0/1128527972 wait complete. 2024-09-18T10:19:54.801 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:19:54.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:20:14.802 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:20:14.802 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:20:14.802 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:20:14.803 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:20:14.806 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:20:14.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:20:14.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: get_monmap_and_config 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: build_initial_monmap 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: monmap: 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:14.916971+0000 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:14.916971+0000 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:14.919 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:14.920 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 Processor -- start 2024-09-18T10:20:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 -- start start 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: init 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): picked mon.noname-b con 0x7f737c09d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f737c0989c0 0x7f737c098d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): picked mon.noname-c con 0x7f737c0989c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f737c0992d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:14.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c069c70 con 0x7f737c0992d0 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c06f9d0 con 0x7f737c0989c0 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c09dc40 con 0x7f737c09d870 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f738095b700 10 monclient(hunting): authenticate will time out at 1503.842606s 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49008/0 (socket says 172.21.15.142:49008) 2024-09-18T10:20:14.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 -- 172.21.15.142:0/2885041937 learned_addr learned my addr 172.21.15.142:0/2885041937 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): get_auth_request con 0x7f737c09d870 auth_method 0 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f7373fff700 1 --2- 172.21.15.142:0/2885041937 >> v2:172.21.15.150:3300/0 conn(0x7f737c0989c0 0x7f737c098d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f7373fff700 10 monclient(hunting): get_auth_request con 0x7f737c0989c0 auth_method 0 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f7373fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f7373fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f7373fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:14.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient(hunting): handle_auth_done global_id 75107 payload 274 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 -- 172.21.15.142:0/2885041937 >> v2:172.21.15.150:3300/0 conn(0x7f737c0989c0 msgr2=0x7f737c098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 --2- 172.21.15.142:0/2885041937 >> v2:172.21.15.150:3300/0 conn(0x7f737c0989c0 0x7f737c098d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 -- 172.21.15.142:0/2885041937 >> v1:172.21.15.142:6789/0 conn(0x7f737c0992d0 legacy=0x7f737c135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 monclient: found mon.noname-b 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.915+0000 7f737a59c700 1 -- 172.21.15.142:0/2885041937 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c06fc20 con 0x7f737c09d870 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f737a59c700 1 -- 172.21.15.142:0/2885041937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 msgr2=0x7f737c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f737a59c700 1 --2- 172.21.15.142:0/2885041937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f737c09de50 tx=0x7f736c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08bf9e589595eb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 1 -- 172.21.15.142:0/2885041937 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c00ba70 con 0x7f737c09d870 2024-09-18T10:20:14.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: dump: 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:14.929 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: _finish_auth 0 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 5 monclient: authenticate success, global_id 75107 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 1 -- 172.21.15.142:0/2885041937 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c00bbd0 con 0x7f737c09d870 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 1 -- 172.21.15.142:0/2885041937 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c011420 con 0x7f737c09d870 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 4 set_mon_vals no callback set 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:14.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f7379d9b700 10 monclient: dump: 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:14.931 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 10 monclient: shutdown 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 -- 172.21.15.142:0/2885041937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 msgr2=0x7f737c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 --2- 172.21.15.142:0/2885041937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f737c09de50 tx=0x7f736c009250 comp rx=0 tx=0).stop 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 -- 172.21.15.142:0/2885041937 shutdown_connections 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 --2- 172.21.15.142:0/2885041937 >> v2:172.21.15.150:3300/0 conn(0x7f737c0989c0 0x7f737c098d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 --2- 172.21.15.142:0/2885041937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c0954b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 -- 172.21.15.142:0/2885041937 >> 172.21.15.142:0/2885041937 conn(0x7f737c0a14e0 msgr2=0x7f737c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 -- 172.21.15.142:0/2885041937 shutdown_connections 2024-09-18T10:20:14.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.919+0000 7f738095b700 1 -- 172.21.15.142:0/2885041937 wait complete. 2024-09-18T10:20:14.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient: build_initial_monmap 2024-09-18T10:20:14.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient: monmap: 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 Processor -- start 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 -- start start 2024-09-18T10:20:14.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient: init 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): picked mon.noname-a con 0x7f737c0989c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): picked mon.noname-b con 0x7f737c0992d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c08a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): picked mon.noname-c con 0x7f737c09d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c097b80 con 0x7f737c0989c0 2024-09-18T10:20:14.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c097180 con 0x7f737c0992d0 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c08d660 con 0x7f737c09d870 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f738095b700 10 monclient(hunting): authenticate will time out at 1503.850839s 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c08a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52770/0 (socket says 172.21.15.142:52770) 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 -- 172.21.15.142:0/137074957 learned_addr learned my addr 172.21.15.142:0/137074957 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): get_auth_request con 0x7f737c0989c0 auth_method 0 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:14.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): get_auth_request con 0x7f737c09d870 auth_method 0 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 10 monclient(hunting): get_auth_request con 0x7f737c0992d0 auth_method 0 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:14.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 msgr2=0x7f737c08a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 msgr2=0x7f737c08a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f7373fff700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient(hunting): handle_auth_done global_id 124188 payload 824 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 msgr2=0x7f737c08a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c08a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 msgr2=0x7f737c08a470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:14.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 monclient: found mon.noname-a 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737a59c700 1 -- 172.21.15.142:0/137074957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c0094f0 con 0x7f737c0989c0 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.923+0000 7f737ad9d700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c08a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f737a59c700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 msgr2=0x7f737c089f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f737a59c700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f736c00b3c0 tx=0x7f736c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c01f070 con 0x7f737c0989c0 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: dump: 2024-09-18T10:20:14.945 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: _finish_auth 0 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 5 monclient: authenticate success, global_id 124188 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c011e80 con 0x7f737c0989c0 2024-09-18T10:20:14.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _renew_subs 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c08c4b0 con 0x7f737c0989c0 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c01a860 con 0x7f737c0989c0 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f737959a700 4 set_mon_vals no callback set 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f73727fc700 10 monclient: dump: 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:14.947 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _renew_subs 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c08e040 con 0x7f737c0989c0 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.927+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c08e2c0 con 0x7f737c0989c0 2024-09-18T10:20:14.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.931+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f736c01ac90 con 0x7f737c0989c0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.931+0000 7f73727fc700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 0x7f735c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.931+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f736c04a330 con 0x7f737c0989c0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.931+0000 7f7373fff700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 0x7f735c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.931+0000 7f7373fff700 10 monclient: get_auth_request con 0x7f735c035ac0 auth_method 0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.935+0000 7f7373fff700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 0x7f735c037f70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7364006fd0 tx=0x7f7364008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.939+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f736c0225a0 con 0x7f737c0989c0 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.939+0000 7f73727fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:14.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:14.939+0000 7f73727fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:15.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f738095b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f738095b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f738095b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f737c08d8a0 con 0x7f737c0989c0 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f73727fc700 1 -- 172.21.15.142:0/137074957 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f736c028070 con 0x7f737c0989c0 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f73727fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:15.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.159+0000 7f73727fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:15.165 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 msgr2=0x7f735c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 0x7f735c037f70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7364006fd0 tx=0x7f7364008040 comp rx=0 tx=0).stop 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 10 monclient: shutdown 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 msgr2=0x7f737c089f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f736c00b3c0 tx=0x7f736c017000 comp rx=0 tx=0).stop 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 shutdown_connections 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f735c035ac0 0x7f735c037f70 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:15.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f737c09d870 0x7f737c08a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:15.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f737c0992d0 0x7f737c08a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:15.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 --2- 172.21.15.142:0/137074957 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f737c0989c0 0x7f737c089f30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:15.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 >> 172.21.15.142:0/137074957 conn(0x7f737c0a14e0 msgr2=0x7f737c097390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:15.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 shutdown_connections 2024-09-18T10:20:15.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:15.163+0000 7f738095b700 1 -- 172.21.15.142:0/137074957 wait complete. 2024-09-18T10:20:18.179 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: get_monmap_and_config 2024-09-18T10:20:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: build_initial_monmap 2024-09-18T10:20:18.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: monmap: 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:18.292871+0000 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:18.292871+0000 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:18.295 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 Processor -- start 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 -- start start 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: init 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:20:18.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7dd40a1cf0 0x7f7dd412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-a con 0x7f7dd40a1cf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-b con 0x7f7dd412f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7dd40a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd409d990 con 0x7f7dd40a02d0 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd409dc70 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40a20c0 con 0x7f7dd412f810 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd9029700 10 monclient(hunting): authenticate will time out at 1507.219335s 2024-09-18T10:20:18.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7dd40a1cf0 0x7f7dd412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49034/0 (socket says 172.21.15.142:49034) 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7dd40a1cf0 0x7f7dd412f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52772/0 (socket says 172.21.15.142:52772) 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 1 -- 172.21.15.142:0/2773916207 learned_addr learned my addr 172.21.15.142:0/2773916207 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 10 monclient(hunting): get_auth_request con 0x7f7dd412f810 auth_method 0 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 10 monclient(hunting): get_auth_request con 0x7f7dd40a1cf0 auth_method 0 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:18.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.291+0000 7f7dd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient(hunting): handle_auth_done global_id 124191 payload 274 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 -- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 msgr2=0x7f7dd409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 -- 172.21.15.142:0/2773916207 >> v1:172.21.15.150:6789/0 conn(0x7f7dd40a02d0 legacy=0x7f7dd41320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 monclient: found mon.noname-a 2024-09-18T10:20:18.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 -- 172.21.15.142:0/2773916207 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd409f9c0 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd37fe700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:18.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 -- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd40a1cf0 msgr2=0x7f7dd412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:18.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd40a1cf0 0x7f7dd412f030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40098e0 tx=0x7f7dc40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce361f42cda7b661 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:18.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 1 -- 172.21.15.142:0/2773916207 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400b850 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: dump: 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd9029700 5 monclient: authenticate success, global_id 124191 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd9029700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 1 -- 172.21.15.142:0/2773916207 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc400b9b0 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd9029700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd9029700 4 set_mon_vals no callback set 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 1 -- 172.21.15.142:0/2773916207 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400bd50 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.295+0000 7f7dd2ffd700 10 monclient: dump: 2024-09-18T10:20:18.306 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 10 monclient: shutdown 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd40a1cf0 msgr2=0x7f7dd412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd40a1cf0 0x7f7dd412f030 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40098e0 tx=0x7f7dc40091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- 172.21.15.142:0/2773916207 shutdown_connections 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd412f810 0x7f7dd409f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 --2- 172.21.15.142:0/2773916207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd40a1cf0 0x7f7dd412f030 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- 172.21.15.142:0/2773916207 >> 172.21.15.142:0/2773916207 conn(0x7f7dd4071820 msgr2=0x7f7dd40a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- 172.21.15.142:0/2773916207 shutdown_connections 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- 172.21.15.142:0/2773916207 wait complete. 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 10 monclient: build_initial_monmap 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 10 monclient: monmap: 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:18.308 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 Processor -- start 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 1 -- start start 2024-09-18T10:20:18.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 10 monclient: init 2024-09-18T10:20:18.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.299+0000 7f7dd9029700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-b con 0x7f7dd40a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd40a1cf0 0x7f7dd409b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-c con 0x7f7dd40a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): picked mon.noname-a con 0x7f7dd412f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd412ef50 con 0x7f7dd412f810 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd409cdc0 con 0x7f7dd40a02d0 2024-09-18T10:20:18.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd406bc90 con 0x7f7dd40a1cf0 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd9029700 10 monclient(hunting): authenticate will time out at 1507.228255s 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd40a1cf0 0x7f7dd409b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52774/0 (socket says 172.21.15.142:52774) 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:51796/0 (socket says 172.21.15.142:51796) 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 -- 172.21.15.142:0/1666890260 learned_addr learned my addr 172.21.15.142:0/1666890260 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): get_auth_request con 0x7f7dd412f810 auth_method 0 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:18.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 10 monclient(hunting): get_auth_request con 0x7f7dd40a02d0 auth_method 0 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): get_auth_request con 0x7f7dd40a1cf0 auth_method 0 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 msgr2=0x7f7dd409b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:18.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 msgr2=0x7f7dd409b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient(hunting): handle_auth_done global_id 124194 payload 824 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd40a1cf0 msgr2=0x7f7dd409b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd40a1cf0 0x7f7dd409b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 msgr2=0x7f7dd409b0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient: _finish_hunting 0 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 monclient: found mon.noname-a 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 -- 172.21.15.142:0/1666890260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80094f0 con 0x7f7dd412f810 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 msgr2=0x7f7dd409bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dd8828700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c930 tx=0x7f7dcc0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc012840 con 0x7f7dd412f810 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:18.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 10 monclient: dump: 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 10 monclient: _finish_auth 0 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.303+0000 7f7dbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:18.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 5 monclient: authenticate success, global_id 124194 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _renew_subs 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc0129c0 con 0x7f7dd412f810 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc4009420 con 0x7f7dd412f810 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc012d80 con 0x7f7dd412f810 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 10 monclient: dump: 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.320 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd1ffb700 4 set_mon_vals no callback set 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _renew_subs 2024-09-18T10:20:18.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd406cae0 con 0x7f7dd412f810 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4072a80 con 0x7f7dd412f810 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7dcc00a5c0 con 0x7f7dd412f810 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 0x7f7dc0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7dcc049010 con 0x7f7dd412f810 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 0x7f7dc0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:18.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.307+0000 7f7dd3fff700 10 monclient: get_auth_request con 0x7f7dc0035ac0 auth_method 0 2024-09-18T10:20:18.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.311+0000 7f7dd3fff700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 0x7f7dc0037f70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40093f0 tx=0x7f7dc4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:18.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.319+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7dcc010d50 con 0x7f7dd412f810 2024-09-18T10:20:18.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.319+0000 7f7dbbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:18.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.319+0000 7f7dbbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:18.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.539+0000 7f7dd9029700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:18.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.539+0000 7f7dd9029700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:18.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.539+0000 7f7dd9029700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:18.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.539+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7dd40a1a90 con 0x7f7dd412f810 2024-09-18T10:20:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dbbfff700 1 -- 172.21.15.142:0/1666890260 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f7dcc010750 con 0x7f7dd412f810 2024-09-18T10:20:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dbbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:18.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dbbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:18.546 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 msgr2=0x7f7dc0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 0x7f7dc0037f70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40093f0 tx=0x7f7dc4016000 comp rx=0 tx=0).stop 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 10 monclient: shutdown 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 msgr2=0x7f7dd409bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:18.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c930 tx=0x7f7dcc0026e0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 shutdown_connections 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7dc0035ac0 0x7f7dc0037f70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dd412f810 0x7f7dd409bb40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dd40a1cf0 0x7f7dd409b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 --2- 172.21.15.142:0/1666890260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dd40a02d0 0x7f7dd409b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 >> 172.21.15.142:0/1666890260 conn(0x7f7dd4071820 msgr2=0x7f7dd4131fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 shutdown_connections 2024-09-18T10:20:18.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:18.543+0000 7f7dd9029700 1 -- 172.21.15.142:0/1666890260 wait complete. 2024-09-18T10:20:21.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: get_monmap_and_config 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: build_initial_monmap 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: monmap: 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:21.668445+0000 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:21.668445+0000 2024-09-18T10:20:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:21.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 Processor -- start 2024-09-18T10:20:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 -- start start 2024-09-18T10:20:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: init 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f872c09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f872c097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c09db80 0x7f872c1386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): picked mon.noname-a con 0x7f872c09db80 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c0958c0 con 0x7f872c097880 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872c095ef0 con 0x7f872c09abd0 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f872c06fa60 con 0x7f872c09db80 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f873312b700 10 monclient(hunting): authenticate will time out at 1510.593894s 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c09db80 0x7f872c1386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f872c09db80 0x7f872c1386a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52802/0 (socket says 172.21.15.142:52802) 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f872bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f872c097880 0x7f872c09c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:56036/0 (socket says 172.21.15.142:56036) 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 -- 172.21.15.142:0/1876136729 learned_addr learned my addr 172.21.15.142:0/1876136729 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): get_auth_request con 0x7f872c09db80 auth_method 0 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient(hunting): handle_auth_done global_id 124197 payload 274 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 -- 172.21.15.142:0/1876136729 >> v1:172.21.15.150:6789/0 conn(0x7f872c09abd0 legacy=0x7f872c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 -- 172.21.15.142:0/1876136729 >> v1:172.21.15.142:6789/0 conn(0x7f872c097880 legacy=0x7f872c09c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient: _finish_hunting 0 2024-09-18T10:20:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 monclient: found mon.noname-a 2024-09-18T10:20:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.666+0000 7f87316c8700 1 -- 172.21.15.142:0/1876136729 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c09afa0 con 0x7f872c09db80 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 1 -- 172.21.15.142:0/1876136729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c09db80 msgr2=0x7f872c1386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 1 --2- 172.21.15.142:0/1876136729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c09db80 0x7f872c1386a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f87240099b0 tx=0x7f8724009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35992e8561167d5e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 1 -- 172.21.15.142:0/1876136729 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872400bb50 con 0x7f872c09db80 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: dump: 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 5 monclient: authenticate success, global_id 124197 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 1 -- 172.21.15.142:0/1876136729 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872400bcb0 con 0x7f872c09db80 2024-09-18T10:20:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 4 set_mon_vals no callback set 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 1 -- 172.21.15.142:0/1876136729 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8724011670 con 0x7f872c09db80 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872b7fe700 10 monclient: dump: 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: shutdown 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- 172.21.15.142:0/1876136729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c09db80 msgr2=0x7f872c1386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 --2- 172.21.15.142:0/1876136729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c09db80 0x7f872c1386a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f87240099b0 tx=0x7f8724009250 comp rx=0 tx=0).stop 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- 172.21.15.142:0/1876136729 shutdown_connections 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 --2- 172.21.15.142:0/1876136729 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c09db80 0x7f872c1386a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- 172.21.15.142:0/1876136729 >> 172.21.15.142:0/1876136729 conn(0x7f872c0a14e0 msgr2=0x7f872c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- 172.21.15.142:0/1876136729 shutdown_connections 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- 172.21.15.142:0/1876136729 wait complete. 2024-09-18T10:20:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: build_initial_monmap 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: monmap: 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 Processor -- start 2024-09-18T10:20:21.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- start start 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: init 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c097880 0x7f872c0908b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): picked mon.noname-a con 0x7f872c097880 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): picked mon.noname-c con 0x7f872c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): picked mon.noname-b con 0x7f872c09db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c091670 con 0x7f872c097880 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c0918e0 con 0x7f872c09db80 2024-09-18T10:20:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872c097c90 con 0x7f872c09abd0 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f873312b700 10 monclient(hunting): authenticate will time out at 1510.599389s 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c097880 0x7f872c0908b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49056/0 (socket says 172.21.15.142:49056) 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 1 -- 172.21.15.142:0/3842194447 learned_addr learned my addr 172.21.15.142:0/3842194447 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): get_auth_request con 0x7f872c09abd0 auth_method 0 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): get_auth_request con 0x7f872c097880 auth_method 0 2024-09-18T10:20:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f872bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 10 monclient(hunting): get_auth_request con 0x7f872c09db80 auth_method 0 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f87316c8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.670+0000 7f8730ec7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8730ec7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 10 monclient(hunting): handle_auth_done global_id 75119 payload 824 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f87316c8700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 msgr2=0x7f872c098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f87316c8700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 msgr2=0x7f872c098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 msgr2=0x7f872c098f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f87316c8700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f87316c8700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c097880 msgr2=0x7f872c0908b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c097880 0x7f872c0908b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:20:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 monclient: found mon.noname-c 2024-09-18T10:20:21.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:21.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:21.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 -- 172.21.15.142:0/3842194447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240094f0 con 0x7f872c09abd0 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 msgr2=0x7f872c098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872bfff700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f871c00cd90 tx=0x7f871c005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c00a920 con 0x7f872c09abd0 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: dump: 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:21.690 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: _finish_auth 0 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 5 monclient: authenticate success, global_id 75119 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c00aaa0 con 0x7f872c09abd0 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 10 monclient: _renew_subs 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c08fcd0 con 0x7f872c09abd0 2024-09-18T10:20:21.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c00ae60 con 0x7f872c09abd0 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f8728ff9700 10 monclient: dump: 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:21.692 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 10 monclient: _renew_subs 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f872a7fc700 4 set_mon_vals no callback set 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.674+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c094650 con 0x7f872c09abd0 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f871c007300 con 0x7f872c09abd0 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f873312b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f873312b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:21.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f873312b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f872c06eaa0 con 0x7f872c09abd0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f8728ff9700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 0x7f8718037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f871c0491a0 con 0x7f872c09abd0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f8730ec7700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 0x7f8718037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.678+0000 7f8730ec7700 10 monclient: get_auth_request con 0x7f8718035ac0 auth_method 0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.682+0000 7f8730ec7700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 0x7f8718037f70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8720003a30 tx=0x7f8720007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.686+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f871c01a360 con 0x7f872c09abd0 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.686+0000 7f8728ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:21.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.686+0000 7f8728ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:21.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f873312b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:21.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f873312b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:21.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f873312b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:21.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f872c0696a0 con 0x7f872c09abd0 2024-09-18T10:20:21.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f8728ff9700 1 -- 172.21.15.142:0/3842194447 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f871c010a60 con 0x7f872c09abd0 2024-09-18T10:20:21.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f8728ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:21.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.906+0000 7f8728ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:21.913 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:21.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 msgr2=0x7f8718037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:21.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 0x7f8718037f70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8720003a30 tx=0x7f8720007ea0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 10 monclient: shutdown 2024-09-18T10:20:21.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 msgr2=0x7f872c098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f871c00cd90 tx=0x7f871c005480 comp rx=0 tx=0).stop 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 shutdown_connections 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8718035ac0 0x7f8718037f70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f872c09db80 0x7f872c098f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f872c09abd0 0x7f872c098a40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 --2- 172.21.15.142:0/3842194447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f872c097880 0x7f872c0908b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:21.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 >> 172.21.15.142:0/3842194447 conn(0x7f872c0a14e0 msgr2=0x7f872c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:21.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 shutdown_connections 2024-09-18T10:20:21.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:21.910+0000 7f873312b700 1 -- 172.21.15.142:0/3842194447 wait complete. 2024-09-18T10:20:21.926 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:20:21.926 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:20:21.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:20:22.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.022+0000 7fdb33669700 10 monclient: get_monmap_and_config 2024-09-18T10:20:22.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.022+0000 7fdb33669700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.022+0000 7fdb33669700 10 monclient: monmap: 2024-09-18T10:20:22.050 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.050 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.050 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:22.027170+0000 2024-09-18T10:20:22.050 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:22.027170+0000 2024-09-18T10:20:22.051 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.051 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.051 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:22.051 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:22.051 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 Processor -- start 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 -- start start 2024-09-18T10:20:22.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient: init 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-a con 0x7fdb2c09abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 0x7fdb2c09c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-b con 0x7fdb2c09db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:22.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-c con 0x7fdb2c09c950 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:22.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c06fa60 con 0x7fdb2c09abd0 2024-09-18T10:20:22.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c0958c0 con 0x7fdb2c09c950 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c095ef0 con 0x7fdb2c09db80 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb33669700 10 monclient(hunting): authenticate will time out at 1510.952934s 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 0x7fdb2c09c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52824/0 (socket says 172.21.15.142:52824) 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 1 -- 172.21.15.142:0/2502640249 learned_addr learned my addr 172.21.15.142:0/2502640249 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): get_auth_request con 0x7fdb2c09abd0 auth_method 0 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): get_auth_request con 0x7fdb2c09db80 auth_method 0 2024-09-18T10:20:22.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 10 monclient(hunting): get_auth_request con 0x7fdb2c09c950 auth_method 0 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 1 -- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 msgr2=0x7fdb2c096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:20:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 1 -- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 msgr2=0x7fdb2c096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient(hunting): handle_auth_done global_id 75122 payload 274 2024-09-18T10:20:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31c06700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 -- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 msgr2=0x7fdb2c096870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 -- 172.21.15.142:0/2502640249 >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 msgr2=0x7fdb2c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 monclient: found mon.noname-b 2024-09-18T10:20:22.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb30c04700 1 -- 172.21.15.142:0/2502640249 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c09afa0 con 0x7fdb2c09db80 2024-09-18T10:20:22.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.026+0000 7fdb31405700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:22.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb30c04700 1 -- 172.21.15.142:0/2502640249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 msgr2=0x7fdb2c09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb30c04700 1 --2- 172.21.15.142:0/2502640249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 0x7fdb2c09c410 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c06fdb0 tx=0x7fdb1c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f71bbd96a36a44 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 1 -- 172.21.15.142:0/2502640249 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011840 con 0x7fdb2c09db80 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: dump: 2024-09-18T10:20:22.063 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 5 monclient: authenticate success, global_id 75122 2024-09-18T10:20:22.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 1 -- 172.21.15.142:0/2502640249 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c0119a0 con 0x7fdb2c09db80 2024-09-18T10:20:22.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:22.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 1 -- 172.21.15.142:0/2502640249 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011d40 con 0x7fdb2c09db80 2024-09-18T10:20:22.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 4 set_mon_vals no callback set 2024-09-18T10:20:22.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:22.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb23fff700 10 monclient: dump: 2024-09-18T10:20:22.067 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.068 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.069 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.069 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.069 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: shutdown 2024-09-18T10:20:22.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- 172.21.15.142:0/2502640249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 msgr2=0x7fdb2c09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 --2- 172.21.15.142:0/2502640249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 0x7fdb2c09c410 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c06fdb0 tx=0x7fdb1c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- 172.21.15.142:0/2502640249 shutdown_connections 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.150:3300/0 conn(0x7fdb2c09c950 0x7fdb2c096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 --2- 172.21.15.142:0/2502640249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09db80 0x7fdb2c09c410 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 --2- 172.21.15.142:0/2502640249 >> v2:172.21.15.142:3300/0 conn(0x7fdb2c09abd0 0x7fdb2c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- 172.21.15.142:0/2502640249 >> 172.21.15.142:0/2502640249 conn(0x7fdb2c0a14e0 msgr2=0x7fdb2c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:22.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- 172.21.15.142:0/2502640249 shutdown_connections 2024-09-18T10:20:22.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- 172.21.15.142:0/2502640249 wait complete. 2024-09-18T10:20:22.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: monmap: 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:22.072 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.073 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.073 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:22.073 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:22.073 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:22.073 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 Processor -- start 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 -- start start 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: init 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:20:22.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 0x7fdb2c1366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.030+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-b con 0x7fdb2c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 0x7fdb2c136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-c con 0x7fdb2c09c950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): picked mon.noname-a con 0x7fdb2c09db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:22.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c08d1b0 con 0x7fdb2c09db80 2024-09-18T10:20:22.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c097c90 con 0x7fdb2c09abd0 2024-09-18T10:20:22.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c09a350 con 0x7fdb2c09c950 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient(hunting): authenticate will time out at 1510.960087s 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 0x7fdb2c136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52830/0 (socket says 172.21.15.142:52830) 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 learned_addr learned my addr 172.21.15.142:0/3942336928 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 0x7fdb2c136bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49074/0 (socket says 172.21.15.142:49074) 2024-09-18T10:20:22.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): get_auth_request con 0x7fdb2c09db80 auth_method 0 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): get_auth_request con 0x7fdb2c09c950 auth_method 0 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 0x7fdb2c1366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 10 monclient(hunting): get_auth_request con 0x7fdb2c09abd0 auth_method 0 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 msgr2=0x7fdb2c1366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 msgr2=0x7fdb2c1366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 0x7fdb2c1366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 0x7fdb2c1366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient(hunting): handle_auth_done global_id 124206 payload 824 2024-09-18T10:20:22.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 msgr2=0x7fdb2c136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 0x7fdb2c136bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 msgr2=0x7fdb2c1366b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb2c09abd0 0x7fdb2c1366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 monclient: found mon.noname-a 2024-09-18T10:20:22.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280094f0 con 0x7fdb2c09db80 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb30c04700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb2c09c950 0x7fdb2c136bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 msgr2=0x7fdb2c137130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb31c06700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009fd0 tx=0x7fdb24009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400ee50 con 0x7fdb2c09db80 2024-09-18T10:20:22.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:22.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: dump: 2024-09-18T10:20:22.084 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.084 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.085 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.086 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.086 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 5 monclient: authenticate success, global_id 124206 2024-09-18T10:20:22.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient: _renew_subs 2024-09-18T10:20:22.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb24014420 con 0x7fdb2c09db80 2024-09-18T10:20:22.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:22.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c009420 con 0x7fdb2c09db80 2024-09-18T10:20:22.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24014820 con 0x7fdb2c09db80 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb217fa700 10 monclient: dump: 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.088 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.089 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient: _renew_subs 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c08b3a0 con 0x7fdb2c09db80 2024-09-18T10:20:22.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.034+0000 7fdb237fe700 4 set_mon_vals no callback set 2024-09-18T10:20:22.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb33669700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:22.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdb24014c50 con 0x7fdb2c09db80 2024-09-18T10:20:22.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb33669700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 0x7fdb2c06d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c08d070 con 0x7fdb2c06d410 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 0x7fdb2c06d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient: get_auth_request con 0x7fdb2c06d410 auth_method 0 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb217fa700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb10035ac0 0x7fdb10037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdb240496a0 con 0x7fdb2c09db80 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 10 monclient(hunting): handle_auth_done global_id 124209 payload 824 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb30c04700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb10035ac0 0x7fdb10037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb2c08d3f0 con 0x7fdb2c06d410 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.038+0000 7fdb30c04700 10 monclient: get_auth_request con 0x7fdb10035ac0 auth_method 0 2024-09-18T10:20:22.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 msgr2=0x7fdb2c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 0x7fdb2c06d7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b370 tx=0x7fdb28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2801b070 con 0x7fdb2c06d410 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 10 monclient: dump: 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.094 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb30c04700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb10035ac0 0x7fdb10037f70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c09da10 tx=0x7fdb1c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdb28014d80 con 0x7fdb2c06d410 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 msgr2=0x7fdb2c06d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 0x7fdb2c06d7e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800b370 tx=0x7fdb28008040 comp rx=0 tx=0).stop 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 msgr2=0x7fdb2c06d7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:22.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.042+0000 7fdb217fa700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06d410 0x7fdb2c06d7e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb33669700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb33669700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 0x7fdb2c0898b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb33669700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c064950 con 0x7fdb2c06dd20 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 0x7fdb2c0898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient: get_auth_request con 0x7fdb2c06dd20 auth_method 0 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 10 monclient(hunting): handle_auth_done global_id 124212 payload 824 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.050+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdb2c0696a0 con 0x7fdb2c06dd20 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb31405700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 msgr2=0x7fdb2c0898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb31405700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 0x7fdb2c0898b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb28016c40 tx=0x7fdb28021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2802f070 con 0x7fdb2c06dd20 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 10 monclient: dump: 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.099 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fdb28014b70 con 0x7fdb2c06dd20 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 msgr2=0x7fdb2c0898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 0x7fdb2c0898b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdb28016c40 tx=0x7fdb28021000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 msgr2=0x7fdb2c0898b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb217fa700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c06dd20 0x7fdb2c0898b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:22.101 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.102 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.103 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:22.104 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.105 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.106 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:22.107 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.108 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:22.109 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.110 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:22.111 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:22.112 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:22.113 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:22.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb10035ac0 msgr2=0x7fdb10037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb10035ac0 0x7fdb10037f70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c09da10 tx=0x7fdb1c017000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 10 monclient: shutdown 2024-09-18T10:20:22.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb31c06700 1 -- 172.21.15.142:0/3942336928 reap_dead start 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 msgr2=0x7fdb2c137130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009fd0 tx=0x7fdb24009870 comp rx=0 tx=0).stop 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 shutdown_connections 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 --2- 172.21.15.142:0/3942336928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb2c09db80 0x7fdb2c137130 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 >> 172.21.15.142:0/3942336928 conn(0x7fdb2c0a14e0 msgr2=0x7fdb2c0988c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 shutdown_connections 2024-09-18T10:20:22.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.054+0000 7fdb33669700 1 -- 172.21.15.142:0/3942336928 wait complete. 2024-09-18T10:20:22.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:22.116 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:20:22.117 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:20:22.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 10 monclient: get_monmap_and_config 2024-09-18T10:20:22.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.231 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 10 monclient: monmap: 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:22.229695+0000 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:22.229695+0000 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:22.232 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 1 Processor -- start 2024-09-18T10:20:22.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 1 -- start start 2024-09-18T10:20:22.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.226+0000 7fc44f7f2700 10 monclient: init 2024-09-18T10:20:22.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc44809e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-b con 0x7fc44809cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc4480a02e0 0x7fc4480a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-a con 0x7fc4480a02e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-c con 0x7fc4480a0bf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44809d1a0 con 0x7fc4480a02e0 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44809e980 con 0x7fc4480a0bf0 2024-09-18T10:20:22.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44809d780 con 0x7fc44809cdd0 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient(hunting): authenticate will time out at 1511.156265s 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc4480a02e0 0x7fc4480a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc44809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc4480a02e0 0x7fc4480a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60884/0 (socket says 172.21.15.142:60884) 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1415370248 learned_addr learned my addr 172.21.15.142:0/1415370248 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request con 0x7fc4480a02e0 auth_method 0 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): get_auth_request con 0x7fc44809cdd0 auth_method 0 2024-09-18T10:20:22.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 10 monclient(hunting): get_auth_request con 0x7fc4480a0bf0 auth_method 0 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_done global_id 124215 payload 274 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 1 -- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 msgr2=0x7fc448138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 1 -- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 msgr2=0x7fc448138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 msgr2=0x7fc44809e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc44809e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 msgr2=0x7fc448138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 monclient: found mon.noname-a 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:22.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1415370248 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44809dd40 con 0x7fc4480a02e0 2024-09-18T10:20:22.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 msgr2=0x7fc4480a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4480a06b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc44809c9f0 tx=0x7fc444009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef06197d36f5c1a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 1 -- 172.21.15.142:0/1415370248 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444009e70 con 0x7fc4480a02e0 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: dump: 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.245 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 5 monclient: authenticate success, global_id 124215 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 1 -- 172.21.15.142:0/1415370248 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44400ba70 con 0x7fc4480a02e0 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc44f7f2700 4 set_mon_vals no callback set 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 1 -- 172.21.15.142:0/1415370248 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44400be30 con 0x7fc4480a02e0 2024-09-18T10:20:22.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.230+0000 7fc43ffff700 10 monclient: dump: 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: shutdown 2024-09-18T10:20:22.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 msgr2=0x7fc4480a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4480a06b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc44809c9f0 tx=0x7fc444009250 comp rx=0 tx=0).stop 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1415370248 shutdown_connections 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1415370248 >> v2:172.21.15.150:3300/0 conn(0x7fc4480a0bf0 0x7fc448138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4480a06b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1415370248 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc44809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1415370248 >> 172.21.15.142:0/1415370248 conn(0x7fc448071680 msgr2=0x7fc4480a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1415370248 shutdown_connections 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1415370248 wait complete. 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: monmap: 2024-09-18T10:20:22.248 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 Processor -- start 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- start start 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: init 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-c con 0x7fc44809cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4481422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-a con 0x7fc4480a02e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4480a0bf0 0x7fc44808d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): picked mon.noname-b con 0x7fc4480a0bf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448098500 con 0x7fc4480a02e0 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480905f0 con 0x7fc4480a0bf0 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448090310 con 0x7fc44809cdd0 2024-09-18T10:20:22.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44f7f2700 10 monclient(hunting): authenticate will time out at 1511.163308s 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4481422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41984/0 (socket says 172.21.15.142:41984) 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 -- 172.21.15.142:0/1474955284 learned_addr learned my addr 172.21.15.142:0/1474955284 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): get_auth_request con 0x7fc44809cdd0 auth_method 0 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request con 0x7fc4480a02e0 auth_method 0 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4480a0bf0 0x7fc44808d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient(hunting): handle_auth_done global_id 75131 payload 824 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4480a0bf0 msgr2=0x7fc44808d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4480a0bf0 0x7fc44808d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 msgr2=0x7fc4481422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4480a02e0 0x7fc4481422a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 monclient: found mon.noname-c 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44d58e700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4440094f0 con 0x7fc44809cdd0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44dd8f700 10 monclient: get_auth_request con 0x7fc4480a0bf0 auth_method 0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.234+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4480a0bf0 0x7fc44808d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44d58e700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 msgr2=0x7fc448141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44d58e700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e930 tx=0x7fc4380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438013890 con 0x7fc44809cdd0 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:22.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: dump: 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 5 monclient: authenticate success, global_id 75131 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4380139f0 con 0x7fc44809cdd0 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc438013db0 con 0x7fc44809cdd0 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 10 monclient: _renew_subs 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4480908d0 con 0x7fc44809cdd0 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43effd700 4 set_mon_vals no callback set 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 10 monclient: dump: 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.255 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 10 monclient: _renew_subs 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc448091510 con 0x7fc44809cdd0 2024-09-18T10:20:22.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc43800c670 con 0x7fc44809cdd0 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.238+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 0x7fc44806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480648c0 con 0x7fc44806cbc0 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc42c035ac0 0x7fc42c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc438049140 con 0x7fc44809cdd0 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 0x7fc44806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient: get_auth_request con 0x7fc44806cbc0 auth_method 0 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc42c035ac0 0x7fc42c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44dd8f700 10 monclient: get_auth_request con 0x7fc42c035ac0 auth_method 0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_done global_id 75134 payload 824 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc448090480 con 0x7fc44806cbc0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 msgr2=0x7fc44806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 0x7fc44806cf90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc448071e80 tx=0x7fc444017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc44dd8f700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc42c035ac0 0x7fc42c037f70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc440009590 tx=0x7fc44000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc444011cd0 con 0x7fc44806cbc0 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.242+0000 7fc43d7fa700 10 monclient: dump: 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc444020040 con 0x7fc44806cbc0 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:22.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:22.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 msgr2=0x7fc44806cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 0x7fc44806cf90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc448071e80 tx=0x7fc444017000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 msgr2=0x7fc44806cf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:22.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.246+0000 7fc43d7fa700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806cbc0 0x7fc44806cf90 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc448071e80 tx=0x7fc444017000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44f7f2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:22.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 0x7fc44808d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44f7f2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448090cd0 con 0x7fc44806d4d0 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 0x7fc44808d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient: get_auth_request con 0x7fc44806d4d0 auth_method 0 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.254+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc44cd8d700 10 monclient(hunting): handle_auth_done global_id 75137 payload 824 2024-09-18T10:20:22.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1474955284 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc4480696a0 con 0x7fc44806d4d0 2024-09-18T10:20:22.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc44cd8d700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 msgr2=0x7fc44808d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc44cd8d700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 0x7fc44808d900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc448072200 tx=0x7fc444021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44401aaf0 con 0x7fc44806d4d0 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 10 monclient: dump: 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.266 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fc4440263f0 con 0x7fc44806d4d0 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 msgr2=0x7fc44808d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 0x7fc44808d900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc448072200 tx=0x7fc444021000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 msgr2=0x7fc44808d900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.258+0000 7fc43d7fa700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44806d4d0 0x7fc44808d900 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc448072200 tx=0x7fc444021000 comp rx=0 tx=0).stop 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:22.267 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:22.268 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.269 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:22.270 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:22.271 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.272 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.273 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.274 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:22.275 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:22.276 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:22.277 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.278 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:22.279 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:22.280 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:22.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc42c035ac0 msgr2=0x7fc42c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc42c035ac0 0x7fc42c037f70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc440009590 tx=0x7fc44000b040 comp rx=0 tx=0).stop 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 10 monclient: shutdown 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 msgr2=0x7fc448141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc43800e930 tx=0x7fc4380091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44dd8f700 1 -- 172.21.15.142:0/1474955284 reap_dead start 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 shutdown_connections 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 --2- 172.21.15.142:0/1474955284 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc44809cdd0 0x7fc448141d60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 >> 172.21.15.142:0/1474955284 conn(0x7fc448071680 msgr2=0x7fc44809eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 shutdown_connections 2024-09-18T10:20:22.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.262+0000 7fc44f7f2700 1 -- 172.21.15.142:0/1474955284 wait complete. 2024-09-18T10:20:22.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:22.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:20:22.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: get_monmap_and_config 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: monmap: 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:22.388319+0000 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:22.388319+0000 2024-09-18T10:20:22.390 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:22.391 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 Processor -- start 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 -- start start 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: init 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c0c0999e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-c con 0x7f6c0c09c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6c0c137910 0x7f6c0c139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c137910 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:22.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0c0a20c0 con 0x7f6c0c0999e0 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c09a000 con 0x7f6c0c137910 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0c0a1c80 con 0x7f6c0c09c410 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c11fa5700 10 monclient(hunting): authenticate will time out at 1511.314772s 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6c0c137910 0x7f6c0c139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6c0c137910 0x7f6c0c139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60892/0 (socket says 172.21.15.142:60892) 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 -- 172.21.15.142:0/1599573345 learned_addr learned my addr 172.21.15.142:0/1599573345 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): get_auth_request con 0x7f6c0c137910 auth_method 0 2024-09-18T10:20:22.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 10 monclient(hunting): get_auth_request con 0x7f6c0c09c410 auth_method 0 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 1 -- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 msgr2=0x7f6c0c0a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 1 -- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 msgr2=0x7f6c0c0a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient(hunting): handle_auth_done global_id 124221 payload 274 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 -- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 msgr2=0x7f6c0c0a0890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 --2- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 -- 172.21.15.142:0/1599573345 >> v1:172.21.15.150:6789/0 conn(0x7f6c0c0999e0 legacy=0x7f6c0c0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 monclient: found mon.noname-a 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:22.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.386+0000 7f6c117a4700 1 -- 172.21.15.142:0/1599573345 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c09a580 con 0x7f6c0c137910 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c117a4700 1 -- 172.21.15.142:0/1599573345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c137910 msgr2=0x7f6c0c139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c117a4700 1 --2- 172.21.15.142:0/1599573345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c137910 0x7f6c0c139cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c06fbb0 tx=0x7f6c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc1a627c8052a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 1 -- 172.21.15.142:0/1599573345 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400e070 con 0x7f6c0c137910 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: dump: 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.402 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 5 monclient: authenticate success, global_id 124221 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 1 -- 172.21.15.142:0/1599573345 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0400bc60 con 0x7f6c0c137910 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 4 set_mon_vals no callback set 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 1 -- 172.21.15.142:0/1599573345 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040092e0 con 0x7f6c0c137910 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c0b7fe700 10 monclient: dump: 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 10 monclient: shutdown 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 -- 172.21.15.142:0/1599573345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c137910 msgr2=0x7f6c0c139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/1599573345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c137910 0x7f6c0c139cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c06fbb0 tx=0x7f6c04009840 comp rx=0 tx=0).stop 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 -- 172.21.15.142:0/1599573345 shutdown_connections 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/1599573345 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c137910 0x7f6c0c139cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/1599573345 >> v2:172.21.15.150:3300/0 conn(0x7f6c0c09c410 0x7f6c0c0a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 -- 172.21.15.142:0/1599573345 >> 172.21.15.142:0/1599573345 conn(0x7f6c0c0a5790 msgr2=0x7f6c0c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 -- 172.21.15.142:0/1599573345 shutdown_connections 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.390+0000 7f6c11fa5700 1 -- 172.21.15.142:0/1599573345 wait complete. 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient: build_initial_monmap 2024-09-18T10:20:22.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient: monmap: 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 Processor -- start 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 -- start start 2024-09-18T10:20:22.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient: init 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c0999e0 0x7f6c0c12af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-a con 0x7f6c0c0999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 0x7f6c0c12b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-b con 0x7f6c0c09c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): picked mon.noname-c con 0x7f6c0c137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c09eea0 con 0x7f6c0c0999e0 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c12eb70 con 0x7f6c0c09c410 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c12e890 con 0x7f6c0c137910 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c11fa5700 10 monclient(hunting): authenticate will time out at 1511.322040s 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c0999e0 0x7f6c0c12af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41996/0 (socket says 172.21.15.142:41996) 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 learned_addr learned my addr 172.21.15.142:0/3031502816 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): get_auth_request con 0x7f6c0c137910 auth_method 0 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request con 0x7f6c0c0999e0 auth_method 0 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 0x7f6c0c12b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 10 monclient(hunting): get_auth_request con 0x7f6c0c09c410 auth_method 0 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:22.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient(hunting): handle_auth_done global_id 75140 payload 824 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 msgr2=0x7f6c0c12b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 msgr2=0x7f6c0c12b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 0x7f6c0c12b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 msgr2=0x7f6c0c12b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 0x7f6c0c12b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c09c410 0x7f6c0c12b4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c0999e0 msgr2=0x7f6c0c12af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6c0c0999e0 0x7f6c0c12af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient: _finish_hunting 0 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 monclient: found mon.noname-c 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.394+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04009ae0 con 0x7f6c0c137910 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 msgr2=0x7f6c0c12ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c117a4700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c04017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04012b60 con 0x7f6c0c137910 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: dump: 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.413 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: _finish_auth 0 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 5 monclient: authenticate success, global_id 75140 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 monclient: _renew_subs 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c04012cc0 con 0x7f6c0c137910 2024-09-18T10:20:22.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c1366d0 con 0x7f6c0c137910 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0401d670 con 0x7f6c0c137910 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 10 monclient: dump: 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:22.415 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 monclient: _renew_subs 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c0affd700 4 set_mon_vals no callback set 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c12d650 con 0x7f6c0c137910 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6c0401daa0 con 0x7f6c0c137910 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c12e750 con 0x7f6c0c12bf40 2024-09-18T10:20:22.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 0x7f6bf8037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6c0404a5c0 con 0x7f6c0c137910 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c12bf40 auth_method 0 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 0x7f6bf8037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c0bfff700 10 monclient: get_auth_request con 0x7f6bf8035af0 auth_method 0 2024-09-18T10:20:22.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.402+0000 7f6c0bfff700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 0x7f6bf8037fa0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006f30 tx=0x7f6bfc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c12bf40 auth_method 0 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:22.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:22.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:22.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:22.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:22.606+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c12bf40 auth_method 0 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:23.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.010+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:20:23.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:23.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c12bf40 auth_method 0 2024-09-18T10:20:23.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:23.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:23.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:23.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:23.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:23.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:23.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:23.814+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:20:25.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.394+0000 7f6bf37fe700 10 monclient: tick 2024-09-18T10:20:25.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.394+0000 7f6bf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:25.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.394+0000 7f6bf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c12bf40 auth_method 0 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_done global_id 134098 payload 824 2024-09-18T10:20:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c0c072a80 con 0x7f6c0c12bf40 2024-09-18T10:20:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f6c00017980 tx=0x7f6c00009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:25.418+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-18T10:20:26.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 10 monclient: tick 2024-09-18T10:20:26.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:26.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 5 monclient: _check_tell_commands timeout tell command 1 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 0x7f6be4005a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 msgr2=0x7f6c0c06cac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c12bf40 0x7f6c0c06cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6bf37fe700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c12e750 con 0x7f6be4003590 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 0x7f6be4005a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6be4003590 auth_method 0 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_done global_id 134101 payload 824 2024-09-18T10:20:26.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.394+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be40060c0 con 0x7f6be4003590 2024-09-18T10:20:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 msgr2=0x7f6be4005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:26.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 0x7f6be4005a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015630 tx=0x7f6c00009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c00023840 con 0x7f6be4003590 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 10 monclient: dump: 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6c000239a0 con 0x7f6be4003590 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.398+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 msgr2=0x7f6be4005a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.402+0000 7f6c08ff9700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 0x7f6be4005a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015630 tx=0x7f6c00009040 comp rx=0 tx=0).stop 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.402+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 msgr2=0x7f6be4005a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:26.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.402+0000 7f6c08ff9700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be4003590 0x7f6be4005a10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c11fa5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:26.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 0x7f6c0c06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c11fa5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c0647f0 con 0x7f6c0c06d000 2024-09-18T10:20:26.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 0x7f6c0c06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient: get_auth_request con 0x7f6c0c06d000 auth_method 0 2024-09-18T10:20:26.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 10 monclient(hunting): handle_auth_done global_id 134104 payload 824 2024-09-18T10:20:26.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.418+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c0c09eb00 con 0x7f6c0c06d000 2024-09-18T10:20:26.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c10fa3700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 msgr2=0x7f6c0c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c10fa3700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 0x7f6c0c06d400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015630 tx=0x7f6c00023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c00033a50 con 0x7f6c0c06d000 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 10 monclient: dump: 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.427 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f6c00023630 con 0x7f6c0c06d000 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 msgr2=0x7f6c0c06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 0x7f6c0c06d400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6c00015630 tx=0x7f6c00023000 comp rx=0 tx=0).stop 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c117a4700 1 -- 172.21.15.142:0/3031502816 reap_dead start 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 msgr2=0x7f6c0c06d400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:26.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c08ff9700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c0c06d000 0x7f6c0c06d400 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:26.429 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:26.430 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:26.432 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:26.433 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:26.434 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.435 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.436 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.437 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:26.438 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:26.439 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:26.440 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 msgr2=0x7f6bf8037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 0x7f6bf8037fa0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006f30 tx=0x7f6bfc008040 comp rx=0 tx=0).stop 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 10 monclient: shutdown 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 msgr2=0x7f6c0c12ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c04017000 comp rx=0 tx=0).stop 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 shutdown_connections 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6bf8035af0 0x7f6bf8037fa0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 --2- 172.21.15.142:0/3031502816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6c0c137910 0x7f6c0c12ba00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 >> 172.21.15.142:0/3031502816 conn(0x7f6c0c0a5790 msgr2=0x7f6c0c136e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:26.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 shutdown_connections 2024-09-18T10:20:26.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.422+0000 7f6c11fa5700 1 -- 172.21.15.142:0/3031502816 wait complete. 2024-09-18T10:20:26.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:26.443 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:20:26.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:20:26.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: get_monmap_and_config 2024-09-18T10:20:26.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: build_initial_monmap 2024-09-18T10:20:26.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: monmap: 2024-09-18T10:20:26.554 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:26.552486+0000 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:26.552486+0000 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:26.555 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 Processor -- start 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 -- start start 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: init 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:20:26.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f999409aae0 0x7f9994096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-c con 0x7f999409aae0 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99940967a0 0x7f999409bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-a con 0x7f99940967a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f999409c320 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999409aeb0 con 0x7f999409c320 2024-09-18T10:20:26.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999406f6d0 con 0x7f99940967a0 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940955a0 con 0x7f999409aae0 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999ab91700 10 monclient(hunting): authenticate will time out at 1515.478378s 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99940967a0 0x7f999409bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999a390700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f999409c320 0x7f9994136590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47750/0 (socket says 172.21.15.142:47750) 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999a390700 1 -- 172.21.15.142:0/2612692448 learned_addr learned my addr 172.21.15.142:0/2612692448 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): get_auth_request con 0x7f99940967a0 auth_method 0 2024-09-18T10:20:26.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f9999b8f700 1 --2- 172.21.15.142:0/2612692448 >> v2:172.21.15.150:3300/0 conn(0x7f999409aae0 0x7f9994096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f9999b8f700 10 monclient(hunting): get_auth_request con 0x7f999409aae0 auth_method 0 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f9999b8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f9999b8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f9999b8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient(hunting): handle_auth_done global_id 124227 payload 274 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 -- 172.21.15.142:0/2612692448 >> v2:172.21.15.150:3300/0 conn(0x7f999409aae0 msgr2=0x7f9994096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:26.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 --2- 172.21.15.142:0/2612692448 >> v2:172.21.15.150:3300/0 conn(0x7f999409aae0 0x7f9994096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 -- 172.21.15.142:0/2612692448 >> v1:172.21.15.142:6789/0 conn(0x7f999409c320 legacy=0x7f9994136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient: _finish_hunting 0 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 monclient: found mon.noname-a 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.550+0000 7f999938e700 1 -- 172.21.15.142:0/2612692448 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994095b60 con 0x7f99940967a0 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999938e700 1 -- 172.21.15.142:0/2612692448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99940967a0 msgr2=0x7f999409bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999938e700 1 --2- 172.21.15.142:0/2612692448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99940967a0 0x7f999409bde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9994071c60 tx=0x7f99900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfccf4637250f4e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 1 -- 172.21.15.142:0/2612692448 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999000b8c0 con 0x7f99940967a0 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: dump: 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: _finish_auth 0 2024-09-18T10:20:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 5 monclient: authenticate success, global_id 124227 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 1 -- 172.21.15.142:0/2612692448 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999000ba20 con 0x7f99940967a0 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 4 set_mon_vals no callback set 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 1 -- 172.21.15.142:0/2612692448 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999000bdc0 con 0x7f99940967a0 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f9998b8d700 10 monclient: dump: 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 10 monclient: shutdown 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 -- 172.21.15.142:0/2612692448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99940967a0 msgr2=0x7f999409bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 --2- 172.21.15.142:0/2612692448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99940967a0 0x7f999409bde0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9994071c60 tx=0x7f99900091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 -- 172.21.15.142:0/2612692448 shutdown_connections 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 --2- 172.21.15.142:0/2612692448 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99940967a0 0x7f999409bde0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 --2- 172.21.15.142:0/2612692448 >> v2:172.21.15.150:3300/0 conn(0x7f999409aae0 0x7f9994096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 -- 172.21.15.142:0/2612692448 >> 172.21.15.142:0/2612692448 conn(0x7f99940a1fe0 msgr2=0x7f9994070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 -- 172.21.15.142:0/2612692448 shutdown_connections 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 1 -- 172.21.15.142:0/2612692448 wait complete. 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 10 monclient: build_initial_monmap 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.554+0000 7f999ab91700 10 monclient: monmap: 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 Processor -- start 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 -- start start 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient: init 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:20:26.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-c con 0x7f99940967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 0x7f999413fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-b con 0x7f999409aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 0x7f999408bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): picked mon.noname-a con 0x7f999409c320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999408db80 con 0x7f999409c320 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999408f0e0 con 0x7f999409aae0 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999406f8d0 con 0x7f99940967a0 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:26.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999ab91700 10 monclient(hunting): authenticate will time out at 1515.484937s 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42012/0 (socket says 172.21.15.142:42012) 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 -- 172.21.15.142:0/1504858048 learned_addr learned my addr 172.21.15.142:0/1504858048 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 0x7f999408bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): get_auth_request con 0x7f99940967a0 auth_method 0 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): get_auth_request con 0x7f999409c320 auth_method 0 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 0x7f999413fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 10 monclient(hunting): get_auth_request con 0x7f999409aae0 auth_method 0 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:26.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient(hunting): handle_auth_done global_id 75143 payload 824 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 -- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 msgr2=0x7f999413fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 0x7f999413fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 -- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 msgr2=0x7f999408bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 0x7f999408bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient: _finish_hunting 0 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 monclient: found mon.noname-c 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f9999b8f700 1 -- 172.21.15.142:0/1504858048 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990009420 con 0x7f99940967a0 2024-09-18T10:20:26.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999938e700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 0x7f999413fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.558+0000 7f999a390700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 0x7f999408bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9999b8f700 1 -- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 msgr2=0x7f999413f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9999b8f700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f998800d8c0 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988012800 con 0x7f99940967a0 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: dump: 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.576 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: _finish_auth 0 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9988012980 con 0x7f99940967a0 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 5 monclient: authenticate success, global_id 75143 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _renew_subs 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999406ec60 con 0x7f99940967a0 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988012d40 con 0x7f99940967a0 2024-09-18T10:20:26.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 10 monclient: dump: 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9987fff700 4 set_mon_vals no callback set 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _renew_subs 2024-09-18T10:20:26.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:26.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999408e1f0 con 0x7f99940967a0 2024-09-18T10:20:26.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:26.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:26.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9988014640 con 0x7f99940967a0 2024-09-18T10:20:26.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:26.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994072a80 con 0x7f99940967a0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 0x7f9978037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9988049600 con 0x7f99940967a0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999938e700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 0x7f9978037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.562+0000 7f999938e700 10 monclient: get_auth_request con 0x7f9978035ac0 auth_method 0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.566+0000 7f999938e700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 0x7f9978037f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f99900030c0 tx=0x7f9990016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.570+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f998801c360 con 0x7f99940967a0 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.570+0000 7f9985ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:26.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.570+0000 7f9985ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:26.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.798+0000 7f999ab91700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:20:26.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.798+0000 7f999ab91700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:20:26.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.798+0000 7f999ab91700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:26.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.798+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f99940678a0 con 0x7f99940967a0 2024-09-18T10:20:26.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f9985ffb700 1 -- 172.21.15.142:0/1504858048 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9988011340 con 0x7f99940967a0 2024-09-18T10:20:26.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f9985ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:20:26.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f9985ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=10,kv=8,logm=56} crc {auth=2949799219,config=3140514009,health=3459629563,kv=4261215878,logm=4153378258}) 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=8,logm=57} crc {auth=190127074,config=3140514009,health=3972976114,kv=4261215878,logm=3976201797}) 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=10,kv=8,logm=56} crc {auth=2949799219,config=3140514009,health=3459629563,kv=4261215878,logm=4153378258}) 2024-09-18T10:20:26.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=4283534384,config=3140514009,health=1675380437,kv=264744547,logm=775425776}) 2024-09-18T10:20:26.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 msgr2=0x7f9978037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 0x7f9978037f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f99900030c0 tx=0x7f9990016000 comp rx=0 tx=0).stop 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 10 monclient: shutdown 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 msgr2=0x7f999413f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f998800d8c0 tx=0x7f9988008040 comp rx=0 tx=0).stop 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 shutdown_connections 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9978035ac0 0x7f9978037f70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f999409c320 0x7f999408bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999409aae0 0x7f999413fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 --2- 172.21.15.142:0/1504858048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99940967a0 0x7f999413f7f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:26.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 >> 172.21.15.142:0/1504858048 conn(0x7f99940a1fe0 msgr2=0x7f999406c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:26.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.802+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 shutdown_connections 2024-09-18T10:20:26.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f999ab91700 1 -- 172.21.15.142:0/1504858048 wait complete. 2024-09-18T10:20:26.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621015319}) 2024-09-18T10:20:26.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1163327422}) 2024-09-18T10:20:26.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621015319}) 2024-09-18T10:20:26.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.806+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678736531}) 2024-09-18T10:20:26.816 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.816 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148648728}) 2024-09-18T10:20:26.816 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1291122522}) 2024-09-18T10:20:26.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148648728}) 2024-09-18T10:20:26.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.810+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470026275}) 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561248658}) 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515389287}) 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561248658}) 2024-09-18T10:20:26.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545883465}) 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086957884}) 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413287807}) 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086957884}) 2024-09-18T10:20:26.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1601124973}) 2024-09-18T10:20:26.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:20:26.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098591903}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561962836}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.814+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098591903}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551926354}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440402130}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3733234753}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440402130}) 2024-09-18T10:20:26.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=989689437}) 2024-09-18T10:20:26.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859341845}) 2024-09-18T10:20:26.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557134390}) 2024-09-18T10:20:26.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859341845}) 2024-09-18T10:20:26.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093390275}) 2024-09-18T10:20:26.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480869513}) 2024-09-18T10:20:26.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418261230}) 2024-09-18T10:20:26.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480869513}) 2024-09-18T10:20:26.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.818+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066532481}) 2024-09-18T10:20:26.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259097271}) 2024-09-18T10:20:26.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772113373}) 2024-09-18T10:20:26.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259097271}) 2024-09-18T10:20:26.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355136034}) 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109908739}) 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111429476}) 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2109908739}) 2024-09-18T10:20:26.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567453690}) 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458113495}) 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2934992289}) 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458113495}) 2024-09-18T10:20:26.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.822+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494330819}) 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806282563}) 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666240106}) 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=806282563}) 2024-09-18T10:20:26.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428353269}) 2024-09-18T10:20:26.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14161376}) 2024-09-18T10:20:26.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=67262969}) 2024-09-18T10:20:26.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14161376}) 2024-09-18T10:20:26.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106984353}) 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308284618}) 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461044939}) 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308284618}) 2024-09-18T10:20:26.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.826+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882185279}) 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957289954}) 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130587809}) 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957289954}) 2024-09-18T10:20:26.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713866995}) 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984806565}) 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3897223025}) 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984806565}) 2024-09-18T10:20:26.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344134634}) 2024-09-18T10:20:26.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812199114}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122820313}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812199114}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1982329832}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73206049}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622670962}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73206049}) 2024-09-18T10:20:26.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.830+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345344603}) 2024-09-18T10:20:26.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145077232}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275563847}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145077232}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3368741537}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160863157}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612313218}) 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160863157}) 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884250878}) 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570273389}) 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2418982824}) 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570273389}) 2024-09-18T10:20:26.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.834+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3633349564}) 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349049426}) 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476205704}) 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349049426}) 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3202505899}) 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207141124}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4167653043}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207141124}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=272084319}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833484443}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2739516587}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833484443}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835521903}) 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094984899}) 2024-09-18T10:20:26.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443415980}) 2024-09-18T10:20:26.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094984899}) 2024-09-18T10:20:26.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.838+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673061336}) 2024-09-18T10:20:26.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790824225}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1790780560}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790824225}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515130199}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311314249}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458748180}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311314249}) 2024-09-18T10:20:26.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562964678}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380930422}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832597180}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380930422}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803607584}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124693861}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1633778547}) 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124693861}) 2024-09-18T10:20:26.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.842+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126829385}) 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830847043}) 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230795231}) 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=830847043}) 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321631471}) 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818822610}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152583947}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818822610}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708515495}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754157460}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521967702}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754157460}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.846+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396362132}) 2024-09-18T10:20:26.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042943636}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1213373013}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042943636}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382275085}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485714433}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2424089563}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485714433}) 2024-09-18T10:20:26.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283606351}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999717761}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712426737}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999717761}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829787715}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427158959}) 2024-09-18T10:20:26.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377186515}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427158959}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.850+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573526291}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953788394}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213174986}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953788394}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696476735}) 2024-09-18T10:20:26.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529529522}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899938052}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529529522}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219997872}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818586618}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614594870}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818586618}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.854+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2624948350}) 2024-09-18T10:20:26.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727389961}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3632349808}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727389961}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702260766}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806743302}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882488197}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806743302}) 2024-09-18T10:20:26.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574421225}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339483088}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199637076}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339483088}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995373904}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661175530}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558409336}) 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661175530}) 2024-09-18T10:20:26.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.858+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=240949314}) 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272669645}) 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1264672214}) 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272669645}) 2024-09-18T10:20:26.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428525991}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987132283}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941852040}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987132283}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525912857}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840896933}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467634365}) 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840896933}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879522537}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617559303}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605428043}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617559303}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.862+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252814603}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=234121144}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709649}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=234121144}) 2024-09-18T10:20:26.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885484372}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227012103}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838849695}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227012103}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898092687}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150419455}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2151282266}) 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3150419455}) 2024-09-18T10:20:26.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.866+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2815839446}) 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443326138}) 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216899236}) 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443326138}) 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414703439}) 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371145640}) 2024-09-18T10:20:26.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167041405}) 2024-09-18T10:20:26.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371145640}) 2024-09-18T10:20:26.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.870+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2652199794}) 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492494329}) 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057221155}) 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492494329}) 2024-09-18T10:20:26.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875744222}) 2024-09-18T10:20:26.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353434160}) 2024-09-18T10:20:26.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3683289598}) 2024-09-18T10:20:26.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1353434160}) 2024-09-18T10:20:26.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.874+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807157645}) 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918988762}) 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003630667}) 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918988762}) 2024-09-18T10:20:26.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760643307}) 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487042792}) 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=187618668}) 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487042792}) 2024-09-18T10:20:26.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.878+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3175079404}) 2024-09-18T10:20:26.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706723279}) 2024-09-18T10:20:26.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770459132}) 2024-09-18T10:20:26.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.887 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706723279}) 2024-09-18T10:20:26.887 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2802237575}) 2024-09-18T10:20:26.887 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.887 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398003062}) 2024-09-18T10:20:26.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2769098871}) 2024-09-18T10:20:26.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398003062}) 2024-09-18T10:20:26.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548809351}) 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911318733}) 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978624366}) 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911318733}) 2024-09-18T10:20:26.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.882+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263995337}) 2024-09-18T10:20:26.890 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.890 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=2,mgr=2} crc {logm=1692167720,mdsmap=3140514009,mgr=829742442}) 2024-09-18T10:20:26.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=3189836629,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T10:20:26.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=2,mgr=2} crc {logm=1692167720,mdsmap=3140514009,mgr=829742442}) 2024-09-18T10:20:26.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2118004015,mdsmap=1531151339,mgr=3533299129}) 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=1391939807,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2632708035,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4123067486}) 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=68} crc {mgr=1391939807,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2632708035,mon_config_key=840343407,monmap=3505902565,osd_metadata=610694207,osd_pg_creating=69081760,osd_snap=454132730,osdmap=401020310}) 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=65} crc {mgr=1391939807,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2632708035,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4123067486}) 2024-09-18T10:20:26.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.886+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=66} crc {mgr=1391939807,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2632708035,mon_config_key=620546692,monmap=3505902565,osd_metadata=3227851384,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4101896303}) 2024-09-18T10:20:26.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=1989023578}) 2024-09-18T10:20:26.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=80} crc {osdmap=877648743}) 2024-09-18T10:20:26.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:20:26.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=1989023578}) 2024-09-18T10:20:26.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:20:26.894+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=81} crc {osdmap=3855492380}) 2024-09-18T10:20:27.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:20:27.823 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.930+0000 7fc6d2320700 10 monclient: get_monmap_and_config 2024-09-18T10:20:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.930+0000 7fc6d2320700 10 monclient: build_initial_monmap 2024-09-18T10:20:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.930+0000 7fc6d2320700 10 monclient: monmap: 2024-09-18T10:20:27.937 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:27.935696+0000 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:27.935696+0000 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:27.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 Processor -- start 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 -- start start 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient: init 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6cc09aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:27.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc6cc132570 0x7fc6cc134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc132570 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6cc134e90 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc09aee0 con 0x7fc6cc134e90 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc06f6d0 con 0x7fc6cc09aae0 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc0955a0 con 0x7fc6cc132570 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d2320700 10 monclient(hunting): authenticate will time out at 1516.861207s 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d1b1f700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc6cc134e90 0x7fc6cc137270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47762/0 (socket says 172.21.15.142:47762) 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d1b1f700 1 -- 172.21.15.142:0/3837842193 learned_addr learned my addr 172.21.15.142:0/3837842193 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 1 --2- 172.21.15.142:0/3837842193 >> v2:172.21.15.150:3300/0 conn(0x7fc6cc132570 0x7fc6cc134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): get_auth_request con 0x7fc6cc132570 auth_method 0 2024-09-18T10:20:27.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_done global_id 134113 payload 274 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 1 -- 172.21.15.142:0/3837842193 >> v1:172.21.15.150:6789/0 conn(0x7fc6cc09aae0 legacy=0x7fc6cc096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 1 -- 172.21.15.142:0/3837842193 >> v1:172.21.15.142:6789/0 conn(0x7fc6cc134e90 legacy=0x7fc6cc137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient: _finish_hunting 0 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 1 monclient: found mon.noname-c 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:20:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.934+0000 7fc6d0b1d700 1 -- 172.21.15.142:0/3837842193 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc095b60 con 0x7fc6cc132570 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d0b1d700 1 -- 172.21.15.142:0/3837842193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc132570 msgr2=0x7fc6cc134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d0b1d700 1 --2- 172.21.15.142:0/3837842193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc132570 0x7fc6cc134950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009e60 tx=0x7fc6c4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26a9d28f13d6b54b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 1 -- 172.21.15.142:0/3837842193 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400e070 con 0x7fc6cc132570 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: dump: 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: _finish_auth 0 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 1 -- 172.21.15.142:0/3837842193 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4012430 con 0x7fc6cc132570 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 5 monclient: authenticate success, global_id 134113 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:27.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 1 -- 172.21.15.142:0/3837842193 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40127d0 con 0x7fc6cc132570 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 4 set_mon_vals no callback set 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6cbfff700 10 monclient: dump: 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:27.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 10 monclient: shutdown 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 -- 172.21.15.142:0/3837842193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc132570 msgr2=0x7fc6cc134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 --2- 172.21.15.142:0/3837842193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc132570 0x7fc6cc134950 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009e60 tx=0x7fc6c4009780 comp rx=0 tx=0).stop 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 -- 172.21.15.142:0/3837842193 shutdown_connections 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 --2- 172.21.15.142:0/3837842193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc132570 0x7fc6cc134950 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 -- 172.21.15.142:0/3837842193 >> 172.21.15.142:0/3837842193 conn(0x7fc6cc0a1fe0 msgr2=0x7fc6cc070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 -- 172.21.15.142:0/3837842193 shutdown_connections 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 -- 172.21.15.142:0/3837842193 wait complete. 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 10 monclient: build_initial_monmap 2024-09-18T10:20:27.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 10 monclient: monmap: 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.938+0000 7fc6d2320700 1 Processor -- start 2024-09-18T10:20:27.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 -- start start 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient: init 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-a con 0x7fc6cc09aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc132570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc134e90 0x7fc6cc099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): picked mon.noname-b con 0x7fc6cc134e90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc06f310 con 0x7fc6cc09aae0 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:27.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc06d090 con 0x7fc6cc134e90 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc09f260 con 0x7fc6cc132570 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d2320700 10 monclient(hunting): authenticate will time out at 1516.868707s 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42018/0 (socket says 172.21.15.142:42018) 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 1 -- 172.21.15.142:0/1520835105 learned_addr learned my addr 172.21.15.142:0/1520835105 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): get_auth_request con 0x7fc6cc132570 auth_method 0 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60936/0 (socket says 172.21.15.142:60936) 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:27.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): get_auth_request con 0x7fc6cc09aae0 auth_method 0 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc134e90 0x7fc6cc099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:27.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): get_auth_request con 0x7fc6cc134e90 auth_method 0 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d1b1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient(hunting): handle_auth_done global_id 124233 payload 824 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 -- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 msgr2=0x7fc6cc099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 -- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc134e90 msgr2=0x7fc6cc099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc134e90 0x7fc6cc099580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:27.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient: _finish_hunting 0 2024-09-18T10:20:27.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 monclient: found mon.noname-a 2024-09-18T10:20:27.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:27.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:27.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 -- 172.21.15.142:0/1520835105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4009a20 con 0x7fc6cc09aae0 2024-09-18T10:20:27.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d0b1d700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 -- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 msgr2=0x7fc6cc098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.942+0000 7fc6d131e700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c4e0 tx=0x7fc6c0007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000ce90 con 0x7fc6cc09aae0 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: dump: 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.959 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: _finish_auth 0 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 5 monclient: authenticate success, global_id 124233 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c00117a0 con 0x7fc6cc09aae0 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:27.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _renew_subs 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0011b40 con 0x7fc6cc09aae0 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6cb7fe700 4 set_mon_vals no callback set 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc131140 con 0x7fc6cc09aae0 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 10 monclient: dump: 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:27.961 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _renew_subs 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc09a4c0 con 0x7fc6cc09aae0 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc6c001c460 con 0x7fc6cc09aae0 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:27.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc1369c0 con 0x7fc6cc09aae0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 0x7fc6b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc6c0048b60 con 0x7fc6cc09aae0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d0b1d700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 0x7fc6b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.946+0000 7fc6d0b1d700 10 monclient: get_auth_request con 0x7fc6b8035ac0 auth_method 0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.950+0000 7fc6d0b1d700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 0x7fc6b8037f70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc09a2f0 tx=0x7fc6c4018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.954+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc6c0002090 con 0x7fc6cc09aae0 2024-09-18T10:20:27.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.954+0000 7fc6c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:27.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:27.954+0000 7fc6c97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6d2320700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:28.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6d2320700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6d2320700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc6cc072a80 con 0x7fc6cc09aae0 2024-09-18T10:20:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6c97fa700 1 -- 172.21.15.142:0/1520835105 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fc6c00102e0 con 0x7fc6cc09aae0 2024-09-18T10:20:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:28.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.178+0000 7fc6c97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:28.186 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 msgr2=0x7fc6b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 0x7fc6b8037f70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc09a2f0 tx=0x7fc6c4018000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 10 monclient: shutdown 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 msgr2=0x7fc6cc098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c4e0 tx=0x7fc6c0007ea0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 shutdown_connections 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc6b8035ac0 0x7fc6b8037f70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6cc134e90 0x7fc6cc099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6cc132570 0x7fc6cc099040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 --2- 172.21.15.142:0/1520835105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6cc09aae0 0x7fc6cc098b00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 >> 172.21.15.142:0/1520835105 conn(0x7fc6cc0a1fe0 msgr2=0x7fc6cc131d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 shutdown_connections 2024-09-18T10:20:28.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.182+0000 7fc6d2320700 1 -- 172.21.15.142:0/1520835105 wait complete. 2024-09-18T10:20:28.204 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:20:28.204 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:20:28.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:20:28.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.302+0000 7f702351d700 10 monclient: get_monmap_and_config 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.302+0000 7f702351d700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.302+0000 7f702351d700 10 monclient: monmap: 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:28.307401+0000 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:28.307401+0000 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:28.310 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 Processor -- start 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 -- start start 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient: init 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f701c09d870 0x7f701c0954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): picked mon.noname-a con 0x7f701c09d870 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f701c0989c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:28.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c0994d0 0x7f701c136680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): picked mon.noname-b con 0x7f701c0994d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c069c70 con 0x7f701c0989c0 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c06f9d0 con 0x7f701c09d870 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c09dc40 con 0x7f701c0994d0 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f702351d700 10 monclient(hunting): authenticate will time out at 1517.233115s 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f701c09d870 0x7f701c0954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c0994d0 0x7f701c136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f701c09d870 0x7f701c0954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60948/0 (socket says 172.21.15.142:60948) 2024-09-18T10:20:28.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 -- 172.21.15.142:0/1223857823 learned_addr learned my addr 172.21.15.142:0/1223857823 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): get_auth_request con 0x7f701c09d870 auth_method 0 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): get_auth_request con 0x7f701c0994d0 auth_method 0 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient(hunting): handle_auth_done global_id 124236 payload 274 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 -- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c0994d0 msgr2=0x7f701c136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 --2- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c0994d0 0x7f701c136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 -- 172.21.15.142:0/1223857823 >> v1:172.21.15.150:6789/0 conn(0x7f701c0989c0 legacy=0x7f701c098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.306+0000 7f70212b9700 1 -- 172.21.15.142:0/1223857823 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c06fc20 con 0x7f701c09d870 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f70212b9700 1 -- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c09d870 msgr2=0x7f701c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f70212b9700 1 --2- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c09d870 0x7f701c0954b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70180099b0 tx=0x7f7018009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d5c80a932b9ad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 1 -- 172.21.15.142:0/1223857823 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018009ea0 con 0x7f701c09d870 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: dump: 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 5 monclient: authenticate success, global_id 124236 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 1 -- 172.21.15.142:0/1223857823 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701800ba70 con 0x7f701c09d870 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 4 set_mon_vals no callback set 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 1 -- 172.21.15.142:0/1223857823 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701800be30 con 0x7f701c09d870 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f7013fff700 10 monclient: dump: 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: shutdown 2024-09-18T10:20:28.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c09d870 msgr2=0x7f701c0954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 --2- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c09d870 0x7f701c0954b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70180099b0 tx=0x7f7018009250 comp rx=0 tx=0).stop 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- 172.21.15.142:0/1223857823 shutdown_connections 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 --2- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c0994d0 0x7f701c136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 --2- 172.21.15.142:0/1223857823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c09d870 0x7f701c0954b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- 172.21.15.142:0/1223857823 >> 172.21.15.142:0/1223857823 conn(0x7f701c0a14e0 msgr2=0x7f701c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- 172.21.15.142:0/1223857823 shutdown_connections 2024-09-18T10:20:28.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- 172.21.15.142:0/1223857823 wait complete. 2024-09-18T10:20:28.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: monmap: 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 Processor -- start 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 -- start start 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: init 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701c0989c0 0x7f701c13e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 10 monclient(hunting): picked mon.noname-b con 0x7f701c0989c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.310+0000 7f702351d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 0x7f701c13eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): picked mon.noname-a con 0x7f701c0994d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c09d870 0x7f701c08aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): picked mon.noname-c con 0x7f701c09d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c08e7c0 con 0x7f701c0994d0 2024-09-18T10:20:28.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c097330 con 0x7f701c0989c0 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c08e4e0 con 0x7f701c09d870 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient(hunting): authenticate will time out at 1517.240114s 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 0x7f701c13eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 0x7f701c13eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60950/0 (socket says 172.21.15.142:60950) 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 -- 172.21.15.142:0/194030388 learned_addr learned my addr 172.21.15.142:0/194030388 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): get_auth_request con 0x7f701c0994d0 auth_method 0 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c09d870 0x7f701c08aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): get_auth_request con 0x7f701c09d870 auth_method 0 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701c0989c0 0x7f701c13e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7021aba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 10 monclient(hunting): get_auth_request con 0x7f701c0989c0 auth_method 0 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient(hunting): handle_auth_done global_id 124239 payload 824 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c09d870 msgr2=0x7f701c08aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f701c09d870 0x7f701c08aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701c0989c0 msgr2=0x7f701c13e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701c0989c0 0x7f701c13e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70180094f0 con 0x7f701c0994d0 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701c0989c0 0x7f701c13e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:20:28.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 msgr2=0x7f701c13eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7020ab8700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 0x7f701c13eea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f700c00c600 tx=0x7f700c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c0117a0 con 0x7f701c0994d0 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: dump: 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.333 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 5 monclient: authenticate success, global_id 124239 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c011920 con 0x7f701c0994d0 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c011ce0 con 0x7f701c0994d0 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient: _renew_subs 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c08df10 con 0x7f701c0994d0 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f7012ffd700 4 set_mon_vals no callback set 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f70117fa700 10 monclient: dump: 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient: _renew_subs 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.314+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c08c6d0 con 0x7f701c0994d0 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f702351d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f702351d700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 0x7f701c08b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c064950 con 0x7f701c08b0e0 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 0x7f701c08b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f700c012630 con 0x7f701c0994d0 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient: get_auth_request con 0x7f701c08b0e0 auth_method 0 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70117fa700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7000035ac0 0x7f7000037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f700c048d70 con 0x7f701c0994d0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 10 monclient(hunting): handle_auth_done global_id 124242 payload 824 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f70212b9700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f701c0696a0 con 0x7f701c08b0e0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f7021aba700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7000035ac0 0x7f7000037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.318+0000 7f7021aba700 10 monclient: get_auth_request con 0x7f7000035ac0 auth_method 0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70212b9700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 msgr2=0x7f701c08b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 0x7f701c08b4b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7018015000 tx=0x7f7018016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018011e00 con 0x7f701c08b0e0 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 10 monclient: dump: 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.337 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f7021aba700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7000035ac0 0x7f7000037f70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7014009fd0 tx=0x7f701400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f701801b470 con 0x7f701c08b0e0 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 msgr2=0x7f701c08b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 0x7f701c08b4b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7018015000 tx=0x7f7018016000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 msgr2=0x7f701c08b4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.322+0000 7f70117fa700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b0e0 0x7f701c08b4b0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7018015000 tx=0x7f7018016000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f702351d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f702351d700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 0x7f701c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f702351d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c08e930 con 0x7f701c08b9f0 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 0x7f701c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient: get_auth_request con 0x7f701c08b9f0 auth_method 0 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 10 monclient(hunting): handle_auth_done global_id 124245 payload 824 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.330+0000 7f70212b9700 1 -- 172.21.15.142:0/194030388 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f701c08d2a0 con 0x7f701c08b9f0 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70212b9700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 msgr2=0x7f701c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70212b9700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 0x7f701c06d410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7018003080 tx=0x7f701801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018037070 con 0x7f701c08b9f0 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 10 monclient: dump: 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f7018033420 con 0x7f701c08b9f0 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 msgr2=0x7f701c06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 0x7f701c06d410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7018003080 tx=0x7f701801f040 comp rx=0 tx=0).stop 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 msgr2=0x7f701c06d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f70117fa700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c08b9f0 0x7f701c06d410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:20:28.344 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:28.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:28.346 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:28.347 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:28.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:28.352 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.353 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:28.355 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:28.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7000035ac0 msgr2=0x7f7000037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7000035ac0 0x7f7000037f70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7014009fd0 tx=0x7f701400b040 comp rx=0 tx=0).stop 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 10 monclient: shutdown 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 msgr2=0x7f701c13eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 --2- 172.21.15.142:0/194030388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f701c0994d0 0x7f701c13eea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f700c00c600 tx=0x7f700c007f60 comp rx=0 tx=0).stop 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f7021aba700 1 -- 172.21.15.142:0/194030388 reap_dead start 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f7021aba700 1 -- 172.21.15.142:0/194030388 reap_dead start 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 shutdown_connections 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 >> 172.21.15.142:0/194030388 conn(0x7f701c0a14e0 msgr2=0x7f701c0a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 shutdown_connections 2024-09-18T10:20:28.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.334+0000 7f702351d700 1 -- 172.21.15.142:0/194030388 wait complete. 2024-09-18T10:20:28.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:28.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:20:28.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:20:28.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: get_monmap_and_config 2024-09-18T10:20:28.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: monmap: 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:28.469828+0000 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:28.469828+0000 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:28.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 Processor -- start 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 -- start start 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: init 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f711406e770 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): picked mon.noname-b con 0x7f711412ebe0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71141371f0 0x7f71141395d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): picked mon.noname-a con 0x7f71141371f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711412f530 con 0x7f711406e770 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711409dc40 con 0x7f71141371f0 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f711406f2e0 con 0x7f711412ebe0 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711912f700 10 monclient(hunting): authenticate will time out at 1517.395629s 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711892e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71141371f0 0x7f71141395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42038/0 (socket says 172.21.15.142:42038) 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 1 -- 172.21.15.142:0/1434930447 learned_addr learned my addr 172.21.15.142:0/1434930447 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 10 monclient(hunting): get_auth_request con 0x7f711412ebe0 auth_method 0 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f71137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711892e700 10 monclient(hunting): get_auth_request con 0x7f71141371f0 auth_method 0 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711892e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711892e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.466+0000 7f711892e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient(hunting): handle_auth_done global_id 124248 payload 274 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 -- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 msgr2=0x7f711412eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 -- 172.21.15.142:0/1434930447 >> v1:172.21.15.150:6789/0 conn(0x7f711406e770 legacy=0x7f711409b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 -- 172.21.15.142:0/1434930447 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711406eb40 con 0x7f71141371f0 2024-09-18T10:20:28.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f71137fe700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:28.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 -- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71141371f0 msgr2=0x7f71141395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711892e700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71141371f0 0x7f71141395d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f710c009e60 tx=0x7f710c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be39d450945fa2bb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 1 -- 172.21.15.142:0/1434930447 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c00e070 con 0x7f71141371f0 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: dump: 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.484 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711912f700 5 monclient: authenticate success, global_id 124248 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711912f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 1 -- 172.21.15.142:0/1434930447 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c00bd70 con 0x7f71141371f0 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711912f700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f711912f700 4 set_mon_vals no callback set 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 1 -- 172.21.15.142:0/1434930447 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c012660 con 0x7f71141371f0 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.470+0000 7f7112ffd700 10 monclient: dump: 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: shutdown 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71141371f0 msgr2=0x7f71141395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71141371f0 0x7f71141395d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f710c009e60 tx=0x7f710c009780 comp rx=0 tx=0).stop 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- 172.21.15.142:0/1434930447 shutdown_connections 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71141371f0 0x7f71141395d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- 172.21.15.142:0/1434930447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711412ebe0 0x7f711412eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- 172.21.15.142:0/1434930447 >> 172.21.15.142:0/1434930447 conn(0x7f7114070ef0 msgr2=0x7f71140712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- 172.21.15.142:0/1434930447 shutdown_connections 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- 172.21.15.142:0/1434930447 wait complete. 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: monmap: 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.487 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 Processor -- start 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- start start 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: init 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711406e770 0x7f7114132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): picked mon.noname-c con 0x7f711406e770 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): picked mon.noname-a con 0x7f711412ebe0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71141371f0 0x7f7114133920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): picked mon.noname-b con 0x7f71141371f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71141357a0 con 0x7f711412ebe0 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7114135430 con 0x7f71141371f0 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7114132ad0 con 0x7f711406e770 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f711912f700 10 monclient(hunting): authenticate will time out at 1517.403743s 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60984/0 (socket says 172.21.15.142:60984) 2024-09-18T10:20:28.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f7113fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711406e770 0x7f7114132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 1 -- 172.21.15.142:0/1000903653 learned_addr learned my addr 172.21.15.142:0/1000903653 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 10 monclient(hunting): get_auth_request con 0x7f711412ebe0 auth_method 0 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f71137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f7113fff700 10 monclient(hunting): get_auth_request con 0x7f711406e770 auth_method 0 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f7113fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f7113fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.474+0000 7f7113fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711892e700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71141371f0 0x7f7114133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711892e700 10 monclient(hunting): get_auth_request con 0x7f71141371f0 auth_method 0 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711892e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711892e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711892e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient(hunting): handle_auth_done global_id 124251 payload 824 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711406e770 msgr2=0x7f7114132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f711406e770 0x7f7114132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71141371f0 msgr2=0x7f7114133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71141371f0 0x7f7114133920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71040094f0 con 0x7f711412ebe0 2024-09-18T10:20:28.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 msgr2=0x7f71141333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71137fe700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f710400ec50 tx=0x7f7104009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71040097d0 con 0x7f711412ebe0 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: dump: 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.498 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710400cb30 con 0x7f711412ebe0 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 5 monclient: authenticate success, global_id 124251 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 10 monclient: _renew_subs 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c009a20 con 0x7f711412ebe0 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104013470 con 0x7f711412ebe0 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f70fbfff700 10 monclient: dump: 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 10 monclient: _renew_subs 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f71127fc700 4 set_mon_vals no callback set 2024-09-18T10:20:28.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.478+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71141363e0 con 0x7f711412ebe0 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711912f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7104013820 con 0x7f711412ebe0 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711912f700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 0x7f7114096720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71140648a0 con 0x7f7114096350 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 0x7f7114096720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f70fbfff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7100035ac0 0x7f7100037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient: get_auth_request con 0x7f7114096350 auth_method 0 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7104049730 con 0x7f711412ebe0 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711892e700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7100035ac0 0x7f7100037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f711892e700 10 monclient: get_auth_request con 0x7f7100035ac0 auth_method 0 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 10 monclient(hunting): handle_auth_done global_id 75161 payload 824 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7114072a80 con 0x7f7114096350 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 msgr2=0x7f7114096720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.482+0000 7f7113fff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 0x7f7114096720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7108014890 con 0x7f7114096350 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 10 monclient: dump: 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f711892e700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7100035ac0 0x7f7100037f70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f71140a0c80 tx=0x7f710c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f71080149f0 con 0x7f7114096350 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 msgr2=0x7f7114096720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 0x7f7114096720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108007ea0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 msgr2=0x7f7114096720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.486+0000 7f70fbfff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096350 0x7f7114096720 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108007ea0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f711912f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f711912f700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 0x7f711414b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f711912f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7114135150 con 0x7f7114096c60 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 0x7f711414b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient: get_auth_request con 0x7f7114096c60 auth_method 0 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 10 monclient(hunting): handle_auth_done global_id 75164 payload 824 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.494+0000 7f7113fff700 1 -- 172.21.15.142:0/1000903653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7114135910 con 0x7f7114096c60 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f7113fff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 msgr2=0x7f711414b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f7113fff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 0x7f711414b6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710801f940 con 0x7f7114096c60 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 10 monclient: dump: 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.507 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f7108024070 con 0x7f7114096c60 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 msgr2=0x7f711414b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 0x7f711414b6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108019000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 msgr2=0x7f711414b6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f70fbfff700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7114096c60 0x7f711414b6b0 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f710800f7b0 tx=0x7f7108019000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.509 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 7, 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.510 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.511 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:28.512 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.513 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:28.515 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.516 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.518 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.519 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.520 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:28.521 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7100035ac0 msgr2=0x7f7100037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7100035ac0 0x7f7100037f70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f71140a0c80 tx=0x7f710c018000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 10 monclient: shutdown 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 msgr2=0x7f71141333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f710400ec50 tx=0x7f7104009250 comp rx=0 tx=0).stop 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711892e700 1 -- 172.21.15.142:0/1000903653 reap_dead start 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 shutdown_connections 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 --2- 172.21.15.142:0/1000903653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f711412ebe0 0x7f71141333e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 >> 172.21.15.142:0/1000903653 conn(0x7f7114070ef0 msgr2=0x7f711412c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 shutdown_connections 2024-09-18T10:20:28.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.498+0000 7f711912f700 1 -- 172.21.15.142:0/1000903653 wait complete. 2024-09-18T10:20:28.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:28.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:20:28.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:20:28.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: get_monmap_and_config 2024-09-18T10:20:28.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: monmap: 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:28.637596+0000 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:28.637596+0000 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:28.738 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 Processor -- start 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 -- start start 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: init 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba3406b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-b con 0x7fba3409ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba3409cdd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fba3409b940 0x7fba341398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-a con 0x7fba3409b940 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba34064a30 con 0x7fba3409cdd0 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3406b660 con 0x7fba3409b940 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3409cb90 con 0x7fba3409ae60 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3912d700 10 monclient(hunting): authenticate will time out at 1517.563375s 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fba3409b940 0x7fba341398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fba3409b940 0x7fba341398f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60996/0 (socket says 172.21.15.142:60996) 2024-09-18T10:20:28.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 1 -- 172.21.15.142:0/198716611 learned_addr learned my addr 172.21.15.142:0/198716611 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): get_auth_request con 0x7fba3409b940 auth_method 0 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba33fff700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba3406b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba33fff700 10 monclient(hunting): get_auth_request con 0x7fba3409ae60 auth_method 0 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.634+0000 7fba3892c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 10 monclient(hunting): handle_auth_done global_id 124254 payload 274 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 -- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 msgr2=0x7fba3406b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba3406b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 -- 172.21.15.142:0/198716611 >> v1:172.21.15.150:6789/0 conn(0x7fba3409cdd0 legacy=0x7fba3409b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 -- 172.21.15.142:0/198716611 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3409e2d0 con 0x7fba3409b940 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba33fff700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba3406b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:28.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 -- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 msgr2=0x7fba341398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3892c700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba341398f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba240099b0 tx=0x7fba24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3afe6aef3e807af server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 1 -- 172.21.15.142:0/198716611 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24009ea0 con 0x7fba3409b940 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: dump: 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.754 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 5 monclient: authenticate success, global_id 124254 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 1 -- 172.21.15.142:0/198716611 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2400ba70 con 0x7fba3409b940 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 4 set_mon_vals no callback set 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 1 -- 172.21.15.142:0/198716611 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2400be30 con 0x7fba3409b940 2024-09-18T10:20:28.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba32ffd700 10 monclient: dump: 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 10 monclient: shutdown 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 -- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 msgr2=0x7fba341398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba341398f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fba240099b0 tx=0x7fba24009250 comp rx=0 tx=0).stop 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 -- 172.21.15.142:0/198716611 shutdown_connections 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba341398f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 --2- 172.21.15.142:0/198716611 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba3406b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 -- 172.21.15.142:0/198716611 >> 172.21.15.142:0/198716611 conn(0x7fba34071ae0 msgr2=0x7fba340a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.638+0000 7fba3912d700 1 -- 172.21.15.142:0/198716611 shutdown_connections 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 -- 172.21.15.142:0/198716611 wait complete. 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient: monmap: 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 Processor -- start 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 -- start start 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient: init 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba341397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-c con 0x7fba3409ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba3411f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-a con 0x7fba3409b940 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3409cdd0 0x7fba3411f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): picked mon.noname-b con 0x7fba3409cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3409db70 con 0x7fba3409b940 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3406e3b0 con 0x7fba3409cdd0 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba341226d0 con 0x7fba3409ae60 2024-09-18T10:20:28.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3912d700 10 monclient(hunting): authenticate will time out at 1517.570498s 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba341397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba3411f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba3411f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32772/0 (socket says 172.21.15.142:32772) 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 -- 172.21.15.142:0/3031468139 learned_addr learned my addr 172.21.15.142:0/3031468139 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): get_auth_request con 0x7fba3409b940 auth_method 0 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): get_auth_request con 0x7fba3409ae60 auth_method 0 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3892c700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3409cdd0 0x7fba3411f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3892c700 10 monclient(hunting): get_auth_request con 0x7fba3409cdd0 auth_method 0 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3892c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3892c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba3892c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient(hunting): handle_auth_done global_id 124257 payload 824 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 msgr2=0x7fba341397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba3409ae60 0x7fba341397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3409cdd0 msgr2=0x7fba3411f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3409cdd0 0x7fba3411f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 monclient: found mon.noname-a 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.642+0000 7fba337fe700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c009a20 con 0x7fba3409b940 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba337fe700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 msgr2=0x7fba3411f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba337fe700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba3411f0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c5d0 tx=0x7fba1c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c0117a0 con 0x7fba3409b940 2024-09-18T10:20:28.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: dump: 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 5 monclient: authenticate success, global_id 124257 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c011900 con 0x7fba3409b940 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c011cc0 con 0x7fba3409b940 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 10 monclient: dump: 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.764 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba31ffb700 4 set_mon_vals no callback set 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 monclient: _renew_subs 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba240094f0 con 0x7fba3409b940 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 monclient: _renew_subs 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba34120c10 con 0x7fba3409b940 2024-09-18T10:20:28.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fba1c012630 con 0x7fba3409b940 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 0x7fba3402ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34064680 con 0x7fba3411fb40 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba14035ac0 0x7fba14037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.646+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fba1c048cf0 con 0x7fba3409b940 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 0x7fba3402ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba3892c700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba14035ac0 0x7fba14037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba3892c700 10 monclient: get_auth_request con 0x7fba14035ac0 auth_method 0 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient: get_auth_request con 0x7fba3411fb40 auth_method 0 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 10 monclient(hunting): handle_auth_done global_id 134128 payload 824 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3411ff10 con 0x7fba3411fb40 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba3892c700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba14035ac0 0x7fba14037f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fba240099b0 tx=0x7fba24016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 msgr2=0x7fba3402ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba33fff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 0x7fba3402ae80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00e040 con 0x7fba3411fb40 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba2bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba2bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.650+0000 7fba2bfff700 10 monclient: dump: 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.767 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fba2c00f040 con 0x7fba3411fb40 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 msgr2=0x7fba3402ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 0x7fba3402ae80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c00b040 comp rx=0 tx=0).stop 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 msgr2=0x7fba3402ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.654+0000 7fba2bfff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3411fb40 0x7fba3402ae80 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c00b040 comp rx=0 tx=0).stop 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba3912d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba3912d700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 0x7fba3405ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba3912d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3411b9e0 con 0x7fba3405e890 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 0x7fba3405ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient: get_auth_request con 0x7fba3405e890 auth_method 0 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.662+0000 7fba33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba33fff700 10 monclient(hunting): handle_auth_done global_id 134131 payload 824 2024-09-18T10:20:28.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba33fff700 1 -- 172.21.15.142:0/3031468139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fba34072a80 con 0x7fba3405e890 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba33fff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 msgr2=0x7fba3405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba33fff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 0x7fba3405ec90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c021650 con 0x7fba3405e890 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 10 monclient: dump: 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.770 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fba2c0217b0 con 0x7fba3405e890 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 msgr2=0x7fba3405ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 0x7fba3405ec90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c01b000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 msgr2=0x7fba3405ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.666+0000 7fba2bfff700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3405e890 0x7fba3405ec90 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00f3e0 tx=0x7fba2c01b000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.771 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 116, 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 8, 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.772 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:28.773 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:28.774 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:28.775 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.776 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.777 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.778 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.779 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:28.780 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:28.781 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:28.782 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:28.783 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:28.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba14035ac0 msgr2=0x7fba14037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba14035ac0 0x7fba14037f70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fba240099b0 tx=0x7fba24016000 comp rx=0 tx=0).stop 2024-09-18T10:20:28.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 10 monclient: shutdown 2024-09-18T10:20:28.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 msgr2=0x7fba3411f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 --2- 172.21.15.142:0/3031468139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba3409b940 0x7fba3411f0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c5d0 tx=0x7fba1c007f60 comp rx=0 tx=0).stop 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3892c700 1 -- 172.21.15.142:0/3031468139 reap_dead start 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3892c700 1 -- 172.21.15.142:0/3031468139 reap_dead start 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 shutdown_connections 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 >> 172.21.15.142:0/3031468139 conn(0x7fba34071ae0 msgr2=0x7fba3406c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 shutdown_connections 2024-09-18T10:20:28.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.670+0000 7fba3912d700 1 -- 172.21.15.142:0/3031468139 wait complete. 2024-09-18T10:20:28.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:20:28.786 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:20:28.786 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:20:28.786 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:20:28.786 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:20:28.786 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:20:28.786 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:20:28.803 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:20:28.803 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:20:28.803 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:28.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.906+0000 7fec7d4a8700 10 monclient: get_monmap_and_config 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.906+0000 7fec7d4a8700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient: monmap: 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:28.911735+0000 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:28.911735+0000 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:28.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 Processor -- start 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 -- start start 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient: init 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 0x7fec780953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-c con 0x7fec7809b050 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-a con 0x7fec78097970 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 0x7fec7809c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-b con 0x7fec7809bde0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7809a570 con 0x7fec78097970 2024-09-18T10:20:28.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7806f900 con 0x7fec7809b050 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7809b420 con 0x7fec7809bde0 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7d4a8700 10 monclient(hunting): authenticate will time out at 1517.837301s 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec77fff700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 msgr2=0x7fec780953c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec77fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 0x7fec780953c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 0x7fec7809c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32780/0 (socket says 172.21.15.142:32780) 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 1 -- 172.21.15.142:0/3155395799 learned_addr learned my addr 172.21.15.142:0/3155395799 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): get_auth_request con 0x7fec78097970 auth_method 0 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): get_auth_request con 0x7fec7809bde0 auth_method 0 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient(hunting): handle_auth_done global_id 75173 payload 274 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 -- 172.21.15.142:0/3155395799 >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 msgr2=0x7fec780953c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 --2- 172.21.15.142:0/3155395799 >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 0x7fec780953c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 -- 172.21.15.142:0/3155395799 >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 msgr2=0x7fec78097d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 --2- 172.21.15.142:0/3155395799 >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 monclient: found mon.noname-b 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec7cca7700 1 -- 172.21.15.142:0/3155395799 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7809a290 con 0x7fec7809bde0 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.910+0000 7fec777fe700 1 --2- 172.21.15.142:0/3155395799 >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7cca7700 1 -- 172.21.15.142:0/3155395799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 msgr2=0x7fec7809c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7cca7700 1 --2- 172.21.15.142:0/3155395799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 0x7fec7809c1d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fec78067d40 tx=0x7fec6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d493b2c50ac1da server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 1 -- 172.21.15.142:0/3155395799 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c011840 con 0x7fec7809bde0 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: dump: 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 5 monclient: authenticate success, global_id 75173 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 1 -- 172.21.15.142:0/3155395799 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c0119c0 con 0x7fec7809bde0 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 4 set_mon_vals no callback set 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 1 -- 172.21.15.142:0/3155395799 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c011d80 con 0x7fec7809bde0 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec76ffd700 10 monclient: dump: 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.925 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 10 monclient: shutdown 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 -- 172.21.15.142:0/3155395799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 msgr2=0x7fec7809c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/3155395799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 0x7fec7809c1d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fec78067d40 tx=0x7fec6c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 -- 172.21.15.142:0/3155395799 shutdown_connections 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/3155395799 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809bde0 0x7fec7809c1d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/3155395799 >> v2:172.21.15.142:3300/0 conn(0x7fec78097970 0x7fec78097d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/3155395799 >> v2:172.21.15.150:3300/0 conn(0x7fec7809b050 0x7fec780953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 -- 172.21.15.142:0/3155395799 >> 172.21.15.142:0/3155395799 conn(0x7fec780a1fe0 msgr2=0x7fec78070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 -- 172.21.15.142:0/3155395799 shutdown_connections 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 1 -- 172.21.15.142:0/3155395799 wait complete. 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 10 monclient: build_initial_monmap 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.914+0000 7fec7d4a8700 10 monclient: monmap: 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:28.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 Processor -- start 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 -- start start 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient: init 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 0x7fec781410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-b con 0x7fec78097970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-c con 0x7fec7809b050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:28.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 0x7fec7812df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): picked mon.noname-a con 0x7fec7809bde0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7806e800 con 0x7fec7809bde0 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7812bf20 con 0x7fec78097970 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7812c4e0 con 0x7fec7809b050 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7d4a8700 10 monclient(hunting): authenticate will time out at 1517.846094s 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec77fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 msgr2=0x7fec781410d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec77fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 0x7fec781410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:28.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 0x7fec7812df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42106/0 (socket says 172.21.15.142:42106) 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 -- 172.21.15.142:0/1814706716 learned_addr learned my addr 172.21.15.142:0/1814706716 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): get_auth_request con 0x7fec7809b050 auth_method 0 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 0x7fec7812df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32788/0 (socket says 172.21.15.142:32788) 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): get_auth_request con 0x7fec7809bde0 auth_method 0 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec7cca7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient(hunting): handle_auth_done global_id 75176 payload 824 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 -- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 msgr2=0x7fec781410d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 0x7fec781410d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 -- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 msgr2=0x7fec7812df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 0x7fec7812df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient: _finish_hunting 0 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 monclient: found mon.noname-c 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:28.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.918+0000 7fec777fe700 1 -- 172.21.15.142:0/1814706716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7000bcf0 con 0x7fec7809b050 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec777fe700 1 -- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 msgr2=0x7fec7812da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec777fe700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fec7000e440 tx=0x7fec7000d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70010040 con 0x7fec7809b050 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: dump: 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.937 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: _finish_auth 0 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 5 monclient: authenticate success, global_id 75176 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7000b2e0 con 0x7fec7809b050 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70009820 con 0x7fec7809b050 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:28.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 10 monclient: dump: 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _renew_subs 2024-09-18T10:20:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec767fc700 4 set_mon_vals no callback set 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c009420 con 0x7fec7809b050 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _renew_subs 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7806ecd0 con 0x7fec7809b050 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78072a80 con 0x7fec7809b050 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fec70009c50 con 0x7fec7809b050 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 0x7fec64037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:28.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.922+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fec70049430 con 0x7fec7809b050 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.926+0000 7fec77fff700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 0x7fec64037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.926+0000 7fec77fff700 10 monclient: get_auth_request con 0x7fec64035a70 auth_method 0 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.926+0000 7fec77fff700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 0x7fec64037f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fec68006fd0 tx=0x7fec68006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.934+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fec70012510 con 0x7fec7809b050 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.934+0000 7fec5ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:28.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:28.934+0000 7fec5ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:29.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.158+0000 7fec7d4a8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:29.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.158+0000 7fec7d4a8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:29.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.158+0000 7fec7d4a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:29.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.158+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fec780678a0 con 0x7fec7809b050 2024-09-18T10:20:29.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec5ffff700 1 -- 172.21.15.142:0/1814706716 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fec70019020 con 0x7fec7809b050 2024-09-18T10:20:29.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec5ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:29.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec5ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:29.166 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 msgr2=0x7fec64037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 0x7fec64037f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fec68006fd0 tx=0x7fec68006e10 comp rx=0 tx=0).stop 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 10 monclient: shutdown 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 msgr2=0x7fec7812da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:29.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fec7000e440 tx=0x7fec7000d040 comp rx=0 tx=0).stop 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 shutdown_connections 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec64035a70 0x7fec64037f20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec7809bde0 0x7fec7812df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec7809b050 0x7fec7812da10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 --2- 172.21.15.142:0/1814706716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec78097970 0x7fec781410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 >> 172.21.15.142:0/1814706716 conn(0x7fec780a1fe0 msgr2=0x7fec78130950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 shutdown_connections 2024-09-18T10:20:29.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:29.162+0000 7fec7d4a8700 1 -- 172.21.15.142:0/1814706716 wait complete. 2024-09-18T10:20:32.180 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:32.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: get_monmap_and_config 2024-09-18T10:20:32.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: build_initial_monmap 2024-09-18T10:20:32.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: monmap: 2024-09-18T10:20:32.291 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:32.289685+0000 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:32.289685+0000 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:32.292 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 Processor -- start 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 -- start start 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: init 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36000a6260 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 0x7f36000a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): picked mon.noname-c con 0x7f360009e280 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:32.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f36000990b0 0x7f3600139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): picked mon.noname-a con 0x7f36000990b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360009d8a0 con 0x7f36000a6260 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360009df60 con 0x7f36000990b0 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:32.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360009fb20 con 0x7f360009e280 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f3604819700 10 monclient(hunting): authenticate will time out at 1521.215591s 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fdd9b700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 msgr2=0x7f36000a0980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fe59c700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f36000a6260 legacy=0x7f36000a0270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fdd9b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 0x7f36000a0980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f36000990b0 0x7f3600139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f36000990b0 0x7f3600139a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41466/0 (socket says 172.21.15.142:41466) 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 1 -- 172.21.15.142:0/520776986 learned_addr learned my addr 172.21.15.142:0/520776986 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 10 monclient(hunting): get_auth_request con 0x7f36000990b0 auth_method 0 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.286+0000 7f35fed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:32.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient(hunting): handle_auth_done global_id 124266 payload 274 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 -- 172.21.15.142:0/520776986 >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 msgr2=0x7f36000a0980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 --2- 172.21.15.142:0/520776986 >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 0x7f36000a0980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 -- 172.21.15.142:0/520776986 >> v1:172.21.15.150:6789/0 conn(0x7f36000a6260 legacy=0x7f36000a0270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 monclient: found mon.noname-a 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:32.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 -- 172.21.15.142:0/520776986 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36000a6630 con 0x7f36000990b0 2024-09-18T10:20:32.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 -- 172.21.15.142:0/520776986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 msgr2=0x7f3600139a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fed9d700 1 --2- 172.21.15.142:0/520776986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f3600139a10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3600099720 tx=0x7f35f8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96c3e5b9d36380c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 1 -- 172.21.15.142:0/520776986 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f800e070 con 0x7f36000990b0 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: dump: 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: _finish_auth 0 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 5 monclient: authenticate success, global_id 124266 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 1 -- 172.21.15.142:0/520776986 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f800bc60 con 0x7f36000990b0 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:32.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 4 set_mon_vals no callback set 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 1 -- 172.21.15.142:0/520776986 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8012430 con 0x7f36000990b0 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f35fd59a700 10 monclient: dump: 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:32.305 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 10 monclient: shutdown 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 1 -- 172.21.15.142:0/520776986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 msgr2=0x7f3600139a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.290+0000 7f3604819700 1 --2- 172.21.15.142:0/520776986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f3600139a10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3600099720 tx=0x7f35f8009840 comp rx=0 tx=0).stop 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- 172.21.15.142:0/520776986 shutdown_connections 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 --2- 172.21.15.142:0/520776986 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f3600139a10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 --2- 172.21.15.142:0/520776986 >> v2:172.21.15.150:3300/0 conn(0x7f360009e280 0x7f36000a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- 172.21.15.142:0/520776986 >> 172.21.15.142:0/520776986 conn(0x7f3600071680 msgr2=0x7f36000a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- 172.21.15.142:0/520776986 shutdown_connections 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- 172.21.15.142:0/520776986 wait complete. 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient: build_initial_monmap 2024-09-18T10:20:32.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient: monmap: 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 Processor -- start 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- start start 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient: init 2024-09-18T10:20:32.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): picked mon.noname-a con 0x7f36000990b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 0x7f360008cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): picked mon.noname-b con 0x7f360009e280 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36000a6260 0x7f360008d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): picked mon.noname-c con 0x7f36000a6260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360009bfb0 con 0x7f36000990b0 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3600090710 con 0x7f360009e280 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:32.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3600090430 con 0x7f36000a6260 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f3604819700 10 monclient(hunting): authenticate will time out at 1521.223277s 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41468/0 (socket says 172.21.15.142:41468) 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 -- 172.21.15.142:0/3693272009 learned_addr learned my addr 172.21.15.142:0/3693272009 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): get_auth_request con 0x7f36000990b0 auth_method 0 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:32.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fdd9b700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 msgr2=0x7f360008cae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36000a6260 0x7f360008d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fdd9b700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 0x7f360008cae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): get_auth_request con 0x7f36000a6260 auth_method 0 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:32.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient(hunting): handle_auth_done global_id 124269 payload 824 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 -- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36000a6260 msgr2=0x7f360008d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:32.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36000a6260 0x7f360008d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 -- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 msgr2=0x7f360008cae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 0x7f360008cae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient: _finish_hunting 0 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 monclient: found mon.noname-a 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:32.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.294+0000 7f35fe59c700 1 -- 172.21.15.142:0/3693272009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f8009ae0 con 0x7f36000990b0 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35fe59c700 1 -- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 msgr2=0x7f36000980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35fe59c700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b790 tx=0x7f35f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0003dd0 con 0x7f36000990b0 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:32.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: dump: 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: _finish_auth 0 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:32.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0006150 con 0x7f36000990b0 2024-09-18T10:20:32.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 5 monclient: authenticate success, global_id 124269 2024-09-18T10:20:32.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 10 monclient: _renew_subs 2024-09-18T10:20:32.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:32.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3600090150 con 0x7f36000990b0 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0006510 con 0x7f36000990b0 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35f67fc700 10 monclient: dump: 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:32.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f35fcd99700 4 set_mon_vals no callback set 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 10 monclient: _renew_subs 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.298+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360008f4a0 con 0x7f36000990b0 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f35f001b3b0 con 0x7f36000990b0 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f3604819700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f3604819700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f3604819700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36000696a0 con 0x7f36000990b0 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f35f67fc700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 0x7f35e4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:32.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f35f0048490 con 0x7f36000990b0 2024-09-18T10:20:32.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f35fdd9b700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 0x7f35e4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:32.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.302+0000 7f35fdd9b700 10 monclient: get_auth_request con 0x7f35e4035a70 auth_method 0 2024-09-18T10:20:32.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.306+0000 7f35fdd9b700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 0x7f35e4037f20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f35e8006fd0 tx=0x7f35e8006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:32.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:32.310+0000 7f35fe59c700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b790 tx=0x7f35f00091a0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f35f000e860 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:20:33.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.138+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f35f000e860 con 0x7f36000990b0 2024-09-18T10:20:33.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.138+0000 7f35f67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:33.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.138+0000 7f35f67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:33.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.362+0000 7f3604819700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:33.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.362+0000 7f3604819700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.362+0000 7f3604819700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.362+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3600067950 con 0x7f36000990b0 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f35f67fc700 1 -- 172.21.15.142:0/3693272009 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f35f0017030 con 0x7f36000990b0 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f35f67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f35f67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:33.370 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 msgr2=0x7f35e4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 0x7f35e4037f20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f35e8006fd0 tx=0x7f35e8006e10 comp rx=0 tx=0).stop 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 10 monclient: shutdown 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 msgr2=0x7f36000980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35f000b790 tx=0x7f35f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 shutdown_connections 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f35e4035a70 0x7f35e4037f20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:33.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36000a6260 0x7f360008d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:33.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f360009e280 0x7f360008cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:33.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 --2- 172.21.15.142:0/3693272009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36000990b0 0x7f36000980f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:33.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 >> 172.21.15.142:0/3693272009 conn(0x7f3600071680 msgr2=0x7f36000a0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:33.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 shutdown_connections 2024-09-18T10:20:33.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:33.366+0000 7f3604819700 1 -- 172.21.15.142:0/3693272009 wait complete. 2024-09-18T10:20:34.812 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-18T10:20:36.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:36.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: get_monmap_and_config 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: build_initial_monmap 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: monmap: 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:36.493498+0000 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:36.493498+0000 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:36.500 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 1 Processor -- start 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 1 -- start start 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: init 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.490+0000 7f9a842d2700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 0x7f9a7c09e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c09cdd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c0a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c0a0bf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c09d1a0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a7c09e980 con 0x7f9a7c09cdd0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c09d780 con 0x7f9a7c0a02e0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient(hunting): authenticate will time out at 1525.420155s 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8206e700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 msgr2=0x7f9a7c09e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8206e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 0x7f9a7c09e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49186/0 (socket says 172.21.15.142:49186) 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41470/0 (socket says 172.21.15.142:41470) 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 1 -- 172.21.15.142:0/3801897206 learned_addr learned my addr 172.21.15.142:0/3801897206 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): get_auth_request con 0x7f9a7c0a02e0 auth_method 0 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): get_auth_request con 0x7f9a7c0a0bf0 auth_method 0 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:36.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient(hunting): handle_auth_done global_id 124272 payload 274 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 -- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 msgr2=0x7f9a7c0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 -- 172.21.15.142:0/3801897206 >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 msgr2=0x7f9a7c09e440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 --2- 172.21.15.142:0/3801897206 >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 0x7f9a7c09e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient: _finish_hunting 0 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 monclient: found mon.noname-a 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 -- 172.21.15.142:0/3801897206 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c09dd40 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8186d700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 -- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 msgr2=0x7f9a7c138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8286f700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c071e50 tx=0x7f9a74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ff5e2136fcbdaf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 1 -- 172.21.15.142:0/3801897206 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7400e070 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: dump: 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:36.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: _finish_auth 0 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 5 monclient: authenticate success, global_id 124272 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 1 -- 172.21.15.142:0/3801897206 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7400bd70 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 1 -- 172.21.15.142:0/3801897206 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74012660 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:36.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a842d2700 4 set_mon_vals no callback set 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.494+0000 7f9a8106c700 10 monclient: dump: 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.512 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 10 monclient: shutdown 2024-09-18T10:20:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 msgr2=0x7f9a7c138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c071e50 tx=0x7f9a74009780 comp rx=0 tx=0).stop 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- 172.21.15.142:0/3801897206 shutdown_connections 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c138a70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 --2- 172.21.15.142:0/3801897206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c0a02e0 0x7f9a7c0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 --2- 172.21.15.142:0/3801897206 >> v2:172.21.15.150:3300/0 conn(0x7f9a7c09cdd0 0x7f9a7c09e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- 172.21.15.142:0/3801897206 >> 172.21.15.142:0/3801897206 conn(0x7f9a7c071680 msgr2=0x7f9a7c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- 172.21.15.142:0/3801897206 shutdown_connections 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- 172.21.15.142:0/3801897206 wait complete. 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 10 monclient: build_initial_monmap 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 10 monclient: monmap: 2024-09-18T10:20:36.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 Processor -- start 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 1 -- start start 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.498+0000 7f9a842d2700 10 monclient: init 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 0x7f9a7c141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c09cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 0x7f9a7c131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c0a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c0a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c142400 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c12b700 con 0x7f9a7c0a02e0 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:36.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7c12b390 con 0x7f9a7c09cdd0 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a842d2700 10 monclient(hunting): authenticate will time out at 1525.428462s 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8186d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 msgr2=0x7f9a7c131ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8186d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 0x7f9a7c131ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 0x7f9a7c141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41484/0 (socket says 172.21.15.142:41484) 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 0x7f9a7c141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49198/0 (socket says 172.21.15.142:49198) 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 -- 172.21.15.142:0/803494471 learned_addr learned my addr 172.21.15.142:0/803494471 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): get_auth_request con 0x7f9a7c0a0bf0 auth_method 0 2024-09-18T10:20:36.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): get_auth_request con 0x7f9a7c09cdd0 auth_method 0 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8206e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient(hunting): handle_auth_done global_id 124275 payload 824 2024-09-18T10:20:36.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 msgr2=0x7f9a7c141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 0x7f9a7c141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 msgr2=0x7f9a7c131ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 0x7f9a7c131ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient: _finish_hunting 0 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 monclient: found mon.noname-a 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 -- 172.21.15.142:0/803494471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0094f0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 msgr2=0x7f9a7c06cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.502+0000 7f9a8286f700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c068ef0 tx=0x7f9a74019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a740263f0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:36.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: dump: 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: _finish_auth 0 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:36.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 5 monclient: authenticate success, global_id 124275 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a74026570 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _renew_subs 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a74026930 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a74009a20 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8086b700 4 set_mon_vals no callback set 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 10 monclient: dump: 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:36.521 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:36.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:36.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _renew_subs 2024-09-18T10:20:36.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c12b0b0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:36.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c0696a0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9a7401b3f0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9a7404aed0 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8206e700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8206e700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 msgr2=0x7f9a60037f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8206e700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 msgr2=0x7f9a60037f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:20:36.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8206e700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:36.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.506+0000 7f9a8206e700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:36.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.514+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9a74026d60 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.514+0000 7f9a727fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:36.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.514+0000 7f9a727fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:36.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.710+0000 7f9a8206e700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:36.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.710+0000 7f9a8206e700 10 monclient: get_auth_request con 0x7f9a60035ac0 auth_method 0 2024-09-18T10:20:36.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.710+0000 7f9a8206e700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003980 tx=0x7f9a78005040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:36.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:36.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9a7c12b840 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a727fc700 1 -- 172.21.15.142:0/803494471 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f9a74019780 con 0x7f9a7c0a0bf0 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a727fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a727fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:36.742 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 msgr2=0x7f9a60037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003980 tx=0x7f9a78005040 comp rx=0 tx=0).stop 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 10 monclient: shutdown 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 msgr2=0x7f9a7c06cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c068ef0 tx=0x7f9a74019000 comp rx=0 tx=0).stop 2024-09-18T10:20:36.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 shutdown_connections 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a60035ac0 0x7f9a60037f70 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a7c0a0bf0 0x7f9a7c06cdf0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a7c0a02e0 0x7f9a7c131ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 --2- 172.21.15.142:0/803494471 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a7c09cdd0 0x7f9a7c141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 >> 172.21.15.142:0/803494471 conn(0x7f9a7c071680 msgr2=0x7f9a7c0a4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 shutdown_connections 2024-09-18T10:20:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:36.738+0000 7f9a842d2700 1 -- 172.21.15.142:0/803494471 wait complete. 2024-09-18T10:20:39.757 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: get_monmap_and_config 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: build_initial_monmap 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: monmap: 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:39.865343+0000 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:39.865343+0000 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:39.867 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:39.868 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 Processor -- start 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 -- start start 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: init 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-c con 0x7f400c09abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:20:39.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f400c09db80 addr v1:172.21.15.150:6789/0 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f400c097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0958c0 con 0x7f400c097880 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c095ef0 con 0x7f400c09db80 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400c06fa60 con 0x7f400c09abd0 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4013eed700 10 monclient(hunting): authenticate will time out at 1528.790848s 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4011488700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f400c09db80 legacy=0x7f400c09c410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4011c89700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 msgr2=0x7f400c09bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f401248a700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f400c097880 0x7f400c137690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:43904/0 (socket says 172.21.15.142:43904) 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f401248a700 1 -- 172.21.15.142:0/3687492455 learned_addr learned my addr 172.21.15.142:0/3687492455 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:20:39.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.862+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:39.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:39.866+0000 7f401248a700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.142:6789/0 conn(0x7f400c097880 legacy=0x7f400c137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:40.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.066+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.150:6789/0 conn(0x7f400c09db80 legacy=0x7f400c09c410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:20:40.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.066+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 msgr2=0x7f400c09bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:40.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.066+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:20:40.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.470+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 msgr2=0x7f400c09bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:40.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.470+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.150:6789/0 conn(0x7f400c09db80 legacy=0x7f400c09c410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:20:40.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:40.470+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:20:41.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:41.274+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 msgr2=0x7f400c09bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:41.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:41.274+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.150:6789/0 conn(0x7f400c09db80 legacy=0x7f400c09c410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:20:41.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:41.274+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:20:42.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): tick 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 msgr2=0x7f400c09bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.150:3300/0 conn(0x7f400c09abd0 0x7f400c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.150:6789/0 conn(0x7f400c09db80 legacy=0x7f400c09c410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 -- 172.21.15.142:0/3687492455 >> v1:172.21.15.142:6789/0 conn(0x7f400c097880 legacy=0x7f400c137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient: _add_conns ranks=[2,3] 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): picked mon.noname-b con 0x7f3ff0003790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): picked mon.noname-a con 0x7f3ff00062b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:42.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 -- 172.21.15.142:0/3687492455 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff0008df0 con 0x7f3ff00062b0 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 1 -- 172.21.15.142:0/3687492455 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff0008f60 con 0x7f3ff0003790 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f40037fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 10 monclient(hunting): get_auth_request con 0x7f3ff0003790 auth_method 0 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 10 monclient(hunting): get_auth_request con 0x7f3ff00062b0 auth_method 0 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:42.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 msgr2=0x7f3ff0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 msgr2=0x7f3ff0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:42.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:42.862+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:43.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 10 monclient(hunting): get_auth_request con 0x7f3ff00062b0 auth_method 0 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 10 monclient(hunting): get_auth_request con 0x7f3ff0003790 auth_method 0 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:20:43.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 msgr2=0x7f3ff0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 msgr2=0x7f3ff0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:20:43.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.066+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:20:43.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> v2:172.21.15.142:3300/0 conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): get_auth_request con 0x7f3ff00062b0 auth_method 0 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): get_auth_request con 0x7f3ff0003790 auth_method 0 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient(hunting): handle_auth_done global_id 124278 payload 274 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 msgr2=0x7f3ff0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ff0003790 0x7f3ff0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient: _finish_hunting 0 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 monclient: found mon.noname-a 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:43.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c09afa0 con 0x7f3ff00062b0 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.470+0000 7f4011488700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f3ff0008760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4008003980 tx=0x7f4008003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c08b7eaf0dbe89 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 1 -- 172.21.15.142:0/3687492455 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080057a0 con 0x7f3ff00062b0 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: dump: 2024-09-18T10:20:43.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: _finish_auth 0 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 5 monclient: authenticate success, global_id 124278 2024-09-18T10:20:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 1 -- 172.21.15.142:0/3687492455 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4008005900 con 0x7f3ff00062b0 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 4 set_mon_vals no callback set 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 1 -- 172.21.15.142:0/3687492455 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008016690 con 0x7f3ff00062b0 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4010c87700 10 monclient: dump: 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.481 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.481 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 10 monclient: shutdown 2024-09-18T10:20:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 -- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 msgr2=0x7f3ff0008760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f3ff0008760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4008003980 tx=0x7f4008003e20 comp rx=0 tx=0).stop 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f401248a700 1 -- 172.21.15.142:0/3687492455 reap_dead start 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 -- 172.21.15.142:0/3687492455 shutdown_connections 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 --2- 172.21.15.142:0/3687492455 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f3ff0008760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 -- 172.21.15.142:0/3687492455 >> 172.21.15.142:0/3687492455 conn(0x7f400c0a14e0 msgr2=0x7f400c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 -- 172.21.15.142:0/3687492455 shutdown_connections 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 1 -- 172.21.15.142:0/3687492455 wait complete. 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 10 monclient: build_initial_monmap 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.474+0000 7f4013eed700 10 monclient: monmap: 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 Processor -- start 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 -- start start 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient: init 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f400c0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-a con 0x7f3ff00062b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 0x7f400c0992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-b con 0x7f400c098f00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): picked mon.noname-c con 0x7f400c099830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c08fe90 con 0x7f3ff00062b0 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c097d40 con 0x7f400c098f00 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c06e8e0 con 0x7f400c099830 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4013eed700 10 monclient(hunting): authenticate will time out at 1532.404768s 2024-09-18T10:20:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011488700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 msgr2=0x7f400c0992f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011488700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 0x7f400c0992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f400c0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50016/0 (socket says 172.21.15.142:50016) 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 -- 172.21.15.142:0/21423172 learned_addr learned my addr 172.21.15.142:0/21423172 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): get_auth_request con 0x7f400c099830 auth_method 0 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): get_auth_request con 0x7f3ff00062b0 auth_method 0 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f4011c89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient(hunting): handle_auth_done global_id 75191 payload 824 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 -- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 msgr2=0x7f400c0992f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 0x7f400c0992f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 -- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 msgr2=0x7f400c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f400c0989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient: _finish_hunting 0 2024-09-18T10:20:43.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 monclient: found mon.noname-c 2024-09-18T10:20:43.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:43.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.478+0000 7f401248a700 1 -- 172.21.15.142:0/21423172 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008011040 con 0x7f400c099830 2024-09-18T10:20:43.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f401248a700 1 -- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 msgr2=0x7f400c147790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f401248a700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f400c09d0d0 tx=0x7f4004001b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400400aa10 con 0x7f400c099830 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: dump: 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: _finish_auth 0 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 5 monclient: authenticate success, global_id 75191 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _renew_subs 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400400abb0 con 0x7f400c099830 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c091830 con 0x7f400c099830 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40040070d0 con 0x7f400c099830 2024-09-18T10:20:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4003fff700 4 set_mon_vals no callback set 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 10 monclient: dump: 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _renew_subs 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c090f90 con 0x7f400c099830 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4004007540 con 0x7f400c099830 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c0696a0 con 0x7f400c099830 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 0x7f3ff0038f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f40040489e0 con 0x7f400c099830 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4011c89700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 0x7f3ff0038f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.482+0000 7f4011c89700 10 monclient: get_auth_request con 0x7f3ff0036ab0 auth_method 0 2024-09-18T10:20:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.486+0000 7f4011c89700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 0x7f3ff0038f60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009fd0 tx=0x7f3ffc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.494+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4004010d70 con 0x7f400c099830 2024-09-18T10:20:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.494+0000 7f4001ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.494+0000 7f4001ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:20:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f400c067950 con 0x7f400c099830 2024-09-18T10:20:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4001ffb700 1 -- 172.21.15.142:0/21423172 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f4004010770 con 0x7f400c099830 2024-09-18T10:20:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4001ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:20:43.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4001ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:43.723 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":120,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:20:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 msgr2=0x7f3ff0038f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 0x7f3ff0038f60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009fd0 tx=0x7f3ffc00b040 comp rx=0 tx=0).stop 2024-09-18T10:20:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 10 monclient: shutdown 2024-09-18T10:20:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:43.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 msgr2=0x7f400c147790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f400c09d0d0 tx=0x7f4004001b30 comp rx=0 tx=0).stop 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 shutdown_connections 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3ff0036ab0 0x7f3ff0038f60 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f400c099830 0x7f400c147790 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f400c098f00 0x7f400c0992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 --2- 172.21.15.142:0/21423172 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ff00062b0 0x7f400c0989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.718+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 >> 172.21.15.142:0/21423172 conn(0x7f400c0a14e0 msgr2=0x7f400c06e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.722+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 shutdown_connections 2024-09-18T10:20:43.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.722+0000 7f4013eed700 1 -- 172.21.15.142:0/21423172 wait complete. 2024-09-18T10:20:43.736 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:20:43.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:20:43.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:20:43.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: get_monmap_and_config 2024-09-18T10:20:43.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: build_initial_monmap 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: monmap: 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:43.843971+0000 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:43.843971+0000 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:43.846 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:43.847 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 Processor -- start 2024-09-18T10:20:43.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 -- start start 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: init 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-b con 0x7fee6809a800 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee6809abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 0x7fee6813dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-a con 0x7fee6813b6e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6809da40 con 0x7fee6809abd0 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6809afa0 con 0x7fee6813b6e0 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee68099a40 con 0x7fee6809a800 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6ff21700 10 monclient(hunting): authenticate will time out at 1532.769861s 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 0x7fee6813dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 0x7fee6813dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60660/0 (socket says 172.21.15.142:60660) 2024-09-18T10:20:43.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 1 -- 172.21.15.142:0/1917637214 learned_addr learned my addr 172.21.15.142:0/1917637214 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): get_auth_request con 0x7fee6813b6e0 auth_method 0 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50022/0 (socket says 172.21.15.142:50022) 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): get_auth_request con 0x7fee6809a800 auth_method 0 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_done global_id 75194 payload 274 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1917637214 >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 msgr2=0x7fee6813dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 --2- 172.21.15.142:0/1917637214 >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 0x7fee6813dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1917637214 >> v1:172.21.15.142:6789/0 conn(0x7fee6809abd0 legacy=0x7fee68099230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient: _finish_hunting 0 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 monclient: found mon.noname-b 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:20:43.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.842+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1917637214 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6809b8e0 con 0x7fee6809a800 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1917637214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 msgr2=0x7fee68098cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6dcbd700 1 --2- 172.21.15.142:0/1917637214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fee580098e0 tx=0x7fee580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c067e2c6c73a9f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 1 -- 172.21.15.142:0/1917637214 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5800b850 con 0x7fee6809a800 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: dump: 2024-09-18T10:20:43.860 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: _finish_auth 0 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 5 monclient: authenticate success, global_id 75194 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 1 -- 172.21.15.142:0/1917637214 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee5800b9b0 con 0x7fee6809a800 2024-09-18T10:20:43.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 4 set_mon_vals no callback set 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 1 -- 172.21.15.142:0/1917637214 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5800bd50 con 0x7fee6809a800 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ccbb700 10 monclient: dump: 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.862 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 10 monclient: shutdown 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- 172.21.15.142:0/1917637214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 msgr2=0x7fee68098cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1917637214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fee580098e0 tx=0x7fee580091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- 172.21.15.142:0/1917637214 shutdown_connections 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1917637214 >> v2:172.21.15.142:3300/0 conn(0x7fee6813b6e0 0x7fee6813dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1917637214 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6809a800 0x7fee68098cf0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- 172.21.15.142:0/1917637214 >> 172.21.15.142:0/1917637214 conn(0x7fee68071b20 msgr2=0x7fee680a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:43.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- 172.21.15.142:0/1917637214 shutdown_connections 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- 172.21.15.142:0/1917637214 wait complete. 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 10 monclient: build_initial_monmap 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 10 monclient: monmap: 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:43.864 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 Processor -- start 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 1 -- start start 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.846+0000 7fee6ff21700 10 monclient: init 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-a con 0x7fee6809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6813b6e0 0x7fee6813a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-b con 0x7fee6813b6e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6806c120 0x7fee6813ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): picked mon.noname-c con 0x7fee6806c120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee68094810 con 0x7fee6809abd0 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680a1070 con 0x7fee6813b6e0 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6806d080 con 0x7fee6806c120 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient(hunting): authenticate will time out at 1532.776566s 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60672/0 (socket says 172.21.15.142:60672) 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1344319057 learned_addr learned my addr 172.21.15.142:0/1344319057 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): get_auth_request con 0x7fee6809abd0 auth_method 0 2024-09-18T10:20:43.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6806c120 0x7fee6813ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6d4bc700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6813b6e0 msgr2=0x7fee6813a6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): get_auth_request con 0x7fee6806c120 auth_method 0 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6d4bc700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6813b6e0 0x7fee6813a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6e4be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient(hunting): handle_auth_done global_id 124287 payload 824 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6806c120 msgr2=0x7fee6813ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fee6806c120 0x7fee6813ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6813b6e0 msgr2=0x7fee6813a6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6813b6e0 0x7fee6813a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient: _finish_hunting 0 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 monclient: found mon.noname-a 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58009420 con 0x7fee6809abd0 2024-09-18T10:20:43.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 msgr2=0x7fee68093fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6dcbd700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fee580093f0 tx=0x7fee58016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58000d10 con 0x7fee6809abd0 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: dump: 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.869 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: _finish_auth 0 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 5 monclient: authenticate success, global_id 124287 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _renew_subs 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58011e20 con 0x7fee6809abd0 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6806b570 con 0x7fee6809abd0 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5801b8d0 con 0x7fee6809abd0 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5dffb700 10 monclient: dump: 2024-09-18T10:20:43.870 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.871 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee5ffff700 4 set_mon_vals no callback set 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _renew_subs 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.850+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6806e3b0 con 0x7fee6809abd0 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6ff21700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 0x7fee6813b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee68064950 con 0x7fee68071b20 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fee5801bc40 con 0x7fee6809abd0 2024-09-18T10:20:43.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 0x7fee6813b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient: get_auth_request con 0x7fee68071b20 auth_method 0 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee5dffb700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fee4c035a70 0x7fee4c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fee5804a390 con 0x7fee6809abd0 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_done global_id 124290 payload 824 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee680696a0 con 0x7fee68071b20 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6e4be700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fee4c035a70 0x7fee4c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6e4be700 10 monclient: get_auth_request con 0x7fee4c035a70 auth_method 0 2024-09-18T10:20:43.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 msgr2=0x7fee6813b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.854+0000 7fee6d4bc700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 0x7fee6813b170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c7e0 tx=0x7fee64002710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64007300 con 0x7fee68071b20 2024-09-18T10:20:43.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 10 monclient: dump: 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.877 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee6e4be700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fee4c035a70 0x7fee4c037f20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fee6809cf70 tx=0x7fee60009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fee64007460 con 0x7fee68071b20 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 msgr2=0x7fee6813b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 0x7fee6813b170 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c7e0 tx=0x7fee64002710 comp rx=0 tx=0).stop 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 msgr2=0x7fee6813b170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.858+0000 7fee5dffb700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee68071b20 0x7fee6813b170 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c7e0 tx=0x7fee64002710 comp rx=0 tx=0).stop 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6ff21700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 0x7fee6806a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:43.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6ff21700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6806e3b0 con 0x7fee6806a360 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 0x7fee6806a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient: get_auth_request con 0x7fee6806a360 auth_method 0 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 10 monclient(hunting): handle_auth_done global_id 124293 payload 824 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.866+0000 7fee6d4bc700 1 -- 172.21.15.142:0/1344319057 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fee6806d5a0 con 0x7fee6806a360 2024-09-18T10:20:43.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6d4bc700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 msgr2=0x7fee6806a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6d4bc700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 0x7fee6806a760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee64015000 tx=0x7fee64016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64012e20 con 0x7fee6806a360 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 10 monclient: dump: 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:43.880 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7fee64021840 con 0x7fee6806a360 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 msgr2=0x7fee6806a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 0x7fee6806a760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee64015000 tx=0x7fee64016000 comp rx=0 tx=0).stop 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 msgr2=0x7fee6806a760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:43.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee5dffb700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6806a360 0x7fee6806a760 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee64015000 tx=0x7fee64016000 comp rx=0 tx=0).stop 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 120, 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:43.882 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:43.883 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:43.884 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:43.885 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:43.886 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:43.887 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:43.888 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:43.889 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:43.890 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.891 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.892 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:43.893 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:43.894 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:43.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fee4c035a70 msgr2=0x7fee4c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fee4c035a70 0x7fee4c037f20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fee6809cf70 tx=0x7fee60009250 comp rx=0 tx=0).stop 2024-09-18T10:20:43.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 10 monclient: shutdown 2024-09-18T10:20:43.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6e4be700 1 -- 172.21.15.142:0/1344319057 reap_dead start 2024-09-18T10:20:43.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 msgr2=0x7fee68093fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fee580093f0 tx=0x7fee58016000 comp rx=0 tx=0).stop 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 shutdown_connections 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 --2- 172.21.15.142:0/1344319057 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee6809abd0 0x7fee68093fc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 >> 172.21.15.142:0/1344319057 conn(0x7fee6809a800 msgr2=0x7fee680a0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 shutdown_connections 2024-09-18T10:20:43.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.870+0000 7fee6ff21700 1 -- 172.21.15.142:0/1344319057 wait complete. 2024-09-18T10:20:43.896 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:20:43.897 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:20:43.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:20:44.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.998+0000 7f82e6fd1700 10 monclient: get_monmap_and_config 2024-09-18T10:20:44.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.998+0000 7f82e6fd1700 10 monclient: build_initial_monmap 2024-09-18T10:20:44.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:43.998+0000 7f82e6fd1700 10 monclient: monmap: 2024-09-18T10:20:44.005 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:44.005 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:20:44.003620+0000 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:20:44.003620+0000 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:20:44.006 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 Processor -- start 2024-09-18T10:20:44.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 -- start start 2024-09-18T10:20:44.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient: init 2024-09-18T10:20:44.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:44.009 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 0x7f82e009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-a con 0x7f82e009abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-b con 0x7f82e009db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82e009c950 addr v1:172.21.15.142:6789/0 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e00958c0 con 0x7f82e009c950 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e006fa60 con 0x7f82e009abd0 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0095ef0 con 0x7f82e009db80 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e6fd1700 10 monclient(hunting): authenticate will time out at 1532.929613s 2024-09-18T10:20:44.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 0x7f82e009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50044/0 (socket says 172.21.15.142:50044) 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 0x7f82e009bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60686/0 (socket says 172.21.15.142:60686) 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 -- 172.21.15.142:0/2253906080 learned_addr learned my addr 172.21.15.142:0/2253906080 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): get_auth_request con 0x7f82e009db80 auth_method 0 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): get_auth_request con 0x7f82e009abd0 auth_method 0 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient(hunting): handle_auth_done global_id 75200 payload 274 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 -- 172.21.15.142:0/2253906080 >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 msgr2=0x7f82e009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 --2- 172.21.15.142:0/2253906080 >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 0x7f82e009bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 -- 172.21.15.142:0/2253906080 >> v1:172.21.15.142:6789/0 conn(0x7f82e009c950 legacy=0x7f82e0096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient: _finish_hunting 0 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 monclient: found mon.noname-b 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:44.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:20:44.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.002+0000 7f82dffff700 1 -- 172.21.15.142:0/2253906080 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e009afa0 con 0x7f82e009db80 2024-09-18T10:20:44.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82dffff700 1 -- 172.21.15.142:0/2253906080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 msgr2=0x7f82e009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82dffff700 1 --2- 172.21.15.142:0/2253906080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f82d00099b0 tx=0x7f82d0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd7cf9d05a32fdf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 1 -- 172.21.15.142:0/2253906080 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0009ea0 con 0x7f82e009db80 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: dump: 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.019 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: _finish_auth 0 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 5 monclient: authenticate success, global_id 75200 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 1 -- 172.21.15.142:0/2253906080 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d000ba70 con 0x7f82e009db80 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 10 monclient: get_monmap_and_config success 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 1 -- 172.21.15.142:0/2253906080 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d000be30 con 0x7f82e009db80 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 4 set_mon_vals no callback set 2024-09-18T10:20:44.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82df7fe700 10 monclient: dump: 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 10 monclient: shutdown 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:44.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- 172.21.15.142:0/2253906080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 msgr2=0x7f82e009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/2253906080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f82d00099b0 tx=0x7f82d0009250 comp rx=0 tx=0).stop 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- 172.21.15.142:0/2253906080 shutdown_connections 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/2253906080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009db80 0x7f82e009c410 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/2253906080 >> v2:172.21.15.142:3300/0 conn(0x7f82e009abd0 0x7f82e009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- 172.21.15.142:0/2253906080 >> 172.21.15.142:0/2253906080 conn(0x7f82e00a14e0 msgr2=0x7f82e00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- 172.21.15.142:0/2253906080 shutdown_connections 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- 172.21.15.142:0/2253906080 wait complete. 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 10 monclient: build_initial_monmap 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 10 monclient: monmap: 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:20:44.022 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 Processor -- start 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 1 -- start start 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.006+0000 7f82e6fd1700 10 monclient: init 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient: _reopen_session rank -1 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e009abd0 0x7f82e0141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-b con 0x7f82e009abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009c950 0x7f82e0141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-c con 0x7f82e009c950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): picked mon.noname-a con 0x7f82e009db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0091a10 con 0x7f82e009db80 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e009a600 con 0x7f82e009abd0 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0094f60 con 0x7f82e009c950 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): _renew_subs 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e6fd1700 10 monclient(hunting): authenticate will time out at 1532.936972s 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e4d6d700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e009abd0 msgr2=0x7f82e0141760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:20:44.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e4d6d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e009abd0 0x7f82e0141760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009c950 0x7f82e0141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60692/0 (socket says 172.21.15.142:60692) 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009c950 0x7f82e0141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50060/0 (socket says 172.21.15.142:50060) 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 learned_addr learned my addr 172.21.15.142:0/3192249663 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): get_auth_request con 0x7f82e009db80 auth_method 0 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): get_auth_request con 0x7f82e009c950 auth_method 0 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82dffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient(hunting): handle_auth_done global_id 124299 payload 824 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009c950 msgr2=0x7f82e0141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e009c950 0x7f82e0141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e009abd0 msgr2=0x7f82e0141760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82e009abd0 0x7f82e0141760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient: _finish_hunting 0 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 monclient: found mon.noname-a 2024-09-18T10:20:44.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.010+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00094f0 con 0x7f82e009db80 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 msgr2=0x7f82e01421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e556e700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c960 tx=0x7f82d80026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8012840 con 0x7f82e009db80 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: dump: 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.027 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: _finish_auth 0 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d80129c0 con 0x7f82e009db80 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8012d80 con 0x7f82e009db80 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 5 monclient: authenticate success, global_id 124299 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 10 monclient: dump: 2024-09-18T10:20:44.028 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 monclient: _renew_subs 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82de7fc700 4 set_mon_vals no callback set 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e0097c90 con 0x7f82e009db80 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 monclient: _renew_subs 2024-09-18T10:20:44.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:20:44.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e0097700 con 0x7f82e009db80 2024-09-18T10:20:44.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 0x7f82e00883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0064950 con 0x7f82e0087fe0 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 0x7f82e00883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient: get_auth_request con 0x7f82e0087fe0 auth_method 0 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e556e700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c960 tx=0x7f82d80026e0 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f82d8011590 mgrmap(e 5) v1 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f82d8011590 con 0x7f82e009db80 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_done global_id 75206 payload 824 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82e4d6d700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e0092740 con 0x7f82e0087fe0 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dcff9700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f82cc035a70 0x7f82cc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dffff700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f82cc035a70 0x7f82cc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.014+0000 7f82dffff700 10 monclient: get_auth_request con 0x7f82cc035a70 auth_method 0 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82e4d6d700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 msgr2=0x7f82e00883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82e4d6d700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 0x7f82e00883b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e8d0 tx=0x7f82d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400ee70 con 0x7f82e0087fe0 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 10 monclient: dump: 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f82d400caf0 con 0x7f82e0087fe0 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:20:44.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:20:44.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 msgr2=0x7f82e00883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 0x7f82e00883b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e8d0 tx=0x7f82d40091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 msgr2=0x7f82e00883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:44.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dcff9700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e0087fe0 0x7f82e00883b0 secure :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f82d400e8d0 tx=0x7f82d40091a0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.018+0000 7f82dffff700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f82cc035a70 0x7f82cc037f20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f82d00099b0 tx=0x7f82d0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:44.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e6fd1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:20:44.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 0x7f82e0098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:20:44.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e6fd1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:20:44.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e0091850 con 0x7f82e00888f0 2024-09-18T10:20:44.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 0x7f82e0098a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient: get_auth_request con 0x7f82e00888f0 auth_method 0 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 10 monclient(hunting): handle_auth_done global_id 75209 payload 824 2024-09-18T10:20:44.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 1 -- 172.21.15.142:0/3192249663 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f82e00696a0 con 0x7f82e00888f0 2024-09-18T10:20:44.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 msgr2=0x7f82e0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82e4d6d700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 0x7f82e0098a70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f82d4018000 tx=0x7f82d4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4022070 con 0x7f82e00888f0 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.030+0000 7f82dcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 10 monclient: dump: 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:20:44.039 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f82d4013680 con 0x7f82e00888f0 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 msgr2=0x7f82e0098a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 0x7f82e0098a70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f82d4018000 tx=0x7f82d4019000 comp rx=0 tx=0).stop 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 msgr2=0x7f82e0098a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82e00888f0 0x7f82e0098a70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.040 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 120, 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:20:44.041 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:20:44.042 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:20:44.043 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:20:44.044 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:20:44.045 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.046 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.047 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:20:44.048 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:20:44.049 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.050 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:20:44.051 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.052 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:20:44.053 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:20:44.054 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:20:44.055 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f82cc035a70 msgr2=0x7f82cc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f82cc035a70 0x7f82cc037f20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f82d00099b0 tx=0x7f82d0016000 comp rx=0 tx=0).stop 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 10 monclient: shutdown 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 msgr2=0x7f82e01421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:20:44.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 --2- 172.21.15.142:0/3192249663 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82e009db80 0x7f82e01421e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c960 tx=0x7f82d80026e0 comp rx=0 tx=0).stop 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 reap_dead start 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e556e700 1 -- 172.21.15.142:0/3192249663 reap_dead start 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82dcff9700 1 -- 172.21.15.142:0/3192249663 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f82d80492f0 con 0x7f82e009db80 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 shutdown_connections 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 >> 172.21.15.142:0/3192249663 conn(0x7f82e00a14e0 msgr2=0x7f82e006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 shutdown_connections 2024-09-18T10:20:44.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:20:44.034+0000 7f82e6fd1700 1 -- 172.21.15.142:0/3192249663 wait complete. 2024-09-18T10:20:44.058 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:20:44.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:21:04.060 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:21:04.060 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:21:04.060 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:21:04.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:21:04.064 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:21:04.064 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:21:04.064 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 10 monclient: get_monmap_and_config 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 10 monclient: build_initial_monmap 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 10 monclient: monmap: 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:04.176 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:04.174318+0000 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:04.174318+0000 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:04.177 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 1 Processor -- start 2024-09-18T10:21:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 1 -- start start 2024-09-18T10:21:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.170+0000 7f22fb304700 10 monclient: init 2024-09-18T10:21:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:04.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f22f40a1cf0 0x7f22f412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-a con 0x7f22f40a1cf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22f412f810 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-b con 0x7f22f40a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f409fb90 con 0x7f22f412f810 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f409d990 con 0x7f22f40a1cf0 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:04.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f409dc70 con 0x7f22f40a02d0 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient(hunting): authenticate will time out at 1553.100435s 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f22f40a1cf0 0x7f22f412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51074/0 (socket says 172.21.15.142:51074) 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 -- 172.21.15.142:0/3628682816 learned_addr learned my addr 172.21.15.142:0/3628682816 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): get_auth_request con 0x7f22f40a1cf0 auth_method 0 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): get_auth_request con 0x7f22f40a02d0 auth_method 0 2024-09-18T10:21:04.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fa302700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient(hunting): handle_auth_done global_id 75212 payload 274 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 -- 172.21.15.142:0/3628682816 >> v2:172.21.15.142:3300/0 conn(0x7f22f40a1cf0 msgr2=0x7f22f412f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 --2- 172.21.15.142:0/3628682816 >> v2:172.21.15.142:3300/0 conn(0x7f22f40a1cf0 0x7f22f412f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 -- 172.21.15.142:0/3628682816 >> v1:172.21.15.150:6789/0 conn(0x7f22f412f810 legacy=0x7f22f409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:04.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient: _finish_hunting 0 2024-09-18T10:21:04.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 monclient: found mon.noname-b 2024-09-18T10:21:04.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:04.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 -- 172.21.15.142:0/3628682816 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f40a20c0 con 0x7f22f40a02d0 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 -- 172.21.15.142:0/3628682816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 msgr2=0x7f22f4139420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fab03700 1 --2- 172.21.15.142:0/3628682816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f22f4071e60 tx=0x7f22ec009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86bd80366b654902 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 1 -- 172.21.15.142:0/3628682816 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec00e070 con 0x7f22f40a02d0 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: dump: 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: _finish_auth 0 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 5 monclient: authenticate success, global_id 75212 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 1 -- 172.21.15.142:0/3628682816 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec00bc30 con 0x7f22f40a02d0 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:04.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22fb304700 4 set_mon_vals no callback set 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 1 -- 172.21.15.142:0/3628682816 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0092e0 con 0x7f22f40a02d0 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.174+0000 7f22f9300700 10 monclient: dump: 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: shutdown 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- 172.21.15.142:0/3628682816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 msgr2=0x7f22f4139420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- 172.21.15.142:0/3628682816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f22f4071e60 tx=0x7f22ec009840 comp rx=0 tx=0).stop 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- 172.21.15.142:0/3628682816 shutdown_connections 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- 172.21.15.142:0/3628682816 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a02d0 0x7f22f4139420 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- 172.21.15.142:0/3628682816 >> v2:172.21.15.142:3300/0 conn(0x7f22f40a1cf0 0x7f22f412f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- 172.21.15.142:0/3628682816 >> 172.21.15.142:0/3628682816 conn(0x7f22f4071820 msgr2=0x7f22f40a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- 172.21.15.142:0/3628682816 shutdown_connections 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- 172.21.15.142:0/3628682816 wait complete. 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: build_initial_monmap 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: monmap: 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:04.194 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 Processor -- start 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- start start 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: init 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-b con 0x7f22f40a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-c con 0x7f22f40a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22f412f810 0x7f22f406b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): picked mon.noname-a con 0x7f22f412f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:04.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f409cdc0 con 0x7f22f412f810 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f412ee20 con 0x7f22f40a02d0 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f406fa10 con 0x7f22f40a1cf0 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fb304700 10 monclient(hunting): authenticate will time out at 1553.106223s 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51086/0 (socket says 172.21.15.142:51086) 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 -- 172.21.15.142:0/129803596 learned_addr learned my addr 172.21.15.142:0/129803596 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): get_auth_request con 0x7f22f40a1cf0 auth_method 0 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:04.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22f412f810 0x7f22f406b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): get_auth_request con 0x7f22f412f810 auth_method 0 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 10 monclient(hunting): get_auth_request con 0x7f22f40a02d0 auth_method 0 2024-09-18T10:21:04.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 msgr2=0x7f22f406ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 msgr2=0x7f22f406ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fab03700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient(hunting): handle_auth_done global_id 75215 payload 824 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22fa302700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:04.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 msgr2=0x7f22f406ab60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22f412f810 msgr2=0x7f22f406b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22f412f810 0x7f22f406b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient: _finish_hunting 0 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 monclient: found mon.noname-c 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.178+0000 7f22f9b01700 1 -- 172.21.15.142:0/129803596 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec009ae0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22f9b01700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 msgr2=0x7f22f406b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22f9b01700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c5d0 tx=0x7f22e8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e80117a0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:04.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: dump: 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: _finish_auth 0 2024-09-18T10:21:04.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e8011900 con 0x7f22f40a1cf0 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 5 monclient: authenticate success, global_id 75215 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _renew_subs 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f406da60 con 0x7f22f40a1cf0 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8011cc0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 10 monclient: dump: 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.201 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22f8aff700 4 set_mon_vals no callback set 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _renew_subs 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f409c3f0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:04.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f406dbf0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.182+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f22e8012630 con 0x7f22f40a1cf0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.186+0000 7f22e67fc700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 0x7f22d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.186+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f22e8048d90 con 0x7f22f40a1cf0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.186+0000 7f22fa302700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 0x7f22d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.186+0000 7f22fa302700 10 monclient: get_auth_request con 0x7f22d8035ac0 auth_method 0 2024-09-18T10:21:04.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.186+0000 7f22fa302700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 0x7f22d8037f70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009520 tx=0x7f22f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:04.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.194+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f22e8011e20 con 0x7f22f40a1cf0 2024-09-18T10:21:04.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.194+0000 7f22e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:04.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.194+0000 7f22e67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:04.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22fb304700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:21:04.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22fb304700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:04.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22fb304700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:04.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f22f4072a80 con 0x7f22f40a1cf0 2024-09-18T10:21:04.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22e67fc700 1 -- 172.21.15.142:0/129803596 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f22e80102e0 con 0x7f22f40a1cf0 2024-09-18T10:21:04.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:04.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.418+0000 7f22e67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:04.426 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":120,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:21:04.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 msgr2=0x7f22d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:04.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 0x7f22d8037f70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009520 tx=0x7f22f0008040 comp rx=0 tx=0).stop 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 10 monclient: shutdown 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 msgr2=0x7f22f406b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f22e800c5d0 tx=0x7f22e8007f60 comp rx=0 tx=0).stop 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 shutdown_connections 2024-09-18T10:21:04.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f22d8035ac0 0x7f22d8037f70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22f412f810 0x7f22f406b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22f40a1cf0 0x7f22f406b0a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 --2- 172.21.15.142:0/129803596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22f40a02d0 0x7f22f406ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:04.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 >> 172.21.15.142:0/129803596 conn(0x7f22f4071820 msgr2=0x7f22f40a4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:04.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 shutdown_connections 2024-09-18T10:21:04.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:04.422+0000 7f22fb304700 1 -- 172.21.15.142:0/129803596 wait complete. 2024-09-18T10:21:07.438 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:21:07.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: get_monmap_and_config 2024-09-18T10:21:07.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: build_initial_monmap 2024-09-18T10:21:07.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: monmap: 2024-09-18T10:21:07.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:07.545313+0000 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:07.545313+0000 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:07.548 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:07.549 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 Processor -- start 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 -- start start 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: init 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b300a1cf0 0x7f0b3012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-a con 0x7f0b300a1cf0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-b con 0x7f0b3012f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:07.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b300a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3009d990 con 0x7f0b300a02d0 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3009dc70 con 0x7f0b300a1cf0 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300a20c0 con 0x7f0b3012f810 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b3499c700 10 monclient(hunting): authenticate will time out at 1556.471267s 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b300a1cf0 0x7f0b3012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51102/0 (socket says 172.21.15.142:51102) 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 1 -- 172.21.15.142:0/585449824 learned_addr learned my addr 172.21.15.142:0/585449824 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:07.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): get_auth_request con 0x7f0b3012f810 auth_method 0 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f0b300a1cf0 auth_method 0 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:07.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.542+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_done global_id 124308 payload 274 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 msgr2=0x7f0b3009f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/585449824 >> v1:172.21.15.150:6789/0 conn(0x7f0b300a02d0 legacy=0x7f0b301320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 monclient: found mon.noname-a 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/585449824 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3009f9c0 con 0x7f0b300a1cf0 2024-09-18T10:21:07.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2effd700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 msgr2=0x7f0b3012f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2f7fe700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3012f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b180098e0 tx=0x7f0b180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0d2f5159081c37 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 1 -- 172.21.15.142:0/585449824 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800b850 con 0x7f0b300a1cf0 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:07.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: dump: 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 5 monclient: authenticate success, global_id 124308 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 1 -- 172.21.15.142:0/585449824 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1800b9b0 con 0x7f0b300a1cf0 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 4 set_mon_vals no callback set 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 1 -- 172.21.15.142:0/585449824 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800bd50 con 0x7f0b300a1cf0 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b2e7fc700 10 monclient: dump: 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:07.561 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 10 monclient: shutdown 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 1 -- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 msgr2=0x7f0b3012f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:07.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.546+0000 7f0b3499c700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3012f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b180098e0 tx=0x7f0b180091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- 172.21.15.142:0/585449824 shutdown_connections 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b3012f810 0x7f0b3009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 --2- 172.21.15.142:0/585449824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3012f030 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- 172.21.15.142:0/585449824 >> 172.21.15.142:0/585449824 conn(0x7f0b30071820 msgr2=0x7f0b300a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- 172.21.15.142:0/585449824 shutdown_connections 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- 172.21.15.142:0/585449824 wait complete. 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient: build_initial_monmap 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient: monmap: 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:07.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 Processor -- start 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- start start 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient: init 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-c con 0x7f0b300a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3009b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-a con 0x7f0b300a1cf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): picked mon.noname-b con 0x7f0b3012f810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3006c850 con 0x7f0b300a1cf0 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3009cdc0 con 0x7f0b3012f810 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3012ef50 con 0x7f0b300a02d0 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b3499c700 10 monclient(hunting): authenticate will time out at 1556.479842s 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.550+0000 7f0b2effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3009b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51114/0 (socket says 172.21.15.142:51114) 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/722611667 learned_addr learned my addr 172.21.15.142:0/722611667 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f0b300a02d0 auth_method 0 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): get_auth_request con 0x7f0b300a1cf0 auth_method 0 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:07.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 10 monclient(hunting): get_auth_request con 0x7f0b3012f810 auth_method 0 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_done global_id 75221 payload 824 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 msgr2=0x7f0b3009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 msgr2=0x7f0b3009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 msgr2=0x7f0b3009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:07.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2ffff700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 msgr2=0x7f0b3009b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3009b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/722611667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200094f0 con 0x7f0b300a02d0 2024-09-18T10:21:07.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2effd700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3009b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:07.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 msgr2=0x7f0b3009b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:07.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2f7fe700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0b180093f0 tx=0x7f0b18016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:07.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18000d10 con 0x7f0b300a02d0 2024-09-18T10:21:07.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:07.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: dump: 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: _finish_auth 0 2024-09-18T10:21:07.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1801b470 con 0x7f0b300a02d0 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 5 monclient: authenticate success, global_id 75221 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 10 monclient: _renew_subs 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b18009420 con 0x7f0b300a02d0 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1801b830 con 0x7f0b300a02d0 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b277fe700 10 monclient: dump: 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:07.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b2dffb700 4 set_mon_vals no callback set 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 10 monclient: _renew_subs 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:07.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.554+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3006cad0 con 0x7f0b300a02d0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b1801bbe0 con 0x7f0b300a02d0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b3499c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b3499c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b3499c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b30072a80 con 0x7f0b300a02d0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b277fe700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 0x7f0b10037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0b1804aad0 con 0x7f0b300a02d0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b2effd700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 0x7f0b10037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.558+0000 7f0b2effd700 10 monclient: get_auth_request con 0x7f0b10035ac0 auth_method 0 2024-09-18T10:21:07.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.562+0000 7f0b2effd700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 0x7f0b10037f70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0b3006acc0 tx=0x7f0b20009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:07.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.566+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0b180193b0 con 0x7f0b300a02d0 2024-09-18T10:21:07.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.566+0000 7f0b277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:07.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.566+0000 7f0b277fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:07.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:21:07.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0b300a1a90 con 0x7f0b300a02d0 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b277fe700 1 -- 172.21.15.142:0/722611667 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f0b1802d350 con 0x7f0b300a02d0 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b277fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:07.794 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":120,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:21:07.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 msgr2=0x7f0b10037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 0x7f0b10037f70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0b3006acc0 tx=0x7f0b20009250 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 10 monclient: shutdown 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 msgr2=0x7f0b3009b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0b180093f0 tx=0x7f0b18016000 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 shutdown_connections 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035ac0 0x7f0b10037f70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b3012f810 0x7f0b3009be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b300a1cf0 0x7f0b3009b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 --2- 172.21.15.142:0/722611667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b300a02d0 0x7f0b3009b3d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:07.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 >> 172.21.15.142:0/722611667 conn(0x7f0b30071820 msgr2=0x7f0b30132020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:07.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 shutdown_connections 2024-09-18T10:21:07.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:07.790+0000 7f0b3499c700 1 -- 172.21.15.142:0/722611667 wait complete. 2024-09-18T10:21:10.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:21:10.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 10 monclient: get_monmap_and_config 2024-09-18T10:21:10.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 10 monclient: build_initial_monmap 2024-09-18T10:21:10.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 10 monclient: monmap: 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:10.918682+0000 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:10.918682+0000 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:10.921 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 1 Processor -- start 2024-09-18T10:21:10.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 1 -- start start 2024-09-18T10:21:10.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.913+0000 7f66dd164700 10 monclient: init 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66d809cef0 0x7f66d80a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-a con 0x7f66d809cef0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66d809f9c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66d8137980 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d809e7d0 con 0x7f66d8137980 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d809d7e0 con 0x7f66d809f9c0 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d809dda0 con 0x7f66d809cef0 2024-09-18T10:21:10.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66dd164700 10 monclient(hunting): authenticate will time out at 1559.844363s 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66d809cef0 0x7f66d80a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d759e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f66d8137980 0x7f66d8139d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46068/0 (socket says 172.21.15.142:46068) 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d759e700 1 -- 172.21.15.142:0/1597408704 learned_addr learned my addr 172.21.15.142:0/1597408704 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d659c700 1 --1- 172.21.15.142:0/1597408704 >> v1:172.21.15.150:6789/0 conn(0x7f66d809f9c0 0x7f66d80a0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:55222/0 (socket says 172.21.15.142:55222) 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66d809cef0 0x7f66d80a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59316/0 (socket says 172.21.15.142:59316) 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): get_auth_request con 0x7f66d809cef0 auth_method 0 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:10.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d659c700 1 -- 172.21.15.142:0/1597408704 >> v1:172.21.15.150:6789/0 conn(0x7f66d809f9c0 legacy=0x7f66d80a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d759e700 1 -- 172.21.15.142:0/1597408704 >> v1:172.21.15.142:6789/0 conn(0x7f66d8137980 legacy=0x7f66d8139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_done global_id 124314 payload 274 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 -- 172.21.15.142:0/1597408704 >> v1:172.21.15.150:6789/0 conn(0x7f66d809f9c0 legacy=0x7f66d80a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 -- 172.21.15.142:0/1597408704 >> v1:172.21.15.142:6789/0 conn(0x7f66d8137980 legacy=0x7f66d8139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 monclient: found mon.noname-a 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:10.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.917+0000 7f66d6d9d700 1 -- 172.21.15.142:0/1597408704 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d809e080 con 0x7f66d809cef0 2024-09-18T10:21:10.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d6d9d700 1 -- 172.21.15.142:0/1597408704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809cef0 msgr2=0x7f66d80a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:10.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d6d9d700 1 --2- 172.21.15.142:0/1597408704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809cef0 0x7f66d80a01f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0099e0 tx=0x7f66cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406c1c8771da9cb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:10.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 1 -- 172.21.15.142:0/1597408704 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc00bba0 con 0x7f66d809cef0 2024-09-18T10:21:10.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:10.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:10.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: dump: 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: _finish_auth 0 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:10.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 5 monclient: authenticate success, global_id 124314 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 1 -- 172.21.15.142:0/1597408704 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc00bd20 con 0x7f66d809cef0 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 1 -- 172.21.15.142:0/1597408704 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc011670 con 0x7f66d809cef0 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 4 set_mon_vals no callback set 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66d5d9b700 10 monclient: dump: 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.931 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 10 monclient: shutdown 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 -- 172.21.15.142:0/1597408704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809cef0 msgr2=0x7f66d80a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 --2- 172.21.15.142:0/1597408704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809cef0 0x7f66d80a01f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0099e0 tx=0x7f66cc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 -- 172.21.15.142:0/1597408704 shutdown_connections 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 --2- 172.21.15.142:0/1597408704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809cef0 0x7f66d80a01f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:10.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 -- 172.21.15.142:0/1597408704 >> 172.21.15.142:0/1597408704 conn(0x7f66d80a5800 msgr2=0x7f66d80a5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 -- 172.21.15.142:0/1597408704 shutdown_connections 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.921+0000 7f66dd164700 1 -- 172.21.15.142:0/1597408704 wait complete. 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient: build_initial_monmap 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient: monmap: 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:10.933 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:10.934 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 Processor -- start 2024-09-18T10:21:10.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 -- start start 2024-09-18T10:21:10.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient: init 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66d809cef0 0x7f66d8141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-b con 0x7f66d809cef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-a con 0x7f66d809f9c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66d8137980 0x7f66d8142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): picked mon.noname-c con 0x7f66d8137980 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d8130ac0 con 0x7f66d809f9c0 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:10.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d81376f0 con 0x7f66d809cef0 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d812e9c0 con 0x7f66d8137980 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66dd164700 10 monclient(hunting): authenticate will time out at 1559.853835s 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59332/0 (socket says 172.21.15.142:59332) 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66d8137980 0x7f66d8142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 -- 172.21.15.142:0/277005270 learned_addr learned my addr 172.21.15.142:0/277005270 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): get_auth_request con 0x7f66d809f9c0 auth_method 0 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:10.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): get_auth_request con 0x7f66d8137980 auth_method 0 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66d809cef0 0x7f66d8141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): get_auth_request con 0x7f66d809cef0 auth_method 0 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:10.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient(hunting): handle_auth_done global_id 124317 payload 824 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 -- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66d8137980 msgr2=0x7f66d8142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66d8137980 0x7f66d8142870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 -- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66d809cef0 msgr2=0x7f66d8141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66d809cef0 0x7f66d8141df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient: _finish_hunting 0 2024-09-18T10:21:10.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 monclient: found mon.noname-a 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.925+0000 7f66d659c700 1 -- 172.21.15.142:0/277005270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc009420 con 0x7f66d809f9c0 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66d659c700 1 -- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 msgr2=0x7f66d8142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66d659c700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66c800c790 tx=0x7f66c8007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8005c80 con 0x7f66d809f9c0 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: dump: 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:10.943 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: _finish_auth 0 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c8005e00 con 0x7f66d809f9c0 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 5 monclient: authenticate success, global_id 124317 2024-09-18T10:21:10.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _renew_subs 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d812f6e0 con 0x7f66d809f9c0 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8007320 con 0x7f66d809f9c0 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 10 monclient: dump: 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:10.945 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66d559a700 4 set_mon_vals no callback set 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _renew_subs 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d812dbf0 con 0x7f66d809f9c0 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f66c801b3b0 con 0x7f66d809f9c0 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:10.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d80696a0 con 0x7f66d809f9c0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 0x7f66c4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f66c8048350 con 0x7f66d809f9c0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66d6d9d700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 0x7f66c4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.929+0000 7f66d6d9d700 10 monclient: get_auth_request con 0x7f66c4035ac0 auth_method 0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.933+0000 7f66d6d9d700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 0x7f66c4037f70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0099e0 tx=0x7f66cc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.937+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f66c800f260 con 0x7f66d809f9c0 2024-09-18T10:21:10.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.937+0000 7f66beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:10.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:10.937+0000 7f66beffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:11.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66dd164700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:21:11.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66dd164700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:11.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66dd164700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f66d812e7a0 con 0x7f66d809f9c0 2024-09-18T10:21:11.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66beffd700 1 -- 172.21.15.142:0/277005270 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f66c8017070 con 0x7f66d809f9c0 2024-09-18T10:21:11.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:11.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.157+0000 7f66beffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:11.165 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 msgr2=0x7f66c4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 0x7f66c4037f70 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0099e0 tx=0x7f66cc017000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 10 monclient: shutdown 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 msgr2=0x7f66d8142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66c800c790 tx=0x7f66c8007e50 comp rx=0 tx=0).stop 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 shutdown_connections 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f66c4035ac0 0x7f66c4037f70 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66d8137980 0x7f66d8142870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66d809f9c0 0x7f66d8142330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 --2- 172.21.15.142:0/277005270 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66d809cef0 0x7f66d8141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 >> 172.21.15.142:0/277005270 conn(0x7f66d80a5800 msgr2=0x7f66d809a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 shutdown_connections 2024-09-18T10:21:11.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.161+0000 7f66dd164700 1 -- 172.21.15.142:0/277005270 wait complete. 2024-09-18T10:21:11.177 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:21:11.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:21:11.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:21:11.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.277+0000 7fcaaade7700 10 monclient: get_monmap_and_config 2024-09-18T10:21:11.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.277+0000 7fcaaade7700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.277+0000 7fcaaade7700 10 monclient: monmap: 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:11.283488+0000 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:11.283488+0000 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:11.286 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:11.287 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:11.287 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:11.287 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:11.287 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 Processor -- start 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 -- start start 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient: init 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcaa406f3b0 0x7fcaa40989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-c con 0x7fcaa406f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 0x7fcaa4099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-a con 0x7fcaa4098f00 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcaa4133660 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa409c330 con 0x7fcaa4133660 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa4099860 con 0x7fcaa4098f00 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa409b030 con 0x7fcaa406f3b0 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaaade7700 10 monclient(hunting): authenticate will time out at 1560.208911s 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 0x7fcaa4099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 0x7fcaa4099320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59340/0 (socket says 172.21.15.142:59340) 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 1 -- 172.21.15.142:0/138244869 learned_addr learned my addr 172.21.15.142:0/138244869 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): get_auth_request con 0x7fcaa4098f00 auth_method 0 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/138244869 >> v2:172.21.15.150:3300/0 conn(0x7fcaa406f3b0 0x7fcaa40989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request con 0x7fcaa406f3b0 auth_method 0 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_done global_id 144103 payload 274 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 -- 172.21.15.142:0/138244869 >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 msgr2=0x7fcaa4099320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/138244869 >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 0x7fcaa4099320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 -- 172.21.15.142:0/138244869 >> v1:172.21.15.142:6789/0 conn(0x7fcaa4133660 legacy=0x7fcaa4135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 monclient: found mon.noname-c 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:21:11.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.281+0000 7fcaa9de5700 1 -- 172.21.15.142:0/138244869 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa409c530 con 0x7fcaa406f3b0 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa9de5700 1 -- 172.21.15.142:0/138244869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa406f3b0 msgr2=0x7fcaa40989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/138244869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa406f3b0 0x7fcaa40989c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4076de0 tx=0x7fca980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d63316ca0f824d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 1 -- 172.21.15.142:0/138244869 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98011840 con 0x7fcaa406f3b0 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: dump: 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.300 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 1 -- 172.21.15.142:0/138244869 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca980119a0 con 0x7fcaa406f3b0 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 1 -- 172.21.15.142:0/138244869 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98011d40 con 0x7fcaa406f3b0 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 5 monclient: authenticate success, global_id 144103 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 4 set_mon_vals no callback set 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaa8de3700 10 monclient: dump: 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 10 monclient: shutdown 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- 172.21.15.142:0/138244869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa406f3b0 msgr2=0x7fcaa40989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 --2- 172.21.15.142:0/138244869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa406f3b0 0x7fcaa40989c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4076de0 tx=0x7fca980091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- 172.21.15.142:0/138244869 shutdown_connections 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 --2- 172.21.15.142:0/138244869 >> v2:172.21.15.142:3300/0 conn(0x7fcaa4098f00 0x7fcaa4099320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 --2- 172.21.15.142:0/138244869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa406f3b0 0x7fcaa40989c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- 172.21.15.142:0/138244869 >> 172.21.15.142:0/138244869 conn(0x7fcaa4070ef0 msgr2=0x7fcaa40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- 172.21.15.142:0/138244869 shutdown_connections 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- 172.21.15.142:0/138244869 wait complete. 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 10 monclient: monmap: 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:11.304 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 Processor -- start 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 1 -- start start 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.285+0000 7fcaaade7700 10 monclient: init 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 0x7fcaa4141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-c con 0x7fcaa406f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa4098f00 0x7fcaa4142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-b con 0x7fcaa4098f00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 0x7fcaa4142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): picked mon.noname-a con 0x7fcaa4133660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4130030 con 0x7fcaa4133660 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa412f800 con 0x7fcaa4098f00 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa412fd50 con 0x7fcaa406f3b0 2024-09-18T10:21:11.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient(hunting): authenticate will time out at 1560.216358s 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 0x7fcaa4141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 0x7fcaa4142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 0x7fcaa4141b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51138/0 (socket says 172.21.15.142:51138) 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 1 -- 172.21.15.142:0/4208393583 learned_addr learned my addr 172.21.15.142:0/4208393583 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request con 0x7fcaa406f3b0 auth_method 0 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): get_auth_request con 0x7fcaa4133660 auth_method 0 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa4098f00 0x7fcaa4142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): get_auth_request con 0x7fcaa4098f00 auth_method 0 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa95e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient(hunting): handle_auth_done global_id 124323 payload 824 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 msgr2=0x7fcaa4141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 0x7fcaa4141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa4098f00 msgr2=0x7fcaa4142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaa4098f00 0x7fcaa4142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 monclient: found mon.noname-a 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca98009420 con 0x7fcaa4133660 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcaa406f3b0 0x7fcaa4141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 msgr2=0x7fcaa4142590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaa5e6700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 0x7fcaa4142590 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c810 tx=0x7fca9c002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c012840 con 0x7fcaa4133660 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: dump: 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.310 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c0129c0 con 0x7fcaa4133660 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 5 monclient: authenticate success, global_id 124323 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _renew_subs 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa412a390 con 0x7fcaa4133660 2024-09-18T10:21:11.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c012d80 con 0x7fcaa4133660 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca95ffb700 10 monclient: dump: 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fca97fff700 4 set_mon_vals no callback set 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _renew_subs 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.289+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa409cf90 con 0x7fcaa4133660 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaaade7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fca9c011590 con 0x7fcaa4133660 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaaade7700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 0x7fcaa4126980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa412fc10 con 0x7fcaa41265b0 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 0x7fcaa4126980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fca95ffb700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca88035b10 0x7fca88037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient: get_auth_request con 0x7fcaa41265b0 auth_method 0 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fca9c048e70 con 0x7fcaa4133660 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_done global_id 124326 payload 824 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa95e4700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca88035b10 0x7fca88037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa9de5700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcaa4072a80 con 0x7fcaa41265b0 2024-09-18T10:21:11.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.293+0000 7fcaa95e4700 10 monclient: get_auth_request con 0x7fca88035b10 auth_method 0 2024-09-18T10:21:11.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fcaa9de5700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 msgr2=0x7fcaa4126980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 0x7fcaa4126980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98021070 con 0x7fcaa41265b0 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 10 monclient: dump: 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.317 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fcaa95e4700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca88035b10 0x7fca88037fc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4068f60 tx=0x7fcaa0009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fca9801c660 con 0x7fcaa41265b0 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 msgr2=0x7fcaa4126980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 0x7fcaa4126980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca98016000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 msgr2=0x7fcaa4126980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.297+0000 7fca95ffb700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa41265b0 0x7fcaa4126980 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca98016000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaaade7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaaade7700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 0x7fcaa41290d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaaade7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa40648a0 con 0x7fcaa4126ec0 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 0x7fcaa41290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient: get_auth_request con 0x7fcaa4126ec0 auth_method 0 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 10 monclient(hunting): handle_auth_done global_id 124329 payload 824 2024-09-18T10:21:11.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.309+0000 7fcaa9de5700 1 -- 172.21.15.142:0/4208393583 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcaa4130910 con 0x7fcaa4126ec0 2024-09-18T10:21:11.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaa9de5700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 msgr2=0x7fcaa41290d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaa9de5700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 0x7fcaa41290d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca9801e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98032480 con 0x7fcaa4126ec0 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 10 monclient: dump: 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.321 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fca980325e0 con 0x7fcaa4126ec0 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 msgr2=0x7fcaa41290d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 0x7fcaa41290d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca9801e070 comp rx=0 tx=0).stop 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 msgr2=0x7fcaa41290d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fca95ffb700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4126ec0 0x7fcaa41290d0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fca980030c0 tx=0x7fca9801e070 comp rx=0 tx=0).stop 2024-09-18T10:21:11.322 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:11.323 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:11.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:11.325 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:11.326 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.327 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:11.331 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.333 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:21:11.334 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.335 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:11.335 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca88035b10 msgr2=0x7fca88037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fca88035b10 0x7fca88037fc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4068f60 tx=0x7fcaa0009250 comp rx=0 tx=0).stop 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 10 monclient: shutdown 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 msgr2=0x7fcaa4142590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 --2- 172.21.15.142:0/4208393583 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcaa4133660 0x7fcaa4142590 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c810 tx=0x7fca9c002620 comp rx=0 tx=0).stop 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 reap_dead start 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaa5e6700 1 -- 172.21.15.142:0/4208393583 reap_dead start 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 shutdown_connections 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 >> 172.21.15.142:0/4208393583 conn(0x7fcaa4070ef0 msgr2=0x7fcaa40a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 shutdown_connections 2024-09-18T10:21:11.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.313+0000 7fcaaade7700 1 -- 172.21.15.142:0/4208393583 wait complete. 2024-09-18T10:21:11.337 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:11.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:21:11.337 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:21:11.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.437+0000 7fdf1e135700 10 monclient: get_monmap_and_config 2024-09-18T10:21:11.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.437+0000 7fdf1e135700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.437+0000 7fdf1e135700 10 monclient: monmap: 2024-09-18T10:21:11.445 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:11.443385+0000 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:11.443385+0000 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:11.446 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 Processor -- start 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 -- start start 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient: init 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf180a00e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdf1809cdf0 0x7fdf1809d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-c con 0x7fdf1809cdf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf1809d720 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf18069dc0 con 0x7fdf180a00e0 2024-09-18T10:21:11.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf18069ae0 con 0x7fdf1809d720 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1806ad00 con 0x7fdf1809cdf0 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1e135700 10 monclient(hunting): authenticate will time out at 1560.369371s 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1d133700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdf180a00e0 0x7fdf1806b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46088/0 (socket says 172.21.15.142:46088) 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1d133700 1 -- 172.21.15.142:0/3239081662 learned_addr learned my addr 172.21.15.142:0/3239081662 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 1 --2- 172.21.15.142:0/3239081662 >> v2:172.21.15.150:3300/0 conn(0x7fdf1809cdf0 0x7fdf1809d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): get_auth_request con 0x7fdf1809cdf0 auth_method 0 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.441+0000 7fdf1d133700 1 -- 172.21.15.142:0/3239081662 >> v1:172.21.15.142:6789/0 conn(0x7fdf180a00e0 legacy=0x7fdf1806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 1 -- 172.21.15.142:0/3239081662 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33716762 0 0) 0x7fdf18069dc0 con 0x7fdf180a00e0 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient(hunting): my global_id is 124332 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1d934700 1 -- 172.21.15.142:0/3239081662 >> v1:172.21.15.150:6789/0 conn(0x7fdf1809d720 legacy=0x7fdf18139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 1 -- 172.21.15.142:0/3239081662 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04003570 con 0x7fdf180a00e0 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 10 monclient(hunting): handle_auth_done global_id 144109 payload 274 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 -- 172.21.15.142:0/3239081662 >> v1:172.21.15.150:6789/0 conn(0x7fdf1809d720 legacy=0x7fdf18139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 -- 172.21.15.142:0/3239081662 >> v1:172.21.15.142:6789/0 conn(0x7fdf180a00e0 legacy=0x7fdf1806b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 monclient: found mon.noname-c 2024-09-18T10:21:11.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:21:11.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 -- 172.21.15.142:0/3239081662 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1813cab0 con 0x7fdf1809cdf0 2024-09-18T10:21:11.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 -- 172.21.15.142:0/3239081662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf1809cdf0 msgr2=0x7fdf1809d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1c932700 1 --2- 172.21.15.142:0/3239081662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf1809cdf0 0x7fdf1809d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf080099e0 tx=0x7fdf080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea3edc69e12b354 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 1 -- 172.21.15.142:0/3239081662 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800ba70 con 0x7fdf1809cdf0 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: dump: 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.454 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 5 monclient: authenticate success, global_id 144109 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 1 -- 172.21.15.142:0/3239081662 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0800bbd0 con 0x7fdf1809cdf0 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 1 -- 172.21.15.142:0/3239081662 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011420 con 0x7fdf1809cdf0 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 4 set_mon_vals no callback set 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf0ffff700 10 monclient: dump: 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 10 monclient: shutdown 2024-09-18T10:21:11.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 1 -- 172.21.15.142:0/3239081662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf1809cdf0 msgr2=0x7fdf1809d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.445+0000 7fdf1e135700 1 --2- 172.21.15.142:0/3239081662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf1809cdf0 0x7fdf1809d1e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf080099e0 tx=0x7fdf080091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- 172.21.15.142:0/3239081662 shutdown_connections 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 --2- 172.21.15.142:0/3239081662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf1809cdf0 0x7fdf1809d1e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- 172.21.15.142:0/3239081662 >> 172.21.15.142:0/3239081662 conn(0x7fdf18096a50 msgr2=0x7fdf18071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- 172.21.15.142:0/3239081662 shutdown_connections 2024-09-18T10:21:11.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- 172.21.15.142:0/3239081662 wait complete. 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient: monmap: 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.460 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 Processor -- start 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- start start 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient: init 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1809cdf0 0x7fdf180992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-c con 0x7fdf1809cdf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 0x7fdf181343f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-a con 0x7fdf1809d720 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180a00e0 0x7fdf18134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): picked mon.noname-b con 0x7fdf180a00e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1806b7f0 con 0x7fdf1809d720 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1813c200 con 0x7fdf180a00e0 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181359a0 con 0x7fdf1809cdf0 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1e135700 10 monclient(hunting): authenticate will time out at 1560.378298s 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 0x7fdf181343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1809cdf0 0x7fdf180992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 0x7fdf181343f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59386/0 (socket says 172.21.15.142:59386) 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 -- 172.21.15.142:0/2787222853 learned_addr learned my addr 172.21.15.142:0/2787222853 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): get_auth_request con 0x7fdf1809d720 auth_method 0 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): get_auth_request con 0x7fdf1809cdf0 auth_method 0 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180a00e0 0x7fdf18134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 10 monclient(hunting): get_auth_request con 0x7fdf180a00e0 auth_method 0 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient(hunting): handle_auth_done global_id 124335 payload 824 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1809cdf0 msgr2=0x7fdf180992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1809cdf0 0x7fdf180992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180a00e0 msgr2=0x7fdf18134930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180a00e0 0x7fdf18134930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 monclient: found mon.noname-a 2024-09-18T10:21:11.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1c932700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf08009420 con 0x7fdf1809d720 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.449+0000 7fdf1d934700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf180a00e0 0x7fdf18134930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1c932700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 msgr2=0x7fdf181343f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1c932700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 0x7fdf181343f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdf18069f50 tx=0x7fdf08017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0801f070 con 0x7fdf1809d720 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: dump: 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.465 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 5 monclient: authenticate success, global_id 124335 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf08002f70 con 0x7fdf1809d720 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08025670 con 0x7fdf1809d720 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 monclient: _renew_subs 2024-09-18T10:21:11.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18136780 con 0x7fdf1809d720 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0effd700 4 set_mon_vals no callback set 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 10 monclient: dump: 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.467 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 monclient: _renew_subs 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1813a6c0 con 0x7fdf1809d720 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 0x7fdf1802ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18136c00 con 0x7fdf18134e70 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 0x7fdf1802ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient: get_auth_request con 0x7fdf18134e70 auth_method 0 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdf08023070 con 0x7fdf1809d720 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.453+0000 7fdf0d7fa700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdefc035ac0 0x7fdefc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdf0804a060 con 0x7fdf1809d720 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d133700 10 monclient(hunting): handle_auth_done global_id 75233 payload 824 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d133700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf18136d90 con 0x7fdf18134e70 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d934700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdefc035ac0 0x7fdefc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d934700 10 monclient: get_auth_request con 0x7fdefc035ac0 auth_method 0 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d133700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 msgr2=0x7fdf1802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d133700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 0x7fdf1802ae80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf140014d0 tx=0x7fdf14007bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14006030 con 0x7fdf18134e70 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 10 monclient: dump: 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf1d934700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdefc035ac0 0x7fdefc037f70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdf1809f160 tx=0x7fdf10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdf14006190 con 0x7fdf18134e70 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:11.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 msgr2=0x7fdf1802ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 0x7fdf1802ae80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf140014d0 tx=0x7fdf14007bb0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 msgr2=0x7fdf1802ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.457+0000 7fdf0d7fa700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf18134e70 0x7fdf1802ae80 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdf140014d0 tx=0x7fdf14007bb0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1e135700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1e135700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 0x7fdf1805ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1e135700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18064810 con 0x7fdf1805e890 2024-09-18T10:21:11.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 0x7fdf1805ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient: get_auth_request con 0x7fdf1805e890 auth_method 0 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 10 monclient(hunting): handle_auth_done global_id 75236 payload 824 2024-09-18T10:21:11.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 1 -- 172.21.15.142:0/2787222853 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdf18135240 con 0x7fdf1805e890 2024-09-18T10:21:11.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 msgr2=0x7fdf1805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf1d133700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 0x7fdf1805ec90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf14016000 tx=0x7fdf14017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14015e10 con 0x7fdf1805e890 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.473+0000 7fdf0d7fa700 10 monclient: dump: 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.482 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fdf14022620 con 0x7fdf1805e890 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 msgr2=0x7fdf1805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 0x7fdf1805ec90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf14016000 tx=0x7fdf14017000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 msgr2=0x7fdf1805ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf0d7fa700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf1805e890 0x7fdf1805ec90 secure :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf14016000 tx=0x7fdf14017000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:11.483 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.484 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.485 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:11.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:11.487 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.488 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.490 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.492 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:11.493 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.494 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.495 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:21:11.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:11.497 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdefc035ac0 msgr2=0x7fdefc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdefc035ac0 0x7fdefc037f70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fdf1809f160 tx=0x7fdf10009040 comp rx=0 tx=0).stop 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 10 monclient: shutdown 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 msgr2=0x7fdf181343f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 --2- 172.21.15.142:0/2787222853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf1809d720 0x7fdf181343f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdf18069f50 tx=0x7fdf08017000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1d934700 1 -- 172.21.15.142:0/2787222853 reap_dead start 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1d934700 1 -- 172.21.15.142:0/2787222853 reap_dead start 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 shutdown_connections 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 >> 172.21.15.142:0/2787222853 conn(0x7fdf18096a50 msgr2=0x7fdf180a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 shutdown_connections 2024-09-18T10:21:11.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.477+0000 7fdf1e135700 1 -- 172.21.15.142:0/2787222853 wait complete. 2024-09-18T10:21:11.500 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:11.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:21:11.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:21:11.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: get_monmap_and_config 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: monmap: 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:11.612852+0000 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:11.612852+0000 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:11.615 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 Processor -- start 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 -- start start 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: init 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): picked mon.noname-a con 0x7f245809a6a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:11.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 0x7f245809ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): picked mon.noname-b con 0x7f245809aa70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f245809b380 0x7f245813c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): picked mon.noname-c con 0x7f245809b380 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245809dae0 con 0x7f245809a6a0 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2458099bc0 con 0x7f245809b380 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24580998e0 con 0x7f245809aa70 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245e780700 10 monclient(hunting): authenticate will time out at 1560.538523s 2024-09-18T10:21:11.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 0x7f245809ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59402/0 (socket says 172.21.15.142:59402) 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): get_auth_request con 0x7f245809aa70 auth_method 0 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 1 -- 172.21.15.142:0/720956694 learned_addr learned my addr 172.21.15.142:0/720956694 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): get_auth_request con 0x7f245809a6a0 auth_method 0 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.150:3300/0 conn(0x7f245809b380 0x7f245813c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): get_auth_request con 0x7f245809b380 auth_method 0 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient(hunting): handle_auth_done global_id 75239 payload 274 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 -- 172.21.15.142:0/720956694 >> v2:172.21.15.150:3300/0 conn(0x7f245809b380 msgr2=0x7f245813c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.150:3300/0 conn(0x7f245809b380 0x7f245813c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 -- 172.21.15.142:0/720956694 >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 msgr2=0x7f245806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 monclient: found mon.noname-b 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245cf7d700 1 -- 172.21.15.142:0/720956694 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245809b800 con 0x7f245809aa70 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.609+0000 7f245d77e700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245cf7d700 1 -- 172.21.15.142:0/720956694 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 msgr2=0x7f245809ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245cf7d700 1 --2- 172.21.15.142:0/720956694 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 0x7f245809ae40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f24580a2540 tx=0x7f24540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=720c46fdbfe23ca7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 1 -- 172.21.15.142:0/720956694 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454011840 con 0x7f245809aa70 2024-09-18T10:21:11.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: dump: 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 1 -- 172.21.15.142:0/720956694 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540119c0 con 0x7f245809aa70 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 5 monclient: authenticate success, global_id 75239 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 1 -- 172.21.15.142:0/720956694 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454011d80 con 0x7f245809aa70 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 4 set_mon_vals no callback set 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f244ffff700 10 monclient: dump: 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.626 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.627 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 10 monclient: shutdown 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 -- 172.21.15.142:0/720956694 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 msgr2=0x7f245809ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 --2- 172.21.15.142:0/720956694 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 0x7f245809ae40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f24580a2540 tx=0x7f24540091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 -- 172.21.15.142:0/720956694 shutdown_connections 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.150:3300/0 conn(0x7f245809b380 0x7f245813c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 --2- 172.21.15.142:0/720956694 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809aa70 0x7f245809ae40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 --2- 172.21.15.142:0/720956694 >> v2:172.21.15.142:3300/0 conn(0x7f245809a6a0 0x7f245806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 -- 172.21.15.142:0/720956694 >> 172.21.15.142:0/720956694 conn(0x7f2458070ef0 msgr2=0x7f24580712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 -- 172.21.15.142:0/720956694 shutdown_connections 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.613+0000 7f245e780700 1 -- 172.21.15.142:0/720956694 wait complete. 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient: monmap: 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.635 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 Processor -- start 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 -- start start 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient: init 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 0x7f245806c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): picked mon.noname-a con 0x7f245809a6a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245809aa70 0x7f245806c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): picked mon.noname-b con 0x7f245809aa70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 0x7f245806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): picked mon.noname-c con 0x7f245809b380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245813bf20 con 0x7f245809a6a0 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580a06a0 con 0x7f245809aa70 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2458126e70 con 0x7f245809b380 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245e780700 10 monclient(hunting): authenticate will time out at 1560.547016s 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 0x7f245806c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 0x7f245806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 0x7f245806c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59408/0 (socket says 172.21.15.142:59408) 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 1 -- 172.21.15.142:0/1603907092 learned_addr learned my addr 172.21.15.142:0/1603907092 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): get_auth_request con 0x7f245809a6a0 auth_method 0 2024-09-18T10:21:11.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): get_auth_request con 0x7f245809b380 auth_method 0 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245809aa70 0x7f245806c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): get_auth_request con 0x7f245809aa70 auth_method 0 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245cf7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient(hunting): handle_auth_done global_id 75242 payload 824 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245809aa70 msgr2=0x7f245806c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245809aa70 0x7f245806c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 msgr2=0x7f245806c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 0x7f245806c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 monclient: found mon.noname-c 2024-09-18T10:21:11.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480094f0 con 0x7f245809b380 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.617+0000 7f245d77e700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f245809a6a0 0x7f245806c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 msgr2=0x7f245806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245df7f700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 0x7f245806ccb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2450009e50 tx=0x7f245000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245000ccb0 con 0x7f245809b380 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: dump: 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 5 monclient: authenticate success, global_id 75242 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 monclient: _renew_subs 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245000f040 con 0x7f245809b380 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:11.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2454009420 con 0x7f245809b380 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24500146d0 con 0x7f245809b380 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244f7fe700 4 set_mon_vals no callback set 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 10 monclient: dump: 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.642 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 monclient: _renew_subs 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24581268b0 con 0x7f245809b380 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 0x7f2458092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24580648a0 con 0x7f245806d1f0 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2450014a40 con 0x7f245809b380 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.621+0000 7f245d77e700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 0x7f2458092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient: get_auth_request con 0x7f245806d1f0 auth_method 0 2024-09-18T10:21:11.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f243c035ac0 0x7f243c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2450049c30 con 0x7f245809b380 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245cf7d700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f243c035ac0 0x7f243c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245cf7d700 10 monclient: get_auth_request con 0x7f243c035ac0 auth_method 0 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 10 monclient(hunting): handle_auth_done global_id 144124 payload 824 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2458072a80 con 0x7f245806d1f0 2024-09-18T10:21:11.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 msgr2=0x7f2458092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245d77e700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 0x7f2458092800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f2448009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448014990 con 0x7f245806d1f0 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f245cf7d700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f243c035ac0 0x7f243c037f70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f24580a2540 tx=0x7f2454017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.625+0000 7f244d7fa700 10 monclient: dump: 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.645 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f2448014af0 con 0x7f245806d1f0 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 msgr2=0x7f2458092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 0x7f2458092800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f2448009250 comp rx=0 tx=0).stop 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 msgr2=0x7f2458092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.629+0000 7f244d7fa700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f245806d1f0 0x7f2458092800 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f2448009250 comp rx=0 tx=0).stop 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245e780700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245e780700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 0x7f2458093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245e780700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245806db20 con 0x7f2458092d40 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 0x7f2458093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient: get_auth_request con 0x7f2458092d40 auth_method 0 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.637+0000 7f245d77e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f245d77e700 10 monclient(hunting): handle_auth_done global_id 144127 payload 824 2024-09-18T10:21:11.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f245d77e700 1 -- 172.21.15.142:0/1603907092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f245806e6f0 con 0x7f2458092d40 2024-09-18T10:21:11.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f245d77e700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 msgr2=0x7f2458093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f245d77e700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 0x7f2458093140 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f244801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448024040 con 0x7f2458092d40 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 10 monclient: dump: 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f24480137c0 con 0x7f2458092d40 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 msgr2=0x7f2458093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 0x7f2458093140 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f244801c000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 msgr2=0x7f2458093140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.641+0000 7f244d7fa700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2458092d40 0x7f2458093140 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f244800f470 tx=0x7f244801c000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:11.651 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:11.652 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.653 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:11.654 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:11.655 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:11.656 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.657 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.658 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.659 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.660 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.661 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:11.662 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:11.663 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:11.663 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:11.663 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:11.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f243c035ac0 msgr2=0x7f243c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f243c035ac0 0x7f243c037f70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f24580a2540 tx=0x7f2454017000 comp rx=0 tx=0).stop 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 10 monclient: shutdown 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 msgr2=0x7f245806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 --2- 172.21.15.142:0/1603907092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f245809b380 0x7f245806ccb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2450009e50 tx=0x7f245000b040 comp rx=0 tx=0).stop 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 reap_dead start 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245df7f700 1 -- 172.21.15.142:0/1603907092 reap_dead start 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 shutdown_connections 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 >> 172.21.15.142:0/1603907092 conn(0x7f2458070ef0 msgr2=0x7f245809f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 shutdown_connections 2024-09-18T10:21:11.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.645+0000 7f245e780700 1 -- 172.21.15.142:0/1603907092 wait complete. 2024-09-18T10:21:11.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:11.666 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:21:11.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:21:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.765+0000 7f33137b5700 10 monclient: get_monmap_and_config 2024-09-18T10:21:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.765+0000 7f33137b5700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient: monmap: 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:11.771818+0000 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:11.771818+0000 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:11.774 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 Processor -- start 2024-09-18T10:21:11.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 -- start start 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient: init 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f330c09cdd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 0x7f330c136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-b con 0x7f330c134a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 0x7f330c139750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-c con 0x7f330c137370 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c09d1a0 con 0x7f330c09cdd0 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c09d780 con 0x7f330c137370 2024-09-18T10:21:11.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c09dd40 con 0x7f330c134a50 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f33137b5700 10 monclient(hunting): authenticate will time out at 1560.697569s 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 0x7f330c136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3311551700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f330c09cdd0 0x7f330c09e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46098/0 (socket says 172.21.15.142:46098) 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3311551700 1 -- 172.21.15.142:0/4010132982 learned_addr learned my addr 172.21.15.142:0/4010132982 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): get_auth_request con 0x7f330c134a50 auth_method 0 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3311d52700 1 --2- 172.21.15.142:0/4010132982 >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 0x7f330c139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient(hunting): handle_auth_done global_id 75245 payload 274 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 -- 172.21.15.142:0/4010132982 >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 msgr2=0x7f330c139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 --2- 172.21.15.142:0/4010132982 >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 0x7f330c139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 -- 172.21.15.142:0/4010132982 >> v1:172.21.15.142:6789/0 conn(0x7f330c09cdd0 legacy=0x7f330c09e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 monclient: found mon.noname-b 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:11.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3310d50700 1 -- 172.21.15.142:0/4010132982 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0a6710 con 0x7f330c134a50 2024-09-18T10:21:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3311d52700 10 monclient: get_auth_request con 0x7f330c137370 auth_method 0 2024-09-18T10:21:11.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.769+0000 7f3311d52700 1 --2- 172.21.15.142:0/4010132982 >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 0x7f330c139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3310d50700 1 -- 172.21.15.142:0/4010132982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 msgr2=0x7f330c136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3310d50700 1 --2- 172.21.15.142:0/4010132982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 0x7f330c136e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3304009e60 tx=0x7f3304009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb9272c22933195 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 1 -- 172.21.15.142:0/4010132982 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330400e070 con 0x7f330c134a50 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: dump: 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.784 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 5 monclient: authenticate success, global_id 75245 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 1 -- 172.21.15.142:0/4010132982 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304012430 con 0x7f330c134a50 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 4 set_mon_vals no callback set 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 1 -- 172.21.15.142:0/4010132982 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33040127d0 con 0x7f330c134a50 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f3303fff700 10 monclient: dump: 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.786 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 10 monclient: shutdown 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 -- 172.21.15.142:0/4010132982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 msgr2=0x7f330c136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 --2- 172.21.15.142:0/4010132982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 0x7f330c136e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3304009e60 tx=0x7f3304009780 comp rx=0 tx=0).stop 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 -- 172.21.15.142:0/4010132982 shutdown_connections 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 --2- 172.21.15.142:0/4010132982 >> v2:172.21.15.150:3300/0 conn(0x7f330c137370 0x7f330c139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 --2- 172.21.15.142:0/4010132982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c134a50 0x7f330c136e30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 -- 172.21.15.142:0/4010132982 >> 172.21.15.142:0/4010132982 conn(0x7f330c071680 msgr2=0x7f330c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 -- 172.21.15.142:0/4010132982 shutdown_connections 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 1 -- 172.21.15.142:0/4010132982 wait complete. 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 10 monclient: build_initial_monmap 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.773+0000 7f33137b5700 10 monclient: monmap: 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:11.788 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 Processor -- start 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 -- start start 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient: init 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 0x7f330c09bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-b con 0x7f330c09cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:11.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-a con 0x7f330c134a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c137370 0x7f330c06b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): picked mon.noname-c con 0x7f330c137370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c137100 con 0x7f330c134a50 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c06de10 con 0x7f330c09cdd0 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c06db30 con 0x7f330c137370 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:11.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f33137b5700 10 monclient(hunting): authenticate will time out at 1560.705477s 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:59412/0 (socket says 172.21.15.142:59412) 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 -- 172.21.15.142:0/2605812981 learned_addr learned my addr 172.21.15.142:0/2605812981 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): get_auth_request con 0x7f330c134a50 auth_method 0 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c137370 0x7f330c06b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): get_auth_request con 0x7f330c137370 auth_method 0 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 0x7f330c09bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 10 monclient(hunting): get_auth_request con 0x7f330c09cdd0 auth_method 0 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:11.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient(hunting): handle_auth_done global_id 124344 payload 824 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 -- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c137370 msgr2=0x7f330c06b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c137370 0x7f330c06b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 -- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 msgr2=0x7f330c09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 0x7f330c09bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient: _finish_hunting 0 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 monclient: found mon.noname-a 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3310d50700 1 -- 172.21.15.142:0/2605812981 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3304009a20 con 0x7f330c134a50 2024-09-18T10:21:11.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.777+0000 7f3311551700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 0x7f330c09bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:21:11.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f3310d50700 1 -- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 msgr2=0x7f330c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:11.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f3310d50700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f330400b6c0 tx=0x7f3304018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304012dd0 con 0x7f330c134a50 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: dump: 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: _finish_auth 0 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304021070 con 0x7f330c134a50 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 5 monclient: authenticate success, global_id 124344 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _renew_subs 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c06d850 con 0x7f330c134a50 2024-09-18T10:21:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304026680 con 0x7f330c134a50 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33037fe700 4 set_mon_vals no callback set 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 10 monclient: dump: 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:11.798 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _renew_subs 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c06ca90 con 0x7f330c134a50 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3304023030 con 0x7f330c134a50 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 0x7f32f0037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f330404afe0 con 0x7f330c134a50 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c136580 con 0x7f330c134a50 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f3311551700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 0x7f32f0037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.781+0000 7f3311551700 10 monclient: get_auth_request con 0x7f32f0035b10 auth_method 0 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.785+0000 7f3311551700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 0x7f32f0037fc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f32fc005300 tx=0x7f32fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.789+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3304011b70 con 0x7f330c134a50 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.789+0000 7f33017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:11.789+0000 7f33017fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:12.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33137b5700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:21:12.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33137b5700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:21:12.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33137b5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:12.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f330c0696a0 con 0x7f330c134a50 2024-09-18T10:21:12.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33017fa700 1 -- 172.21.15.142:0/2605812981 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f3304011390 con 0x7f330c134a50 2024-09-18T10:21:12.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33017fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:21:12.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.017+0000 7f33017fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=7,logm=57} crc {auth=1831671034,config=3140514009,health=2295952920,kv=1523155648,logm=1180364910}) 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=3251403368,config=3140514009,health=1350913232,kv=4261215878,logm=3766795357}) 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=7,logm=57} crc {auth=1831671034,config=3140514009,health=2295952920,kv=1523155648,logm=1180364910}) 2024-09-18T10:21:12.026 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=9,kv=8,logm=55} crc {auth=3923141778,config=3140514010,health=2706963262,kv=4261215878,logm=536323013}) 2024-09-18T10:21:12.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 msgr2=0x7f32f0037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:12.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 0x7f32f0037fc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f32fc005300 tx=0x7f32fc009040 comp rx=0 tx=0).stop 2024-09-18T10:21:12.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 10 monclient: shutdown 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 msgr2=0x7f330c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f330400b6c0 tx=0x7f3304018000 comp rx=0 tx=0).stop 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 shutdown_connections 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32f0035b10 0x7f32f0037fc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:12.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f330c137370 0x7f330c06b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:12.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f330c134a50 0x7f330c06abb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:12.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 --2- 172.21.15.142:0/2605812981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f330c09cdd0 0x7f330c09bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:12.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 >> 172.21.15.142:0/2605812981 conn(0x7f330c071680 msgr2=0x7f330c134030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:12.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 shutdown_connections 2024-09-18T10:21:12.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f33137b5700 1 -- 172.21.15.142:0/2605812981 wait complete. 2024-09-18T10:21:12.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.030 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195732066}) 2024-09-18T10:21:12.030 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896595062}) 2024-09-18T10:21:12.030 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.030 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195732066}) 2024-09-18T10:21:12.030 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.021+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830249604}) 2024-09-18T10:21:12.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239648168}) 2024-09-18T10:21:12.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=932108895}) 2024-09-18T10:21:12.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239648168}) 2024-09-18T10:21:12.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.025+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226413162}) 2024-09-18T10:21:12.034 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.034 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321500963}) 2024-09-18T10:21:12.034 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=413128984}) 2024-09-18T10:21:12.034 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.034 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1321500963}) 2024-09-18T10:21:12.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3739864321}) 2024-09-18T10:21:12.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.037 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957417790}) 2024-09-18T10:21:12.037 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2060987869}) 2024-09-18T10:21:12.037 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.037 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2957417790}) 2024-09-18T10:21:12.037 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.029+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=538404536}) 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143278969}) 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498342843}) 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143278969}) 2024-09-18T10:21:12.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765820633}) 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540892822}) 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770781780}) 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=540892822}) 2024-09-18T10:21:12.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641541027}) 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528113191}) 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599636891}) 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528113191}) 2024-09-18T10:21:12.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.033+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485633246}) 2024-09-18T10:21:12.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902577654}) 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4146925900}) 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902577654}) 2024-09-18T10:21:12.048 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.041+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952168545}) 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867699742}) 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755300299}) 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2867699742}) 2024-09-18T10:21:12.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716638442}) 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284311992}) 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254266948}) 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284311992}) 2024-09-18T10:21:12.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.045+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235563548}) 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759564098}) 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2743524199}) 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3759564098}) 2024-09-18T10:21:12.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223525224}) 2024-09-18T10:21:12.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676965434}) 2024-09-18T10:21:12.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2245280528}) 2024-09-18T10:21:12.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1676965434}) 2024-09-18T10:21:12.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009103944}) 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885882638}) 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443051704}) 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1885882638}) 2024-09-18T10:21:12.057 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.049+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944275636}) 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060433000}) 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262593182}) 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060433000}) 2024-09-18T10:21:12.059 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927809437}) 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651678147}) 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639849782}) 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651678147}) 2024-09-18T10:21:12.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.053+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433774625}) 2024-09-18T10:21:12.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.063 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677198325}) 2024-09-18T10:21:12.063 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430654568}) 2024-09-18T10:21:12.063 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.063 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=677198325}) 2024-09-18T10:21:12.063 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1491512431}) 2024-09-18T10:21:12.064 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.064 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772758280}) 2024-09-18T10:21:12.064 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2954852526}) 2024-09-18T10:21:12.065 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.065 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772758280}) 2024-09-18T10:21:12.065 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.057+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118234294}) 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377737881}) 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667520942}) 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377737881}) 2024-09-18T10:21:12.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.061+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2224164031}) 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095167294}) 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236772376}) 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095167294}) 2024-09-18T10:21:12.576 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.569+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867848615}) 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=587388769}) 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2718914269}) 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=587388769}) 2024-09-18T10:21:12.578 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=446610590}) 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876313459}) 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008041393}) 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876313459}) 2024-09-18T10:21:12.581 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.573+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916177509}) 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511210703}) 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3708158470}) 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511210703}) 2024-09-18T10:21:12.583 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.577+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149623040}) 2024-09-18T10:21:12.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.585 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=176988164}) 2024-09-18T10:21:12.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830198863}) 2024-09-18T10:21:12.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=176988164}) 2024-09-18T10:21:12.586 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616719607}) 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291617387}) 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2034729929}) 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291617387}) 2024-09-18T10:21:12.588 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.581+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125140093}) 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784496185}) 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4197090525}) 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784496185}) 2024-09-18T10:21:12.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717476649}) 2024-09-18T10:21:12.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2096609399}) 2024-09-18T10:21:12.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895261730}) 2024-09-18T10:21:12.592 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2096609399}) 2024-09-18T10:21:12.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.585+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1185376308}) 2024-09-18T10:21:12.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218355061}) 2024-09-18T10:21:12.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=711658635}) 2024-09-18T10:21:12.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218355061}) 2024-09-18T10:21:12.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332514376}) 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547091213}) 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386668013}) 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547091213}) 2024-09-18T10:21:12.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1819643981}) 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311984256}) 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2414323217}) 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311984256}) 2024-09-18T10:21:12.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1695096141}) 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706037357}) 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176869682}) 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706037357}) 2024-09-18T10:21:12.597 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.589+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005864283}) 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888977072}) 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566356450}) 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888977072}) 2024-09-18T10:21:12.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458167910}) 2024-09-18T10:21:12.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004363249}) 2024-09-18T10:21:12.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=833836834}) 2024-09-18T10:21:12.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004363249}) 2024-09-18T10:21:12.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136819764}) 2024-09-18T10:21:12.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353087161}) 2024-09-18T10:21:12.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153724222}) 2024-09-18T10:21:12.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353087161}) 2024-09-18T10:21:12.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.593+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112738290}) 2024-09-18T10:21:12.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843268147}) 2024-09-18T10:21:12.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159917639}) 2024-09-18T10:21:12.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843268147}) 2024-09-18T10:21:12.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088880971}) 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649124760}) 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982568589}) 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649124760}) 2024-09-18T10:21:12.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1755381151}) 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592240358}) 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964658342}) 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592240358}) 2024-09-18T10:21:12.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.597+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=117514855}) 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611929978}) 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039196128}) 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611929978}) 2024-09-18T10:21:12.613 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.605+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485321777}) 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225519141}) 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267062113}) 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225519141}) 2024-09-18T10:21:12.615 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2868981927}) 2024-09-18T10:21:12.616 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303183587}) 2024-09-18T10:21:12.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479203226}) 2024-09-18T10:21:12.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303183587}) 2024-09-18T10:21:12.617 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.609+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31438723}) 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649372735}) 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942551344}) 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649372735}) 2024-09-18T10:21:12.743 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.737+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026546619}) 2024-09-18T10:21:12.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440037882}) 2024-09-18T10:21:12.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2406598924}) 2024-09-18T10:21:12.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440037882}) 2024-09-18T10:21:12.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.937+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910389638}) 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351926810}) 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201827678}) 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351926810}) 2024-09-18T10:21:12.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.953+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412886291}) 2024-09-18T10:21:12.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756440577}) 2024-09-18T10:21:12.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1746591916}) 2024-09-18T10:21:12.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756440577}) 2024-09-18T10:21:12.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568588257}) 2024-09-18T10:21:12.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313570377}) 2024-09-18T10:21:12.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991258212}) 2024-09-18T10:21:12.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313570377}) 2024-09-18T10:21:12.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1982281730}) 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619828156}) 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=434703717}) 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619828156}) 2024-09-18T10:21:12.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.957+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597874913}) 2024-09-18T10:21:12.967 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.967 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041469590}) 2024-09-18T10:21:12.967 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3823324857}) 2024-09-18T10:21:12.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2041469590}) 2024-09-18T10:21:12.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969722715}) 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413220247}) 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982333840}) 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413220247}) 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675316684}) 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003449161}) 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398524931}) 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003449161}) 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.961+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1959094131}) 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859568679}) 2024-09-18T10:21:12.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858085992}) 2024-09-18T10:21:12.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859568679}) 2024-09-18T10:21:12.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004104836}) 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425001804}) 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=48603578}) 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425001804}) 2024-09-18T10:21:12.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.965+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078487046}) 2024-09-18T10:21:12.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694874443}) 2024-09-18T10:21:12.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835888199}) 2024-09-18T10:21:12.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694874443}) 2024-09-18T10:21:12.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442135119}) 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255632334}) 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103278117}) 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255632334}) 2024-09-18T10:21:12.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873268115}) 2024-09-18T10:21:12.976 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.976 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820049328}) 2024-09-18T10:21:12.976 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1003985288}) 2024-09-18T10:21:12.976 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.976 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820049328}) 2024-09-18T10:21:12.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.969+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3306976591}) 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=966988756}) 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451053681}) 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=966988756}) 2024-09-18T10:21:12.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700524131}) 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433222863}) 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781367258}) 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433222863}) 2024-09-18T10:21:12.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.973+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1214549487}) 2024-09-18T10:21:12.987 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451517134}) 2024-09-18T10:21:12.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544863518}) 2024-09-18T10:21:12.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451517134}) 2024-09-18T10:21:12.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654978000}) 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619464135}) 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079303607}) 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619464135}) 2024-09-18T10:21:12.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.981+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493514491}) 2024-09-18T10:21:12.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084806648}) 2024-09-18T10:21:12.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873285912}) 2024-09-18T10:21:12.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084806648}) 2024-09-18T10:21:12.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550664420}) 2024-09-18T10:21:12.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781352457}) 2024-09-18T10:21:12.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928071869}) 2024-09-18T10:21:12.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781352457}) 2024-09-18T10:21:12.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.985+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218616113}) 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553826641}) 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401471111}) 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553826641}) 2024-09-18T10:21:12.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3596953963}) 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114239085}) 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513119405}) 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4114239085}) 2024-09-18T10:21:12.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576947135}) 2024-09-18T10:21:12.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698701642}) 2024-09-18T10:21:12.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138649410}) 2024-09-18T10:21:12.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698701642}) 2024-09-18T10:21:12.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.989+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400604424}) 2024-09-18T10:21:12.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407982181}) 2024-09-18T10:21:12.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178891118}) 2024-09-18T10:21:12.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407982181}) 2024-09-18T10:21:12.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212127152}) 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3885280491}) 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1483442661}) 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3885280491}) 2024-09-18T10:21:12.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121397560}) 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400435325}) 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3168508768}) 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400435325}) 2024-09-18T10:21:13.000 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933922397}) 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673017828}) 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798914084}) 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673017828}) 2024-09-18T10:21:13.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.993+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147096870}) 2024-09-18T10:21:13.002 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.002 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=4,monmap=1} crc {logm=3797120058,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgrstat=2957389514,mon_config_key=3383142877,monmap=1861363405}) 2024-09-18T10:21:13.002 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2} crc {logm=1858637322,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=1925940116,mon_config_key=1147460298}) 2024-09-18T10:21:13.003 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.003 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=4,monmap=1} crc {logm=3797120058,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgrstat=2957389514,mon_config_key=3383142877,monmap=1861363405}) 2024-09-18T10:21:13.003 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5} crc {logm=721090461,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3714165226,mon_config_key=620546692}) 2024-09-18T10:21:13.004 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1754743539}) 2024-09-18T10:21:13.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=3237279720}) 2024-09-18T10:21:13.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1754743539}) 2024-09-18T10:21:13.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:12.997+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=89} crc {monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=2766904529}) 2024-09-18T10:21:13.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=916953229}) 2024-09-18T10:21:13.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=3468330975}) 2024-09-18T10:21:13.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:21:13.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=916953229}) 2024-09-18T10:21:13.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:21:13.005+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=2316391192}) 2024-09-18T10:21:13.043 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:21:13.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:21:13.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: get_monmap_and_config 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: monmap: 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:13.152313+0000 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:13.152313+0000 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:13.156 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:13.157 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 Processor -- start 2024-09-18T10:21:13.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 -- start start 2024-09-18T10:21:13.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: init 2024-09-18T10:21:13.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c600676a0 0x7f3c60132140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-a con 0x7f3c600676a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c60098650 0x7f3c60098a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-c con 0x7f3c60098650 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c6012e8f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6009e0c0 con 0x7f3c6012e8f0 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c60096e40 con 0x7f3c600676a0 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c60067a70 con 0x7f3c60098650 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c66c3e700 10 monclient(hunting): authenticate will time out at 1562.078742s 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c600676a0 0x7f3c60132140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c60098650 0x7f3c60098a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3c600676a0 0x7f3c60132140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43376/0 (socket says 172.21.15.142:43376) 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): get_auth_request con 0x7f3c60098650 auth_method 0 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 -- 172.21.15.142:0/2259708399 learned_addr learned my addr 172.21.15.142:0/2259708399 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): get_auth_request con 0x7f3c600676a0 auth_method 0 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_done global_id 124347 payload 274 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 -- 172.21.15.142:0/2259708399 >> v2:172.21.15.150:3300/0 conn(0x7f3c60098650 msgr2=0x7f3c60098a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 --2- 172.21.15.142:0/2259708399 >> v2:172.21.15.150:3300/0 conn(0x7f3c60098650 0x7f3c60098a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 -- 172.21.15.142:0/2259708399 >> v1:172.21.15.150:6789/0 conn(0x7f3c6012e8f0 legacy=0x7f3c6012ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 monclient: found mon.noname-a 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.149+0000 7f3c65c3c700 1 -- 172.21.15.142:0/2259708399 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6006f570 con 0x7f3c600676a0 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c65c3c700 1 -- 172.21.15.142:0/2259708399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c600676a0 msgr2=0x7f3c60132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c65c3c700 1 --2- 172.21.15.142:0/2259708399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c600676a0 0x7f3c60132140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c6006aaf0 tx=0x7f3c5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a179dca009225913 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 1 -- 172.21.15.142:0/2259708399 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c009e70 con 0x7f3c600676a0 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: dump: 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.164 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c66c3e700 5 monclient: authenticate success, global_id 124347 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 1 -- 172.21.15.142:0/2259708399 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c00ba70 con 0x7f3c600676a0 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c66c3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c66c3e700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c66c3e700 4 set_mon_vals no callback set 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 1 -- 172.21.15.142:0/2259708399 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c00be30 con 0x7f3c600676a0 2024-09-18T10:21:13.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.153+0000 7f3c64c3a700 10 monclient: dump: 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.166 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: shutdown 2024-09-18T10:21:13.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- 172.21.15.142:0/2259708399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c600676a0 msgr2=0x7f3c60132140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.170 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/2259708399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c600676a0 0x7f3c60132140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c6006aaf0 tx=0x7f3c5c009250 comp rx=0 tx=0).stop 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- 172.21.15.142:0/2259708399 shutdown_connections 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/2259708399 >> v2:172.21.15.150:3300/0 conn(0x7f3c60098650 0x7f3c60098a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/2259708399 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c600676a0 0x7f3c60132140 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- 172.21.15.142:0/2259708399 >> 172.21.15.142:0/2259708399 conn(0x7f3c60071ae0 msgr2=0x7f3c600a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- 172.21.15.142:0/2259708399 shutdown_connections 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- 172.21.15.142:0/2259708399 wait complete. 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: monmap: 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:13.171 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 Processor -- start 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- start start 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: init 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 0x7f3c601414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-c con 0x7f3c600676a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c60098650 0x7f3c6012b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-b con 0x7f3c60098650 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 0x7f3c60099740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): picked mon.noname-a con 0x7f3c6012e8f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c60118b00 con 0x7f3c6012e8f0 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6012daf0 con 0x7f3c60098650 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6009e250 con 0x7f3c600676a0 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c66c3e700 10 monclient(hunting): authenticate will time out at 1562.087406s 2024-09-18T10:21:13.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 0x7f3c601414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 0x7f3c60099740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 0x7f3c601414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42490/0 (socket says 172.21.15.142:42490) 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 1 -- 172.21.15.142:0/1248888783 learned_addr learned my addr 172.21.15.142:0/1248888783 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): get_auth_request con 0x7f3c600676a0 auth_method 0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): get_auth_request con 0x7f3c6012e8f0 auth_method 0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6543b700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c60098650 0x7f3c6012b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6543b700 10 monclient(hunting): get_auth_request con 0x7f3c60098650 auth_method 0 2024-09-18T10:21:13.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6543b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6543b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6543b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c6643d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.157+0000 7f3c65c3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6543b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 10 monclient(hunting): handle_auth_done global_id 124350 payload 824 2024-09-18T10:21:13.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 -- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 msgr2=0x7f3c601414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 0x7f3c601414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 -- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c60098650 msgr2=0x7f3c6012b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c60098650 0x7f3c6012b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 monclient: found mon.noname-a 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 -- 172.21.15.142:0/1248888783 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0094f0 con 0x7f3c6012e8f0 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 -- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 msgr2=0x7f3c60099740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c6643d700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 0x7f3c60099740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c840 tx=0x7f3c58002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58012840 con 0x7f3c6012e8f0 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: dump: 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c580129c0 con 0x7f3c6012e8f0 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 5 monclient: authenticate success, global_id 124350 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 10 monclient: _renew_subs 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6009b570 con 0x7f3c6012e8f0 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58012d80 con 0x7f3c6012e8f0 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c51ffb700 10 monclient: dump: 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.178 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 10 monclient: _renew_subs 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c601422d0 con 0x7f3c6012e8f0 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.161+0000 7f3c53fff700 4 set_mon_vals no callback set 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c66c3e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c66c3e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c66c3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60119020 con 0x7f3c6012e8f0 2024-09-18T10:21:13.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3c58011590 con 0x7f3c6012e8f0 2024-09-18T10:21:13.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c51ffb700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 0x7f3c44037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3c58048b60 con 0x7f3c6012e8f0 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c65c3c700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 0x7f3c44037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.165+0000 7f3c65c3c700 10 monclient: get_auth_request con 0x7f3c44035b10 auth_method 0 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.169+0000 7f3c65c3c700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 0x7f3c44037fc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009fd0 tx=0x7f3c5c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.169+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3c60119020 con 0x7f3c6012e8f0 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.169+0000 7f3c51ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.169+0000 7f3c51ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c66c3e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:21:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c66c3e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c66c3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3c60119020 con 0x7f3c6012e8f0 2024-09-18T10:21:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c51ffb700 1 -- 172.21.15.142:0/1248888783 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f3c60119020 con 0x7f3c6012e8f0 2024-09-18T10:21:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c51ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.389+0000 7f3c51ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:13.397 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 msgr2=0x7f3c44037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 0x7f3c44037fc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009fd0 tx=0x7f3c5c016000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 10 monclient: shutdown 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 msgr2=0x7f3c60099740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 0x7f3c60099740 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800c840 tx=0x7f3c58002620 comp rx=0 tx=0).stop 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 shutdown_connections 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c44035b10 0x7f3c44037fc0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c6012e8f0 0x7f3c60099740 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c60098650 0x7f3c6012b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 --2- 172.21.15.142:0/1248888783 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c600676a0 0x7f3c601414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 >> 172.21.15.142:0/1248888783 conn(0x7f3c60071ae0 msgr2=0x7f3c6012dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 shutdown_connections 2024-09-18T10:21:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.393+0000 7f3c66c3e700 1 -- 172.21.15.142:0/1248888783 wait complete. 2024-09-18T10:21:13.411 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:21:13.411 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:21:13.411 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:21:13.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.513+0000 7f99ae821700 10 monclient: get_monmap_and_config 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.513+0000 7f99ae821700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.513+0000 7f99ae821700 10 monclient: monmap: 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:13.519205+0000 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:13.519205+0000 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.521 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:13.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 Processor -- start 2024-09-18T10:21:13.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 -- start start 2024-09-18T10:21:13.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient: init 2024-09-18T10:21:13.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99a809c2c0 0x7f99a80999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-a con 0x7f99a809c2c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-b con 0x7f99a80a0110 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99a80a0a50 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80a6620 con 0x7f99a80a0a50 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a809ab70 con 0x7f99a809c2c0 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80a1c70 con 0x7f99a80a0110 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae821700 10 monclient(hunting): authenticate will time out at 1562.444955s 2024-09-18T10:21:13.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99a809c2c0 0x7f99a80999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42494/0 (socket says 172.21.15.142:42494) 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 -- 172.21.15.142:0/223596307 learned_addr learned my addr 172.21.15.142:0/223596307 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): get_auth_request con 0x7f99a80a0110 auth_method 0 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ae020700 1 --1- 172.21.15.142:0/223596307 >> v1:172.21.15.142:6789/0 conn(0x7f99a80a0a50 0x7f99a81388a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:58568/0 (socket says 172.21.15.142:58568) 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): get_auth_request con 0x7f99a809c2c0 auth_method 0 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient(hunting): handle_auth_done global_id 75254 payload 274 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 -- 172.21.15.142:0/223596307 >> v2:172.21.15.142:3300/0 conn(0x7f99a809c2c0 msgr2=0x7f99a80999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 --2- 172.21.15.142:0/223596307 >> v2:172.21.15.142:3300/0 conn(0x7f99a809c2c0 0x7f99a80999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 -- 172.21.15.142:0/223596307 >> v1:172.21.15.142:6789/0 conn(0x7f99a80a0a50 legacy=0x7f99a81388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 monclient: found mon.noname-b 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:13.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.517+0000 7f99ad01e700 1 -- 172.21.15.142:0/223596307 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a809c690 con 0x7f99a80a0110 2024-09-18T10:21:13.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 1 -- 172.21.15.142:0/223596307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 msgr2=0x7f99a80a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 1 --2- 172.21.15.142:0/223596307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009f20 tx=0x7f99a4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aab5d84066549d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 1 -- 172.21.15.142:0/223596307 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a400e070 con 0x7f99a80a0110 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: dump: 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.534 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 1 -- 172.21.15.142:0/223596307 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a400bbe0 con 0x7f99a80a0110 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 5 monclient: authenticate success, global_id 75254 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 1 -- 172.21.15.142:0/223596307 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a40092e0 con 0x7f99a80a0110 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 4 set_mon_vals no callback set 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ac81d700 10 monclient: dump: 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: shutdown 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- 172.21.15.142:0/223596307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 msgr2=0x7f99a80a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- 172.21.15.142:0/223596307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009f20 tx=0x7f99a4009840 comp rx=0 tx=0).stop 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- 172.21.15.142:0/223596307 shutdown_connections 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- 172.21.15.142:0/223596307 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a80a0510 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- 172.21.15.142:0/223596307 >> v2:172.21.15.142:3300/0 conn(0x7f99a809c2c0 0x7f99a80999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- 172.21.15.142:0/223596307 >> 172.21.15.142:0/223596307 conn(0x7f99a80a5720 msgr2=0x7f99a80a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- 172.21.15.142:0/223596307 shutdown_connections 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- 172.21.15.142:0/223596307 wait complete. 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: monmap: 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.537 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 Processor -- start 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- start start 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: init 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:21:13.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 0x7f99a8141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-a con 0x7f99a809c2c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a81420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-c con 0x7f99a80a0110 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80a0a50 0x7f99a808def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): picked mon.noname-b con 0x7f99a80a0a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8091b00 con 0x7f99a809c2c0 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80994c0 con 0x7f99a80a0a50 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80a4f40 con 0x7f99a80a0110 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae821700 10 monclient(hunting): authenticate will time out at 1562.450647s 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a81420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a81420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42500/0 (socket says 172.21.15.142:42500) 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 0x7f99a8141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 1 -- 172.21.15.142:0/2177290847 learned_addr learned my addr 172.21.15.142:0/2177290847 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): get_auth_request con 0x7f99a80a0110 auth_method 0 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): get_auth_request con 0x7f99a809c2c0 auth_method 0 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae020700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80a0a50 0x7f99a808def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae020700 10 monclient(hunting): get_auth_request con 0x7f99a80a0a50 auth_method 0 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae020700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae020700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ae020700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient(hunting): handle_auth_done global_id 124356 payload 824 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 msgr2=0x7f99a81420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99a80a0110 0x7f99a81420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80a0a50 msgr2=0x7f99a808def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a80a0a50 0x7f99a808def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 monclient: found mon.noname-a 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.521+0000 7f99ad81f700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4009ae0 con 0x7f99a809c2c0 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad81f700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 msgr2=0x7f99a8141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad81f700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 0x7f99a8141b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f99a000ea50 tx=0x7f99a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a00095d0 con 0x7f99a809c2c0 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: dump: 2024-09-18T10:21:13.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 5 monclient: authenticate success, global_id 124356 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 monclient: _renew_subs 2024-09-18T10:21:13.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a808ff00 con 0x7f99a809c2c0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 monclient: _renew_subs 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a000ca00 con 0x7f99a809c2c0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a80926d0 con 0x7f99a809c2c0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a000cdc0 con 0x7f99a809c2c0 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 10 monclient: dump: 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.544 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f999b7fe700 4 set_mon_vals no callback set 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f99a0013680 con 0x7f99a809c2c0 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 0x7f99a8029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8064750 con 0x7f99a808ec00 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f998c035af0 0x7f998c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f99a00494c0 con 0x7f99a809c2c0 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 0x7f99a8029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient: get_auth_request con 0x7f99a808ec00 auth_method 0 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae020700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f998c035af0 0x7f998c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.525+0000 7f99ae020700 10 monclient: get_auth_request con 0x7f998c035af0 auth_method 0 2024-09-18T10:21:13.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f99ad01e700 10 monclient(hunting): handle_auth_done global_id 124359 payload 824 2024-09-18T10:21:13.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f99ad01e700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99a80918b0 con 0x7f99a808ec00 2024-09-18T10:21:13.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f99ad01e700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 msgr2=0x7f99a8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f99ad01e700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 0x7f99a8029d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f99a8067e20 tx=0x7f99a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4012c30 con 0x7f99a808ec00 2024-09-18T10:21:13.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 10 monclient: dump: 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f99ae020700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f998c035af0 0x7f998c037fa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c8f0 tx=0x7f999c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f99a4026420 con 0x7f99a808ec00 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 msgr2=0x7f99a8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 0x7f99a8029d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f99a8067e20 tx=0x7f99a4017000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 msgr2=0x7f99a8029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.529+0000 7f9999ffb700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808ec00 0x7f99a8029d00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ae821700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ae821700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 0x7f99a802a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ae821700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a80915c0 con 0x7f99a808efd0 2024-09-18T10:21:13.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 0x7f99a802a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient: get_auth_request con 0x7f99a808efd0 auth_method 0 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 10 monclient(hunting): handle_auth_done global_id 124362 payload 824 2024-09-18T10:21:13.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.541+0000 7f99ad01e700 1 -- 172.21.15.142:0/2177290847 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99a806e190 con 0x7f99a808efd0 2024-09-18T10:21:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ad01e700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 msgr2=0x7f99a802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ad01e700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 0x7f99a802a240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99a4025000 tx=0x7f99a4020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a401dd70 con 0x7f99a808efd0 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 10 monclient: dump: 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.553 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f99a401ded0 con 0x7f99a808efd0 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 msgr2=0x7f99a802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 0x7f99a802a240 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99a4025000 tx=0x7f99a4020070 comp rx=0 tx=0).stop 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 msgr2=0x7f99a802a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f9999ffb700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a808efd0 0x7f99a802a240 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:13.554 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:13.555 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:13.556 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.557 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:13.558 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.559 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.561 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.562 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:13.563 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.564 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.565 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.566 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:13.567 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f998c035af0 msgr2=0x7f998c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f998c035af0 0x7f998c037fa0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c8f0 tx=0x7f999c009040 comp rx=0 tx=0).stop 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 10 monclient: shutdown 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 msgr2=0x7f99a8141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 0x7f99a8141b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f99a000ea50 tx=0x7f99a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae020700 1 -- 172.21.15.142:0/2177290847 reap_dead start 2024-09-18T10:21:13.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 shutdown_connections 2024-09-18T10:21:13.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 --2- 172.21.15.142:0/2177290847 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99a809c2c0 0x7f99a8141b90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 >> 172.21.15.142:0/2177290847 conn(0x7f99a80a5720 msgr2=0x7f99a809dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 shutdown_connections 2024-09-18T10:21:13.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.545+0000 7f99ae821700 1 -- 172.21.15.142:0/2177290847 wait complete. 2024-09-18T10:21:13.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:13.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:21:13.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:21:13.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.677+0000 7fcf9c5bf700 10 monclient: get_monmap_and_config 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.677+0000 7fcf9c5bf700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.677+0000 7fcf9c5bf700 10 monclient: monmap: 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:13.682978+0000 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:13.682978+0000 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:13.685 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:13.686 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:13.686 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 Processor -- start 2024-09-18T10:21:13.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 -- start start 2024-09-18T10:21:13.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient: init 2024-09-18T10:21:13.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf9409d870 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcf940954b0 0x7fcf940990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-c con 0x7fcf940954b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 0x7fcf94136a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-b con 0x7fcf94134670 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf94069c70 con 0x7fcf9409d870 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9406f9d0 con 0x7fcf940954b0 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940958b0 con 0x7fcf94134670 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9c5bf700 10 monclient(hunting): authenticate will time out at 1562.608999s 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9a35b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcf9409d870 0x7fcf940989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:58584/0 (socket says 172.21.15.142:58584) 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9a35b700 1 -- 172.21.15.142:0/1590773943 learned_addr learned my addr 172.21.15.142:0/1590773943 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 0x7fcf94136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): get_auth_request con 0x7fcf94134670 auth_method 0 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 1 --2- 172.21.15.142:0/1590773943 >> v2:172.21.15.150:3300/0 conn(0x7fcf940954b0 0x7fcf940990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): get_auth_request con 0x7fcf940954b0 auth_method 0 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient(hunting): handle_auth_done global_id 75260 payload 274 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 -- 172.21.15.142:0/1590773943 >> v2:172.21.15.150:3300/0 conn(0x7fcf940954b0 msgr2=0x7fcf940990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/1590773943 >> v2:172.21.15.150:3300/0 conn(0x7fcf940954b0 0x7fcf940990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 -- 172.21.15.142:0/1590773943 >> v1:172.21.15.142:6789/0 conn(0x7fcf9409d870 legacy=0x7fcf940989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 monclient: found mon.noname-b 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:13.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.681+0000 7fcf9ab5c700 1 -- 172.21.15.142:0/1590773943 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9409dc40 con 0x7fcf94134670 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9ab5c700 1 -- 172.21.15.142:0/1590773943 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 msgr2=0x7fcf94136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/1590773943 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 0x7fcf94136a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009e60 tx=0x7fcf8c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9850b72b570291 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 1 -- 172.21.15.142:0/1590773943 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00e070 con 0x7fcf94134670 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: dump: 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.694 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 5 monclient: authenticate success, global_id 75260 2024-09-18T10:21:13.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 1 -- 172.21.15.142:0/1590773943 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c00bd70 con 0x7fcf94134670 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 1 -- 172.21.15.142:0/1590773943 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c012660 con 0x7fcf94134670 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 4 set_mon_vals no callback set 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf99359700 10 monclient: dump: 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.696 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 10 monclient: shutdown 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/1590773943 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 msgr2=0x7fcf94136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/1590773943 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 0x7fcf94136a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009e60 tx=0x7fcf8c009780 comp rx=0 tx=0).stop 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/1590773943 shutdown_connections 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/1590773943 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf94134670 0x7fcf94136a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/1590773943 >> v2:172.21.15.150:3300/0 conn(0x7fcf940954b0 0x7fcf940990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/1590773943 >> 172.21.15.142:0/1590773943 conn(0x7fcf940a14e0 msgr2=0x7fcf940a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/1590773943 shutdown_connections 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/1590773943 wait complete. 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.685+0000 7fcf9c5bf700 10 monclient: monmap: 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:13.698 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 Processor -- start 2024-09-18T10:21:13.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 -- start start 2024-09-18T10:21:13.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient: init 2024-09-18T10:21:13.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:21:13.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf940954b0 0x7fcf9412e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-a con 0x7fcf940954b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:13.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-c con 0x7fcf9409d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf94134670 0x7fcf9406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): picked mon.noname-b con 0x7fcf94134670 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94127c60 con 0x7fcf940954b0 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94097b80 con 0x7fcf94134670 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94127560 con 0x7fcf9409d870 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9c5bf700 10 monclient(hunting): authenticate will time out at 1562.617468s 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf940954b0 0x7fcf9412e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42520/0 (socket says 172.21.15.142:42520) 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 -- 172.21.15.142:0/4225957892 learned_addr learned my addr 172.21.15.142:0/4225957892 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): get_auth_request con 0x7fcf9409d870 auth_method 0 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): get_auth_request con 0x7fcf940954b0 auth_method 0 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf94134670 0x7fcf9406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 10 monclient(hunting): get_auth_request con 0x7fcf94134670 auth_method 0 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient(hunting): handle_auth_done global_id 75263 payload 824 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf94134670 msgr2=0x7fcf9406d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf94134670 0x7fcf9406d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf940954b0 msgr2=0x7fcf9412e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf940954b0 0x7fcf9412e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 monclient: found mon.noname-c 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf99b5a700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c009a20 con 0x7fcf9409d870 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.689+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf94134670 0x7fcf9406d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf99b5a700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 msgr2=0x7fcf9406cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf99b5a700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcf84010e90 tx=0x7fcf840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011450 con 0x7fcf9409d870 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: dump: 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.708 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 5 monclient: authenticate success, global_id 75263 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf840115d0 con 0x7fcf9409d870 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 monclient: _renew_subs 2024-09-18T10:21:13.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9406e990 con 0x7fcf9409d870 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011990 con 0x7fcf9409d870 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf98b58700 4 set_mon_vals no callback set 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 10 monclient: dump: 2024-09-18T10:21:13.710 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 monclient: _renew_subs 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:21:13.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9406edd0 con 0x7fcf9409d870 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 0x7fcf9406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94064950 con 0x7fcf9412f140 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 0x7fcf9406d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient: get_auth_request con 0x7fcf9412f140 auth_method 0 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcf84011d00 con 0x7fcf9409d870 2024-09-18T10:21:13.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_done global_id 75266 payload 824 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9a35b700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf94127130 con 0x7fcf9412f140 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf78035ac0 0x7fcf78037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fcf8404b550 con 0x7fcf9409d870 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf78035ac0 0x7fcf78037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.693+0000 7fcf9ab5c700 10 monclient: get_auth_request con 0x7fcf78035ac0 auth_method 0 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf9a35b700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 msgr2=0x7fcf9406d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf9a35b700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 0x7fcf9406d690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000eac0 tx=0x7fcf90007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000cc40 con 0x7fcf9412f140 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 10 monclient: dump: 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.713 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcf9000cda0 con 0x7fcf9412f140 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 msgr2=0x7fcf9406d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 0x7fcf9406d690 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000eac0 tx=0x7fcf90007f90 comp rx=0 tx=0).stop 2024-09-18T10:21:13.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 msgr2=0x7fcf9406d690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf8a7fc700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9412f140 0x7fcf9406d690 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.697+0000 7fcf9ab5c700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf78035ac0 0x7fcf78037f70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009e60 tx=0x7fcf8c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9c5bf700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 0x7fcf9409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9c5bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940696a0 con 0x7fcf9409c760 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 0x7fcf9409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient: get_auth_request con 0x7fcf9409c760 auth_method 0 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 10 monclient(hunting): handle_auth_done global_id 75269 payload 824 2024-09-18T10:21:13.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 1 -- 172.21.15.142:0/4225957892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcf9406f2f0 con 0x7fcf9409c760 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 msgr2=0x7fcf9409cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf9a35b700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 0x7fcf9409cb60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcf90007360 tx=0x7fcf90016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90029400 con 0x7fcf9409c760 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf8a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:13.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.709+0000 7fcf8a7fc700 10 monclient: dump: 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fcf9000ca30 con 0x7fcf9409c760 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:13.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 msgr2=0x7fcf9409cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 0x7fcf9409cb60 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcf90007360 tx=0x7fcf90016000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 msgr2=0x7fcf9409cb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf8a7fc700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409c760 0x7fcf9409cb60 secure :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcf90007360 tx=0x7fcf90016000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:13.720 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.721 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.722 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:13.723 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.724 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:13.725 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:13.726 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:13.727 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.728 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.729 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.730 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:13.731 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:21:13.732 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:13.733 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf78035ac0 msgr2=0x7fcf78037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcf78035ac0 0x7fcf78037f70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009e60 tx=0x7fcf8c018000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 10 monclient: shutdown 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 msgr2=0x7fcf9406cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcf84010e90 tx=0x7fcf840091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9ab5c700 1 -- 172.21.15.142:0/4225957892 reap_dead start 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 shutdown_connections 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 --2- 172.21.15.142:0/4225957892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf9409d870 0x7fcf9406cc10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 >> 172.21.15.142:0/4225957892 conn(0x7fcf940a14e0 msgr2=0x7fcf94131a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 shutdown_connections 2024-09-18T10:21:13.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.713+0000 7fcf9c5bf700 1 -- 172.21.15.142:0/4225957892 wait complete. 2024-09-18T10:21:13.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:13.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:21:13.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: get_monmap_and_config 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: monmap: 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:13.849025+0000 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:13.849025+0000 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:13.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:13.852 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:13.852 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:13.852 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:13.852 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 Processor -- start 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 -- start start 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: init 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b4c09abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b4c09b740 0x7f0b4c134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-a con 0x7f0b4c09b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b4c134bb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c09dc90 con 0x7f0b4c09abd0 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c09afa0 con 0x7f0b4c134bb0 2024-09-18T10:21:13.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c09b500 con 0x7f0b4c09b740 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b52f7d700 10 monclient(hunting): authenticate will time out at 1562.774774s 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b4c09b740 0x7f0b4c134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b51f7b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0b4c09abd0 0x7f0b4c0960e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:58596/0 (socket says 172.21.15.142:58596) 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b51f7b700 1 -- 172.21.15.142:0/2426288451 learned_addr learned my addr 172.21.15.142:0/2426288451 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): get_auth_request con 0x7f0b4c09b740 auth_method 0 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient(hunting): handle_auth_done global_id 124368 payload 274 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 1 -- 172.21.15.142:0/2426288451 >> v1:172.21.15.150:6789/0 conn(0x7f0b4c134bb0 legacy=0x7f0b4c136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 1 -- 172.21.15.142:0/2426288451 >> v1:172.21.15.142:6789/0 conn(0x7f0b4c09abd0 legacy=0x7f0b4c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 1 monclient: found mon.noname-a 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.845+0000 7f0b5177a700 1 -- 172.21.15.142:0/2426288451 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c06f810 con 0x7f0b4c09b740 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b5177a700 1 -- 172.21.15.142:0/2426288451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c09b740 msgr2=0x7f0b4c134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b5177a700 1 --2- 172.21.15.142:0/2426288451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c09b740 0x7f0b4c134670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c095730 tx=0x7f0b3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efd22031d94115d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 1 -- 172.21.15.142:0/2426288451 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c00b9a0 con 0x7f0b4c09b740 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: dump: 2024-09-18T10:21:13.860 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 5 monclient: authenticate success, global_id 124368 2024-09-18T10:21:13.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 1 -- 172.21.15.142:0/2426288451 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c00bb00 con 0x7f0b4c09b740 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 4 set_mon_vals no callback set 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 1 -- 172.21.15.142:0/2426288451 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c00bea0 con 0x7f0b4c09b740 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b50f79700 10 monclient: dump: 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.862 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 10 monclient: shutdown 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 -- 172.21.15.142:0/2426288451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c09b740 msgr2=0x7f0b4c134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/2426288451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c09b740 0x7f0b4c134670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c095730 tx=0x7f0b3c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 -- 172.21.15.142:0/2426288451 shutdown_connections 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/2426288451 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c09b740 0x7f0b4c134670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 -- 172.21.15.142:0/2426288451 >> 172.21.15.142:0/2426288451 conn(0x7f0b4c070ef0 msgr2=0x7f0b4c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 -- 172.21.15.142:0/2426288451 shutdown_connections 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 1 -- 172.21.15.142:0/2426288451 wait complete. 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 10 monclient: build_initial_monmap 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.849+0000 7f0b52f7d700 10 monclient: monmap: 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 Processor -- start 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 -- start start 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient: init 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b4c09abd0 0x7f0b4c087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-c con 0x7f0b4c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c09b740 0x7f0b4c088520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-b con 0x7f0b4c09b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): picked mon.noname-a con 0x7f0b4c134bb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c09fa00 con 0x7f0b4c134bb0 2024-09-18T10:21:13.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c089820 con 0x7f0b4c09b740 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c089f70 con 0x7f0b4c09abd0 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b52f7d700 10 monclient(hunting): authenticate will time out at 1562.781318s 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b4c09abd0 0x7f0b4c087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43478/0 (socket says 172.21.15.142:43478) 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 learned_addr learned my addr 172.21.15.142:0/1002243155 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): get_auth_request con 0x7f0b4c134bb0 auth_method 0 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): get_auth_request con 0x7f0b4c09abd0 auth_method 0 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c09b740 0x7f0b4c088520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): get_auth_request con 0x7f0b4c09b740 auth_method 0 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5177a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient(hunting): handle_auth_done global_id 124371 payload 824 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b4c09abd0 msgr2=0x7f0b4c087fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b4c09abd0 0x7f0b4c087fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c09b740 msgr2=0x7f0b4c088520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c09b740 0x7f0b4c088520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient: _finish_hunting 0 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 monclient: found mon.noname-a 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:13.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.853+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c009420 con 0x7f0b4c134bb0 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 msgr2=0x7f0b4c088a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b5277c700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400c890 tx=0x7f0b440026a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44012840 con 0x7f0b4c134bb0 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: dump: 2024-09-18T10:21:13.871 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: _finish_auth 0 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 5 monclient: authenticate success, global_id 124371 2024-09-18T10:21:13.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 monclient: _renew_subs 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b440129c0 con 0x7f0b4c134bb0 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c09d240 con 0x7f0b4c134bb0 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b44012d80 con 0x7f0b4c134bb0 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b437fe700 4 set_mon_vals no callback set 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 10 monclient: dump: 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.873 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 monclient: _renew_subs 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c08aef0 con 0x7f0b4c134bb0 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 0x7f0b4c06b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c08b690 con 0x7f0b4c088fa0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 0x7f0b4c06b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient: get_auth_request con 0x7f0b4c088fa0 auth_method 0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b44011590 con 0x7f0b4c134bb0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b30035ac0 0x7f0b30037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_done global_id 144151 payload 824 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 176..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0b440492e0 con 0x7f0b4c134bb0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b51f7b700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b4c08a470 con 0x7f0b4c088fa0 2024-09-18T10:21:13.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b5177a700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b30035ac0 0x7f0b30037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.857+0000 7f0b5177a700 10 monclient: get_auth_request con 0x7f0b30035ac0 auth_method 0 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b51f7b700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 msgr2=0x7f0b4c06b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b51f7b700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 0x7f0b4c06b370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800eac0 tx=0x7f0b48007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4800cc40 con 0x7f0b4c088fa0 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 10 monclient: dump: 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.876 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b5177a700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b30035ac0 0x7f0b30037f70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0030c0 tx=0x7f0b3c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0b4800cda0 con 0x7f0b4c088fa0 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 msgr2=0x7f0b4c06b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 0x7f0b4c06b370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800eac0 tx=0x7f0b48007f90 comp rx=0 tx=0).stop 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 msgr2=0x7f0b4c06b370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.861+0000 7f0b41ffb700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c088fa0 0x7f0b4c06b370 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800eac0 tx=0x7f0b48007f90 comp rx=0 tx=0).stop 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b52f7d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 0x7f0b4c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b52f7d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c0648a0 con 0x7f0b4c06b8b0 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 0x7f0b4c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient: get_auth_request con 0x7f0b4c06b8b0 auth_method 0 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:13.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:13.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:13.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:13.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:13.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 10 monclient(hunting): handle_auth_done global_id 144154 payload 824 2024-09-18T10:21:13.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.873+0000 7f0b51f7b700 1 -- 172.21.15.142:0/1002243155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b4c072a80 con 0x7f0b4c06b8b0 2024-09-18T10:21:13.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b51f7b700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 msgr2=0x7f0b4c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:13.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b51f7b700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 0x7f0b4c06bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b48007360 tx=0x7f0b48016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48013e70 con 0x7f0b4c06b8b0 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 10 monclient: dump: 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:13.884 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:13.885 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0b480213f0 con 0x7f0b4c06b8b0 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 msgr2=0x7f0b4c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 0x7f0b4c06bcb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b48007360 tx=0x7f0b48016000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 msgr2=0x7f0b4c06bcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.877+0000 7f0b41ffb700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4c06b8b0 0x7f0b4c06bcb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.886 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 122, 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.887 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:13.888 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:13.889 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:13.890 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:13.891 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:13.892 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:13.893 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.894 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:13.895 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.896 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:13.897 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:13.898 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:13.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b30035ac0 msgr2=0x7f0b30037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b30035ac0 0x7f0b30037f70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0030c0 tx=0x7f0b3c016000 comp rx=0 tx=0).stop 2024-09-18T10:21:13.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 10 monclient: shutdown 2024-09-18T10:21:13.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 msgr2=0x7f0b4c088a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b5277c700 1 -- 172.21.15.142:0/1002243155 reap_dead start 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400c890 tx=0x7f0b440026a0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 shutdown_connections 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 --2- 172.21.15.142:0/1002243155 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b4c134bb0 0x7f0b4c088a60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 >> 172.21.15.142:0/1002243155 conn(0x7f0b4c070ef0 msgr2=0x7f0b4c09bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 shutdown_connections 2024-09-18T10:21:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:13.881+0000 7f0b52f7d700 1 -- 172.21.15.142:0/1002243155 wait complete. 2024-09-18T10:21:13.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:21:13.902 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:21:13.902 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T10:21:13.902 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:21:13.902 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:21:13.903 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:21:13.903 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:21:13.919 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:21:13.919 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T10:21:13.920 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T10:21:13.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:21:33.921 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T10:21:33.922 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T10:21:33.922 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:21:33.922 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:21:34.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.025+0000 7f4bdeef8700 10 monclient: get_monmap_and_config 2024-09-18T10:21:34.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.025+0000 7f4bdeef8700 10 monclient: build_initial_monmap 2024-09-18T10:21:34.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.025+0000 7f4bdeef8700 10 monclient: monmap: 2024-09-18T10:21:34.033 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:34.033 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:34.031363+0000 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:34.031363+0000 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:34.034 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:34.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 Processor -- start 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 -- start start 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient: init 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bd80a1cf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-a con 0x7f4bd812f810 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-b con 0x7f4bd80a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd809fb90 con 0x7f4bd80a1cf0 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:34.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd809d990 con 0x7f4bd812f810 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809dc70 con 0x7f4bd80a02d0 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdeef8700 10 monclient(hunting): authenticate will time out at 1582.957136s 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38038/0 (socket says 172.21.15.142:38038) 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bddef6700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 0x7f4bd809f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48394/0 (socket says 172.21.15.142:48394) 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 learned_addr learned my addr 172.21.15.142:0/509092124 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:34.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request con 0x7f4bd812f810 auth_method 0 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bd80a02d0 auth_method 0 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:34.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:34.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.029+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 legacy=0x7f4bd809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:34.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bd80a02d0 auth_method 0 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request con 0x7f4bd812f810 auth_method 0 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:21:34.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.233+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bd80a02d0 auth_method 0 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request con 0x7f4bd812f810 auth_method 0 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:21:34.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.637+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:21:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.809+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 legacy=0x7f4bd809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-18T10:21:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.809+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 legacy=0x7f4bd809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:21:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.809+0000 7f4bddef6700 1 --1- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 0x7f4bd809f480 :-1 s=OPENED pgs=152 cs=1 l=1).handle_message read tag failed 2024-09-18T10:21:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.809+0000 7f4bddef6700 1 --1- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 0x7f4bd809f480 :-1 s=OPENED pgs=152 cs=1 l=1).fault on lossy channel, failing 2024-09-18T10:21:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:34.809+0000 7f4bdcef4700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.142:6789/0 2024-09-18T10:21:35.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:35.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:35.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bd80a02d0 auth_method 0 2024-09-18T10:21:35.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:35.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request con 0x7f4bd812f810 auth_method 0 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:21:35.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:35.441+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:21:37.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): tick 2024-09-18T10:21:37.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:21:37.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 msgr2=0x7f4bd8139420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bd80a02d0 0x7f4bd8139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 msgr2=0x7f4bd812f030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.142:3300/0 conn(0x7f4bd812f810 0x7f4bd812f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 legacy=0x7f4bd809f480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient: _add_conns ranks=[1,4] 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bbc003640 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc0072c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd80a20c0 con 0x7f4bbc003640 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:37.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd809d990 con 0x7f4bbc0072c0 2024-09-18T10:21:37.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bcf7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:37.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.150:6789/0 conn(0x7f4bbc003640 legacy=0x7f4bbc005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:21:37.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:37.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.029+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:37.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.233+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.150:6789/0 conn(0x7f4bbc003640 legacy=0x7f4bbc005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:21:37.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.233+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:37.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.233+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:21:37.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.637+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.150:6789/0 conn(0x7f4bbc003640 legacy=0x7f4bbc005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:21:37.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.637+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:37.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:37.637+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:21:38.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:38.441+0000 7f4bddef6700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.150:6789/0 conn(0x7f4bbc003640 legacy=0x7f4bbc005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:21:38.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:38.441+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:38.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:38.441+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): tick 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc0072c0 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.150:6789/0 conn(0x7f4bbc003640 legacy=0x7f4bbc005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 reap_dead start 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bd80a1cf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 0x7f4bbc009770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc00a550 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 0x7f4bbc005a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc00aa50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc00ae40 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd809d990 con 0x7f4bbc00a550 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 1 -- 172.21.15.142:0/509092124 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd809dc70 con 0x7f4bbc00aa50 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bcf7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bdd6f5700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bdd6f5700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 0x7f4bbc005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bbc00aa50 auth_method 0 2024-09-18T10:21:40.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_done global_id 75275 payload 274 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 msgr2=0x7f4bbc009770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 0x7f4bbc009770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> v1:172.21.15.142:6789/0 conn(0x7f4bd80a1cf0 legacy=0x7f4bbc009e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 monclient: found mon.noname-b 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:40.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.029+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd80a20c0 con 0x7f4bbc00aa50 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bde6f7700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 msgr2=0x7f4bbc005a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 0x7f4bbc005a20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80677a0 tx=0x7f4bd4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9bdf61785da88b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 1 -- 172.21.15.142:0/509092124 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4013780 con 0x7f4bbc00aa50 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: dump: 2024-09-18T10:21:40.041 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.042 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 1 -- 172.21.15.142:0/509092124 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40138e0 con 0x7f4bbc00aa50 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 5 monclient: authenticate success, global_id 75275 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 1 -- 172.21.15.142:0/509092124 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4013ca0 con 0x7f4bbc00aa50 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 4 set_mon_vals no callback set 2024-09-18T10:21:40.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdcef4700 10 monclient: dump: 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.044 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 10 monclient: shutdown 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 -- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 msgr2=0x7f4bbc005a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 0x7f4bbc005a20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80677a0 tx=0x7f4bd4008040 comp rx=0 tx=0).stop 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 -- 172.21.15.142:0/509092124 shutdown_connections 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/509092124 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00aa50 0x7f4bbc005a20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/509092124 >> v2:172.21.15.150:3300/0 conn(0x7f4bbc00a550 0x7f4bbc009770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 -- 172.21.15.142:0/509092124 >> 172.21.15.142:0/509092124 conn(0x7f4bd8071820 msgr2=0x7f4bd80a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 -- 172.21.15.142:0/509092124 shutdown_connections 2024-09-18T10:21:40.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.033+0000 7f4bdeef8700 1 -- 172.21.15.142:0/509092124 wait complete. 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient: build_initial_monmap 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient: monmap: 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:40.049 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 Processor -- start 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 -- start start 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient: init 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 0x7f4bd8141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-c con 0x7f4bbc00a550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 0x7f4bd81422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-b con 0x7f4bbc00aa50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 0x7f4bd81427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): picked mon.noname-a con 0x7f4bd80a1cf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd808cb50 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd808c870 con 0x7f4bbc00aa50 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd808d520 con 0x7f4bbc00a550 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdeef8700 10 monclient(hunting): authenticate will time out at 1588.967266s 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdd6f5700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 msgr2=0x7f4bd81422b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 0x7f4bd8141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bdd6f5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 0x7f4bd81422b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 0x7f4bd81427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 0x7f4bd8141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34214/0 (socket says 172.21.15.142:34214) 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 1 -- 172.21.15.142:0/1345116081 learned_addr learned my addr 172.21.15.142:0/1345116081 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request con 0x7f4bd80a1cf0 auth_method 0 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): get_auth_request con 0x7f4bbc00a550 auth_method 0 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.037+0000 7f4bddef6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 10 monclient(hunting): handle_auth_done global_id 124374 payload 824 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 -- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 msgr2=0x7f4bd8141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 0x7f4bd8141d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 -- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 msgr2=0x7f4bd81422b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 0x7f4bd81422b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 monclient: found mon.noname-a 2024-09-18T10:21:40.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 -- 172.21.15.142:0/1345116081 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd400e040 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 -- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 msgr2=0x7f4bd81427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bde6f7700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 0x7f4bd81427f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4bd812fbe0 tx=0x7f4bd401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4025070 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: dump: 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.054 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd400f4a0 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40168a0 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 5 monclient: authenticate success, global_id 124374 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 10 monclient: _renew_subs 2024-09-18T10:21:40.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcf7fe700 4 set_mon_vals no callback set 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd808c590 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bcdffb700 10 monclient: dump: 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.056 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.057 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.057 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 10 monclient: _renew_subs 2024-09-18T10:21:40.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.041+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd806e080 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4bd4016c90 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bdeef8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bdeef8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bdeef8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd808cce0 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bcdffb700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 0x7f4bbc039510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4bd4000e40 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bddef6700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 0x7f4bbc039510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.045+0000 7f4bddef6700 10 monclient: get_auth_request con 0x7f4bbc037060 auth_method 0 2024-09-18T10:21:40.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.049+0000 7f4bddef6700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 0x7f4bbc039510 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8001870 tx=0x7f4bc8001ab0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.053+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4bd4013c60 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.053+0000 7f4bcdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.053+0000 7f4bcdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:40.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bdeef8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:21:40.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bdeef8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:40.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bdeef8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bd8072a80 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bcdffb700 1 -- 172.21.15.142:0/1345116081 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f4bd401a020 con 0x7f4bd80a1cf0 2024-09-18T10:21:40.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bcdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:21:40.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.277+0000 7f4bcdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:40.285 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":124,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 msgr2=0x7f4bbc039510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 0x7f4bbc039510 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8001870 tx=0x7f4bc8001ab0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 10 monclient: shutdown 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 msgr2=0x7f4bd81427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 0x7f4bd81427f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4bd812fbe0 tx=0x7f4bd401f000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 shutdown_connections 2024-09-18T10:21:40.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4bbc037060 0x7f4bbc039510 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4bd80a1cf0 0x7f4bd81427f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bbc00aa50 0x7f4bd81422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 --2- 172.21.15.142:0/1345116081 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4bbc00a550 0x7f4bd8141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 >> 172.21.15.142:0/1345116081 conn(0x7f4bd8071820 msgr2=0x7f4bd812bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 shutdown_connections 2024-09-18T10:21:40.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.281+0000 7f4bdeef8700 1 -- 172.21.15.142:0/1345116081 wait complete. 2024-09-18T10:21:40.298 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:21:40.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:21:40.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:21:40.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.393+0000 7fe80d812700 10 monclient: get_monmap_and_config 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.393+0000 7fe80d812700 10 monclient: build_initial_monmap 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.393+0000 7fe80d812700 10 monclient: monmap: 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:40.399130+0000 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:40.399130+0000 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.401 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:40.402 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:40.402 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:40.402 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:40.402 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:40.402 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.393+0000 7fe80d812700 1 Processor -- start 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 -- start start 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient: init 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe80809abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe80809b740 0x7fe808134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-a con 0x7fe80809b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 0x7fe808136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-c con 0x7fe808134bb0 addr v2:172.21.15.150:3300/0 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80809dc90 con 0x7fe80809abd0 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80809b500 con 0x7fe80809b740 2024-09-18T10:21:40.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8080967f0 con 0x7fe808134bb0 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d812700 10 monclient(hunting): authenticate will time out at 1589.324562s 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe80809b740 0x7fe808134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe80809b740 0x7fe808134670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38100/0 (socket says 172.21.15.142:38100) 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 -- 172.21.15.142:0/298917881 learned_addr learned my addr 172.21.15.142:0/298917881 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): get_auth_request con 0x7fe80809b740 auth_method 0 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d011700 1 -- 172.21.15.142:0/298917881 >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 msgr2=0x7fe808136f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80d011700 1 --2- 172.21.15.142:0/298917881 >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 0x7fe808136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe80c810700 1 -- 172.21.15.142:0/298917881 >> v1:172.21.15.142:6789/0 conn(0x7fe80809abd0 legacy=0x7fe8080960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient(hunting): handle_auth_done global_id 124377 payload 274 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 -- 172.21.15.142:0/298917881 >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 msgr2=0x7fe808136f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:40.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 --2- 172.21.15.142:0/298917881 >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 0x7fe808136f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 -- 172.21.15.142:0/298917881 >> v1:172.21.15.142:6789/0 conn(0x7fe80809abd0 legacy=0x7fe8080960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 monclient: found mon.noname-a 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.397+0000 7fe807fff700 1 -- 172.21.15.142:0/298917881 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80809afa0 con 0x7fe80809b740 2024-09-18T10:21:40.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe807fff700 1 -- 172.21.15.142:0/298917881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809b740 msgr2=0x7fe808134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe807fff700 1 --2- 172.21.15.142:0/298917881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809b740 0x7fe808134670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe808095730 tx=0x7fe7fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0ea7f2da304387 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 1 -- 172.21.15.142:0/298917881 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00bb10 con 0x7fe80809b740 2024-09-18T10:21:40.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: dump: 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 1 -- 172.21.15.142:0/298917881 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc00bc70 con 0x7fe80809b740 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 5 monclient: authenticate success, global_id 124377 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 4 set_mon_vals no callback set 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 1 -- 172.21.15.142:0/298917881 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc011420 con 0x7fe80809b740 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe8077fe700 10 monclient: dump: 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.412 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 10 monclient: shutdown 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 -- 172.21.15.142:0/298917881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809b740 msgr2=0x7fe808134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 --2- 172.21.15.142:0/298917881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809b740 0x7fe808134670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe808095730 tx=0x7fe7fc009250 comp rx=0 tx=0).stop 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 -- 172.21.15.142:0/298917881 shutdown_connections 2024-09-18T10:21:40.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 --2- 172.21.15.142:0/298917881 >> v2:172.21.15.150:3300/0 conn(0x7fe808134bb0 0x7fe808136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 --2- 172.21.15.142:0/298917881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809b740 0x7fe808134670 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 -- 172.21.15.142:0/298917881 >> 172.21.15.142:0/298917881 conn(0x7fe808070ef0 msgr2=0x7fe8080712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 -- 172.21.15.142:0/298917881 shutdown_connections 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.401+0000 7fe80d812700 1 -- 172.21.15.142:0/298917881 wait complete. 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient: build_initial_monmap 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient: monmap: 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:40.414 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:40.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 Processor -- start 2024-09-18T10:21:40.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 -- start start 2024-09-18T10:21:40.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient: init 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-a con 0x7fe80809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80809b740 0x7fe808089720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-b con 0x7fe80809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe808134bb0 0x7fe808089c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): picked mon.noname-c con 0x7fe808134bb0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80808ae20 con 0x7fe80809abd0 2024-09-18T10:21:40.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80808c4b0 con 0x7fe80809b740 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80808ca70 con 0x7fe808134bb0 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d812700 10 monclient(hunting): authenticate will time out at 1589.334590s 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe807fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80809b740 msgr2=0x7fe808089720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe807fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80809b740 0x7fe808089720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38106/0 (socket says 172.21.15.142:38106) 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 -- 172.21.15.142:0/1623646579 learned_addr learned my addr 172.21.15.142:0/1623646579 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): get_auth_request con 0x7fe80809abd0 auth_method 0 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe808134bb0 0x7fe808089c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): get_auth_request con 0x7fe808134bb0 auth_method 0 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80d011700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient(hunting): handle_auth_done global_id 124380 payload 824 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe808134bb0 msgr2=0x7fe808089c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe808134bb0 0x7fe808089c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80809b740 msgr2=0x7fe808089720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe80809b740 0x7fe808089720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 monclient: found mon.noname-a 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.405+0000 7fe80c810700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0094f0 con 0x7fe80809abd0 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80c810700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 msgr2=0x7fe8080891e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80c810700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c7c0 tx=0x7fe7f4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40074f0 con 0x7fe80809abd0 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: dump: 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.426 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 5 monclient: authenticate success, global_id 124380 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4007650 con 0x7fe80809abd0 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 monclient: _renew_subs 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80808aa00 con 0x7fe80809abd0 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4005a60 con 0x7fe80809abd0 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe806ffd700 4 set_mon_vals no callback set 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 10 monclient: dump: 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 monclient: _renew_subs 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80808b550 con 0x7fe80809abd0 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe7f401b3b0 con 0x7fe80809abd0 2024-09-18T10:21:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 0x7fe80806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.409+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80808d870 con 0x7fe80808a1a0 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7ec035a70 0x7fe7ec037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 0x7fe80806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe7f4048320 con 0x7fe80809abd0 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient: get_auth_request con 0x7fe80808a1a0 auth_method 0 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe80d011700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7ec035a70 0x7fe7ec037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 10 monclient(hunting): handle_auth_done global_id 124383 payload 824 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe80808cc30 con 0x7fe80808a1a0 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe80d011700 10 monclient: get_auth_request con 0x7fe7ec035a70 auth_method 0 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 msgr2=0x7fe80806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe807fff700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 0x7fe80806c370 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009fd0 tx=0x7fe7fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc020070 con 0x7fe80808a1a0 2024-09-18T10:21:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 10 monclient: dump: 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe80d011700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7ec035a70 0x7fe7ec037f20 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe80809fda0 tx=0x7fe80000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe7fc011e60 con 0x7fe80808a1a0 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.413+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 msgr2=0x7fe80806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.417+0000 7fe804ff9700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 0x7fe80806c370 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009fd0 tx=0x7fe7fc017000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.417+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 msgr2=0x7fe80806c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:40.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.417+0000 7fe804ff9700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80808a1a0 0x7fe80806c370 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009fd0 tx=0x7fe7fc017000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe80d812700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe80d812700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 0x7fe80806ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe80d812700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8080648a0 con 0x7fe80806c8b0 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 0x7fe80806ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient: get_auth_request con 0x7fe80806c8b0 auth_method 0 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 10 monclient(hunting): handle_auth_done global_id 124386 payload 824 2024-09-18T10:21:40.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.425+0000 7fe807fff700 1 -- 172.21.15.142:0/1623646579 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe808072a80 con 0x7fe80806c8b0 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe807fff700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 msgr2=0x7fe80806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe807fff700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 0x7fe80806ccb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8080721d0 tx=0x7fe7fc020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc024ce0 con 0x7fe80806c8b0 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 10 monclient: dump: 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.435 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7fe7fc0233b0 con 0x7fe80806c8b0 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 msgr2=0x7fe80806ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 0x7fe80806ccb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8080721d0 tx=0x7fe7fc020000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 msgr2=0x7fe80806ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe804ff9700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80806c8b0 0x7fe80806ccb0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe8080721d0 tx=0x7fe7fc020000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.437 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 124, 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:40.438 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.439 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:40.440 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:40.441 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.442 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:40.443 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.444 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:40.445 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.447 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.448 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.449 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.450 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:40.450 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7ec035a70 msgr2=0x7fe7ec037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe7ec035a70 0x7fe7ec037f20 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe80809fda0 tx=0x7fe80000d040 comp rx=0 tx=0).stop 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 10 monclient: shutdown 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 msgr2=0x7fe8080891e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d011700 1 -- 172.21.15.142:0/1623646579 reap_dead start 2024-09-18T10:21:40.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400c7c0 tx=0x7fe7f4007df0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 shutdown_connections 2024-09-18T10:21:40.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 --2- 172.21.15.142:0/1623646579 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe80809abd0 0x7fe8080891e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 >> 172.21.15.142:0/1623646579 conn(0x7fe808070ef0 msgr2=0x7fe808135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 shutdown_connections 2024-09-18T10:21:40.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.429+0000 7fe80d812700 1 -- 172.21.15.142:0/1623646579 wait complete. 2024-09-18T10:21:40.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:21:40.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:21:40.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:21:40.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: get_monmap_and_config 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: build_initial_monmap 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: monmap: 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:21:40.561089+0000 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:21:40.561089+0000 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:21:40.563 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:21:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:21:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:21:40.564 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 Processor -- start 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 -- start start 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: init 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 0x7f49a40960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-b con 0x7f49a409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49a409b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49a409c5e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a409dc90 con 0x7f49a409b740 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a409afa0 con 0x7f49a409c5e0 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a409b500 con 0x7f49a409abd0 2024-09-18T10:21:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49ab341700 10 monclient(hunting): authenticate will time out at 1589.486907s 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49a9b3e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f49a409b740 0x7f49a409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:48430/0 (socket says 172.21.15.142:48430) 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49a9b3e700 1 -- 172.21.15.142:0/3732328909 learned_addr learned my addr 172.21.15.142:0/3732328909 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aab40700 1 -- 172.21.15.142:0/3732328909 >> v1:172.21.15.150:6789/0 conn(0x7f49a409c5e0 legacy=0x7f49a4137690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 1 --2- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 0x7f49a40960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): get_auth_request con 0x7f49a409abd0 auth_method 0 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient(hunting): handle_auth_done global_id 75284 payload 274 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 1 -- 172.21.15.142:0/3732328909 >> v1:172.21.15.150:6789/0 conn(0x7f49a409c5e0 legacy=0x7f49a4137690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 1 -- 172.21.15.142:0/3732328909 >> v1:172.21.15.142:6789/0 conn(0x7f49a409b740 legacy=0x7f49a409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 1 monclient: found mon.noname-b 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:21:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.557+0000 7f49aa33f700 1 -- 172.21.15.142:0/3732328909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a406f810 con 0x7f49a409abd0 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49aa33f700 1 -- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 msgr2=0x7f49a40960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49aa33f700 1 --2- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 0x7f49a40960e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4998009910 tx=0x7f49980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=182318786fab16be server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 1 -- 172.21.15.142:0/3732328909 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499800ba70 con 0x7f49a409abd0 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: dump: 2024-09-18T10:21:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 5 monclient: authenticate success, global_id 75284 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 1 -- 172.21.15.142:0/3732328909 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499800bbd0 con 0x7f49a409abd0 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 10 monclient: get_monmap_and_config success 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 4 set_mon_vals no callback set 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 1 -- 172.21.15.142:0/3732328909 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998011420 con 0x7f49a409abd0 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49a933d700 10 monclient: dump: 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 10 monclient: shutdown 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 -- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 msgr2=0x7f49a40960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 --2- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 0x7f49a40960e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4998009910 tx=0x7f49980091a0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 -- 172.21.15.142:0/3732328909 shutdown_connections 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 --2- 172.21.15.142:0/3732328909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409abd0 0x7f49a40960e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 -- 172.21.15.142:0/3732328909 >> 172.21.15.142:0/3732328909 conn(0x7f49a4070ef0 msgr2=0x7f49a40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 -- 172.21.15.142:0/3732328909 shutdown_connections 2024-09-18T10:21:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 1 -- 172.21.15.142:0/3732328909 wait complete. 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 10 monclient: build_initial_monmap 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.561+0000 7f49ab341700 10 monclient: monmap: 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 Processor -- start 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 -- start start 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient: init 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient: _reopen_session rank -1 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 0x7f49a413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-a con 0x7f49a409abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a409b740 0x7f49a4094480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-b con 0x7f49a409b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 0x7f49a408c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): picked mon.noname-c con 0x7f49a409c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a409d240 con 0x7f49a409abd0 2024-09-18T10:21:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40a01e0 con 0x7f49a409b740 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a408f960 con 0x7f49a409c5e0 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): _renew_subs 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49ab341700 10 monclient(hunting): authenticate will time out at 1589.493004s 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 0x7f49a413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49a9b3e700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a409b740 msgr2=0x7f49a4094480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 0x7f49a408c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49a9b3e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a409b740 0x7f49a4094480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 0x7f49a408c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:34232/0 (socket says 172.21.15.142:34232) 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 0x7f49a413f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38126/0 (socket says 172.21.15.142:38126) 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 1 -- 172.21.15.142:0/1097628944 learned_addr learned my addr 172.21.15.142:0/1097628944 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:21:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): get_auth_request con 0x7f49a409c5e0 auth_method 0 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): get_auth_request con 0x7f49a409abd0 auth_method 0 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient(hunting): handle_auth_done global_id 124389 payload 824 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 msgr2=0x7f49a408c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 0x7f49a408c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a409b740 msgr2=0x7f49a4094480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49a409b740 0x7f49a4094480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient: _finish_hunting 0 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 monclient: found mon.noname-a 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aa33f700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4998009420 con 0x7f49a409abd0 2024-09-18T10:21:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.565+0000 7f49aab40700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a409c5e0 0x7f49a408c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49aa33f700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 msgr2=0x7f49a413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49aa33f700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 0x7f49a413f920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49a4069270 tx=0x7f4998016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499801b450 con 0x7f49a409abd0 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: dump: 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: _finish_auth 0 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 5 monclient: authenticate success, global_id 124389 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4998020070 con 0x7f49a409abd0 2024-09-18T10:21:40.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499801b7e0 con 0x7f49a409abd0 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 10 monclient: dump: 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a8b3c700 4 set_mon_vals no callback set 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 monclient: _renew_subs 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a408f680 con 0x7f49a409abd0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 monclient: _renew_subs 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a408e720 con 0x7f49a409abd0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 0x7f49a408c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a40648a0 con 0x7f49a413fe60 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 0x7f49a408c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4998022030 con 0x7f49a409abd0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient: get_auth_request con 0x7f49a413fe60 auth_method 0 2024-09-18T10:21:40.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4990035aa0 0x7f4990037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f499804acf0 con 0x7f49a409abd0 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_done global_id 75290 payload 824 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49a9b3e700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49a408e720 con 0x7f49a413fe60 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49aab40700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4990035aa0 0x7f4990037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.569+0000 7f49aab40700 10 monclient: get_auth_request con 0x7f4990035aa0 auth_method 0 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a9b3e700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 msgr2=0x7f49a408c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a9b3e700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 0x7f49a408c630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49a408d6f0 tx=0x7f4994007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994005e20 con 0x7f49a413fe60 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 10 monclient: dump: 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49aab40700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4990035aa0 0x7f4990037f50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f499c009fd0 tx=0x7f499c0097d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f49940070d0 con 0x7f49a413fe60 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 msgr2=0x7f49a408c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 0x7f49a408c630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49a408d6f0 tx=0x7f4994007f90 comp rx=0 tx=0).stop 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 msgr2=0x7f49a408c630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:40.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.573+0000 7f49a27fc700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a413fe60 0x7f49a408c630 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f49a408d6f0 tx=0x7f4994007f90 comp rx=0 tx=0).stop 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49ab341700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49ab341700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 0x7f49a408cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49ab341700 10 monclient(hunting): start opening mon connection 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a408fba0 con 0x7f49a40949c0 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 0x7f49a408cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:21:40.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient: get_auth_request con 0x7f49a40949c0 auth_method 0 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 10 monclient(hunting): handle_auth_done global_id 75293 payload 824 2024-09-18T10:21:40.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 1 -- 172.21.15.142:0/1097628944 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f49a4072a80 con 0x7f49a40949c0 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 msgr2=0x7f49a408cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a9b3e700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 0x7f49a408cb70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4994014000 tx=0x7f4994015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994023070 con 0x7f49a40949c0 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:21:40.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.585+0000 7f49a27fc700 10 monclient: dump: 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:21:40.594 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f499401f610 con 0x7f49a40949c0 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 msgr2=0x7f49a408cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 0x7f49a408cb70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4994014000 tx=0x7f4994015000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 msgr2=0x7f49a408cb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:21:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49a27fc700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49a40949c0 0x7f49a408cb70 secure :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4994014000 tx=0x7f4994015000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 124, 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:21:40.596 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:21:40.597 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:21:40.598 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:21:40.599 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.600 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.601 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.602 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.603 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.604 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.605 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.606 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:21:40.607 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:21:40.608 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:21:40.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4990035aa0 msgr2=0x7f4990037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4990035aa0 0x7f4990037f50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f499c009fd0 tx=0x7f499c0097d0 comp rx=0 tx=0).stop 2024-09-18T10:21:40.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 10 monclient: shutdown 2024-09-18T10:21:40.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 msgr2=0x7f49a413f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 --2- 172.21.15.142:0/1097628944 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49a409abd0 0x7f49a413f920 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f49a4069270 tx=0x7f4998016000 comp rx=0 tx=0).stop 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49aab40700 1 -- 172.21.15.142:0/1097628944 reap_dead start 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49aab40700 1 -- 172.21.15.142:0/1097628944 reap_dead start 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 shutdown_connections 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 >> 172.21.15.142:0/1097628944 conn(0x7f49a4070ef0 msgr2=0x7f49a409def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 shutdown_connections 2024-09-18T10:21:40.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:21:40.589+0000 7f49ab341700 1 -- 172.21.15.142:0/1097628944 wait complete. 2024-09-18T10:21:40.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:21:40.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:22:00.612 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:22:00.612 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:22:00.612 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:22:00.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:22:00.616 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:22:00.616 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T10:22:00.616 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T10:22:00.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:22:20.617 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T10:22:20.618 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T10:22:20.618 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:22:20.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: get_monmap_and_config 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: build_initial_monmap 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: monmap: 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:20.733022+0000 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:20.733022+0000 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.735 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:20.736 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:20.736 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:20.736 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:20.736 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:20.736 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 Processor -- start 2024-09-18T10:22:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 -- start start 2024-09-18T10:22:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: init 2024-09-18T10:22:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:20.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): picked mon.noname-c con 0x7f19540a1cf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f195412f810 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19540a02d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195409fb90 con 0x7f195412f810 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195409d990 con 0x7f19540a02d0 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195409dc70 con 0x7f19540a1cf0 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1959179700 10 monclient(hunting): authenticate will time out at 1629.658490s 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f19537fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f195412f810 0x7f195409f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47848/0 (socket says 172.21.15.142:47848) 2024-09-18T10:22:20.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f19537fe700 1 -- 172.21.15.142:0/2681222753 learned_addr learned my addr 172.21.15.142:0/2681222753 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): get_auth_request con 0x7f19540a1cf0 auth_method 0 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.728+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f19537fe700 1 -- 172.21.15.142:0/2681222753 >> v1:172.21.15.142:6789/0 conn(0x7f195412f810 legacy=0x7f195409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978393894 0 0) 0x7f195409fb90 con 0x7f195412f810 2024-09-18T10:22:20.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 10 monclient(hunting): my global_id is 124392 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1934003570 con 0x7f195412f810 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1958978700 1 -- 172.21.15.142:0/2681222753 >> v1:172.21.15.150:6789/0 conn(0x7f19540a02d0 legacy=0x7f1954139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265403386 0 0) 0x7f1934003570 con 0x7f195412f810 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 10 monclient(hunting): handle_auth_done global_id 154099 payload 274 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> v1:172.21.15.150:6789/0 conn(0x7f19540a02d0 legacy=0x7f1954139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> v1:172.21.15.142:6789/0 conn(0x7f195412f810 legacy=0x7f195409f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:20.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 monclient: found mon.noname-c 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540a20c0 con 0x7f19540a1cf0 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1952ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 msgr2=0x7f195412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 msgr2=0x7f195412f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f193c0099e0 tx=0x7f193c0091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:20.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.732+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient: get_auth_request con 0x7f19540a1cf0 auth_method 0 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 0 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient: ms_handle_reset current mon v2:172.21.15.150:3300/0 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 msgr2=0x7f195412f030 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T10:22:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.150:3300/0 conn(0x7f19540a1cf0 0x7f195412f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient: _add_conns ranks=[3,2] 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.142:3300/0 conn(0x7f1934003810 0x7f1934003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): picked mon.noname-a con 0x7f1934003810 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934004810 0x7f1934004cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): picked mon.noname-b con 0x7f1934004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c001960 con 0x7f1934003810 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19340051f0 con 0x7f1934004810 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1952ffd700 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.150:3300/0 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> v2:172.21.15.142:3300/0 conn(0x7f1934003810 0x7f1934003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934004810 0x7f1934004cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): get_auth_request con 0x7f1934003810 auth_method 0 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): get_auth_request con 0x7f1934004810 auth_method 0 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient(hunting): handle_auth_done global_id 154099 payload 293 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934004810 msgr2=0x7f1934004cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934004810 0x7f1934004cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 monclient: found mon.noname-a 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.936+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934003570 con 0x7f1934003810 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1953fff700 1 -- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1934003810 msgr2=0x7f1934003c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1953fff700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1934003810 0x7f1934003c10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f193c0093f0 tx=0x7f193c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3ea00b7603fab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c020070 con 0x7f1934003810 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: dump: 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: _finish_auth 0 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 5 monclient: authenticate success, global_id 154099 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c015070 con 0x7f1934003810 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 4 set_mon_vals no callback set 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 1 -- 172.21.15.142:0/2681222753 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c001be0 con 0x7f1934003810 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1952ffd700 10 monclient: dump: 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:20.951 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 10 monclient: shutdown 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 -- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1934003810 msgr2=0x7f1934003c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1934003810 0x7f1934003c10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f193c0093f0 tx=0x7f193c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1958978700 1 -- 172.21.15.142:0/2681222753 reap_dead start 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 -- 172.21.15.142:0/2681222753 shutdown_connections 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 --2- 172.21.15.142:0/2681222753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1934003810 0x7f1934003c10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 -- 172.21.15.142:0/2681222753 >> 172.21.15.142:0/2681222753 conn(0x7f1954071820 msgr2=0x7f19540a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 -- 172.21.15.142:0/2681222753 shutdown_connections 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 1 -- 172.21.15.142:0/2681222753 wait complete. 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 10 monclient: build_initial_monmap 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.940+0000 7f1959179700 10 monclient: monmap: 2024-09-18T10:22:20.952 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:20.953 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 Processor -- start 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 -- start start 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient: init 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): picked mon.noname-c con 0x7f1934003810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): picked mon.noname-b con 0x7f1954142630 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f195406b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): picked mon.noname-a con 0x7f195406b420 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195409ce30 con 0x7f195406b420 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195406e9f0 con 0x7f1954142630 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195412ef50 con 0x7f1934003810 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1959179700 10 monclient(hunting): authenticate will time out at 1629.873467s 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f195406b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51310/0 (socket says 172.21.15.142:51310) 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 -- 172.21.15.142:0/867925370 learned_addr learned my addr 172.21.15.142:0/867925370 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 10 monclient(hunting): get_auth_request con 0x7f1934003810 auth_method 0 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): get_auth_request con 0x7f195406b420 auth_method 0 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): get_auth_request con 0x7f1954142630 auth_method 0 2024-09-18T10:22:20.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 msgr2=0x7f19541420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 msgr2=0x7f19541420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1953fff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:20.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient(hunting): handle_auth_done global_id 154105 payload 824 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 msgr2=0x7f19541420f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f19541420f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 msgr2=0x7f195406b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f195406b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 monclient: found mon.noname-b 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954142f40 con 0x7f1954142630 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.944+0000 7f1958978700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f195406b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 msgr2=0x7f1954142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f194c00ae40 tx=0x7f194c00a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:20.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c012690 con 0x7f1954142630 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: dump: 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:20.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: _finish_auth 0 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 5 monclient: authenticate success, global_id 154105 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _renew_subs 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c0127f0 con 0x7f1954142630 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1954139a30 con 0x7f1954142630 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c012bb0 con 0x7f1954142630 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:20.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 10 monclient: dump: 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _renew_subs 2024-09-18T10:22:20.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195406dcd0 con 0x7f1954142630 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f19527fc700 4 set_mon_vals no callback set 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f194c00e040 con 0x7f1954142630 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195406cf20 con 0x7f1954142630 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.948+0000 7f194bfff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 0x7f1938037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.952+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f194c00b040 con 0x7f1954142630 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.952+0000 7f1953fff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 0x7f1938037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:20.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.952+0000 7f1953fff700 10 monclient: get_auth_request con 0x7f1938035af0 auth_method 0 2024-09-18T10:22:20.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.952+0000 7f1953fff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 0x7f1938037fa0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f193c009fa0 tx=0x7f193c01b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:20.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.960+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f194c00f550 con 0x7f1954142630 2024-09-18T10:22:20.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.960+0000 7f194bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:20.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:20.960+0000 7f194bfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1959179700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1959179700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1959179700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1954072a80 con 0x7f1954142630 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 msgr2=0x7f1954142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 msgr2=0x7f1954142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f194c00ae40 tx=0x7f194c00a440 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f194c00ae40 tx=0x7f194c00a440 comp rx=0 tx=0).stop 2024-09-18T10:22:21.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 msgr2=0x7f1954142a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1954142630 0x7f1954142a00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f1938040aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): picked mon.c con 0x7f1934003810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1938041270 0x7f19380436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): picked mon.b con 0x7f1938041270 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f1938046190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): picked mon.a con 0x7f195406b420 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1938046b10 con 0x7f195406b420 2024-09-18T10:22:21.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1938046c50 con 0x7f1938041270 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1938046de0 con 0x7f1934003810 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f194bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f1938040aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): get_auth_request con 0x7f1934003810 auth_method 0 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f1938046190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): get_auth_request con 0x7f195406b420 auth_method 0 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1938041270 0x7f19380436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): get_auth_request con 0x7f1938041270 auth_method 0 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1953fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f1958978700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient(hunting): handle_auth_done global_id 154105 payload 293 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1938041270 msgr2=0x7f19380436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1938041270 0x7f19380436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 msgr2=0x7f1938046190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f1938046190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 monclient: found mon.c 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f1938047000 con 0x7f1934003810 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:21.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1954072a80 con 0x7f1934003810 2024-09-18T10:22:21.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.188+0000 7f19537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f19537fe700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 msgr2=0x7f1938040aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f19537fe700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f1938040aa0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f194c047dc0 tx=0x7f194c04e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c00f550 con 0x7f1934003810 2024-09-18T10:22:21.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: dump: 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194c047ed0 con 0x7f1934003810 2024-09-18T10:22:21.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f19527fc700 4 set_mon_vals no callback set 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f194c019310 con 0x7f1934003810 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 1 -- 172.21.15.142:0/867925370 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f194c047850 con 0x7f1934003810 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.192+0000 7f194bfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:21.205 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:22:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 msgr2=0x7f1938037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 0x7f1938037fa0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f193c009fa0 tx=0x7f193c01b000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 10 monclient: shutdown 2024-09-18T10:22:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 msgr2=0x7f1938040aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f1938040aa0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f194c047dc0 tx=0x7f194c04e000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 shutdown_connections 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f195406b420 0x7f1938046190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1938041270 0x7f19380436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1934003810 0x7f1938040aa0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 --2- 172.21.15.142:0/867925370 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1938035af0 0x7f1938037fa0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 >> 172.21.15.142:0/867925370 conn(0x7f1954071820 msgr2=0x7f19540a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 shutdown_connections 2024-09-18T10:22:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.196+0000 7f1959179700 1 -- 172.21.15.142:0/867925370 wait complete. 2024-09-18T10:22:21.214 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:22:21.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:22:21.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:22:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: get_monmap_and_config 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: monmap: 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:21.321919+0000 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:21.321919+0000 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:21.325 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 Processor -- start 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 -- start start 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: init 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:22:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f896009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-b con 0x7f896009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-a con 0x7f896009db80 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 0x7f8960096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-c con 0x7f896009c950 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896006fa60 con 0x7f896009db80 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89600958c0 con 0x7f896009c950 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8960095ef0 con 0x7f896009abd0 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f89660bf700 10 monclient(hunting): authenticate will time out at 1630.247309s 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f896009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41612/0 (socket says 172.21.15.142:41612) 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 1 -- 172.21.15.142:0/1550962586 learned_addr learned my addr 172.21.15.142:0/1550962586 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): get_auth_request con 0x7f896009db80 auth_method 0 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): get_auth_request con 0x7f896009abd0 auth_method 0 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895ffff700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 0x7f8960096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895ffff700 10 monclient(hunting): get_auth_request con 0x7f896009c950 auth_method 0 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.316+0000 7f895ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 10 monclient(hunting): handle_auth_done global_id 75299 payload 274 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 -- 172.21.15.142:0/1550962586 >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 msgr2=0x7f8960096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 0x7f8960096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 -- 172.21.15.142:0/1550962586 >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 msgr2=0x7f896009c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 -- 172.21.15.142:0/1550962586 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896009afa0 con 0x7f896009abd0 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895effd700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:22:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895ffff700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 0x7f8960096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:22:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 -- 172.21.15.142:0/1550962586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 msgr2=0x7f896009bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895f7fe700 1 --2- 172.21.15.142:0/1550962586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f896009bed0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f896006fdb0 tx=0x7f89540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=befedbeda0392de7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 1 -- 172.21.15.142:0/1550962586 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954011840 con 0x7f896009abd0 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: dump: 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f89660bf700 5 monclient: authenticate success, global_id 75299 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 1 -- 172.21.15.142:0/1550962586 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540119a0 con 0x7f896009abd0 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f89660bf700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 1 -- 172.21.15.142:0/1550962586 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954011d40 con 0x7f896009abd0 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f89660bf700 4 set_mon_vals no callback set 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.320+0000 7f895e7fc700 10 monclient: dump: 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.337 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: shutdown 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- 172.21.15.142:0/1550962586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 msgr2=0x7f896009bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- 172.21.15.142:0/1550962586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f896009bed0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f896006fdb0 tx=0x7f89540091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- 172.21.15.142:0/1550962586 shutdown_connections 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.150:3300/0 conn(0x7f896009c950 0x7f8960096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- 172.21.15.142:0/1550962586 >> v2:172.21.15.142:3300/0 conn(0x7f896009db80 0x7f896009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- 172.21.15.142:0/1550962586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f896009bed0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- 172.21.15.142:0/1550962586 >> 172.21.15.142:0/1550962586 conn(0x7f89600a14e0 msgr2=0x7f89600a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- 172.21.15.142:0/1550962586 shutdown_connections 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- 172.21.15.142:0/1550962586 wait complete. 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: monmap: 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:21.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 Processor -- start 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- start start 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: init 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f8960141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-c con 0x7f896009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896009c950 0x7f8960141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-b con 0x7f896009c950 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): picked mon.noname-a con 0x7f896009db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8960091910 con 0x7f896009db80 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8960097c90 con 0x7f896009c950 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896006efd0 con 0x7f896009abd0 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.324+0000 7f89660bf700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 10 monclient(hunting): authenticate will time out at 1630.256022s 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f8960141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41614/0 (socket says 172.21.15.142:41614) 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 learned_addr learned my addr 172.21.15.142:0/4141416683 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): get_auth_request con 0x7f896009abd0 auth_method 0 2024-09-18T10:22:21.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): get_auth_request con 0x7f896009db80 auth_method 0 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896009c950 0x7f8960141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): get_auth_request con 0x7f896009c950 auth_method 0 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient(hunting): handle_auth_done global_id 124401 payload 824 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 msgr2=0x7f8960141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f896009abd0 0x7f8960141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896009c950 msgr2=0x7f8960141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896009c950 0x7f8960141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 monclient: found mon.noname-a 2024-09-18T10:22:21.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8950009ae0 con 0x7f896009db80 2024-09-18T10:22:21.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 msgr2=0x7f89601421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895ffff700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f895800e870 tx=0x7f8958008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958013620 con 0x7f896009db80 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: dump: 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 5 monclient: authenticate success, global_id 124401 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958013780 con 0x7f896009db80 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958013b20 con 0x7f896009db80 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 10 monclient: _renew_subs 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8954009420 con 0x7f896009db80 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f895d7fa700 4 set_mon_vals no callback set 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89477fe700 10 monclient: dump: 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.350 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 10 monclient: _renew_subs 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.328+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89600925e0 con 0x7f896009db80 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89660bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89660bf700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 0x7f896008a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8960064950 con 0x7f8960089ff0 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895ffff700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f895800e870 tx=0x7f8958008040 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f8958012370 mgrmap(e 5) v1 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8958012370 con 0x7f896009db80 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 0x7f896008a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient: get_auth_request con 0x7f8960089ff0 auth_method 0 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f89477fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f894c035ac0 0x7f894c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 10 monclient(hunting): handle_auth_done global_id 124404 payload 824 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895f7fe700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89600696a0 con 0x7f8960089ff0 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895effd700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f894c035ac0 0x7f894c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.332+0000 7f895effd700 10 monclient: get_auth_request con 0x7f894c035ac0 auth_method 0 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f895f7fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 msgr2=0x7f896008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f895f7fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 0x7f896008a3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f896006fdb0 tx=0x7f8954017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954021070 con 0x7f8960089ff0 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 10 monclient: dump: 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.353 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f895400bc50 con 0x7f8960089ff0 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 msgr2=0x7f896008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 0x7f896008a3c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f896006fdb0 tx=0x7f8954017000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 msgr2=0x7f896008a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f89477fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8960089ff0 0x7f896008a3c0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f896006fdb0 tx=0x7f8954017000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.336+0000 7f895effd700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f894c035ac0 0x7f894c037f70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8960064050 tx=0x7f895000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f89660bf700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:21.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f89660bf700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 0x7f896008dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f89660bf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8960091320 con 0x7f896008a900 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 0x7f896008dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient: get_auth_request con 0x7f896008a900 auth_method 0 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 10 monclient(hunting): handle_auth_done global_id 124407 payload 824 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.344+0000 7f895f7fe700 1 -- 172.21.15.142:0/4141416683 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8960067950 con 0x7f896008a900 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f895f7fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 msgr2=0x7f896008dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f895f7fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 0x7f896008dca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8960064700 tx=0x7f895401e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954031960 con 0x7f896008a900 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 10 monclient: dump: 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.358 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f8954025410 con 0x7f896008a900 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 msgr2=0x7f896008dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 0x7f896008dca0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8960064700 tx=0x7f895401e070 comp rx=0 tx=0).stop 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 msgr2=0x7f896008dca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896008a900 0x7f896008dca0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8960064700 tx=0x7f895401e070 comp rx=0 tx=0).stop 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:21.359 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.360 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.361 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.362 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.365 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.369 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:21.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:21.371 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f894c035ac0 msgr2=0x7f894c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f894c035ac0 0x7f894c037f70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f8960064050 tx=0x7f895000b040 comp rx=0 tx=0).stop 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 10 monclient: shutdown 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f895ffff700 1 -- 172.21.15.142:0/4141416683 reap_dead start 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 msgr2=0x7f89601421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f895800e870 tx=0x7f8958008040 comp rx=0 tx=0).stop 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89477fe700 1 -- 172.21.15.142:0/4141416683 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f8958048c10 con 0x7f896009db80 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 shutdown_connections 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 --2- 172.21.15.142:0/4141416683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f896009db80 0x7f89601421e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 >> 172.21.15.142:0/4141416683 conn(0x7f89600a14e0 msgr2=0x7f896009f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 shutdown_connections 2024-09-18T10:22:21.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.348+0000 7f89660bf700 1 -- 172.21.15.142:0/4141416683 wait complete. 2024-09-18T10:22:21.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:21.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:22:21.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:22:21.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:22:21.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: monmap: 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:21.487894+0000 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:21.487894+0000 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:21.490 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 Processor -- start 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 -- start start 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: init 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55bc09c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55bc134880 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f55bc138380 0x7f55bc13a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f55bc138380 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc0a6620 con 0x7f55bc134880 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc0a1c70 con 0x7f55bc09c2c0 2024-09-18T10:22:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55bc09ab90 con 0x7f55bc138380 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bbfff700 10 monclient(hunting): authenticate will time out at 1630.413416s 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f55bc138380 0x7f55bc13a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55ba7fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f55bc134880 0x7f55bc136c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47860/0 (socket says 172.21.15.142:47860) 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55ba7fc700 1 -- 172.21.15.142:0/1575822117 learned_addr learned my addr 172.21.15.142:0/1575822117 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): get_auth_request con 0x7f55bc138380 auth_method 0 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_done global_id 124410 payload 274 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 1 -- 172.21.15.142:0/1575822117 >> v1:172.21.15.150:6789/0 conn(0x7f55bc09c2c0 legacy=0x7f55bc0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 1 -- 172.21.15.142:0/1575822117 >> v1:172.21.15.142:6789/0 conn(0x7f55bc134880 legacy=0x7f55bc136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 1 monclient: found mon.noname-a 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.484+0000 7f55bb7fe700 1 -- 172.21.15.142:0/1575822117 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc09c690 con 0x7f55bc138380 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bb7fe700 1 -- 172.21.15.142:0/1575822117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc138380 msgr2=0x7f55bc13a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bb7fe700 1 --2- 172.21.15.142:0/1575822117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc138380 0x7f55bc13a760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009e60 tx=0x7f55b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394c1eb883ba08e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 1 -- 172.21.15.142:0/1575822117 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b400e070 con 0x7f55bc138380 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: dump: 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 5 monclient: authenticate success, global_id 124410 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 1 -- 172.21.15.142:0/1575822117 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4012430 con 0x7f55bc138380 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:21.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 4 set_mon_vals no callback set 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 1 -- 172.21.15.142:0/1575822117 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40127d0 con 0x7f55bc138380 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55b9ffb700 10 monclient: dump: 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.499 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 10 monclient: shutdown 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- 172.21.15.142:0/1575822117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc138380 msgr2=0x7f55bc13a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 --2- 172.21.15.142:0/1575822117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc138380 0x7f55bc13a760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009e60 tx=0x7f55b4009780 comp rx=0 tx=0).stop 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- 172.21.15.142:0/1575822117 shutdown_connections 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 --2- 172.21.15.142:0/1575822117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc138380 0x7f55bc13a760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- 172.21.15.142:0/1575822117 >> 172.21.15.142:0/1575822117 conn(0x7f55bc0a5720 msgr2=0x7f55bc0a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- 172.21.15.142:0/1575822117 shutdown_connections 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- 172.21.15.142:0/1575822117 wait complete. 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 10 monclient: monmap: 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 Processor -- start 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 1 -- start start 2024-09-18T10:22:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.488+0000 7f55bbfff700 10 monclient: init 2024-09-18T10:22:21.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:22:21.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 0x7f55bc06b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f55bc09c2c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc134880 0x7f55bc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f55bc134880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc138380 0x7f55bc06bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f55bc138380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc06ee20 con 0x7f55bc09c2c0 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc13a9c0 con 0x7f55bc134880 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc06eb40 con 0x7f55bc138380 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient(hunting): authenticate will time out at 1630.420204s 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 0x7f55bc06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 0x7f55bc06b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41650/0 (socket says 172.21.15.142:41650) 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 -- 172.21.15.142:0/4041115684 learned_addr learned my addr 172.21.15.142:0/4041115684 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): get_auth_request con 0x7f55bc09c2c0 auth_method 0 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc138380 0x7f55bc06bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): get_auth_request con 0x7f55bc138380 auth_method 0 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55ba7fc700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc134880 0x7f55bc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55ba7fc700 10 monclient(hunting): get_auth_request con 0x7f55bc134880 auth_method 0 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient(hunting): handle_auth_done global_id 124413 payload 824 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc138380 msgr2=0x7f55bc06bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc138380 0x7f55bc06bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc134880 msgr2=0x7f55bc06b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc134880 0x7f55bc06b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 monclient: found mon.noname-a 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4009a20 con 0x7f55bc09c2c0 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 msgr2=0x7f55bc06b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55baffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 0x7f55bc06b280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b5e0 tx=0x7f55ac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00f720 con 0x7f55bc09c2c0 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: dump: 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.509 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac00f880 con 0x7f55bc09c2c0 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 5 monclient: authenticate success, global_id 124413 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _renew_subs 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55bc06e8c0 con 0x7f55bc09c2c0 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00fc40 con 0x7f55bc09c2c0 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55a2ffd700 10 monclient: dump: 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55b8ff9700 4 set_mon_vals no callback set 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _renew_subs 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.492+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc06d900 con 0x7f55bc09c2c0 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f55ac011610 con 0x7f55bc09c2c0 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bbfff700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 0x7f55bc09cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc064750 con 0x7f55bc06c240 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 0x7f55bc09cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient: get_auth_request con 0x7f55bc06c240 auth_method 0 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55a2ffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55a8035ac0 0x7f55a8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f55ac048af0 con 0x7f55bc09c2c0 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_done global_id 75308 payload 824 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bb7fe700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55a8035ac0 0x7f55a8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55bc06db80 con 0x7f55bc06c240 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55bb7fe700 10 monclient: get_auth_request con 0x7f55a8035ac0 auth_method 0 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 msgr2=0x7f55bc09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.496+0000 7f55ba7fc700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 0x7f55bc09cb00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55b000e740 tx=0x7f55b0009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0007170 con 0x7f55bc06c240 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 10 monclient: dump: 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.514 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55bb7fe700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55a8035ac0 0x7f55a8037f70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f55b40099f0 tx=0x7f55b4018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f55b00072d0 con 0x7f55bc06c240 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 msgr2=0x7f55bc09cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 0x7f55bc09cb00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55b000e740 tx=0x7f55b0009040 comp rx=0 tx=0).stop 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 msgr2=0x7f55bc09cb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.500+0000 7f55a2ffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc06c240 0x7f55bc09cb00 secure :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55b000e740 tx=0x7f55b0009040 comp rx=0 tx=0).stop 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55bbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55bbfff700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 0x7f55bc09d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55bbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc06ecb0 con 0x7f55bc09d040 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 0x7f55bc09d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient: get_auth_request con 0x7f55bc09d040 auth_method 0 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 10 monclient(hunting): handle_auth_done global_id 75311 payload 824 2024-09-18T10:22:21.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.508+0000 7f55ba7fc700 1 -- 172.21.15.142:0/4041115684 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55bc06ef60 con 0x7f55bc09d040 2024-09-18T10:22:21.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55ba7fc700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 msgr2=0x7f55bc09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55ba7fc700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 0x7f55bc09d440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55b0017000 tx=0x7f55b001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b00208a0 con 0x7f55bc09d040 2024-09-18T10:22:21.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 10 monclient: dump: 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f55b001c400 con 0x7f55bc09d040 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 msgr2=0x7f55bc09d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 0x7f55bc09d440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55b0017000 tx=0x7f55b001a000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 msgr2=0x7f55bc09d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55a2ffd700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55bc09d040 0x7f55bc09d440 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:21.521 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.522 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.523 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:21.524 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:21.526 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.528 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.529 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.530 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.531 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.532 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:21.533 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55a8035ac0 msgr2=0x7f55a8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f55a8035ac0 0x7f55a8037f70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f55b40099f0 tx=0x7f55b4018000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 10 monclient: shutdown 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 msgr2=0x7f55bc06b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 --2- 172.21.15.142:0/4041115684 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55bc09c2c0 0x7f55bc06b280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f55ac00b5e0 tx=0x7f55ac009120 comp rx=0 tx=0).stop 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bb7fe700 1 -- 172.21.15.142:0/4041115684 reap_dead start 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bb7fe700 1 -- 172.21.15.142:0/4041115684 reap_dead start 2024-09-18T10:22:21.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 shutdown_connections 2024-09-18T10:22:21.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 >> 172.21.15.142:0/4041115684 conn(0x7f55bc0a5720 msgr2=0x7f55bc0a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 shutdown_connections 2024-09-18T10:22:21.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.512+0000 7f55bbfff700 1 -- 172.21.15.142:0/4041115684 wait complete. 2024-09-18T10:22:21.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:21.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:22:21.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:22:21.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: get_monmap_and_config 2024-09-18T10:22:21.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: monmap: 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:21.648738+0000 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:21.648738+0000 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:21.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 Processor -- start 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 -- start start 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: init 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 0x7f6f340989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-b con 0x7f6f3406f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f34098f00 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f340992f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3409c330 con 0x7f6f340992f0 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3409c530 con 0x7f6f34098f00 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3409b030 con 0x7f6f3406f3b0 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3ac14700 10 monclient(hunting): authenticate will time out at 1630.574579s 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3a413700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6f340992f0 0x7f6f34136d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47864/0 (socket says 172.21.15.142:47864) 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f3a413700 1 -- 172.21.15.142:0/3176296236 learned_addr learned my addr 172.21.15.142:0/3176296236 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 0x7f6f340989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): get_auth_request con 0x7f6f3406f3b0 auth_method 0 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient(hunting): handle_auth_done global_id 75314 payload 274 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 1 -- 172.21.15.142:0/3176296236 >> v1:172.21.15.150:6789/0 conn(0x7f6f34098f00 legacy=0x7f6f34133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 1 -- 172.21.15.142:0/3176296236 >> v1:172.21.15.142:6789/0 conn(0x7f6f340992f0 legacy=0x7f6f34136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 1 monclient: found mon.noname-b 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.644+0000 7f6f39c12700 1 -- 172.21.15.142:0/3176296236 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3409c0f0 con 0x7f6f3406f3b0 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f39c12700 1 -- 172.21.15.142:0/3176296236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 msgr2=0x7f6f340989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f39c12700 1 --2- 172.21.15.142:0/3176296236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 0x7f6f340989c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009910 tx=0x7f6f300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b59da0d828ea02b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 1 -- 172.21.15.142:0/3176296236 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3000ba70 con 0x7f6f3406f3b0 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: dump: 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.660 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 5 monclient: authenticate success, global_id 75314 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 1 -- 172.21.15.142:0/3176296236 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3000bbd0 con 0x7f6f3406f3b0 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:21.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 4 set_mon_vals no callback set 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 1 -- 172.21.15.142:0/3176296236 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30011420 con 0x7f6f3406f3b0 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f38c10700 10 monclient: dump: 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.662 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 10 monclient: shutdown 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- 172.21.15.142:0/3176296236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 msgr2=0x7f6f340989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/3176296236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 0x7f6f340989c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009910 tx=0x7f6f300091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- 172.21.15.142:0/3176296236 shutdown_connections 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/3176296236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f3406f3b0 0x7f6f340989c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- 172.21.15.142:0/3176296236 >> 172.21.15.142:0/3176296236 conn(0x7f6f34070ef0 msgr2=0x7f6f340712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- 172.21.15.142:0/3176296236 shutdown_connections 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- 172.21.15.142:0/3176296236 wait complete. 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 10 monclient: monmap: 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 Processor -- start 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.648+0000 7f6f3ac14700 1 -- start start 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient: init 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f34098f00 0x7f6f341418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-a con 0x7f6f34098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f340992f0 0x7f6f34094440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-b con 0x7f6f340992f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 0x7f6f3408e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): picked mon.noname-c con 0x7f6f34094980 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f34091660 con 0x7f6f34098f00 2024-09-18T10:22:21.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3406fa20 con 0x7f6f340992f0 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340973b0 con 0x7f6f34094980 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient(hunting): authenticate will time out at 1630.580494s 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f34098f00 0x7f6f341418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 0x7f6f3408e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f34098f00 0x7f6f341418d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41666/0 (socket says 172.21.15.142:41666) 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 0x7f6f3408e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51384/0 (socket says 172.21.15.142:51384) 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 1 -- 172.21.15.142:0/453498703 learned_addr learned my addr 172.21.15.142:0/453498703 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): get_auth_request con 0x7f6f34098f00 auth_method 0 2024-09-18T10:22:21.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): get_auth_request con 0x7f6f34094980 auth_method 0 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f340992f0 0x7f6f34094440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): get_auth_request con 0x7f6f340992f0 auth_method 0 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f39411700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient(hunting): handle_auth_done global_id 75317 payload 824 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f340992f0 msgr2=0x7f6f34094440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f340992f0 0x7f6f34094440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f34098f00 msgr2=0x7f6f341418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6f34098f00 0x7f6f341418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 monclient: found mon.noname-c 2024-09-18T10:22:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009420 con 0x7f6f34094980 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 msgr2=0x7f6f3408e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3a413700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 0x7f6f3408e080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00cb90 tx=0x7f6f2c005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c00a9b0 con 0x7f6f34094980 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:21.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 10 monclient: dump: 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 5 monclient: authenticate success, global_id 75317 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c00ab30 con 0x7f6f34094980 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient: _renew_subs 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.652+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f34142700 con 0x7f6f34094980 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0070f0 con 0x7f6f34094980 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f27fff700 4 set_mon_vals no callback set 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 10 monclient: dump: 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.675 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 10 monclient: _renew_subs 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3408fb30 con 0x7f6f34094980 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 0x7f6f3408e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340648a0 con 0x7f6f3408e5c0 2024-09-18T10:22:21.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6f2c0074e0 con 0x7f6f34094980 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 0x7f6f3408e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient: get_auth_request con 0x7f6f3408e5c0 auth_method 0 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6f18035ac0 0x7f6f18037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 10 monclient(hunting): handle_auth_done global_id 154120 payload 824 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39c12700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f3408eed0 con 0x7f6f3408e5c0 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6f2c049260 con 0x7f6f34094980 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39411700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6f18035ac0 0x7f6f18037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.656+0000 7f6f39411700 10 monclient: get_auth_request con 0x7f6f18035ac0 auth_method 0 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f39c12700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 msgr2=0x7f6f3408e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f39c12700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 0x7f6f3408e990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009860 tx=0x7f6f30016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30021070 con 0x7f6f3408e5c0 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 10 monclient: dump: 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.678 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f39411700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6f18035ac0 0x7f6f18037f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f340a0110 tx=0x7f6f28007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6f3001c660 con 0x7f6f3408e5c0 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 msgr2=0x7f6f3408e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 0x7f6f3408e990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009860 tx=0x7f6f30016000 comp rx=0 tx=0).stop 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 msgr2=0x7f6f3408e990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.660+0000 7f6f25ffb700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f3408e5c0 0x7f6f3408e990 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f3ac14700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 0x7f6f3408a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f3ac14700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f34072a80 con 0x7f6f34089ff0 2024-09-18T10:22:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 0x7f6f3408a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient: get_auth_request con 0x7f6f34089ff0 auth_method 0 2024-09-18T10:22:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.672+0000 7f6f39c12700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f39c12700 10 monclient(hunting): handle_auth_done global_id 154123 payload 824 2024-09-18T10:22:21.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f39c12700 1 -- 172.21.15.142:0/453498703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f340678a0 con 0x7f6f34089ff0 2024-09-18T10:22:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f39c12700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 msgr2=0x7f6f3408a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f39c12700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 0x7f6f3408a3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f340721d0 tx=0x7f6f3001e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3001b650 con 0x7f6f34089ff0 2024-09-18T10:22:21.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 10 monclient: dump: 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.685 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f6f3001c450 con 0x7f6f34089ff0 2024-09-18T10:22:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 msgr2=0x7f6f3408a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 0x7f6f3408a3f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f340721d0 tx=0x7f6f3001e070 comp rx=0 tx=0).stop 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 msgr2=0x7f6f3408a3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.676+0000 7f6f25ffb700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f34089ff0 0x7f6f3408a3f0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6f340721d0 tx=0x7f6f3001e070 comp rx=0 tx=0).stop 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:21.687 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.688 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.689 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:21.690 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:21.691 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:21.692 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.694 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:21.695 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.696 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.697 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:21.698 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:21.699 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6f18035ac0 msgr2=0x7f6f18037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6f18035ac0 0x7f6f18037f70 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6f340a0110 tx=0x7f6f28007ea0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 10 monclient: shutdown 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 msgr2=0x7f6f3408e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 --2- 172.21.15.142:0/453498703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6f34094980 0x7f6f3408e080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00cb90 tx=0x7f6f2c005340 comp rx=0 tx=0).stop 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 reap_dead start 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3a413700 1 -- 172.21.15.142:0/453498703 reap_dead start 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 shutdown_connections 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 >> 172.21.15.142:0/453498703 conn(0x7f6f34070ef0 msgr2=0x7f6f340a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 shutdown_connections 2024-09-18T10:22:21.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.680+0000 7f6f3ac14700 1 -- 172.21.15.142:0/453498703 wait complete. 2024-09-18T10:22:21.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:21.705 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:22:21.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:22:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: get_monmap_and_config 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: monmap: 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:21.809916+0000 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:21.809916+0000 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:21.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 Processor -- start 2024-09-18T10:22:21.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 -- start start 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: init 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc10009b050 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc100098080 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc1000953c0 0x7fc100136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): picked mon.noname-c con 0x7fc1000953c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10009a570 con 0x7fc10009b050 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc10006f900 con 0x7fc100098080 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc100095800 con 0x7fc1000953c0 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.804+0000 7fc105612700 10 monclient(hunting): authenticate will time out at 1630.735981s 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0fffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc10009b050 0x7fc100097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47872/0 (socket says 172.21.15.142:47872) 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0fffff700 1 -- 172.21.15.142:0/2289856544 learned_addr learned my addr 172.21.15.142:0/2289856544 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 1 --2- 172.21.15.142:0/2289856544 >> v2:172.21.15.150:3300/0 conn(0x7fc1000953c0 0x7fc100136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): get_auth_request con 0x7fc1000953c0 auth_method 0 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0fffff700 1 -- 172.21.15.142:0/2289856544 >> v1:172.21.15.142:6789/0 conn(0x7fc10009b050 legacy=0x7fc100097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0ff7fe700 1 -- 172.21.15.142:0/2289856544 >> v1:172.21.15.150:6789/0 conn(0x7fc100098080 legacy=0x7fc100133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0feffd700 1 -- 172.21.15.142:0/2289856544 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170805225 0 0) 0x7fc10009a570 con 0x7fc10009b050 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0feffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0feffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0feffd700 10 monclient(hunting): my global_id is 124419 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc0feffd700 1 -- 172.21.15.142:0/2289856544 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0e8003570 con 0x7fc10009b050 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient(hunting): handle_auth_done global_id 154126 payload 274 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 1 -- 172.21.15.142:0/2289856544 >> v1:172.21.15.150:6789/0 conn(0x7fc100098080 legacy=0x7fc100133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 1 -- 172.21.15.142:0/2289856544 >> v1:172.21.15.142:6789/0 conn(0x7fc10009b050 legacy=0x7fc100097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 1 monclient: found mon.noname-c 2024-09-18T10:22:21.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:22:21.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.808+0000 7fc104e11700 1 -- 172.21.15.142:0/2289856544 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10009b420 con 0x7fc1000953c0 2024-09-18T10:22:21.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc104e11700 1 -- 172.21.15.142:0/2289856544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1000953c0 msgr2=0x7fc100136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc104e11700 1 --2- 172.21.15.142:0/2289856544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1000953c0 0x7fc100136eb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009f60 tx=0x7fc0f8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82eedf3195452aa8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 1 -- 172.21.15.142:0/2289856544 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800e070 con 0x7fc1000953c0 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: dump: 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.821 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 1 -- 172.21.15.142:0/2289856544 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800be40 con 0x7fc1000953c0 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 1 -- 172.21.15.142:0/2289856544 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8012660 con 0x7fc1000953c0 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 5 monclient: authenticate success, global_id 154126 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 4 set_mon_vals no callback set 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc0feffd700 10 monclient: dump: 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.823 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 10 monclient: shutdown 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 -- 172.21.15.142:0/2289856544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1000953c0 msgr2=0x7fc100136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 --2- 172.21.15.142:0/2289856544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1000953c0 0x7fc100136eb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8009f60 tx=0x7fc0f8009780 comp rx=0 tx=0).stop 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 -- 172.21.15.142:0/2289856544 shutdown_connections 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 --2- 172.21.15.142:0/2289856544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1000953c0 0x7fc100136eb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 -- 172.21.15.142:0/2289856544 >> 172.21.15.142:0/2289856544 conn(0x7fc1000a1fe0 msgr2=0x7fc100070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.812+0000 7fc105612700 1 -- 172.21.15.142:0/2289856544 shutdown_connections 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 -- 172.21.15.142:0/2289856544 wait complete. 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient: build_initial_monmap 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient: monmap: 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:21.827 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 Processor -- start 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 -- start start 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient: init 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): picked mon.noname-a con 0x7fc1000953c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100098080 0x7fc10006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): picked mon.noname-b con 0x7fc100098080 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc10009b050 0x7fc10006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): picked mon.noname-c con 0x7fc10009b050 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:21.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc100127c60 con 0x7fc1000953c0 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc100127e60 con 0x7fc100098080 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc100128140 con 0x7fc10009b050 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc105612700 10 monclient(hunting): authenticate will time out at 1630.746985s 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc10009b050 0x7fc10006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41670/0 (socket says 172.21.15.142:41670) 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 -- 172.21.15.142:0/377728094 learned_addr learned my addr 172.21.15.142:0/377728094 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): get_auth_request con 0x7fc1000953c0 auth_method 0 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100098080 0x7fc10006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): get_auth_request con 0x7fc10009b050 auth_method 0 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): get_auth_request con 0x7fc100098080 auth_method 0 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc104e11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0ff7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient(hunting): handle_auth_done global_id 124422 payload 824 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 -- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc10009b050 msgr2=0x7fc10006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc10009b050 0x7fc10006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 -- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100098080 msgr2=0x7fc10006d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100098080 0x7fc10006d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 1 monclient: found mon.noname-a 2024-09-18T10:22:21.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:21.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.816+0000 7fc0fffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0fffff700 1 -- 172.21.15.142:0/377728094 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8009a20 con 0x7fc1000953c0 2024-09-18T10:22:21.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0fffff700 1 -- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 msgr2=0x7fc100128560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0fffff700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c740 tx=0x7fc0f0007c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0005d40 con 0x7fc1000953c0 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: dump: 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.834 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: _finish_auth 0 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00070d0 con 0x7fc1000953c0 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 5 monclient: authenticate success, global_id 124422 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 10 monclient: _renew_subs 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc100126520 con 0x7fc1000953c0 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0005ea0 con 0x7fc1000953c0 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:21.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0e7fff700 10 monclient: dump: 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 10 monclient: _renew_subs 2024-09-18T10:22:21.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc0fe7fc700 4 set_mon_vals no callback set 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.820+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc10006dd40 con 0x7fc1000953c0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc105612700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc105612700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc105612700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc0f001b3b0 con 0x7fc1000953c0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc100072a80 con 0x7fc1000953c0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc0e7fff700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 0x7fc0ec037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc0f0048310 con 0x7fc1000953c0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc0ff7fe700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 0x7fc0ec037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.824+0000 7fc0ff7fe700 10 monclient: get_auth_request con 0x7fc0ec035ac0 auth_method 0 2024-09-18T10:22:21.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.828+0000 7fc0ff7fe700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 0x7fc0ec037f70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4000c00 tx=0x7fc0f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:21.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.832+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc0f00196e0 con 0x7fc1000953c0 2024-09-18T10:22:21.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.832+0000 7fc0e7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:21.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:21.832+0000 7fc0e7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:22.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc105612700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:22:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc105612700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:22:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc105612700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc1000678a0 con 0x7fc1000953c0 2024-09-18T10:22:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc0e7fff700 1 -- 172.21.15.142:0/377728094 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc0f0017070 con 0x7fc1000953c0 2024-09-18T10:22:22.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc0e7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:22:22.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.052+0000 7fc0e7fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:22.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3792087174,config=3140514009,health=2496539594,kv=4261215878,logm=4139536204}) 2024-09-18T10:22:22.061 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=2381653535,config=3140514009,health=2496539594,kv=1758494360,logm=84930809}) 2024-09-18T10:22:22.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3792087174,config=3140514009,health=2496539594,kv=4261215878,logm=4139536204}) 2024-09-18T10:22:22.062 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.052+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=8,logm=54} crc {auth=2951596307,config=3140514010,health=2473239523,kv=4261215878,logm=4208752292}) 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 msgr2=0x7fc0ec037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 0x7fc0ec037f70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4000c00 tx=0x7fc0f4009040 comp rx=0 tx=0).stop 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 10 monclient: shutdown 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 msgr2=0x7fc100128560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000c740 tx=0x7fc0f0007c10 comp rx=0 tx=0).stop 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 shutdown_connections 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc0ec035ac0 0x7fc0ec037f70 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc10009b050 0x7fc10006d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:22.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc100098080 0x7fc10006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 --2- 172.21.15.142:0/377728094 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc1000953c0 0x7fc100128560 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 >> 172.21.15.142:0/377728094 conn(0x7fc1000a1fe0 msgr2=0x7fc10009f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 shutdown_connections 2024-09-18T10:22:22.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:22.056+0000 7fc105612700 1 -- 172.21.15.142:0/377728094 wait complete. 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272808386}) 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610327436}) 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3272808386}) 2024-09-18T10:22:22.066 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.056+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371102298}) 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572692951}) 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405106405}) 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572692951}) 2024-09-18T10:22:22.068 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.060+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587717638}) 2024-09-18T10:22:22.070 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700038939}) 2024-09-18T10:22:22.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827476615}) 2024-09-18T10:22:22.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700038939}) 2024-09-18T10:22:22.071 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232366137}) 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641366148}) 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222019768}) 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2641366148}) 2024-09-18T10:22:22.073 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.064+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156644089}) 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543376289}) 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766700196}) 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543376289}) 2024-09-18T10:22:22.074 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.068+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=934664196}) 2024-09-18T10:22:22.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936557739}) 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580622646}) 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936557739}) 2024-09-18T10:22:22.082 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=92767171}) 2024-09-18T10:22:22.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.083 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258704908}) 2024-09-18T10:22:22.084 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630442300}) 2024-09-18T10:22:22.084 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.084 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258704908}) 2024-09-18T10:22:22.084 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=119838346}) 2024-09-18T10:22:22.085 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.085 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120373817}) 2024-09-18T10:22:22.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543500449}) 2024-09-18T10:22:22.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120373817}) 2024-09-18T10:22:22.086 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.076+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562368636}) 2024-09-18T10:22:22.410 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851803093}) 2024-09-18T10:22:22.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599834745}) 2024-09-18T10:22:22.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851803093}) 2024-09-18T10:22:22.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2731794651}) 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215060185}) 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3470439074}) 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3215060185}) 2024-09-18T10:22:22.413 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.404+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1870654467}) 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554730730}) 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936300055}) 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554730730}) 2024-09-18T10:22:22.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.716+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195955362}) 2024-09-18T10:22:22.733 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466693091}) 2024-09-18T10:22:22.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3713331830}) 2024-09-18T10:22:22.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466693091}) 2024-09-18T10:22:22.734 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3701611726}) 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898057378}) 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496676734}) 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898057378}) 2024-09-18T10:22:22.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676283113}) 2024-09-18T10:22:22.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273301115}) 2024-09-18T10:22:22.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2181467877}) 2024-09-18T10:22:22.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273301115}) 2024-09-18T10:22:22.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421214900}) 2024-09-18T10:22:22.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069476807}) 2024-09-18T10:22:22.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520184067}) 2024-09-18T10:22:22.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069476807}) 2024-09-18T10:22:22.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=834312858}) 2024-09-18T10:22:22.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052984192}) 2024-09-18T10:22:22.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291187757}) 2024-09-18T10:22:22.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052984192}) 2024-09-18T10:22:22.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485395436}) 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838576513}) 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695452616}) 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838576513}) 2024-09-18T10:22:22.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970539802}) 2024-09-18T10:22:22.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837417449}) 2024-09-18T10:22:22.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892087330}) 2024-09-18T10:22:22.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837417449}) 2024-09-18T10:22:22.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220050056}) 2024-09-18T10:22:22.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627204127}) 2024-09-18T10:22:22.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292157216}) 2024-09-18T10:22:22.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627204127}) 2024-09-18T10:22:22.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1921569001}) 2024-09-18T10:22:22.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278747212}) 2024-09-18T10:22:22.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2782015854}) 2024-09-18T10:22:22.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278747212}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290716603}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714340554}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369304973}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714340554}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943108661}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849471756}) 2024-09-18T10:22:22.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4202940680}) 2024-09-18T10:22:22.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849471756}) 2024-09-18T10:22:22.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3975543889}) 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003077956}) 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=192018843}) 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003077956}) 2024-09-18T10:22:22.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.872+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743163696}) 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872163269}) 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351867434}) 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872163269}) 2024-09-18T10:22:22.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556367887}) 2024-09-18T10:22:22.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065851741}) 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176024241}) 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065851741}) 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022370029}) 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157537253}) 2024-09-18T10:22:22.885 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1618556346}) 2024-09-18T10:22:22.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157537253}) 2024-09-18T10:22:22.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.876+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284089699}) 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409509206}) 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440656955}) 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409509206}) 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964259576}) 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191785237}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129860840}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191785237}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923526703}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350169298}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239687367}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350169298}) 2024-09-18T10:22:22.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365536892}) 2024-09-18T10:22:22.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136797943}) 2024-09-18T10:22:22.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219601293}) 2024-09-18T10:22:22.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136797943}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.880+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127950388}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714777906}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923028738}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714777906}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527322991}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780286487}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388544300}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780286487}) 2024-09-18T10:22:22.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3634822787}) 2024-09-18T10:22:22.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411499004}) 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189646370}) 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411499004}) 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.884+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8261836}) 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409592544}) 2024-09-18T10:22:22.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871479572}) 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409592544}) 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899172666}) 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318632900}) 2024-09-18T10:22:22.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943759663}) 2024-09-18T10:22:22.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318632900}) 2024-09-18T10:22:22.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114189969}) 2024-09-18T10:22:22.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543029852}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1344666781}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543029852}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323744573}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859350799}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1737902562}) 2024-09-18T10:22:22.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859350799}) 2024-09-18T10:22:22.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.888+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485888560}) 2024-09-18T10:22:22.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601874569}) 2024-09-18T10:22:22.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271413251}) 2024-09-18T10:22:22.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:22.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601874569}) 2024-09-18T10:22:22.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.892+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845397879}) 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1317080581}) 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838148246}) 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1317080581}) 2024-09-18T10:22:23.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:22.992+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620342982}) 2024-09-18T10:22:23.077 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:22:23.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.184+0000 7f116a087700 10 monclient: get_monmap_and_config 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.184+0000 7f116a087700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.184+0000 7f116a087700 10 monclient: monmap: 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:23.191111+0000 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:23.191111+0000 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.193 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:23.194 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:23.194 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:23.194 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:23.194 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:23.194 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.184+0000 7f116a087700 1 Processor -- start 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 -- start start 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient: init 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f116409aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 0x7f1164134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): picked mon.noname-a con 0x7f1164132570 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 0x7f1164137270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): picked mon.noname-b con 0x7f1164134e90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116409aeb0 con 0x7f116409aae0 2024-09-18T10:22:23.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116406f6d0 con 0x7f1164132570 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11640955a0 con 0x7f1164134e90 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f116a087700 10 monclient(hunting): authenticate will time out at 1632.116574s 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 0x7f1164134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 0x7f1164137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 0x7f1164134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38370/0 (socket says 172.21.15.142:38370) 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 1 -- 172.21.15.142:0/2584280005 learned_addr learned my addr 172.21.15.142:0/2584280005 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): get_auth_request con 0x7f1164132570 auth_method 0 2024-09-18T10:22:23.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): get_auth_request con 0x7f1164134e90 auth_method 0 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient(hunting): handle_auth_done global_id 75323 payload 274 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 -- 172.21.15.142:0/2584280005 >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 msgr2=0x7f1164134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 --2- 172.21.15.142:0/2584280005 >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 0x7f1164134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 -- 172.21.15.142:0/2584280005 >> v1:172.21.15.150:6789/0 conn(0x7f116409aae0 legacy=0x7f1164096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 monclient: found mon.noname-b 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 -- 172.21.15.142:0/2584280005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1164095b60 con 0x7f1164134e90 2024-09-18T10:22:23.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 -- 172.21.15.142:0/2584280005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 msgr2=0x7f1164137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.188+0000 7f1169886700 1 --2- 172.21.15.142:0/2584280005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 0x7f1164137270 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f115c009e60 tx=0x7f115c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1be4d388289a377 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 1 -- 172.21.15.142:0/2584280005 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00e070 con 0x7f1164134e90 2024-09-18T10:22:23.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: dump: 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 5 monclient: authenticate success, global_id 75323 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 1 -- 172.21.15.142:0/2584280005 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c00bd70 con 0x7f1164134e90 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 4 set_mon_vals no callback set 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 1 -- 172.21.15.142:0/2584280005 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c012660 con 0x7f1164134e90 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f115bfff700 10 monclient: dump: 2024-09-18T10:22:23.207 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: shutdown 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- 172.21.15.142:0/2584280005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 msgr2=0x7f1164137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- 172.21.15.142:0/2584280005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 0x7f1164137270 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f115c009e60 tx=0x7f115c009780 comp rx=0 tx=0).stop 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- 172.21.15.142:0/2584280005 shutdown_connections 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- 172.21.15.142:0/2584280005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1164134e90 0x7f1164137270 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- 172.21.15.142:0/2584280005 >> v2:172.21.15.142:3300/0 conn(0x7f1164132570 0x7f1164134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- 172.21.15.142:0/2584280005 >> 172.21.15.142:0/2584280005 conn(0x7f11640a1fe0 msgr2=0x7f1164070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- 172.21.15.142:0/2584280005 shutdown_connections 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- 172.21.15.142:0/2584280005 wait complete. 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: monmap: 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:23.209 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 Processor -- start 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- start start 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: init 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:22:23.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f116409aae0 0x7f1164131ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): picked mon.noname-c con 0x7f116409aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): picked mon.noname-a con 0x7f1164132570 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1164134e90 0x7f116406de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): picked mon.noname-b con 0x7f1164134e90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1164134a90 con 0x7f1164132570 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1164128c00 con 0x7f1164134e90 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1164127f40 con 0x7f116409aae0 2024-09-18T10:22:23.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f116a087700 10 monclient(hunting): authenticate will time out at 1632.123776s 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1169085700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f116409aae0 0x7f1164131ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38386/0 (socket says 172.21.15.142:38386) 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 1 -- 172.21.15.142:0/2762899089 learned_addr learned my addr 172.21.15.142:0/2762899089 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 10 monclient(hunting): get_auth_request con 0x7f1164132570 auth_method 0 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.192+0000 7f1168884700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): get_auth_request con 0x7f116409aae0 auth_method 0 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169886700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1164134e90 0x7f116406de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169085700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169886700 10 monclient(hunting): get_auth_request con 0x7f1164134e90 auth_method 0 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169886700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169886700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1169886700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient(hunting): handle_auth_done global_id 124428 payload 824 2024-09-18T10:22:23.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 -- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f116409aae0 msgr2=0x7f1164131ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f116409aae0 0x7f1164131ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 -- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1164134e90 msgr2=0x7f116406de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1164134e90 0x7f116406de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 monclient: found mon.noname-a 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 -- 172.21.15.142:0/2762899089 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c009a20 con 0x7f1164132570 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 -- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 msgr2=0x7f116406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f1168884700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f115400ea20 tx=0x7f1154009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115400eeb0 con 0x7f1164132570 2024-09-18T10:22:23.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: dump: 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 5 monclient: authenticate success, global_id 124428 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115400ca60 con 0x7f1164132570 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115400ce20 con 0x7f1164132570 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 10 monclient: _renew_subs 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1164131460 con 0x7f1164132570 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f115b7fe700 4 set_mon_vals no callback set 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:23.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f11597fa700 10 monclient: dump: 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 10 monclient: _renew_subs 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:23.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.196+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11641264c0 con 0x7f1164132570 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f116a087700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f116a087700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f116a087700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1164072a80 con 0x7f1164132570 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1154013680 con 0x7f1164132570 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f11597fa700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 0x7f1148037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1154049280 con 0x7f1164132570 2024-09-18T10:22:23.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f1169085700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 0x7f1148037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.200+0000 7f1169085700 10 monclient: get_auth_request con 0x7f1148035b10 auth_method 0 2024-09-18T10:22:23.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.204+0000 7f1169085700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 0x7f1148037fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1160003a30 tx=0x7f1160007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.208+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f11540234d0 con 0x7f1164132570 2024-09-18T10:22:23.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.208+0000 7f11597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.208+0000 7f11597fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:23.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:22:23.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:23.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:22:23.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f11640678a0 con 0x7f1164132570 2024-09-18T10:22:23.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f11597fa700 1 -- 172.21.15.142:0/2762899089 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f11540232f0 con 0x7f1164132570 2024-09-18T10:22:23.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f11597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:23.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f11597fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:23.435 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:22:23.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 msgr2=0x7f1148037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 0x7f1148037fc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1160003a30 tx=0x7f1160007ea0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 10 monclient: shutdown 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 msgr2=0x7f116406d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f115400ea20 tx=0x7f1154009250 comp rx=0 tx=0).stop 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 shutdown_connections 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1148035b10 0x7f1148037fc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1164134e90 0x7f116406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1164132570 0x7f116406d900 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.428+0000 7f116a087700 1 --2- 172.21.15.142:0/2762899089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f116409aae0 0x7f1164131ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.432+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 >> 172.21.15.142:0/2762899089 conn(0x7f11640a1fe0 msgr2=0x7f11641369c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.432+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 shutdown_connections 2024-09-18T10:22:23.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.432+0000 7f116a087700 1 -- 172.21.15.142:0/2762899089 wait complete. 2024-09-18T10:22:23.449 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:22:23.449 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:22:23.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:22:23.450 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: get_monmap_and_config 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: monmap: 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:23.553088+0000 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:23.553088+0000 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.555 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:23.556 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:23.556 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:23.556 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:23.556 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:23.556 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 Processor -- start 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 -- start start 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: init 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:22:23.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efc1009a830 0x7efc10098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-a con 0x7efc1009a830 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 0x7efc10099110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-b con 0x7efc1009ac00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc1009bed0 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc10099d50 con 0x7efc1009bed0 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1009afd0 con 0x7efc1009a830 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1009dca0 con 0x7efc1009ac00 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc181e3700 10 monclient(hunting): authenticate will time out at 1632.478550s 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 0x7efc10099110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc16780700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7efc1009bed0 0x7efc1009c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:53176/0 (socket says 172.21.15.142:53176) 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc16780700 1 -- 172.21.15.142:0/485341514 learned_addr learned my addr 172.21.15.142:0/485341514 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 1 --2- 172.21.15.142:0/485341514 >> v2:172.21.15.142:3300/0 conn(0x7efc1009a830 0x7efc10098bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): get_auth_request con 0x7efc1009ac00 auth_method 0 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): get_auth_request con 0x7efc1009a830 auth_method 0 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.560 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient(hunting): handle_auth_done global_id 75329 payload 274 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 -- 172.21.15.142:0/485341514 >> v2:172.21.15.142:3300/0 conn(0x7efc1009a830 msgr2=0x7efc10098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 --2- 172.21.15.142:0/485341514 >> v2:172.21.15.142:3300/0 conn(0x7efc1009a830 0x7efc10098bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 -- 172.21.15.142:0/485341514 >> v1:172.21.15.142:6789/0 conn(0x7efc1009bed0 legacy=0x7efc1009c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 monclient: found mon.noname-b 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.548+0000 7efc1577e700 1 -- 172.21.15.142:0/485341514 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10099a70 con 0x7efc1009ac00 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc1577e700 1 -- 172.21.15.142:0/485341514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 msgr2=0x7efc10099110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc1577e700 1 --2- 172.21.15.142:0/485341514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 0x7efc10099110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e60 tx=0x7efc08009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cf0ebf266631400 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 1 -- 172.21.15.142:0/485341514 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800e070 con 0x7efc1009ac00 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: dump: 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.568 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 5 monclient: authenticate success, global_id 75329 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:23.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 1 -- 172.21.15.142:0/485341514 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0800bd70 con 0x7efc1009ac00 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 4 set_mon_vals no callback set 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 1 -- 172.21.15.142:0/485341514 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08012680 con 0x7efc1009ac00 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc14f7d700 10 monclient: dump: 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.570 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 10 monclient: shutdown 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- 172.21.15.142:0/485341514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 msgr2=0x7efc10099110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 --2- 172.21.15.142:0/485341514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 0x7efc10099110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e60 tx=0x7efc08009780 comp rx=0 tx=0).stop 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- 172.21.15.142:0/485341514 shutdown_connections 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 --2- 172.21.15.142:0/485341514 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009ac00 0x7efc10099110 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 --2- 172.21.15.142:0/485341514 >> v2:172.21.15.142:3300/0 conn(0x7efc1009a830 0x7efc10098bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- 172.21.15.142:0/485341514 >> 172.21.15.142:0/485341514 conn(0x7efc100a14e0 msgr2=0x7efc100a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- 172.21.15.142:0/485341514 shutdown_connections 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- 172.21.15.142:0/485341514 wait complete. 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 10 monclient: monmap: 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:23.572 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 Processor -- start 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 1 -- start start 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.552+0000 7efc181e3700 10 monclient: init 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-c con 0x7efc1009a830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1009bed0 0x7efc10138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-b con 0x7efc1009bed0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10146200 0x7efc1006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): picked mon.noname-a con 0x7efc10146200 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:23.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10146bf0 con 0x7efc10146200 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100947f0 con 0x7efc1009bed0 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1006ebf0 con 0x7efc1009a830 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc181e3700 10 monclient(hunting): authenticate will time out at 1632.485035s 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54984/0 (socket says 172.21.15.142:54984) 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 -- 172.21.15.142:0/1083495721 learned_addr learned my addr 172.21.15.142:0/1083495721 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): get_auth_request con 0x7efc1009a830 auth_method 0 2024-09-18T10:22:23.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10146200 0x7efc1006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): get_auth_request con 0x7efc10146200 auth_method 0 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc1577e700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1009bed0 0x7efc10138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc1577e700 10 monclient(hunting): get_auth_request con 0x7efc1009bed0 auth_method 0 2024-09-18T10:22:23.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc1577e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc1577e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc1577e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc16780700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient(hunting): handle_auth_done global_id 75332 payload 824 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1009bed0 msgr2=0x7efc10138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc1009bed0 0x7efc10138910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10146200 msgr2=0x7efc1006b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10146200 0x7efc1006b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 monclient: found mon.noname-c 2024-09-18T10:22:23.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc08009a20 con 0x7efc1009a830 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 msgr2=0x7efc10145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.556+0000 7efc15f7f700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7efc0000ec80 tx=0x7efc000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0000ca60 con 0x7efc1009a830 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: dump: 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.577 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 5 monclient: authenticate success, global_id 75332 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 monclient: _renew_subs 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0000cbc0 con 0x7efc1009a830 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1006e910 con 0x7efc1009a830 2024-09-18T10:22:23.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00013470 con 0x7efc1009a830 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 10 monclient: dump: 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.579 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc077fe700 4 set_mon_vals no callback set 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 monclient: _renew_subs 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1006dbc0 con 0x7efc1009a830 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 0x7efc1006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10064950 con 0x7efc10138e50 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efc000137e0 con 0x7efc1009a830 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 0x7efc1006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient: get_auth_request con 0x7efc10138e50 auth_method 0 2024-09-18T10:22:23.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efbf4035ac0 0x7efbf4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7efc00049380 con 0x7efc1009a830 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 10 monclient(hunting): handle_auth_done global_id 124437 payload 824 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc1577e700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc100696a0 con 0x7efc10138e50 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc16780700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efbf4035ac0 0x7efbf4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.560+0000 7efc16780700 10 monclient: get_auth_request con 0x7efbf4035ac0 auth_method 0 2024-09-18T10:22:23.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc1577e700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 msgr2=0x7efc1006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc1577e700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 0x7efc1006b8c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e60 tx=0x7efc08018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800bee0 con 0x7efc10138e50 2024-09-18T10:22:23.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 10 monclient: dump: 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efc08012e90 con 0x7efc10138e50 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 msgr2=0x7efc1006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 0x7efc1006b8c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efc08009e60 tx=0x7efc08018000 comp rx=0 tx=0).stop 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 msgr2=0x7efc1006b8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc05ffb700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc10138e50 0x7efc1006b8c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.564+0000 7efc16780700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efbf4035ac0 0x7efbf4037f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efc0c007860 tx=0x7efc0c007f70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc181e3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc181e3700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 0x7efc10091800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc181e3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10139220 con 0x7efc1006be00 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 0x7efc10091800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient: get_auth_request con 0x7efc1006be00 auth_method 0 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.572+0000 7efc1577e700 10 monclient(hunting): handle_auth_done global_id 124440 payload 824 2024-09-18T10:22:23.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc1577e700 1 -- 172.21.15.142:0/1083495721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc1006c200 con 0x7efc1006be00 2024-09-18T10:22:23.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc1577e700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 msgr2=0x7efc10091800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc1577e700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 0x7efc10091800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc08020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08027be0 con 0x7efc1006be00 2024-09-18T10:22:23.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 10 monclient: dump: 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7efc08012c80 con 0x7efc1006be00 2024-09-18T10:22:23.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 msgr2=0x7efc10091800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 0x7efc10091800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc08020040 comp rx=0 tx=0).stop 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 msgr2=0x7efc10091800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc05ffb700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc1006be00 0x7efc10091800 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc08020040 comp rx=0 tx=0).stop 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:22:23.589 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.590 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.591 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:23.592 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.593 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:23.594 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.596 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.597 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.600 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.601 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:23.602 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efbf4035ac0 msgr2=0x7efbf4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efbf4035ac0 0x7efbf4037f70 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efc0c007860 tx=0x7efc0c007f70 comp rx=0 tx=0).stop 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 10 monclient: shutdown 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 msgr2=0x7efc10145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7efc0000ec80 tx=0x7efc000091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc16780700 1 -- 172.21.15.142:0/1083495721 reap_dead start 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 shutdown_connections 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 --2- 172.21.15.142:0/1083495721 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efc1009a830 0x7efc10145cc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.576+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 >> 172.21.15.142:0/1083495721 conn(0x7efc100a14e0 msgr2=0x7efc10139910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.580+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 shutdown_connections 2024-09-18T10:22:23.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.580+0000 7efc181e3700 1 -- 172.21.15.142:0/1083495721 wait complete. 2024-09-18T10:22:23.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:23.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:22:23.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151900109}) 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679902741}) 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151900109}) 2024-09-18T10:22:23.624 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.616+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574092350}) 2024-09-18T10:22:23.626 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.626 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502988359}) 2024-09-18T10:22:23.626 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297267755}) 2024-09-18T10:22:23.626 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.627 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502988359}) 2024-09-18T10:22:23.627 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.620+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686831013}) 2024-09-18T10:22:23.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: get_monmap_and_config 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: monmap: 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:23.718323+0000 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:23.718323+0000 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:23.721 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:23.722 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 Processor -- start 2024-09-18T10:22:23.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 -- start start 2024-09-18T10:22:23.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: init 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 0x7f83b80960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-a con 0x7f83b809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-b con 0x7f83b809b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 0x7f83b809c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-c con 0x7f83b809c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b809b500 con 0x7f83b809abd0 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b809dc90 con 0x7f83b809c410 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b809afa0 con 0x7f83b809b740 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.712+0000 7f83bd92a700 10 monclient(hunting): authenticate will time out at 1632.643877s 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 0x7f83b80960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 0x7f83b80960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38412/0 (socket says 172.21.15.142:38412) 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54990/0 (socket says 172.21.15.142:54990) 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 1 -- 172.21.15.142:0/3052637338 learned_addr learned my addr 172.21.15.142:0/3052637338 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): get_auth_request con 0x7f83b809abd0 auth_method 0 2024-09-18T10:22:23.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): get_auth_request con 0x7f83b809b740 auth_method 0 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 0x7f83b809c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 10 monclient(hunting): get_auth_request con 0x7f83b809c410 auth_method 0 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient(hunting): handle_auth_done global_id 75335 payload 274 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 -- 172.21.15.142:0/3052637338 >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 msgr2=0x7f83b809c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 0x7f83b809c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 -- 172.21.15.142:0/3052637338 >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 msgr2=0x7f83b80960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 0x7f83b80960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 monclient: found mon.noname-b 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 -- 172.21.15.142:0/3052637338 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b806f810 con 0x7f83b809b740 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd129700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 0x7f83b809c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 -- 172.21.15.142:0/3052637338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 msgr2=0x7f83b809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3052637338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f83b806fd20 tx=0x7f83a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2edba035eea95f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 1 -- 172.21.15.142:0/3052637338 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011840 con 0x7f83b809b740 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: dump: 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.730 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd92a700 5 monclient: authenticate success, global_id 75335 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 1 -- 172.21.15.142:0/3052637338 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a80119a0 con 0x7f83b809b740 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd92a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd92a700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83bd92a700 4 set_mon_vals no callback set 2024-09-18T10:22:23.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 1 -- 172.21.15.142:0/3052637338 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8011d40 con 0x7f83b809b740 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.716+0000 7f83b77fe700 10 monclient: dump: 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.732 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: shutdown 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- 172.21.15.142:0/3052637338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 msgr2=0x7f83b809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3052637338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f83b806fd20 tx=0x7f83a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- 172.21.15.142:0/3052637338 shutdown_connections 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.150:3300/0 conn(0x7f83b809c410 0x7f83b809c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3052637338 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809b740 0x7f83b809bed0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3052637338 >> v2:172.21.15.142:3300/0 conn(0x7f83b809abd0 0x7f83b80960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- 172.21.15.142:0/3052637338 >> 172.21.15.142:0/3052637338 conn(0x7f83b8070ef0 msgr2=0x7f83b80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- 172.21.15.142:0/3052637338 shutdown_connections 2024-09-18T10:22:23.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- 172.21.15.142:0/3052637338 wait complete. 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: monmap: 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:23.734 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:23.735 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022446037}) 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1799699061}) 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022446037}) 2024-09-18T10:22:23.736 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464023882}) 2024-09-18T10:22:23.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 Processor -- start 2024-09-18T10:22:23.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- start start 2024-09-18T10:22:23.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: init 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-c con 0x7f83b809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b809b740 0x7f83b8094000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-b con 0x7f83b809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 0x7f83b808bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): picked mon.noname-a con 0x7f83b809c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8136540 con 0x7f83b809c410 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8135df0 con 0x7f83b809b740 2024-09-18T10:22:23.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b808de60 con 0x7f83b809abd0 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd92a700 10 monclient(hunting): authenticate will time out at 1632.651529s 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 0x7f83b808bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55006/0 (socket says 172.21.15.142:55006) 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 0x7f83b808bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38422/0 (socket says 172.21.15.142:38422) 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 1 -- 172.21.15.142:0/3748656139 learned_addr learned my addr 172.21.15.142:0/3748656139 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): get_auth_request con 0x7f83b809abd0 auth_method 0 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): get_auth_request con 0x7f83b809c410 auth_method 0 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b809b740 0x7f83b8094000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bc928700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.720+0000 7f83bd129700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bd129700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83b7fff700 10 monclient(hunting): get_auth_request con 0x7f83b809b740 auth_method 0 2024-09-18T10:22:23.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 10 monclient(hunting): handle_auth_done global_id 75338 payload 824 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b809b740 msgr2=0x7f83b8094000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b809b740 0x7f83b8094000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 msgr2=0x7f83b808bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 0x7f83b808bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 monclient: found mon.noname-c 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8009420 con 0x7f83b809abd0 2024-09-18T10:22:23.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b809b740 0x7f83b8094000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:22:23.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bd129700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83b809c410 0x7f83b808bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:22:23.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 msgr2=0x7f83b81358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.724+0000 7f83bc928700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f83a000ea80 tx=0x7f83a0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.744 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256489626}) 2024-09-18T10:22:23.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853219275}) 2024-09-18T10:22:23.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256489626}) 2024-09-18T10:22:23.745 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585329379}) 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0009640 con 0x7f83b809abd0 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: dump: 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.748 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a000ca60 con 0x7f83b809abd0 2024-09-18T10:22:23.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 5 monclient: authenticate success, global_id 75338 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a000cde0 con 0x7f83b809abd0 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 10 monclient: _renew_subs 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b806eb80 con 0x7f83b809abd0 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b6ffd700 4 set_mon_vals no callback set 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83b4ff9700 10 monclient: dump: 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.750 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 10 monclient: _renew_subs 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.728+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b808d6d0 con 0x7f83b809abd0 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bc928700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f83a000ea80 tx=0x7f83a0009250 comp rx=0 tx=0).queue_received will delay after 1.72665e+09 on 0x7f83a0013680 mgrmap(e 5) v1 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd92a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 0x7f83b808c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f83a0013680 con 0x7f83b809abd0 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80648a0 con 0x7f83b8094540 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 0x7f83b808c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient: get_auth_request con 0x7f83b8094540 auth_method 0 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b4ff9700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8398035b10 0x7f8398037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 10 monclient(hunting): handle_auth_done global_id 75341 payload 824 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd129700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8398035b10 0x7f8398037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83b808e300 con 0x7f83b8094540 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83bd129700 10 monclient: get_auth_request con 0x7f8398035b10 auth_method 0 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 msgr2=0x7f83b808c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.732+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 0x7f83b808c120 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f83b806fd20 tx=0x7f83a8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8020070 con 0x7f83b8094540 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 10 monclient: dump: 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.753 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83bd129700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8398035b10 0x7f8398037fc0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f83b808f590 tx=0x7f83b0009990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f83a800bca0 con 0x7f83b8094540 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 msgr2=0x7f83b808c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 0x7f83b808c120 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f83b806fd20 tx=0x7f83a8017000 comp rx=0 tx=0).stop 2024-09-18T10:22:23.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 msgr2=0x7f83b808c120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.736+0000 7f83b4ff9700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b8094540 0x7f83b808c120 secure :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f83b806fd20 tx=0x7f83a8017000 comp rx=0 tx=0).stop 2024-09-18T10:22:23.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183752839}) 2024-09-18T10:22:23.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4116683820}) 2024-09-18T10:22:23.756 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.756 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183752839}) 2024-09-18T10:22:23.756 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.740+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329536083}) 2024-09-18T10:22:23.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83bd92a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 0x7f83b8087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83bd92a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8094910 con 0x7f83b808c660 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 0x7f83b8087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient: get_auth_request con 0x7f83b808c660 auth_method 0 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 10 monclient(hunting): handle_auth_done global_id 75344 payload 824 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.744+0000 7f83b7fff700 1 -- 172.21.15.142:0/3748656139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f83b808ca30 con 0x7f83b808c660 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b7fff700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 msgr2=0x7f83b8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b7fff700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 0x7f83b8087fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83a80093a0 tx=0x7f83a801d070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8031a30 con 0x7f83b808c660 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 10 monclient: dump: 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f83a8010380 con 0x7f83b808c660 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 msgr2=0x7f83b8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 0x7f83b8087fe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83a80093a0 tx=0x7f83a801d070 comp rx=0 tx=0).stop 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 msgr2=0x7f83b8087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b808c660 0x7f83b8087fe0 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f83a80093a0 tx=0x7f83a801d070 comp rx=0 tx=0).stop 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:22:23.761 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:23.763 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:23.764 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:23.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:23.767 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.768 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.771 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.772 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:23.773 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8398035b10 msgr2=0x7f8398037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8398035b10 0x7f8398037fc0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f83b808f590 tx=0x7f83b0009990 comp rx=0 tx=0).stop 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 10 monclient: shutdown 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 msgr2=0x7f83b81358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f83a000ea80 tx=0x7f83a0009250 comp rx=0 tx=0).stop 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd129700 1 -- 172.21.15.142:0/3748656139 reap_dead start 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83b4ff9700 1 -- 172.21.15.142:0/3748656139 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f83a0049190 con 0x7f83b809abd0 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 shutdown_connections 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 --2- 172.21.15.142:0/3748656139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f83b809abd0 0x7f83b81358b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 >> 172.21.15.142:0/3748656139 conn(0x7f83b8070ef0 msgr2=0x7f83b80a04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 shutdown_connections 2024-09-18T10:22:23.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.748+0000 7f83bd92a700 1 -- 172.21.15.142:0/3748656139 wait complete. 2024-09-18T10:22:23.776 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:23.776 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:22:23.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:22:23.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635021622}) 2024-09-18T10:22:23.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4196147822}) 2024-09-18T10:22:23.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635021622}) 2024-09-18T10:22:23.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.856+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717441899}) 2024-09-18T10:22:23.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724706561}) 2024-09-18T10:22:23.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1746337788}) 2024-09-18T10:22:23.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724706561}) 2024-09-18T10:22:23.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600569170}) 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225398050}) 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1380537445}) 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225398050}) 2024-09-18T10:22:23.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.860+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698452603}) 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050423185}) 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901358800}) 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050423185}) 2024-09-18T10:22:23.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.864+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428055090}) 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499274148}) 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1836229774}) 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499274148}) 2024-09-18T10:22:23.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:22:23.868+0000 7f3497567700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=85736847}) 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.876+0000 7f0ef38a3700 10 monclient: get_monmap_and_config 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.876+0000 7f0ef38a3700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.876+0000 7f0ef38a3700 10 monclient: monmap: 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.884 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:23.882611+0000 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:23.882611+0000 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:23.885 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 Processor -- start 2024-09-18T10:22:23.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 -- start start 2024-09-18T10:22:23.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient: init 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0eec06f1b0 0x7f0eec06f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-c con 0x7f0eec06f1b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0eec09c990 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec09d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-b con 0x7f0eec09d4a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec098b30 con 0x7f0eec09c990 2024-09-18T10:22:23.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec06f060 con 0x7f0eec06f1b0 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec098150 con 0x7f0eec09d4a0 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef38a3700 10 monclient(hunting): authenticate will time out at 1632.809395s 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef0e3e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0eec09c990 0x7f0eec09cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:53190/0 (socket says 172.21.15.142:53190) 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef0e3e700 1 -- 172.21.15.142:0/2134671139 learned_addr learned my addr 172.21.15.142:0/2134671139 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec09d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2134671139 >> v2:172.21.15.150:3300/0 conn(0x7f0eec06f1b0 0x7f0eec06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): get_auth_request con 0x7f0eec09d4a0 auth_method 0 2024-09-18T10:22:23.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): get_auth_request con 0x7f0eec06f1b0 auth_method 0 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_done global_id 75347 payload 274 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2134671139 >> v2:172.21.15.150:3300/0 conn(0x7f0eec06f1b0 msgr2=0x7f0eec06f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2134671139 >> v2:172.21.15.150:3300/0 conn(0x7f0eec06f1b0 0x7f0eec06f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2134671139 >> v1:172.21.15.142:6789/0 conn(0x7f0eec09c990 legacy=0x7f0eec09cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 monclient: found mon.noname-b 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.880+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2134671139 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec098f00 con 0x7f0eec09d4a0 2024-09-18T10:22:23.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 msgr2=0x7f0eec09d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec09d900 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009e60 tx=0x7f0ee4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d598b7af642752d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 1 -- 172.21.15.142:0/2134671139 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee400e070 con 0x7f0eec09d4a0 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: dump: 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.899 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 5 monclient: authenticate success, global_id 75347 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 1 -- 172.21.15.142:0/2134671139 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee400bd70 con 0x7f0eec09d4a0 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 4 set_mon_vals no callback set 2024-09-18T10:22:23.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 1 -- 172.21.15.142:0/2134671139 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4012660 con 0x7f0eec09d4a0 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ee3fff700 10 monclient: dump: 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 10 monclient: shutdown 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 msgr2=0x7f0eec09d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec09d900 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009e60 tx=0x7f0ee4009780 comp rx=0 tx=0).stop 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2134671139 shutdown_connections 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2134671139 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec09d900 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2134671139 >> v2:172.21.15.150:3300/0 conn(0x7f0eec06f1b0 0x7f0eec06f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2134671139 >> 172.21.15.142:0/2134671139 conn(0x7f0eec0a1fa0 msgr2=0x7f0eec0a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2134671139 shutdown_connections 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2134671139 wait complete. 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 10 monclient: build_initial_monmap 2024-09-18T10:22:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 10 monclient: monmap: 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.884+0000 7f0ef38a3700 1 Processor -- start 2024-09-18T10:22:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 -- start start 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient: init 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec06f1b0 0x7f0eec13c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-b con 0x7f0eec06f1b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 0x7f0eec13bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-a con 0x7f0eec09c990 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec13fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): picked mon.noname-c con 0x7f0eec09d4a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1213e0 con 0x7f0eec09c990 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec121000 con 0x7f0eec06f1b0 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec11f2f0 con 0x7f0eec09d4a0 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef38a3700 10 monclient(hunting): authenticate will time out at 1632.817285s 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 0x7f0eec13bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec13fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec13fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:55038/0 (socket says 172.21.15.142:55038) 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 0x7f0eec13bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38426/0 (socket says 172.21.15.142:38426) 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 learned_addr learned my addr 172.21.15.142:0/2547042421 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): get_auth_request con 0x7f0eec09c990 auth_method 0 2024-09-18T10:22:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec06f1b0 0x7f0eec13c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): get_auth_request con 0x7f0eec09d4a0 auth_method 0 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): get_auth_request con 0x7f0eec06f1b0 auth_method 0 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient(hunting): handle_auth_done global_id 75350 payload 824 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec06f1b0 msgr2=0x7f0eec13c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec06f1b0 0x7f0eec13c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 msgr2=0x7f0eec13bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 0x7f0eec13bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient: _finish_hunting 0 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 monclient: found mon.noname-c 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4009a20 con 0x7f0eec09d4a0 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.888+0000 7f0ef0e3e700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0eec09c990 0x7f0eec13bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 msgr2=0x7f0eec13fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef1e40700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec13fb40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b6c0 tx=0x7f0ee4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40070d0 con 0x7f0eec09d4a0 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: dump: 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: _finish_auth 0 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4021070 con 0x7f0eec09d4a0 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 5 monclient: authenticate success, global_id 75350 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 monclient: _renew_subs 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec120d20 con 0x7f0eec09d4a0 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4026430 con 0x7f0eec09d4a0 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 10 monclient: dump: 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee37fe700 4 set_mon_vals no callback set 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 monclient: _renew_subs 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:23.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec13cfe0 con 0x7f0eec09d4a0 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 0x7f0eec029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec064900 con 0x7f0eec11fe50 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0ee4023030 con 0x7f0eec09d4a0 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 0x7f0eec029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient: get_auth_request con 0x7f0eec11fe50 auth_method 0 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ed0035ac0 0x7f0ed0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0ee404b1a0 con 0x7f0eec09d4a0 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 10 monclient(hunting): handle_auth_done global_id 154153 payload 824 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef0e3e700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ed0035ac0 0x7f0ed0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef163f700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eec1211c0 con 0x7f0eec11fe50 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.892+0000 7f0ef0e3e700 10 monclient: get_auth_request con 0x7f0ed0035ac0 auth_method 0 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ef163f700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 msgr2=0x7f0eec029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 0x7f0eec029d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011c20 tx=0x7f0ee80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800fe40 con 0x7f0eec11fe50 2024-09-18T10:22:23.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 10 monclient: dump: 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ef0e3e700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ed0035ac0 0x7f0ed0037f70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0eec098d70 tx=0x7f0edc007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0ee8014440 con 0x7f0eec11fe50 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 msgr2=0x7f0eec029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 0x7f0eec029d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8011c20 tx=0x7f0ee80091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 msgr2=0x7f0eec029d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.896+0000 7f0ee17fa700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec11fe50 0x7f0eec029d00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef38a3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 0x7f0eec02a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef38a3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec11ec30 con 0x7f0eec02a240 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 0x7f0eec02a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient: get_auth_request con 0x7f0eec02a240 auth_method 0 2024-09-18T10:22:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 10 monclient(hunting): handle_auth_done global_id 154156 payload 824 2024-09-18T10:22:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.908+0000 7f0ef163f700 1 -- 172.21.15.142:0/2547042421 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0eec13d1d0 con 0x7f0eec02a240 2024-09-18T10:22:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef163f700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 msgr2=0x7f0eec02a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef163f700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 0x7f0eec02a640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009940 tx=0x7f0ee801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee802b070 con 0x7f0eec02a240 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 10 monclient: dump: 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0ee8023040 con 0x7f0eec02a240 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 msgr2=0x7f0eec02a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 0x7f0eec02a640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009940 tx=0x7f0ee801c000 comp rx=0 tx=0).stop 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 msgr2=0x7f0eec02a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ee17fa700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0eec02a240 0x7f0eec02a640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 128, 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.923 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.924 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.925 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:23.926 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:23.927 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.928 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.929 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:23.930 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.931 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.932 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.933 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:23.934 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:23.935 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:23.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ed0035ac0 msgr2=0x7f0ed0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0ed0035ac0 0x7f0ed0037f70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0eec098d70 tx=0x7f0edc007f90 comp rx=0 tx=0).stop 2024-09-18T10:22:23.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 10 monclient: shutdown 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 msgr2=0x7f0eec13fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 --2- 172.21.15.142:0/2547042421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0eec09d4a0 0x7f0eec13fb40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b6c0 tx=0x7f0ee4018000 comp rx=0 tx=0).stop 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 reap_dead start 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef1e40700 1 -- 172.21.15.142:0/2547042421 reap_dead start 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 shutdown_connections 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 >> 172.21.15.142:0/2547042421 conn(0x7f0eec0a1fa0 msgr2=0x7f0eec099d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 shutdown_connections 2024-09-18T10:22:23.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:23.912+0000 7f0ef38a3700 1 -- 172.21.15.142:0/2547042421 wait complete. 2024-09-18T10:22:23.937 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:22:23.938 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:22:23.938 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-18T10:22:23.938 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:22:23.938 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:22:23.938 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:22:23.938 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:22:23.956 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:22:23.956 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T10:22:23.957 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T10:22:23.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:22:43.958 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T10:22:43.959 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T10:22:43.959 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:22:43.960 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: get_monmap_and_config 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: build_initial_monmap 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: monmap: 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:44.065908+0000 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:44.065908+0000 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:44.068 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:44.069 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:44.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 Processor -- start 2024-09-18T10:22:44.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 -- start start 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: init 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): picked mon.noname-b con 0x7ff03809d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): picked mon.noname-c con 0x7ff0380989c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0380992d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff038069c70 con 0x7ff0380992d0 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:44.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03806f9d0 con 0x7ff0380989c0 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03809dc40 con 0x7ff03809d870 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.059+0000 7ff040606700 10 monclient(hunting): authenticate will time out at 1652.991886s 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49048/0 (socket says 172.21.15.142:49048) 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 learned_addr learned my addr 172.21.15.142:0/3916725686 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff03809d870 auth_method 0 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff0380989c0 auth_method 0 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:44.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.063+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 legacy=0x7ff038135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:44.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff03809d870 auth_method 0 2024-09-18T10:22:44.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff0380989c0 auth_method 0 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.267+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:22:44.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff03809d870 auth_method 0 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff0380989c0 auth_method 0 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:22:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:22:44.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:44.671+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff03809d870 auth_method 0 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff0380989c0 auth_method 0 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:45.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:22:45.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.475+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:22:45.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.715+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 legacy=0x7ff038135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-18T10:22:45.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.715+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 legacy=0x7ff038135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:22:45.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.715+0000 7ff03eba3700 1 --1- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 0x7ff038135670 :-1 s=OPENED pgs=39 cs=1 l=1).handle_message read tag failed 2024-09-18T10:22:45.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.715+0000 7ff03eba3700 1 --1- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 0x7ff038135670 :-1 s=OPENED pgs=39 cs=1 l=1).fault on lossy channel, failing 2024-09-18T10:22:45.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:45.715+0000 7ff03d3a0700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.150:6789/0 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.059+0000 7ff02ffff700 10 monclient(hunting): tick 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.059+0000 7ff02ffff700 1 monclient(hunting): continuing hunt 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.059+0000 7ff02ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.059+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 msgr2=0x7ff0380954b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.059+0000 7ff02ffff700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff03809d870 0x7ff0380954b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 msgr2=0x7ff038098d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.150:3300/0 conn(0x7ff0380989c0 0x7ff038098d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff0380992d0 legacy=0x7ff038135670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 10 monclient: _add_conns ranks=[0,3] 2024-09-18T10:22:47.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff01c003670 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 10 monclient(hunting): picked mon.noname-a con 0x7ff01c007390 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03806fc20 con 0x7ff01c003670 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03809dc40 con 0x7ff01c007390 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff02ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c003670 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:47.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.063+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:47.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.267+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:47.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.267+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c003670 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:47.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.267+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:22:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.671+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:47.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.671+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c003670 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:47.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:47.671+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:22:48.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:48.475+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:48.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:48.475+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c003670 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:48.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:48.475+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:22:50.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): tick 2024-09-18T10:22:50.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 monclient(hunting): continuing hunt 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c007390 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c003670 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 reap_dead start 2024-09-18T10:22:50.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c009f10 0x7ff01c009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): picked mon.noname-a con 0x7ff01c009f10 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff01c00a380 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff01c00a860 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01c00af70 con 0x7ff01c00a860 2024-09-18T10:22:50.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03806fc20 con 0x7ff01c00a380 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c009f10 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 1 -- 172.21.15.142:0/3916725686 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03809dc40 con 0x7ff01c009f10 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff02ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c00a860 legacy=0x7ff01c00d330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:50.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c009f10 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff01c00a380 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880865920 0 0) 0x7ff03806fc20 con 0x7ff01c00a380 2024-09-18T10:22:50.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): my global_id is 154159 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024003570 con 0x7ff01c00a380 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619168877 0 0) 0x7ff024003570 con 0x7ff01c00a380 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c009f10 msgr2=0x7ff01c009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 --2- 172.21.15.142:0/3916725686 >> v2:172.21.15.142:3300/0 conn(0x7ff01c009f10 0x7ff01c009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.142:6789/0 conn(0x7ff01c00a860 legacy=0x7ff01c00d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c010b10 con 0x7ff01c00a380 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff040606700 5 monclient: authenticate success, global_id 154159 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff040606700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff02800f5b0 con 0x7ff01c00a380 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:22:50.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: dump: 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:22:50.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 >> v1:172.21.15.150:6789/0 conn(0x7ff01c00a380 legacy=0x7ff01c005ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff024003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): picked mon.c con 0x7ff024003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff024004810 0x7ff024004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): picked mon.b con 0x7ff024004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0240052c0 0x7ff02400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): picked mon.a con 0x7ff0240052c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03809dc40 con 0x7ff0240052c0 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02400c280 con 0x7ff024004810 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02400c440 con 0x7ff024003a60 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0240052c0 msgr2=0x7ff02400ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03d3a0700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff024003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03eba3700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0240052c0 0x7ff02400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff040606700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff024003a60 auth_method 0 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff024004810 0x7ff024004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff024004810 auth_method 0 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_done global_id 154159 payload 293 2024-09-18T10:22:50.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff024004810 msgr2=0x7ff024004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff024004810 0x7ff024004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0240052c0 msgr2=0x7ff02400ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0240052c0 0x7ff02400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 monclient: found mon.c 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03eba3700 1 -- 172.21.15.142:0/3916725686 reap_dead start 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024003570 con 0x7ff024003a60 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.067+0000 7ff03e3a2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03e3a2700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 msgr2=0x7ff024003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff024003e50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff034009650 tx=0x7ff034009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d05bcb573414ca server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340097a0 con 0x7ff024003a60 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: dump: 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.097 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03400f040 con 0x7ff024003a60 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 1 -- 172.21.15.142:0/3916725686 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034017470 con 0x7ff024003a60 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 4 set_mon_vals no callback set 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff03d3a0700 10 monclient: dump: 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.098 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 10 monclient: shutdown 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 -- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 msgr2=0x7ff024003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff024003e50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff034009650 tx=0x7ff034009040 comp rx=0 tx=0).stop 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 -- 172.21.15.142:0/3916725686 shutdown_connections 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 --2- 172.21.15.142:0/3916725686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff024003e50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 -- 172.21.15.142:0/3916725686 >> 172.21.15.142:0/3916725686 conn(0x7ff0380a14e0 msgr2=0x7ff0380a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 -- 172.21.15.142:0/3916725686 shutdown_connections 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.071+0000 7ff040606700 1 -- 172.21.15.142:0/3916725686 wait complete. 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient: build_initial_monmap 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient: monmap: 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:50.100 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 Processor -- start 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 -- start start 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient: init 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): picked mon.noname-c con 0x7ff024003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:50.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff038141e40 0x7ff038142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): picked mon.noname-b con 0x7ff038141e40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 0x7ff03808e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): picked mon.noname-a con 0x7ff03808dca0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038097ee0 con 0x7ff03808dca0 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038092510 con 0x7ff038141e40 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03808fc60 con 0x7ff024003a60 2024-09-18T10:22:50.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff040606700 10 monclient(hunting): authenticate will time out at 1659.007135s 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03eba3700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 msgr2=0x7ff03808e110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03eba3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 0x7ff03808e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49126/0 (socket says 172.21.15.142:49126) 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 1 -- 172.21.15.142:0/1507818487 learned_addr learned my addr 172.21.15.142:0/1507818487 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request con 0x7ff024003a60 auth_method 0 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff038141e40 0x7ff038142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): get_auth_request con 0x7ff038141e40 auth_method 0 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.075+0000 7ff03dba1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 10 monclient(hunting): handle_auth_done global_id 75353 payload 824 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 -- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff038141e40 msgr2=0x7ff038142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff038141e40 0x7ff038142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 -- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 msgr2=0x7ff03808e110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 0x7ff03808e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 monclient: found mon.noname-c 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 -- 172.21.15.142:0/1507818487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff038091560 con 0x7ff024003a60 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 -- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 msgr2=0x7ff038141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03e3a2700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c5d0 tx=0x7ff03401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034017e10 con 0x7ff024003a60 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: dump: 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.107 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034023630 con 0x7ff024003a60 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 5 monclient: authenticate success, global_id 75353 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 10 monclient: _renew_subs 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0380a0b50 con 0x7ff024003a60 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340239f0 con 0x7ff024003a60 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff02e7fc700 10 monclient: dump: 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 10 monclient: _renew_subs 2024-09-18T10:22:50.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff03cb9f700 4 set_mon_vals no callback set 2024-09-18T10:22:50.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.079+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff038091280 con 0x7ff024003a60 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff040606700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff034003050 con 0x7ff024003a60 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff040606700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff040606700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0380696a0 con 0x7ff024003a60 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff02e7fc700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 0x7ff01c039060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff03404b610 con 0x7ff024003a60 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff03dba1700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 0x7ff01c039060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.083+0000 7ff03dba1700 10 monclient: get_auth_request con 0x7ff01c036bb0 auth_method 0 2024-09-18T10:22:50.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.087+0000 7ff03dba1700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 0x7ff01c039060 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff028010010 tx=0x7ff028008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.091+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff034033b40 con 0x7ff024003a60 2024-09-18T10:22:50.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.091+0000 7ff02e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.091+0000 7ff02e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:50.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff040606700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:22:50.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff040606700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:50.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff040606700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff03806fb20 con 0x7ff024003a60 2024-09-18T10:22:50.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff02e7fc700 1 -- 172.21.15.142:0/1507818487 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7ff034033b40 con 0x7ff024003a60 2024-09-18T10:22:50.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff02e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:22:50.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.315+0000 7ff02e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:50.326 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":130,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 msgr2=0x7ff01c039060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 0x7ff01c039060 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff028010010 tx=0x7ff028008040 comp rx=0 tx=0).stop 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 10 monclient: shutdown 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 msgr2=0x7ff038141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff02400c5d0 tx=0x7ff03401c000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 shutdown_connections 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff01c036bb0 0x7ff01c039060 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff03808dca0 0x7ff03808e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff038141e40 0x7ff038142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 --2- 172.21.15.142:0/1507818487 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff024003a60 0x7ff038141900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 >> 172.21.15.142:0/1507818487 conn(0x7ff0380a14e0 msgr2=0x7ff038087fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 shutdown_connections 2024-09-18T10:22:50.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.319+0000 7ff040606700 1 -- 172.21.15.142:0/1507818487 wait complete. 2024-09-18T10:22:50.339 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:22:50.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:22:50.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:22:50.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 10 monclient: get_monmap_and_config 2024-09-18T10:22:50.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 10 monclient: build_initial_monmap 2024-09-18T10:22:50.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 10 monclient: monmap: 2024-09-18T10:22:50.444 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:50.442583+0000 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:50.442583+0000 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:50.445 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 1 Processor -- start 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 1 -- start start 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 10 monclient: init 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.435+0000 7ffa8a7d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-b con 0x7ffa8409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa8409b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa8409c5e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8409dc90 con 0x7ffa8409c5e0 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8409afa0 con 0x7ffa8409b740 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8409b500 con 0x7ffa8409abd0 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient(hunting): authenticate will time out at 1659.368129s 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa89fd1700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7ffa8409c5e0 legacy=0x7ffa84137690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49142/0 (socket says 172.21.15.142:49142) 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 -- 172.21.15.142:0/3997507533 learned_addr learned my addr 172.21.15.142:0/3997507533 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): get_auth_request con 0x7ffa8409abd0 auth_method 0 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient(hunting): handle_auth_done global_id 75356 payload 274 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 -- 172.21.15.142:0/3997507533 >> v1:172.21.15.150:6789/0 conn(0x7ffa8409b740 legacy=0x7ffa8409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 -- 172.21.15.142:0/3997507533 >> v1:172.21.15.142:6789/0 conn(0x7ffa8409c5e0 legacy=0x7ffa84137690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 monclient: found mon.noname-b 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 -- 172.21.15.142:0/3997507533 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8406f810 con 0x7ffa8409abd0 2024-09-18T10:22:50.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 -- 172.21.15.142:0/3997507533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 msgr2=0x7ffa840960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa897d0700 1 --2- 172.21.15.142:0/3997507533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa740099e0 tx=0x7ffa74009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac8c17161051ac8a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 1 -- 172.21.15.142:0/3997507533 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400bb50 con 0x7ffa8409abd0 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: dump: 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.462 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 5 monclient: authenticate success, global_id 75356 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 1 -- 172.21.15.142:0/3997507533 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7400bcb0 con 0x7ffa8409abd0 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 1 -- 172.21.15.142:0/3997507533 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74011670 con 0x7ffa8409abd0 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa83fff700 10 monclient: dump: 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.463 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.439+0000 7ffa8a7d2700 4 set_mon_vals no callback set 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: shutdown 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/3997507533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 msgr2=0x7ffa840960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/3997507533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa740099e0 tx=0x7ffa74009250 comp rx=0 tx=0).stop 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/3997507533 shutdown_connections 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/3997507533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409abd0 0x7ffa840960e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/3997507533 >> 172.21.15.142:0/3997507533 conn(0x7ffa84070ef0 msgr2=0x7ffa840712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/3997507533 shutdown_connections 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/3997507533 wait complete. 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: build_initial_monmap 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: monmap: 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:50.465 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 Processor -- start 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- start start 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: init 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa8409abd0 0x7ffa8413f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-b con 0x7ffa8409abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa8409b740 0x7ffa8413fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-a con 0x7ffa8409b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:50.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): picked mon.noname-c con 0x7ffa8409c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8408e340 con 0x7ffa8409b740 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8409d240 con 0x7ffa8409abd0 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840a0200 con 0x7ffa8409c5e0 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa8a7d2700 10 monclient(hunting): authenticate will time out at 1659.375303s 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa88fcf700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa8409b740 msgr2=0x7ffa8413fc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa88fcf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa8409b740 0x7ffa8413fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49144/0 (socket says 172.21.15.142:49144) 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 learned_addr learned my addr 172.21.15.142:0/917224923 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): get_auth_request con 0x7ffa8409c5e0 auth_method 0 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa897d0700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa8409abd0 0x7ffa8413f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa897d0700 10 monclient(hunting): get_auth_request con 0x7ffa8409abd0 auth_method 0 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa897d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa897d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa897d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient(hunting): handle_auth_done global_id 75359 payload 824 2024-09-18T10:22:50.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa8409abd0 msgr2=0x7ffa8413f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa8409abd0 0x7ffa8413f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa8409b740 msgr2=0x7ffa8413fc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa8409b740 0x7ffa8413fc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 monclient: found mon.noname-c 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.443+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740094f0 con 0x7ffa8409c5e0 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 msgr2=0x7ffa841401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa89fd1700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00bf80 tx=0x7ffa7c009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c016430 con 0x7ffa8409c5e0 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: dump: 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.471 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c0165b0 con 0x7ffa8409c5e0 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 5 monclient: authenticate success, global_id 75359 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 monclient: _renew_subs 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8408f860 con 0x7ffa8409c5e0 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c016760 con 0x7ffa8409c5e0 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa817fa700 10 monclient: dump: 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.473 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa837fe700 4 set_mon_vals no callback set 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 monclient: _renew_subs 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8408d770 con 0x7ffa8409c5e0 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 0x7ffa840883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8408dd30 con 0x7ffa84087fe0 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 0x7ffa840883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient: get_auth_request con 0x7ffa84087fe0 auth_method 0 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.447+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffa7c012020 con 0x7ffa8409c5e0 2024-09-18T10:22:50.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa897d0700 10 monclient(hunting): handle_auth_done global_id 75362 payload 824 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa897d0700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa84072a80 con 0x7ffa84087fe0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa70035a70 0x7ffa70037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ffa7c0491a0 con 0x7ffa8409c5e0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa88fcf700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa70035a70 0x7ffa70037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa88fcf700 10 monclient: get_auth_request con 0x7ffa70035a70 auth_method 0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa897d0700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 msgr2=0x7ffa840883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa897d0700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 0x7ffa840883b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ffa8406fd20 tx=0x7ffa74016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74011e40 con 0x7ffa84087fe0 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 10 monclient: dump: 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ffa7401c450 con 0x7ffa84087fe0 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 msgr2=0x7ffa840883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 0x7ffa840883b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ffa8406fd20 tx=0x7ffa74016000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 msgr2=0x7ffa840883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:50.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa817fa700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84087fe0 0x7ffa840883b0 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ffa8406fd20 tx=0x7ffa74016000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.451+0000 7ffa88fcf700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa70035a70 0x7ffa70037f20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ffa78007fd0 tx=0x7ffa78007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa8a7d2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:50.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 0x7ffa8408bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa8a7d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa84090590 con 0x7ffa840888f0 2024-09-18T10:22:50.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 0x7ffa8408bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient: get_auth_request con 0x7ffa840888f0 auth_method 0 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 10 monclient(hunting): handle_auth_done global_id 75365 payload 824 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.463+0000 7ffa897d0700 1 -- 172.21.15.142:0/917224923 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa840678a0 con 0x7ffa840888f0 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa897d0700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 msgr2=0x7ffa8408bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa897d0700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 0x7ffa8408bc30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009930 tx=0x7ffa7401f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74025850 con 0x7ffa840888f0 2024-09-18T10:22:50.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 10 monclient: dump: 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7ffa740259b0 con 0x7ffa840888f0 2024-09-18T10:22:50.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 msgr2=0x7ffa8408bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 0x7ffa8408bc30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009930 tx=0x7ffa7401f040 comp rx=0 tx=0).stop 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 msgr2=0x7ffa8408bc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa817fa700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840888f0 0x7ffa8408bc30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:22:50.483 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 130, 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.484 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.485 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:50.486 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:50.487 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:50.488 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:50.489 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:50.490 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.491 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.492 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.493 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:50.494 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.495 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.496 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:50.497 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:50.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa70035a70 msgr2=0x7ffa70037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffa70035a70 0x7ffa70037f20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ffa78007fd0 tx=0x7ffa78007b00 comp rx=0 tx=0).stop 2024-09-18T10:22:50.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 10 monclient: shutdown 2024-09-18T10:22:50.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 msgr2=0x7ffa841401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa89fd1700 1 -- 172.21.15.142:0/917224923 reap_dead start 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00bf80 tx=0x7ffa7c009730 comp rx=0 tx=0).stop 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 shutdown_connections 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 --2- 172.21.15.142:0/917224923 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa8409c5e0 0x7ffa841401c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 >> 172.21.15.142:0/917224923 conn(0x7ffa84070ef0 msgr2=0x7ffa8409def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 shutdown_connections 2024-09-18T10:22:50.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.467+0000 7ffa8a7d2700 1 -- 172.21.15.142:0/917224923 wait complete. 2024-09-18T10:22:50.500 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:22:50.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:22:50.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: get_monmap_and_config 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: build_initial_monmap 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: monmap: 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:22:50.613953+0000 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:22:50.613953+0000 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.616 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:22:50.617 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:22:50.617 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:22:50.617 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:22:50.617 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:22:50.617 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 Processor -- start 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 -- start start 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: init 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 0x7f5a04098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-a con 0x7f5a0409a830 addr v2:172.21.15.142:3300/0 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a0409ac00 addr v1:172.21.15.150:6789/0 2024-09-18T10:22:50.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5a0413b6e0 0x7f5a0413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-c con 0x7f5a0413b6e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a04099820 con 0x7f5a0409ac00 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a04099d50 con 0x7f5a0409a830 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0409dca0 con 0x7f5a0413b6e0 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a0ab96700 10 monclient(hunting): authenticate will time out at 1659.539791s 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a08932700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 msgr2=0x7f5a04098bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.607+0000 7f5a08932700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 0x7f5a04098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5a0413b6e0 0x7f5a0413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a03fff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5a0409ac00 0x7f5a04099110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:60186/0 (socket says 172.21.15.142:60186) 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a03fff700 1 -- 172.21.15.142:0/504770332 learned_addr learned my addr 172.21.15.142:0/504770332 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): get_auth_request con 0x7f5a0413b6e0 auth_method 0 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient(hunting): handle_auth_done global_id 154168 payload 274 2024-09-18T10:22:50.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 -- 172.21.15.142:0/504770332 >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 msgr2=0x7f5a04098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 --2- 172.21.15.142:0/504770332 >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 0x7f5a04098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 -- 172.21.15.142:0/504770332 >> v1:172.21.15.150:6789/0 conn(0x7f5a0409ac00 legacy=0x7f5a04099110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 monclient: found mon.noname-c 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:22:50.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 -- 172.21.15.142:0/504770332 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04099a70 con 0x7f5a0413b6e0 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 -- 172.21.15.142:0/504770332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 msgr2=0x7f5a0413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a09133700 1 --2- 172.21.15.142:0/504770332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a0413dac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a0406fdb0 tx=0x7f59f40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be41bb3ca1d7c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 1 -- 172.21.15.142:0/504770332 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400b9a0 con 0x7f5a0413b6e0 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: dump: 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.629 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 1 -- 172.21.15.142:0/504770332 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f400bb20 con 0x7f5a0413b6e0 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a0ab96700 5 monclient: authenticate success, global_id 154168 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a0ab96700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:22:50.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a0ab96700 10 monclient: get_monmap_and_config success 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a0ab96700 4 set_mon_vals no callback set 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 1 -- 172.21.15.142:0/504770332 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4011420 con 0x7f5a0413b6e0 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.611+0000 7f5a037fe700 10 monclient: dump: 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: shutdown 2024-09-18T10:22:50.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- 172.21.15.142:0/504770332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 msgr2=0x7f5a0413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/504770332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a0413dac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5a0406fdb0 tx=0x7f59f40091a0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- 172.21.15.142:0/504770332 shutdown_connections 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/504770332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a0413dac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/504770332 >> v2:172.21.15.142:3300/0 conn(0x7f5a0409a830 0x7f5a04098bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- 172.21.15.142:0/504770332 >> 172.21.15.142:0/504770332 conn(0x7f5a040a14e0 msgr2=0x7f5a040a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- 172.21.15.142:0/504770332 shutdown_connections 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- 172.21.15.142:0/504770332 wait complete. 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: build_initial_monmap 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: monmap: 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:22:50.634 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 Processor -- start 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- start start 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: init 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: _reopen_session rank -1 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a0409a830 0x7f5a04094190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-a con 0x7f5a0409a830 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 0x7f5a04137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-c con 0x7f5a0409ac00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a04138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): picked mon.noname-b con 0x7f5a0413b6e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0413d250 con 0x7f5a0409a830 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04139640 con 0x7f5a0413b6e0 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0413b3a0 con 0x7f5a0409ac00 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): _renew_subs 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a0ab96700 10 monclient(hunting): authenticate will time out at 1659.546963s 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a08932700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a0409a830 msgr2=0x7f5a04094190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a08932700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a0409a830 0x7f5a04094190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 0x7f5a04137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 0x7f5a04137cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49174/0 (socket says 172.21.15.142:49174) 2024-09-18T10:22:50.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 -- 172.21.15.142:0/1266475417 learned_addr learned my addr 172.21.15.142:0/1266475417 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): get_auth_request con 0x7f5a0409ac00 auth_method 0 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a04138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): get_auth_request con 0x7f5a0413b6e0 auth_method 0 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a09133700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient(hunting): handle_auth_done global_id 75368 payload 824 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 msgr2=0x7f5a04138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413b6e0 0x7f5a04138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a0409a830 msgr2=0x7f5a04094190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a0409a830 0x7f5a04094190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient: _finish_hunting 0 2024-09-18T10:22:50.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 monclient: found mon.noname-c 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.615+0000 7f5a03fff700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4009420 con 0x7f5a0409ac00 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a03fff700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 msgr2=0x7f5a04137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a03fff700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 0x7f5a04137cc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00cca0 tx=0x7f59fc005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00abd0 con 0x7f5a0409ac00 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: dump: 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.639 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: _finish_auth 0 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc00ad50 con 0x7f5a0409ac00 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 5 monclient: authenticate success, global_id 75368 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 10 monclient: _renew_subs 2024-09-18T10:22:50.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0413b0c0 con 0x7f5a0409ac00 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc007340 con 0x7f5a0409ac00 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a00ff9700 10 monclient: dump: 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a02ffd700 4 set_mon_vals no callback set 2024-09-18T10:22:50.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 10 monclient: _renew_subs 2024-09-18T10:22:50.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:22:50.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.619+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a040946d0 con 0x7f5a0409ac00 2024-09-18T10:22:50.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a0ab96700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 0x7f5a04138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04064950 con 0x7f5a0413a160 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f59fc007730 con 0x7f5a0409ac00 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 0x7f5a04138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient: get_auth_request con 0x7f5a0413a160 auth_method 0 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a00ff9700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f59f0035aa0 0x7f59f0037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f59fc049480 con 0x7f5a0409ac00 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 10 monclient(hunting): handle_auth_done global_id 154174 payload 824 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a04090bf0 con 0x7f5a0413a160 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a09133700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f59f0035aa0 0x7f59f0037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a09133700 10 monclient: get_auth_request con 0x7f59f0035aa0 auth_method 0 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 msgr2=0x7f5a04138740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.623+0000 7f5a08932700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 0x7f5a04138740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59f80039d0 tx=0x7f59f8009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f80050f0 con 0x7f5a0413a160 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:50.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 10 monclient: dump: 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a09133700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f59f0035aa0 0x7f59f0037f50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5a0409d750 tx=0x7f59f4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f59f8005250 con 0x7f5a0413a160 2024-09-18T10:22:50.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 msgr2=0x7f5a04138740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 0x7f5a04138740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59f80039d0 tx=0x7f59f8009220 comp rx=0 tx=0).stop 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 msgr2=0x7f5a04138740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.627+0000 7f5a00ff9700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a0413a160 0x7f5a04138740 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a0ab96700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 0x7f5a040957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a0ab96700 10 monclient(hunting): start opening mon connection 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04093820 con 0x7f5a040953b0 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 0x7f5a040957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient: get_auth_request con 0x7f5a040953b0 auth_method 0 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:22:50.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:22:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:22:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:22:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:22:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 10 monclient(hunting): handle_auth_done global_id 154177 payload 824 2024-09-18T10:22:50.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.635+0000 7f5a08932700 1 -- 172.21.15.142:0/1266475417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5a04090ef0 con 0x7f5a040953b0 2024-09-18T10:22:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a08932700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 msgr2=0x7f5a040957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:22:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a08932700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 0x7f5a040957b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59f8014000 tx=0x7f59f8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:22:50.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8020840 con 0x7f5a040953b0 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 10 monclient: dump: 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:22:50.652 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f59f8003d90 con 0x7f5a040953b0 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:22:50.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:22:50.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 msgr2=0x7f5a040957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 0x7f5a040957b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59f8014000 tx=0x7f59f8015000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 msgr2=0x7f5a040957b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:22:50.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.639+0000 7f5a00ff9700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a040953b0 0x7f5a040957b0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59f8014000 tx=0x7f59f8015000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 130, 2024-09-18T10:22:50.655 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.656 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.657 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.661 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:22:50.662 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:22:50.663 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:22:50.664 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.665 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:22:50.666 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.667 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.668 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:22:50.668 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.669 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.671 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.672 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.673 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:22:50.674 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:22:50.675 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:22:50.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f59f0035aa0 msgr2=0x7f59f0037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f59f0035aa0 0x7f59f0037f50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5a0409d750 tx=0x7f59f4016000 comp rx=0 tx=0).stop 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 10 monclient: shutdown 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 msgr2=0x7f5a04137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 --2- 172.21.15.142:0/1266475417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a0409ac00 0x7f5a04137cc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00cca0 tx=0x7f59fc005480 comp rx=0 tx=0).stop 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a09133700 1 -- 172.21.15.142:0/1266475417 reap_dead start 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a09133700 1 -- 172.21.15.142:0/1266475417 reap_dead start 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 shutdown_connections 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 >> 172.21.15.142:0/1266475417 conn(0x7f5a040a14e0 msgr2=0x7f5a0409dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 shutdown_connections 2024-09-18T10:22:50.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:22:50.643+0000 7f5a0ab96700 1 -- 172.21.15.142:0/1266475417 wait complete. 2024-09-18T10:22:50.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:22:50.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:23:10.679 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:23:10.679 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:23:10.679 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:23:10.679 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:23:10.682 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:23:10.682 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-18T10:23:10.683 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-18T10:23:10.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-18T10:23:30.683 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-18T10:23:30.684 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-18T10:23:30.685 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:23:30.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:30.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: get_monmap_and_config 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: build_initial_monmap 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: monmap: 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:30.796756+0000 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:30.796756+0000 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:30.799 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:30.800 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:30.800 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:30.800 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 Processor -- start 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 -- start start 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: init 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 0x7f7e0409afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-c con 0x7f7e0409b6f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:30.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e0409bed0 0x7f7e0412e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-a con 0x7f7e0409bed0 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e0409c2f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e04098cd0 con 0x7f7e0409c2f0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0406f940 con 0x7f7e0409bed0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0409b4f0 con 0x7f7e0409b6f0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e0974c700 10 monclient(hunting): authenticate will time out at 1699.722291s 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e0409bed0 0x7f7e0412e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e037fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7e0409c2f0 0x7f7e0412ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33176/0 (socket says 172.21.15.142:33176) 2024-09-18T10:23:30.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7e0409bed0 0x7f7e0412e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33542/0 (socket says 172.21.15.142:33542) 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e037fe700 1 -- 172.21.15.142:0/4665258 learned_addr learned my addr 172.21.15.142:0/4665258 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): get_auth_request con 0x7f7e0409bed0 auth_method 0 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 1 --2- 172.21.15.142:0/4665258 >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 0x7f7e0409afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 10 monclient(hunting): get_auth_request con 0x7f7e0409b6f0 auth_method 0 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:30.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient(hunting): handle_auth_done global_id 164097 payload 274 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 -- 172.21.15.142:0/4665258 >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 msgr2=0x7f7e0409afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 --2- 172.21.15.142:0/4665258 >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 0x7f7e0409afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 -- 172.21.15.142:0/4665258 >> v1:172.21.15.142:6789/0 conn(0x7f7e0409c2f0 legacy=0x7f7e0412ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient: _finish_hunting 0 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 monclient: found mon.noname-a 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e027fc700 1 -- 172.21.15.142:0/4665258 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04099110 con 0x7f7e0409bed0 2024-09-18T10:23:30.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.794+0000 7f7e02ffd700 1 --2- 172.21.15.142:0/4665258 >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 0x7f7e0409afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:23:30.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e027fc700 1 -- 172.21.15.142:0/4665258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409bed0 msgr2=0x7f7e0412e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e027fc700 1 --2- 172.21.15.142:0/4665258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409bed0 0x7f7e0412e6a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e0409b630 tx=0x7f7dec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f730b957e2f16b21 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 1 -- 172.21.15.142:0/4665258 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00b890 con 0x7f7e0409bed0 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: dump: 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:30.815 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: _finish_auth 0 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 5 monclient: authenticate success, global_id 164097 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 1 -- 172.21.15.142:0/4665258 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec00b9f0 con 0x7f7e0409bed0 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 4 set_mon_vals no callback set 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 1 -- 172.21.15.142:0/4665258 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00bd90 con 0x7f7e0409bed0 2024-09-18T10:23:30.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e01ffb700 10 monclient: dump: 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 10 monclient: shutdown 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- 172.21.15.142:0/4665258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409bed0 msgr2=0x7f7e0412e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 --2- 172.21.15.142:0/4665258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409bed0 0x7f7e0412e6a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e0409b630 tx=0x7f7dec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- 172.21.15.142:0/4665258 shutdown_connections 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 --2- 172.21.15.142:0/4665258 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409bed0 0x7f7e0412e6a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 --2- 172.21.15.142:0/4665258 >> v2:172.21.15.150:3300/0 conn(0x7f7e0409b6f0 0x7f7e0409afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- 172.21.15.142:0/4665258 >> 172.21.15.142:0/4665258 conn(0x7f7e040a14e0 msgr2=0x7f7e040a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- 172.21.15.142:0/4665258 shutdown_connections 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- 172.21.15.142:0/4665258 wait complete. 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 10 monclient: build_initial_monmap 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 10 monclient: monmap: 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:30.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 Processor -- start 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 1 -- start start 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.798+0000 7f7e0974c700 10 monclient: init 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:23:30.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-a con 0x7f7e0409b6f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e0409bed0 0x7f7e041293a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-b con 0x7f7e0409bed0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 0x7f7e041298e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): picked mon.noname-c con 0x7f7e0409c2f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0412b210 con 0x7f7e0409b6f0 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0406fac0 con 0x7f7e0409bed0 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0412e7f0 con 0x7f7e0409c2f0 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:30.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e0974c700 10 monclient(hunting): authenticate will time out at 1699.729135s 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33556/0 (socket says 172.21.15.142:33556) 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 -- 172.21.15.142:0/39959599 learned_addr learned my addr 172.21.15.142:0/39959599 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): get_auth_request con 0x7f7e0409b6f0 auth_method 0 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 0x7f7e041298e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 10 monclient(hunting): get_auth_request con 0x7f7e0409c2f0 auth_method 0 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e027fc700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e0409bed0 0x7f7e041293a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e027fc700 10 monclient(hunting): get_auth_request con 0x7f7e0409bed0 auth_method 0 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e027fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient(hunting): handle_auth_done global_id 164100 payload 824 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 -- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 msgr2=0x7f7e041298e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 0x7f7e041298e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 -- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e0409bed0 msgr2=0x7f7e041293a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e0409bed0 0x7f7e041293a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:30.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 monclient: found mon.noname-a 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e02ffd700 1 -- 172.21.15.142:0/39959599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec009420 con 0x7f7e0409b6f0 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.802+0000 7f7e037fe700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 0x7f7e041298e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e02ffd700 1 -- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 msgr2=0x7f7e04128e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e02ffd700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00d1d0 tx=0x7f7dfc009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00de70 con 0x7f7e0409b6f0 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: dump: 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:30.823 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: _finish_auth 0 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc00f070 con 0x7f7e0409b6f0 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 5 monclient: authenticate success, global_id 164100 2024-09-18T10:23:30.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _renew_subs 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0412a9e0 con 0x7f7e0409b6f0 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc019550 con 0x7f7e0409b6f0 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 10 monclient: dump: 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:30.825 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e017fa700 4 set_mon_vals no callback set 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _renew_subs 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0412b5b0 con 0x7f7e0409b6f0 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0412c600 con 0x7f7e0409b6f0 2024-09-18T10:23:30.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7dfc019980 con 0x7f7e0409b6f0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 0x7f7de4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7dfc049710 con 0x7f7e0409b6f0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e027fc700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 0x7f7de4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.806+0000 7f7e027fc700 10 monclient: get_auth_request con 0x7f7de4035ac0 auth_method 0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.810+0000 7f7e027fc700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 0x7f7de4037f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0093f0 tx=0x7f7dec017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.818+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7e0412c600 con 0x7f7e0409b6f0 2024-09-18T10:23:30.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.818+0000 7f7dfaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:30.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:30.818+0000 7f7dfaffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:31.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7e0974c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:31.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7e0974c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:31.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7e0974c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7e0412c600 con 0x7f7e0409b6f0 2024-09-18T10:23:31.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7dfaffd700 1 -- 172.21.15.142:0/39959599 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f7e0412c600 con 0x7f7e0409b6f0 2024-09-18T10:23:31.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7dfaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:31.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.042+0000 7f7dfaffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:31.049 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 msgr2=0x7f7de4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 0x7f7de4037f70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7dec0093f0 tx=0x7f7dec017000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 10 monclient: shutdown 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 msgr2=0x7f7e04128e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00d1d0 tx=0x7f7dfc009a90 comp rx=0 tx=0).stop 2024-09-18T10:23:31.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 shutdown_connections 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7de4035ac0 0x7f7de4037f70 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7e0409c2f0 0x7f7e041298e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e0409bed0 0x7f7e041293a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 --2- 172.21.15.142:0/39959599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e0409b6f0 0x7f7e04128e60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 >> 172.21.15.142:0/39959599 conn(0x7f7e040a14e0 msgr2=0x7f7e0406ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 shutdown_connections 2024-09-18T10:23:31.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.046+0000 7f7e0974c700 1 -- 172.21.15.142:0/39959599 wait complete. 2024-09-18T10:23:31.063 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:23:31.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:23:31.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: get_monmap_and_config 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: monmap: 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:31.166212+0000 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:31.166212+0000 2024-09-18T10:23:31.168 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:31.169 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 Processor -- start 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 -- start start 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: init 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdcb409cdd0 0x7fdcb409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-c con 0x7fdcb409cdd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcb40a02e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdcb40a0df0 0x7fdcb4139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-a con 0x7fdcb40a0df0 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb409da60 con 0x7fdcb40a02e0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb409d1a0 con 0x7fdcb40a0df0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb409e980 con 0x7fdcb409cdd0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb98e7700 10 monclient(hunting): authenticate will time out at 1700.091648s 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdcb40a0df0 0x7fdcb4139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdcb40a0df0 0x7fdcb4139a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33562/0 (socket says 172.21.15.142:33562) 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb27fc700 1 --1- 172.21.15.142:0/3777231922 >> v1:172.21.15.142:6789/0 conn(0x7fdcb40a02e0 0x7fdcb40a06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33184/0 (socket says 172.21.15.142:33184) 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3777231922 learned_addr learned my addr 172.21.15.142:0/3777231922 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request con 0x7fdcb40a0df0 auth_method 0 2024-09-18T10:23:31.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.162+0000 7fdcb37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/3777231922 >> v2:172.21.15.150:3300/0 conn(0x7fdcb409cdd0 0x7fdcb409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request con 0x7fdcb409cdd0 auth_method 0 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_done global_id 164103 payload 274 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3777231922 >> v2:172.21.15.150:3300/0 conn(0x7fdcb409cdd0 msgr2=0x7fdcb409e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/3777231922 >> v2:172.21.15.150:3300/0 conn(0x7fdcb409cdd0 0x7fdcb409e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3777231922 >> v1:172.21.15.142:6789/0 conn(0x7fdcb40a02e0 legacy=0x7fdcb40a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 monclient: found mon.noname-a 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3777231922 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb409d780 con 0x7fdcb40a0df0 2024-09-18T10:23:31.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3777231922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 msgr2=0x7fdcb4139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/3777231922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb4139a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdcb409dbf0 tx=0x7fdcac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa96aed46b38fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 1 -- 172.21.15.142:0/3777231922 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009e70 con 0x7fdcb40a0df0 2024-09-18T10:23:31.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: dump: 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb98e7700 5 monclient: authenticate success, global_id 164103 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 1 -- 172.21.15.142:0/3777231922 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac00ba40 con 0x7fdcb40a0df0 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb98e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb98e7700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 1 -- 172.21.15.142:0/3777231922 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00be00 con 0x7fdcb40a0df0 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb98e7700 4 set_mon_vals no callback set 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.166+0000 7fdcb1ffb700 10 monclient: dump: 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.183 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: shutdown 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- 172.21.15.142:0/3777231922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 msgr2=0x7fdcb4139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/3777231922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb4139a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdcb409dbf0 tx=0x7fdcac009250 comp rx=0 tx=0).stop 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- 172.21.15.142:0/3777231922 shutdown_connections 2024-09-18T10:23:31.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/3777231922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb4139a80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/3777231922 >> v2:172.21.15.150:3300/0 conn(0x7fdcb409cdd0 0x7fdcb409e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- 172.21.15.142:0/3777231922 >> 172.21.15.142:0/3777231922 conn(0x7fdcb4071680 msgr2=0x7fdcb40a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- 172.21.15.142:0/3777231922 shutdown_connections 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- 172.21.15.142:0/3777231922 wait complete. 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: monmap: 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:31.185 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 Processor -- start 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- start start 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: init 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb409cdd0 0x7fdcb4141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-b con 0x7fdcb409cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb40a02e0 0x7fdcb4142070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-c con 0x7fdcb40a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb41425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): picked mon.noname-a con 0x7fdcb40a0df0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40984b0 con 0x7fdcb40a0df0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb409bed0 con 0x7fdcb409cdd0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40905d0 con 0x7fdcb40a02e0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb98e7700 10 monclient(hunting): authenticate will time out at 1700.098611s 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb41425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb40a02e0 0x7fdcb4142070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb41425b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33572/0 (socket says 172.21.15.142:33572) 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 learned_addr learned my addr 172.21.15.142:0/4068499445 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request con 0x7fdcb40a0df0 auth_method 0 2024-09-18T10:23:31.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): get_auth_request con 0x7fdcb40a02e0 auth_method 0 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb409cdd0 0x7fdcb4141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request con 0x7fdcb409cdd0 auth_method 0 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_done global_id 164106 payload 824 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb40a02e0 msgr2=0x7fdcb4142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb40a02e0 0x7fdcb4142070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb409cdd0 msgr2=0x7fdcb4141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb409cdd0 0x7fdcb4141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 monclient: found mon.noname-a 2024-09-18T10:23:31.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac0094f0 con 0x7fdcb40a0df0 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.170+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcb409cdd0 0x7fdcb4141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 msgr2=0x7fdcb41425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb41425b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40a12c0 tx=0x7fdcac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009e70 con 0x7fdcb40a0df0 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: dump: 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac006a90 con 0x7fdcb40a0df0 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 5 monclient: authenticate success, global_id 164106 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 monclient: _renew_subs 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb40902f0 con 0x7fdcb40a0df0 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac025420 con 0x7fdcb40a0df0 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 10 monclient: dump: 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.192 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb0ff9700 4 set_mon_vals no callback set 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 monclient: _renew_subs 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb408f390 con 0x7fdcb40a0df0 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 0x7fdcb405e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb408e260 con 0x7fdcb408de90 2024-09-18T10:23:31.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 0x7fdcb405e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient: get_auth_request con 0x7fdcb408de90 auth_method 0 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdcac023070 con 0x7fdcb40a0df0 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_done global_id 164109 payload 824 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.174+0000 7fdcb2ffd700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdcb4090760 con 0x7fdcb408de90 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdca0035ac0 0x7fdca0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdcac04a070 con 0x7fdcb40a0df0 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdcb27fc700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdca0035ac0 0x7fdca0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdcb27fc700 10 monclient: get_auth_request con 0x7fdca0035ac0 auth_method 0 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdcb2ffd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 msgr2=0x7fdcb405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 0x7fdcb405e890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011840 tx=0x7fdca8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca800fad0 con 0x7fdcb408de90 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 10 monclient: dump: 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.198 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdca800fc30 con 0x7fdcb408de90 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 msgr2=0x7fdcb405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 0x7fdcb405e890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdca8011840 tx=0x7fdca8008040 comp rx=0 tx=0).stop 2024-09-18T10:23:31.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 msgr2=0x7fdcb405e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdc9affd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb408de90 0x7fdcb405e890 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.178+0000 7fdcb27fc700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdca0035ac0 0x7fdca0037f70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4099350 tx=0x7fdca40053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb98e7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 0x7fdcb4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb98e7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40648c0 con 0x7fdcb405edd0 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 0x7fdcb4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient: get_auth_request con 0x7fdcb405edd0 auth_method 0 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 10 monclient(hunting): handle_auth_done global_id 164112 payload 824 2024-09-18T10:23:31.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.190+0000 7fdcb2ffd700 1 -- 172.21.15.142:0/4068499445 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdcb406d320 con 0x7fdcb405edd0 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb2ffd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 msgr2=0x7fdcb4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb2ffd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 0x7fdcb4029d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdca801c000 tx=0x7fdca801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8006930 con 0x7fdcb405edd0 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 10 monclient: dump: 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.202 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fdca8015880 con 0x7fdcb405edd0 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 msgr2=0x7fdcb4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 0x7fdcb4029d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdca801c000 tx=0x7fdca801d000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:31.204 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.205 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.206 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:31.207 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.208 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:31.209 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.210 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.211 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.212 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:31.213 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.214 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.214 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.215 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.216 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.217 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.218 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.218 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:23:31.218 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.219 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.219 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.219 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:31.219 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:31.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 msgr2=0x7fdcb4029d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdc9affd700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb405edd0 0x7fdcb4029d00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdca0035ac0 msgr2=0x7fdca0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdca0035ac0 0x7fdca0037f70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4099350 tx=0x7fdca40053c0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 10 monclient: shutdown 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 msgr2=0x7fdcb41425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 --2- 172.21.15.142:0/4068499445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb40a0df0 0x7fdcb41425b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40a12c0 tx=0x7fdcac017000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 reap_dead start 2024-09-18T10:23:31.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb37fe700 1 -- 172.21.15.142:0/4068499445 reap_dead start 2024-09-18T10:23:31.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 shutdown_connections 2024-09-18T10:23:31.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 >> 172.21.15.142:0/4068499445 conn(0x7fdcb4071680 msgr2=0x7fdcb40a4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 shutdown_connections 2024-09-18T10:23:31.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.194+0000 7fdcb98e7700 1 -- 172.21.15.142:0/4068499445 wait complete. 2024-09-18T10:23:31.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:31.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:23:31.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:23:31.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 10 monclient: get_monmap_and_config 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 10 monclient: monmap: 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:31.338818+0000 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:31.338818+0000 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:31.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:31.342 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:31.342 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:31.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 1 Processor -- start 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 1 -- start start 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.334+0000 7f555a1c3700 10 monclient: init 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f555406f3b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55540990d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55540994f0 0x7f5554137da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-c con 0x7f55540994f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555409c330 con 0x7f555406f3b0 2024-09-18T10:23:31.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555409c530 con 0x7f55540990d0 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555409b380 con 0x7f55540994f0 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f555a1c3700 10 monclient(hunting): authenticate will time out at 1700.264453s 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55591c1700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f555406f3b0 0x7f55540989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33200/0 (socket says 172.21.15.142:33200) 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55591c1700 1 -- 172.21.15.142:0/1290965398 learned_addr learned my addr 172.21.15.142:0/1290965398 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 1 --2- 172.21.15.142:0/1290965398 >> v2:172.21.15.150:3300/0 conn(0x7f55540994f0 0x7f5554137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): get_auth_request con 0x7f55540994f0 auth_method 0 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient(hunting): handle_auth_done global_id 154192 payload 274 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 1 -- 172.21.15.142:0/1290965398 >> v1:172.21.15.150:6789/0 conn(0x7f55540990d0 legacy=0x7f5554134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 1 -- 172.21.15.142:0/1290965398 >> v1:172.21.15.142:6789/0 conn(0x7f555406f3b0 legacy=0x7f55540989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 1 monclient: found mon.noname-c 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:23:31.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.338+0000 7f55599c2700 1 -- 172.21.15.142:0/1290965398 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555409c0f0 con 0x7f55540994f0 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 1 -- 172.21.15.142:0/1290965398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 msgr2=0x7f5554137da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 1 --2- 172.21.15.142:0/1290965398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f5554137da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f554c009e60 tx=0x7f554c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3baa2567d053c890 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 1 -- 172.21.15.142:0/1290965398 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c00e070 con 0x7f55540994f0 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: dump: 2024-09-18T10:23:31.351 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 5 monclient: authenticate success, global_id 154192 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 1 -- 172.21.15.142:0/1290965398 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554c012430 con 0x7f55540994f0 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 4 set_mon_vals no callback set 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 1 -- 172.21.15.142:0/1290965398 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554c0127d0 con 0x7f55540994f0 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f554bfff700 10 monclient: dump: 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: shutdown 2024-09-18T10:23:31.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- 172.21.15.142:0/1290965398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 msgr2=0x7f5554137da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1290965398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f5554137da0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f554c009e60 tx=0x7f554c009780 comp rx=0 tx=0).stop 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- 172.21.15.142:0/1290965398 shutdown_connections 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1290965398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f5554137da0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- 172.21.15.142:0/1290965398 >> 172.21.15.142:0/1290965398 conn(0x7f5554070ef0 msgr2=0x7f55540712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- 172.21.15.142:0/1290965398 shutdown_connections 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- 172.21.15.142:0/1290965398 wait complete. 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: monmap: 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:31.355 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 Processor -- start 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- start start 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: init 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f555406f3b0 0x7f5554141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-a con 0x7f555406f3b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:31.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 0x7f5554141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-c con 0x7f55540990d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f555408dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): picked mon.noname-b con 0x7f55540994f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5554096fa0 con 0x7f555406f3b0 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55540917c0 con 0x7f55540994f0 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555408f9b0 con 0x7f55540990d0 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f555a1c3700 10 monclient(hunting): authenticate will time out at 1700.271320s 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f555406f3b0 0x7f5554141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 0x7f5554141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 0x7f5554141e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39332/0 (socket says 172.21.15.142:39332) 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f555406f3b0 0x7f5554141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33596/0 (socket says 172.21.15.142:33596) 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 1 -- 172.21.15.142:0/1547053012 learned_addr learned my addr 172.21.15.142:0/1547053012 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): get_auth_request con 0x7f55540990d0 auth_method 0 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): get_auth_request con 0x7f555406f3b0 auth_method 0 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f555408dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): get_auth_request con 0x7f55540994f0 auth_method 0 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55589c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.342+0000 7f55599c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 10 monclient(hunting): handle_auth_done global_id 75377 payload 824 2024-09-18T10:23:31.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 msgr2=0x7f555408dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55540994f0 0x7f555408dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f555406f3b0 msgr2=0x7f5554141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f555406f3b0 0x7f5554141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 monclient: found mon.noname-c 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c009a20 con 0x7f55540990d0 2024-09-18T10:23:31.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 msgr2=0x7f5554141e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55589c0700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 0x7f5554141e40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f555000c550 tx=0x7f5550007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55500117a0 con 0x7f55540990d0 2024-09-18T10:23:31.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: dump: 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.364 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 5 monclient: authenticate success, global_id 75377 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 10 monclient: _renew_subs 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55540914e0 con 0x7f55540990d0 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5550011900 con 0x7f55540990d0 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5550011cc0 con 0x7f55540990d0 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f55497fa700 10 monclient: dump: 2024-09-18T10:23:31.365 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f554b7fe700 4 set_mon_vals no callback set 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 10 monclient: _renew_subs 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.346+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5554142380 con 0x7f55540990d0 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f555a1c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5550012630 con 0x7f55540990d0 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 0x7f555408ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55540648a0 con 0x7f55540926d0 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 0x7f555408ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient: get_auth_request con 0x7f55540926d0 auth_method 0 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 10 monclient(hunting): handle_auth_done global_id 75380 payload 824 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5538035ac0 0x7f5538037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5554091a00 con 0x7f55540926d0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f5550048a00 con 0x7f55540990d0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55599c2700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5538035ac0 0x7f5538037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55599c2700 10 monclient: get_auth_request con 0x7f5538035ac0 auth_method 0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 msgr2=0x7f555408ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55591c1700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 0x7f555408ebd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f554400eac0 tx=0x7f5544007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554400cc40 con 0x7f55540926d0 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.350+0000 7f55497fa700 10 monclient: dump: 2024-09-18T10:23:31.368 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f554400cda0 con 0x7f55540926d0 2024-09-18T10:23:31.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55599c2700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5538035ac0 0x7f5538037f70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f554c0099f0 tx=0x7f554c018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 msgr2=0x7f555408ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 0x7f555408ebd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f554400eac0 tx=0x7f5544007f90 comp rx=0 tx=0).stop 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 msgr2=0x7f555408ebd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.354+0000 7f55497fa700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540926d0 0x7f555408ebd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f555a1c3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 0x7f5554029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f555a1c3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55541425a0 con 0x7f555408f110 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 0x7f5554029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient: get_auth_request con 0x7f555408f110 auth_method 0 2024-09-18T10:23:31.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 10 monclient(hunting): handle_auth_done global_id 75383 payload 824 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.362+0000 7f55591c1700 1 -- 172.21.15.142:0/1547053012 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f555406dad0 con 0x7f555408f110 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55591c1700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 msgr2=0x7f5554029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55591c1700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 0x7f5554029d00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5544007360 tx=0x7f5544016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544013e00 con 0x7f555408f110 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:31.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 10 monclient: dump: 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f554400ca30 con 0x7f555408f110 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:31.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 msgr2=0x7f5554029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 0x7f5554029d00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5544007360 tx=0x7f5544016000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 msgr2=0x7f5554029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55497fa700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f555408f110 0x7f5554029d00 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5544007360 tx=0x7f5544016000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:31.375 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.376 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.377 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.378 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.379 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.383 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.384 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.385 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:23:31.386 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:31.387 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5538035ac0 msgr2=0x7f5538037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5538035ac0 0x7f5538037f70 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f554c0099f0 tx=0x7f554c018000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 10 monclient: shutdown 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 msgr2=0x7f5554141e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 --2- 172.21.15.142:0/1547053012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55540990d0 0x7f5554141e40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f555000c550 tx=0x7f5550007f60 comp rx=0 tx=0).stop 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55599c2700 1 -- 172.21.15.142:0/1547053012 reap_dead start 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f55599c2700 1 -- 172.21.15.142:0/1547053012 reap_dead start 2024-09-18T10:23:31.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 shutdown_connections 2024-09-18T10:23:31.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 >> 172.21.15.142:0/1547053012 conn(0x7f5554070ef0 msgr2=0x7f555409f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 shutdown_connections 2024-09-18T10:23:31.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.366+0000 7f555a1c3700 1 -- 172.21.15.142:0/1547053012 wait complete. 2024-09-18T10:23:31.390 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:31.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:23:31.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.498+0000 7f04426d9700 10 monclient: get_monmap_and_config 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.498+0000 7f04426d9700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.498+0000 7f04426d9700 10 monclient: monmap: 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:31.503091+0000 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:31.503091+0000 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.505 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:31.506 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:31.506 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:31.506 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:31.506 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:31.506 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.498+0000 7f04426d9700 1 Processor -- start 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 -- start start 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient: init 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 0x7f043c0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-c con 0x7f043c09abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f043c09b740 0x7f043c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-a con 0x7f043c09b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f043c09c410 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c09dc90 con 0x7f043c09c410 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c09b500 con 0x7f043c09b740 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c09afa0 con 0x7f043c09abd0 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient(hunting): authenticate will time out at 1700.428625s 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f043c09b740 0x7f043c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f043c09b740 0x7f043c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33600/0 (socket says 172.21.15.142:33600) 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 -- 172.21.15.142:0/1616183231 learned_addr learned my addr 172.21.15.142:0/1616183231 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): get_auth_request con 0x7f043c09b740 auth_method 0 2024-09-18T10:23:31.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04416d7700 1 --2- 172.21.15.142:0/1616183231 >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 0x7f043c0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient(hunting): handle_auth_done global_id 164118 payload 274 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 -- 172.21.15.142:0/1616183231 >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 msgr2=0x7f043c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 --2- 172.21.15.142:0/1616183231 >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 0x7f043c0960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 -- 172.21.15.142:0/1616183231 >> v1:172.21.15.142:6789/0 conn(0x7f043c09c410 legacy=0x7f043c136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 monclient: found mon.noname-a 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 -- 172.21.15.142:0/1616183231 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c06f810 con 0x7f043c09b740 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04416d7700 10 monclient: get_auth_request con 0x7f043c09abd0 auth_method 0 2024-09-18T10:23:31.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04416d7700 1 --2- 172.21.15.142:0/1616183231 >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 0x7f043c0960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 -- 172.21.15.142:0/1616183231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09b740 msgr2=0x7f043c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f0440ed6700 1 --2- 172.21.15.142:0/1616183231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09b740 0x7f043c09bed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f043c06a310 tx=0x7f0430009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6465b66111e553b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 1 -- 172.21.15.142:0/1616183231 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043000ba40 con 0x7f043c09b740 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: dump: 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.522 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 5 monclient: authenticate success, global_id 164118 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 1 -- 172.21.15.142:0/1616183231 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043000bba0 con 0x7f043c09b740 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f04426d9700 4 set_mon_vals no callback set 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 1 -- 172.21.15.142:0/1616183231 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430011420 con 0x7f043c09b740 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.502+0000 7f043bfff700 10 monclient: dump: 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.524 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: shutdown 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- 172.21.15.142:0/1616183231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09b740 msgr2=0x7f043c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 --2- 172.21.15.142:0/1616183231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09b740 0x7f043c09bed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f043c06a310 tx=0x7f0430009250 comp rx=0 tx=0).stop 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- 172.21.15.142:0/1616183231 shutdown_connections 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 --2- 172.21.15.142:0/1616183231 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09b740 0x7f043c09bed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 --2- 172.21.15.142:0/1616183231 >> v2:172.21.15.150:3300/0 conn(0x7f043c09abd0 0x7f043c0960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- 172.21.15.142:0/1616183231 >> 172.21.15.142:0/1616183231 conn(0x7f043c070ef0 msgr2=0x7f043c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- 172.21.15.142:0/1616183231 shutdown_connections 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- 172.21.15.142:0/1616183231 wait complete. 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: monmap: 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:31.526 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 Processor -- start 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 -- start start 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: init 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09abd0 0x7f043c089740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.506+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-a con 0x7f043c09abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 0x7f043c089c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-c con 0x7f043c09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c09c410 0x7f043c08a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): picked mon.noname-b con 0x7f043c09c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:31.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c08d070 con 0x7f043c09abd0 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c09d240 con 0x7f043c09c410 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c09feb0 con 0x7f043c09b740 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient(hunting): authenticate will time out at 1700.436186s 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 0x7f043c089c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 0x7f043c089c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39366/0 (socket says 172.21.15.142:39366) 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 -- 172.21.15.142:0/398394547 learned_addr learned my addr 172.21.15.142:0/398394547 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): get_auth_request con 0x7f043c09b740 auth_method 0 2024-09-18T10:23:31.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09abd0 0x7f043c089740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): get_auth_request con 0x7f043c09abd0 auth_method 0 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c09c410 0x7f043c08a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): get_auth_request con 0x7f043c09c410 auth_method 0 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0441ed8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient(hunting): handle_auth_done global_id 75386 payload 824 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c09c410 msgr2=0x7f043c08a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c09c410 0x7f043c08a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09abd0 msgr2=0x7f043c089740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f043c09abd0 0x7f043c089740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 monclient: found mon.noname-c 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04300094f0 con 0x7f043c09b740 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 msgr2=0x7f043c089c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f0440ed6700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 0x7f043c089c80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0430011420 tx=0x7f0430017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043001f070 con 0x7f043c09b740 2024-09-18T10:23:31.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: dump: 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.532 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043001a420 con 0x7f043c09b740 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 5 monclient: authenticate success, global_id 75386 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient: _renew_subs 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c08bf40 con 0x7f043c09b740 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043001a7e0 con 0x7f043c09b740 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04397fa700 10 monclient: dump: 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.534 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f043b7fe700 4 set_mon_vals no callback set 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient: _renew_subs 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.510+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c08cd90 con 0x7f043c09b740 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04426d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f043001abd0 con 0x7f043c09b740 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04426d9700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 0x7f043c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0648a0 con 0x7f043c08a700 2024-09-18T10:23:31.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04397fa700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0428035ac0 0x7f0428037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f043004a5c0 con 0x7f043c09b740 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 0x7f043c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f0441ed8700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0428035ac0 0x7f0428037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient: get_auth_request con 0x7f043c08a700 auth_method 0 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f0441ed8700 10 monclient: get_auth_request con 0x7f0428035ac0 auth_method 0 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 10 monclient(hunting): handle_auth_done global_id 154201 payload 824 2024-09-18T10:23:31.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.514+0000 7f04416d7700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f043c072a80 con 0x7f043c08a700 2024-09-18T10:23:31.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f0441ed8700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0428035ac0 0x7f0428037f70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f04340039b0 tx=0x7f0434007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04416d7700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 msgr2=0x7f043c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04416d7700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 0x7f043c06c370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c0077b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c00a790 con 0x7f043c08a700 2024-09-18T10:23:31.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 10 monclient: dump: 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f042c00a8f0 con 0x7f043c08a700 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 msgr2=0x7f043c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 0x7f043c06c370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c0077b0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 msgr2=0x7f043c06c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.518+0000 7f04397fa700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c08a700 0x7f043c06c370 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c0077b0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.530+0000 7f04426d9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.530+0000 7f04426d9700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 0x7f043c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.530+0000 7f04426d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.530+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c08aad0 con 0x7f043c06c8b0 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 0x7f043c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient: get_auth_request con 0x7f043c06c8b0 auth_method 0 2024-09-18T10:23:31.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 10 monclient(hunting): handle_auth_done global_id 154204 payload 824 2024-09-18T10:23:31.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 1 -- 172.21.15.142:0/398394547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f043c0678a0 con 0x7f043c06c8b0 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 msgr2=0x7f043c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04416d7700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 0x7f043c06ccb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c01a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c00ae60 con 0x7f043c06c8b0 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.534+0000 7f04397fa700 10 monclient: dump: 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f042c023390 con 0x7f043c06c8b0 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 msgr2=0x7f043c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 0x7f043c06ccb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c01a000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 msgr2=0x7f043c06ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04397fa700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c06c8b0 0x7f043c06ccb0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f042c00d5e0 tx=0x7f042c01a000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:31.544 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.545 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.546 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:31.547 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:31.548 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.549 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:31.550 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:31.551 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:31.553 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.554 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.555 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:31.556 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:31.557 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:31.557 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:31.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:31.557 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:31.557 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:31.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0428035ac0 msgr2=0x7f0428037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0428035ac0 0x7f0428037f70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f04340039b0 tx=0x7f0434007ea0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 10 monclient: shutdown 2024-09-18T10:23:31.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 msgr2=0x7f043c089c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 --2- 172.21.15.142:0/398394547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f043c09b740 0x7f043c089c80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0430011420 tx=0x7f0430017000 comp rx=0 tx=0).stop 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f0441ed8700 1 -- 172.21.15.142:0/398394547 reap_dead start 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f0441ed8700 1 -- 172.21.15.142:0/398394547 reap_dead start 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 shutdown_connections 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 >> 172.21.15.142:0/398394547 conn(0x7f043c070ef0 msgr2=0x7f043c099ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 shutdown_connections 2024-09-18T10:23:31.559 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.538+0000 7f04426d9700 1 -- 172.21.15.142:0/398394547 wait complete. 2024-09-18T10:23:31.559 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:31.559 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:23:31.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.666+0000 7f04316d6700 10 monclient: get_monmap_and_config 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.666+0000 7f04316d6700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.666+0000 7f04316d6700 10 monclient: monmap: 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:31.671099+0000 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:31.671099+0000 2024-09-18T10:23:31.673 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:31.674 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.666+0000 7f04316d6700 1 Processor -- start 2024-09-18T10:23:31.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 -- start start 2024-09-18T10:23:31.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient: init 2024-09-18T10:23:31.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f042c0a5f90 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-b con 0x7f042c0990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f042c09e360 0x7f042c139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-c con 0x7f042c09e360 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042c09d980 con 0x7f042c0a5f90 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042c09d780 con 0x7f042c09e360 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c09fc70 con 0x7f042c0990b0 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient(hunting): authenticate will time out at 1700.596575s 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39372/0 (socket says 172.21.15.142:39372) 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 -- 172.21.15.142:0/1468466653 learned_addr learned my addr 172.21.15.142:0/1468466653 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): get_auth_request con 0x7f042c0990b0 auth_method 0 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042b7fe700 1 --2- 172.21.15.142:0/1468466653 >> v2:172.21.15.150:3300/0 conn(0x7f042c09e360 0x7f042c139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042b7fe700 10 monclient(hunting): get_auth_request con 0x7f042c09e360 auth_method 0 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient(hunting): handle_auth_done global_id 75389 payload 274 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 -- 172.21.15.142:0/1468466653 >> v2:172.21.15.150:3300/0 conn(0x7f042c09e360 msgr2=0x7f042c139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 --2- 172.21.15.142:0/1468466653 >> v2:172.21.15.150:3300/0 conn(0x7f042c09e360 0x7f042c139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 -- 172.21.15.142:0/1468466653 >> v1:172.21.15.150:6789/0 conn(0x7f042c0a5f90 legacy=0x7f042c0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 monclient: found mon.noname-b 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:31.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 -- 172.21.15.142:0/1468466653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c0a6360 con 0x7f042c0990b0 2024-09-18T10:23:31.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 -- 172.21.15.142:0/1468466653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 msgr2=0x7f042c0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f042a7fc700 1 --2- 172.21.15.142:0/1468466653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f041c0099b0 tx=0x7f041c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8caa4987cd2238 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 1 -- 172.21.15.142:0/1468466653 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00ba70 con 0x7f042c0990b0 2024-09-18T10:23:31.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: dump: 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 1 -- 172.21.15.142:0/1468466653 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c00bbd0 con 0x7f042c0990b0 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 1 -- 172.21.15.142:0/1468466653 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c011420 con 0x7f042c0990b0 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 5 monclient: authenticate success, global_id 75389 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f04316d6700 4 set_mon_vals no callback set 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.670+0000 7f0429ffb700 10 monclient: dump: 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 10 monclient: shutdown 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- 172.21.15.142:0/1468466653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 msgr2=0x7f042c0a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 --2- 172.21.15.142:0/1468466653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f041c0099b0 tx=0x7f041c009250 comp rx=0 tx=0).stop 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- 172.21.15.142:0/1468466653 shutdown_connections 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 --2- 172.21.15.142:0/1468466653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c0990b0 0x7f042c0a0980 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 --2- 172.21.15.142:0/1468466653 >> v2:172.21.15.150:3300/0 conn(0x7f042c09e360 0x7f042c139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- 172.21.15.142:0/1468466653 >> 172.21.15.142:0/1468466653 conn(0x7f042c071680 msgr2=0x7f042c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- 172.21.15.142:0/1468466653 shutdown_connections 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- 172.21.15.142:0/1468466653 wait complete. 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 10 monclient: build_initial_monmap 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 10 monclient: monmap: 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:31.685 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:31.686 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:31.686 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 Processor -- start 2024-09-18T10:23:31.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 1 -- start start 2024-09-18T10:23:31.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.674+0000 7f04316d6700 10 monclient: init 2024-09-18T10:23:31.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:31.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:23:31.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f042c0990b0 0x7f042c141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-b con 0x7f042c0990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:31.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 0x7f042c1421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-c con 0x7f042c09e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): picked mon.noname-a con 0x7f042c0a5f90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c08ea80 con 0x7f042c0a5f90 2024-09-18T10:23:31.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c091740 con 0x7f042c0990b0 2024-09-18T10:23:31.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:31.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c09c0e0 con 0x7f042c09e360 2024-09-18T10:23:31.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:31.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f04316d6700 10 monclient(hunting): authenticate will time out at 1700.604758s 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 0x7f042c1421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 0x7f042c1421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39384/0 (socket says 172.21.15.142:39384) 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33624/0 (socket says 172.21.15.142:33624) 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 1 -- 172.21.15.142:0/2541525005 learned_addr learned my addr 172.21.15.142:0/2541525005 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:31.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): get_auth_request con 0x7f042c09e360 auth_method 0 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): get_auth_request con 0x7f042c0a5f90 auth_method 0 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f042c0990b0 0x7f042c141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): get_auth_request con 0x7f042c0990b0 auth_method 0 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient(hunting): handle_auth_done global_id 164124 payload 824 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 -- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 msgr2=0x7f042c1421a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 0x7f042c1421a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 -- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f042c0990b0 msgr2=0x7f042c141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f042c0990b0 0x7f042c141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.678+0000 7f042b7fe700 1 -- 172.21.15.142:0/2541525005 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0094f0 con 0x7f042c0a5f90 2024-09-18T10:23:31.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f042b7fe700 1 -- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 msgr2=0x7f042c08cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f042b7fe700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0424007f00 tx=0x7f0424009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042400eda0 con 0x7f042c0a5f90 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: dump: 2024-09-18T10:23:31.704 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.705 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.705 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.705 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.705 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.705 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: _finish_auth 0 2024-09-18T10:23:31.706 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 5 monclient: authenticate success, global_id 164124 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _renew_subs 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c090790 con 0x7f042c0a5f90 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0424014420 con 0x7f042c0a5f90 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:31.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04240147e0 con 0x7f042c0a5f90 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 10 monclient: dump: 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:31.708 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04297fa700 4 set_mon_vals no callback set 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _renew_subs 2024-09-18T10:23:31.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c08f5c0 con 0x7f042c0a5f90 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0424014bd0 con 0x7f042c0a5f90 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042c0696a0 con 0x7f042c0a5f90 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 0x7f0418037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:31.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0424049570 con 0x7f042c0a5f90 2024-09-18T10:23:31.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f042affd700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 0x7f0418037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:31.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.682+0000 7f042affd700 10 monclient: get_auth_request con 0x7f0418035ac0 auth_method 0 2024-09-18T10:23:31.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.686+0000 7f042affd700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 0x7f0418037f70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0420003a30 tx=0x7f0420007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:31.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.694+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f042401ecb0 con 0x7f042c0a5f90 2024-09-18T10:23:31.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.694+0000 7f0412ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:31.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.694+0000 7f0412ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:31.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.914+0000 7f04316d6700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:23:31.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.914+0000 7f04316d6700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:23:31.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.914+0000 7f04316d6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:31.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.914+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f042c067950 con 0x7f042c0a5f90 2024-09-18T10:23:31.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f0412ffd700 1 -- 172.21.15.142:0/2541525005 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f042401b310 con 0x7f042c0a5f90 2024-09-18T10:23:31.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f0412ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:23:31.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f0412ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:31.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=2840858195,config=3140514009,health=2464607163,kv=4261215878,logm=651688745}) 2024-09-18T10:23:31.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=1,health=12,kv=8,logm=50} crc {auth=2840858195,config=1228700967,health=2247699913,kv=4261215878,logm=1448665133}) 2024-09-18T10:23:31.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=2840858195,config=3140514009,health=2464607163,kv=4261215878,logm=651688745}) 2024-09-18T10:23:31.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.918+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=306555429,config=3140514009,health=2247699913,kv=4261215878,logm=2001775426}) 2024-09-18T10:23:31.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817486101}) 2024-09-18T10:23:31.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839286335}) 2024-09-18T10:23:31.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817486101}) 2024-09-18T10:23:31.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3840399717}) 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 msgr2=0x7f0418037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 0x7f0418037f70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0420003a30 tx=0x7f0420007ea0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 10 monclient: shutdown 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 msgr2=0x7f042c08cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0424007f00 tx=0x7f0424009850 comp rx=0 tx=0).stop 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 shutdown_connections 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0418035ac0 0x7f0418037f70 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f042c0a5f90 0x7f042c08cef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f042c09e360 0x7f042c1421a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 --2- 172.21.15.142:0/2541525005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f042c0990b0 0x7f042c141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:31.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 >> 172.21.15.142:0/2541525005 conn(0x7f042c071680 msgr2=0x7f042c06ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:31.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.922+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 shutdown_connections 2024-09-18T10:23:31.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f04316d6700 1 -- 172.21.15.142:0/2541525005 wait complete. 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129658139}) 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1653256538}) 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129658139}) 2024-09-18T10:23:31.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.926+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421713785}) 2024-09-18T10:23:31.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359591788}) 2024-09-18T10:23:31.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3048858220}) 2024-09-18T10:23:31.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359591788}) 2024-09-18T10:23:31.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587153929}) 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996107570}) 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082252185}) 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996107570}) 2024-09-18T10:23:31.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147225657}) 2024-09-18T10:23:31.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447572597}) 2024-09-18T10:23:31.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003150168}) 2024-09-18T10:23:31.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447572597}) 2024-09-18T10:23:31.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.930+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110582037}) 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846859877}) 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935009679}) 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846859877}) 2024-09-18T10:23:31.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078454343}) 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890901877}) 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690228715}) 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890901877}) 2024-09-18T10:23:31.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.934+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934309955}) 2024-09-18T10:23:31.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832720670}) 2024-09-18T10:23:31.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752468222}) 2024-09-18T10:23:31.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832720670}) 2024-09-18T10:23:31.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=687687180}) 2024-09-18T10:23:31.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:23:31.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019027439}) 2024-09-18T10:23:31.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691659066}) 2024-09-18T10:23:31.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019027439}) 2024-09-18T10:23:31.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.938+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023961917}) 2024-09-18T10:23:31.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692931165}) 2024-09-18T10:23:31.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947834499}) 2024-09-18T10:23:31.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692931165}) 2024-09-18T10:23:31.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=852679036}) 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105035761}) 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854856733}) 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105035761}) 2024-09-18T10:23:31.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.942+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109291689}) 2024-09-18T10:23:31.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504167207}) 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2760368371}) 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504167207}) 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337714307}) 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632674386}) 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468984234}) 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632674386}) 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782750203}) 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96523189}) 2024-09-18T10:23:31.952 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764824501}) 2024-09-18T10:23:31.952 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.952 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=96523189}) 2024-09-18T10:23:31.952 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987968927}) 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311528139}) 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2349872253}) 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311528139}) 2024-09-18T10:23:31.953 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.946+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210441135}) 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258855392}) 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983211411}) 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258855392}) 2024-09-18T10:23:31.954 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531751896}) 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969747438}) 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899205996}) 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969747438}) 2024-09-18T10:23:31.955 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994910081}) 2024-09-18T10:23:31.956 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.956 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778904620}) 2024-09-18T10:23:31.956 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383379204}) 2024-09-18T10:23:31.956 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.956 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778904620}) 2024-09-18T10:23:31.957 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757398888}) 2024-09-18T10:23:31.958 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.958 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195092099}) 2024-09-18T10:23:31.958 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796438545}) 2024-09-18T10:23:31.958 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195092099}) 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.950+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244111851}) 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592422594}) 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=456848519}) 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592422594}) 2024-09-18T10:23:31.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975402274}) 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490797839}) 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1103727961}) 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490797839}) 2024-09-18T10:23:31.960 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3388528460}) 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435901847}) 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579864745}) 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3435901847}) 2024-09-18T10:23:31.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.954+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440566079}) 2024-09-18T10:23:31.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279065696}) 2024-09-18T10:23:31.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374007705}) 2024-09-18T10:23:31.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279065696}) 2024-09-18T10:23:31.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786394272}) 2024-09-18T10:23:31.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836124765}) 2024-09-18T10:23:31.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=183527236}) 2024-09-18T10:23:31.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836124765}) 2024-09-18T10:23:31.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231117448}) 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1219213348}) 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698443352}) 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1219213348}) 2024-09-18T10:23:31.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.958+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=46858856}) 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860051804}) 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1486780570}) 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860051804}) 2024-09-18T10:23:31.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.962+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1185173792}) 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673359328}) 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071941996}) 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673359328}) 2024-09-18T10:23:31.977 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.970+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=426898736}) 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706763013}) 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3383643430}) 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706763013}) 2024-09-18T10:23:31.978 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964918373}) 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648846229}) 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977600585}) 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648846229}) 2024-09-18T10:23:31.979 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273090054}) 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685801099}) 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614408846}) 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685801099}) 2024-09-18T10:23:31.981 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.974+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318434650}) 2024-09-18T10:23:31.982 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352917964}) 2024-09-18T10:23:31.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256581539}) 2024-09-18T10:23:31.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352917964}) 2024-09-18T10:23:31.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115682270}) 2024-09-18T10:23:31.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996151595}) 2024-09-18T10:23:31.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776142923}) 2024-09-18T10:23:31.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996151595}) 2024-09-18T10:23:31.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.978+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=103330308}) 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063849755}) 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366120696}) 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063849755}) 2024-09-18T10:23:31.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132537857}) 2024-09-18T10:23:31.987 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146250687}) 2024-09-18T10:23:31.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956900}) 2024-09-18T10:23:31.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146250687}) 2024-09-18T10:23:31.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4103120088}) 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903927381}) 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270623471}) 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1903927381}) 2024-09-18T10:23:31.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.982+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1366513504}) 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511710935}) 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700383600}) 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511710935}) 2024-09-18T10:23:31.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214980736}) 2024-09-18T10:23:31.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315738411}) 2024-09-18T10:23:31.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317800715}) 2024-09-18T10:23:31.991 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=315738411}) 2024-09-18T10:23:31.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554941260}) 2024-09-18T10:23:31.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601467829}) 2024-09-18T10:23:31.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705406108}) 2024-09-18T10:23:31.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601467829}) 2024-09-18T10:23:31.993 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.986+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251113797}) 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443054648}) 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702778139}) 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443054648}) 2024-09-18T10:23:31.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345930991}) 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613262007}) 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397672220}) 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613262007}) 2024-09-18T10:23:31.995 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999177314}) 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236784932}) 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=492413766}) 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236784932}) 2024-09-18T10:23:31.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.990+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078065241}) 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9586313}) 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3050021129}) 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9586313}) 2024-09-18T10:23:32.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:31.998+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335213268}) 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528745354}) 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1698399553}) 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528745354}) 2024-09-18T10:23:32.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.002+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1761023171}) 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46820255}) 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3038677280}) 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46820255}) 2024-09-18T10:23:32.703 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545812980}) 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243436087}) 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268937740}) 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243436087}) 2024-09-18T10:23:32.705 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.698+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=892679313}) 2024-09-18T10:23:32.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2185657219}) 2024-09-18T10:23:32.707 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556858438}) 2024-09-18T10:23:32.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2185657219}) 2024-09-18T10:23:32.708 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.702+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734573005}) 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081670632}) 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2872409266}) 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081670632}) 2024-09-18T10:23:32.710 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2234038336}) 2024-09-18T10:23:32.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817738989}) 2024-09-18T10:23:32.712 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154009119}) 2024-09-18T10:23:32.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817738989}) 2024-09-18T10:23:32.713 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.706+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415832976}) 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733844245}) 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445735062}) 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733844245}) 2024-09-18T10:23:32.715 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396139002}) 2024-09-18T10:23:32.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850258}) 2024-09-18T10:23:32.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702423170}) 2024-09-18T10:23:32.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3850258}) 2024-09-18T10:23:32.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.710+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485721618}) 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219833806}) 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2496431619}) 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219833806}) 2024-09-18T10:23:32.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972819602}) 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527529027}) 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4063089825}) 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527529027}) 2024-09-18T10:23:32.720 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2468949677}) 2024-09-18T10:23:32.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110441377}) 2024-09-18T10:23:32.721 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210793480}) 2024-09-18T10:23:32.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110441377}) 2024-09-18T10:23:32.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.714+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789691904}) 2024-09-18T10:23:32.722 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248057258}) 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=436033934}) 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248057258}) 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147434430}) 2024-09-18T10:23:32.723 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984254310}) 2024-09-18T10:23:32.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650576685}) 2024-09-18T10:23:32.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984254310}) 2024-09-18T10:23:32.724 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.718+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760744225}) 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742264489}) 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220082282}) 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742264489}) 2024-09-18T10:23:32.728 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.722+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534701630}) 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760147363}) 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806674104}) 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760147363}) 2024-09-18T10:23:32.752 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544076592}) 2024-09-18T10:23:32.753 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673493331}) 2024-09-18T10:23:32.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.746+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=833027195}) 2024-09-18T10:23:32.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673493331}) 2024-09-18T10:23:32.754 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540902638}) 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533403742}) 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=613300790}) 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533403742}) 2024-09-18T10:23:32.755 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040236298}) 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001987674}) 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953620512}) 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001987674}) 2024-09-18T10:23:32.757 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.750+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578866707}) 2024-09-18T10:23:32.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475754294}) 2024-09-18T10:23:32.758 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845494081}) 2024-09-18T10:23:32.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=475754294}) 2024-09-18T10:23:32.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096712574}) 2024-09-18T10:23:32.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665023267}) 2024-09-18T10:23:32.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=963115895}) 2024-09-18T10:23:32.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.760 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665023267}) 2024-09-18T10:23:32.761 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.754+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243598255}) 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306254759}) 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384255729}) 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=306254759}) 2024-09-18T10:23:32.762 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=68161605}) 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577816294}) 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585906657}) 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577816294}) 2024-09-18T10:23:32.764 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345019807}) 2024-09-18T10:23:32.765 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.765 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155595848}) 2024-09-18T10:23:32.766 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1731769557}) 2024-09-18T10:23:32.766 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.766 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155595848}) 2024-09-18T10:23:32.766 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.758+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829196}) 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204910414}) 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=474155840}) 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204910414}) 2024-09-18T10:23:32.767 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2050743490}) 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874079869}) 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2185137970}) 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874079869}) 2024-09-18T10:23:32.768 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.762+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=886909399}) 2024-09-18T10:23:32.769 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.769 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960932517}) 2024-09-18T10:23:32.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=110881793}) 2024-09-18T10:23:32.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960932517}) 2024-09-18T10:23:32.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926187293}) 2024-09-18T10:23:32.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124005006}) 2024-09-18T10:23:32.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696708576}) 2024-09-18T10:23:32.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124005006}) 2024-09-18T10:23:32.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900723265}) 2024-09-18T10:23:32.772 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.772 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467539416}) 2024-09-18T10:23:32.772 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044935085}) 2024-09-18T10:23:32.772 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.773 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467539416}) 2024-09-18T10:23:32.773 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029300630}) 2024-09-18T10:23:32.773 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.773 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256080584}) 2024-09-18T10:23:32.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910321507}) 2024-09-18T10:23:32.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256080584}) 2024-09-18T10:23:32.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.766+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874101204}) 2024-09-18T10:23:32.776 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.776 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496316449}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984568222}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496316449}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626743285}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630453219}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3871522293}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630453219}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466237953}) 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=548576771}) 2024-09-18T10:23:32.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182228619}) 2024-09-18T10:23:32.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=548576771}) 2024-09-18T10:23:32.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.770+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045718803}) 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=2112351866,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3188546452,mon_config_key=3383142877,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=668940925}) 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=2112351866,mdsmap=1531151339,mgr=1051400072,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2571379053,mon_config_key=620546692,monmap=3505902565,osd_metadata=2166527243,osd_pg_creating=69081760,osd_snap=454132730,osdmap=282725898}) 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {logm=2112351866,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3188546452,mon_config_key=3383142877,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=668940925}) 2024-09-18T10:23:32.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:23:32.782+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {logm=3605876395,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3272508607,mon_config_key=620546692,monmap=3505902565,osd_metadata=4002491971,osd_pg_creating=69081760,osd_snap=454132730,osdmap=2717758269}) 2024-09-18T10:23:32.945 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:23:32.945 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.046+0000 7f4f82243700 10 monclient: get_monmap_and_config 2024-09-18T10:23:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.046+0000 7f4f82243700 10 monclient: build_initial_monmap 2024-09-18T10:23:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.046+0000 7f4f82243700 10 monclient: monmap: 2024-09-18T10:23:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:33.051439+0000 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:33.051439+0000 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:33.055 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:33.055 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 Processor -- start 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 -- start start 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient: init 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:23:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f7c0990b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 0x7f4f7c139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c09e360 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7c09d980 con 0x7f4f7c0990b0 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7c09d780 con 0x7f4f7c09e360 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c09fc70 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f82243700 10 monclient(hunting): authenticate will time out at 1701.977293s 2024-09-18T10:23:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 0x7f4f7c139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f72ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4f7c0990b0 0x7f4f7c0a07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38088/0 (socket says 172.21.15.142:38088) 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40528/0 (socket says 172.21.15.142:40528) 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f72ffd700 1 -- 172.21.15.142:0/2414947605 learned_addr learned my addr 172.21.15.142:0/2414947605 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): get_auth_request con 0x7f4f7c09e360 auth_method 0 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): get_auth_request con 0x7f4f7c0a5f90 auth_method 0 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_done global_id 75395 payload 274 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 -- 172.21.15.142:0/2414947605 >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 msgr2=0x7f4f7c139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 --2- 172.21.15.142:0/2414947605 >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 0x7f4f7c139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 -- 172.21.15.142:0/2414947605 >> v1:172.21.15.142:6789/0 conn(0x7f4f7c0990b0 legacy=0x7f4f7c0a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 monclient: found mon.noname-b 2024-09-18T10:23:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7b7fe700 1 -- 172.21.15.142:0/2414947605 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0a6360 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.050+0000 7f4f7bfff700 1 --2- 172.21.15.142:0/2414947605 >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 0x7f4f7c139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:33.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7b7fe700 1 -- 172.21.15.142:0/2414947605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 msgr2=0x7f4f7c0a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:33.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7b7fe700 1 --2- 172.21.15.142:0/2414947605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0098e0 tx=0x7f4f6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4eaffb26cc8f774 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:33.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 1 -- 172.21.15.142:0/2414947605 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c00b850 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: dump: 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:33.069 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: _finish_auth 0 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 5 monclient: authenticate success, global_id 75395 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 1 -- 172.21.15.142:0/2414947605 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c00b9b0 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 4 set_mon_vals no callback set 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 1 -- 172.21.15.142:0/2414947605 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c00bd50 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f7affd700 10 monclient: dump: 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:33.071 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 10 monclient: shutdown 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 -- 172.21.15.142:0/2414947605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 msgr2=0x7f4f7c0a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 --2- 172.21.15.142:0/2414947605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0098e0 tx=0x7f4f6c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 -- 172.21.15.142:0/2414947605 shutdown_connections 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 --2- 172.21.15.142:0/2414947605 >> v2:172.21.15.142:3300/0 conn(0x7f4f7c09e360 0x7f4f7c139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 --2- 172.21.15.142:0/2414947605 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c0a5f90 0x7f4f7c0a0270 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 -- 172.21.15.142:0/2414947605 >> 172.21.15.142:0/2414947605 conn(0x7f4f7c071680 msgr2=0x7f4f7c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 -- 172.21.15.142:0/2414947605 shutdown_connections 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 1 -- 172.21.15.142:0/2414947605 wait complete. 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 10 monclient: build_initial_monmap 2024-09-18T10:23:33.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.054+0000 7f4f82243700 10 monclient: monmap: 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 Processor -- start 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 -- start start 2024-09-18T10:23:33.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient: init 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0990b0 0x7f4f7c141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c0990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c09e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c0a5f90 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c090760 con 0x7f4f7c0a5f90 2024-09-18T10:23:33.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c08eed0 con 0x7f4f7c0990b0 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c08f590 con 0x7f4f7c09e360 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f82243700 10 monclient(hunting): authenticate will time out at 1701.986273s 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56500/0 (socket says 172.21.15.142:56500) 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 1 -- 172.21.15.142:0/3492918462 learned_addr learned my addr 172.21.15.142:0/3492918462 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): get_auth_request con 0x7f4f7c0a5f90 auth_method 0 2024-09-18T10:23:33.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): get_auth_request con 0x7f4f7c09e360 auth_method 0 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:33.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0990b0 0x7f4f7c141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): get_auth_request con 0x7f4f7c0990b0 auth_method 0 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:33.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient(hunting): handle_auth_done global_id 75398 payload 824 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 -- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0990b0 msgr2=0x7f4f7c141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0990b0 0x7f4f7c141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 -- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 msgr2=0x7f4f7c08d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:23:33.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 monclient: found mon.noname-c 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f72ffd700 1 -- 172.21.15.142:0/3492918462 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f74009a20 con 0x7f4f7c09e360 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.058+0000 7f4f7bfff700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f72ffd700 1 -- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 msgr2=0x7f4f7c142230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f72ffd700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400c530 tx=0x7f4f64007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400ce80 con 0x7f4f7c09e360 2024-09-18T10:23:33.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: dump: 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:33.081 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: _finish_auth 0 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 5 monclient: authenticate success, global_id 75398 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f64005d40 con 0x7f4f7c09e360 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _renew_subs 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c009420 con 0x7f4f7c09e360 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64007320 con 0x7f4f7c09e360 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:33.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f7a7fc700 4 set_mon_vals no callback set 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 10 monclient: dump: 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:33.083 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:33.084 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:33.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _renew_subs 2024-09-18T10:23:33.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:33.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c08e9c0 con 0x7f4f7c09e360 2024-09-18T10:23:33.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f72ffd700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400c530 tx=0x7f4f64007f90 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f4f64048340 osd_map(264..264 src has 239..264) v4 2024-09-18T10:23:33.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4f6401b3b0 con 0x7f4f7c09e360 2024-09-18T10:23:33.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:33.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.062+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c0696a0 con 0x7f4f7c09e360 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.066+0000 7f4f73fff700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 0x7f4f5c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.066+0000 7f4f7b7fe700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 0x7f4f5c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.066+0000 7f4f7b7fe700 10 monclient: get_auth_request con 0x7f4f5c035ac0 auth_method 0 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.066+0000 7f4f7b7fe700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 0x7f4f5c037f70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0030c0 tx=0x7f4f6c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:33.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.070+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4f64048340 con 0x7f4f7c09e360 2024-09-18T10:23:33.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.890+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4f640196e0 con 0x7f4f7c09e360 2024-09-18T10:23:33.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.890+0000 7f4f73fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:33.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:33.890+0000 7f4f73fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:34.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f82243700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:34.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f82243700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:34.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f82243700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4f7c090920 con 0x7f4f7c09e360 2024-09-18T10:23:34.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f73fff700 1 -- 172.21.15.142:0/3492918462 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4f64017070 con 0x7f4f7c09e360 2024-09-18T10:23:34.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f73fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:34.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.122+0000 7f4f73fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:34.129 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:34.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 msgr2=0x7f4f5c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 0x7f4f5c037f70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c0030c0 tx=0x7f4f6c016000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 10 monclient: shutdown 2024-09-18T10:23:34.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 msgr2=0x7f4f7c142230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400c530 tx=0x7f4f64007f90 comp rx=0 tx=0).stop 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 shutdown_connections 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4f5c035ac0 0x7f4f5c037f70 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4f7c0a5f90 0x7f4f7c08d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4f7c09e360 0x7f4f7c142230 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 --2- 172.21.15.142:0/3492918462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0990b0 0x7f4f7c141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 >> 172.21.15.142:0/3492918462 conn(0x7f4f7c071680 msgr2=0x7f4f7c06ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 shutdown_connections 2024-09-18T10:23:34.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.126+0000 7f4f82243700 1 -- 172.21.15.142:0/3492918462 wait complete. 2024-09-18T10:23:34.142 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:23:34.142 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:23:34.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:23:34.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 10 monclient: get_monmap_and_config 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 10 monclient: monmap: 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:34.246776+0000 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:34.246776+0000 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.251 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 1 Processor -- start 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 1 -- start start 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.242+0000 7f63fdc3f700 10 monclient: init 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 0x7f63f80954b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-c con 0x7f63f809d870 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:34.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63f80989c0 0x7f63f8098dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-a con 0x7f63f80989c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f8099300 0x7f63f8135670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-b con 0x7f63f8099300 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f806f9d0 con 0x7f63f80989c0 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f8069c70 con 0x7f63f809d870 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f809dc40 con 0x7f63f8099300 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient(hunting): authenticate will time out at 1703.172266s 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63f80989c0 0x7f63f8098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f8099300 0x7f63f8135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f63f80989c0 0x7f63f8098dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56516/0 (socket says 172.21.15.142:56516) 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 -- 172.21.15.142:0/1996528113 learned_addr learned my addr 172.21.15.142:0/1996528113 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): get_auth_request con 0x7f63f80989c0 auth_method 0 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): get_auth_request con 0x7f63f8099300 auth_method 0 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 1 --2- 172.21.15.142:0/1996528113 >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 0x7f63f80954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 10 monclient(hunting): get_auth_request con 0x7f63f809d870 auth_method 0 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_done global_id 164133 payload 274 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 -- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f8099300 msgr2=0x7f63f8135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f8099300 0x7f63f8135670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 -- 172.21.15.142:0/1996528113 >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 msgr2=0x7f63f80954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/1996528113 >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 0x7f63f80954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 -- 172.21.15.142:0/1996528113 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f806fc20 con 0x7f63f80989c0 2024-09-18T10:23:34.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f77fe700 1 --2- 172.21.15.142:0/1996528113 >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 0x7f63f80954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 -- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f80989c0 msgr2=0x7f63f8098dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f80989c0 0x7f63f8098dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f63f8076e90 tx=0x7f63e800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2cc400ad3fa771 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 1 -- 172.21.15.142:0/1996528113 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8010070 con 0x7f63f80989c0 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: dump: 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.261 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 5 monclient: authenticate success, global_id 164133 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 1 -- 172.21.15.142:0/1996528113 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e800b0d0 con 0x7f63f80989c0 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63fdc3f700 4 set_mon_vals no callback set 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 1 -- 172.21.15.142:0/1996528113 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800b490 con 0x7f63f80989c0 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.246+0000 7f63f67fc700 10 monclient: dump: 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: shutdown 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f80989c0 msgr2=0x7f63f8098dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f80989c0 0x7f63f8098dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f63f8076e90 tx=0x7f63e800ba50 comp rx=0 tx=0).stop 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- 172.21.15.142:0/1996528113 shutdown_connections 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f8099300 0x7f63f8135670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/1996528113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f80989c0 0x7f63f8098dc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/1996528113 >> v2:172.21.15.150:3300/0 conn(0x7f63f809d870 0x7f63f80954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- 172.21.15.142:0/1996528113 >> 172.21.15.142:0/1996528113 conn(0x7f63f80a14e0 msgr2=0x7f63f80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- 172.21.15.142:0/1996528113 shutdown_connections 2024-09-18T10:23:34.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- 172.21.15.142:0/1996528113 wait complete. 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: monmap: 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 Processor -- start 2024-09-18T10:23:34.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- start start 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: init 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f80989c0 0x7f63f8089980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-b con 0x7f63f80989c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-a con 0x7f63f8099300 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f809d870 0x7f63f808a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): picked mon.noname-c con 0x7f63f809d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f808d0f0 con 0x7f63f8099300 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8097bf0 con 0x7f63f80989c0 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8097180 con 0x7f63f809d870 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63fdc3f700 10 monclient(hunting): authenticate will time out at 1703.179364s 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f809d870 0x7f63f808a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f809d870 0x7f63f808a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40552/0 (socket says 172.21.15.142:40552) 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56524/0 (socket says 172.21.15.142:56524) 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 1 -- 172.21.15.142:0/3781132184 learned_addr learned my addr 172.21.15.142:0/3781132184 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): get_auth_request con 0x7f63f809d870 auth_method 0 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): get_auth_request con 0x7f63f8099300 auth_method 0 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f80989c0 0x7f63f8089980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): get_auth_request con 0x7f63f80989c0 auth_method 0 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.250+0000 7f63f6ffd700 10 monclient(hunting): handle_auth_done global_id 164136 payload 824 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f809d870 msgr2=0x7f63f808a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63f809d870 0x7f63f808a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f80989c0 msgr2=0x7f63f8089980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63f80989c0 0x7f63f8089980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e800bcf0 con 0x7f63f8099300 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 msgr2=0x7f63f8089ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63e8017000 tx=0x7f63e801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800ad50 con 0x7f63f8099300 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: dump: 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 5 monclient: authenticate success, global_id 164136 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8028460 con 0x7f63f8099300 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 10 monclient: _renew_subs 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f808bf20 con 0x7f63f8099300 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8028820 con 0x7f63f8099300 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63df7fe700 10 monclient: dump: 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63f5ffb700 4 set_mon_vals no callback set 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 10 monclient: _renew_subs 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.254+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f808b370 con 0x7f63f8099300 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f6ffd700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63e8017000 tx=0x7f63e801a000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f63e8003040 mgrmap(e 5) v1 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63fdc3f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f63e8003040 con 0x7f63f8099300 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 0x7f63f806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f808d490 con 0x7f63f808a940 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 0x7f63f806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient: get_auth_request con 0x7f63f808a940 auth_method 0 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63e4035b10 0x7f63e4037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 10 monclient(hunting): handle_auth_done global_id 164139 payload 824 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63f80696a0 con 0x7f63f808a940 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f7fff700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63e4035b10 0x7f63e4037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f7fff700 10 monclient: get_auth_request con 0x7f63e4035b10 auth_method 0 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 msgr2=0x7f63f806d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f77fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 0x7f63f806d410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63ec006fd0 tx=0x7f63ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00fa00 con 0x7f63f808a940 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 10 monclient: dump: 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.278 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f63ec00fb60 con 0x7f63f808a940 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 msgr2=0x7f63f806d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 0x7f63f806d410 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63ec006fd0 tx=0x7f63ec008040 comp rx=0 tx=0).stop 2024-09-18T10:23:34.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 msgr2=0x7f63f806d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63df7fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f808a940 0x7f63f806d410 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f63ec006fd0 tx=0x7f63ec008040 comp rx=0 tx=0).stop 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.258+0000 7f63f7fff700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63e4035b10 0x7f63e4037fc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009eb0 tx=0x7f63f00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63fdc3f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 0x7f63f806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63fdc3f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f8064950 con 0x7f63f806d950 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 0x7f63f806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient: get_auth_request con 0x7f63f806d950 auth_method 0 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 10 monclient(hunting): handle_auth_done global_id 164142 payload 824 2024-09-18T10:23:34.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.270+0000 7f63f77fe700 1 -- 172.21.15.142:0/3781132184 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f63f808ad10 con 0x7f63f806d950 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63f77fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 msgr2=0x7f63f806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63f77fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 0x7f63f806dd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec01a000 tx=0x7f63ec01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec015ab0 con 0x7f63f806d950 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 10 monclient: dump: 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f63ec015c10 con 0x7f63f806d950 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 msgr2=0x7f63f806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 0x7f63f806dd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec01a000 tx=0x7f63ec01d000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 msgr2=0x7f63f806dd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f806d950 0x7f63f806dd50 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f63ec01a000 tx=0x7f63ec01d000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:34.284 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.285 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:34.287 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.288 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:34.289 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.291 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:34.292 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.293 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.294 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:34.295 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:34.296 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63e4035b10 msgr2=0x7f63e4037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63e4035b10 0x7f63e4037fc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009eb0 tx=0x7f63f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 10 monclient: shutdown 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 msgr2=0x7f63f8089ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63f7fff700 1 -- 172.21.15.142:0/3781132184 reap_dead start 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f63e8017000 tx=0x7f63e801a000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63df7fe700 1 -- 172.21.15.142:0/3781132184 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f63e804ce10 con 0x7f63f8099300 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 shutdown_connections 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 --2- 172.21.15.142:0/3781132184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63f8099300 0x7f63f8089ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 >> 172.21.15.142:0/3781132184 conn(0x7f63f80a14e0 msgr2=0x7f63f8095a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 shutdown_connections 2024-09-18T10:23:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.274+0000 7f63fdc3f700 1 -- 172.21.15.142:0/3781132184 wait complete. 2024-09-18T10:23:34.299 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:34.299 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:23:34.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:23:34.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: get_monmap_and_config 2024-09-18T10:23:34.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: monmap: 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:34.408393+0000 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:34.408393+0000 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:34.411 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:34.412 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:34.412 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 Processor -- start 2024-09-18T10:23:34.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 -- start start 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: init 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e7006f3b0 0x7f6e700989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-c con 0x7f6e7006f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6e70098f00 0x7f6e70099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-a con 0x7f6e70098f00 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e70133660 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7009c330 con 0x7f6e70133660 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e70099860 con 0x7f6e70098f00 2024-09-18T10:23:34.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7009b030 con 0x7f6e7006f3b0 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e75773700 10 monclient(hunting): authenticate will time out at 1703.334271s 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6e70098f00 0x7f6e70099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6e70098f00 0x7f6e70099320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56556/0 (socket says 172.21.15.142:56556) 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 -- 172.21.15.142:0/925274854 learned_addr learned my addr 172.21.15.142:0/925274854 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): get_auth_request con 0x7f6e70098f00 auth_method 0 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/925274854 >> v2:172.21.15.150:3300/0 conn(0x7f6e7006f3b0 0x7f6e700989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request con 0x7f6e7006f3b0 auth_method 0 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient(hunting): handle_auth_done global_id 164145 payload 274 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 -- 172.21.15.142:0/925274854 >> v2:172.21.15.150:3300/0 conn(0x7f6e7006f3b0 msgr2=0x7f6e700989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 --2- 172.21.15.142:0/925274854 >> v2:172.21.15.150:3300/0 conn(0x7f6e7006f3b0 0x7f6e700989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 -- 172.21.15.142:0/925274854 >> v1:172.21.15.150:6789/0 conn(0x7f6e70133660 legacy=0x7f6e70135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 monclient: found mon.noname-a 2024-09-18T10:23:34.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.406+0000 7f6e67fff700 1 -- 172.21.15.142:0/925274854 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7009c530 con 0x7f6e70098f00 2024-09-18T10:23:34.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e67fff700 1 -- 172.21.15.142:0/925274854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 msgr2=0x7f6e70099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e67fff700 1 --2- 172.21.15.142:0/925274854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e70099320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6e70076de0 tx=0x7f6e58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de45f70e8da98bd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 1 -- 172.21.15.142:0/925274854 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800ba70 con 0x7f6e70098f00 2024-09-18T10:23:34.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:34.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: dump: 2024-09-18T10:23:34.427 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 1 -- 172.21.15.142:0/925274854 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5800bbd0 con 0x7f6e70098f00 2024-09-18T10:23:34.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 5 monclient: authenticate success, global_id 164145 2024-09-18T10:23:34.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:34.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 1 -- 172.21.15.142:0/925274854 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58011420 con 0x7f6e70098f00 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 4 set_mon_vals no callback set 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e6f7fe700 10 monclient: dump: 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 10 monclient: shutdown 2024-09-18T10:23:34.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 -- 172.21.15.142:0/925274854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 msgr2=0x7f6e70099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 --2- 172.21.15.142:0/925274854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e70099320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6e70076de0 tx=0x7f6e58009250 comp rx=0 tx=0).stop 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 -- 172.21.15.142:0/925274854 shutdown_connections 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 --2- 172.21.15.142:0/925274854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e70099320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 --2- 172.21.15.142:0/925274854 >> v2:172.21.15.150:3300/0 conn(0x7f6e7006f3b0 0x7f6e700989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 -- 172.21.15.142:0/925274854 >> 172.21.15.142:0/925274854 conn(0x7f6e70070ef0 msgr2=0x7f6e700712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 -- 172.21.15.142:0/925274854 shutdown_connections 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 1 -- 172.21.15.142:0/925274854 wait complete. 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.410+0000 7f6e75773700 10 monclient: monmap: 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.434 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:34.435 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:34.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:34.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 Processor -- start 2024-09-18T10:23:34.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 -- start start 2024-09-18T10:23:34.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient: init 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e7006f3b0 0x7f6e7013f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-b con 0x7f6e7006f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e701319c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-a con 0x7f6e70098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:34.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 0x7f6e7006c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): picked mon.noname-c con 0x7f6e70133660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7006fa20 con 0x7f6e70098f00 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7009eb10 con 0x7f6e7006f3b0 2024-09-18T10:23:34.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7013faf0 con 0x7f6e70133660 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e75773700 10 monclient(hunting): authenticate will time out at 1703.341287s 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 0x7f6e7006c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e701319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 0x7f6e7006c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40568/0 (socket says 172.21.15.142:40568) 2024-09-18T10:23:34.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 learned_addr learned my addr 172.21.15.142:0/1330536005 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): get_auth_request con 0x7f6e70133660 auth_method 0 2024-09-18T10:23:34.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): get_auth_request con 0x7f6e70098f00 auth_method 0 2024-09-18T10:23:34.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e7006f3b0 0x7f6e7013f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request con 0x7f6e7006f3b0 auth_method 0 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient(hunting): handle_auth_done global_id 75407 payload 824 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e7006f3b0 msgr2=0x7f6e7013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e7006f3b0 0x7f6e7013f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 msgr2=0x7f6e701319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e701319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 monclient: found mon.noname-c 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e580094f0 con 0x7f6e70133660 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.414+0000 7f6e67fff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e70098f00 0x7f6e701319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 msgr2=0x7f6e7006c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e74f72700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 0x7f6e7006c4b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c890 tx=0x7f6e680026a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68012840 con 0x7f6e70133660 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: dump: 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.446 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e680129c0 con 0x7f6e70133660 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 5 monclient: authenticate success, global_id 75407 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 monclient: _renew_subs 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70140410 con 0x7f6e70133660 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68012d80 con 0x7f6e70133660 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 10 monclient: dump: 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.449 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6effd700 4 set_mon_vals no callback set 2024-09-18T10:23:34.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 monclient: _renew_subs 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e70127120 con 0x7f6e70133660 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 0x7f6e7006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700648a0 con 0x7f6e70131f00 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 0x7f6e7006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient: get_auth_request con 0x7f6e70131f00 auth_method 0 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_done global_id 75410 payload 824 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6ffff700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e7006e600 con 0x7f6e70131f00 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6e68011590 con 0x7f6e70133660 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6e50035ac0 0x7f6e50037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.418+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6e68049240 con 0x7f6e70133660 2024-09-18T10:23:34.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e67fff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6e50035ac0 0x7f6e50037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e67fff700 10 monclient: get_auth_request con 0x7f6e50035ac0 auth_method 0 2024-09-18T10:23:34.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6ffff700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 msgr2=0x7f6e7006c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 0x7f6e7006c9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6e60010b80 tx=0x7f6e60008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6000eb70 con 0x7f6e70131f00 2024-09-18T10:23:34.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 10 monclient: dump: 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6e6000ecd0 con 0x7f6e70131f00 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 msgr2=0x7f6e7006c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 0x7f6e7006c9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6e60010b80 tx=0x7f6e60008040 comp rx=0 tx=0).stop 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 msgr2=0x7f6e7006c9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e6cff9700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70131f00 0x7f6e7006c9f0 secure :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6e60010b80 tx=0x7f6e60008040 comp rx=0 tx=0).stop 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.422+0000 7f6e67fff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6e50035ac0 0x7f6e50037f70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6e70076de0 tx=0x7f6e58016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e75773700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e75773700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 0x7f6e7006a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e75773700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7006d330 con 0x7f6e7006cf30 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 0x7f6e7006a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient: get_auth_request con 0x7f6e7006cf30 auth_method 0 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 10 monclient(hunting): handle_auth_done global_id 75413 payload 824 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.430+0000 7f6e6ffff700 1 -- 172.21.15.142:0/1330536005 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e70072a80 con 0x7f6e7006cf30 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6ffff700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 msgr2=0x7f6e7006a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6ffff700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 0x7f6e7006a360 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6e6001a000 tx=0x7f6e6001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6002b070 con 0x7f6e7006cf30 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 10 monclient: dump: 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f6e60023040 con 0x7f6e7006cf30 2024-09-18T10:23:34.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 msgr2=0x7f6e7006a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 0x7f6e7006a360 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6e6001a000 tx=0x7f6e6001d000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 msgr2=0x7f6e7006a360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e6cff9700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e7006cf30 0x7f6e7006a360 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:34.460 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.461 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.462 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:34.463 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.464 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:34.465 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.466 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.467 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.468 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.469 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.470 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.471 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.472 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.473 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:34.473 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:34.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6e50035ac0 msgr2=0x7f6e50037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6e50035ac0 0x7f6e50037f70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6e70076de0 tx=0x7f6e58016000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 10 monclient: shutdown 2024-09-18T10:23:34.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 msgr2=0x7f6e7006c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 --2- 172.21.15.142:0/1330536005 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6e70133660 0x7f6e7006c4b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c890 tx=0x7f6e680026a0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 reap_dead start 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e74f72700 1 -- 172.21.15.142:0/1330536005 reap_dead start 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 shutdown_connections 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.434+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 >> 172.21.15.142:0/1330536005 conn(0x7f6e70070ef0 msgr2=0x7f6e70132880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.438+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 shutdown_connections 2024-09-18T10:23:34.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.438+0000 7f6e75773700 1 -- 172.21.15.142:0/1330536005 wait complete. 2024-09-18T10:23:34.475 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:34.475 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:23:34.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:23:34.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 10 monclient: get_monmap_and_config 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 10 monclient: monmap: 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:34.589763+0000 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:34.589763+0000 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:34.592 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:34.593 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:34.593 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:34.593 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:34.593 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 1 Processor -- start 2024-09-18T10:23:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 1 -- start start 2024-09-18T10:23:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.586+0000 7fd57332a700 10 monclient: init 2024-09-18T10:23:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-b con 0x7fd56c09cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd56c0a02e0 0x7fd56c0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-a con 0x7fd56c0a02e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd56c0a0bf0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56c09d1a0 con 0x7fd56c0a0bf0 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56c09e980 con 0x7fd56c0a02e0 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c09d780 con 0x7fd56c09cdd0 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient(hunting): authenticate will time out at 1703.516262s 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40578/0 (socket says 172.21.15.142:40578) 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 -- 172.21.15.142:0/2646170772 learned_addr learned my addr 172.21.15.142:0/2646170772 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd56c0a02e0 0x7fd56c0a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): get_auth_request con 0x7fd56c09cdd0 auth_method 0 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): get_auth_request con 0x7fd56c0a02e0 auth_method 0 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient(hunting): handle_auth_done global_id 75416 payload 274 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 -- 172.21.15.142:0/2646170772 >> v2:172.21.15.142:3300/0 conn(0x7fd56c0a02e0 msgr2=0x7fd56c0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 --2- 172.21.15.142:0/2646170772 >> v2:172.21.15.142:3300/0 conn(0x7fd56c0a02e0 0x7fd56c0a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 -- 172.21.15.142:0/2646170772 >> v1:172.21.15.142:6789/0 conn(0x7fd56c0a0bf0 legacy=0x7fd56c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 monclient: found mon.noname-b 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 -- 172.21.15.142:0/2646170772 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c09dd40 con 0x7fd56c09cdd0 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 -- 172.21.15.142:0/2646170772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 msgr2=0x7fd56c09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd5710c6700 1 --2- 172.21.15.142:0/2646170772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd5680099b0 tx=0x7fd568009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2151dd4f05ea0b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 1 -- 172.21.15.142:0/2646170772 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568009ea0 con 0x7fd56c09cdd0 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:34.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: dump: 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 5 monclient: authenticate success, global_id 75416 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 1 -- 172.21.15.142:0/2646170772 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56800ba70 con 0x7fd56c09cdd0 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd57332a700 4 set_mon_vals no callback set 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 1 -- 172.21.15.142:0/2646170772 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56800be30 con 0x7fd56c09cdd0 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.590+0000 7fd563fff700 10 monclient: dump: 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.607 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: shutdown 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- 172.21.15.142:0/2646170772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 msgr2=0x7fd56c09e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- 172.21.15.142:0/2646170772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd5680099b0 tx=0x7fd568009250 comp rx=0 tx=0).stop 2024-09-18T10:23:34.608 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- 172.21.15.142:0/2646170772 shutdown_connections 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- 172.21.15.142:0/2646170772 >> v2:172.21.15.142:3300/0 conn(0x7fd56c0a02e0 0x7fd56c0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- 172.21.15.142:0/2646170772 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c09cdd0 0x7fd56c09e440 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- 172.21.15.142:0/2646170772 >> 172.21.15.142:0/2646170772 conn(0x7fd56c071680 msgr2=0x7fd56c0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- 172.21.15.142:0/2646170772 shutdown_connections 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- 172.21.15.142:0/2646170772 wait complete. 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: monmap: 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.609 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 Processor -- start 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- start start 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: init 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c09cdd0 0x7fd56c08cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-b con 0x7fd56c09cdd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 0x7fd56c08d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-c con 0x7fd56c0a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56c0a0bf0 0x7fd56c08da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): picked mon.noname-a con 0x7fd56c0a0bf0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c090660 con 0x7fd56c0a0bf0 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c090380 con 0x7fd56c09cdd0 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c090940 con 0x7fd56c0a02e0 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd57332a700 10 monclient(hunting): authenticate will time out at 1703.523423s 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 0x7fd56c08d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 0x7fd56c08d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40594/0 (socket says 172.21.15.142:40594) 2024-09-18T10:23:34.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56c0a0bf0 0x7fd56c08da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 1 -- 172.21.15.142:0/438272610 learned_addr learned my addr 172.21.15.142:0/438272610 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): get_auth_request con 0x7fd56c0a02e0 auth_method 0 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): get_auth_request con 0x7fd56c0a0bf0 auth_method 0 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5710c6700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c09cdd0 0x7fd56c08cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5710c6700 10 monclient(hunting): get_auth_request con 0x7fd56c09cdd0 auth_method 0 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5710c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5710c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5710c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5718c7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.594+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 10 monclient(hunting): handle_auth_done global_id 75419 payload 824 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c09cdd0 msgr2=0x7fd56c08cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c09cdd0 0x7fd56c08cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56c0a0bf0 msgr2=0x7fd56c08da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56c0a0bf0 0x7fd56c08da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 monclient: found mon.noname-c 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680094f0 con 0x7fd56c0a02e0 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5718c7700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56c0a0bf0 0x7fd56c08da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 msgr2=0x7fd56c08d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5708c5700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 0x7fd56c08d510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00ec80 tx=0x7fd55c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0096b0 con 0x7fd56c0a02e0 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: dump: 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 5 monclient: authenticate success, global_id 75419 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00ca60 con 0x7fd56c0a02e0 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00ce20 con 0x7fd56c0a02e0 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5617fa700 10 monclient: dump: 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.616 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd5637fe700 4 set_mon_vals no callback set 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 10 monclient: _renew_subs 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c08e850 con 0x7fd56c0a02e0 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 10 monclient: _renew_subs 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.598+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c08ed00 con 0x7fd56c0a02e0 2024-09-18T10:23:34.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd57332a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd57332a700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 0x7fd56c02ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0648c0 con 0x7fd56c08df90 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 0x7fd56c02ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient: get_auth_request con 0x7fd56c08df90 auth_method 0 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd55c013680 con 0x7fd56c0a02e0 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 10 monclient(hunting): handle_auth_done global_id 154231 payload 824 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd56c0904f0 con 0x7fd56c08df90 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5617fa700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd550035b10 0x7fd550037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd55c049230 con 0x7fd56c0a02e0 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5718c7700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd550035b10 0x7fd550037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5718c7700 10 monclient: get_auth_request con 0x7fd550035b10 auth_method 0 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 msgr2=0x7fd56c02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.602+0000 7fd5710c6700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 0x7fd56c02ae80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd568015000 tx=0x7fd568016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568021070 con 0x7fd56c08df90 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 10 monclient: dump: 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.622 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5718c7700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd550035b10 0x7fd550037fc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd56801c450 con 0x7fd56c08df90 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 msgr2=0x7fd56c02ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 0x7fd56c02ae80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd568015000 tx=0x7fd568016000 comp rx=0 tx=0).stop 2024-09-18T10:23:34.623 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 msgr2=0x7fd56c02ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.606+0000 7fd5617fa700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c08df90 0x7fd56c02ae80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd57332a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:34.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd57332a700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 0x7fd56c05ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd57332a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c09c070 con 0x7fd56c05e890 2024-09-18T10:23:34.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd5710c6700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 0x7fd56c05ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd5710c6700 10 monclient: get_auth_request con 0x7fd56c05e890 auth_method 0 2024-09-18T10:23:34.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd5710c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd5710c6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.614+0000 7fd5710c6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 10 monclient(hunting): handle_auth_done global_id 154234 payload 824 2024-09-18T10:23:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 1 -- 172.21.15.142:0/438272610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd56c06d320 con 0x7fd56c05e890 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 msgr2=0x7fd56c05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5710c6700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 0x7fd56c05ec90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd568003080 tx=0x7fd56801e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56801ce40 con 0x7fd56c05e890 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 10 monclient: dump: 2024-09-18T10:23:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.631 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7fd568011d50 con 0x7fd56c05e890 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 msgr2=0x7fd56c05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 0x7fd56c05ec90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd568003080 tx=0x7fd56801e070 comp rx=0 tx=0).stop 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 msgr2=0x7fd56c05ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:34.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.618+0000 7fd5617fa700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56c05e890 0x7fd56c05ec90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.633 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:34.633 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:34.633 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:34.633 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:23:34.633 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 136, 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.634 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.635 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:34.636 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.637 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:34.638 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:34.639 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:34.640 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.641 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.642 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.643 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.644 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:34.645 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:34.646 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.647 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:34.649 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:34.649 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:34.649 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd550035b10 msgr2=0x7fd550037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd550035b10 0x7fd550037fc0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd564007fd0 tx=0x7fd564002620 comp rx=0 tx=0).stop 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 10 monclient: shutdown 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 msgr2=0x7fd56c08d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 --2- 172.21.15.142:0/438272610 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56c0a02e0 0x7fd56c08d510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00ec80 tx=0x7fd55c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd5718c7700 1 -- 172.21.15.142:0/438272610 reap_dead start 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd5718c7700 1 -- 172.21.15.142:0/438272610 reap_dead start 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 shutdown_connections 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 >> 172.21.15.142:0/438272610 conn(0x7fd56c071680 msgr2=0x7fd56c0a47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 shutdown_connections 2024-09-18T10:23:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.622+0000 7fd57332a700 1 -- 172.21.15.142:0/438272610 wait complete. 2024-09-18T10:23:34.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:23:34.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:23:34.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:23:34.652 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:23:34.652 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:23:34.652 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:23:34.652 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:23:34.670 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:23:34.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:23:34.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:34.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: get_monmap_and_config 2024-09-18T10:23:34.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: monmap: 2024-09-18T10:23:34.775 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.775 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:34.773379+0000 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:34.773379+0000 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:34.776 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 Processor -- start 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 -- start start 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: init 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6f809aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6f8132570 0x7fe6f8134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-a con 0x7fe6f8132570 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6f8134e90 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f809aee0 con 0x7fe6f8134e90 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f806f6d0 con 0x7fe6f809aae0 2024-09-18T10:23:34.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80955a0 con 0x7fe6f8132570 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6ff425700 10 monclient(hunting): authenticate will time out at 1703.698976s 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fe423700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7fe6f809aae0 legacy=0x7fe6f8096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6f8132570 0x7fe6f8134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe6f8132570 0x7fe6f8134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56618/0 (socket says 172.21.15.142:56618) 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 -- 172.21.15.142:0/4021668949 learned_addr learned my addr 172.21.15.142:0/4021668949 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): get_auth_request con 0x7fe6f8132570 auth_method 0 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_done global_id 164157 payload 274 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 -- 172.21.15.142:0/4021668949 >> v1:172.21.15.150:6789/0 conn(0x7fe6f809aae0 legacy=0x7fe6f8096260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 -- 172.21.15.142:0/4021668949 >> v1:172.21.15.142:6789/0 conn(0x7fe6f8134e90 legacy=0x7fe6f8137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 monclient: found mon.noname-a 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:34.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.770+0000 7fe6fdc22700 1 -- 172.21.15.142:0/4021668949 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f8095b60 con 0x7fe6f8132570 2024-09-18T10:23:34.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fdc22700 1 -- 172.21.15.142:0/4021668949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 msgr2=0x7fe6f8134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fdc22700 1 --2- 172.21.15.142:0/4021668949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f8134950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009f20 tx=0x7fe6f4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe4022bfd07f2a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 1 -- 172.21.15.142:0/4021668949 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400e070 con 0x7fe6f8132570 2024-09-18T10:23:34.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: dump: 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 5 monclient: authenticate success, global_id 164157 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 1 -- 172.21.15.142:0/4021668949 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f400bc90 con 0x7fe6f8132570 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 1 -- 172.21.15.142:0/4021668949 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4012660 con 0x7fe6f8132570 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6fd421700 10 monclient: dump: 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.785 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 4 set_mon_vals no callback set 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 10 monclient: shutdown 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 -- 172.21.15.142:0/4021668949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 msgr2=0x7fe6f8134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 --2- 172.21.15.142:0/4021668949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f8134950 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009f20 tx=0x7fe6f4009840 comp rx=0 tx=0).stop 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 -- 172.21.15.142:0/4021668949 shutdown_connections 2024-09-18T10:23:34.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 --2- 172.21.15.142:0/4021668949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f8134950 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 -- 172.21.15.142:0/4021668949 >> 172.21.15.142:0/4021668949 conn(0x7fe6f80a1fe0 msgr2=0x7fe6f8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 -- 172.21.15.142:0/4021668949 shutdown_connections 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.774+0000 7fe6ff425700 1 -- 172.21.15.142:0/4021668949 wait complete. 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient: build_initial_monmap 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient: monmap: 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.787 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 Processor -- start 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 -- start start 2024-09-18T10:23:34.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient: init 2024-09-18T10:23:34.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-c con 0x7fe6f809aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-a con 0x7fe6f8132570 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 0x7fe6f806c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): picked mon.noname-b con 0x7fe6f8134e90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f809f260 con 0x7fe6f8132570 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f8126ef0 con 0x7fe6f8134e90 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f806f8d0 con 0x7fe6f809aae0 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6ff425700 10 monclient(hunting): authenticate will time out at 1703.707475s 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fec24700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 msgr2=0x7fe6f806c450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56628/0 (socket says 172.21.15.142:56628) 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 1 -- 172.21.15.142:0/755538909 learned_addr learned my addr 172.21.15.142:0/755538909 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40622/0 (socket says 172.21.15.142:40622) 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): get_auth_request con 0x7fe6f8132570 auth_method 0 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fec24700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 0x7fe6f806c450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): get_auth_request con 0x7fe6f809aae0 auth_method 0 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fe423700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.778+0000 7fe6fdc22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 10 monclient(hunting): handle_auth_done global_id 75422 payload 824 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 -- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 msgr2=0x7fe6f806c450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 0x7fe6f806c450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 -- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 msgr2=0x7fe6f806bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 10 monclient: _finish_hunting 0 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 monclient: found mon.noname-c 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 -- 172.21.15.142:0/755538909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009ae0 con 0x7fe6f809aae0 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fdc22700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:34.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 -- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 msgr2=0x7fe6f806b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fe423700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800b5d0 tx=0x7fe6e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8003e20 con 0x7fe6f809aae0 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: dump: 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: _finish_auth 0 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 5 monclient: authenticate success, global_id 75422 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e8006150 con 0x7fe6f809aae0 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 10 monclient: _renew_subs 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f806d470 con 0x7fe6f809aae0 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8006510 con 0x7fe6f809aae0 2024-09-18T10:23:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ee7fc700 10 monclient: dump: 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 10 monclient: _renew_subs 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f806e1e0 con 0x7fe6f809aae0 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.782+0000 7fe6fcc20700 4 set_mon_vals no callback set 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe6e801b3b0 con 0x7fe6f809aae0 2024-09-18T10:23:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ff425700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ff425700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ff425700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f81306a0 con 0x7fe6f809aae0 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ee7fc700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 0x7fe6e4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe6e8048540 con 0x7fe6f809aae0 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6fdc22700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 0x7fe6e4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.786+0000 7fe6fdc22700 10 monclient: get_auth_request con 0x7fe6e4035a70 auth_method 0 2024-09-18T10:23:34.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.790+0000 7fe6fdc22700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 0x7fe6e4037f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c010 tx=0x7fe6f4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:34.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.794+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe6e800e680 con 0x7fe6f809aae0 2024-09-18T10:23:34.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.794+0000 7fe6ee7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:34.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:34.794+0000 7fe6ee7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:35.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ff425700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:35.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ff425700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:35.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ff425700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:35.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6f81369c0 con 0x7fe6f809aae0 2024-09-18T10:23:35.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ee7fc700 1 -- 172.21.15.142:0/755538909 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fe6e8017030 con 0x7fe6f809aae0 2024-09-18T10:23:35.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ee7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:35.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.022+0000 7fe6ee7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:35.029 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:35.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 msgr2=0x7fe6e4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:35.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 0x7fe6e4037f20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400c010 tx=0x7fe6f4017000 comp rx=0 tx=0).stop 2024-09-18T10:23:35.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 10 monclient: shutdown 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 msgr2=0x7fe6f806b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800b5d0 tx=0x7fe6e80091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 shutdown_connections 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe6e4035a70 0x7fe6e4037f20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6f8134e90 0x7fe6f806c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6f8132570 0x7fe6f806bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 --2- 172.21.15.142:0/755538909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe6f809aae0 0x7fe6f806b9d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 >> 172.21.15.142:0/755538909 conn(0x7fe6f80a1fe0 msgr2=0x7fe6f8131ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 shutdown_connections 2024-09-18T10:23:35.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:35.026+0000 7fe6ff425700 1 -- 172.21.15.142:0/755538909 wait complete. 2024-09-18T10:23:38.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:38.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.146+0000 7f2bf5f5d700 10 monclient: get_monmap_and_config 2024-09-18T10:23:38.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.146+0000 7f2bf5f5d700 10 monclient: build_initial_monmap 2024-09-18T10:23:38.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.146+0000 7f2bf5f5d700 10 monclient: monmap: 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:38.151556+0000 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:38.151556+0000 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:38.154 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 Processor -- start 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 -- start start 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient: init 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2bf009aae0 0x7f2bf0096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-a con 0x7f2bf009aae0 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-b con 0x7f2bf00967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bf009c320 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf0095880 con 0x7f2bf009c320 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:38.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf009aeb0 con 0x7f2bf009aae0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf006f6d0 con 0x7f2bf00967a0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf5f5d700 10 monclient(hunting): authenticate will time out at 1707.077033s 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf575c700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f2bf009c320 legacy=0x7f2bf0136590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2bf009aae0 0x7f2bf0096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2bf009aae0 0x7f2bf0096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56632/0 (socket says 172.21.15.142:56632) 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/2185109020 learned_addr learned my addr 172.21.15.142:0/2185109020 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): get_auth_request con 0x7f2bf009aae0 auth_method 0 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:38.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): get_auth_request con 0x7f2bf00967a0 auth_method 0 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2beffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_done global_id 164163 payload 274 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 msgr2=0x7f2bf009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:38.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 --2- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf009bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/2185109020 >> v1:172.21.15.150:6789/0 conn(0x7f2bf009c320 legacy=0x7f2bf0136590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient: _finish_hunting 0 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 monclient: found mon.noname-a 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.150+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/2185109020 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00955a0 con 0x7f2bf009aae0 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009aae0 msgr2=0x7f2bf0096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf4f5b700 1 --2- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009aae0 0x7f2bf0096260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2be00099b0 tx=0x7f2be0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee98acad18b6469 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 1 -- 172.21.15.142:0/2185109020 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0009ea0 con 0x7f2bf009aae0 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: dump: 2024-09-18T10:23:38.163 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: _finish_auth 0 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 5 monclient: authenticate success, global_id 164163 2024-09-18T10:23:38.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 1 -- 172.21.15.142:0/2185109020 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be000ba70 con 0x7f2bf009aae0 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 4 set_mon_vals no callback set 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 1 -- 172.21.15.142:0/2185109020 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be000be30 con 0x7f2bf009aae0 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bef7fe700 10 monclient: dump: 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.165 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 10 monclient: shutdown 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009aae0 msgr2=0x7f2bf0096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009aae0 0x7f2bf0096260 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2be00099b0 tx=0x7f2be0009250 comp rx=0 tx=0).stop 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/2185109020 shutdown_connections 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf009bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/2185109020 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009aae0 0x7f2bf0096260 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/2185109020 >> 172.21.15.142:0/2185109020 conn(0x7f2bf00a1fe0 msgr2=0x7f2bf0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:38.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/2185109020 shutdown_connections 2024-09-18T10:23:38.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/2185109020 wait complete. 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 10 monclient: build_initial_monmap 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 10 monclient: monmap: 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:38.172 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.154+0000 7f2bf5f5d700 1 Processor -- start 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 -- start start 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient: init 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-c con 0x7f2bf00967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:38.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 0x7f2bf013fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-b con 0x7f2bf009aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009c320 0x7f2bf01400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): picked mon.noname-a con 0x7f2bf009c320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0090510 con 0x7f2bf009c320 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf006f8d0 con 0x7f2bf009aae0 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf006ee00 con 0x7f2bf00967a0 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf5f5d700 10 monclient(hunting): authenticate will time out at 1707.084752s 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:38.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2beffff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 msgr2=0x7f2bf013fb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2beffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 0x7f2bf013fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009c320 0x7f2bf01400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40636/0 (socket says 172.21.15.142:40636) 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/3815465906 learned_addr learned my addr 172.21.15.142:0/3815465906 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): get_auth_request con 0x7f2bf00967a0 auth_method 0 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): get_auth_request con 0x7f2bf009c320 auth_method 0 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:38.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf575c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient(hunting): handle_auth_done global_id 75428 payload 824 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 msgr2=0x7f2bf013fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 0x7f2bf013fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009c320 msgr2=0x7f2bf01400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009c320 0x7f2bf01400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient: _finish_hunting 0 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 monclient: found mon.noname-c 2024-09-18T10:23:38.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/3815465906 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be00094f0 con 0x7f2bf00967a0 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 -- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 msgr2=0x7f2bf013f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.158+0000 7f2bf4f5b700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0071c90 tx=0x7f2be0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0006b60 con 0x7f2bf00967a0 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: dump: 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.177 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: _finish_auth 0 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 5 monclient: authenticate success, global_id 75428 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be0020070 con 0x7f2bf00967a0 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be0011d80 con 0x7f2bf00967a0 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _renew_subs 2024-09-18T10:23:38.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf008f2a0 con 0x7f2bf00967a0 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 10 monclient: dump: 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:38.179 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2beeffd700 4 set_mon_vals no callback set 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _renew_subs 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf008d770 con 0x7f2bf00967a0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf008f4c0 con 0x7f2bf00967a0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2be0022030 con 0x7f2bf00967a0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 0x7f2bdc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2be004ad00 con 0x7f2bf00967a0 2024-09-18T10:23:38.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2beffff700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 0x7f2bdc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:38.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.162+0000 7f2beffff700 10 monclient: get_auth_request con 0x7f2bdc035a70 auth_method 0 2024-09-18T10:23:38.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.166+0000 7f2beffff700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 0x7f2bdc037f20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2be40094c0 tx=0x7f2be40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:38.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.170+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2bf008f4c0 con 0x7f2bf00967a0 2024-09-18T10:23:38.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.170+0000 7f2becff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:38.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.170+0000 7f2becff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:38.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2bf5f5d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:38.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2bf5f5d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:38.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2bf5f5d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:38.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2bf008f4c0 con 0x7f2bf00967a0 2024-09-18T10:23:38.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2becff9700 1 -- 172.21.15.142:0/3815465906 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2bf008f4c0 con 0x7f2bf00967a0 2024-09-18T10:23:38.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2becff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:38.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.386+0000 7f2becff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:38.393 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:38.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 msgr2=0x7f2bdc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:38.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 0x7f2bdc037f20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2be40094c0 tx=0x7f2be40091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 10 monclient: shutdown 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 msgr2=0x7f2bf013f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0071c90 tx=0x7f2be0016000 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 shutdown_connections 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2bdc035a70 0x7f2bdc037f20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bf009c320 0x7f2bf01400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf009aae0 0x7f2bf013fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 --2- 172.21.15.142:0/3815465906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2bf00967a0 0x7f2bf013f650 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 >> 172.21.15.142:0/3815465906 conn(0x7f2bf00a1fe0 msgr2=0x7f2bf009fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 shutdown_connections 2024-09-18T10:23:38.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:38.390+0000 7f2bf5f5d700 1 -- 172.21.15.142:0/3815465906 wait complete. 2024-09-18T10:23:41.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:41.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 10 monclient: get_monmap_and_config 2024-09-18T10:23:41.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 10 monclient: build_initial_monmap 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 10 monclient: monmap: 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:41.522570+0000 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:41.522570+0000 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:41.525 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 1 Processor -- start 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 1 -- start start 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.518+0000 7fa10ffff700 10 monclient: init 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa11009b050 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 0x7fa1100953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fa110098080 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa110135b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fa110133780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11009a570 con 0x7fa11009b050 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11006f900 con 0x7fa110098080 2024-09-18T10:23:41.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11009b420 con 0x7fa110133780 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10ffff700 10 monclient(hunting): authenticate will time out at 1710.448897s 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10effd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa11009b050 0x7fa110097970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38112/0 (socket says 172.21.15.142:38112) 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10effd700 1 -- 172.21.15.142:0/2170679043 learned_addr learned my addr 172.21.15.142:0/2170679043 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa110135b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 1 --2- 172.21.15.142:0/2170679043 >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 0x7fa1100953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): get_auth_request con 0x7fa110133780 auth_method 0 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): get_auth_request con 0x7fa110098080 auth_method 0 2024-09-18T10:23:41.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:41.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:41.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:41.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:41.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:41.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_done global_id 75431 payload 274 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 -- 172.21.15.142:0/2170679043 >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 msgr2=0x7fa1100953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/2170679043 >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 0x7fa1100953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 -- 172.21.15.142:0/2170679043 >> v1:172.21.15.142:6789/0 conn(0x7fa11009b050 legacy=0x7fa110097970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:41.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa10f7fe700 1 -- 172.21.15.142:0/2170679043 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11009a290 con 0x7fa110133780 2024-09-18T10:23:41.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.522+0000 7fa1067fc700 1 --2- 172.21.15.142:0/2170679043 >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 0x7fa1100953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10f7fe700 1 -- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 msgr2=0x7fa110135b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa110135b60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa11006fc80 tx=0x7fa108009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb4bd82a93aeb18 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 1 -- 172.21.15.142:0/2170679043 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10800e070 con 0x7fa110133780 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: dump: 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:41.537 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 1 -- 172.21.15.142:0/2170679043 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10800bd70 con 0x7fa110133780 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 5 monclient: authenticate success, global_id 75431 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:41.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 1 -- 172.21.15.142:0/2170679043 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108012660 con 0x7fa110133780 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 4 set_mon_vals no callback set 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10e7fc700 10 monclient: dump: 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.539 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 10 monclient: shutdown 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 -- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 msgr2=0x7fa110135b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 --2- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa110135b60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa11006fc80 tx=0x7fa108009780 comp rx=0 tx=0).stop 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 -- 172.21.15.142:0/2170679043 shutdown_connections 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 --2- 172.21.15.142:0/2170679043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa110135b60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 --2- 172.21.15.142:0/2170679043 >> v2:172.21.15.142:3300/0 conn(0x7fa110098080 0x7fa1100953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 -- 172.21.15.142:0/2170679043 >> 172.21.15.142:0/2170679043 conn(0x7fa1100a1fe0 msgr2=0x7fa110070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 -- 172.21.15.142:0/2170679043 shutdown_connections 2024-09-18T10:23:41.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 -- 172.21.15.142:0/2170679043 wait complete. 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 10 monclient: build_initial_monmap 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 10 monclient: monmap: 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.526+0000 7fa10ffff700 1 Processor -- start 2024-09-18T10:23:41.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 -- start start 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient: init 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 0x7fa11013f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fa110098080 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fa11009b050 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fa110133780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110129da0 con 0x7fa11009b050 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11012a360 con 0x7fa110098080 2024-09-18T10:23:41.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11012a640 con 0x7fa110133780 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10ffff700 10 monclient(hunting): authenticate will time out at 1710.456700s 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10effd700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 msgr2=0x7fa11013f2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:40662/0 (socket says 172.21.15.142:40662) 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:56660/0 (socket says 172.21.15.142:56660) 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 -- 172.21.15.142:0/717134028 learned_addr learned my addr 172.21.15.142:0/717134028 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): get_auth_request con 0x7fa110133780 auth_method 0 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:41.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10effd700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 0x7fa11013f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): get_auth_request con 0x7fa11009b050 auth_method 0 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient(hunting): handle_auth_done global_id 75434 payload 824 2024-09-18T10:23:41.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 -- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 msgr2=0x7fa11013f2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 0x7fa11013f2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 -- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 msgr2=0x7fa11012fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 -- 172.21.15.142:0/717134028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f800a5d0 con 0x7fa110133780 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1067fc700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 -- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 msgr2=0x7fa11006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:41.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa10f7fe700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa108003380 tx=0x7fa108019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108027450 con 0x7fa110133780 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1077fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.530+0000 7fa1077fe700 10 monclient: dump: 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 10 monclient: _finish_auth 0 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 5 monclient: authenticate success, global_id 75434 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _renew_subs 2024-09-18T10:23:41.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1080275f0 con 0x7fa110133780 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa108009a20 con 0x7fa110133780 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1080279b0 con 0x7fa110133780 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10dffb700 4 set_mon_vals no callback set 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 10 monclient: dump: 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:41.549 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _renew_subs 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa110128e40 con 0x7fa110133780 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa110129060 con 0x7fa110133780 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa1080303f0 con 0x7fa110133780 2024-09-18T10:23:41.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 0x7fa0f0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa10804acb0 con 0x7fa110133780 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10effd700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 0x7fa0f0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.534+0000 7fa10effd700 10 monclient: get_auth_request con 0x7fa0f0035a70 auth_method 0 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.538+0000 7fa10effd700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 0x7fa0f0037f20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa100004860 tx=0x7fa100007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.542+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa10801b750 con 0x7fa110133780 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.542+0000 7fa1077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:41.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.542+0000 7fa1077fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:41.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa10ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:41.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa10ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa10ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa110072a80 con 0x7fa110133780 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa1077fe700 1 -- 172.21.15.142:0/717134028 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fa1080257c0 con 0x7fa110133780 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa1077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.762+0000 7fa1077fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:41.768 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":136,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 msgr2=0x7fa0f0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 0x7fa0f0037f20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa100004860 tx=0x7fa100007c10 comp rx=0 tx=0).stop 2024-09-18T10:23:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 10 monclient: shutdown 2024-09-18T10:23:41.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 msgr2=0x7fa11006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa108003380 tx=0x7fa108019000 comp rx=0 tx=0).stop 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 shutdown_connections 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa0f0035a70 0x7fa0f0037f20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa110133780 0x7fa11006d4d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa11009b050 0x7fa11012fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 --2- 172.21.15.142:0/717134028 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa110098080 0x7fa11013f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 >> 172.21.15.142:0/717134028 conn(0x7fa1100a1fe0 msgr2=0x7fa110130810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 shutdown_connections 2024-09-18T10:23:41.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:41.766+0000 7fa10ffff700 1 -- 172.21.15.142:0/717134028 wait complete. 2024-09-18T10:23:44.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: get_monmap_and_config 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: build_initial_monmap 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: monmap: 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:44.892134+0000 2024-09-18T10:23:44.894 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:44.892134+0000 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:44.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 Processor -- start 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 -- start start 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: init 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9888097a30 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-a con 0x7f988809ce70 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:44.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f988809d270 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988809a7d0 con 0x7f9888097a30 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988809a4f0 con 0x7f988809d270 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988809db60 con 0x7f988809ce70 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f988ceb2700 10 monclient(hunting): authenticate will time out at 1713.818498s 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f9886d9d700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f988809d270 legacy=0x7f9888136f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60734/0 (socket says 172.21.15.142:60734) 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 learned_addr learned my addr 172.21.15.142:0/2605510812 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 10 monclient(hunting): get_auth_request con 0x7f988809ce70 auth_method 0 2024-09-18T10:23:44.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.890+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:44.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:44.894+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.142:6789/0 conn(0x7f9888097a30 legacy=0x7f9888097e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f9886d9d700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.150:6789/0 conn(0x7f988809d270 legacy=0x7f9888136f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 10 monclient(hunting): get_auth_request con 0x7f988809ce70 auth_method 0 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:45.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:23:45.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:45.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:45.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.094+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:23:45.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f9886d9d700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.150:6789/0 conn(0x7f988809d270 legacy=0x7f9888136f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:45.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:45.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 10 monclient(hunting): get_auth_request con 0x7f988809ce70 auth_method 0 2024-09-18T10:23:45.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:45.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:45.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:45.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:23:45.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:45.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:45.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:45.498+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:23:46.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f9886d9d700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.150:6789/0 conn(0x7f988809d270 legacy=0x7f9888136f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:46.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 10 monclient(hunting): get_auth_request con 0x7f988809ce70 auth_method 0 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:46.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:46.302+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:23:47.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): tick 2024-09-18T10:23:47.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 monclient(hunting): continuing hunt 2024-09-18T10:23:47.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 msgr2=0x7f9888134a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.142:3300/0 conn(0x7f988809ce70 0x7f9888134a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.150:6789/0 conn(0x7f988809d270 legacy=0x7f9888136f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 -- 172.21.15.142:0/2605510812 >> v1:172.21.15.142:6789/0 conn(0x7f9888097a30 legacy=0x7f9888097e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient: _add_conns ranks=[2,4] 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): picked mon.noname-b con 0x7f986c003770 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): picked mon.noname-c con 0x7f986c006290 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 -- 172.21.15.142:0/2605510812 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c008db0 con 0x7f986c006290 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:47.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 1 -- 172.21.15.142:0/2605510812 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c008f20 con 0x7f986c003770 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f9884d99700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 msgr2=0x7f986c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 10 monclient(hunting): get_auth_request con 0x7f986c003770 auth_method 0 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:47.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:47.890+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 msgr2=0x7f986c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 10 monclient(hunting): get_auth_request con 0x7f986c003770 auth_method 0 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:23:48.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:48.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.094+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:23:48.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 msgr2=0x7f986c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:48.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:23:48.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:48.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 10 monclient(hunting): get_auth_request con 0x7f986c003770 auth_method 0 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:48.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:48.498+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f987ffff700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 msgr2=0x7f986c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f987ffff700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): get_auth_request con 0x7f986c003770 auth_method 0 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient(hunting): handle_auth_done global_id 75437 payload 274 2024-09-18T10:23:49.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 msgr2=0x7f986c008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> v2:172.21.15.150:3300/0 conn(0x7f986c006290 0x7f986c008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 monclient: found mon.noname-b 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988809a210 con 0x7f986c003770 2024-09-18T10:23:49.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.302+0000 7f988659c700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9878007bd0 tx=0x7f9878009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a96decf840798 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 1 -- 172.21.15.142:0/2605510812 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98780073b0 con 0x7f986c003770 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: dump: 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 1 -- 172.21.15.142:0/2605510812 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9878007510 con 0x7f986c003770 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 5 monclient: authenticate success, global_id 75437 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 1 -- 172.21.15.142:0/2605510812 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878005a20 con 0x7f986c003770 2024-09-18T10:23:49.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 4 set_mon_vals no callback set 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9885d9b700 10 monclient: dump: 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 10 monclient: shutdown 2024-09-18T10:23:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 -- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 msgr2=0x7f986c005bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 --2- 172.21.15.142:0/2605510812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f986c003770 0x7f986c005bb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9878007bd0 tx=0x7f9878009040 comp rx=0 tx=0).stop 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f9886d9d700 1 -- 172.21.15.142:0/2605510812 reap_dead start 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 -- 172.21.15.142:0/2605510812 shutdown_connections 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 -- 172.21.15.142:0/2605510812 >> 172.21.15.142:0/2605510812 conn(0x7f98880a1470 msgr2=0x7f98880a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 -- 172.21.15.142:0/2605510812 shutdown_connections 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 1 -- 172.21.15.142:0/2605510812 wait complete. 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 10 monclient: build_initial_monmap 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.306+0000 7f988ceb2700 10 monclient: monmap: 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:49.314 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:49.315 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.315 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:49.315 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:49.315 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:49.315 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 Processor -- start 2024-09-18T10:23:49.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 -- start start 2024-09-18T10:23:49.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient: init 2024-09-18T10:23:49.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 0x7f988812e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-a con 0x7f986c003770 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 0x7f988812a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-b con 0x7f988812a0f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): picked mon.noname-c con 0x7f988812aa20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988812ddf0 con 0x7f986c003770 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988806f670 con 0x7f988812a0f0 2024-09-18T10:23:49.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9888096f10 con 0x7f988812aa20 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988ceb2700 10 monclient(hunting): authenticate will time out at 1718.237898s 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 0x7f988812e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f987ffff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 msgr2=0x7f988812a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39510/0 (socket says 172.21.15.142:39510) 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 0x7f988812e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60778/0 (socket says 172.21.15.142:60778) 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 -- 172.21.15.142:0/1928518660 learned_addr learned my addr 172.21.15.142:0/1928518660 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): get_auth_request con 0x7f988812aa20 auth_method 0 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): get_auth_request con 0x7f986c003770 auth_method 0 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f987ffff700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 0x7f988812a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f988659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient(hunting): handle_auth_done global_id 75440 payload 824 2024-09-18T10:23:49.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 -- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 msgr2=0x7f988812a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 0x7f988812a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 -- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 msgr2=0x7f988812e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 0x7f988812e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 monclient: found mon.noname-c 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.310+0000 7f9886d9d700 1 -- 172.21.15.142:0/1928518660 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987800f040 con 0x7f988812aa20 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f9886d9d700 1 -- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 msgr2=0x7f988813f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f9886d9d700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9888064b10 tx=0x7f988000a3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880016030 con 0x7f988812aa20 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: dump: 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.323 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 5 monclient: authenticate success, global_id 75440 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98800107a0 con 0x7f988812aa20 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _renew_subs 2024-09-18T10:23:49.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988812d4c0 con 0x7f988812aa20 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880010b60 con 0x7f988812aa20 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f987e7fc700 10 monclient: dump: 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.325 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988559a700 4 set_mon_vals no callback set 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _renew_subs 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988812ae80 con 0x7f988812aa20 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.314+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98880696a0 con 0x7f988812aa20 2024-09-18T10:23:49.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.318+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9880019070 con 0x7f988812aa20 2024-09-18T10:23:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.318+0000 7f987e7fc700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 0x7f986c038d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.318+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9880017070 con 0x7f988812aa20 2024-09-18T10:23:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.318+0000 7f988659c700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 0x7f986c038d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.318+0000 7f988659c700 10 monclient: get_auth_request con 0x7f986c036890 auth_method 0 2024-09-18T10:23:49.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.322+0000 7f988659c700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 0x7f986c038d40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9878000c00 tx=0x7f987801c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.322+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9880011420 con 0x7f988812aa20 2024-09-18T10:23:49.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.322+0000 7f987e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.322+0000 7f987e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:49.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f988ceb2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:23:49.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f988ceb2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:49.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f988ceb2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f988812d650 con 0x7f988812aa20 2024-09-18T10:23:49.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f987e7fc700 1 -- 172.21.15.142:0/1928518660 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f9880019320 con 0x7f988812aa20 2024-09-18T10:23:49.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f987e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:23:49.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.542+0000 7f987e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:49.550 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":140,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 msgr2=0x7f986c038d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 0x7f986c038d40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9878000c00 tx=0x7f987801c000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 10 monclient: shutdown 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 msgr2=0x7f988813f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9888064b10 tx=0x7f988000a3e0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 shutdown_connections 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f986c036890 0x7f986c038d40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f988812aa20 0x7f988813f9d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f988812a0f0 0x7f988812a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 --2- 172.21.15.142:0/1928518660 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f986c003770 0x7f988812e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 >> 172.21.15.142:0/1928518660 conn(0x7f98880a1470 msgr2=0x7f988809def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 shutdown_connections 2024-09-18T10:23:49.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.546+0000 7f988ceb2700 1 -- 172.21.15.142:0/1928518660 wait complete. 2024-09-18T10:23:49.563 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:23:49.563 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:23:49.563 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: get_monmap_and_config 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: build_initial_monmap 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: monmap: 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:49.669995+0000 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:49.669995+0000 2024-09-18T10:23:49.674 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 Processor -- start 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 -- start start 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: init 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-18T10:23:49.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-b con 0x7f97a809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 0x7f97a809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-c con 0x7f97a809b740 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97a809c410 addr v1:172.21.15.150:6789/0 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a809dc90 con 0x7f97a809c410 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a809b500 con 0x7f97a809b740 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a809afa0 con 0x7f97a809abd0 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97ac8d9700 10 monclient(hunting): authenticate will time out at 1718.595449s 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a659c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 msgr2=0x7f97a809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a659c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 0x7f97a809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a759e700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f97a809c410 legacy=0x7f97a8136680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39520/0 (socket says 172.21.15.142:39520) 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 1 -- 172.21.15.142:0/4095772410 learned_addr learned my addr 172.21.15.142:0/4095772410 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request con 0x7f97a809abd0 auth_method 0 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.666+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_done global_id 75443 payload 274 2024-09-18T10:23:49.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 -- 172.21.15.142:0/4095772410 >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 msgr2=0x7f97a809bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/4095772410 >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 0x7f97a809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 -- 172.21.15.142:0/4095772410 >> v1:172.21.15.150:6789/0 conn(0x7f97a809c410 legacy=0x7f97a8136680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 monclient: found mon.noname-b 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 -- 172.21.15.142:0/4095772410 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a806f810 con 0x7f97a809abd0 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 -- 172.21.15.142:0/4095772410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 msgr2=0x7f97a80960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/4095772410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f97980099b0 tx=0x7f9798009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=367cbc459466b7c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 1 -- 172.21.15.142:0/4095772410 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979800bb50 con 0x7f97a809abd0 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: dump: 2024-09-18T10:23:49.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 1 -- 172.21.15.142:0/4095772410 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979800bcb0 con 0x7f97a809abd0 2024-09-18T10:23:49.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 5 monclient: authenticate success, global_id 75443 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 1 -- 172.21.15.142:0/4095772410 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798011670 con 0x7f97a809abd0 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 4 set_mon_vals no callback set 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97a5d9b700 10 monclient: dump: 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.685 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 10 monclient: shutdown 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 -- 172.21.15.142:0/4095772410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 msgr2=0x7f97a80960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/4095772410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f97980099b0 tx=0x7f9798009250 comp rx=0 tx=0).stop 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 -- 172.21.15.142:0/4095772410 shutdown_connections 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/4095772410 >> v2:172.21.15.150:3300/0 conn(0x7f97a809b740 0x7f97a809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/4095772410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809abd0 0x7f97a80960e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 -- 172.21.15.142:0/4095772410 >> 172.21.15.142:0/4095772410 conn(0x7f97a8070ef0 msgr2=0x7f97a80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 -- 172.21.15.142:0/4095772410 shutdown_connections 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.670+0000 7f97ac8d9700 1 -- 172.21.15.142:0/4095772410 wait complete. 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient: build_initial_monmap 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient: monmap: 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:49.687 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 Processor -- start 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 -- start start 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient: init 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a809b740 0x7f97a813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-a con 0x7f97a809b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-c con 0x7f97a809c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a808bdb0 0x7f97a808c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): picked mon.noname-b con 0x7f97a808bdb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a808f990 con 0x7f97a809b740 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a809d240 con 0x7f97a808bdb0 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80a0150 con 0x7f97a809c410 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97ac8d9700 10 monclient(hunting): authenticate will time out at 1718.601939s 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a809b740 0x7f97a813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39534/0 (socket says 172.21.15.142:39534) 2024-09-18T10:23:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a759e700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a808bdb0 msgr2=0x7f97a808c1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 -- 172.21.15.142:0/443112807 learned_addr learned my addr 172.21.15.142:0/443112807 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): get_auth_request con 0x7f97a809c410 auth_method 0 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a759e700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a808bdb0 0x7f97a808c1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request con 0x7f97a809b740 auth_method 0 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient(hunting): handle_auth_done global_id 75446 payload 824 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a808bdb0 msgr2=0x7f97a808c1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a808bdb0 0x7f97a808c1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a809b740 msgr2=0x7f97a813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a809b740 0x7f97a813f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 monclient: found mon.noname-c 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.674+0000 7f97a659c700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980094f0 con 0x7f97a809c410 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a659c700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 msgr2=0x7f97a813feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a659c700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f979000b840 tx=0x7f9790009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9790003e20 con 0x7f97a809c410 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a659c700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f979000b840 tx=0x7f9790009220 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f9790006150 config(0 keys) v1 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: dump: 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.692 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 5 monclient: authenticate success, global_id 75446 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 monclient: _renew_subs 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9790006150 con 0x7f97a809c410 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a808dd20 con 0x7f97a809c410 2024-09-18T10:23:49.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a559a700 4 set_mon_vals no callback set 2024-09-18T10:23:49.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 monclient: _renew_subs 2024-09-18T10:23:49.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a808d810 con 0x7f97a809c410 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 0x7f97a808caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80648a0 con 0x7f97a808c720 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 0x7f97a808caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9790006510 con 0x7f97a809c410 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 10 monclient: dump: 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.696 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient: get_auth_request con 0x7f97a808c720 auth_method 0 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f979001a360 con 0x7f97a809c410 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_done global_id 164181 payload 824 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a6d9d700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97a808e520 con 0x7f97a808c720 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f979effd700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9788035a70 0x7f9788037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a759e700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9788035a70 0x7f9788037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.678+0000 7f97a759e700 10 monclient: get_auth_request con 0x7f9788035a70 auth_method 0 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f97a6d9d700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 msgr2=0x7f97a808caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 0x7f97a808caf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97a806fd20 tx=0x7f9798016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798021070 con 0x7f97a808c720 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 10 monclient: dump: 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f979801c660 con 0x7f97a808c720 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 msgr2=0x7f97a808caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 0x7f97a808caf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97a806fd20 tx=0x7f9798016000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 msgr2=0x7f97a808caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f979effd700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808c720 0x7f97a808caf0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97a806fd20 tx=0x7f9798016000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.682+0000 7f97a759e700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9788035a70 0x7f9788037f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f97a0003d00 tx=0x7f97a0009310 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.690+0000 7f97ac8d9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.690+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 0x7f97a8087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.690+0000 7f97ac8d9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.690+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a808f6b0 con 0x7f97a808d030 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 0x7f97a8087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient: get_auth_request con 0x7f97a808d030 auth_method 0 2024-09-18T10:23:49.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 10 monclient(hunting): handle_auth_done global_id 164184 payload 824 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 1 -- 172.21.15.142:0/443112807 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f97a8090620 con 0x7f97a808d030 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 msgr2=0x7f97a8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f97a6d9d700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 0x7f97a8087fe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97980094c0 tx=0x7f9798021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798032480 con 0x7f97a808d030 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:49.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 10 monclient: dump: 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f979801b440 con 0x7f97a808d030 2024-09-18T10:23:49.704 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 msgr2=0x7f97a8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 0x7f97a8087fe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97980094c0 tx=0x7f9798021000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 msgr2=0x7f97a8087fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.694+0000 7f979effd700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97a808d030 0x7f97a8087fe0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:23:49.705 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 140, 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.706 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.707 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:49.708 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.709 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:49.710 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.711 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.712 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:49.713 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.714 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.715 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:49.716 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.717 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.718 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:49.718 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9788035a70 msgr2=0x7f9788037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9788035a70 0x7f9788037f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f97a0003d00 tx=0x7f97a0009310 comp rx=0 tx=0).stop 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 10 monclient: shutdown 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 msgr2=0x7f97a813feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 --2- 172.21.15.142:0/443112807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f97a809c410 0x7f97a813feb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f979000b840 tx=0x7f9790009220 comp rx=0 tx=0).stop 2024-09-18T10:23:49.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97a759e700 1 -- 172.21.15.142:0/443112807 reap_dead start 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f979effd700 1 -- 172.21.15.142:0/443112807 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f9790048270 con 0x7f97a809c410 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97a759e700 1 -- 172.21.15.142:0/443112807 reap_dead start 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 shutdown_connections 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 >> 172.21.15.142:0/443112807 conn(0x7f97a8070ef0 msgr2=0x7f97a8096930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 shutdown_connections 2024-09-18T10:23:49.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.698+0000 7f97ac8d9700 1 -- 172.21.15.142:0/443112807 wait complete. 2024-09-18T10:23:49.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:23:49.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:23:49.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:23:49.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 10 monclient: get_monmap_and_config 2024-09-18T10:23:49.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 10 monclient: build_initial_monmap 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 10 monclient: monmap: 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:23:49.829567+0000 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:23:49.829567+0000 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:49.832 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:49.833 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 1 Processor -- start 2024-09-18T10:23:49.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 1 -- start start 2024-09-18T10:23:49.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.826+0000 7f31c5220700 10 monclient: init 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 0x7f31c0070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-c con 0x7f31c009bcc0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 0x7f31c0071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-a con 0x7f31c0071430 addr v2:172.21.15.142:3300/0 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-b con 0x7f31c0134750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c0071d70 con 0x7f31c0071430 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c009ddf0 con 0x7f31c009bcc0 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c009c090 con 0x7f31c0134750 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31c5220700 10 monclient(hunting): authenticate will time out at 1718.756862s 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bed9d700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 msgr2=0x7f31c0070ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bed9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 0x7f31c0070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 0x7f31c0071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39538/0 (socket says 172.21.15.142:39538) 2024-09-18T10:23:49.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 0x7f31c0071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60798/0 (socket says 172.21.15.142:60798) 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 -- 172.21.15.142:0/96943868 learned_addr learned my addr 172.21.15.142:0/96943868 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): get_auth_request con 0x7f31c0134750 auth_method 0 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): get_auth_request con 0x7f31c0071430 auth_method 0 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient(hunting): handle_auth_done global_id 75449 payload 274 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 -- 172.21.15.142:0/96943868 >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 msgr2=0x7f31c0070ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 --2- 172.21.15.142:0/96943868 >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 0x7f31c0070ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 -- 172.21.15.142:0/96943868 >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 msgr2=0x7f31c0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 --2- 172.21.15.142:0/96943868 >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 0x7f31c0071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 monclient: found mon.noname-b 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:49.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 -- 172.21.15.142:0/96943868 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c009eb70 con 0x7f31c0134750 2024-09-18T10:23:49.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 -- 172.21.15.142:0/96943868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 msgr2=0x7f31c0136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.830+0000 7f31bf59e700 1 --2- 172.21.15.142:0/96943868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f31c0064700 tx=0x7f31b8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e194fd91f2d67a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 1 -- 172.21.15.142:0/96943868 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800e070 con 0x7f31c0134750 2024-09-18T10:23:49.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: dump: 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 5 monclient: authenticate success, global_id 75449 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 1 -- 172.21.15.142:0/96943868 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b800bc30 con 0x7f31c0134750 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 4 set_mon_vals no callback set 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 1 -- 172.21.15.142:0/96943868 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b80092e0 con 0x7f31c0134750 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31be59c700 10 monclient: dump: 2024-09-18T10:23:49.848 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: shutdown 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- 172.21.15.142:0/96943868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 msgr2=0x7f31c0136b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- 172.21.15.142:0/96943868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f31c0064700 tx=0x7f31b8009840 comp rx=0 tx=0).stop 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- 172.21.15.142:0/96943868 shutdown_connections 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- 172.21.15.142:0/96943868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0134750 0x7f31c0136b30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- 172.21.15.142:0/96943868 >> v2:172.21.15.142:3300/0 conn(0x7f31c0071430 0x7f31c0071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- 172.21.15.142:0/96943868 >> v2:172.21.15.150:3300/0 conn(0x7f31c009bcc0 0x7f31c0070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- 172.21.15.142:0/96943868 >> 172.21.15.142:0/96943868 conn(0x7f31c00a25d0 msgr2=0x7f31c00a29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- 172.21.15.142:0/96943868 shutdown_connections 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- 172.21.15.142:0/96943868 wait complete. 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: build_initial_monmap 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: monmap: 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:23:49.850 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 Processor -- start 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- start start 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: init 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0071430 0x7f31c0099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-c con 0x7f31c0071430 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-a con 0x7f31c009bcc0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31c0134750 0x7f31c009a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): picked mon.noname-b con 0x7f31c0134750 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c012f800 con 0x7f31c009bcc0 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0133520 con 0x7f31c0134750 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c012f520 con 0x7f31c0071430 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31c5220700 10 monclient(hunting): authenticate will time out at 1718.763118s 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bf59e700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31c0134750 msgr2=0x7f31c009a530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60806/0 (socket says 172.21.15.142:60806) 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0071430 0x7f31c0099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 -- 172.21.15.142:0/222827484 learned_addr learned my addr 172.21.15.142:0/222827484 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): get_auth_request con 0x7f31c009bcc0 auth_method 0 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bf59e700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31c0134750 0x7f31c009a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): get_auth_request con 0x7f31c0071430 auth_method 0 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient(hunting): handle_auth_done global_id 164190 payload 824 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0071430 msgr2=0x7f31c0099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c0071430 0x7f31c0099ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31c0134750 msgr2=0x7f31c009a530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31c0134750 0x7f31c009a530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 monclient: found mon.noname-a 2024-09-18T10:23:49.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.834+0000 7f31b7fff700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8009ae0 con 0x7f31c009bcc0 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b7fff700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 msgr2=0x7f31c0099ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b7fff700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f31a800e930 tx=0x7f31a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8013890 con 0x7f31c009bcc0 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: dump: 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.855 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a80139f0 con 0x7f31c009bcc0 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 5 monclient: authenticate success, global_id 164190 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8013db0 con 0x7f31c009bcc0 2024-09-18T10:23:49.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31b6ffd700 10 monclient: dump: 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 monclient: _renew_subs 2024-09-18T10:23:49.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31bdd9b700 4 set_mon_vals no callback set 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c012fae0 con 0x7f31c009bcc0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 monclient: _renew_subs 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c012e630 con 0x7f31c009bcc0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.838+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c0064a50 con 0x7f31c002ae80 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient: get_auth_request con 0x7f31c002ae80 auth_method 0 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_done global_id 75455 payload 824 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31c006f400 con 0x7f31c002ae80 2024-09-18T10:23:49.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f31a800c670 con 0x7f31c009bcc0 2024-09-18T10:23:49.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 0x7f31a0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f31a8049120 con 0x7f31c009bcc0 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bf59e700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 0x7f31a0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bf59e700 10 monclient: get_auth_request con 0x7f31a0035a70 auth_method 0 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 msgr2=0x7f31c009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009650 tx=0x7f31b00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 msgr2=0x7f31c009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 msgr2=0x7f31c009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009650 tx=0x7f31b00091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f31b0009650 tx=0x7f31b00091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 msgr2=0x7f31c009aa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 0x7f31a0040820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 msgr2=0x7f31c009aa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c002ae80 0x7f31c009aa70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0040e80 con 0x7f31a003e410 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 0x7f31a0040820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient: get_auth_request con 0x7f31a003e410 auth_method 0 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.842+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31bed9d700 10 monclient(hunting): handle_auth_done global_id 75458 payload 824 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31a0040ff0 con 0x7f31a003e410 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31bf59e700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 0x7f31a0037f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f31c0064700 tx=0x7f31b8018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 msgr2=0x7f31a0040820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 0x7f31a0040820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f31b00093d0 tx=0x7f31b0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b001fea0 con 0x7f31a003e410 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 10 monclient: dump: 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.864 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f31b001b660 con 0x7f31a003e410 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 msgr2=0x7f31a0040820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 0x7f31a0040820 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f31b00093d0 tx=0x7f31b0015000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 msgr2=0x7f31a0040820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.865 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.846+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31a003e410 0x7f31a0040820 secure :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f31b00093d0 tx=0x7f31b0015000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31c5220700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:23:49.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 0x7f31c005ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:49.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31c5220700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:49.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c012fcd0 con 0x7f31c005e890 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 0x7f31c005ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient: get_auth_request con 0x7f31c005e890 auth_method 0 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 10 monclient(hunting): handle_auth_done global_id 75461 payload 824 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.858+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f31c00696a0 con 0x7f31c005e890 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31bed9d700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 msgr2=0x7f31c005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31bed9d700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 0x7f31c005ec90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f31a0040fc0 tx=0x7f31b0021050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:49.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0024070 con 0x7f31c005e890 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 10 monclient: dump: 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:23:49.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f31b001b450 con 0x7f31c005e890 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 msgr2=0x7f31c005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 0x7f31c005ec90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f31a0040fc0 tx=0x7f31b0021050 comp rx=0 tx=0).stop 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 msgr2=0x7f31c005ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31bf59e700 1 -- 172.21.15.142:0/222827484 reap_dead start 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31b6ffd700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f31c005e890 0x7f31c005ec90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:23:49.870 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 140, 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.871 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.872 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.873 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.873 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.873 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.873 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:23:49.877 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:23:49.878 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.879 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:23:49.880 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:23:49.881 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.882 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:23:49.883 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.884 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.885 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:23:49.886 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:23:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:23:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:23:49.887 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:23:49.887 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:23:49.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 msgr2=0x7f31a0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 0x7f31a0037f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f31c0064700 tx=0x7f31b8018000 comp rx=0 tx=0).stop 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 10 monclient: shutdown 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 msgr2=0x7f31c0099ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f31a800e930 tx=0x7f31a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 shutdown_connections 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f31a0035a70 0x7f31a0037f20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 --2- 172.21.15.142:0/222827484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31c009bcc0 0x7f31c0099ff0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 >> 172.21.15.142:0/222827484 conn(0x7f31c00a25d0 msgr2=0x7f31c0133c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 shutdown_connections 2024-09-18T10:23:49.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:23:49.862+0000 7f31c5220700 1 -- 172.21.15.142:0/222827484 wait complete. 2024-09-18T10:23:49.889 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:23:49.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:23:57.645 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=110 2024-09-18T10:23:57.645 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 110 -ne 110 ]] 2024-09-18T10:23:57.646 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:23:57.646 INFO:tasks.workunit.client.0.smithi150.stderr:+ export CEPH_ARGS= 2024-09-18T10:23:57.646 INFO:tasks.workunit.client.0.smithi150.stderr:+ CEPH_ARGS= 2024-09-18T10:23:57.646 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph auth del client.foo 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: get_monmap_and_config 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: build_initial_monmap 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: monmap: 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:23:57.740647+0000 2024-09-18T10:23:57.742 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:23:57.740647+0000 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:23:57.743 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 Processor -- start 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 -- start start 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: init 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 0x7f3c1c1320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c0713f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c1c0987a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 0x7f3c1c09d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.746 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c098bc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c096a60 con 0x7f3c1c0987a0 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c099020 con 0x7f3c1c0713f0 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0717c0 con 0x7f3c1c098bc0 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c209d2700 10 monclient(hunting): authenticate will time out at 1719.834228s 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1ad9d700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 msgr2=0x7f3c1c1320e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1ad9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 0x7f3c1c1320e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 0x7f3c1c09d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1a59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3c1c0987a0 0x7f3c1c09ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:33640/0 (socket says 172.21.15.150:33640) 2024-09-18T10:23:57.747 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1a59c700 1 -- 172.21.15.150:0/3581207364 learned_addr learned my addr 172.21.15.150:0/3581207364 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): get_auth_request con 0x7f3c1c098bc0 auth_method 0 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient(hunting): handle_auth_done global_id 75464 payload 274 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 -- 172.21.15.150:0/3581207364 >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 msgr2=0x7f3c1c1320e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 --2- 172.21.15.150:0/3581207364 >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 0x7f3c1c1320e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.748 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 -- 172.21.15.150:0/3581207364 >> v1:172.21.15.142:6789/0 conn(0x7f3c1c0987a0 legacy=0x7f3c1c09ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:23:57.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient: _finish_hunting 0 2024-09-18T10:23:57.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 monclient: found mon.noname-b 2024-09-18T10:23:57.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:57.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:23:57.749 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.737+0000 7f3c1b59e700 1 -- 172.21.15.150:0/3581207364 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c09e160 con 0x7f3c1c098bc0 2024-09-18T10:23:57.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c1b59e700 1 -- 172.21.15.150:0/3581207364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 msgr2=0x7f3c1c09d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:57.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c1b59e700 1 --2- 172.21.15.150:0/3581207364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 0x7f3c1c09d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3c140098e0 tx=0x7f3c140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca25450f812044d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:57.752 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 1 -- 172.21.15.150:0/3581207364 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400ba70 con 0x7f3c1c098bc0 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: dump: 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:57.753 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: _finish_auth 0 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 5 monclient: authenticate success, global_id 75464 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 1 -- 172.21.15.150:0/3581207364 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1400bbf0 con 0x7f3c1c098bc0 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 10 monclient: get_monmap_and_config success 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 4 set_mon_vals no callback set 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 1 -- 172.21.15.150:0/3581207364 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14011420 con 0x7f3c1c098bc0 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:23:57.754 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c19d9b700 10 monclient: dump: 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 10 monclient: shutdown 2024-09-18T10:23:57.755 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 -- 172.21.15.150:0/3581207364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 msgr2=0x7f3c1c09d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 --2- 172.21.15.150:0/3581207364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 0x7f3c1c09d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3c140098e0 tx=0x7f3c140091a0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 -- 172.21.15.150:0/3581207364 shutdown_connections 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 --2- 172.21.15.150:0/3581207364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c098bc0 0x7f3c1c09d190 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 --2- 172.21.15.150:0/3581207364 >> v2:172.21.15.150:3300/0 conn(0x7f3c1c0713f0 0x7f3c1c1320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 -- 172.21.15.150:0/3581207364 >> 172.21.15.150:0/3581207364 conn(0x7f3c1c0a2090 msgr2=0x7f3c1c0a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 -- 172.21.15.150:0/3581207364 shutdown_connections 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 1 -- 172.21.15.150:0/3581207364 wait complete. 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 10 monclient: build_initial_monmap 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.741+0000 7f3c209d2700 10 monclient: monmap: 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:23:57.756 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 Processor -- start 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 -- start start 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient: init 2024-09-18T10:23:57.757 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-a con 0x7f3c1c0713f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c0987a0 0x7f3c1c12bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-c con 0x7f3c1c0987a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 0x7f3c1c12f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): picked mon.noname-b con 0x7f3c1c098bc0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c1192b0 con 0x7f3c1c0713f0 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c118ca0 con 0x7f3c1c098bc0 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c118f40 con 0x7f3c1c0987a0 2024-09-18T10:23:57.758 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c209d2700 10 monclient(hunting): authenticate will time out at 1719.841266s 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1b59e700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 msgr2=0x7f3c1c12f3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 0x7f3c1c12f3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c0987a0 0x7f3c1c12bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.150:47268/0 (socket says 172.21.15.150:47268) 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 -- 172.21.15.150:0/1741152035 learned_addr learned my addr 172.21.15.150:0/1741152035 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): get_auth_request con 0x7f3c1c0713f0 auth_method 0 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:57.759 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): get_auth_request con 0x7f3c1c0987a0 auth_method 0 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient(hunting): handle_auth_done global_id 164193 payload 824 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 -- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c0987a0 msgr2=0x7f3c1c12bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:57.760 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c0987a0 0x7f3c1c12bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 -- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 msgr2=0x7f3c1c12f3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 0x7f3c1c12f3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 monclient: found mon.noname-a 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:23:57.761 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.745+0000 7f3c1ad9d700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c14009420 con 0x7f3c1c0713f0 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c1ad9d700 1 -- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 msgr2=0x7f3c1c141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c1ad9d700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000b7e0 tx=0x7f3c10009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10003e20 con 0x7f3c1c0713f0 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: dump: 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:23:57.763 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: _finish_auth 0 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c10006150 con 0x7f3c1c0713f0 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 5 monclient: authenticate success, global_id 164193 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _renew_subs 2024-09-18T10:23:57.764 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c130fa0 con 0x7f3c1c0713f0 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10006510 con 0x7f3c1c0713f0 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 10 monclient: dump: 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:23:57.765 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _renew_subs 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c1959a700 4 set_mon_vals no callback set 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c142100 con 0x7f3c1c0713f0 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c04c430 con 0x7f3c1c0713f0 2024-09-18T10:23:57.766 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3c100103c0 con 0x7f3c1c0713f0 2024-09-18T10:23:57.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 0x7f3c08037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:23:57.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3c10048720 con 0x7f3c1c0713f0 2024-09-18T10:23:57.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c1a59c700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 0x7f3c08037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:23:57.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.749+0000 7f3c1a59c700 10 monclient: get_auth_request con 0x7f3c08035a70 auth_method 0 2024-09-18T10:23:57.767 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.753+0000 7f3c1a59c700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 0x7f3c08037f20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c006e80 tx=0x7f3c0c0053c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:23:57.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.761+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3c1000ed30 con 0x7f3c1c0713f0 2024-09-18T10:23:57.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.761+0000 7f3c02ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:23:57.769 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.761+0000 7f3c02ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:23:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.985+0000 7f3c209d2700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.985+0000 7f3c209d2700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.985+0000 7f3c209d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:57.992 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.985+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f3c1c096f90 con 0x7f3c1c0713f0 2024-09-18T10:23:57.998 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.993+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c10017030 con 0x7f3c1c0713f0 2024-09-18T10:23:57.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.993+0000 7f3c02ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:57.999 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:57.993+0000 7f3c02ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:23:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.133+0000 7f3c209d2700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.133+0000 7f3c209d2700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.133+0000 7f3c209d2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:23:58.141 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.133+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.foo"} v 0) v1 -- 0x7f3c1c0008d0 con 0x7f3c1c0713f0 2024-09-18T10:23:58.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c02ffd700 1 -- 172.21.15.150:0/1741152035 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.foo"}]=0 v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c10017030 con 0x7f3c1c0713f0 2024-09-18T10:23:58.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c02ffd700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.foo"}] 2024-09-18T10:23:58.142 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c02ffd700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 msgr2=0x7f3c08037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 0x7f3c08037f20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c006e80 tx=0x7f3c0c0053c0 comp rx=0 tx=0).stop 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 10 monclient: shutdown 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 msgr2=0x7f3c1c141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000b7e0 tx=0x7f3c10009220 comp rx=0 tx=0).stop 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 shutdown_connections 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3c08035a70 0x7f3c08037f20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c1c098bc0 0x7f3c1c12f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:58.145 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3c1c0987a0 0x7f3c1c12bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:58.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 --2- 172.21.15.150:0/1741152035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3c1c0713f0 0x7f3c1c141450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:23:58.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 >> 172.21.15.150:0/1741152035 conn(0x7f3c1c0a2090 msgr2=0x7f3c1c135100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:23:58.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 shutdown_connections 2024-09-18T10:23:58.146 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:23:58.137+0000 7f3c209d2700 1 -- 172.21.15.150:0/1741152035 wait complete. 2024-09-18T10:23:58.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13 2024-09-18T10:23:58.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 2024-09-18T10:23:58.154 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:23:58.154 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:23:58.155 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:23:58.155 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status 2024-09-18T10:24:07.252 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ c=''\''allow command service with prefix=list, allow command quorum_status'\''' 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 0 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph auth get-or-create client.bar mon '\''allow command service with prefix=list, allow command quorum_status'\'' > /tmp/cephtest-mon-caps-madness.bar.keyring' 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T10:24:07.253 INFO:tasks.workunit.client.0.smithi150.stdout:ceph auth get-or-create client.bar mon 'allow command service with prefix=list, allow command quorum_status' > /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T10:24:07.254 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph auth get-or-create client.bar mon ''\''allow' command service with prefix=list, allow command 'quorum_status'\''' '>' /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T10:24:07.778 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 0 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=0 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:24:07.779 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:24:07.780 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=0 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 0 -ne 0 ]] 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 13 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls' 2024-09-18T10:24:09.077 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:24:09.078 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T10:24:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T10:24:09.078 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth ls 2024-09-18T10:24:09.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:24:09.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:24:09.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:09.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 13 2024-09-18T10:24:09.421 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export' 2024-09-18T10:24:09.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:24:09.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T10:24:09.422 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T10:24:09.422 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth export 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 13 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo' 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:24:09.746 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T10:24:09.747 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T10:24:09.747 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar auth del client.foo 2024-09-18T10:24:09.891 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:24:09.891 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:24:09.891 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:24:09.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:24:09.894 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:24:09.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:24:09.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:09.998+0000 7fe2313b4700 10 monclient: get_monmap_and_config 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:09.998+0000 7fe2313b4700 10 monclient: build_initial_monmap 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:09.998+0000 7fe2313b4700 10 monclient: monmap: 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:10.003172+0000 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:10.003172+0000 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:10.006 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 Processor -- start 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 -- start start 2024-09-18T10:24:10.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient: init 2024-09-18T10:24:10.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:10.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-18T10:24:10.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-c con 0x7fe22c06eec0 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:10.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe22c09c770 0x7fe22c09cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-a con 0x7fe22c09c770 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:10.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe22c1365a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:10.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22c098f30 con 0x7fe22c1365a0 2024-09-18T10:24:10.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:10.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c06f290 con 0x7fe22c09c770 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22c098970 con 0x7fe22c06eec0 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe2313b4700 10 monclient(hunting): authenticate will time out at 1738.929050s 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe22c09c770 0x7fe22c09cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe230bb3700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fe22c1365a0 0x7fe22c138980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38410/0 (socket says 172.21.15.142:38410) 2024-09-18T10:24:10.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe230bb3700 1 -- 172.21.15.142:0/1405510902 learned_addr learned my addr 172.21.15.142:0/1405510902 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:10.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): get_auth_request con 0x7fe22c09c770 auth_method 0 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:10.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:10.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:10.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 10 monclient(hunting): get_auth_request con 0x7fe22c06eec0 auth_method 0 2024-09-18T10:24:10.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:10.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:10.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 1 -- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 msgr2=0x7fe22c09c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 1 -- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 msgr2=0x7fe22c09c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient(hunting): handle_auth_done global_id 164232 payload 274 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 msgr2=0x7fe22c09c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:10.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 --2- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1405510902 >> v1:172.21.15.142:6789/0 conn(0x7fe22c1365a0 legacy=0x7fe22c138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:24:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 monclient: found mon.noname-a 2024-09-18T10:24:10.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.002+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1405510902 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c098c50 con 0x7fe22c09c770 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1405510902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c09c770 msgr2=0x7fe22c09cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 --2- 172.21.15.142:0/1405510902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c09c770 0x7fe22c09cb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe22c068da0 tx=0x7fe21c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40370b1abac1d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 1 -- 172.21.15.142:0/1405510902 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00b8c0 con 0x7fe22c09c770 2024-09-18T10:24:10.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:10.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:10.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: dump: 2024-09-18T10:24:10.036 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:10.036 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:10.036 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:10.037 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: _finish_auth 0 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 1 -- 172.21.15.142:0/1405510902 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c00ba20 con 0x7fe22c09c770 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 5 monclient: authenticate success, global_id 164232 2024-09-18T10:24:10.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 1 -- 172.21.15.142:0/1405510902 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00bdc0 con 0x7fe22c09c770 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 4 set_mon_vals no callback set 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:10.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22affd700 10 monclient: dump: 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:10.040 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: shutdown 2024-09-18T10:24:10.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:10.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- 172.21.15.142:0/1405510902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c09c770 msgr2=0x7fe22c09cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:10.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1405510902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c09c770 0x7fe22c09cb90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe22c068da0 tx=0x7fe21c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- 172.21.15.142:0/1405510902 shutdown_connections 2024-09-18T10:24:10.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1405510902 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c09c770 0x7fe22c09cb90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1405510902 >> v2:172.21.15.150:3300/0 conn(0x7fe22c06eec0 0x7fe22c09c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- 172.21.15.142:0/1405510902 >> 172.21.15.142:0/1405510902 conn(0x7fe22c0a13f0 msgr2=0x7fe22c0a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- 172.21.15.142:0/1405510902 shutdown_connections 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- 172.21.15.142:0/1405510902 wait complete. 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: build_initial_monmap 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: monmap: 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:10.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:10.044 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:10.044 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.044 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:10.044 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:10.044 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 Processor -- start 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- start start 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: init 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:10.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:24:10.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-a con 0x7fe22c06eec0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:10.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-b con 0x7fe22c09c770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:10.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 0x7fe22c12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): picked mon.noname-c con 0x7fe22c1365a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c09d470 con 0x7fe22c06eec0 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c09fd50 con 0x7fe22c09c770 2024-09-18T10:24:10.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c1311e0 con 0x7fe22c1365a0 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe2313b4700 10 monclient(hunting): authenticate will time out at 1738.935437s 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:60268/0 (socket says 172.21.15.142:60268) 2024-09-18T10:24:10.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 0x7fe22c12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 1 -- 172.21.15.142:0/1380151731 learned_addr learned my addr 172.21.15.142:0/1380151731 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): get_auth_request con 0x7fe22c06eec0 auth_method 0 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): get_auth_request con 0x7fe22c1365a0 auth_method 0 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:10.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 10 monclient(hunting): get_auth_request con 0x7fe22c09c770 auth_method 0 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:10.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe230bb3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 msgr2=0x7fe22c12e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 msgr2=0x7fe22c12e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.006+0000 7fe22b7fe700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe22b7fe700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 10 monclient(hunting): handle_auth_done global_id 75494 payload 824 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 msgr2=0x7fe22c12e0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 msgr2=0x7fe22c12db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 10 monclient: _finish_hunting 0 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 monclient: found mon.noname-c 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 -- 172.21.15.142:0/1380151731 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c009420 con 0x7fe22c1365a0 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 msgr2=0x7fe22c12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe230bb3700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 0x7fe22c12e5f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe22400cb60 tx=0x7fe224005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22400aa60 con 0x7fe22c1365a0 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: dump: 2024-09-18T10:24:10.052 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: _finish_auth 0 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:10.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22400abe0 con 0x7fe22c1365a0 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 5 monclient: authenticate success, global_id 75494 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 10 monclient: _renew_subs 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c134ed0 con 0x7fe22c1365a0 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2240070f0 con 0x7fe22c1365a0 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe223fff700 10 monclient: dump: 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.054 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe22a7fc700 4 set_mon_vals no callback set 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 10 monclient: _renew_subs 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.010+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c130fb0 con 0x7fe22c1365a0 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe2313b4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe2313b4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe2313b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c131400 con 0x7fe22c1365a0 2024-09-18T10:24:10.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe2240074e0 con 0x7fe22c1365a0 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe223fff700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 0x7fe20c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe224049370 con 0x7fe22c1365a0 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 0x7fe20c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.014+0000 7fe22bfff700 10 monclient: get_auth_request con 0x7fe20c035ac0 auth_method 0 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.018+0000 7fe22bfff700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 0x7fe20c037f70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe214006f00 tx=0x7fe214008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:10.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.022+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe22c131400 con 0x7fe22c1365a0 2024-09-18T10:24:10.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.022+0000 7fe223fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:10.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.022+0000 7fe223fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:10.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe22c131400 con 0x7fe22c1365a0 2024-09-18T10:24:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe223fff700 1 -- 172.21.15.142:0/1380151731 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fe22c131400 con 0x7fe22c1365a0 2024-09-18T10:24:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe223fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:10.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe223fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:10.247 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":140,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:10.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 msgr2=0x7fe20c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 0x7fe20c037f70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe214006f00 tx=0x7fe214008040 comp rx=0 tx=0).stop 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 10 monclient: shutdown 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 msgr2=0x7fe22c12e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 0x7fe22c12e5f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe22400cb60 tx=0x7fe224005340 comp rx=0 tx=0).stop 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 shutdown_connections 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe20c035ac0 0x7fe20c037f70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe22c1365a0 0x7fe22c12e5f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe22c09c770 0x7fe22c12e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 --2- 172.21.15.142:0/1380151731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe22c06eec0 0x7fe22c12db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 >> 172.21.15.142:0/1380151731 conn(0x7fe22c0a13f0 msgr2=0x7fe22c0a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.242+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 shutdown_connections 2024-09-18T10:24:10.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:10.246+0000 7fe2313b4700 1 -- 172.21.15.142:0/1380151731 wait complete. 2024-09-18T10:24:10.291 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 13 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump' 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:24:10.292 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T10:24:10.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T10:24:10.293 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar osd dump 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ export CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ CEPH_ARGS=--rados-mon-op-timeout=300 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 110 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump' 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=110 2024-09-18T10:24:10.576 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T10:24:10.577 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T10:24:10.577 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar pg dump 2024-09-18T10:24:13.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: monmap: 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:13.375819+0000 2024-09-18T10:24:13.378 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:13.375819+0000 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:13.379 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:13.380 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:13.380 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 Processor -- start 2024-09-18T10:24:13.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 -- start start 2024-09-18T10:24:13.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: init 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3d8132030 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 0x7ff3d8137360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d8132400 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3d81378a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d80a17b0 con 0x7ff3d8132030 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8132b90 con 0x7ff3d81378a0 2024-09-18T10:24:13.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d80a1a90 con 0x7ff3d8132400 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d7fff700 10 monclient(hunting): authenticate will time out at 1742.301375s 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d6ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff3d8132030 0x7ff3d809ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46928/0 (socket says 172.21.15.142:46928) 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 learned_addr learned my addr 172.21.15.142:0/2592203662 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 0x7ff3d8137360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request con 0x7ff3d8132400 auth_method 0 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 1 -- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 msgr2=0x7ff3d8137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 1 -- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 msgr2=0x7ff3d8137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 0x7ff3d8137360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 0x7ff3d8137360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.374+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 >> v1:172.21.15.142:6789/0 conn(0x7ff3d8132030 legacy=0x7ff3d809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 1 -- 172.21.15.142:0/2592203662 >> v1:172.21.15.150:6789/0 conn(0x7ff3d81378a0 legacy=0x7ff3d8139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:13.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239592080 0 0) 0x7ff3d80a17b0 con 0x7ff3d8132030 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): my global_id is 164253 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4003570 con 0x7ff3d8132030 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414572013 0 0) 0x7ff3c4003570 con 0x7ff3d8132030 2024-09-18T10:24:13.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 msgr2=0x7ff3d8137360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 --2- 172.21.15.142:0/2592203662 >> v2:172.21.15.150:3300/0 conn(0x7ff3d8132400 0x7ff3d8137360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 >> v1:172.21.15.150:6789/0 conn(0x7ff3d81378a0 legacy=0x7ff3d8139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: _finish_hunting 0 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:13.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80a00f0 con 0x7ff3d8132030 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff3d0001b20 con 0x7ff3d8132030 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d7fff700 5 monclient: authenticate success, global_id 164253 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:13.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: dump: 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 >> v1:172.21.15.142:6789/0 conn(0x7ff3d8132030 legacy=0x7ff3d809ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 0x7ff3c4004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): picked mon.c con 0x7ff3c4003c50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c4004810 0x7ff3c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): picked mon.b con 0x7ff3c4004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4009a50 0x7ff3c400be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): picked mon.a con 0x7ff3c4009a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:13.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d80a1a90 con 0x7ff3c4009a50 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c400c4b0 con 0x7ff3c4004810 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c400c670 con 0x7ff3c4003c50 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d5ffb700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d7fff700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:13.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 0x7ff3c4004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): get_auth_request con 0x7ff3c4003c50 auth_method 0 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c4004810 0x7ff3c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request con 0x7ff3c4004810 auth_method 0 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:13.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4009a50 0x7ff3c400be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 10 monclient(hunting): get_auth_request con 0x7ff3c4009a50 auth_method 0 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:13.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:13.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_done global_id 164253 payload 293 2024-09-18T10:24:13.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c4004810 msgr2=0x7ff3c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:13.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c4004810 0x7ff3c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4009a50 msgr2=0x7ff3c400be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:13.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4009a50 0x7ff3c400be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 monclient: found mon.c 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4003570 con 0x7ff3c4003c50 2024-09-18T10:24:13.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d77fe700 1 -- 172.21.15.142:0/2592203662 reap_dead start 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.378+0000 7ff3d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 1 -- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 msgr2=0x7ff3c4004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 0x7ff3c4004060 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0001850 tx=0x7ff3d000a240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77901f774fc501ee server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d000c5d0 con 0x7ff3c4003c50 2024-09-18T10:24:13.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: dump: 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:13.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d000c730 con 0x7ff3c4003c50 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 1 -- 172.21.15.142:0/2592203662 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d000caf0 con 0x7ff3c4003c50 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:13.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d5ffb700 10 monclient: dump: 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 4 set_mon_vals no callback set 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: shutdown 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 msgr2=0x7ff3c4004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 0x7ff3c4004060 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0001850 tx=0x7ff3d000a240 comp rx=0 tx=0).stop 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- 172.21.15.142:0/2592203662 shutdown_connections 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/2592203662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3c4003c50 0x7ff3c4004060 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- 172.21.15.142:0/2592203662 >> 172.21.15.142:0/2592203662 conn(0x7ff3d80a5700 msgr2=0x7ff3d80a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- 172.21.15.142:0/2592203662 shutdown_connections 2024-09-18T10:24:13.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- 172.21.15.142:0/2592203662 wait complete. 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: monmap: 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 Processor -- start 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- start start 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: init 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4003c50 0x7ff3d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff3c4003c50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d806dca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d806e5d0 0x7ff3d8147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d806e5d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8131400 con 0x7ff3c4003c50 2024-09-18T10:24:13.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d8131e70 con 0x7ff3d806e5d0 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d811d680 con 0x7ff3d806dca0 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d7fff700 10 monclient(hunting): authenticate will time out at 1742.311184s 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4003c50 0x7ff3d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45022/0 (socket says 172.21.15.142:45022) 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 -- 172.21.15.142:0/4051389344 learned_addr learned my addr 172.21.15.142:0/4051389344 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request con 0x7ff3d806dca0 auth_method 0 2024-09-18T10:24:13.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): get_auth_request con 0x7ff3c4003c50 auth_method 0 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d77fe700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d806e5d0 0x7ff3d8147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:13.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d77fe700 10 monclient(hunting): get_auth_request con 0x7ff3d806e5d0 auth_method 0 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient(hunting): handle_auth_done global_id 75509 payload 824 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 -- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d806e5d0 msgr2=0x7ff3d8147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d806e5d0 0x7ff3d8147d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 -- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4003c50 msgr2=0x7ff3d806d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4003c50 0x7ff3d806d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 monclient: found mon.noname-c 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.382+0000 7ff3d67fc700 1 -- 172.21.15.142:0/4051389344 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d809dd20 con 0x7ff3d806dca0 2024-09-18T10:24:13.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d67fc700 1 -- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 msgr2=0x7ff3d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d67fc700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0010980 tx=0x7ff3c0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000eae0 con 0x7ff3d806dca0 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: dump: 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:13.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 5 monclient: authenticate success, global_id 75509 2024-09-18T10:24:13.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000ec40 con 0x7ff3d806dca0 2024-09-18T10:24:13.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:13.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0016480 con 0x7ff3d806dca0 2024-09-18T10:24:13.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 10 monclient: _renew_subs 2024-09-18T10:24:13.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d809f6e0 con 0x7ff3d806dca0 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d57fa700 4 set_mon_vals no callback set 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3ceffd700 10 monclient: dump: 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:13.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:13.421 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.421 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:13.421 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:13.421 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:13.421 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:13.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:13.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:13.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:13.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 10 monclient: _renew_subs 2024-09-18T10:24:13.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.386+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d806f190 con 0x7ff3d806dca0 2024-09-18T10:24:13.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff3c0016830 con 0x7ff3d806dca0 2024-09-18T10:24:13.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:13.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:13.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d8072a80 con 0x7ff3d806dca0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3ceffd700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 0x7ff3b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff3c0049b70 con 0x7ff3d806dca0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 0x7ff3b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d6ffd700 10 monclient: get_auth_request con 0x7ff3b8035ac0 auth_method 0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.390+0000 7ff3d6ffd700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 0x7ff3b8037f70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000a0d0 tx=0x7ff3d0013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.398+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff3c0014430 con 0x7ff3d806dca0 2024-09-18T10:24:13.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.398+0000 7ff3ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:13.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.398+0000 7ff3ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3d7fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:13.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3d7fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3d7fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff3d80a0310 con 0x7ff3d806dca0 2024-09-18T10:24:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3ceffd700 1 -- 172.21.15.142:0/4051389344 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ff3c0027b40 con 0x7ff3d806dca0 2024-09-18T10:24:13.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3ceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:13.622 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.614+0000 7ff3ceffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:13.622 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":140,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:13.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 msgr2=0x7ff3b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 0x7ff3b8037f70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000a0d0 tx=0x7ff3d0013000 comp rx=0 tx=0).stop 2024-09-18T10:24:13.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 10 monclient: shutdown 2024-09-18T10:24:13.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 msgr2=0x7ff3d806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0010980 tx=0x7ff3c0008040 comp rx=0 tx=0).stop 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 shutdown_connections 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff3b8035ac0 0x7ff3b8037f70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d806e5d0 0x7ff3d8147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3d806dca0 0x7ff3d806e090 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 --2- 172.21.15.142:0/4051389344 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff3c4003c50 0x7ff3d806d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 >> 172.21.15.142:0/4051389344 conn(0x7ff3d80a5700 msgr2=0x7ff3d80a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 shutdown_connections 2024-09-18T10:24:13.626 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:13.618+0000 7ff3d7fff700 1 -- 172.21.15.142:0/4051389344 wait complete. 2024-09-18T10:24:16.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 10 monclient: get_monmap_and_config 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 10 monclient: build_initial_monmap 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 10 monclient: monmap: 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:16.754904+0000 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:16.754904+0000 2024-09-18T10:24:16.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 1 Processor -- start 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 1 -- start start 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.750+0000 7fb43b1d0700 10 monclient: init 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 0x7fb434132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-a con 0x7fb434132030 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:16.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-b con 0x7fb43409ece0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb43409ae60 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb434132940 con 0x7fb43409ae60 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43409f960 con 0x7fb434132030 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340a1a90 con 0x7fb43409ece0 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient(hunting): authenticate will time out at 1745.680487s 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45034/0 (socket says 172.21.15.142:45034) 2024-09-18T10:24:16.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 -- 172.21.15.142:0/20020821 learned_addr learned my addr 172.21.15.142:0/20020821 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): get_auth_request con 0x7fb43409ece0 auth_method 0 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 1 --2- 172.21.15.142:0/20020821 >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 0x7fb434132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 10 monclient(hunting): get_auth_request con 0x7fb434132030 auth_method 0 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:16.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient(hunting): handle_auth_done global_id 75512 payload 274 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 -- 172.21.15.142:0/20020821 >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 msgr2=0x7fb434132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 --2- 172.21.15.142:0/20020821 >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 0x7fb434132400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 -- 172.21.15.142:0/20020821 >> v1:172.21.15.150:6789/0 conn(0x7fb43409ae60 legacy=0x7fb43409b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 monclient: found mon.noname-b 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 -- 172.21.15.142:0/20020821 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340a17b0 con 0x7fb43409ece0 2024-09-18T10:24:16.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43a1ce700 1 --2- 172.21.15.142:0/20020821 >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 0x7fb434132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 -- 172.21.15.142:0/20020821 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 msgr2=0x7fb43409f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4399cd700 1 --2- 172.21.15.142:0/20020821 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb430009f20 tx=0x7fb430009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1254effcc35c2a97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 1 -- 172.21.15.142:0/20020821 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000e070 con 0x7fb43409ece0 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: dump: 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.768 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: _finish_auth 0 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 5 monclient: authenticate success, global_id 75512 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 1 -- 172.21.15.142:0/20020821 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43000bc90 con 0x7fb43409ece0 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:16.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb43b1d0700 4 set_mon_vals no callback set 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 1 -- 172.21.15.142:0/20020821 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4300092e0 con 0x7fb43409ece0 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.754+0000 7fb4391cc700 10 monclient: dump: 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:16.770 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: shutdown 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- 172.21.15.142:0/20020821 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 msgr2=0x7fb43409f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/20020821 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb430009f20 tx=0x7fb430009840 comp rx=0 tx=0).stop 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- 172.21.15.142:0/20020821 shutdown_connections 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/20020821 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb43409ece0 0x7fb43409f0b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/20020821 >> v2:172.21.15.142:3300/0 conn(0x7fb434132030 0x7fb434132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- 172.21.15.142:0/20020821 >> 172.21.15.142:0/20020821 conn(0x7fb4340a5700 msgr2=0x7fb4340a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- 172.21.15.142:0/20020821 shutdown_connections 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- 172.21.15.142:0/20020821 wait complete. 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: build_initial_monmap 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: monmap: 2024-09-18T10:24:16.771 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 Processor -- start 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- start start 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: init 2024-09-18T10:24:16.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 0x7fb43406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-a con 0x7fb43409ae60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-c con 0x7fb434132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): picked mon.noname-b con 0x7fb434141c20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43411e4e0 con 0x7fb43409ae60 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43411e200 con 0x7fb434141c20 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:16.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43411e7c0 con 0x7fb434132030 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.758+0000 7fb43b1d0700 10 monclient(hunting): authenticate will time out at 1745.687962s 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 0x7fb43406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45050/0 (socket says 172.21.15.142:45050) 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 0x7fb43406e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35642/0 (socket says 172.21.15.142:35642) 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 -- 172.21.15.142:0/266905336 learned_addr learned my addr 172.21.15.142:0/266905336 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): get_auth_request con 0x7fb434132030 auth_method 0 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): get_auth_request con 0x7fb43409ae60 auth_method 0 2024-09-18T10:24:16.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 10 monclient(hunting): get_auth_request con 0x7fb434141c20 auth_method 0 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:16.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a1ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 msgr2=0x7fb43411bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 msgr2=0x7fb43411bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient(hunting): handle_auth_done global_id 75515 payload 824 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43a9cf700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 msgr2=0x7fb43411bd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 msgr2=0x7fb43406e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 0x7fb43406e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 monclient: found mon.noname-c 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:16.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 -- 172.21.15.142:0/266905336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb430009ae0 con 0x7fb434132030 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 msgr2=0x7fb4341310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4399cd700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb4340a1bd0 tx=0x7fb430017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4300070c0 con 0x7fb434132030 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: dump: 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:16.779 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43b1d0700 5 monclient: authenticate success, global_id 75515 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430012b00 con 0x7fb434132030 2024-09-18T10:24:16.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43b1d0700 10 monclient: _renew_subs 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43b1d0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb434142530 con 0x7fb434132030 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43001e430 con 0x7fb434132030 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb42a7fc700 10 monclient: dump: 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:16.781 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.762+0000 7fb4389cb700 4 set_mon_vals no callback set 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 10 monclient: _renew_subs 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43411d440 con 0x7fb434132030 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb43001e820 con 0x7fb434132030 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:16.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4340a4940 con 0x7fb434132030 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb42a7fc700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 0x7fb420037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb43004a880 con 0x7fb434132030 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43a1ce700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 0x7fb420037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.766+0000 7fb43a1ce700 10 monclient: get_auth_request con 0x7fb420035ac0 auth_method 0 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.770+0000 7fb43a1ce700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 0x7fb420037f70 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb424006f00 tx=0x7fb424008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.774+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb4340a4940 con 0x7fb434132030 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.774+0000 7fb42a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:16.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:16.774+0000 7fb42a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:17.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb4340a4940 con 0x7fb434132030 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb42a7fc700 1 -- 172.21.15.142:0/266905336 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb4340a4940 con 0x7fb434132030 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb42a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:17.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb42a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:17.008 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":142,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 msgr2=0x7fb420037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 0x7fb420037f70 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb424006f00 tx=0x7fb424008040 comp rx=0 tx=0).stop 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 10 monclient: shutdown 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.002+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 msgr2=0x7fb4341310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb4340a1bd0 tx=0x7fb430017000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 shutdown_connections 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb420035ac0 0x7fb420037f70 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb434141c20 0x7fb43411bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb434132030 0x7fb4341310e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 --2- 172.21.15.142:0/266905336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb43409ae60 0x7fb43406e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 >> 172.21.15.142:0/266905336 conn(0x7fb4340a5700 msgr2=0x7fb434135350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 shutdown_connections 2024-09-18T10:24:17.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.006+0000 7fb43b1d0700 1 -- 172.21.15.142:0/266905336 wait complete. 2024-09-18T10:24:17.021 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:24:17.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:24:17.022 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: get_monmap_and_config 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: monmap: 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:17.125028+0000 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:17.125028+0000 2024-09-18T10:24:17.127 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:17.128 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 Processor -- start 2024-09-18T10:24:17.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 -- start start 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: init 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-b con 0x7fcb3806f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb38098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 0x7fcb38136d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-a con 0x7fcb380992f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3809c330 con 0x7fcb38098f00 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3809b380 con 0x7fcb380992f0 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3809b030 con 0x7fcb3806f3b0 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3ebe6700 10 monclient(hunting): authenticate will time out at 1746.050644s 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 0x7fcb38136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3d3e3700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcb38098f00 0x7fcb38133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46942/0 (socket says 172.21.15.142:46942) 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 0x7fcb38136d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35658/0 (socket says 172.21.15.142:35658) 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3d3e3700 1 -- 172.21.15.142:0/142763394 learned_addr learned my addr 172.21.15.142:0/142763394 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45066/0 (socket says 172.21.15.142:45066) 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request con 0x7fcb3806f3b0 auth_method 0 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): get_auth_request con 0x7fcb380992f0 auth_method 0 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_done global_id 75518 payload 274 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/142763394 >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 msgr2=0x7fcb38136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/142763394 >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 0x7fcb38136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/142763394 >> v1:172.21.15.142:6789/0 conn(0x7fcb38098f00 legacy=0x7fcb38133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 monclient: found mon.noname-b 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.121+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/142763394 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3809c0f0 con 0x7fcb3806f3b0 2024-09-18T10:24:17.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/142763394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 msgr2=0x7fcb380989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/142763394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bb90 tx=0x7fcb2800b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=722486c02f84f0d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 1 -- 172.21.15.142:0/142763394 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800dd50 con 0x7fcb3806f3b0 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: dump: 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.140 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 5 monclient: authenticate success, global_id 75518 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 1 -- 172.21.15.142:0/142763394 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28010470 con 0x7fcb3806f3b0 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 4 set_mon_vals no callback set 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 1 -- 172.21.15.142:0/142763394 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28010830 con 0x7fcb3806f3b0 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3cbe2700 10 monclient: dump: 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 10 monclient: shutdown 2024-09-18T10:24:17.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/142763394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 msgr2=0x7fcb380989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/142763394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bb90 tx=0x7fcb2800b430 comp rx=0 tx=0).stop 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/142763394 shutdown_connections 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/142763394 >> v2:172.21.15.142:3300/0 conn(0x7fcb380992f0 0x7fcb38136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/142763394 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb3806f3b0 0x7fcb380989c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/142763394 >> 172.21.15.142:0/142763394 conn(0x7fcb38070ef0 msgr2=0x7fcb380712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/142763394 shutdown_connections 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.125+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/142763394 wait complete. 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient: monmap: 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.143 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 Processor -- start 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 -- start start 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient: init 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 0x7fcb3813d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-b con 0x7fcb3806f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 0x7fcb3813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-a con 0x7fcb38098f00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 0x7fcb3813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): picked mon.noname-c con 0x7fcb380992f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3808d090 con 0x7fcb38098f00 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3806fa20 con 0x7fcb3806f3b0 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3808b5d0 con 0x7fcb380992f0 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3ebe6700 10 monclient(hunting): authenticate will time out at 1746.057570s 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 0x7fcb3813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 0x7fcb3813dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35672/0 (socket says 172.21.15.142:35672) 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 0x7fcb3813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 1 -- 172.21.15.142:0/1594618207 learned_addr learned my addr 172.21.15.142:0/1594618207 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): get_auth_request con 0x7fcb38098f00 auth_method 0 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 0x7fcb3813d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): get_auth_request con 0x7fcb380992f0 auth_method 0 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request con 0x7fcb3806f3b0 auth_method 0 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 msgr2=0x7fcb3813d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:24:17.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 msgr2=0x7fcb3813d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 0x7fcb3813d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 0x7fcb3813d720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient(hunting): handle_auth_done global_id 75521 payload 824 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 msgr2=0x7fcb3813d720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb3806f3b0 0x7fcb3813d720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 msgr2=0x7fcb3813dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 0x7fcb3813dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 monclient: found mon.noname-c 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2800b6d0 con 0x7fcb380992f0 2024-09-18T10:24:17.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.129+0000 7fcb3d3e3700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb38098f00 0x7fcb3813dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 msgr2=0x7fcb3813e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3e3e5700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 0x7fcb3813e1a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009e50 tx=0x7fcb3000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3000cd60 con 0x7fcb380992f0 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: dump: 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.149 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 5 monclient: authenticate success, global_id 75521 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3000f040 con 0x7fcb380992f0 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 monclient: _renew_subs 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3808e3e0 con 0x7fcb380992f0 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300146d0 con 0x7fcb380992f0 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 10 monclient: dump: 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2ffff700 4 set_mon_vals no callback set 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 monclient: _renew_subs 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb380971d0 con 0x7fcb380992f0 2024-09-18T10:24:17.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 0x7fcb3805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380648a0 con 0x7fcb3808ac00 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 0x7fcb3805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient: get_auth_request con 0x7fcb3808ac00 auth_method 0 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcb30014a40 con 0x7fcb380992f0 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcb1c035ac0 0x7fcb1c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fcb30049cd0 con 0x7fcb380992f0 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_done global_id 164271 payload 824 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb3808c260 con 0x7fcb3808ac00 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3d3e3700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcb1c035ac0 0x7fcb1c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.133+0000 7fcb3d3e3700 10 monclient: get_auth_request con 0x7fcb1c035ac0 auth_method 0 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 msgr2=0x7fcb3805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 0x7fcb3805e890 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bb90 tx=0x7fcb2801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2801c030 con 0x7fcb3808ac00 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 10 monclient: dump: 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.157 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb3d3e3700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcb1c035ac0 0x7fcb1c037f70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcb3808b260 tx=0x7fcb34004a80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcb28018420 con 0x7fcb3808ac00 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 msgr2=0x7fcb3805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 0x7fcb3805e890 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bb90 tx=0x7fcb2801a000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 msgr2=0x7fcb3805e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.137+0000 7fcb2dffb700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3808ac00 0x7fcb3805e890 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3ebe6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 0x7fcb38029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3ebe6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.158 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3808d3a0 con 0x7fcb3805edd0 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 0x7fcb38029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient: get_auth_request con 0x7fcb3805edd0 auth_method 0 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.145+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb3dbe4700 10 monclient(hunting): handle_auth_done global_id 164274 payload 824 2024-09-18T10:24:17.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcb3806dad0 con 0x7fcb3805edd0 2024-09-18T10:24:17.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb3dbe4700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 msgr2=0x7fcb38029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb3dbe4700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 0x7fcb38029d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bfd0 tx=0x7fcb28023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb280319d0 con 0x7fcb3805edd0 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 10 monclient: dump: 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.161 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fcb2802d420 con 0x7fcb3805edd0 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 msgr2=0x7fcb38029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 0x7fcb38029d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800bfd0 tx=0x7fcb28023000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 msgr2=0x7fcb38029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.149+0000 7fcb2dffb700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb3805edd0 0x7fcb38029d00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:17.162 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.163 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.164 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:17.165 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.166 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.167 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.168 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.169 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.170 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.171 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:17.172 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.173 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:17.174 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:17.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcb1c035ac0 msgr2=0x7fcb1c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcb1c035ac0 0x7fcb1c037f70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcb3808b260 tx=0x7fcb34004a80 comp rx=0 tx=0).stop 2024-09-18T10:24:17.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 10 monclient: shutdown 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 msgr2=0x7fcb3813e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 --2- 172.21.15.142:0/1594618207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb380992f0 0x7fcb3813e1a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009e50 tx=0x7fcb3000b040 comp rx=0 tx=0).stop 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 reap_dead start 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3e3e5700 1 -- 172.21.15.142:0/1594618207 reap_dead start 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 shutdown_connections 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 >> 172.21.15.142:0/1594618207 conn(0x7fcb38070ef0 msgr2=0x7fcb380959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 shutdown_connections 2024-09-18T10:24:17.176 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.153+0000 7fcb3ebe6700 1 -- 172.21.15.142:0/1594618207 wait complete. 2024-09-18T10:24:17.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:17.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:24:17.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.273+0000 7efedbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.273+0000 7efedbfff700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.273+0000 7efedbfff700 10 monclient: monmap: 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.281 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:17.279301+0000 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:17.279301+0000 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:17.282 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 Processor -- start 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 -- start start 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient: init 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efedc09abd0 0x7efedc0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-a con 0x7efedc09abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efedc09b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-c con 0x7efedc09c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efedc09dc90 con 0x7efedc09b740 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efedc09b500 con 0x7efedc09abd0 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efedc09afa0 con 0x7efedc09c5e0 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedbfff700 10 monclient(hunting): authenticate will time out at 1746.205037s 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7efedc09abd0 0x7efedc0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efeda7fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7efedc09b740 0x7efedc09bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46946/0 (socket says 172.21.15.142:46946) 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efeda7fc700 1 -- 172.21.15.142:0/1524728346 learned_addr learned my addr 172.21.15.142:0/1524728346 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): get_auth_request con 0x7efedc09abd0 auth_method 0 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 1 --2- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 10 monclient(hunting): get_auth_request con 0x7efedc09c5e0 auth_method 0 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient(hunting): handle_auth_done global_id 164277 payload 274 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 1 -- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 msgr2=0x7efedc137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:24:17.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 1 -- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 msgr2=0x7efedc137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 1 --2- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 -- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 msgr2=0x7efedc137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedb7fe700 1 --2- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 --2- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 -- 172.21.15.142:0/1524728346 >> v1:172.21.15.142:6789/0 conn(0x7efedc09b740 legacy=0x7efedc09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 monclient: found mon.noname-a 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.277+0000 7efedaffd700 1 -- 172.21.15.142:0/1524728346 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc06f810 con 0x7efedc09abd0 2024-09-18T10:24:17.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedaffd700 1 -- 172.21.15.142:0/1524728346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 msgr2=0x7efedc0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedaffd700 1 --2- 172.21.15.142:0/1524728346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc0960e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efedc076de0 tx=0x7efed00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e05baf03c6cbaa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 1 -- 172.21.15.142:0/1524728346 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000b890 con 0x7efedc09abd0 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: dump: 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 5 monclient: authenticate success, global_id 164277 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 1 -- 172.21.15.142:0/1524728346 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed000b9f0 con 0x7efedc09abd0 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 4 set_mon_vals no callback set 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 1 -- 172.21.15.142:0/1524728346 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed000bd90 con 0x7efedc09abd0 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efed9ffb700 10 monclient: dump: 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 10 monclient: shutdown 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- 172.21.15.142:0/1524728346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 msgr2=0x7efedc0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 --2- 172.21.15.142:0/1524728346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc0960e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efedc076de0 tx=0x7efed00091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- 172.21.15.142:0/1524728346 shutdown_connections 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 --2- 172.21.15.142:0/1524728346 >> v2:172.21.15.150:3300/0 conn(0x7efedc09c5e0 0x7efedc137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 --2- 172.21.15.142:0/1524728346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc0960e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- 172.21.15.142:0/1524728346 >> 172.21.15.142:0/1524728346 conn(0x7efedc070ef0 msgr2=0x7efedc0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- 172.21.15.142:0/1524728346 shutdown_connections 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- 172.21.15.142:0/1524728346 wait complete. 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 10 monclient: monmap: 2024-09-18T10:24:17.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 Processor -- start 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 1 -- start start 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.281+0000 7efedbfff700 10 monclient: init 2024-09-18T10:24:17.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc13fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-a con 0x7efedc09abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 0x7efedc13ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-c con 0x7efedc09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 0x7efedc08bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): picked mon.noname-b con 0x7efedc09c5e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc08dc10 con 0x7efedc09abd0 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc09d240 con 0x7efedc09c5e0 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc0a0100 con 0x7efedc09b740 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedbfff700 10 monclient(hunting): authenticate will time out at 1746.212890s 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc13fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 0x7efedc13ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc13fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35712/0 (socket says 172.21.15.142:35712) 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 1 -- 172.21.15.142:0/3876530686 learned_addr learned my addr 172.21.15.142:0/3876530686 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): get_auth_request con 0x7efedc09abd0 auth_method 0 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): get_auth_request con 0x7efedc09b740 auth_method 0 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 0x7efedc08bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 10 monclient(hunting): get_auth_request con 0x7efedc09c5e0 auth_method 0 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 msgr2=0x7efedc08bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 msgr2=0x7efedc08bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 0x7efedc08bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedb7fe700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 0x7efedc08bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient(hunting): handle_auth_done global_id 75524 payload 824 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 msgr2=0x7efedc08bb00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efedc09c5e0 0x7efedc08bb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 msgr2=0x7efedc13fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc13fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 monclient: found mon.noname-c 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efeda7fc700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0009420 con 0x7efedc09b740 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.285+0000 7efedaffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efedc09abd0 0x7efedc13fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efeda7fc700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 msgr2=0x7efedc13ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efeda7fc700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 0x7efedc13ff90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efecc00c560 tx=0x7efecc007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc005d40 con 0x7efedc09b740 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: dump: 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.308 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 5 monclient: authenticate success, global_id 75524 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efecc0070d0 con 0x7efedc09b740 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 monclient: _renew_subs 2024-09-18T10:24:17.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efecc007490 con 0x7efedc09b740 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc08f150 con 0x7efedc09b740 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efed97fa700 4 set_mon_vals no callback set 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 10 monclient: dump: 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.310 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 monclient: _renew_subs 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc1404d0 con 0x7efedc09b740 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 0x7efedc08c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc0648a0 con 0x7efedc090420 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 0x7efedc08c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient: get_auth_request con 0x7efedc090420 auth_method 0 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7efecc01b3b0 con 0x7efedc09b740 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 10 monclient(hunting): handle_auth_done global_id 75527 payload 824 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedaffd700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efedc072a80 con 0x7efedc090420 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efec8035ac0 0x7efec8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7efecc048430 con 0x7efedc09b740 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedb7fe700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efec8035ac0 0x7efec8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.289+0000 7efedb7fe700 10 monclient: get_auth_request con 0x7efec8035ac0 auth_method 0 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efedaffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 msgr2=0x7efedc08c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efedaffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 0x7efedc08c040 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efedc095e60 tx=0x7efed0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0021070 con 0x7efedc090420 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 10 monclient: dump: 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.314 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.314 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7efed0000ca0 con 0x7efedc090420 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 msgr2=0x7efedc08c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 0x7efedc08c040 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efedc095e60 tx=0x7efed0017000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 msgr2=0x7efedc08c040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efec2ffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc090420 0x7efedc08c040 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7efedc095e60 tx=0x7efed0017000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.293+0000 7efedb7fe700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efec8035ac0 0x7efec8037f70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efed4009590 tx=0x7efed400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedbfff700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 0x7efedc08c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efedc0907f0 con 0x7efedc08c580 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 0x7efedc08c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient: get_auth_request con 0x7efedc08c580 auth_method 0 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 10 monclient(hunting): handle_auth_done global_id 75530 payload 824 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.305+0000 7efedaffd700 1 -- 172.21.15.142:0/3876530686 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efedc08cec0 con 0x7efedc08c580 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedaffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 msgr2=0x7efedc08c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedaffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 0x7efedc08c980 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efed0025000 tx=0x7efed0003050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed001a630 con 0x7efedc08c580 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 10 monclient: dump: 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.318 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7efed001a790 con 0x7efedc08c580 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 msgr2=0x7efedc08c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 0x7efedc08c980 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efed0025000 tx=0x7efed0003050 comp rx=0 tx=0).stop 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 msgr2=0x7efedc08c980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efec2ffd700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc08c580 0x7efedc08c980 secure :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efed0025000 tx=0x7efed0003050 comp rx=0 tx=0).stop 2024-09-18T10:24:17.319 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:24:17.320 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.321 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.322 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:17.323 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.324 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.325 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.326 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.327 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:17.328 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.329 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.330 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.331 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.332 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:17.333 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:17.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efec8035ac0 msgr2=0x7efec8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7efec8035ac0 0x7efec8037f70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efed4009590 tx=0x7efed400b040 comp rx=0 tx=0).stop 2024-09-18T10:24:17.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 10 monclient: shutdown 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 msgr2=0x7efedc13ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 0x7efedc13ff90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efecc00c560 tx=0x7efecc007f90 comp rx=0 tx=0).stop 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedb7fe700 1 -- 172.21.15.142:0/3876530686 reap_dead start 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 shutdown_connections 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 --2- 172.21.15.142:0/3876530686 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efedc09b740 0x7efedc13ff90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 >> 172.21.15.142:0/3876530686 conn(0x7efedc070ef0 msgr2=0x7efedc09def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 shutdown_connections 2024-09-18T10:24:17.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.309+0000 7efedbfff700 1 -- 172.21.15.142:0/3876530686 wait complete. 2024-09-18T10:24:17.335 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:17.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:24:17.336 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 10 monclient: get_monmap_and_config 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 10 monclient: monmap: 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:17.442868+0000 2024-09-18T10:24:17.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:17.442868+0000 2024-09-18T10:24:17.446 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.446 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.446 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:17.446 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:17.446 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:17.447 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:17.447 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:17.447 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 1 Processor -- start 2024-09-18T10:24:17.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 1 -- start start 2024-09-18T10:24:17.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.437+0000 7ff23d7d4700 10 monclient: init 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 0x7ff238098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-c con 0x7ff23809a830 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff23809ac00 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff23813b6e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff238099820 con 0x7ff23809ac00 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff238099d50 con 0x7ff23813b6e0 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23809dca0 con 0x7ff23809a830 2024-09-18T10:24:17.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff23d7d4700 10 monclient(hunting): authenticate will time out at 1746.368701s 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff22ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff23809ac00 0x7ff238099110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46950/0 (socket says 172.21.15.142:46950) 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 learned_addr learned my addr 172.21.15.142:0/2702604318 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 0x7ff238098bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 10 monclient(hunting): get_auth_request con 0x7ff23809a830 auth_method 0 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 1 -- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 msgr2=0x7ff238098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 1 -- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 msgr2=0x7ff238098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 0x7ff238098bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:17.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 0x7ff238098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 >> v1:172.21.15.142:6789/0 conn(0x7ff23809ac00 legacy=0x7ff238099110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2377fe700 1 -- 172.21.15.142:0/2702604318 >> v1:172.21.15.150:6789/0 conn(0x7ff23813b6e0 legacy=0x7ff23813dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32925 0 0) 0x7ff238099820 con 0x7ff23809ac00 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 10 monclient(hunting): my global_id is 164283 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224003570 con 0x7ff23809ac00 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.441+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3991474893 0 0) 0x7ff224003570 con 0x7ff23809ac00 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 msgr2=0x7ff238098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 --2- 172.21.15.142:0/2702604318 >> v2:172.21.15.150:3300/0 conn(0x7ff23809a830 0x7ff238098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 >> v1:172.21.15.150:6789/0 conn(0x7ff23813b6e0 legacy=0x7ff23813dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238099a70 con 0x7ff23809ac00 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff23d7d4700 5 monclient: authenticate success, global_id 164283 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff23d7d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff2280017a0 con 0x7ff23809ac00 2024-09-18T10:24:17.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: dump: 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.465 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.466 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:24:17.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 >> v1:172.21.15.142:6789/0 conn(0x7ff23809ac00 legacy=0x7ff238099110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff224003c50 0x7ff224004080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): picked mon.c con 0x7ff224003c50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff224004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): picked mon.a con 0x7ff224004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224009a50 0x7ff22400be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): picked mon.b con 0x7ff224009a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23809dca0 con 0x7ff224004810 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22400c420 con 0x7ff224009a50 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22400c5e0 con 0x7ff224003c50 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff224003c50 0x7ff224004080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff224004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:24:17.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): get_auth_request con 0x7ff224003c50 auth_method 0 2024-09-18T10:24:17.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): get_auth_request con 0x7ff224004810 auth_method 0 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224009a50 0x7ff22400be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff23d7d4700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): get_auth_request con 0x7ff224009a50 auth_method 0 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient(hunting): handle_auth_done global_id 164283 payload 293 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff224003c50 msgr2=0x7ff224004080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff224003c50 0x7ff224004080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224009a50 msgr2=0x7ff22400be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224009a50 0x7ff22400be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 monclient: found mon.a 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2377fe700 1 -- 172.21.15.142:0/2702604318 reap_dead start 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224003570 con 0x7ff224004810 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 -- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 msgr2=0x7ff224004c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff224004c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff2280014d0 tx=0x7ff23809a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fe2464d016f2fb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800ce40 con 0x7ff224004810 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.445+0000 7ff2367fc700 10 monclient: dump: 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.475 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228005920 con 0x7ff224004810 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 4 set_mon_vals no callback set 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 1 -- 172.21.15.142:0/2702604318 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228005cc0 con 0x7ff224004810 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff2367fc700 10 monclient: dump: 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 10 monclient: shutdown 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 msgr2=0x7ff224004c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff224004c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff2280014d0 tx=0x7ff23809a7c0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2702604318 shutdown_connections 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2702604318 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff224004c60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2702604318 >> 172.21.15.142:0/2702604318 conn(0x7ff2380a14e0 msgr2=0x7ff2380a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2702604318 shutdown_connections 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2702604318 wait complete. 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 10 monclient: monmap: 2024-09-18T10:24:17.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 Processor -- start 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 1 -- start start 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.449+0000 7ff23d7d4700 10 monclient: init 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff2381461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-a con 0x7ff224004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 0x7ff23806c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-c con 0x7ff238146710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 0x7ff23806cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): picked mon.noname-b con 0x7ff23806c6f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff238094830 con 0x7ff224004810 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380a0440 con 0x7ff23806c6f0 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23806dca0 con 0x7ff238146710 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff23d7d4700 10 monclient(hunting): authenticate will time out at 1746.380406s 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff2381461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 0x7ff23806c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 0x7ff23806c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45120/0 (socket says 172.21.15.142:45120) 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 -- 172.21.15.142:0/2579690658 learned_addr learned my addr 172.21.15.142:0/2579690658 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff2381461d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35730/0 (socket says 172.21.15.142:35730) 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): get_auth_request con 0x7ff238146710 auth_method 0 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): get_auth_request con 0x7ff224004810 auth_method 0 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 0x7ff23806cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 10 monclient(hunting): get_auth_request con 0x7ff23806c6f0 auth_method 0 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient(hunting): handle_auth_done global_id 75533 payload 824 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 msgr2=0x7ff23806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 msgr2=0x7ff23806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:17.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 msgr2=0x7ff23806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 0x7ff23806cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 0x7ff23806cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806c6f0 0x7ff23806cb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 msgr2=0x7ff2381461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff2381461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 monclient: found mon.noname-c 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238126990 con 0x7ff238146710 2024-09-18T10:24:17.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff224004810 0x7ff2381461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:17.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 msgr2=0x7ff23806c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.453+0000 7ff22ffff700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 0x7ff23806c1b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff2240057f0 tx=0x7ff228015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800ce40 con 0x7ff238146710 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: dump: 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.490 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 5 monclient: authenticate success, global_id 75533 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2280074f0 con 0x7ff238146710 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22801e430 con 0x7ff238146710 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 monclient: _renew_subs 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23806e870 con 0x7ff238146710 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff235ffb700 4 set_mon_vals no callback set 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 10 monclient: dump: 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 monclient: _renew_subs 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2381265b0 con 0x7ff238146710 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 0x7ff238092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff238064950 con 0x7ff23806d040 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff22801e7e0 con 0x7ff238146710 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 0x7ff238092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient: get_auth_request con 0x7ff23806d040 auth_method 0 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff218035b40 0x7ff218037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff22804a4f0 con 0x7ff238146710 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 10 monclient(hunting): handle_auth_done global_id 174100 payload 824 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff236ffd700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2380696a0 con 0x7ff23806d040 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff218035b40 0x7ff218037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.457+0000 7ff2377fe700 10 monclient: get_auth_request con 0x7ff218035b40 auth_method 0 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff236ffd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 msgr2=0x7ff238092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 0x7ff238092800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff23804d340 tx=0x7ff220008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220011e60 con 0x7ff23806d040 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 10 monclient: dump: 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.496 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff2377fe700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff218035b40 0x7ff218037ff0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff238065410 tx=0x7ff230001920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff220015840 con 0x7ff23806d040 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 msgr2=0x7ff238092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 0x7ff238092800 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff23804d340 tx=0x7ff220008040 comp rx=0 tx=0).stop 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 msgr2=0x7ff238092800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.461+0000 7ff22effd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff23806d040 0x7ff238092800 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff23d7d4700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 0x7ff238093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff23d7d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff238126b20 con 0x7ff238092d40 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 0x7ff238093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient: get_auth_request con 0x7ff238092d40 auth_method 0 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 10 monclient(hunting): handle_auth_done global_id 174103 payload 824 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.473+0000 7ff236ffd700 1 -- 172.21.15.142:0/2579690658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff238126d50 con 0x7ff238092d40 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff236ffd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 msgr2=0x7ff238093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff236ffd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 0x7ff238093140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff22001e000 tx=0x7ff220021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220027040 con 0x7ff238092d40 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 10 monclient: dump: 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7ff220015630 con 0x7ff238092d40 2024-09-18T10:24:17.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 msgr2=0x7ff238093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 0x7ff238093140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff22001e000 tx=0x7ff220021000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 msgr2=0x7ff238093140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.477+0000 7ff22effd700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff238092d40 0x7ff238093140 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff22001e000 tx=0x7ff220021000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.502 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:17.503 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:17.504 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.505 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.506 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:17.507 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:17.508 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.509 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:17.510 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.512 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:17.515 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:17.516 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:17.516 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:17.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff218035b40 msgr2=0x7ff218037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff218035b40 0x7ff218037ff0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff238065410 tx=0x7ff230001920 comp rx=0 tx=0).stop 2024-09-18T10:24:17.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 10 monclient: shutdown 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 msgr2=0x7ff23806c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 --2- 172.21.15.142:0/2579690658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff238146710 0x7ff23806c1b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff2240057f0 tx=0x7ff228015000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff2377fe700 1 -- 172.21.15.142:0/2579690658 reap_dead start 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff2377fe700 1 -- 172.21.15.142:0/2579690658 reap_dead start 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 shutdown_connections 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 >> 172.21.15.142:0/2579690658 conn(0x7ff2380a14e0 msgr2=0x7ff23809dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 shutdown_connections 2024-09-18T10:24:17.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.481+0000 7ff23d7d4700 1 -- 172.21.15.142:0/2579690658 wait complete. 2024-09-18T10:24:17.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:17.519 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:24:17.519 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:24:17.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.621+0000 7f4084a59700 10 monclient: get_monmap_and_config 2024-09-18T10:24:17.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.621+0000 7f4084a59700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.621+0000 7f4084a59700 10 monclient: monmap: 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:17.627223+0000 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:17.627223+0000 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.630 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:17.631 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:17.631 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:17.631 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:17.631 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:17.631 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 Processor -- start 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 -- start start 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient: init 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f408009abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4080097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f408009db80 0x7f40801386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-c con 0x7f408009db80 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40800958c0 con 0x7f4080097880 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4080095ef0 con 0x7f408009abd0 2024-09-18T10:24:17.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408006fa60 con 0x7f408009db80 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 10 monclient(hunting): authenticate will time out at 1746.552723s 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4077fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4080097880 0x7f408009c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46952/0 (socket says 172.21.15.142:46952) 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4077fff700 1 -- 172.21.15.142:0/3323036368 learned_addr learned my addr 172.21.15.142:0/3323036368 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 1 --2- 172.21.15.142:0/3323036368 >> v2:172.21.15.150:3300/0 conn(0x7f408009db80 0x7f40801386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): get_auth_request con 0x7f408009db80 auth_method 0 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4077fff700 1 -- 172.21.15.142:0/3323036368 >> v1:172.21.15.142:6789/0 conn(0x7f4080097880 legacy=0x7f408009c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888581595 0 0) 0x7f40800958c0 con 0x7f4080097880 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient(hunting): my global_id is 164289 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c003570 con 0x7f4080097880 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407e59c700 1 -- 172.21.15.142:0/3323036368 >> v1:172.21.15.150:6789/0 conn(0x7f408009abd0 legacy=0x7f408009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3692440031 0 0) 0x7f406c003570 con 0x7f4080097880 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 >> v2:172.21.15.150:3300/0 conn(0x7f408009db80 msgr2=0x7f40801386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 --2- 172.21.15.142:0/3323036368 >> v2:172.21.15.150:3300/0 conn(0x7f408009db80 0x7f40801386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 >> v1:172.21.15.150:6789/0 conn(0x7f408009abd0 legacy=0x7f408009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408009afa0 con 0x7f4080097880 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 5 monclient: authenticate success, global_id 164289 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f4084a59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f4068001860 con 0x7f4080097880 2024-09-18T10:24:17.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407ed9d700 1 --2- 172.21.15.142:0/3323036368 >> v2:172.21.15.150:3300/0 conn(0x7f408009db80 0x7f40801386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: dump: 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.650 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 >> v1:172.21.15.142:6789/0 conn(0x7f4080097880 legacy=0x7f408009c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 0x7f406c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 10 monclient(hunting): picked mon.c con 0x7f406c003b80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.625+0000 7f407dd9b700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f406c004810 0x7f406c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): picked mon.b con 0x7f406c004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c0053c0 0x7f406c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): picked mon.a con 0x7f406c0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c2f0 con 0x7f406c0053c0 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c460 con 0x7f406c004810 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00c620 con 0x7f406c003b80 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4084a59700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 0x7f406c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c0053c0 0x7f406c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): get_auth_request con 0x7f406c0053c0 auth_method 0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f406c004810 0x7f406c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): get_auth_request con 0x7f406c003b80 auth_method 0 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): get_auth_request con 0x7f406c004810 auth_method 0 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:24:17.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient(hunting): handle_auth_done global_id 164289 payload 293 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 -- 172.21.15.142:0/3323036368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f406c004810 msgr2=0x7f406c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f406c004810 0x7f406c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 -- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c0053c0 msgr2=0x7f406c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c0053c0 0x7f406c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 monclient: found mon.c 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407ed9d700 1 -- 172.21.15.142:0/3323036368 reap_dead start 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 -- 172.21.15.142:0/3323036368 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c003570 con 0x7f406c003b80 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 -- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 msgr2=0x7f406c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407e59c700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 0x7f406c003f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4070001670 tx=0x7f4070007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b58ed6e293fb2e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700070d0 con 0x7f406c003b80 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: dump: 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.656 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4084a59700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4070007250 con 0x7f406c003b80 2024-09-18T10:24:17.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 1 -- 172.21.15.142:0/3323036368 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070007400 con 0x7f406c003b80 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f407dd9b700 10 monclient: dump: 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.658 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4084a59700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.629+0000 7f4084a59700 4 set_mon_vals no callback set 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: shutdown 2024-09-18T10:24:17.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 msgr2=0x7f406c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 0x7f406c003f90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4070001670 tx=0x7f4070007d10 comp rx=0 tx=0).stop 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- 172.21.15.142:0/3323036368 shutdown_connections 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 --2- 172.21.15.142:0/3323036368 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f406c003b80 0x7f406c003f90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- 172.21.15.142:0/3323036368 >> 172.21.15.142:0/3323036368 conn(0x7f40800a14e0 msgr2=0x7f40800a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- 172.21.15.142:0/3323036368 shutdown_connections 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- 172.21.15.142:0/3323036368 wait complete. 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: build_initial_monmap 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: monmap: 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:17.660 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 Processor -- start 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- start start 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: init 2024-09-18T10:24:17.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-a con 0x7f406c003b80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 0x7f408008a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-c con 0x7f408008a1a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408008aab0 0x7f4080143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): picked mon.noname-b con 0x7f408008aab0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080097c90 con 0x7f406c003b80 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408008e300 con 0x7f408008aab0 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:17.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408009fcd0 con 0x7f408008a1a0 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4084a59700 10 monclient(hunting): authenticate will time out at 1746.563380s 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 0x7f408008a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35736/0 (socket says 172.21.15.142:35736) 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 1 -- 172.21.15.142:0/4289045320 learned_addr learned my addr 172.21.15.142:0/4289045320 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): get_auth_request con 0x7f406c003b80 auth_method 0 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): get_auth_request con 0x7f408008a1a0 auth_method 0 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407ed9d700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408008aab0 0x7f4080143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f4077fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407ed9d700 10 monclient(hunting): get_auth_request con 0x7f408008aab0 auth_method 0 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.633+0000 7f407ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 10 monclient(hunting): handle_auth_done global_id 164292 payload 824 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 -- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 msgr2=0x7f408008a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 0x7f408008a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 -- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408008aab0 msgr2=0x7f4080143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408008aab0 0x7f4080143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:24:17.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 monclient: found mon.noname-a 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 -- 172.21.15.142:0/4289045320 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408008d610 con 0x7f406c003b80 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4077fff700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 0x7f408008a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 -- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 msgr2=0x7f4080089c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407e59c700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4070008ff0 tx=0x7f4070016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070020070 con 0x7f406c003b80 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: dump: 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: _finish_auth 0 2024-09-18T10:24:17.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:17.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40700070d0 con 0x7f406c003b80 2024-09-18T10:24:17.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 5 monclient: authenticate success, global_id 164292 2024-09-18T10:24:17.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 10 monclient: _renew_subs 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070011ce0 con 0x7f406c003b80 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408008ba30 con 0x7f406c003b80 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f407d59a700 4 set_mon_vals no callback set 2024-09-18T10:24:17.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f40767fc700 10 monclient: dump: 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:17.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:17.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:17.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 10 monclient: _renew_subs 2024-09-18T10:24:17.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.637+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408008d050 con 0x7f406c003b80 2024-09-18T10:24:17.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4084a59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4084a59700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4084a59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408006db60 con 0x7f406c003b80 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4070017020 con 0x7f406c003b80 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f40767fc700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 0x7f4060037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f407004cc40 con 0x7f406c003b80 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4077fff700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 0x7f4060037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.641+0000 7f4077fff700 10 monclient: get_auth_request con 0x7f4060035ac0 auth_method 0 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.645+0000 7f4077fff700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 0x7f4060037f70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f40800a0e40 tx=0x7f4068007c30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.649+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f407001a420 con 0x7f406c003b80 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.649+0000 7f40767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:17.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.649+0000 7f40767fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f4084a59700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:24:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f4084a59700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:24:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f4084a59700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f40800696a0 con 0x7f406c003b80 2024-09-18T10:24:17.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f40767fc700 1 -- 172.21.15.142:0/4289045320 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f40700172d0 con 0x7f406c003b80 2024-09-18T10:24:17.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f40767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:24:17.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.873+0000 7f40767fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:17.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 msgr2=0x7f4060037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 0x7f4060037f70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f40800a0e40 tx=0x7f4068007c30 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 10 monclient: shutdown 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 msgr2=0x7f4080089c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4070008ff0 tx=0x7f4070016000 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 shutdown_connections 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4060035ac0 0x7f4060037f70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f408008aab0 0x7f4080143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f408008a1a0 0x7f408008a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 --2- 172.21.15.142:0/4289045320 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f406c003b80 0x7f4080089c60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 >> 172.21.15.142:0/4289045320 conn(0x7f40800a14e0 msgr2=0x7f4080098c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 shutdown_connections 2024-09-18T10:24:17.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:17.877+0000 7f4084a59700 1 -- 172.21.15.142:0/4289045320 wait complete. 2024-09-18T10:24:17.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:24:18.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=8,logm=47} crc {auth=3901282598,config=55734535,health=3687809884,kv=4261215878,logm=1541808181}) 2024-09-18T10:24:18.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=11,kv=8,logm=48} crc {auth=2028530482,config=3140514009,health=3468056279,kv=4261215878,logm=1461269165}) 2024-09-18T10:24:18.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=8,logm=47} crc {auth=3901282598,config=55734535,health=3687809884,kv=4261215878,logm=1541808181}) 2024-09-18T10:24:18.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.165+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=908348339,config=3140514009,health=3468056279,kv=4261215878,logm=551731338}) 2024-09-18T10:24:18.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981287618}) 2024-09-18T10:24:18.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690133535}) 2024-09-18T10:24:18.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981287618}) 2024-09-18T10:24:18.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.173+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967921756}) 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709907855}) 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966974668}) 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709907855}) 2024-09-18T10:24:18.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.309+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3768686164}) 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886245276}) 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657463605}) 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886245276}) 2024-09-18T10:24:18.318 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1888958085}) 2024-09-18T10:24:18.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603984405}) 2024-09-18T10:24:18.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663600564}) 2024-09-18T10:24:18.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603984405}) 2024-09-18T10:24:18.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.313+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511805469}) 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706248904}) 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4001045632}) 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706248904}) 2024-09-18T10:24:18.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916558286}) 2024-09-18T10:24:18.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492507420}) 2024-09-18T10:24:18.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639804794}) 2024-09-18T10:24:18.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:18.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492507420}) 2024-09-18T10:24:18.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:18.317+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479224062}) 2024-09-18T10:24:18.895 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:24:18.895 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: get_monmap_and_config 2024-09-18T10:24:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: monmap: 2024-09-18T10:24:19.006 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:19.004096+0000 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:19.004096+0000 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:19.007 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:19.008 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 Processor -- start 2024-09-18T10:24:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 -- start start 2024-09-18T10:24:19.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: init 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 0x7f2c6c070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c0a2e40 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c6c071430 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 0x7f2c6c139e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c071830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c09ec10 con 0x7f2c6c071430 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c09a340 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c09d370 con 0x7f2c6c071830 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c725e4700 10 monclient(hunting): authenticate will time out at 1747.929536s 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c70de1700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2c6c071430 0x7f2c6c136700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46954/0 (socket says 172.21.15.142:46954) 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c70de1700 1 -- 172.21.15.142:0/3797517845 learned_addr learned my addr 172.21.15.142:0/3797517845 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 --2- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 0x7f2c6c139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 1 --2- 172.21.15.142:0/3797517845 >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 0x7f2c6c070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.015 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): get_auth_request con 0x7f2c6c0a2e40 auth_method 0 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): get_auth_request con 0x7f2c6c071830 auth_method 0 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.016 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient(hunting): handle_auth_done global_id 75539 payload 274 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 -- 172.21.15.142:0/3797517845 >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 msgr2=0x7f2c6c070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 --2- 172.21.15.142:0/3797517845 >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 0x7f2c6c070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 -- 172.21.15.142:0/3797517845 >> v1:172.21.15.142:6789/0 conn(0x7f2c6c071430 legacy=0x7f2c6c136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 monclient: found mon.noname-b 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c71de3700 1 -- 172.21.15.142:0/3797517845 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c09e930 con 0x7f2c6c071830 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.001+0000 7f2c715e2700 1 --2- 172.21.15.142:0/3797517845 >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 0x7f2c6c070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:19.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c71de3700 1 -- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 msgr2=0x7f2c6c139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c71de3700 1 --2- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 0x7f2c6c139e30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098e0 tx=0x7f2c5c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e1b585f52211cb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 1 -- 172.21.15.142:0/3797517845 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00b850 con 0x7f2c6c071830 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: dump: 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.018 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 5 monclient: authenticate success, global_id 75539 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 1 -- 172.21.15.142:0/3797517845 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c00b9b0 con 0x7f2c6c071830 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 4 set_mon_vals no callback set 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 1 -- 172.21.15.142:0/3797517845 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c00bd50 con 0x7f2c6c071830 2024-09-18T10:24:19.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c63fff700 10 monclient: dump: 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 10 monclient: shutdown 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 -- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 msgr2=0x7f2c6c139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 --2- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 0x7f2c6c139e30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098e0 tx=0x7f2c5c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 -- 172.21.15.142:0/3797517845 shutdown_connections 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 --2- 172.21.15.142:0/3797517845 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c071830 0x7f2c6c139e30 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 --2- 172.21.15.142:0/3797517845 >> v2:172.21.15.142:3300/0 conn(0x7f2c6c0a2e40 0x7f2c6c070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 -- 172.21.15.142:0/3797517845 >> 172.21.15.142:0/3797517845 conn(0x7f2c6c06a2e0 msgr2=0x7f2c6c0a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 -- 172.21.15.142:0/3797517845 shutdown_connections 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 1 -- 172.21.15.142:0/3797517845 wait complete. 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.005+0000 7f2c725e4700 10 monclient: monmap: 2024-09-18T10:24:19.021 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:19.022 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 Processor -- start 2024-09-18T10:24:19.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 -- start start 2024-09-18T10:24:19.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient: init 2024-09-18T10:24:19.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6c071430 0x7f2c6c1417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-b con 0x7f2c6c071430 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-a con 0x7f2c6c071830 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): picked mon.noname-c con 0x7f2c6c0a2e40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c09e4e0 con 0x7f2c6c071830 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c08f000 con 0x7f2c6c071430 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0a2460 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c725e4700 10 monclient(hunting): authenticate will time out at 1747.938002s 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45168/0 (socket says 172.21.15.142:45168) 2024-09-18T10:24:19.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35756/0 (socket says 172.21.15.142:35756) 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 -- 172.21.15.142:0/1788797089 learned_addr learned my addr 172.21.15.142:0/1788797089 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): get_auth_request con 0x7f2c6c0a2e40 auth_method 0 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): get_auth_request con 0x7f2c6c071830 auth_method 0 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6c071430 0x7f2c6c1417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): get_auth_request con 0x7f2c6c071430 auth_method 0 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c715e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient(hunting): handle_auth_done global_id 75542 payload 824 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 -- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6c071430 msgr2=0x7f2c6c1417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6c071430 0x7f2c6c1417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 -- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 msgr2=0x7f2c6c141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 monclient: found mon.noname-c 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c71de3700 1 -- 172.21.15.142:0/1788797089 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009420 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.009+0000 7f2c70de1700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c71de3700 1 -- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 msgr2=0x7f2c6c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c71de3700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0091e0 tx=0x7f2c5c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c01b470 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: dump: 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.034 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 5 monclient: authenticate success, global_id 75542 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _renew_subs 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c01b5d0 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5c009420 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c01b970 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c62ffd700 4 set_mon_vals no callback set 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 10 monclient: dump: 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.036 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _renew_subs 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c0905b0 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2c5c01bd20 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c072a80 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 0x7f2c58037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.013+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2c5c04abf0 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.017+0000 7f2c715e2700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 0x7f2c58037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.017+0000 7f2c715e2700 10 monclient: get_auth_request con 0x7f2c58035ac0 auth_method 0 2024-09-18T10:24:19.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.017+0000 7f2c715e2700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 0x7f2c58037f70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009f40 tx=0x7f2c640091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.025+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2c5c018950 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.025+0000 7f2c617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.025+0000 7f2c617fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:19.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394208353}) 2024-09-18T10:24:19.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4017186485}) 2024-09-18T10:24:19.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394208353}) 2024-09-18T10:24:19.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.113+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629838668}) 2024-09-18T10:24:19.122 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166918903}) 2024-09-18T10:24:19.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3022380751}) 2024-09-18T10:24:19.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166918903}) 2024-09-18T10:24:19.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733443058}) 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330740100}) 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655345057}) 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330740100}) 2024-09-18T10:24:19.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.117+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337305062}) 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860896558}) 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875928204}) 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860896558}) 2024-09-18T10:24:19.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.169+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993188022}) 2024-09-18T10:24:19.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427053608}) 2024-09-18T10:24:19.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3292576541}) 2024-09-18T10:24:19.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427053608}) 2024-09-18T10:24:19.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.177+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781827908}) 2024-09-18T10:24:19.195 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527832567}) 2024-09-18T10:24:19.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201102580}) 2024-09-18T10:24:19.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527832567}) 2024-09-18T10:24:19.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.189+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300684385}) 2024-09-18T10:24:19.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.249+0000 7f2c725e4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:19.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.249+0000 7f2c725e4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:19.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.249+0000 7f2c725e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.249+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2c6c08eaf0 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c617fa700 1 -- 172.21.15.142:0/1788797089 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2c5c02d350 con 0x7f2c6c0a2e40 2024-09-18T10:24:19.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:19.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c617fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:19.259 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":142,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 msgr2=0x7f2c58037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 0x7f2c58037f70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009f40 tx=0x7f2c640091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 10 monclient: shutdown 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 msgr2=0x7f2c6c142250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0091e0 tx=0x7f2c5c016000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 shutdown_connections 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2c58035ac0 0x7f2c58037f70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c6c0a2e40 0x7f2c6c142250 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c6c071830 0x7f2c6c141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 --2- 172.21.15.142:0/1788797089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6c071430 0x7f2c6c1417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 >> 172.21.15.142:0/1788797089 conn(0x7f2c6c06a2e0 msgr2=0x7f2c6c09b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 shutdown_connections 2024-09-18T10:24:19.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.253+0000 7f2c725e4700 1 -- 172.21.15.142:0/1788797089 wait complete. 2024-09-18T10:24:19.272 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:24:19.273 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:24:19.273 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:24:19.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 10 monclient: get_monmap_and_config 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 10 monclient: monmap: 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:19.373797+0000 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:19.373797+0000 2024-09-18T10:24:19.376 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:19.377 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 1 Processor -- start 2024-09-18T10:24:19.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 1 -- start start 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.369+0000 7f20317c6700 10 monclient: init 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f202c070ef0 0x7f202c09d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-c con 0x7f202c070ef0 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f202c0712c0 0x7f202c071690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-a con 0x7f202c0712c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f202c134670 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202c09db90 con 0x7f202c134670 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c06fcc0 con 0x7f202c0712c0 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c068360 con 0x7f202c070ef0 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient(hunting): authenticate will time out at 1748.300287s 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f202c0712c0 0x7f202c071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f202c0712c0 0x7f202c071690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35758/0 (socket says 172.21.15.142:35758) 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 -- 172.21.15.142:0/481265718 learned_addr learned my addr 172.21.15.142:0/481265718 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): get_auth_request con 0x7f202c0712c0 auth_method 0 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202affd700 1 --2- 172.21.15.142:0/481265718 >> v2:172.21.15.150:3300/0 conn(0x7f202c070ef0 0x7f202c09d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202affd700 10 monclient(hunting): get_auth_request con 0x7f202c070ef0 auth_method 0 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient(hunting): handle_auth_done global_id 164301 payload 274 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 -- 172.21.15.142:0/481265718 >> v2:172.21.15.150:3300/0 conn(0x7f202c070ef0 msgr2=0x7f202c09d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 --2- 172.21.15.142:0/481265718 >> v2:172.21.15.150:3300/0 conn(0x7f202c070ef0 0x7f202c09d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 -- 172.21.15.142:0/481265718 >> v1:172.21.15.150:6789/0 conn(0x7f202c134670 legacy=0x7f202c136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 monclient: found mon.noname-a 2024-09-18T10:24:19.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 -- 172.21.15.142:0/481265718 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c09de70 con 0x7f202c0712c0 2024-09-18T10:24:19.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 -- 172.21.15.142:0/481265718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 msgr2=0x7f202c071690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f202a7fc700 1 --2- 172.21.15.142:0/481265718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c071690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f202c09da90 tx=0x7f2020009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28b8668021ed665 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 1 -- 172.21.15.142:0/481265718 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000ba40 con 0x7f202c0712c0 2024-09-18T10:24:19.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: dump: 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.391 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 5 monclient: authenticate success, global_id 164301 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 1 -- 172.21.15.142:0/481265718 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202000bba0 con 0x7f202c0712c0 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f20317c6700 4 set_mon_vals no callback set 2024-09-18T10:24:19.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 1 -- 172.21.15.142:0/481265718 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020011420 con 0x7f202c0712c0 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.373+0000 7f2029ffb700 10 monclient: dump: 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.393 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: shutdown 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- 172.21.15.142:0/481265718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 msgr2=0x7f202c071690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- 172.21.15.142:0/481265718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c071690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f202c09da90 tx=0x7f2020009250 comp rx=0 tx=0).stop 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- 172.21.15.142:0/481265718 shutdown_connections 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- 172.21.15.142:0/481265718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c071690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- 172.21.15.142:0/481265718 >> v2:172.21.15.150:3300/0 conn(0x7f202c070ef0 0x7f202c09d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- 172.21.15.142:0/481265718 >> 172.21.15.142:0/481265718 conn(0x7f202c0a24f0 msgr2=0x7f202c0a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- 172.21.15.142:0/481265718 shutdown_connections 2024-09-18T10:24:19.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- 172.21.15.142:0/481265718 wait complete. 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: monmap: 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:19.395 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 Processor -- start 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- start start 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: init 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 0x7f202c141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-c con 0x7f202c070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-a con 0x7f202c0712c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:19.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f202c134670 0x7f202c0999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): picked mon.noname-b con 0x7f202c134670 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c097ef0 con 0x7f202c0712c0 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c12fab0 con 0x7f202c134670 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c130910 con 0x7f202c070ef0 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f20317c6700 10 monclient(hunting): authenticate will time out at 1748.306855s 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 0x7f202c141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35768/0 (socket says 172.21.15.142:35768) 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 0x7f202c141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45172/0 (socket says 172.21.15.142:45172) 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 -- 172.21.15.142:0/3005625325 learned_addr learned my addr 172.21.15.142:0/3005625325 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): get_auth_request con 0x7f202c0712c0 auth_method 0 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): get_auth_request con 0x7f202c070ef0 auth_method 0 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202b7fe700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f202c134670 0x7f202c0999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202b7fe700 10 monclient(hunting): get_auth_request con 0x7f202c134670 auth_method 0 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient(hunting): handle_auth_done global_id 164304 payload 824 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 msgr2=0x7f202c141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 0x7f202c141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f202c134670 msgr2=0x7f202c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f202c134670 0x7f202c0999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 monclient: found mon.noname-a 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.400 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202a7fc700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20200094f0 con 0x7f202c0712c0 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.377+0000 7f202affd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f202c070ef0 0x7f202c141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202a7fc700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 msgr2=0x7f202c142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202a7fc700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c142280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2020006fd0 tx=0x7f2020017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202001f070 con 0x7f202c0712c0 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: dump: 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.401 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 5 monclient: authenticate success, global_id 164304 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020011e00 con 0x7f202c0712c0 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 monclient: _renew_subs 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c130bf0 con 0x7f202c0712c0 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202001a8a0 con 0x7f202c0712c0 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f2012ffd700 10 monclient: dump: 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20297fa700 4 set_mon_vals no callback set 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 monclient: _renew_subs 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c142960 con 0x7f202c0712c0 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 0x7f202c099f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c064a50 con 0x7f202c09b470 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202affd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 0x7f202c099f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202affd700 10 monclient: get_auth_request con 0x7f202c09b470 auth_method 0 2024-09-18T10:24:19.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.381+0000 7f202affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f202001ac10 con 0x7f202c0712c0 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 10 monclient(hunting): handle_auth_done global_id 164307 payload 824 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f202c0696a0 con 0x7f202c09b470 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2018035ac0 0x7f2018037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f202004a5f0 con 0x7f202c0712c0 2024-09-18T10:24:19.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202b7fe700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2018035ac0 0x7f2018037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202b7fe700 10 monclient: get_auth_request con 0x7f2018035ac0 auth_method 0 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 msgr2=0x7f202c099f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202affd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 0x7f202c099f10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f201c010cb0 tx=0x7f201c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c014470 con 0x7f202c09b470 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 10 monclient: dump: 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f201c0145d0 con 0x7f202c09b470 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 msgr2=0x7f202c099f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 0x7f202c099f10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f201c010cb0 tx=0x7f201c008040 comp rx=0 tx=0).stop 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 msgr2=0x7f202c099f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f2012ffd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09b470 0x7f202c099f10 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f201c010cb0 tx=0x7f201c008040 comp rx=0 tx=0).stop 2024-09-18T10:24:19.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f202b7fe700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2018035ac0 0x7f2018037f70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2024007fd0 tx=0x7f2024009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.385+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.389+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618390661}) 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.389+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984500399}) 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.389+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.389+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618390661}) 2024-09-18T10:24:19.411 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.389+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532930821}) 2024-09-18T10:24:19.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f20317c6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:19.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f20317c6700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 0x7f202c09a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f20317c6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c097a40 con 0x7f202c09a450 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 0x7f202c09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient: get_auth_request con 0x7f202c09a450 auth_method 0 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 10 monclient(hunting): handle_auth_done global_id 164310 payload 824 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.397+0000 7f202affd700 1 -- 172.21.15.142:0/3005625325 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f202c067fa0 con 0x7f202c09a450 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f202affd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 msgr2=0x7f202c09a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f202affd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 0x7f202c09a850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f201c019000 tx=0x7f201c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c020440 con 0x7f202c09a450 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 10 monclient: dump: 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.415 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f201c0205a0 con 0x7f202c09a450 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:19.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 msgr2=0x7f202c09a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 0x7f202c09a850 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f201c019000 tx=0x7f201c01c000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 msgr2=0x7f202c09a850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f2012ffd700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c09a450 0x7f202c09a850 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f201c019000 tx=0x7f201c01c000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:19.417 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.418 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.419 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.420 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:19.421 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.422 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:19.423 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.424 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.425 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.426 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.427 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.428 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.429 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:19.430 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.431 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.432 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:19.432 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:19.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2018035ac0 msgr2=0x7f2018037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2018035ac0 0x7f2018037f70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2024007fd0 tx=0x7f2024009040 comp rx=0 tx=0).stop 2024-09-18T10:24:19.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 10 monclient: shutdown 2024-09-18T10:24:19.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 msgr2=0x7f202c142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 --2- 172.21.15.142:0/3005625325 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f202c0712c0 0x7f202c142280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2020006fd0 tx=0x7f2020017000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f202b7fe700 1 -- 172.21.15.142:0/3005625325 reap_dead start 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f202b7fe700 1 -- 172.21.15.142:0/3005625325 reap_dead start 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 shutdown_connections 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 >> 172.21.15.142:0/3005625325 conn(0x7f202c0a24f0 msgr2=0x7f202c0966d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 shutdown_connections 2024-09-18T10:24:19.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.401+0000 7f20317c6700 1 -- 172.21.15.142:0/3005625325 wait complete. 2024-09-18T10:24:19.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:19.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:24:19.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: get_monmap_and_config 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: monmap: 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:19.545887+0000 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:19.545887+0000 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.548 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:19.549 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:19.549 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:19.549 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:19.549 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:19.549 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 Processor -- start 2024-09-18T10:24:19.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 -- start start 2024-09-18T10:24:19.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: init 2024-09-18T10:24:19.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa14809a6a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa14806f7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa14813c6f0 0x7fa14813ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa14813c6f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14809dae0 con 0x7fa14809a6a0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa148099bc0 con 0x7fa14806f7f0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1480998e0 con 0x7fa14813c6f0 2024-09-18T10:24:19.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa147fff700 10 monclient(hunting): authenticate will time out at 1748.471381s 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa14813c6f0 0x7fa14813ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa14813c6f0 0x7fa14813ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35782/0 (socket says 172.21.15.142:35782) 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 1 -- 172.21.15.142:0/80318278 learned_addr learned my addr 172.21.15.142:0/80318278 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): get_auth_request con 0x7fa14813c6f0 auth_method 0 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa146ffd700 1 --1- 172.21.15.142:0/80318278 >> v1:172.21.15.142:6789/0 conn(0x7fa14809a6a0 0x7fa14809aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46966/0 (socket says 172.21.15.142:46966) 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.541+0000 7fa1477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 10 monclient(hunting): handle_auth_done global_id 164313 payload 274 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 -- 172.21.15.142:0/80318278 >> v1:172.21.15.150:6789/0 conn(0x7fa14806f7f0 legacy=0x7fa14809b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 -- 172.21.15.142:0/80318278 >> v1:172.21.15.142:6789/0 conn(0x7fa14809a6a0 legacy=0x7fa14809aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 monclient: found mon.noname-a 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 -- 172.21.15.142:0/80318278 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14809b890 con 0x7fa14813c6f0 2024-09-18T10:24:19.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 -- 172.21.15.142:0/80318278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14813c6f0 msgr2=0x7fa14813ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa1477fe700 1 --2- 172.21.15.142:0/80318278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14813c6f0 0x7fa14813ead0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1380099e0 tx=0x7fa138009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25c3e70e2a0f5df server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 1 -- 172.21.15.142:0/80318278 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13800bb10 con 0x7fa14813c6f0 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: dump: 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.562 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 5 monclient: authenticate success, global_id 164313 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 1 -- 172.21.15.142:0/80318278 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13800bc70 con 0x7fa14813c6f0 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 4 set_mon_vals no callback set 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 1 -- 172.21.15.142:0/80318278 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa138011420 con 0x7fa14813c6f0 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa145ffb700 10 monclient: dump: 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.564 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 10 monclient: shutdown 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 -- 172.21.15.142:0/80318278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14813c6f0 msgr2=0x7fa14813ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 --2- 172.21.15.142:0/80318278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14813c6f0 0x7fa14813ead0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa1380099e0 tx=0x7fa138009250 comp rx=0 tx=0).stop 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 -- 172.21.15.142:0/80318278 shutdown_connections 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 --2- 172.21.15.142:0/80318278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14813c6f0 0x7fa14813ead0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 -- 172.21.15.142:0/80318278 >> 172.21.15.142:0/80318278 conn(0x7fa148070ef0 msgr2=0x7fa1480712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 -- 172.21.15.142:0/80318278 shutdown_connections 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.545+0000 7fa147fff700 1 -- 172.21.15.142:0/80318278 wait complete. 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient: monmap: 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.566 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 Processor -- start 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 -- start start 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient: init 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14806f7f0 0x7fa14813c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa14806f7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 0x7fa148092ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa14809a6a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa14813c6f0 0x7fa148093010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa14813c6f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480a04c0 con 0x7fa14806f7f0 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148093f60 con 0x7fa14813c6f0 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148094800 con 0x7fa14809a6a0 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa147fff700 10 monclient(hunting): authenticate will time out at 1748.477684s 2024-09-18T10:24:19.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 0x7fa148092ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 0x7fa148092ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45188/0 (socket says 172.21.15.142:45188) 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 -- 172.21.15.142:0/2347354776 learned_addr learned my addr 172.21.15.142:0/2347354776 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): get_auth_request con 0x7fa14809a6a0 auth_method 0 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14806f7f0 0x7fa14813c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): get_auth_request con 0x7fa14806f7f0 auth_method 0 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1477fe700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa14813c6f0 0x7fa148093010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1477fe700 10 monclient(hunting): get_auth_request con 0x7fa14813c6f0 auth_method 0 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1477fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient(hunting): handle_auth_done global_id 75548 payload 824 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa14813c6f0 msgr2=0x7fa148093010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa14813c6f0 0x7fa148093010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14806f7f0 msgr2=0x7fa14813c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa14806f7f0 0x7fa14813c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 monclient: found mon.noname-c 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.549+0000 7fa1467fc700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380094f0 con 0x7fa14809a6a0 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa1467fc700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 msgr2=0x7fa148092ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa1467fc700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 0x7fa148092ad0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c980 tx=0x7fa140002760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000ce70 con 0x7fa14809a6a0 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: dump: 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.572 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14000a930 con 0x7fa14809a6a0 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 5 monclient: authenticate success, global_id 75548 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14000ad10 con 0x7fa14809a6a0 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 monclient: _renew_subs 2024-09-18T10:24:19.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148137ce0 con 0x7fa14809a6a0 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa1457fa700 4 set_mon_vals no callback set 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 10 monclient: dump: 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 monclient: _renew_subs 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1480944f0 con 0x7fa14809a6a0 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 0x7fa1480973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1480648a0 con 0x7fa148093550 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa140012630 con 0x7fa14809a6a0 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 0x7fa1480973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient: get_auth_request con 0x7fa148093550 auth_method 0 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa134035ac0 0x7fa134037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa140048cc0 con 0x7fa14809a6a0 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 10 monclient(hunting): handle_auth_done global_id 75551 payload 824 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa146ffd700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa148093920 con 0x7fa148093550 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa1477fe700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa134035ac0 0x7fa134037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.553+0000 7fa1477fe700 10 monclient: get_auth_request con 0x7fa134035ac0 auth_method 0 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa146ffd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 msgr2=0x7fa1480973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa146ffd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 0x7fa1480973a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e740 tx=0x7fa13c007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c005260 con 0x7fa148093550 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 10 monclient: dump: 2024-09-18T10:24:19.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fa13c0053c0 con 0x7fa148093550 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 msgr2=0x7fa1480973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 0x7fa1480973a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e740 tx=0x7fa13c007ed0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 msgr2=0x7fa1480973a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa12effd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa148093550 0x7fa1480973a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.557+0000 7fa1477fe700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa134035ac0 0x7fa134037f70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1380099e0 tx=0x7fa138016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa147fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa147fff700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 0x7fa148097ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa147fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1481398c0 con 0x7fa1480978e0 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 0x7fa148097ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient: get_auth_request con 0x7fa1480978e0 auth_method 0 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 10 monclient(hunting): handle_auth_done global_id 75554 payload 824 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.565+0000 7fa146ffd700 1 -- 172.21.15.142:0/2347354776 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa148139670 con 0x7fa1480978e0 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa146ffd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 msgr2=0x7fa148097ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa146ffd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 0x7fa148097ce0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa13c017000 tx=0x7fa13c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c01c610 con 0x7fa1480978e0 2024-09-18T10:24:19.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 10 monclient: dump: 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.584 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7fa13c004ee0 con 0x7fa1480978e0 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 msgr2=0x7fa148097ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 0x7fa148097ce0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa13c017000 tx=0x7fa13c018000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 msgr2=0x7fa148097ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.569+0000 7fa12effd700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa1480978e0 0x7fa148097ce0 secure :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa13c017000 tx=0x7fa13c018000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.585 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.588 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.589 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.590 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:19.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.593 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.595 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.596 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.597 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:19.598 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:24:19.599 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:19.600 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:19.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa134035ac0 msgr2=0x7fa134037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa134035ac0 0x7fa134037f70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1380099e0 tx=0x7fa138016000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 10 monclient: shutdown 2024-09-18T10:24:19.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 msgr2=0x7fa148092ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 --2- 172.21.15.142:0/2347354776 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa14809a6a0 0x7fa148092ad0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c980 tx=0x7fa140002760 comp rx=0 tx=0).stop 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa1477fe700 1 -- 172.21.15.142:0/2347354776 reap_dead start 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa1477fe700 1 -- 172.21.15.142:0/2347354776 reap_dead start 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 shutdown_connections 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 >> 172.21.15.142:0/2347354776 conn(0x7fa148070ef0 msgr2=0x7fa148136790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 shutdown_connections 2024-09-18T10:24:19.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.573+0000 7fa147fff700 1 -- 172.21.15.142:0/2347354776 wait complete. 2024-09-18T10:24:19.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:19.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:24:19.604 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:24:19.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828237303}) 2024-09-18T10:24:19.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598479607}) 2024-09-18T10:24:19.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.718 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828237303}) 2024-09-18T10:24:19.719 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.713+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206352205}) 2024-09-18T10:24:19.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: get_monmap_and_config 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: monmap: 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:19.719781+0000 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:19.719781+0000 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:19.722 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:19.723 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:19.723 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:19.723 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:19.723 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 Processor -- start 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 -- start start 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: init 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f011809abd0 0x7f01180960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-a con 0x7f011809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f011809b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f011809c5e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:19.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011809dc90 con 0x7f011809b740 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011809afa0 con 0x7f011809c5e0 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011809b500 con 0x7f011809abd0 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011e43d700 10 monclient(hunting): authenticate will time out at 1748.645327s 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f011809abd0 0x7f01180960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f011809abd0 0x7f01180960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35796/0 (socket says 172.21.15.142:35796) 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 -- 172.21.15.142:0/1033645651 learned_addr learned my addr 172.21.15.142:0/1033645651 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): get_auth_request con 0x7f011809abd0 auth_method 0 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011cc3a700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f011809b740 0x7f011809bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:46976/0 (socket says 172.21.15.142:46976) 2024-09-18T10:24:19.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient(hunting): handle_auth_done global_id 164319 payload 274 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 -- 172.21.15.142:0/1033645651 >> v1:172.21.15.150:6789/0 conn(0x7f011809c5e0 legacy=0x7f0118137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 -- 172.21.15.142:0/1033645651 >> v1:172.21.15.142:6789/0 conn(0x7f011809b740 legacy=0x7f011809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 monclient: found mon.noname-a 2024-09-18T10:24:19.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.717+0000 7f011d43b700 1 -- 172.21.15.142:0/1033645651 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011806f810 con 0x7f011809abd0 2024-09-18T10:24:19.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011d43b700 1 -- 172.21.15.142:0/1033645651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 msgr2=0x7f01180960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011d43b700 1 --2- 172.21.15.142:0/1033645651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f01180960e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0118095e60 tx=0x7f0108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde16083b78c83d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 1 -- 172.21.15.142:0/1033645651 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800ba70 con 0x7f011809abd0 2024-09-18T10:24:19.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:19.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: dump: 2024-09-18T10:24:19.735 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 5 monclient: authenticate success, global_id 164319 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 1 -- 172.21.15.142:0/1033645651 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010800bbd0 con 0x7f011809abd0 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 4 set_mon_vals no callback set 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 1 -- 172.21.15.142:0/1033645651 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011420 con 0x7f011809abd0 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f010ffff700 10 monclient: dump: 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.737 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 10 monclient: shutdown 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 -- 172.21.15.142:0/1033645651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 msgr2=0x7f01180960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 --2- 172.21.15.142:0/1033645651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f01180960e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0118095e60 tx=0x7f0108009250 comp rx=0 tx=0).stop 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 -- 172.21.15.142:0/1033645651 shutdown_connections 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 --2- 172.21.15.142:0/1033645651 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f01180960e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 -- 172.21.15.142:0/1033645651 >> 172.21.15.142:0/1033645651 conn(0x7f0118070ef0 msgr2=0x7f01180712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 -- 172.21.15.142:0/1033645651 shutdown_connections 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 1 -- 172.21.15.142:0/1033645651 wait complete. 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.721+0000 7f011e43d700 10 monclient: monmap: 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 Processor -- start 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 -- start start 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient: init 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-a con 0x7f011809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011809b740 0x7f011813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-b con 0x7f011809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f011809c5e0 0x7f011808bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): picked mon.noname-c con 0x7f011809c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011808f4f0 con 0x7f011809abd0 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011808fa20 con 0x7f011809b740 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0118094f90 con 0x7f011809c5e0 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011e43d700 10 monclient(hunting): authenticate will time out at 1748.653801s 2024-09-18T10:24:19.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35798/0 (socket says 172.21.15.142:35798) 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 -- 172.21.15.142:0/867034817 learned_addr learned my addr 172.21.15.142:0/867034817 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): get_auth_request con 0x7f011809abd0 auth_method 0 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f011809c5e0 0x7f011808bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): get_auth_request con 0x7f011809c5e0 auth_method 0 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011809b740 0x7f011813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): get_auth_request con 0x7f011809b740 auth_method 0 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011dc3c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient(hunting): handle_auth_done global_id 164322 payload 824 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f011809c5e0 msgr2=0x7f011808bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f011809c5e0 0x7f011808bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011809b740 msgr2=0x7f011813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011809b740 0x7f011813feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 monclient: found mon.noname-a 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.725+0000 7f011d43b700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080094f0 con 0x7f011809abd0 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011d43b700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 msgr2=0x7f011813f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011d43b700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f01180957b0 tx=0x7f0108016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800ba70 con 0x7f011809abd0 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: dump: 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 5 monclient: authenticate success, global_id 164322 2024-09-18T10:24:19.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 monclient: _renew_subs 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108020070 con 0x7f011809abd0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01180a0040 con 0x7f011809abd0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010f7fe700 4 set_mon_vals no callback set 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 monclient: _renew_subs 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011808e490 con 0x7f011809abd0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010801c680 con 0x7f011809abd0 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 10 monclient: dump: 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 0x7f011808c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01180648a0 con 0x7f011808c310 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 0x7f011808c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient: get_auth_request con 0x7f011808c310 auth_method 0 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0108022030 con 0x7f011809abd0 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00fc035ac0 0x7f00fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 10 monclient(hunting): handle_auth_done global_id 174127 payload 824 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f010804ae10 con 0x7f011809abd0 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.729+0000 7f011cc3a700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f011809d2d0 con 0x7f011808c310 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f011dc3c700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00fc035ac0 0x7f00fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f011dc3c700 10 monclient: get_auth_request con 0x7f00fc035ac0 auth_method 0 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f011cc3a700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 msgr2=0x7f011808c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f011cc3a700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 0x7f011808c6e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f011400e740 tx=0x7f0114007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114005260 con 0x7f011808c310 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 10 monclient: dump: 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.755 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f011dc3c700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00fc035ac0 0x7f00fc037f70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0110007fd0 tx=0x7f0110002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f01140053c0 con 0x7f011808c310 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 msgr2=0x7f011808c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 0x7f011808c6e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f011400e740 tx=0x7f0114007ed0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 msgr2=0x7f011808c6e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.733+0000 7f010d7fa700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808c310 0x7f011808c6e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011e43d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011e43d700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 0x7f0118087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011e43d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011808e490 con 0x7f011808cc20 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 0x7f0118087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient: get_auth_request con 0x7f011808cc20 auth_method 0 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 10 monclient(hunting): handle_auth_done global_id 174130 payload 824 2024-09-18T10:24:19.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.745+0000 7f011cc3a700 1 -- 172.21.15.142:0/867034817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0118072a80 con 0x7f011808cc20 2024-09-18T10:24:19.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011cc3a700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 msgr2=0x7f0118087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011cc3a700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 0x7f0118087fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0114017000 tx=0x7f0114018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240961882}) 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043802681}) 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240961882}) 2024-09-18T10:24:19.759 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1192725645}) 2024-09-18T10:24:19.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011401c610 con 0x7f011808cc20 2024-09-18T10:24:19.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 10 monclient: dump: 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0114004ee0 con 0x7f011808cc20 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 msgr2=0x7f0118087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 0x7f0118087fe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0114017000 tx=0x7f0114018000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 msgr2=0x7f0118087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f010d7fa700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f011808cc20 0x7f0118087fe0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0114017000 tx=0x7f0114018000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.762 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 142, 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 4, 2024-09-18T10:24:19.763 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.764 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.765 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.766 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:19.767 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.768 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.770 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.772 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.773 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:19.774 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.775 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:19.776 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:19.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00fc035ac0 msgr2=0x7f00fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f00fc035ac0 0x7f00fc037f70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0110007fd0 tx=0x7f0110002620 comp rx=0 tx=0).stop 2024-09-18T10:24:19.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 10 monclient: shutdown 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011dc3c700 1 -- 172.21.15.142:0/867034817 reap_dead start 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 msgr2=0x7f011813f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f01180957b0 tx=0x7f0108016000 comp rx=0 tx=0).stop 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 shutdown_connections 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 --2- 172.21.15.142:0/867034817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f011809abd0 0x7f011813f970 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 >> 172.21.15.142:0/867034817 conn(0x7f0118070ef0 msgr2=0x7f011809de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 shutdown_connections 2024-09-18T10:24:19.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f011e43d700 1 -- 172.21.15.142:0/867034817 wait complete. 2024-09-18T10:24:19.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567855786}) 2024-09-18T10:24:19.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041582086}) 2024-09-18T10:24:19.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567855786}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.749+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237589867}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337095730}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110357958}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337095730}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.753+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3916836450}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362441160}) 2024-09-18T10:24:19.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181407334}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362441160}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913833368}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827597731}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290404616}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827597731}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.757+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2412964894}) 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1455726798}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125064179}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1455726798}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690286164}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343035568}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833382707}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3343035568}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.761+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363244917}) 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.785 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512676566}) 2024-09-18T10:24:19.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=533587422}) 2024-09-18T10:24:19.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512676566}) 2024-09-18T10:24:19.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307920683}) 2024-09-18T10:24:19.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:24:19.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:24:19.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:24:19.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:24:19.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:24:19.788 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:24:19.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515070024}) 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235066375}) 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515070024}) 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3927986670}) 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81396623}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339777816}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81396623}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544720028}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780208804}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350755262}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780208804}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.765+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808561931}) 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953842036}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3858103434}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953842036}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1514355831}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852760803}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639609923}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=852760803}) 2024-09-18T10:24:19.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252779304}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46781523}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964155334}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46781523}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.769+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3252950268}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703444684}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925400342}) 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703444684}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587084271}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510670828}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1637545932}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510670828}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.777+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363937250}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532144219}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162345009}) 2024-09-18T10:24:19.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532144219}) 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70911725}) 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306231744}) 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480797475}) 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306231744}) 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682191068}) 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811375448}) 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570225348}) 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811375448}) 2024-09-18T10:24:19.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:24:19.781+0000 7f46353f0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822396818}) 2024-09-18T10:24:19.801 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:24:19.801 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:24:19.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:19.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: get_monmap_and_config 2024-09-18T10:24:19.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: monmap: 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:19.910372+0000 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:19.910372+0000 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:19.913 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 Processor -- start 2024-09-18T10:24:19.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 -- start start 2024-09-18T10:24:19.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: init 2024-09-18T10:24:19.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcbc070ef0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-b con 0x7fdcbc09a380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcbc09df70 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcbc09e720 con 0x7fdcbc09df70 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcbc09e9e0 con 0x7fdcbc070ef0 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc06fc50 con 0x7fdcbc09a380 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc174d700 10 monclient(hunting): authenticate will time out at 1748.835889s 2024-09-18T10:24:19.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.905+0000 7fdcc0f4c700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fdcbc09df70 legacy=0x7fdcbc139bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcbbfff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdcbc070ef0 0x7fdcbc0712c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:36364/0 (socket says 172.21.15.142:36364) 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcbbfff700 1 -- 172.21.15.142:0/3580819325 learned_addr learned my addr 172.21.15.142:0/3580819325 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45230/0 (socket says 172.21.15.142:45230) 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request con 0x7fdcbc09a380 auth_method 0 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient(hunting): handle_auth_done global_id 75560 payload 274 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3580819325 >> v1:172.21.15.150:6789/0 conn(0x7fdcbc070ef0 legacy=0x7fdcbc0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3580819325 >> v1:172.21.15.142:6789/0 conn(0x7fdcbc09df70 legacy=0x7fdcbc139bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 monclient: found mon.noname-b 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3580819325 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc09ecc0 con 0x7fdcbc09a380 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 -- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 msgr2=0x7fdcbc1376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009910 tx=0x7fdca40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b740deed2e38c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcbb7fe700 1 -- 172.21.15.142:0/3580819325 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400ba70 con 0x7fdcbc09a380 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.909+0000 7fdcbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: dump: 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 5 monclient: authenticate success, global_id 75560 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 1 -- 172.21.15.142:0/3580819325 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca400bbd0 con 0x7fdcbc09a380 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 4 set_mon_vals no callback set 2024-09-18T10:24:19.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 1 -- 172.21.15.142:0/3580819325 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4011420 con 0x7fdcbc09a380 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcbb7fe700 10 monclient: dump: 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 10 monclient: shutdown 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 msgr2=0x7fdcbc1376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 --2- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009910 tx=0x7fdca40091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- 172.21.15.142:0/3580819325 shutdown_connections 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 --2- 172.21.15.142:0/3580819325 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc09a380 0x7fdcbc1376a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- 172.21.15.142:0/3580819325 >> 172.21.15.142:0/3580819325 conn(0x7fdcbc06a2e0 msgr2=0x7fdcbc0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- 172.21.15.142:0/3580819325 shutdown_connections 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- 172.21.15.142:0/3580819325 wait complete. 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 10 monclient: build_initial_monmap 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 10 monclient: monmap: 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:19.924 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 Processor -- start 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 1 -- start start 2024-09-18T10:24:19.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.913+0000 7fdcc174d700 10 monclient: init 2024-09-18T10:24:19.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:19.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:24:19.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-c con 0x7fdcbc070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 0x7fdcbc094a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-a con 0x7fdcbc09a380 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcbc09df70 0x7fdcbc08d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): picked mon.noname-b con 0x7fdcbc09df70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc090c60 con 0x7fdcbc09a380 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc08f720 con 0x7fdcbc09df70 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb37fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 msgr2=0x7fdcbc094a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0a0da0 con 0x7fdcbc070ef0 2024-09-18T10:24:19.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient(hunting): authenticate will time out at 1748.844395s 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 0x7fdcbc094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45242/0 (socket says 172.21.15.142:45242) 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 -- 172.21.15.142:0/1024710180 learned_addr learned my addr 172.21.15.142:0/1024710180 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): get_auth_request con 0x7fdcbc070ef0 auth_method 0 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc0f4c700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcbc09df70 0x7fdcbc08d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc0f4c700 10 monclient(hunting): get_auth_request con 0x7fdcbc09df70 auth_method 0 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc0f4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc0f4c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc0f4c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient(hunting): handle_auth_done global_id 75563 payload 824 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 -- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcbc09df70 msgr2=0x7fdcbc08d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcbc09df70 0x7fdcbc08d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 -- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 msgr2=0x7fdcbc094a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 0x7fdcbc094a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:19.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient: _finish_hunting 0 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 monclient: found mon.noname-c 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 -- 172.21.15.142:0/1024710180 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009420 con 0x7fdcbc070ef0 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 -- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 msgr2=0x7fdcbc1419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c5e0 tx=0x7fdcac004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcbbfff700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c5e0 tx=0x7fdcac004a80 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fdcac005d40 mon_map magic: 0 v1 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac005d40 con 0x7fdcbc070ef0 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 10 monclient: dump: 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 10 monclient: _finish_auth 0 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac0070d0 con 0x7fdcbc070ef0 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 5 monclient: authenticate success, global_id 75563 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _renew_subs 2024-09-18T10:24:19.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcbc09e470 con 0x7fdcbc070ef0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcb8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _renew_subs 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcba7fc700 4 set_mon_vals no callback set 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.917+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcbc08f1a0 con 0x7fdcbc070ef0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcc174d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcc174d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcc174d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcbc072a80 con 0x7fdcbc070ef0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac007490 con 0x7fdcbc070ef0 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:19.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 10 monclient: dump: 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdcac01b3b0 con 0x7fdcbc070ef0 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb8ff9700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 0x7fdc9c037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:19.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 0x7fdc9c037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:19.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.921+0000 7fdcb37fe700 10 monclient: get_auth_request con 0x7fdc9c035af0 auth_method 0 2024-09-18T10:24:19.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.925+0000 7fdcb37fe700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 0x7fdc9c037fa0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc08eca0 tx=0x7fdca4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:19.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:19.929+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdcac048580 con 0x7fdcbc070ef0 2024-09-18T10:24:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.541+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdcac00f820 con 0x7fdcbc070ef0 2024-09-18T10:24:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.541+0000 7fdcb8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:20.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.541+0000 7fdcb8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:20.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcc174d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcc174d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcc174d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdcbc09f960 con 0x7fdcbc070ef0 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcb8ff9700 1 -- 172.21.15.142:0/1024710180 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fdcac00f260 con 0x7fdcbc070ef0 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcb8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.765+0000 7fdcb8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:20.773 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":142,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:20.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 msgr2=0x7fdc9c037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:20.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 0x7fdc9c037fa0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc08eca0 tx=0x7fdca4016000 comp rx=0 tx=0).stop 2024-09-18T10:24:20.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 10 monclient: shutdown 2024-09-18T10:24:20.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:20.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 msgr2=0x7fdcbc1419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00c5e0 tx=0x7fdcac004a80 comp rx=0 tx=0).stop 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 shutdown_connections 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc9c035af0 0x7fdc9c037fa0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcbc09df70 0x7fdcbc08d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcbc09a380 0x7fdcbc094a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 --2- 172.21.15.142:0/1024710180 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcbc070ef0 0x7fdcbc1419e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 >> 172.21.15.142:0/1024710180 conn(0x7fdcbc06a2e0 msgr2=0x7fdcbc098550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 shutdown_connections 2024-09-18T10:24:20.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:20.769+0000 7fdcc174d700 1 -- 172.21.15.142:0/1024710180 wait complete. 2024-09-18T10:24:23.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:23.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: get_monmap_and_config 2024-09-18T10:24:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: build_initial_monmap 2024-09-18T10:24:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: monmap: 2024-09-18T10:24:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:23.902 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:23.896398+0000 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:23.896398+0000 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:23.903 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 Processor -- start 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 -- start start 2024-09-18T10:24:23.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: init 2024-09-18T10:24:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-18T10:24:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 0x7fd9f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-a con 0x7fd9f0070ef0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:23.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-b con 0x7fd9f009a380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9f009df70 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f0071d40 con 0x7fd9f009df70 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f006fc50 con 0x7fd9f0070ef0 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:23.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f009e720 con 0x7fd9f009a380 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f69c5700 10 monclient(hunting): authenticate will time out at 1752.822533s 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f59c3700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 msgr2=0x7fd9f00712c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f61c4700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7fd9f009df70 legacy=0x7fd9f01386a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f59c3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 0x7fd9f00712c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46458/0 (socket says 172.21.15.142:46458) 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 -- 172.21.15.142:0/3440779571 learned_addr learned my addr 172.21.15.142:0/3440779571 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): get_auth_request con 0x7fd9f009a380 auth_method 0 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:23.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient(hunting): handle_auth_done global_id 75566 payload 274 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 -- 172.21.15.142:0/3440779571 >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 msgr2=0x7fd9f00712c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 --2- 172.21.15.142:0/3440779571 >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 0x7fd9f00712c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 -- 172.21.15.142:0/3440779571 >> v1:172.21.15.142:6789/0 conn(0x7fd9f009df70 legacy=0x7fd9f01386a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient: _finish_hunting 0 2024-09-18T10:24:23.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 monclient: found mon.noname-b 2024-09-18T10:24:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:23.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.893+0000 7fd9f51c2700 1 -- 172.21.15.142:0/3440779571 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f009e9e0 con 0x7fd9f009a380 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f51c2700 1 -- 172.21.15.142:0/3440779571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 msgr2=0x7fd9f0071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f51c2700 1 --2- 172.21.15.142:0/3440779571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0064bc0 tx=0x7fd9e0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9755989e8e86d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 1 -- 172.21.15.142:0/3440779571 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e000ba70 con 0x7fd9f009a380 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: dump: 2024-09-18T10:24:23.916 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: _finish_auth 0 2024-09-18T10:24:23.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 1 -- 172.21.15.142:0/3440779571 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e000bbd0 con 0x7fd9f009a380 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 5 monclient: authenticate success, global_id 75566 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 1 -- 172.21.15.142:0/3440779571 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0011420 con 0x7fd9f009a380 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f49c1700 10 monclient: dump: 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 4 set_mon_vals no callback set 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 10 monclient: shutdown 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3440779571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 msgr2=0x7fd9f0071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3440779571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0064bc0 tx=0x7fd9e0009250 comp rx=0 tx=0).stop 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3440779571 shutdown_connections 2024-09-18T10:24:23.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3440779571 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009a380 0x7fd9f0071800 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3440779571 >> v2:172.21.15.142:3300/0 conn(0x7fd9f0070ef0 0x7fd9f00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3440779571 >> 172.21.15.142:0/3440779571 conn(0x7fd9f006a2e0 msgr2=0x7fd9f00a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3440779571 shutdown_connections 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.897+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3440779571 wait complete. 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient: build_initial_monmap 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient: monmap: 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:23.920 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 Processor -- start 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 -- start start 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient: init 2024-09-18T10:24:23.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9f0070ef0 0x7fd9f01417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-b con 0x7fd9f0070ef0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 0x7fd9f0094820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-a con 0x7fd9f009a380 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): picked mon.noname-c con 0x7fd9f009df70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:23.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f0090d10 con 0x7fd9f009a380 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f009e470 con 0x7fd9f0070ef0 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f00a0ba0 con 0x7fd9f009df70 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f69c5700 10 monclient(hunting): authenticate will time out at 1752.830774s 2024-09-18T10:24:23.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f51c2700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 msgr2=0x7fd9f0094820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f51c2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 0x7fd9f0094820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46474/0 (socket says 172.21.15.142:46474) 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 -- 172.21.15.142:0/3943494153 learned_addr learned my addr 172.21.15.142:0/3943494153 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): get_auth_request con 0x7fd9f009df70 auth_method 0 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:23.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9f0070ef0 0x7fd9f01417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): get_auth_request con 0x7fd9f0070ef0 auth_method 0 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:23.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f59c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient(hunting): handle_auth_done global_id 75569 payload 824 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 -- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9f0070ef0 msgr2=0x7fd9f01417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9f0070ef0 0x7fd9f01417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 -- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 msgr2=0x7fd9f0094820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 0x7fd9f0094820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient: _finish_hunting 0 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 monclient: found mon.noname-c 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:23.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.901+0000 7fd9f61c4700 1 -- 172.21.15.142:0/3943494153 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e00094f0 con 0x7fd9f009df70 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f61c4700 1 -- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 msgr2=0x7fd9f008d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f61c4700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800bf80 tx=0x7fd9e8009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800ce50 con 0x7fd9f009df70 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: dump: 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:23.927 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 5 monclient: authenticate success, global_id 75569 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e8016430 con 0x7fd9f009df70 2024-09-18T10:24:23.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _renew_subs 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f0090a30 con 0x7fd9f009df70 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8016620 con 0x7fd9f009df70 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 10 monclient: dump: 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:23.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e7fff700 4 set_mon_vals no callback set 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _renew_subs 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:23.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f0142580 con 0x7fd9f009df70 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f0072a80 con 0x7fd9f009df70 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.905+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd9e8012020 con 0x7fd9f009df70 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.909+0000 7fd9e5ffb700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 0x7fd9d4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.909+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd9e8048fc0 con 0x7fd9f009df70 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.909+0000 7fd9f59c3700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 0x7fd9d4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:23.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.909+0000 7fd9f59c3700 10 monclient: get_auth_request con 0x7fd9d4035ac0 auth_method 0 2024-09-18T10:24:23.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.909+0000 7fd9f59c3700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 0x7fd9d4037f70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0039d0 tx=0x7fd9ec0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:23.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.917+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd9e80078f0 con 0x7fd9f009df70 2024-09-18T10:24:23.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.917+0000 7fd9e5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:23.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:23.917+0000 7fd9e5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd9f00678a0 con 0x7fd9f009df70 2024-09-18T10:24:24.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9e5ffb700 1 -- 172.21.15.142:0/3943494153 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd9e801d300 con 0x7fd9f009df70 2024-09-18T10:24:24.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9e5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:24.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9e5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:24.145 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":142,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 msgr2=0x7fd9d4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 0x7fd9d4037f70 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0039d0 tx=0x7fd9ec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 10 monclient: shutdown 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 msgr2=0x7fd9f008d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.137+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800bf80 tx=0x7fd9e8009730 comp rx=0 tx=0).stop 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 shutdown_connections 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd9d4035ac0 0x7fd9d4037f70 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd9f009df70 0x7fd9f008d4a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd9f009a380 0x7fd9f0094820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:24.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 --2- 172.21.15.142:0/3943494153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9f0070ef0 0x7fd9f01417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:24.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 >> 172.21.15.142:0/3943494153 conn(0x7fd9f006a2e0 msgr2=0x7fd9f00984a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:24.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 shutdown_connections 2024-09-18T10:24:24.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:24.141+0000 7fd9f69c5700 1 -- 172.21.15.142:0/3943494153 wait complete. 2024-09-18T10:24:27.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: get_monmap_and_config 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: build_initial_monmap 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: monmap: 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:27.267895+0000 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:27.267895+0000 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.270 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 Processor -- start 2024-09-18T10:24:27.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 -- start start 2024-09-18T10:24:27.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: init 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f012ec80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1f009bf90 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb1f009c3a0 0x7fb1f012be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f009c3a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f009ce70 con 0x7fb1f009bf90 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f009e470 con 0x7fb1f009c3a0 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f009e8b0 con 0x7fb1f012ec80 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1effff700 10 monclient(hunting): authenticate will time out at 1756.194831s 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46476/0 (socket says 172.21.15.142:46476) 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 -- 172.21.15.142:0/147754915 learned_addr learned my addr 172.21.15.142:0/147754915 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): get_auth_request con 0x7fb1f012ec80 auth_method 0 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:27.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1ef7fe700 1 --2- 172.21.15.142:0/147754915 >> v2:172.21.15.150:3300/0 conn(0x7fb1f009c3a0 0x7fb1f012be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1ef7fe700 10 monclient(hunting): get_auth_request con 0x7fb1f009c3a0 auth_method 0 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_done global_id 75572 payload 274 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 -- 172.21.15.142:0/147754915 >> v2:172.21.15.150:3300/0 conn(0x7fb1f009c3a0 msgr2=0x7fb1f012be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 --2- 172.21.15.142:0/147754915 >> v2:172.21.15.150:3300/0 conn(0x7fb1f009c3a0 0x7fb1f012be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 -- 172.21.15.142:0/147754915 >> v1:172.21.15.150:6789/0 conn(0x7fb1f009bf90 legacy=0x7fb1f012b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:27.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:27.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 monclient: found mon.noname-b 2024-09-18T10:24:27.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:27.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.265+0000 7fb1eeffd700 1 -- 172.21.15.142:0/147754915 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f012f050 con 0x7fb1f012ec80 2024-09-18T10:24:27.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1eeffd700 1 -- 172.21.15.142:0/147754915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 msgr2=0x7fb1f012f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:27.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1eeffd700 1 --2- 172.21.15.142:0/147754915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000bc50 tx=0x7fb1e000b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64fdd6ad163c44d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 1 -- 172.21.15.142:0/147754915 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000be60 con 0x7fb1f012ec80 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: dump: 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:27.281 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: _finish_auth 0 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 1 -- 172.21.15.142:0/147754915 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e000de00 con 0x7fb1f012ec80 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 5 monclient: authenticate success, global_id 75572 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 1 -- 172.21.15.142:0/147754915 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00108d0 con 0x7fb1f012ec80 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 4 set_mon_vals no callback set 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:27.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1edffb700 10 monclient: dump: 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 10 monclient: shutdown 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 -- 172.21.15.142:0/147754915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 msgr2=0x7fb1f012f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:27.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 --2- 172.21.15.142:0/147754915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000bc50 tx=0x7fb1e000b4f0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 -- 172.21.15.142:0/147754915 shutdown_connections 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 --2- 172.21.15.142:0/147754915 >> v2:172.21.15.150:3300/0 conn(0x7fb1f009c3a0 0x7fb1f012be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 --2- 172.21.15.142:0/147754915 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f012ec80 0x7fb1f012f700 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 -- 172.21.15.142:0/147754915 >> 172.21.15.142:0/147754915 conn(0x7fb1f006a2e0 msgr2=0x7fb1f00a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 -- 172.21.15.142:0/147754915 shutdown_connections 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.269+0000 7fb1effff700 1 -- 172.21.15.142:0/147754915 wait complete. 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient: build_initial_monmap 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient: monmap: 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:27.284 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 Processor -- start 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 -- start start 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient: init 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f009bf90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 0x7fb1f0129d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f009c3a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 0x7fb1f012a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): picked mon.noname-a con 0x7fb1f012ec80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f009d720 con 0x7fb1f012ec80 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f006ce90 con 0x7fb1f009c3a0 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f012c490 con 0x7fb1f009bf90 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1effff700 10 monclient(hunting): authenticate will time out at 1756.201553s 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ef7fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 msgr2=0x7fb1f012a2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ef7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 0x7fb1f012a2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:27.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46482/0 (socket says 172.21.15.142:46482) 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 -- 172.21.15.142:0/808416220 learned_addr learned my addr 172.21.15.142:0/808416220 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): get_auth_request con 0x7fb1f009bf90 auth_method 0 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 0x7fb1f0129d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 10 monclient(hunting): get_auth_request con 0x7fb1f009c3a0 auth_method 0 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:27.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient(hunting): handle_auth_done global_id 75575 payload 824 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 -- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 msgr2=0x7fb1f0129d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 0x7fb1f0129d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 -- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 msgr2=0x7fb1f012a2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 0x7fb1f012a2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 monclient: found mon.noname-c 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1eeffd700 1 -- 172.21.15.142:0/808416220 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e000b790 con 0x7fb1f009bf90 2024-09-18T10:24:27.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.273+0000 7fb1ee7fc700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 0x7fb1f0129d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:24:27.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1eeffd700 1 -- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 msgr2=0x7fb1f006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:27.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1eeffd700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0006fd0 tx=0x7fb1e001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0018840 con 0x7fb1f009bf90 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: dump: 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:27.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e00189a0 con 0x7fb1f009bf90 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 5 monclient: authenticate success, global_id 75575 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _renew_subs 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f006c8d0 con 0x7fb1f009bf90 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0018d60 con 0x7fb1f009bf90 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 10 monclient: dump: 2024-09-18T10:24:27.292 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1ed7fa700 4 set_mon_vals no callback set 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _renew_subs 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f006eda0 con 0x7fb1f009bf90 2024-09-18T10:24:27.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f0072a80 con 0x7fb1f009bf90 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb1e00233d0 con 0x7fb1f009bf90 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 0x7fb1dc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.277+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb1e0014070 con 0x7fb1f009bf90 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.281+0000 7fb1ee7fc700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 0x7fb1dc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.281+0000 7fb1ee7fc700 10 monclient: get_auth_request con 0x7fb1dc035a70 auth_method 0 2024-09-18T10:24:27.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.281+0000 7fb1ee7fc700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 0x7fb1dc037f20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4002860 tx=0x7fb1e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:27.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.293+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb1e000edd0 con 0x7fb1f009bf90 2024-09-18T10:24:27.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.293+0000 7fb1d6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:27.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.293+0000 7fb1d6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:27.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.521+0000 7fb1effff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:27.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.521+0000 7fb1effff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.521+0000 7fb1effff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.521+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb1f0128420 con 0x7fb1f009bf90 2024-09-18T10:24:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1d6ffd700 1 -- 172.21.15.142:0/808416220 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fb1e0029070 con 0x7fb1f009bf90 2024-09-18T10:24:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1d6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:27.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1d6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:27.531 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":142,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 msgr2=0x7fb1dc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 0x7fb1dc037f20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4002860 tx=0x7fb1e4009040 comp rx=0 tx=0).stop 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 10 monclient: shutdown 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 msgr2=0x7fb1f006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:27.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0006fd0 tx=0x7fb1e001a000 comp rx=0 tx=0).stop 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 shutdown_connections 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb1dc035a70 0x7fb1dc037f20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f012ec80 0x7fb1f012a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1f009c3a0 0x7fb1f0129d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 --2- 172.21.15.142:0/808416220 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f009bf90 0x7fb1f006f810 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 >> 172.21.15.142:0/808416220 conn(0x7fb1f006a2e0 msgr2=0x7fb1f00a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 shutdown_connections 2024-09-18T10:24:27.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:27.525+0000 7fb1effff700 1 -- 172.21.15.142:0/808416220 wait complete. 2024-09-18T10:24:30.546 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.653+0000 7ff73b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:24:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.653+0000 7ff73b59e700 10 monclient: build_initial_monmap 2024-09-18T10:24:30.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.653+0000 7ff73b59e700 10 monclient: monmap: 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:30.659473+0000 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:30.659473+0000 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:30.662 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:30.663 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:30.663 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:30.663 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 Processor -- start 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 -- start start 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient: init 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff73c09aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff73c132570 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff73c134e90 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c09aeb0 con 0x7ff73c134e90 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff73c06f6d0 con 0x7ff73c09aae0 2024-09-18T10:24:30.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0955a0 con 0x7ff73c132570 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73b59e700 10 monclient(hunting): authenticate will time out at 1759.585092s 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73ad9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7ff73c134e90 legacy=0x7ff73c137270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46498/0 (socket says 172.21.15.142:46498) 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 learned_addr learned my addr 172.21.15.142:0/1522772659 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff73c132570 auth_method 0 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:30.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:30.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:30.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:30.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:30.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.657+0000 7ff73a59c700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.150:6789/0 conn(0x7ff73c09aae0 legacy=0x7ff73c096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:30.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff73ad9d700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.142:6789/0 conn(0x7ff73c134e90 legacy=0x7ff73c137270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff73c132570 auth_method 0 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:30.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:30.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:24:31.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff73ad9d700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.142:6789/0 conn(0x7ff73c134e90 legacy=0x7ff73c137270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:31.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:31.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff73c132570 auth_method 0 2024-09-18T10:24:31.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:31.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:31.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:24:32.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff73ad9d700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.142:6789/0 conn(0x7ff73c134e90 legacy=0x7ff73c137270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:32.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:32.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff73c132570 auth_method 0 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:32.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:32.069+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:24:33.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): tick 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 monclient(hunting): continuing hunt 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 msgr2=0x7ff73c134950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c132570 0x7ff73c134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.150:6789/0 conn(0x7ff73c09aae0 legacy=0x7ff73c096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 -- 172.21.15.142:0/1522772659 >> v1:172.21.15.142:6789/0 conn(0x7ff73c134e90 legacy=0x7ff73c137270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient: _add_conns ranks=[3,4] 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 0x7ff71c005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff71c0037a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:33.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff71c006300 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 -- 172.21.15.142:0/1522772659 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c0955a0 con 0x7ff71c0037a0 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 1 -- 172.21.15.142:0/1522772659 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c008e70 con 0x7ff71c006300 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff733fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff73a59c700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 msgr2=0x7ff71c005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff73a59c700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 0x7ff71c005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff71c006300 auth_method 0 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:24:33.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:33.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:33.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.657+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:33.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff73a59c700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 msgr2=0x7ff71c005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff73a59c700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 0x7ff71c005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff71c006300 auth_method 0 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:33.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:33.861+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:24:34.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff73a59c700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 msgr2=0x7ff71c005c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff73a59c700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 0x7ff71c005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.150:3300/0 conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff71c006300 auth_method 0 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient(hunting): handle_auth_done global_id 174145 payload 274 2024-09-18T10:24:34.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 msgr2=0x7ff71c005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> v2:172.21.15.142:3300/0 conn(0x7ff71c0037a0 0x7ff71c005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 monclient: found mon.noname-c 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:24:34.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c095b60 con 0x7ff71c006300 2024-09-18T10:24:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.265+0000 7ff739d9b700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c006300 0x7ff71c0087b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0062d0 tx=0x7ff734009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c381f31d82fbb80f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 1 -- 172.21.15.142:0/1522772659 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff734019430 con 0x7ff71c006300 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: dump: 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.274 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 1 -- 172.21.15.142:0/1522772659 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff73400f040 con 0x7ff71c006300 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 1 -- 172.21.15.142:0/1522772659 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7340197a0 con 0x7ff71c006300 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 5 monclient: authenticate success, global_id 174145 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 4 set_mon_vals no callback set 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73959a700 10 monclient: dump: 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.276 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.277 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 10 monclient: shutdown 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 -- 172.21.15.142:0/1522772659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c006300 msgr2=0x7ff71c0087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c006300 0x7ff71c0087b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0062d0 tx=0x7ff734009040 comp rx=0 tx=0).stop 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73ad9d700 1 -- 172.21.15.142:0/1522772659 reap_dead start 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 -- 172.21.15.142:0/1522772659 shutdown_connections 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 --2- 172.21.15.142:0/1522772659 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c006300 0x7ff71c0087b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 -- 172.21.15.142:0/1522772659 >> 172.21.15.142:0/1522772659 conn(0x7ff73c0a1fe0 msgr2=0x7ff73c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 -- 172.21.15.142:0/1522772659 shutdown_connections 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 1 -- 172.21.15.142:0/1522772659 wait complete. 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 10 monclient: build_initial_monmap 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.269+0000 7ff73b59e700 10 monclient: monmap: 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:34.283 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 Processor -- start 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 -- start start 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient: init 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 0x7ff73c098a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff71c006300 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 0x7ff73c0993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff73c098fb0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:34.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff73c099900 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c06d380 con 0x7ff71c006300 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c06d9b0 con 0x7ff73c098fb0 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c06d0a0 con 0x7ff73c099900 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73b59e700 10 monclient(hunting): authenticate will time out at 1763.201050s 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73a59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 msgr2=0x7ff73c098a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 0x7ff73c098a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37578/0 (socket says 172.21.15.142:37578) 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 -- 172.21.15.142:0/3936411759 learned_addr learned my addr 172.21.15.142:0/3936411759 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): get_auth_request con 0x7ff73c099900 auth_method 0 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 0x7ff73c0993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): get_auth_request con 0x7ff73c098fb0 auth_method 0 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient(hunting): handle_auth_done global_id 75578 payload 824 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 -- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 msgr2=0x7ff73c0993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 0x7ff73c0993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 -- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 msgr2=0x7ff73c098a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 0x7ff73c098a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 monclient: found mon.noname-c 2024-09-18T10:24:34.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff73ad9d700 1 -- 172.21.15.142:0/3936411759 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734009930 con 0x7ff73c099900 2024-09-18T10:24:34.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.273+0000 7ff739d9b700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 0x7ff73c0993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73ad9d700 1 -- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 msgr2=0x7ff73c147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73ad9d700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff71c006270 tx=0x7ff72c0093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c014030 con 0x7ff73c099900 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: dump: 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.290 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72c00c040 con 0x7ff73c099900 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72c0067a0 con 0x7ff73c099900 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 10 monclient: dump: 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 5 monclient: authenticate success, global_id 75578 2024-09-18T10:24:34.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff738d99700 4 set_mon_vals no callback set 2024-09-18T10:24:34.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _renew_subs 2024-09-18T10:24:34.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c06bec0 con 0x7ff73c099900 2024-09-18T10:24:34.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _renew_subs 2024-09-18T10:24:34.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c06bb20 con 0x7ff73c099900 2024-09-18T10:24:34.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c06d5a0 con 0x7ff73c099900 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff72c017070 con 0x7ff73c099900 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 0x7ff71c038cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.277+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff72c015070 con 0x7ff73c099900 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.281+0000 7ff73a59c700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 0x7ff71c038cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.281+0000 7ff73a59c700 10 monclient: get_auth_request con 0x7ff71c036840 auth_method 0 2024-09-18T10:24:34.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.281+0000 7ff73a59c700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 0x7ff71c038cf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff724007fd0 tx=0x7ff724007b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.289+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff72c018b50 con 0x7ff73c099900 2024-09-18T10:24:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.289+0000 7ff7327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.289+0000 7ff7327fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:34.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff73c1369c0 con 0x7ff73c099900 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff7327fc700 1 -- 172.21.15.142:0/3936411759 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7ff72c018370 con 0x7ff73c099900 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff7327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:34.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff7327fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:34.512 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":144,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 msgr2=0x7ff71c038cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 0x7ff71c038cf0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff724007fd0 tx=0x7ff724007b20 comp rx=0 tx=0).stop 2024-09-18T10:24:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 10 monclient: shutdown 2024-09-18T10:24:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 msgr2=0x7ff73c147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff71c006270 tx=0x7ff72c0093e0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 shutdown_connections 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff71c036840 0x7ff71c038cf0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff73c099900 0x7ff73c147b60 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff73c098fb0 0x7ff73c0993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 --2- 172.21.15.142:0/3936411759 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff71c006300 0x7ff73c098a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 >> 172.21.15.142:0/3936411759 conn(0x7ff73c0a1fe0 msgr2=0x7ff73c131d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 shutdown_connections 2024-09-18T10:24:34.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.505+0000 7ff73b59e700 1 -- 172.21.15.142:0/3936411759 wait complete. 2024-09-18T10:24:34.524 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:24:34.525 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:24:34.525 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 10 monclient: get_monmap_and_config 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 10 monclient: build_initial_monmap 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 10 monclient: monmap: 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:34.626824+0000 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:34.626824+0000 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:34.629 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:34.630 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.633 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 1 Processor -- start 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 1 -- start start 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.621+0000 7f23d0cd6700 10 monclient: init 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23cc0a2df0 0x7f23cc070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-c con 0x7f23cc0a2df0 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-b con 0x7f23cc09df10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 0x7f23cc138690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-a con 0x7f23cc071970 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.634 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc096060 con 0x7f23cc071970 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23cc099a30 con 0x7f23cc0a2df0 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc09ec60 con 0x7f23cc09df10 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23d0cd6700 10 monclient(hunting): authenticate will time out at 1763.552504s 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cbfff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 msgr2=0x7f23cc138690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cbfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 0x7f23cc138690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37588/0 (socket says 172.21.15.142:37588) 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 -- 172.21.15.142:0/3728787958 learned_addr learned my addr 172.21.15.142:0/3728787958 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): get_auth_request con 0x7f23cc09df10 auth_method 0 2024-09-18T10:24:34.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/3728787958 >> v2:172.21.15.150:3300/0 conn(0x7f23cc0a2df0 0x7f23cc070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request con 0x7f23cc0a2df0 auth_method 0 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.636 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient(hunting): handle_auth_done global_id 75581 payload 274 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 -- 172.21.15.142:0/3728787958 >> v2:172.21.15.150:3300/0 conn(0x7f23cc0a2df0 msgr2=0x7f23cc070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 --2- 172.21.15.142:0/3728787958 >> v2:172.21.15.150:3300/0 conn(0x7f23cc0a2df0 0x7f23cc070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 -- 172.21.15.142:0/3728787958 >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 msgr2=0x7f23cc138690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 --2- 172.21.15.142:0/3728787958 >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 0x7f23cc138690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 monclient: found mon.noname-b 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 -- 172.21.15.142:0/3728787958 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0a31c0 con 0x7f23cc09df10 2024-09-18T10:24:34.637 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 -- 172.21.15.142:0/3728787958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 msgr2=0x7f23cc071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23caffd700 1 --2- 172.21.15.142:0/3728787958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f23cc070190 tx=0x7f23bc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693239631be53c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.625+0000 7f23ca7fc700 1 -- 172.21.15.142:0/3728787958 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc009e70 con 0x7f23cc09df10 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: dump: 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.638 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 5 monclient: authenticate success, global_id 75581 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 1 -- 172.21.15.142:0/3728787958 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc00ba70 con 0x7f23cc09df10 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 1 -- 172.21.15.142:0/3728787958 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc00be30 con 0x7f23cc09df10 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 4 set_mon_vals no callback set 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23ca7fc700 10 monclient: dump: 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.640 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: shutdown 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- 172.21.15.142:0/3728787958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 msgr2=0x7f23cc071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/3728787958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f23cc070190 tx=0x7f23bc009250 comp rx=0 tx=0).stop 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- 172.21.15.142:0/3728787958 shutdown_connections 2024-09-18T10:24:34.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/3728787958 >> v2:172.21.15.142:3300/0 conn(0x7f23cc071970 0x7f23cc138690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/3728787958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc071430 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/3728787958 >> v2:172.21.15.150:3300/0 conn(0x7f23cc0a2df0 0x7f23cc070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- 172.21.15.142:0/3728787958 >> 172.21.15.142:0/3728787958 conn(0x7f23cc06a140 msgr2=0x7f23cc0a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- 172.21.15.142:0/3728787958 shutdown_connections 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- 172.21.15.142:0/3728787958 wait complete. 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: build_initial_monmap 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: monmap: 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:34.648 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 Processor -- start 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- start start 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: init 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23cc071970 0x7f23cc141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-b con 0x7f23cc071970 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:34.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-c con 0x7f23cc09df10 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f23cc0a2df0 0x7f23cc08d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): picked mon.noname-a con 0x7f23cc0a2df0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc09e440 con 0x7f23cc0a2df0 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0a0720 con 0x7f23cc071970 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc08ecf0 con 0x7f23cc09df10 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23d0cd6700 10 monclient(hunting): authenticate will time out at 1763.559490s 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23cbfff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f23cc0a2df0 msgr2=0x7f23cc08d160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23cbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f23cc0a2df0 0x7f23cc08d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc141ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37598/0 (socket says 172.21.15.142:37598) 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 1 -- 172.21.15.142:0/4004567416 learned_addr learned my addr 172.21.15.142:0/4004567416 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): get_auth_request con 0x7f23cc09df10 auth_method 0 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23cc071970 0x7f23cc141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.629+0000 7f23caffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request con 0x7f23cc071970 auth_method 0 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 10 monclient(hunting): handle_auth_done global_id 75584 payload 824 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23cc071970 msgr2=0x7f23cc141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23cc071970 0x7f23cc141980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f23cc0a2df0 msgr2=0x7f23cc08d160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f23cc0a2df0 0x7f23cc08d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 monclient: found mon.noname-c 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc0094f0 con 0x7f23cc09df10 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 msgr2=0x7f23cc141ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23caffd700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc141ec0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0090b0 tx=0x7f23bc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc006a90 con 0x7f23cc09df10 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:34.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: dump: 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 5 monclient: authenticate success, global_id 75584 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc011af0 con 0x7f23cc09df10 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 10 monclient: _renew_subs 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc091a00 con 0x7f23cc09df10 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc025420 con 0x7f23cc09df10 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23b37fe700 10 monclient: dump: 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.655 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 10 monclient: _renew_subs 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc08f7d0 con 0x7f23cc09df10 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.633+0000 7f23c97fa700 4 set_mon_vals no callback set 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23d0cd6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 0x7f23cc029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc064950 con 0x7f23cc08de70 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 0x7f23cc029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient: get_auth_request con 0x7f23cc08de70 auth_method 0 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_done global_id 75587 payload 824 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23cc090bb0 con 0x7f23cc08de70 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f23bc023070 con 0x7f23cc09df10 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23b8035a70 0x7f23b8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f23bc049dc0 con 0x7f23cc09df10 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cbfff700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23b8035a70 0x7f23b8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cbfff700 10 monclient: get_auth_request con 0x7f23b8035a70 auth_method 0 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 msgr2=0x7f23cc029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 0x7f23cc029d00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f23c0011c20 tx=0x7f23c00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c000fe80 con 0x7f23cc08de70 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.637+0000 7f23b37fe700 10 monclient: dump: 2024-09-18T10:24:34.658 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f23c0014440 con 0x7f23cc08de70 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 msgr2=0x7f23cc029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 0x7f23cc029d00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f23c0011c20 tx=0x7f23c00091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 msgr2=0x7f23cc029d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:34.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23b37fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08de70 0x7f23cc029d00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.641+0000 7f23cbfff700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23b8035a70 0x7f23b8037f20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f23cc090d30 tx=0x7f23c40092d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23d0cd6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:34.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 0x7f23cc02a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23d0cd6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc06d2d0 con 0x7f23cc08e240 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 0x7f23cc02a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient: get_auth_request con 0x7f23cc08e240 auth_method 0 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 10 monclient(hunting): handle_auth_done global_id 75590 payload 824 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.649+0000 7f23cb7fe700 1 -- 172.21.15.142:0/4004567416 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f23cc072a80 con 0x7f23cc08e240 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23cb7fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 msgr2=0x7f23cc02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23cb7fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 0x7f23cc02a240 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f23c0011c20 tx=0x7f23c001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c002b070 con 0x7f23cc08e240 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 10 monclient: dump: 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.664 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.665 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.665 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.665 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.665 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.665 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f23c0023040 con 0x7f23cc08e240 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 msgr2=0x7f23cc02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 0x7f23cc02a240 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f23c0011c20 tx=0x7f23c001c000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 msgr2=0x7f23cc02a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:34.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23b37fe700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc08e240 0x7f23cc02a240 secure :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f23c0011c20 tx=0x7f23c001c000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.667 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:34.667 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:34.667 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 144, 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:34.668 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.669 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.670 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:34.671 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:34.672 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.673 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.674 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.675 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:34.676 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.677 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.678 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.679 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.680 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:34.681 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:34.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.683 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:34.684 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23b8035a70 msgr2=0x7f23b8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23b8035a70 0x7f23b8037f20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f23cc090d30 tx=0x7f23c40092d0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 10 monclient: shutdown 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 msgr2=0x7f23cc141ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 --2- 172.21.15.142:0/4004567416 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f23cc09df10 0x7f23cc141ec0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f23bc0090b0 tx=0x7f23bc017000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23cbfff700 1 -- 172.21.15.142:0/4004567416 reap_dead start 2024-09-18T10:24:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23cbfff700 1 -- 172.21.15.142:0/4004567416 reap_dead start 2024-09-18T10:24:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 shutdown_connections 2024-09-18T10:24:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 >> 172.21.15.142:0/4004567416 conn(0x7f23cc06a140 msgr2=0x7f23cc0a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 shutdown_connections 2024-09-18T10:24:34.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.653+0000 7f23d0cd6700 1 -- 172.21.15.142:0/4004567416 wait complete. 2024-09-18T10:24:34.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:24:34.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:24:34.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: get_monmap_and_config 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: build_initial_monmap 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: monmap: 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:34.791879+0000 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:34.791879+0000 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:34.794 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:34.795 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:34.795 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 Processor -- start 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 -- start start 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: init 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 0x7f0fb4097ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-a con 0x7f0fb4097380 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:34.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fb409be60 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-b con 0x7f0fb409c970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb409b3d0 con 0x7f0fb409be60 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb409b610 con 0x7f0fb4097380 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40955e0 con 0x7f0fb409c970 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb9dae700 10 monclient(hunting): authenticate will time out at 1763.717451s 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb8dac700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 msgr2=0x7f0fb4097ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb8dac700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 0x7f0fb4097ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37640/0 (socket says 172.21.15.142:37640) 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 -- 172.21.15.142:0/917957378 learned_addr learned my addr 172.21.15.142:0/917957378 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): get_auth_request con 0x7f0fb409c970 auth_method 0 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_done global_id 75593 payload 274 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 -- 172.21.15.142:0/917957378 >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 msgr2=0x7f0fb4097ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 --2- 172.21.15.142:0/917957378 >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 0x7f0fb4097ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 -- 172.21.15.142:0/917957378 >> v1:172.21.15.150:6789/0 conn(0x7f0fb409be60 legacy=0x7f0fb409c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 monclient: found mon.noname-b 2024-09-18T10:24:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.789+0000 7f0fb95ad700 1 -- 172.21.15.142:0/917957378 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4097750 con 0x7f0fb409c970 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb95ad700 1 -- 172.21.15.142:0/917957378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 msgr2=0x7f0fb4137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb95ad700 1 --2- 172.21.15.142:0/917957378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009f20 tx=0x7f0fac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=710ef0b9d3827ffb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 1 -- 172.21.15.142:0/917957378 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac00e070 con 0x7f0fb409c970 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: dump: 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 5 monclient: authenticate success, global_id 75593 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 1 -- 172.21.15.142:0/917957378 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac00bc90 con 0x7f0fb409c970 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:34.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 4 set_mon_vals no callback set 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 1 -- 172.21.15.142:0/917957378 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac012660 con 0x7f0fb409c970 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb37fe700 10 monclient: dump: 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.807 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 10 monclient: shutdown 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- 172.21.15.142:0/917957378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 msgr2=0x7f0fb4137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/917957378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009f20 tx=0x7f0fac009840 comp rx=0 tx=0).stop 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- 172.21.15.142:0/917957378 shutdown_connections 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/917957378 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb4137620 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/917957378 >> v2:172.21.15.142:3300/0 conn(0x7f0fb4097380 0x7f0fb4097ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- 172.21.15.142:0/917957378 >> 172.21.15.142:0/917957378 conn(0x7f0fb40a1ec0 msgr2=0x7f0fb4070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- 172.21.15.142:0/917957378 shutdown_connections 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- 172.21.15.142:0/917957378 wait complete. 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 10 monclient: build_initial_monmap 2024-09-18T10:24:34.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 10 monclient: monmap: 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 Processor -- start 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 1 -- start start 2024-09-18T10:24:34.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.793+0000 7f0fb9dae700 10 monclient: init 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fb4097380 0x7f0fb408bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-a con 0x7f0fb4097380 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb409be60 0x7f0fb408c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-b con 0x7f0fb409be60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb408c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): picked mon.noname-c con 0x7f0fb409c970 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb8dac700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fb4097380 msgr2=0x7f0fb408bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb4090190 con 0x7f0fb4097380 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb409d0e0 con 0x7f0fb409be60 2024-09-18T10:24:34.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb406ef00 con 0x7f0fb409c970 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient(hunting): authenticate will time out at 1763.724576s 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb8dac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fb4097380 0x7f0fb408bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb408c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb408c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37650/0 (socket says 172.21.15.142:37650) 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 learned_addr learned my addr 172.21.15.142:0/1833494196 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): get_auth_request con 0x7f0fb409c970 auth_method 0 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb3fff700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb409be60 0x7f0fb408c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb3fff700 10 monclient(hunting): get_auth_request con 0x7f0fb409be60 auth_method 0 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient(hunting): handle_auth_done global_id 75596 payload 824 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb409be60 msgr2=0x7f0fb408c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb409be60 0x7f0fb408c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fb4097380 msgr2=0x7f0fb408bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0fb4097380 0x7f0fb408bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient: _finish_hunting 0 2024-09-18T10:24:34.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 monclient: found mon.noname-c 2024-09-18T10:24:34.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:34.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac009ae0 con 0x7f0fb409c970 2024-09-18T10:24:34.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 msgr2=0x7f0fb408c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb95ad700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb408c5b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fac003000 tx=0x7f0fac017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac012e70 con 0x7f0fb409c970 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: dump: 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: _finish_auth 0 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 5 monclient: authenticate success, global_id 75596 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient: _renew_subs 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac01e430 con 0x7f0fb409c970 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb408d440 con 0x7f0fb409c970 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac01e7b0 con 0x7f0fb409c970 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb2ffd700 4 set_mon_vals no callback set 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.797+0000 7f0fb0ff9700 10 monclient: dump: 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 10 monclient: _renew_subs 2024-09-18T10:24:34.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:34.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb408e010 con 0x7f0fb409c970 2024-09-18T10:24:34.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 0x7f0fb4087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb408f110 con 0x7f0fb408caf0 2024-09-18T10:24:34.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 0x7f0fb4087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient: get_auth_request con 0x7f0fb408caf0 auth_method 0 2024-09-18T10:24:34.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0fac01eb60 con 0x7f0fb409c970 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb0ff9700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f98035a70 0x7f0f98037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_done global_id 174160 payload 824 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0fac04aba0 con 0x7f0fb409c970 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb8dac700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb4072a80 con 0x7f0fb408caf0 2024-09-18T10:24:34.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb3fff700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f98035a70 0x7f0f98037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.801+0000 7f0fb3fff700 10 monclient: get_auth_request con 0x7f0f98035a70 auth_method 0 2024-09-18T10:24:34.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb8dac700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 msgr2=0x7f0fb4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb8dac700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 0x7f0fb4087fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b690 tx=0x7f0fa80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80050f0 con 0x7f0fb408caf0 2024-09-18T10:24:34.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 10 monclient: dump: 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.826 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.826 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.826 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.826 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.826 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb3fff700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f98035a70 0x7f0f98037f20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4095100 tx=0x7f0fa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0fa8005250 con 0x7f0fb408caf0 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:34.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 msgr2=0x7f0fb4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 0x7f0fb4087fe0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b690 tx=0x7f0fa80091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 msgr2=0x7f0fb4087fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:34.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.805+0000 7f0fb0ff9700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb408caf0 0x7f0fb4087fe0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:34.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.813+0000 7f0fb9dae700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:24:34.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.813+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 0x7f0fb4088920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.813+0000 7f0fb9dae700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.813+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40648a0 con 0x7f0fb4088520 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.813+0000 7f0fb8dac700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 0x7f0fb4088920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient: get_auth_request con 0x7f0fb4088520 auth_method 0 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:34.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:34.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 10 monclient(hunting): handle_auth_done global_id 174163 payload 824 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 1 -- 172.21.15.142:0/1833494196 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0fb408f390 con 0x7f0fb4088520 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 msgr2=0x7f0fb4088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb8dac700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 0x7f0fb4088920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b690 tx=0x7f0fa8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8020840 con 0x7f0fb4088520 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:34.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.817+0000 7f0fb0ff9700 10 monclient: dump: 2024-09-18T10:24:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.835 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:34.836 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f0fa80209a0 con 0x7f0fb4088520 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 msgr2=0x7f0fb4088920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 0x7f0fb4088920 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b690 tx=0x7f0fa8015000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 msgr2=0x7f0fb4088920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:24:34.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb0ff9700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0fb4088520 0x7f0fb4088920 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b690 tx=0x7f0fa8015000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.838 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:24:34.838 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 144, 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:24:34.839 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:24:34.840 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.840 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:24:34.840 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:34.840 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:24:34.840 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.841 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:24:34.843 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:24:34.843 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.843 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.843 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.843 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.844 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:24:34.845 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:24:34.846 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:24:34.847 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.848 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:24:34.849 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:24:34.850 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.851 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:24:34.852 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.853 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.854 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:24:34.855 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:24:34.856 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:24:34.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f98035a70 msgr2=0x7f0f98037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.858 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0f98035a70 0x7f0f98037f20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4095100 tx=0x7f0fa0009040 comp rx=0 tx=0).stop 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 10 monclient: shutdown 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 msgr2=0x7f0fb408c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 --2- 172.21.15.142:0/1833494196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0fb409c970 0x7f0fb408c5b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fac003000 tx=0x7f0fac017000 comp rx=0 tx=0).stop 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 reap_dead start 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb95ad700 1 -- 172.21.15.142:0/1833494196 reap_dead start 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 shutdown_connections 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 >> 172.21.15.142:0/1833494196 conn(0x7f0fb40a1ec0 msgr2=0x7f0fb406e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 shutdown_connections 2024-09-18T10:24:34.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:34.821+0000 7f0fb9dae700 1 -- 172.21.15.142:0/1833494196 wait complete. 2024-09-18T10:24:34.860 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:24:34.860 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:24:54.861 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:24:54.862 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:24:54.862 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:24:54.862 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:24:54.865 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:24:54.865 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:24:54.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:54.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: get_monmap_and_config 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: build_initial_monmap 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: monmap: 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:54.969673+0000 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:54.969673+0000 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:54.973 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:54.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 1 Processor -- start 2024-09-18T10:24:54.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 1 -- start start 2024-09-18T10:24:54.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: init 2024-09-18T10:24:54.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:54.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:24:54.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d3809d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:24:54.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d3812f510 0x7f3d38134130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.965+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-c con 0x7f3d3812f510 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d3812f930 addr v1:172.21.15.142:6789/0 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3809c440 con 0x7f3d3812f930 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3df7b700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f3d3812f930 legacy=0x7f3d38136680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3809a6c0 con 0x7f3d3809d870 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3809dc40 con 0x7f3d3812f510 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3f9de700 10 monclient(hunting): authenticate will time out at 1783.896174s 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3d3812f510 0x7f3d38134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3d77a700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3d3809d870 0x7f3d3812ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:53250/0 (socket says 172.21.15.142:53250) 2024-09-18T10:24:54.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3d77a700 1 -- 172.21.15.142:0/2062429258 learned_addr learned my addr 172.21.15.142:0/2062429258 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): get_auth_request con 0x7f3d3812f510 auth_method 0 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_done global_id 174166 payload 274 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 -- 172.21.15.142:0/2062429258 >> v1:172.21.15.150:6789/0 conn(0x7f3d3809d870 legacy=0x7f3d3812ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 -- 172.21.15.142:0/2062429258 >> v1:172.21.15.142:6789/0 conn(0x7f3d3812f930 legacy=0x7f3d38136680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient: _finish_hunting 0 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 monclient: found mon.noname-c 2024-09-18T10:24:54.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:54.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:24:54.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 -- 172.21.15.142:0/2062429258 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3809a9a0 con 0x7f3d3812f510 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 -- 172.21.15.142:0/2062429258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 msgr2=0x7f3d38134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.969+0000 7f3d3cf79700 1 --2- 172.21.15.142:0/2062429258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d38134130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d3809ab90 tx=0x7f3d340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc83a4051b09ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 1 -- 172.21.15.142:0/2062429258 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400b9a0 con 0x7f3d3812f510 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:54.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: dump: 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: _finish_auth 0 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:54.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 1 -- 172.21.15.142:0/2062429258 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3400bb00 con 0x7f3d3812f510 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 1 -- 172.21.15.142:0/2062429258 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400bea0 con 0x7f3d3812f510 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 5 monclient: authenticate success, global_id 174166 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 4 set_mon_vals no callback set 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d2ffff700 10 monclient: dump: 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.982 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 10 monclient: shutdown 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 -- 172.21.15.142:0/2062429258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 msgr2=0x7f3d38134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/2062429258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d38134130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d3809ab90 tx=0x7f3d340091a0 comp rx=0 tx=0).stop 2024-09-18T10:24:54.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 -- 172.21.15.142:0/2062429258 shutdown_connections 2024-09-18T10:24:54.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/2062429258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d38134130 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:54.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 -- 172.21.15.142:0/2062429258 >> 172.21.15.142:0/2062429258 conn(0x7f3d38071820 msgr2=0x7f3d380a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:54.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 -- 172.21.15.142:0/2062429258 shutdown_connections 2024-09-18T10:24:54.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 1 -- 172.21.15.142:0/2062429258 wait complete. 2024-09-18T10:24:54.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 10 monclient: build_initial_monmap 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.973+0000 7f3d3f9de700 10 monclient: monmap: 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:54.988 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 Processor -- start 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 -- start start 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient: init 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-c con 0x7f3d3809d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d3812ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-b con 0x7f3d3812f510 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:54.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): picked mon.noname-a con 0x7f3d3812f930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3806b680 con 0x7f3d3812f930 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3806d400 con 0x7f3d3812f510 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380a1220 con 0x7f3d3809d870 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3f9de700 10 monclient(hunting): authenticate will time out at 1783.907726s 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41516/0 (socket says 172.21.15.142:41516) 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 1 -- 172.21.15.142:0/3139452252 learned_addr learned my addr 172.21.15.142:0/3139452252 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:54.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 10 monclient(hunting): get_auth_request con 0x7f3d3809d870 auth_method 0 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3d77a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.977+0000 7f3d3df7b700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 10 monclient(hunting): get_auth_request con 0x7f3d3812f930 auth_method 0 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d3812ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:54.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): get_auth_request con 0x7f3d3812f510 auth_method 0 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 msgr2=0x7f3d3812b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 msgr2=0x7f3d3812b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3cf79700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3df7b700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient(hunting): handle_auth_done global_id 75599 payload 824 2024-09-18T10:24:54.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 msgr2=0x7f3d3812ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d3812ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 msgr2=0x7f3d3812b3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient: _finish_hunting 0 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 monclient: found mon.noname-c 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:54.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 -- 172.21.15.142:0/3139452252 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34009420 con 0x7f3d3809d870 2024-09-18T10:24:54.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 msgr2=0x7f3d3806f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:54.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3d77a700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c610 tx=0x7f3d28004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28005d40 con 0x7f3d3809d870 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: dump: 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:54.996 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:54.997 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 5 monclient: authenticate success, global_id 75599 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 10 monclient: _renew_subs 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38099e30 con 0x7f3d3809d870 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d280070d0 con 0x7f3d3809d870 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:54.998 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28007490 con 0x7f3d3809d870 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2d7fa700 10 monclient: dump: 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:54.999 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:55.000 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d2f7fe700 4 set_mon_vals no callback set 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 10 monclient: _renew_subs 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.981+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3806ac80 con 0x7f3d3809d870 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3f9de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3f9de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3f9de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d380696a0 con 0x7f3d3809d870 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3d28012400 con 0x7f3d3809d870 2024-09-18T10:24:55.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d2d7fa700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 0x7f3d1c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3d280483b0 con 0x7f3d3809d870 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3cf79700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 0x7f3d1c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.985+0000 7f3d3cf79700 10 monclient: get_auth_request con 0x7f3d1c035ac0 auth_method 0 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.989+0000 7f3d3cf79700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 0x7f3d1c037f70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3d3812c490 tx=0x7f3d34016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.989+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3d2800f860 con 0x7f3d3809d870 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.989+0000 7f3d2d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:55.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:54.989+0000 7f3d2d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:55.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d3f9de700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:55.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d3f9de700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:55.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d3f9de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:55.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3d38067950 con 0x7f3d3809d870 2024-09-18T10:24:55.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d2d7fa700 1 -- 172.21.15.142:0/3139452252 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f3d28017020 con 0x7f3d3809d870 2024-09-18T10:24:55.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d2d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:55.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.209+0000 7f3d2d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:55.216 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":144,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 msgr2=0x7f3d1c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 0x7f3d1c037f70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3d3812c490 tx=0x7f3d34016000 comp rx=0 tx=0).stop 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 10 monclient: shutdown 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 msgr2=0x7f3d3806f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:55.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c610 tx=0x7f3d28004a80 comp rx=0 tx=0).stop 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 shutdown_connections 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3d1c035ac0 0x7f3d1c037f70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3812f930 0x7f3d3812b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3812f510 0x7f3d3812ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 --2- 172.21.15.142:0/3139452252 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3809d870 0x7f3d3806f680 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 >> 172.21.15.142:0/3139452252 conn(0x7f3d38071820 msgr2=0x7f3d3812c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 shutdown_connections 2024-09-18T10:24:55.219 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:55.213+0000 7f3d3f9de700 1 -- 172.21.15.142:0/3139452252 wait complete. 2024-09-18T10:24:58.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:24:58.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.333+0000 7f091901b700 10 monclient: get_monmap_and_config 2024-09-18T10:24:58.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.333+0000 7f091901b700 10 monclient: build_initial_monmap 2024-09-18T10:24:58.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.333+0000 7f091901b700 10 monclient: monmap: 2024-09-18T10:24:58.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:24:58.339209+0000 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:24:58.339209+0000 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:24:58.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 Processor -- start 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 -- start start 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient: init 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): picked mon.noname-b con 0x7f091409aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): picked mon.noname-a con 0x7f09140967a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 0x7f091409c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): picked mon.noname-c con 0x7f091409c320 addr v2:172.21.15.150:3300/0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0914095880 con 0x7f09140967a0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091409aeb0 con 0x7f091409c320 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091406f6d0 con 0x7f091409aae0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091901b700 10 monclient(hunting): authenticate will time out at 1787.264765s 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41568/0 (socket says 172.21.15.142:41568) 2024-09-18T10:24:58.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 -- 172.21.15.142:0/3375448534 learned_addr learned my addr 172.21.15.142:0/3375448534 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): get_auth_request con 0x7f091409aae0 auth_method 0 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 10 monclient(hunting): get_auth_request con 0x7f09140967a0 auth_method 0 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 1 -- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 msgr2=0x7f091409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 1 -- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 msgr2=0x7f091409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 0x7f091409c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f09137fe700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 10 monclient(hunting): get_auth_request con 0x7f091409c320 auth_method 0 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient(hunting): handle_auth_done global_id 75602 payload 274 2024-09-18T10:24:58.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 -- 172.21.15.142:0/3375448534 >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 msgr2=0x7f091409c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 0x7f091409c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 -- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 msgr2=0x7f091409bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient: _finish_hunting 0 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 monclient: found mon.noname-b 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 -- 172.21.15.142:0/3375448534 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140955a0 con 0x7f091409aae0 2024-09-18T10:24:58.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f091881a700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 0x7f091409c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:24:58.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 -- 172.21.15.142:0/3375448534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 msgr2=0x7f0914096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:58.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.337+0000 7f0913fff700 1 --2- 172.21.15.142:0/3375448534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0914095210 tx=0x7f0904009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6145a308c09025fa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:58.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 1 -- 172.21.15.142:0/3375448534 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090400ba70 con 0x7f091409aae0 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: dump: 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:58.356 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: _finish_auth 0 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 5 monclient: authenticate success, global_id 75602 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 1 -- 172.21.15.142:0/3375448534 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090400bbd0 con 0x7f091409aae0 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: get_monmap_and_config success 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 4 set_mon_vals no callback set 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 1 -- 172.21.15.142:0/3375448534 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904011420 con 0x7f091409aae0 2024-09-18T10:24:58.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0912ffd700 10 monclient: dump: 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:58.358 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: shutdown 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- 172.21.15.142:0/3375448534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 msgr2=0x7f0914096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- 172.21.15.142:0/3375448534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0914095210 tx=0x7f0904009250 comp rx=0 tx=0).stop 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- 172.21.15.142:0/3375448534 shutdown_connections 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.150:3300/0 conn(0x7f091409c320 0x7f091409c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- 172.21.15.142:0/3375448534 >> v2:172.21.15.142:3300/0 conn(0x7f09140967a0 0x7f091409bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- 172.21.15.142:0/3375448534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914096260 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- 172.21.15.142:0/3375448534 >> 172.21.15.142:0/3375448534 conn(0x7f09140a1fe0 msgr2=0x7f0914070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- 172.21.15.142:0/3375448534 shutdown_connections 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- 172.21.15.142:0/3375448534 wait complete. 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: build_initial_monmap 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: monmap: 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:24:58.360 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 Processor -- start 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- start start 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: init 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: _reopen_session rank -1 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): picked mon.noname-a con 0x7f09140967a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): picked mon.noname-c con 0x7f091409aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 0x7f091408d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): picked mon.noname-b con 0x7f091409c320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:24:58.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091406f8d0 con 0x7f09140967a0 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091406ea20 con 0x7f091409c320 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091408fa30 con 0x7f091409aae0 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): _renew_subs 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091901b700 10 monclient(hunting): authenticate will time out at 1787.270717s 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39806/0 (socket says 172.21.15.142:39806) 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 -- 172.21.15.142:0/2235487515 learned_addr learned my addr 172.21.15.142:0/2235487515 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 10 monclient(hunting): get_auth_request con 0x7f09140967a0 auth_method 0 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 msgr2=0x7f09141357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 msgr2=0x7f09141357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f0913fff700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): get_auth_request con 0x7f091409aae0 auth_method 0 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 0x7f091408d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): get_auth_request con 0x7f091409c320 auth_method 0 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient(hunting): handle_auth_done global_id 75605 payload 824 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 msgr2=0x7f091408d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 0x7f091408d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 msgr2=0x7f09141357e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:24:58.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient: _finish_hunting 0 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 monclient: found mon.noname-c 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f09137fe700 1 -- 172.21.15.142:0/2235487515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040094f0 con 0x7f091409aae0 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.341+0000 7f091881a700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 0x7f091408d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f09137fe700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 msgr2=0x7f0914135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f09137fe700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914135d20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f090800d890 tx=0x7f0908008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908012800 con 0x7f091409aae0 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: dump: 2024-09-18T10:24:58.365 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: _finish_auth 0 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908012980 con 0x7f091409aae0 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 5 monclient: authenticate success, global_id 75605 2024-09-18T10:24:58.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 10 monclient: _renew_subs 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091408f520 con 0x7f091409aae0 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908012d40 con 0x7f091409aae0 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f08fbfff700 10 monclient: dump: 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:24:58.367 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f09127fc700 4 set_mon_vals no callback set 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 10 monclient: _renew_subs 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.345+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0914136260 con 0x7f091409aae0 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f091901b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f091901b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f091901b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0914091160 con 0x7f091409aae0 2024-09-18T10:24:58.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0908014640 con 0x7f091409aae0 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f08fbfff700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 0x7f0900037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f09080494c0 con 0x7f091409aae0 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f0913fff700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 0x7f0900037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.349+0000 7f0913fff700 10 monclient: get_auth_request con 0x7f0900035b10 auth_method 0 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.353+0000 7f0913fff700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 0x7f0900037fc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0904009340 tx=0x7f0904017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:24:58.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.357+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0908011db0 con 0x7f091409aae0 2024-09-18T10:24:58.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.357+0000 7f08fbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:24:58.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.357+0000 7f08fbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:24:58.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f091901b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:24:58.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f091901b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:58.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f091901b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:24:58.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0914072a80 con 0x7f091409aae0 2024-09-18T10:24:58.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f08fbfff700 1 -- 172.21.15.142:0/2235487515 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f09080117b0 con 0x7f091409aae0 2024-09-18T10:24:58.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f08fbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:24:58.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.573+0000 7f08fbfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:24:58.581 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":144,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 msgr2=0x7f0900037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 0x7f0900037fc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0904009340 tx=0x7f0904017000 comp rx=0 tx=0).stop 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 10 monclient: shutdown 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 msgr2=0x7f0914135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914135d20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f090800d890 tx=0x7f0908008040 comp rx=0 tx=0).stop 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 shutdown_connections 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0900035b10 0x7f0900037fc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f091409c320 0x7f091408d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f091409aae0 0x7f0914135d20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 --2- 172.21.15.142:0/2235487515 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09140967a0 0x7f09141357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:24:58.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 >> 172.21.15.142:0/2235487515 conn(0x7f09140a1fe0 msgr2=0x7f091409f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:24:58.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 shutdown_connections 2024-09-18T10:24:58.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:24:58.577+0000 7f091901b700 1 -- 172.21.15.142:0/2235487515 wait complete. 2024-09-18T10:25:01.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: get_monmap_and_config 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: build_initial_monmap 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: monmap: 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:01.704865+0000 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:01.704865+0000 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:01.707 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:01.708 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 Processor -- start 2024-09-18T10:25:01.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 -- start start 2024-09-18T10:25:01.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: init 2024-09-18T10:25:01.711 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f65c80a27d0 0x7f65c812ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-a con 0x7f65c80a27d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65c8098730 0x7f65c8098b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-c con 0x7f65c8098730 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65c809c810 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c812f940 con 0x7f65c809c810 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c809e040 con 0x7f65c80a27d0 2024-09-18T10:25:01.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c8096f40 con 0x7f65c8098730 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65cda35700 10 monclient(hunting): authenticate will time out at 1790.630775s 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f65c809c810 0x7f65c809cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:52498/0 (socket says 172.21.15.142:52498) 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c77fe700 1 -- 172.21.15.142:0/3142050968 learned_addr learned my addr 172.21.15.142:0/3142050968 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c6ffd700 1 --2- 172.21.15.142:0/3142050968 >> v2:172.21.15.142:3300/0 conn(0x7f65c80a27d0 0x7f65c812ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request con 0x7f65c80a27d0 auth_method 0 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 1 --2- 172.21.15.142:0/3142050968 >> v2:172.21.15.150:3300/0 conn(0x7f65c8098730 0x7f65c8098b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): get_auth_request con 0x7f65c8098730 auth_method 0 2024-09-18T10:25:01.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.701+0000 7f65c67fc700 10 monclient(hunting): handle_auth_done global_id 174178 payload 274 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 -- 172.21.15.142:0/3142050968 >> v2:172.21.15.142:3300/0 conn(0x7f65c80a27d0 msgr2=0x7f65c812ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 --2- 172.21.15.142:0/3142050968 >> v2:172.21.15.142:3300/0 conn(0x7f65c80a27d0 0x7f65c812ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 -- 172.21.15.142:0/3142050968 >> v1:172.21.15.142:6789/0 conn(0x7f65c809c810 legacy=0x7f65c809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 monclient: found mon.noname-c 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:01.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:25:01.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 -- 172.21.15.142:0/3142050968 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c80a2ba0 con 0x7f65c8098730 2024-09-18T10:25:01.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 -- 172.21.15.142:0/3142050968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c8098730 msgr2=0x7f65c8098b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c67fc700 1 --2- 172.21.15.142:0/3142050968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c8098730 0x7f65c8098b70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f65c8068e70 tx=0x7f65b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c3afa6eb74eefb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 1 -- 172.21.15.142:0/3142050968 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b000b8c0 con 0x7f65c8098730 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: dump: 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.719 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 5 monclient: authenticate success, global_id 174178 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 1 -- 172.21.15.142:0/3142050968 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b000ba20 con 0x7f65c8098730 2024-09-18T10:25:01.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 4 set_mon_vals no callback set 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 1 -- 172.21.15.142:0/3142050968 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b000bdc0 con 0x7f65c8098730 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65c5ffb700 10 monclient: dump: 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:01.721 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 10 monclient: shutdown 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 1 -- 172.21.15.142:0/3142050968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c8098730 msgr2=0x7f65c8098b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.705+0000 7f65cda35700 1 --2- 172.21.15.142:0/3142050968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c8098730 0x7f65c8098b70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f65c8068e70 tx=0x7f65b00091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- 172.21.15.142:0/3142050968 shutdown_connections 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 --2- 172.21.15.142:0/3142050968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c8098730 0x7f65c8098b70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 --2- 172.21.15.142:0/3142050968 >> v2:172.21.15.142:3300/0 conn(0x7f65c80a27d0 0x7f65c812ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- 172.21.15.142:0/3142050968 >> 172.21.15.142:0/3142050968 conn(0x7f65c8071740 msgr2=0x7f65c80a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- 172.21.15.142:0/3142050968 shutdown_connections 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- 172.21.15.142:0/3142050968 wait complete. 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient: build_initial_monmap 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient: monmap: 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:01.723 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 Processor -- start 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- start start 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient: init 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65c8141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-c con 0x7f65c8098730 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-a con 0x7f65c809c810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65c812fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): picked mon.noname-b con 0x7f65c80a27d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c809e200 con 0x7f65c809c810 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c812dd80 con 0x7f65c80a27d0 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c8141e20 con 0x7f65c8098730 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65cda35700 10 monclient(hunting): authenticate will time out at 1790.638352s 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65c8141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39840/0 (socket says 172.21.15.142:39840) 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65c8141560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41594/0 (socket says 172.21.15.142:41594) 2024-09-18T10:25:01.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 learned_addr learned my addr 172.21.15.142:0/4260636474 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): get_auth_request con 0x7f65c809c810 auth_method 0 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request con 0x7f65c8098730 auth_method 0 2024-09-18T10:25:01.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65c812fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): get_auth_request con 0x7f65c80a27d0 auth_method 0 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:01.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient(hunting): handle_auth_done global_id 184101 payload 824 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 msgr2=0x7f65c8141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65c8141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 msgr2=0x7f65c812fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65c812fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 monclient: found mon.noname-a 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:01.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.709+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b0009420 con 0x7f65c809c810 2024-09-18T10:25:01.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 msgr2=0x7f65c812baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f65c80678c0 tx=0x7f65b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 msgr2=0x7f65c812baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 msgr2=0x7f65c812baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f65c80678c0 tx=0x7f65b0016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f65c80678c0 tx=0x7f65b0016000 comp rx=0 tx=0).stop 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 msgr2=0x7f65c812baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65c809c810 0x7f65c812baf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:01.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65a8005520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): picked mon.noname-b con 0x7f65c80a27d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65a8007f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): picked mon.noname-c con 0x7f65c8098730 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 0x7f65a800ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): picked mon.noname-a con 0x7f65a80088e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800b860 con 0x7f65a80088e0 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800b9d0 con 0x7f65c80a27d0 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800bb90 con 0x7f65c8098730 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65beffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:01.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65a8007f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 0x7f65a800ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): get_auth_request con 0x7f65c8098730 auth_method 0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65a8005520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): get_auth_request con 0x7f65a80088e0 auth_method 0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request con 0x7f65c80a27d0 auth_method 0 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:01.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:01.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient(hunting): handle_auth_done global_id 184101 payload 293 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 msgr2=0x7f65a8005520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65a8005520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 msgr2=0x7f65a800ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 0x7f65a800ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 monclient: found mon.noname-c 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a800be10 con 0x7f65c8098730 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.713+0000 7f65c77fe700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 0x7f65a800ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65c67fc700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 msgr2=0x7f65a8007f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:01.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65c67fc700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65a8007f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f65c812f360 tx=0x7f65b001e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0009110 con 0x7f65c8098730 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: dump: 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:01.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 5 monclient: authenticate success, global_id 184101 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b0020070 con 0x7f65c8098730 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 10 monclient: _renew_subs 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c8131ee0 con 0x7f65c8098730 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b001b450 con 0x7f65c8098730 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 10 monclient: dump: 2024-09-18T10:25:01.742 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65c4ff9700 4 set_mon_vals no callback set 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 10 monclient: _renew_subs 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:01.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c8131a50 con 0x7f65c8098730 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.717+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f65b0022030 con 0x7f65c8098730 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65cda35700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65cda35700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65cda35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65beffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 0x7f65a803fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f65b004c030 con 0x7f65c8098730 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c8131a50 con 0x7f65c8098730 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65c6ffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 0x7f65a803fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65c6ffd700 10 monclient: get_auth_request con 0x7f65c809c810 auth_method 0 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.721+0000 7f65c6ffd700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 0x7f65a803fce0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f65b8006fd0 tx=0x7f65b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.729+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f65b0035080 con 0x7f65c8098730 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.729+0000 7f65beffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:01.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.729+0000 7f65beffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:01.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f65c8131a50 con 0x7f65c8098730 2024-09-18T10:25:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65beffd700 1 -- 172.21.15.142:0/4260636474 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f65b0025790 con 0x7f65c8098730 2024-09-18T10:25:01.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65beffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:01.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65beffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:01.969 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 msgr2=0x7f65a803fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 0x7f65a803fce0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f65b8006fd0 tx=0x7f65b8008040 comp rx=0 tx=0).stop 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 10 monclient: shutdown 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 msgr2=0x7f65a8007f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65a8007f10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f65c812f360 tx=0x7f65b001e070 comp rx=0 tx=0).stop 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 shutdown_connections 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f65c809c810 0x7f65a803fce0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f65a80088e0 0x7f65a800ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65c8098730 0x7f65a8007f10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 --2- 172.21.15.142:0/4260636474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65c80a27d0 0x7f65a8005520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.961+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 >> 172.21.15.142:0/4260636474 conn(0x7f65c8071740 msgr2=0x7f65c809cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.965+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 shutdown_connections 2024-09-18T10:25:01.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:01.965+0000 7f65cda35700 1 -- 172.21.15.142:0/4260636474 wait complete. 2024-09-18T10:25:01.980 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:25:01.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:25:01.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.073+0000 7f73abfff700 10 monclient: get_monmap_and_config 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.073+0000 7f73abfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient: monmap: 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:02.079749+0000 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:02.079749+0000 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:02.082 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:02.083 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:02.083 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:02.083 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 Processor -- start 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 -- start start 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient: init 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73ac1320a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-b con 0x7f73ac09f820 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f73ac137410 0x7f73ac1397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-c con 0x7f73ac137410 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac132f10 con 0x7f73ac1320a0 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0a1eb0 con 0x7f73ac137410 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac09fbf0 con 0x7f73ac09f820 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73abfff700 10 monclient(hunting): authenticate will time out at 1791.005542s 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37462/0 (socket says 172.21.15.142:37462) 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 -- 172.21.15.142:0/1132104447 learned_addr learned my addr 172.21.15.142:0/1132104447 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): get_auth_request con 0x7f73ac09f820 auth_method 0 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aaffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f73ac1320a0 0x7f73ac09f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37678/0 (socket says 172.21.15.142:37678) 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73ab7fe700 1 --2- 172.21.15.142:0/1132104447 >> v2:172.21.15.150:3300/0 conn(0x7f73ac137410 0x7f73ac1397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73ab7fe700 10 monclient(hunting): get_auth_request con 0x7f73ac137410 auth_method 0 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_done global_id 75611 payload 274 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 -- 172.21.15.142:0/1132104447 >> v2:172.21.15.150:3300/0 conn(0x7f73ac137410 msgr2=0x7f73ac1397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/1132104447 >> v2:172.21.15.150:3300/0 conn(0x7f73ac137410 0x7f73ac1397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 -- 172.21.15.142:0/1132104447 >> v1:172.21.15.142:6789/0 conn(0x7f73ac1320a0 legacy=0x7f73ac09f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 monclient: found mon.noname-b 2024-09-18T10:25:02.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:02.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.077+0000 7f73aa7fc700 1 -- 172.21.15.142:0/1132104447 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0a1bd0 con 0x7f73ac09f820 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73aa7fc700 1 -- 172.21.15.142:0/1132104447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 msgr2=0x7f73ac132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/1132104447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f739c009f20 tx=0x7f739c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c32a26e8fc21faa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 1 -- 172.21.15.142:0/1132104447 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c00e070 con 0x7f73ac09f820 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: dump: 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.090 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 5 monclient: authenticate success, global_id 75611 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 1 -- 172.21.15.142:0/1132104447 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c00bc90 con 0x7f73ac09f820 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 4 set_mon_vals no callback set 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 1 -- 172.21.15.142:0/1132104447 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c0092e0 con 0x7f73ac09f820 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73a9ffb700 10 monclient: dump: 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 10 monclient: shutdown 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 -- 172.21.15.142:0/1132104447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 msgr2=0x7f73ac132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 --2- 172.21.15.142:0/1132104447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f739c009f20 tx=0x7f739c009840 comp rx=0 tx=0).stop 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 -- 172.21.15.142:0/1132104447 shutdown_connections 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 --2- 172.21.15.142:0/1132104447 >> v2:172.21.15.150:3300/0 conn(0x7f73ac137410 0x7f73ac1397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 --2- 172.21.15.142:0/1132104447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac09f820 0x7f73ac132850 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 -- 172.21.15.142:0/1132104447 >> 172.21.15.142:0/1132104447 conn(0x7f73ac0a5700 msgr2=0x7f73ac0a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 -- 172.21.15.142:0/1132104447 shutdown_connections 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 1 -- 172.21.15.142:0/1132104447 wait complete. 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.081+0000 7f73abfff700 10 monclient: monmap: 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:02.094 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:02.095 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 Processor -- start 2024-09-18T10:25:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 -- start start 2024-09-18T10:25:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient: init 2024-09-18T10:25:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73ac09f820 0x7f73ac06b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-b con 0x7f73ac09f820 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-a con 0x7f73ac1320a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac137410 0x7f73ac06bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): picked mon.noname-c con 0x7f73ac137410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0a0ff0 con 0x7f73ac1320a0 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac09f610 con 0x7f73ac09f820 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac06ea90 con 0x7f73ac137410 2024-09-18T10:25:02.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73abfff700 10 monclient(hunting): authenticate will time out at 1791.013722s 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42390/0 (socket says 172.21.15.142:42390) 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 -- 172.21.15.142:0/2506218546 learned_addr learned my addr 172.21.15.142:0/2506218546 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): get_auth_request con 0x7f73ac1320a0 auth_method 0 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac137410 0x7f73ac06bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73ac09f820 0x7f73ac06b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 10 monclient(hunting): get_auth_request con 0x7f73ac09f820 auth_method 0 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 10 monclient(hunting): get_auth_request con 0x7f73ac137410 auth_method 0 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient(hunting): handle_auth_done global_id 184104 payload 824 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac137410 msgr2=0x7f73ac06bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac137410 0x7f73ac06bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73ac09f820 msgr2=0x7f73ac06b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73ac09f820 0x7f73ac06b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 monclient: found mon.noname-a 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aa7fc700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c009ae0 con 0x7f73ac1320a0 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73ab7fe700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f73ac137410 0x7f73ac06bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.085+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73ac09f820 0x7f73ac06b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aa7fc700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 msgr2=0x7f73ac06b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73ac09ce50 tx=0x7f739c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c0070c0 con 0x7f73ac1320a0 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: dump: 2024-09-18T10:25:02.103 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c012b00 con 0x7f73ac1320a0 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 5 monclient: authenticate success, global_id 184104 2024-09-18T10:25:02.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 monclient: _renew_subs 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac06cff0 con 0x7f73ac1320a0 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c012ea0 con 0x7f73ac1320a0 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 10 monclient: dump: 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.105 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a97fa700 4 set_mon_vals no callback set 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 monclient: _renew_subs 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac06f720 con 0x7f73ac1320a0 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 0x7f73ac06c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac131f50 con 0x7f73ac02ae80 2024-09-18T10:25:02.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 0x7f73ac06c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient: get_auth_request con 0x7f73ac02ae80 auth_method 0 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aa7fc700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73ac09ce50 tx=0x7f739c017000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f739c04a4b0 osd_map(264..264 src has 239..264) v4 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f739c01d630 con 0x7f73ac1320a0 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 10 monclient(hunting): handle_auth_done global_id 184107 payload 824 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73ac06f910 con 0x7f73ac02ae80 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 0x7f738c037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.089+0000 7f73ab7fe700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 0x7f738c037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73ab7fe700 10 monclient: get_auth_request con 0x7f738c035b10 auth_method 0 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 msgr2=0x7f73ac06c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 0x7f73ac06c450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f7394009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394005c80 con 0x7f73ac02ae80 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 10 monclient: dump: 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.108 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7394005de0 con 0x7f73ac02ae80 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:02.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 msgr2=0x7f73ac06c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 0x7f73ac06c450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f7394009040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 msgr2=0x7f73ac06c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac02ae80 0x7f73ac06c450 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f7394009040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.093+0000 7f73ab7fe700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 0x7f738c037fc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006fd0 tx=0x7f73a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73abfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73abfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac064680 con 0x7f73ac05e890 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient: get_auth_request con 0x7f73ac05e890 auth_method 0 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 10 monclient(hunting): handle_auth_done global_id 184110 payload 824 2024-09-18T10:25:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.105+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f73ac06c990 con 0x7f73ac05e890 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 msgr2=0x7f73ac05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f739401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 msgr2=0x7f73ac05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 msgr2=0x7f73ac05ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f739401a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f739400e740 tx=0x7f739401a000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 msgr2=0x7f73ac05ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 0x7f738c03ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 msgr2=0x7f73ac05ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac05e890 0x7f73ac05ec90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c03b130 con 0x7f738c038770 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 0x7f738c03ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient: get_auth_request con 0x7f738c038770 auth_method 0 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 10 monclient(hunting): handle_auth_done global_id 184113 payload 824 2024-09-18T10:25:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7394005de0 con 0x7f738c038770 2024-09-18T10:25:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 msgr2=0x7f738c03ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.109+0000 7f73aaffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 0x7f738c03ab50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0647c0 tx=0x7f7394021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739402c070 con 0x7f738c038770 2024-09-18T10:25:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 10 monclient: dump: 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f73940203f0 con 0x7f738c038770 2024-09-18T10:25:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 msgr2=0x7f738c03ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 0x7f738c03ab50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0647c0 tx=0x7f7394021000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73ab7fe700 1 -- 172.21.15.142:0/2506218546 reap_dead start 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 msgr2=0x7f738c03ab50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:02.120 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.121 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.122 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.123 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.124 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:02.125 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.126 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.127 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.128 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.129 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.130 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:25:02.131 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:02.132 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:02.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f738c038770 0x7f738c03ab50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 msgr2=0x7f738c037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 0x7f738c037fc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f73a4006fd0 tx=0x7f73a4008040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 10 monclient: shutdown 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 msgr2=0x7f73ac06b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73ac09ce50 tx=0x7f739c017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73a2ffd700 1 -- 172.21.15.142:0/2506218546 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7f739c04a4b0 con 0x7f73ac1320a0 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 shutdown_connections 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f738c035b10 0x7f738c037fc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 --2- 172.21.15.142:0/2506218546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f73ac1320a0 0x7f73ac06b9d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 >> 172.21.15.142:0/2506218546 conn(0x7f73ac0a5700 msgr2=0x7f73ac138870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 shutdown_connections 2024-09-18T10:25:02.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.113+0000 7f73abfff700 1 -- 172.21.15.142:0/2506218546 wait complete. 2024-09-18T10:25:02.134 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:02.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:25:02.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.233+0000 7f8991c54700 10 monclient: get_monmap_and_config 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.233+0000 7f8991c54700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.233+0000 7f8991c54700 10 monclient: monmap: 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.241 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:02.239228+0000 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:02.239228+0000 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:02.242 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 Processor -- start 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 -- start start 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient: init 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:25:02.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f898c09abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f898c097880 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f898c09db80 0x7f898c1386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-a con 0x7f898c09db80 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c0958c0 con 0x7f898c097880 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c095ef0 con 0x7f898c09abd0 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c06fa60 con 0x7f898c09db80 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f8991c54700 10 monclient(hunting): authenticate will time out at 1791.165487s 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898affd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f898c097880 0x7f898c09c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37682/0 (socket says 172.21.15.142:37682) 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898affd700 1 -- 172.21.15.142:0/2493975633 learned_addr learned my addr 172.21.15.142:0/2493975633 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898bfff700 1 --2- 172.21.15.142:0/2493975633 >> v2:172.21.15.142:3300/0 conn(0x7f898c09db80 0x7f898c1386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898bfff700 10 monclient(hunting): get_auth_request con 0x7f898c09db80 auth_method 0 2024-09-18T10:25:02.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.237+0000 7f898bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898b7fe700 1 -- 172.21.15.142:0/2493975633 >> v1:172.21.15.150:6789/0 conn(0x7f898c09abd0 legacy=0x7f898c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898affd700 1 -- 172.21.15.142:0/2493975633 >> v1:172.21.15.142:6789/0 conn(0x7f898c097880 legacy=0x7f898c09c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 1 -- 172.21.15.142:0/2493975633 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917876984 0 0) 0x7f898c095ef0 con 0x7f898c09abd0 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient(hunting): my global_id is 174190 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 1 -- 172.21.15.142:0/2493975633 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8974003570 con 0x7f898c09abd0 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient(hunting): handle_auth_done global_id 184116 payload 274 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 -- 172.21.15.142:0/2493975633 >> v1:172.21.15.150:6789/0 conn(0x7f898c09abd0 legacy=0x7f898c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 -- 172.21.15.142:0/2493975633 >> v1:172.21.15.142:6789/0 conn(0x7f898c097880 legacy=0x7f898c09c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 monclient: found mon.noname-a 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 -- 172.21.15.142:0/2493975633 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c09afa0 con 0x7f898c09db80 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 -- 172.21.15.142:0/2493975633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09db80 msgr2=0x7f898c1386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898bfff700 1 --2- 172.21.15.142:0/2493975633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09db80 0x7f898c1386a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8984009f60 tx=0x7f8984009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2a0cfab50bf5a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 1 -- 172.21.15.142:0/2493975633 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898400e070 con 0x7f898c09db80 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: dump: 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.250 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f8991c54700 5 monclient: authenticate success, global_id 184116 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f8991c54700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 1 -- 172.21.15.142:0/2493975633 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898400be40 con 0x7f898c09db80 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f8991c54700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f8991c54700 4 set_mon_vals no callback set 2024-09-18T10:25:02.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 1 -- 172.21.15.142:0/2493975633 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984012660 con 0x7f898c09db80 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f898a7fc700 10 monclient: dump: 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.252 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.241+0000 7f8991c54700 10 monclient: shutdown 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- 172.21.15.142:0/2493975633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09db80 msgr2=0x7f898c1386a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 --2- 172.21.15.142:0/2493975633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09db80 0x7f898c1386a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8984009f60 tx=0x7f8984009780 comp rx=0 tx=0).stop 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- 172.21.15.142:0/2493975633 shutdown_connections 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 --2- 172.21.15.142:0/2493975633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09db80 0x7f898c1386a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- 172.21.15.142:0/2493975633 >> 172.21.15.142:0/2493975633 conn(0x7f898c0a14e0 msgr2=0x7f898c0a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- 172.21.15.142:0/2493975633 shutdown_connections 2024-09-18T10:25:02.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- 172.21.15.142:0/2493975633 wait complete. 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient: monmap: 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:02.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 Processor -- start 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- start start 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient: init 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f898c097880 0x7f898c094910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-b con 0x7f898c097880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-a con 0x7f898c09abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c09db80 0x7f898c08bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): picked mon.noname-c con 0x7f898c09db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c08fb20 con 0x7f898c09abd0 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c097c90 con 0x7f898c097880 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c06eb20 con 0x7f898c09db80 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f8991c54700 10 monclient(hunting): authenticate will time out at 1791.173826s 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42418/0 (socket says 172.21.15.142:42418) 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 -- 172.21.15.142:0/2975225022 learned_addr learned my addr 172.21.15.142:0/2975225022 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): get_auth_request con 0x7f898c09abd0 auth_method 0 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c09db80 0x7f898c08bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): get_auth_request con 0x7f898c09db80 auth_method 0 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f898c097880 0x7f898c094910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient(hunting): handle_auth_done global_id 184122 payload 824 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c09db80 msgr2=0x7f898c08bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c09db80 0x7f898c08bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f898c097880 msgr2=0x7f898c094910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f898c097880 0x7f898c094910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 monclient: found mon.noname-a 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898affd700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984009a20 con 0x7f898c09abd0 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898b7fe700 10 monclient: get_auth_request con 0x7f898c097880 auth_method 0 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.245+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f898c097880 0x7f898c094910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898affd700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 msgr2=0x7f898c08b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898affd700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8980010bd0 tx=0x7f8980007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980005d40 con 0x7f898c09abd0 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: dump: 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.262 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 5 monclient: authenticate success, global_id 184122 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89800070f0 con 0x7f898c09abd0 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 monclient: _renew_subs 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c08ee60 con 0x7f898c09abd0 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980005ea0 con 0x7f898c09abd0 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 10 monclient: dump: 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 monclient: _renew_subs 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8989ffb700 4 set_mon_vals no callback set 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c094e50 con 0x7f898c09abd0 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 0x7f898c08c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c064950 con 0x7f898c08df00 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 0x7f898c08c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient: get_auth_request con 0x7f898c08df00 auth_method 0 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f898000a590 con 0x7f898c09abd0 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 10 monclient(hunting): handle_auth_done global_id 75620 payload 824 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898b7fe700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f898c09a9d0 con 0x7f898c08df00 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8978035ac0 0x7f8978037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f898004c810 con 0x7f898c09abd0 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898bfff700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8978035ac0 0x7f8978037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.249+0000 7f898bfff700 10 monclient: get_auth_request con 0x7f8978035ac0 auth_method 0 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f898b7fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 msgr2=0x7f898c08c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 0x7f898c08c300 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c007510 con 0x7f898c08df00 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 10 monclient: dump: 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.267 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f897c007670 con 0x7f898c08df00 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 msgr2=0x7f898c08c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 0x7f898c08c300 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c009040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 msgr2=0x7f898c08c300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f89737fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c08df00 0x7f898c08c300 secure :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c009040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.253+0000 7f898bfff700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8978035ac0 0x7f8978037f70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f898c09dfa0 tx=0x7f8984017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f8991c54700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:02.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f8991c54700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 0x7f898c087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f8991c54700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0696a0 con 0x7f898c094130 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 0x7f898c087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient: get_auth_request con 0x7f898c094130 auth_method 0 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 10 monclient(hunting): handle_auth_done global_id 75623 payload 824 2024-09-18T10:25:02.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 1 -- 172.21.15.142:0/2975225022 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f898c067950 con 0x7f898c094130 2024-09-18T10:25:02.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 msgr2=0x7f898c087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f898b7fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 0x7f898c087fe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c012e40 con 0x7f898c094130 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f89737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f89737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.265+0000 7f89737fe700 10 monclient: dump: 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f897c01b610 con 0x7f898c094130 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 msgr2=0x7f898c087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 0x7f898c087fe0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 msgr2=0x7f898c087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f89737fe700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f898c094130 0x7f898c087fe0 secure :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f897c001520 tx=0x7f897c017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.275 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.276 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:02.277 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:02.278 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.279 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.280 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:02.283 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.284 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.285 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.286 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:02.287 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.288 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:02.289 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.290 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:02.290 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8978035ac0 msgr2=0x7f8978037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8978035ac0 0x7f8978037f70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f898c09dfa0 tx=0x7f8984017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 10 monclient: shutdown 2024-09-18T10:25:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 msgr2=0x7f898c08b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f898bfff700 1 -- 172.21.15.142:0/2975225022 reap_dead start 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8980010bd0 tx=0x7f8980007e50 comp rx=0 tx=0).stop 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 shutdown_connections 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 --2- 172.21.15.142:0/2975225022 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f898c09abd0 0x7f898c08b880 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 >> 172.21.15.142:0/2975225022 conn(0x7f898c0a14e0 msgr2=0x7f898c06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 shutdown_connections 2024-09-18T10:25:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.269+0000 7f8991c54700 1 -- 172.21.15.142:0/2975225022 wait complete. 2024-09-18T10:25:02.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:02.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:25:02.293 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: get_monmap_and_config 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: monmap: 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:02.402047+0000 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:02.402047+0000 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:02.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 1 Processor -- start 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 1 -- start start 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: init 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f21b406f3b0 0x7f21b40989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-c con 0x7f21b406f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:02.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21b4098f00 0x7f21b4099320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.397+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-a con 0x7f21b4098f00 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21b4133660 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b409c330 con 0x7f21b4133660 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b4099860 con 0x7f21b4098f00 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b409b030 con 0x7f21b406f3b0 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21ba311700 10 monclient(hunting): authenticate will time out at 1791.328140s 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21b4098f00 0x7f21b4099320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f21b4098f00 0x7f21b4099320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42434/0 (socket says 172.21.15.142:42434) 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 -- 172.21.15.142:0/4270800169 learned_addr learned my addr 172.21.15.142:0/4270800169 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request con 0x7f21b4098f00 auth_method 0 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 1 --2- 172.21.15.142:0/4270800169 >> v2:172.21.15.150:3300/0 conn(0x7f21b406f3b0 0x7f21b40989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): get_auth_request con 0x7f21b406f3b0 auth_method 0 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_done global_id 184125 payload 274 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 -- 172.21.15.142:0/4270800169 >> v2:172.21.15.150:3300/0 conn(0x7f21b406f3b0 msgr2=0x7f21b40989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/4270800169 >> v2:172.21.15.150:3300/0 conn(0x7f21b406f3b0 0x7f21b40989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 -- 172.21.15.142:0/4270800169 >> v1:172.21.15.150:6789/0 conn(0x7f21b4133660 legacy=0x7f21b4135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 monclient: found mon.noname-a 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 -- 172.21.15.142:0/4270800169 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b409c530 con 0x7f21b4098f00 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 -- 172.21.15.142:0/4270800169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4098f00 msgr2=0x7f21b4099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/4270800169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4098f00 0x7f21b4099320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f21b4076de0 tx=0x7f21a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50e38b21d22405c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b3fff700 1 -- 172.21.15.142:0/4270800169 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a400e070 con 0x7f21b4098f00 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.401+0000 7f21b3fff700 10 monclient: dump: 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 5 monclient: authenticate success, global_id 184125 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 1 -- 172.21.15.142:0/4270800169 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a400bc30 con 0x7f21b4098f00 2024-09-18T10:25:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 4 set_mon_vals no callback set 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 1 -- 172.21.15.142:0/4270800169 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a40092e0 con 0x7f21b4098f00 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b3fff700 10 monclient: dump: 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: shutdown 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- 172.21.15.142:0/4270800169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4098f00 msgr2=0x7f21b4099320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- 172.21.15.142:0/4270800169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4098f00 0x7f21b4099320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f21b4076de0 tx=0x7f21a4009840 comp rx=0 tx=0).stop 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- 172.21.15.142:0/4270800169 shutdown_connections 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- 172.21.15.142:0/4270800169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4098f00 0x7f21b4099320 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- 172.21.15.142:0/4270800169 >> v2:172.21.15.150:3300/0 conn(0x7f21b406f3b0 0x7f21b40989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- 172.21.15.142:0/4270800169 >> 172.21.15.142:0/4270800169 conn(0x7f21b4070ef0 msgr2=0x7f21b40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- 172.21.15.142:0/4270800169 shutdown_connections 2024-09-18T10:25:02.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- 172.21.15.142:0/4270800169 wait complete. 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: monmap: 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 Processor -- start 2024-09-18T10:25:02.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- start start 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: init 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 0x7f21b4127f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-c con 0x7f21b406f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4098f00 0x7f21b4128490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-b con 0x7f21b4098f00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4133660 0x7f21b41289d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): picked mon.noname-a con 0x7f21b4133660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b412b900 con 0x7f21b4133660 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b412a010 con 0x7f21b4098f00 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b41298e0 con 0x7f21b406f3b0 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21ba311700 10 monclient(hunting): authenticate will time out at 1791.335517s 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 0x7f21b4127f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b9b10700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4133660 0x7f21b41289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 0x7f21b4127f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37484/0 (socket says 172.21.15.142:37484) 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 1 -- 172.21.15.142:0/990057386 learned_addr learned my addr 172.21.15.142:0/990057386 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): get_auth_request con 0x7f21b406f3b0 auth_method 0 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b9b10700 10 monclient(hunting): get_auth_request con 0x7f21b4133660 auth_method 0 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b9b10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b9b10700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b9b10700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4098f00 0x7f21b4128490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b930f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request con 0x7f21b4098f00 auth_method 0 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.405+0000 7f21b8b0e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b8b0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b9b10700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b9b10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b9b10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 10 monclient(hunting): handle_auth_done global_id 75626 payload 824 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4098f00 msgr2=0x7f21b4128490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4098f00 0x7f21b4128490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4133660 msgr2=0x7f21b41289d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21b4133660 0x7f21b41289d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 monclient: found mon.noname-c 2024-09-18T10:25:02.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:02.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4009ae0 con 0x7f21b406f3b0 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 msgr2=0x7f21b4127f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b930f700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 0x7f21b4127f50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21a800e950 tx=0x7f21a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8013890 con 0x7f21b406f3b0 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: dump: 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 5 monclient: authenticate success, global_id 75626 2024-09-18T10:25:02.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 10 monclient: _renew_subs 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8013a10 con 0x7f21b406f3b0 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b406fa20 con 0x7f21b406f3b0 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8013dd0 con 0x7f21b406f3b0 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b37fe700 4 set_mon_vals no callback set 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21b17fa700 10 monclient: dump: 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.430 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 10 monclient: _renew_subs 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.409+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b412a570 con 0x7f21b406f3b0 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21ba311700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21ba311700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 0x7f21b409cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b40648a0 con 0x7f21b4128f10 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f21a80125e0 con 0x7f21b406f3b0 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21a0035ac0 0x7f21a0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 0x7f21b409cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient: get_auth_request con 0x7f21b4128f10 auth_method 0 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f21a8048e70 con 0x7f21b406f3b0 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b9b10700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21a0035ac0 0x7f21a0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b9b10700 10 monclient: get_auth_request con 0x7f21a0035ac0 auth_method 0 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_done global_id 174199 payload 824 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b4072a80 con 0x7f21b4128f10 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 msgr2=0x7f21b409cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 0x7f21b409cf30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21b4076de0 tx=0x7f21a4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4012c10 con 0x7f21b4128f10 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b17fa700 10 monclient: dump: 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.433 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.413+0000 7f21b9b10700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21a0035ac0 0x7f21a0037f70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21b409bff0 tx=0x7f21ac007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f21a4026420 con 0x7f21b4128f10 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 msgr2=0x7f21b409cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 0x7f21b409cf30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21b4076de0 tx=0x7f21a4017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 msgr2=0x7f21b409cf30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.417+0000 7f21b17fa700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b4128f10 0x7f21b409cf30 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f21b4076de0 tx=0x7f21a4017000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21ba311700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:02.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21ba311700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 0x7f21b4029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21ba311700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b412a570 con 0x7f21b409d470 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 0x7f21b4029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient: get_auth_request con 0x7f21b409d470 auth_method 0 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 10 monclient(hunting): handle_auth_done global_id 174202 payload 824 2024-09-18T10:25:02.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.425+0000 7f21b8b0e700 1 -- 172.21.15.142:0/990057386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21b406dad0 con 0x7f21b409d470 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b8b0e700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 msgr2=0x7f21b4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b8b0e700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 0x7f21b4029d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21b40677a0 tx=0x7f21a4020070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a401e820 con 0x7f21b409d470 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 10 monclient: dump: 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f21a401e980 con 0x7f21b409d470 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 msgr2=0x7f21b4029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 0x7f21b4029d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21b40677a0 tx=0x7f21a4020070 comp rx=0 tx=0).stop 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 msgr2=0x7f21b4029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:02.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.429+0000 7f21b17fa700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f21b409d470 0x7f21b4029d00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21b40677a0 tx=0x7f21a4020070 comp rx=0 tx=0).stop 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.441 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.442 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:02.443 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:02.444 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.446 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:02.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.450 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:25:02.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:02.452 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21a0035ac0 msgr2=0x7f21a0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f21a0035ac0 0x7f21a0037f70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21b409bff0 tx=0x7f21ac007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 10 monclient: shutdown 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 msgr2=0x7f21b4127f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 --2- 172.21.15.142:0/990057386 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21b406f3b0 0x7f21b4127f50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21a800e950 tx=0x7f21a80091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21b9b10700 1 -- 172.21.15.142:0/990057386 reap_dead start 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21b9b10700 1 -- 172.21.15.142:0/990057386 reap_dead start 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 shutdown_connections 2024-09-18T10:25:02.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 >> 172.21.15.142:0/990057386 conn(0x7f21b4070ef0 msgr2=0x7f21b4132d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 shutdown_connections 2024-09-18T10:25:02.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.433+0000 7f21ba311700 1 -- 172.21.15.142:0/990057386 wait complete. 2024-09-18T10:25:02.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:02.455 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:25:02.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: monmap: 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:02.563810+0000 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:02.563810+0000 2024-09-18T10:25:02.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:02.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 Processor -- start 2024-09-18T10:25:02.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 -- start start 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: init 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc2e8132030 0x7fc2e8132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc2e8132030 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc2e809ece0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2e809ae60 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e8132940 con 0x7fc2e809ae60 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e809f960 con 0x7fc2e8132030 2024-09-18T10:25:02.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80a1a90 con 0x7fc2e809ece0 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e7fff700 10 monclient(hunting): authenticate will time out at 1791.489356s 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37496/0 (socket says 172.21.15.142:37496) 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 -- 172.21.15.142:0/725014737 learned_addr learned my addr 172.21.15.142:0/725014737 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): get_auth_request con 0x7fc2e809ece0 auth_method 0 2024-09-18T10:25:02.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 1 --2- 172.21.15.142:0/725014737 >> v2:172.21.15.150:3300/0 conn(0x7fc2e8132030 0x7fc2e8132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): get_auth_request con 0x7fc2e8132030 auth_method 0 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_done global_id 75629 payload 274 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 -- 172.21.15.142:0/725014737 >> v2:172.21.15.150:3300/0 conn(0x7fc2e8132030 msgr2=0x7fc2e8132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 --2- 172.21.15.142:0/725014737 >> v2:172.21.15.150:3300/0 conn(0x7fc2e8132030 0x7fc2e8132400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 -- 172.21.15.142:0/725014737 >> v1:172.21.15.150:6789/0 conn(0x7fc2e809ae60 legacy=0x7fc2e809b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 monclient: found mon.noname-b 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:02.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.561+0000 7fc2e67fc700 1 -- 172.21.15.142:0/725014737 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e80a17b0 con 0x7fc2e809ece0 2024-09-18T10:25:02.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e67fc700 1 -- 172.21.15.142:0/725014737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 msgr2=0x7fc2e809f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e67fc700 1 --2- 172.21.15.142:0/725014737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00099b0 tx=0x7fc2d0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27977c27f6530779 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 1 -- 172.21.15.142:0/725014737 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0009ea0 con 0x7fc2e809ece0 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: dump: 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 5 monclient: authenticate success, global_id 75629 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 1 -- 172.21.15.142:0/725014737 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d000ba70 con 0x7fc2e809ece0 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 4 set_mon_vals no callback set 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 1 -- 172.21.15.142:0/725014737 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d000be30 con 0x7fc2e809ece0 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:02.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e5ffb700 10 monclient: dump: 2024-09-18T10:25:02.582 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.582 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.582 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.582 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 10 monclient: shutdown 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- 172.21.15.142:0/725014737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 msgr2=0x7fc2e809f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/725014737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00099b0 tx=0x7fc2d0009250 comp rx=0 tx=0).stop 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- 172.21.15.142:0/725014737 shutdown_connections 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/725014737 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e809ece0 0x7fc2e809f0b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/725014737 >> v2:172.21.15.150:3300/0 conn(0x7fc2e8132030 0x7fc2e8132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- 172.21.15.142:0/725014737 >> 172.21.15.142:0/725014737 conn(0x7fc2e80a5700 msgr2=0x7fc2e80a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- 172.21.15.142:0/725014737 shutdown_connections 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- 172.21.15.142:0/725014737 wait complete. 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 10 monclient: build_initial_monmap 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 10 monclient: monmap: 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:02.584 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 Processor -- start 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 1 -- start start 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.565+0000 7fc2e7fff700 10 monclient: init 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:02.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fc2e809ae60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2e809ece0 0x7fc2e8131050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fc2e809ece0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e8132030 0x7fc2e8131590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fc2e8132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e811e850 con 0x7fc2e809ae60 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80a4840 con 0x7fc2e809ece0 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e806f5c0 con 0x7fc2e8132030 2024-09-18T10:25:02.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e7fff700 10 monclient(hunting): authenticate will time out at 1791.496855s 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42456/0 (socket says 172.21.15.142:42456) 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 -- 172.21.15.142:0/887770494 learned_addr learned my addr 172.21.15.142:0/887770494 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): get_auth_request con 0x7fc2e809ae60 auth_method 0 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e8132030 0x7fc2e8131590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): get_auth_request con 0x7fc2e8132030 auth_method 0 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2e809ece0 0x7fc2e8131050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): get_auth_request con 0x7fc2e809ece0 auth_method 0 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient(hunting): handle_auth_done global_id 184131 payload 824 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 -- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e8132030 msgr2=0x7fc2e8131590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e8132030 0x7fc2e8131590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 -- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2e809ece0 msgr2=0x7fc2e8131050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2e809ece0 0x7fc2e8131050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 monclient: found mon.noname-a 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.569+0000 7fc2e6ffd700 1 -- 172.21.15.142:0/887770494 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d00094f0 con 0x7fc2e809ae60 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e6ffd700 1 -- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 msgr2=0x7fc2e806ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e6ffd700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009f40 tx=0x7fc2e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000cd80 con 0x7fc2e809ae60 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: dump: 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.590 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.591 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.591 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000f040 con 0x7fc2e809ae60 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 5 monclient: authenticate success, global_id 184131 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _renew_subs 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e811e190 con 0x7fc2e809ae60 2024-09-18T10:25:02.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000cee0 con 0x7fc2e809ae60 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 10 monclient: dump: 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.593 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:02.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:02.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:02.595 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:02.595 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:02.595 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:02.596 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:02.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e57fa700 4 set_mon_vals no callback set 2024-09-18T10:25:02.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _renew_subs 2024-09-18T10:25:02.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8141bb0 con 0x7fc2e809ae60 2024-09-18T10:25:02.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc2e0014920 con 0x7fc2e809ae60 2024-09-18T10:25:02.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 0x7fc2c8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:02.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc2e0049930 con 0x7fc2e809ae60 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e8141e00 con 0x7fc2e809ae60 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e67fc700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 0x7fc2c8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.573+0000 7fc2e67fc700 10 monclient: get_auth_request con 0x7fc2c8035ac0 auth_method 0 2024-09-18T10:25:02.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.577+0000 7fc2e67fc700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 0x7fc2c8037f70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00099b0 tx=0x7fc2d0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:02.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.585+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc2e0018070 con 0x7fc2e809ae60 2024-09-18T10:25:02.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.585+0000 7fc2deffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:02.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.585+0000 7fc2deffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:02.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.801+0000 7fc2e7fff700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.801+0000 7fc2e7fff700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.801+0000 7fc2e7fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.801+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc2e81425a0 con 0x7fc2e809ae60 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2deffd700 1 -- 172.21.15.142:0/887770494 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fc2e0012340 con 0x7fc2e809ae60 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2deffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:25:02.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2deffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:02.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=6,logm=52} crc {auth=3875052248,config=3140514009,health=3405284703,kv=1327097124,logm=1747466251}) 2024-09-18T10:25:02.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=7,logm=52} crc {auth=1916591536,config=3140514009,health=3405284703,kv=1523155648,logm=78194744}) 2024-09-18T10:25:02.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=6,logm=52} crc {auth=3875052248,config=3140514009,health=3405284703,kv=1327097124,logm=1747466251}) 2024-09-18T10:25:02.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3189895749,config=3140514009,health=59277812,kv=4261215878,logm=3923021271}) 2024-09-18T10:25:02.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 msgr2=0x7fc2c8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 0x7fc2c8037f70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00099b0 tx=0x7fc2d0016000 comp rx=0 tx=0).stop 2024-09-18T10:25:02.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 10 monclient: shutdown 2024-09-18T10:25:02.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:02.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 msgr2=0x7fc2e806ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009f40 tx=0x7fc2e000b040 comp rx=0 tx=0).stop 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 shutdown_connections 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc2c8035ac0 0x7fc2c8037f70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2e8132030 0x7fc2e8131590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc2e809ece0 0x7fc2e8131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 --2- 172.21.15.142:0/887770494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2e809ae60 0x7fc2e806ea90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 >> 172.21.15.142:0/887770494 conn(0x7fc2e80a5700 msgr2=0x7fc2e80a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:02.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 shutdown_connections 2024-09-18T10:25:02.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:02.805+0000 7fc2e7fff700 1 -- 172.21.15.142:0/887770494 wait complete. 2024-09-18T10:25:02.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1150315872}) 2024-09-18T10:25:02.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1241958906}) 2024-09-18T10:25:02.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.818 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1150315872}) 2024-09-18T10:25:02.818 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.809+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818239497}) 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435036386}) 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279296116}) 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435036386}) 2024-09-18T10:25:02.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.813+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342273661}) 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278993775}) 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202337089}) 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278993775}) 2024-09-18T10:25:02.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3822061196}) 2024-09-18T10:25:02.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245928992}) 2024-09-18T10:25:02.824 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418296968}) 2024-09-18T10:25:02.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245928992}) 2024-09-18T10:25:02.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.817+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104083555}) 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769010432}) 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809266882}) 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769010432}) 2024-09-18T10:25:02.826 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864467196}) 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12020303}) 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079257703}) 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12020303}) 2024-09-18T10:25:02.827 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700474904}) 2024-09-18T10:25:02.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186516365}) 2024-09-18T10:25:02.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29143699}) 2024-09-18T10:25:02.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186516365}) 2024-09-18T10:25:02.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.821+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337271026}) 2024-09-18T10:25:02.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:25:02.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887230549}) 2024-09-18T10:25:02.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664912331}) 2024-09-18T10:25:02.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887230549}) 2024-09-18T10:25:02.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1637276746}) 2024-09-18T10:25:02.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708539037}) 2024-09-18T10:25:02.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058045816}) 2024-09-18T10:25:02.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708539037}) 2024-09-18T10:25:02.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.825+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871716861}) 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130366384}) 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783503044}) 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130366384}) 2024-09-18T10:25:02.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713372900}) 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515972992}) 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225310367}) 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515972992}) 2024-09-18T10:25:02.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.829+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771237804}) 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335214756}) 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676360693}) 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335214756}) 2024-09-18T10:25:02.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164266966}) 2024-09-18T10:25:02.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276080402}) 2024-09-18T10:25:02.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421564703}) 2024-09-18T10:25:02.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276080402}) 2024-09-18T10:25:02.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3692074841}) 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.833+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106951333}) 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379518449}) 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106951333}) 2024-09-18T10:25:02.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972178944}) 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384359159}) 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1456821097}) 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384359159}) 2024-09-18T10:25:02.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953641762}) 2024-09-18T10:25:02.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896899427}) 2024-09-18T10:25:02.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967858207}) 2024-09-18T10:25:02.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896899427}) 2024-09-18T10:25:02.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.837+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972685981}) 2024-09-18T10:25:02.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481473032}) 2024-09-18T10:25:02.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2646092288}) 2024-09-18T10:25:02.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481473032}) 2024-09-18T10:25:02.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626130029}) 2024-09-18T10:25:02.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067474721}) 2024-09-18T10:25:02.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071205481}) 2024-09-18T10:25:02.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067474721}) 2024-09-18T10:25:02.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428748352}) 2024-09-18T10:25:02.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3514289157}) 2024-09-18T10:25:02.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109948658}) 2024-09-18T10:25:02.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3514289157}) 2024-09-18T10:25:02.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.841+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309818206}) 2024-09-18T10:25:02.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829620934}) 2024-09-18T10:25:02.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2694501998}) 2024-09-18T10:25:02.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829620934}) 2024-09-18T10:25:02.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698603954}) 2024-09-18T10:25:02.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396507377}) 2024-09-18T10:25:02.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421302418}) 2024-09-18T10:25:02.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396507377}) 2024-09-18T10:25:02.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15404059}) 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493819965}) 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3990365435}) 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493819965}) 2024-09-18T10:25:02.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.845+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3351504155}) 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305127314}) 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174996859}) 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305127314}) 2024-09-18T10:25:02.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537267997}) 2024-09-18T10:25:02.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421317502}) 2024-09-18T10:25:02.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1704174785}) 2024-09-18T10:25:02.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421317502}) 2024-09-18T10:25:02.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211138703}) 2024-09-18T10:25:02.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267550793}) 2024-09-18T10:25:02.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491185838}) 2024-09-18T10:25:02.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2267550793}) 2024-09-18T10:25:02.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.849+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761264738}) 2024-09-18T10:25:02.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234665852}) 2024-09-18T10:25:02.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=485775667}) 2024-09-18T10:25:02.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1234665852}) 2024-09-18T10:25:02.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3216508116}) 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596475441}) 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435570397}) 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596475441}) 2024-09-18T10:25:02.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.853+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1772421878}) 2024-09-18T10:25:02.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427414246}) 2024-09-18T10:25:02.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591253105}) 2024-09-18T10:25:02.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427414246}) 2024-09-18T10:25:02.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2524426056}) 2024-09-18T10:25:02.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580929075}) 2024-09-18T10:25:02.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502099465}) 2024-09-18T10:25:02.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580929075}) 2024-09-18T10:25:02.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.857+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612622794}) 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950929540}) 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056563855}) 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1950929540}) 2024-09-18T10:25:02.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922988352}) 2024-09-18T10:25:02.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411438638}) 2024-09-18T10:25:02.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3405999067}) 2024-09-18T10:25:02.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411438638}) 2024-09-18T10:25:02.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639611907}) 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419066805}) 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1536758981}) 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419066805}) 2024-09-18T10:25:02.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.861+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936142874}) 2024-09-18T10:25:02.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173855450}) 2024-09-18T10:25:02.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=840350249}) 2024-09-18T10:25:02.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173855450}) 2024-09-18T10:25:02.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352497369}) 2024-09-18T10:25:02.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212264526}) 2024-09-18T10:25:02.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2618483402}) 2024-09-18T10:25:02.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212264526}) 2024-09-18T10:25:02.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355869609}) 2024-09-18T10:25:02.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494772805}) 2024-09-18T10:25:02.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2614281460}) 2024-09-18T10:25:02.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494772805}) 2024-09-18T10:25:02.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.865+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237925206}) 2024-09-18T10:25:02.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285103677}) 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122632630}) 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=285103677}) 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3741414967}) 2024-09-18T10:25:02.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281365046}) 2024-09-18T10:25:02.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231832536}) 2024-09-18T10:25:02.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281365046}) 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.869+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723375604}) 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365181671}) 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134669030}) 2024-09-18T10:25:02.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365181671}) 2024-09-18T10:25:02.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612709018}) 2024-09-18T10:25:02.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068648230}) 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507986453}) 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068648230}) 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211645035}) 2024-09-18T10:25:02.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080374813}) 2024-09-18T10:25:02.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935188151}) 2024-09-18T10:25:02.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080374813}) 2024-09-18T10:25:02.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.873+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178565740}) 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291541531}) 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141657970}) 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291541531}) 2024-09-18T10:25:02.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045353922}) 2024-09-18T10:25:02.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534443506}) 2024-09-18T10:25:02.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579009901}) 2024-09-18T10:25:02.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534443506}) 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1378192186}) 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799290128}) 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30821333}) 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799290128}) 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.877+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651498387}) 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266720100}) 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992030973}) 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266720100}) 2024-09-18T10:25:02.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581862280}) 2024-09-18T10:25:02.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200544171}) 2024-09-18T10:25:02.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381150383}) 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200544171}) 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1086331226}) 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.881+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932121028}) 2024-09-18T10:25:02.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2425331659}) 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=932121028}) 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159834509}) 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208943049}) 2024-09-18T10:25:02.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060136391}) 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208943049}) 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109802359}) 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559065450}) 2024-09-18T10:25:02.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731324790}) 2024-09-18T10:25:02.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559065450}) 2024-09-18T10:25:02.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.885+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433931102}) 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447617466}) 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943472336}) 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447617466}) 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957166497}) 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037711042}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11027017}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037711042}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1944357287}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124214827}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599864506}) 2024-09-18T10:25:02.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124214827}) 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.889+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149047791}) 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936793443}) 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730720901}) 2024-09-18T10:25:02.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936793443}) 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333543716}) 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486749178}) 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=233548825}) 2024-09-18T10:25:02.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486749178}) 2024-09-18T10:25:02.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.893+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146097536}) 2024-09-18T10:25:02.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305493532}) 2024-09-18T10:25:02.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4016691587}) 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305493532}) 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909454933}) 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934904293}) 2024-09-18T10:25:02.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4197236532}) 2024-09-18T10:25:02.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934904293}) 2024-09-18T10:25:02.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783554045}) 2024-09-18T10:25:02.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146075032}) 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086900082}) 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146075032}) 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.897+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548000273}) 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559040227}) 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2298859071}) 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559040227}) 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=121456940}) 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546240890}) 2024-09-18T10:25:02.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153485627}) 2024-09-18T10:25:02.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546240890}) 2024-09-18T10:25:02.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4108275530}) 2024-09-18T10:25:02.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.901+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680806729}) 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928282668}) 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680806729}) 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691525168}) 2024-09-18T10:25:02.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841139079}) 2024-09-18T10:25:02.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655203286}) 2024-09-18T10:25:02.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841139079}) 2024-09-18T10:25:02.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191559849}) 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580290609}) 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384563335}) 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580290609}) 2024-09-18T10:25:02.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.905+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347364402}) 2024-09-18T10:25:02.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045673856}) 2024-09-18T10:25:02.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951812714}) 2024-09-18T10:25:02.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045673856}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8432297}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241741766}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512519627}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241741766}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197272501}) 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698645248}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3949350617}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698645248}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.909+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521093717}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502929629}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1694161895}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502929629}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556042080}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106543136}) 2024-09-18T10:25:02.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526937030}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4106543136}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068345891}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931185276}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=895552667}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931185276}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.913+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723064274}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693937561}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90064663}) 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693937561}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126375756}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=420408394}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502090170}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=420408394}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575286307}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592172039}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151498539}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592172039}) 2024-09-18T10:25:02.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.917+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2050501900}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163864301}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1634156602}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2163864301}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785687353}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188893909}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=592986451}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188893909}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3044211287}) 2024-09-18T10:25:02.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.921+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528512793}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509737905}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528512793}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618657163}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910828876}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859205857}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910828876}) 2024-09-18T10:25:02.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287968492}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828477906}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3923211659}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3828477906}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.925+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860562162}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168835102}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818095818}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168835102}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166597832}) 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764123137}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745119750}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764123137}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128447941}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=317395764,mdsmap=1531151339,mgr=2679965908,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=110012503,mon_config_key=620546692,monmap=3505902565,osd_metadata=500417744,osd_pg_creating=69081760,osd_snap=454132730,osdmap=285589789}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=1342486581,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=110012503,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=874817075}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=26} crc {logm=317395764,mdsmap=1531151339,mgr=2679965908,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=110012503,mon_config_key=620546692,monmap=3505902565,osd_metadata=500417744,osd_pg_creating=69081760,osd_snap=454132730,osdmap=285589789}) 2024-09-18T10:25:02.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.929+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=27} crc {logm=3673737237,mdsmap=1531151339,mgr=1051400072,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=110012503,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=811057539}) 2024-09-18T10:25:02.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=4176436111}) 2024-09-18T10:25:02.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=3343908320}) 2024-09-18T10:25:02.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:02.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=4176436111}) 2024-09-18T10:25:02.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:02.937+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=4113219245}) 2024-09-18T10:25:03.831 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:25:03.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: get_monmap_and_config 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: build_initial_monmap 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: monmap: 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:03.937695+0000 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:03.937695+0000 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:03.941 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:03.941 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:03.941 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:03.941 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:03.941 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 Processor -- start 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 -- start start 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: init 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f810809a830 0x7f8108098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-a con 0x7f810809a830 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f810809ac00 0x7f8108099110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-c con 0x7f810809ac00 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f810809bed0 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108099d50 con 0x7f810809bed0 2024-09-18T10:25:03.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810809dca0 con 0x7f810809a830 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8108099a70 con 0x7f810809ac00 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810fda7700 10 monclient(hunting): authenticate will time out at 1792.863288s 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f810809a830 0x7f8108098bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f810809a830 0x7f8108098bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42470/0 (socket says 172.21.15.142:42470) 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 1 -- 172.21.15.142:0/1293603000 learned_addr learned my addr 172.21.15.142:0/1293603000 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): get_auth_request con 0x7f810809a830 auth_method 0 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:03.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 1 --2- 172.21.15.142:0/1293603000 >> v2:172.21.15.150:3300/0 conn(0x7f810809ac00 0x7f8108099110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): get_auth_request con 0x7f810809ac00 auth_method 0 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.932+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 10 monclient(hunting): handle_auth_done global_id 184134 payload 274 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 -- 172.21.15.142:0/1293603000 >> v2:172.21.15.150:3300/0 conn(0x7f810809ac00 msgr2=0x7f8108099110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:03.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 --2- 172.21.15.142:0/1293603000 >> v2:172.21.15.150:3300/0 conn(0x7f810809ac00 0x7f8108099110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 -- 172.21.15.142:0/1293603000 >> v1:172.21.15.150:6789/0 conn(0x7f810809bed0 legacy=0x7f810809c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 10 monclient: _finish_hunting 0 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 monclient: found mon.noname-a 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 -- 172.21.15.142:0/1293603000 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810809a030 con 0x7f810809a830 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 -- 172.21.15.142:0/1293603000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809a830 msgr2=0x7f8108098bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810db43700 1 --2- 172.21.15.142:0/1293603000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809a830 0x7f8108098bd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f810809d420 tx=0x7f80f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab8096926057b1f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 1 -- 172.21.15.142:0/1293603000 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8009e70 con 0x7f810809a830 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: dump: 2024-09-18T10:25:03.949 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: _finish_auth 0 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 5 monclient: authenticate success, global_id 184134 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 1 -- 172.21.15.142:0/1293603000 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f800ba70 con 0x7f810809a830 2024-09-18T10:25:03.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 4 set_mon_vals no callback set 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 1 -- 172.21.15.142:0/1293603000 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f800be10 con 0x7f810809a830 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810cb41700 10 monclient: dump: 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:03.951 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 10 monclient: shutdown 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 -- 172.21.15.142:0/1293603000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809a830 msgr2=0x7f8108098bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 --2- 172.21.15.142:0/1293603000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809a830 0x7f8108098bd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f810809d420 tx=0x7f80f8009250 comp rx=0 tx=0).stop 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 -- 172.21.15.142:0/1293603000 shutdown_connections 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 --2- 172.21.15.142:0/1293603000 >> v2:172.21.15.150:3300/0 conn(0x7f810809ac00 0x7f8108099110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 --2- 172.21.15.142:0/1293603000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809a830 0x7f8108098bd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.936+0000 7f810fda7700 1 -- 172.21.15.142:0/1293603000 >> 172.21.15.142:0/1293603000 conn(0x7f81080a14e0 msgr2=0x7f81080a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:03.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- 172.21.15.142:0/1293603000 shutdown_connections 2024-09-18T10:25:03.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- 172.21.15.142:0/1293603000 wait complete. 2024-09-18T10:25:03.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient: build_initial_monmap 2024-09-18T10:25:03.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient: monmap: 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 Processor -- start 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- start start 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient: init 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809a830 0x7f8108138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-c con 0x7f810809a830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-a con 0x7f810809ac00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 0x7f810806b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): picked mon.noname-b con 0x7f810809bed0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:03.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810806eb40 con 0x7f810809ac00 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810806da70 con 0x7f810809bed0 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8108146a00 con 0x7f810809a830 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810fda7700 10 monclient(hunting): authenticate will time out at 1792.869876s 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42478/0 (socket says 172.21.15.142:42478) 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 -- 172.21.15.142:0/905144852 learned_addr learned my addr 172.21.15.142:0/905144852 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): get_auth_request con 0x7f810809ac00 auth_method 0 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:03.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809a830 0x7f8108138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): get_auth_request con 0x7f810809a830 auth_method 0 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 0x7f810806b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 10 monclient(hunting): get_auth_request con 0x7f810809bed0 auth_method 0 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:03.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810db43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient(hunting): handle_auth_done global_id 184137 payload 824 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 -- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809a830 msgr2=0x7f8108138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809a830 0x7f8108138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 -- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 msgr2=0x7f810806b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 0x7f810806b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient: _finish_hunting 0 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 monclient: found mon.noname-a 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:03.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810d342700 1 -- 172.21.15.142:0/905144852 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f80094f0 con 0x7f810809ac00 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.940+0000 7f810e344700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 0x7f810806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810d342700 1 -- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 msgr2=0x7f810806b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810d342700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8104009f10 tx=0x7f810400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810400ce30 con 0x7f810809ac00 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: dump: 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:03.963 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810400f040 con 0x7f810809ac00 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 5 monclient: authenticate success, global_id 184137 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _renew_subs 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810806ee20 con 0x7f810809ac00 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:03.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040146d0 con 0x7f810809ac00 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 10 monclient: dump: 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:03.965 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _renew_subs 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81080946d0 con 0x7f810809ac00 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80ff7fe700 4 set_mon_vals no callback set 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810806e960 con 0x7f810809ac00 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.944+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f8104014a80 con 0x7f810809ac00 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.948+0000 7f80fdffb700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 0x7f80ec037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.948+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f8104049990 con 0x7f810809ac00 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.948+0000 7f810db43700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 0x7f80ec037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:03.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.948+0000 7f810db43700 10 monclient: get_auth_request con 0x7f80ec035b10 auth_method 0 2024-09-18T10:25:03.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.948+0000 7f810db43700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 0x7f80ec037fc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f810809d420 tx=0x7f80f8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:03.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.956+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f81040112c0 con 0x7f810809ac00 2024-09-18T10:25:03.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.956+0000 7f80fdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:03.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:03.956+0000 7f80fdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:04.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.180+0000 7f810fda7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:04.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.180+0000 7f810fda7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:04.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.180+0000 7f810fda7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.180+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f81080696a0 con 0x7f810809ac00 2024-09-18T10:25:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f80fdffb700 1 -- 172.21.15.142:0/905144852 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f8104018030 con 0x7f810809ac00 2024-09-18T10:25:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f80fdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:04.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f80fdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:04.190 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:04.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 msgr2=0x7f80ec037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 0x7f80ec037fc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f810809d420 tx=0x7f80f8016000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 10 monclient: shutdown 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 msgr2=0x7f810806b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8104009f10 tx=0x7f810400b040 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 shutdown_connections 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80ec035b10 0x7f80ec037fc0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809bed0 0x7f810806b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f810809ac00 0x7f810806b3c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 --2- 172.21.15.142:0/905144852 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809a830 0x7f8108138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 >> 172.21.15.142:0/905144852 conn(0x7f81080a14e0 msgr2=0x7f81081398f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 shutdown_connections 2024-09-18T10:25:04.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.184+0000 7f810fda7700 1 -- 172.21.15.142:0/905144852 wait complete. 2024-09-18T10:25:04.204 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:25:04.204 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:25:04.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:25:04.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: get_monmap_and_config 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: monmap: 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:04.307962+0000 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:04.307962+0000 2024-09-18T10:25:04.310 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:04.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 Processor -- start 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 -- start start 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: init 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:25:04.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-b con 0x7f7c5809eee0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c58132b00 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 0x7f7c58139a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-a con 0x7f7c58137660 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c580a0660 con 0x7f7c58132b00 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c580a1c40 con 0x7f7c58137660 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580a1600 con 0x7f7c5809eee0 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5ddfc700 10 monclient(hunting): authenticate will time out at 1793.233413s 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 0x7f7c58139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 0x7f7c58139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42480/0 (socket says 172.21.15.142:42480) 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37518/0 (socket says 172.21.15.142:37518) 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 1 -- 172.21.15.142:0/3021696189 learned_addr learned my addr 172.21.15.142:0/3021696189 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): get_auth_request con 0x7f7c58137660 auth_method 0 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): get_auth_request con 0x7f7c5809eee0 auth_method 0 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_done global_id 75638 payload 274 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3021696189 >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 msgr2=0x7f7c58139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 --2- 172.21.15.142:0/3021696189 >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 0x7f7c58139a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3021696189 >> v1:172.21.15.142:6789/0 conn(0x7f7c58132b00 legacy=0x7f7c5809f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 monclient: found mon.noname-b 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:04.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.304+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3021696189 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c580a1f20 con 0x7f7c5809eee0 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3021696189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 msgr2=0x7f7c581325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 --2- 172.21.15.142:0/3021696189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c440098e0 tx=0x7f7c440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c208a4080272d9d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 1 -- 172.21.15.142:0/3021696189 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400b850 con 0x7f7c5809eee0 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: dump: 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 5 monclient: authenticate success, global_id 75638 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 1 -- 172.21.15.142:0/3021696189 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4400b9b0 con 0x7f7c5809eee0 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 1 -- 172.21.15.142:0/3021696189 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400bd50 con 0x7f7c5809eee0 2024-09-18T10:25:04.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 4 set_mon_vals no callback set 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c577fe700 10 monclient: dump: 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.323 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: shutdown 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3021696189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 msgr2=0x7f7c581325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3021696189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7c440098e0 tx=0x7f7c440091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3021696189 shutdown_connections 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3021696189 >> v2:172.21.15.142:3300/0 conn(0x7f7c58137660 0x7f7c58139a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3021696189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c5809eee0 0x7f7c581325c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3021696189 >> 172.21.15.142:0/3021696189 conn(0x7f7c580a5700 msgr2=0x7f7c580a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3021696189 shutdown_connections 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3021696189 wait complete. 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: monmap: 2024-09-18T10:25:04.324 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 Processor -- start 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- start start 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: init 2024-09-18T10:25:04.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 0x7f7c5806b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-a con 0x7f7c5809eee0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c58132b00 0x7f7c5806ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-b con 0x7f7c58132b00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c58137660 0x7f7c5806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): picked mon.noname-c con 0x7f7c58137660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c58131f80 con 0x7f7c5809eee0 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580a11c0 con 0x7f7c58132b00 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5809ded0 con 0x7f7c58137660 2024-09-18T10:25:04.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5ddfc700 10 monclient(hunting): authenticate will time out at 1793.239120s 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 0x7f7c5806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c58137660 0x7f7c5806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 0x7f7c5806b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42484/0 (socket says 172.21.15.142:42484) 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3049440615 learned_addr learned my addr 172.21.15.142:0/3049440615 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): get_auth_request con 0x7f7c5809eee0 auth_method 0 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): get_auth_request con 0x7f7c58137660 auth_method 0 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5d5fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c58132b00 0x7f7c5806ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 10 monclient(hunting): get_auth_request con 0x7f7c58132b00 auth_method 0 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient(hunting): handle_auth_done global_id 184143 payload 824 2024-09-18T10:25:04.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c58137660 msgr2=0x7f7c5806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c58137660 0x7f7c5806bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c58132b00 msgr2=0x7f7c5806ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c58132b00 0x7f7c5806ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 monclient: found mon.noname-a 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44009420 con 0x7f7c5809eee0 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.308+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c58132b00 0x7f7c5806ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5cdfa700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 msgr2=0x7f7c5806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5cdfa700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 0x7f7c5806b550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c580646f0 tx=0x7f7c44016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44006af0 con 0x7f7c5809eee0 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: dump: 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.330 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c44011d40 con 0x7f7c5809eee0 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 5 monclient: authenticate success, global_id 184143 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 monclient: _renew_subs 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5806eb30 con 0x7f7c5809eee0 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4401a6d0 con 0x7f7c5809eee0 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c54ff9700 10 monclient: dump: 2024-09-18T10:25:04.331 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c567fc700 4 set_mon_vals no callback set 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 monclient: _renew_subs 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5806f0f0 con 0x7f7c5809eee0 2024-09-18T10:25:04.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 0x7f7c5806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c580647a0 con 0x7f7c5802ae80 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.312+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 0x7f7c5806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient: get_auth_request con 0x7f7c5802ae80 auth_method 0 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 10 monclient(hunting): handle_auth_done global_id 184146 payload 824 2024-09-18T10:25:04.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c5806f610 con 0x7f7c5802ae80 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7c4401ab00 con 0x7f7c5809eee0 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c3c035ac0 0x7f7c3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7c4404adb0 con 0x7f7c5809eee0 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c5d5fb700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c3c035ac0 0x7f7c3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c5d5fb700 10 monclient: get_auth_request con 0x7f7c3c035ac0 auth_method 0 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 msgr2=0x7f7c5806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 0x7f7c5806c510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c5809f8d0 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c006820 con 0x7f7c5802ae80 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 10 monclient: dump: 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.338 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7c4c006980 con 0x7f7c5802ae80 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:04.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 msgr2=0x7f7c5806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 0x7f7c5806c510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c5809f8d0 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 msgr2=0x7f7c5806c510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.316+0000 7f7c54ff9700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5802ae80 0x7f7c5806c510 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.320+0000 7f7c5d5fb700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c3c035ac0 0x7f7c3c037f70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7c5806dc10 tx=0x7f7c5000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c5ddfc700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 0x7f7c5805ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c5ddfc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c58134b90 con 0x7f7c5805e890 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 0x7f7c5805ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient: get_auth_request con 0x7f7c5805e890 auth_method 0 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 10 monclient(hunting): handle_auth_done global_id 184149 payload 824 2024-09-18T10:25:04.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.328+0000 7f7c57fff700 1 -- 172.21.15.142:0/3049440615 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7c5806d060 con 0x7f7c5805e890 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c57fff700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 msgr2=0x7f7c5805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c57fff700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 0x7f7c5805ec90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c5809f8d0 tx=0x7f7c4c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c015430 con 0x7f7c5805e890 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 10 monclient: dump: 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f7c4c0135f0 con 0x7f7c5805e890 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 msgr2=0x7f7c5805ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 0x7f7c5805ec90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c5809f8d0 tx=0x7f7c4c01b000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 msgr2=0x7f7c5805ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c54ff9700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5805e890 0x7f7c5805ec90 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7c5809f8d0 tx=0x7f7c4c01b000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.344 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.345 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:04.347 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.348 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:04.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:04.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:04.355 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c3c035ac0 msgr2=0x7f7c3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7c3c035ac0 0x7f7c3c037f70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7c5806dc10 tx=0x7f7c5000b040 comp rx=0 tx=0).stop 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 10 monclient: shutdown 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 msgr2=0x7f7c5806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 --2- 172.21.15.142:0/3049440615 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c5809eee0 0x7f7c5806b550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c580646f0 tx=0x7f7c44016000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5d5fb700 1 -- 172.21.15.142:0/3049440615 reap_dead start 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5d5fb700 1 -- 172.21.15.142:0/3049440615 reap_dead start 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 shutdown_connections 2024-09-18T10:25:04.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 >> 172.21.15.142:0/3049440615 conn(0x7f7c580a5700 msgr2=0x7f7c580a42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 shutdown_connections 2024-09-18T10:25:04.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.332+0000 7f7c5ddfc700 1 -- 172.21.15.142:0/3049440615 wait complete. 2024-09-18T10:25:04.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:04.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:25:04.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:25:04.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: get_monmap_and_config 2024-09-18T10:25:04.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: monmap: 2024-09-18T10:25:04.463 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.463 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:04.461306+0000 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:04.461306+0000 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:04.464 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 Processor -- start 2024-09-18T10:25:04.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 -- start start 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: init 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68b009b0b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f68b009b6f0 0x7f68b012edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-a con 0x7f68b009b6f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68b01339d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b009b480 con 0x7f68b009b0b0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b0098cd0 con 0x7f68b01339d0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b006f940 con 0x7f68b009b6f0 2024-09-18T10:25:04.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b841f700 10 monclient(hunting): authenticate will time out at 1793.387665s 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b61bb700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f68b009b0b0 0x7f68b012e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37702/0 (socket says 172.21.15.142:37702) 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b61bb700 1 -- 172.21.15.142:0/3297645930 learned_addr learned my addr 172.21.15.142:0/3297645930 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b59ba700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f68b009b6f0 0x7f68b012edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b59ba700 10 monclient(hunting): get_auth_request con 0x7f68b009b6f0 auth_method 0 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b59ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b59ba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.456+0000 7f68b59ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient(hunting): handle_auth_done global_id 184152 payload 274 2024-09-18T10:25:04.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 -- 172.21.15.142:0/3297645930 >> v1:172.21.15.150:6789/0 conn(0x7f68b01339d0 legacy=0x7f68b0135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 -- 172.21.15.142:0/3297645930 >> v1:172.21.15.142:6789/0 conn(0x7f68b009b0b0 legacy=0x7f68b012e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 monclient: found mon.noname-a 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.469 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 -- 172.21.15.142:0/3297645930 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0099110 con 0x7f68b009b6f0 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 -- 172.21.15.142:0/3297645930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 msgr2=0x7f68b012edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b59ba700 1 --2- 172.21.15.142:0/3297645930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b012edb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098e0 tx=0x7f68a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46287290fa7d312 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 1 -- 172.21.15.142:0/3297645930 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000ba70 con 0x7f68b009b6f0 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: dump: 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.471 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 1 -- 172.21.15.142:0/3297645930 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a000bbd0 con 0x7f68b009b6f0 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b841f700 5 monclient: authenticate success, global_id 184152 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b841f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b841f700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 1 -- 172.21.15.142:0/3297645930 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0011420 con 0x7f68b009b6f0 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b841f700 4 set_mon_vals no callback set 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.460+0000 7f68b51b9700 10 monclient: dump: 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.473 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 10 monclient: shutdown 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- 172.21.15.142:0/3297645930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 msgr2=0x7f68b012edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 --2- 172.21.15.142:0/3297645930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b012edb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098e0 tx=0x7f68a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- 172.21.15.142:0/3297645930 shutdown_connections 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 --2- 172.21.15.142:0/3297645930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b012edb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- 172.21.15.142:0/3297645930 >> 172.21.15.142:0/3297645930 conn(0x7f68b00a14e0 msgr2=0x7f68b00a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- 172.21.15.142:0/3297645930 shutdown_connections 2024-09-18T10:25:04.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- 172.21.15.142:0/3297645930 wait complete. 2024-09-18T10:25:04.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 10 monclient: monmap: 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 Processor -- start 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 1 -- start start 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.464+0000 7f68b841f700 10 monclient: init 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 0x7f68b01332a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-c con 0x7f68b009b0b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b0127580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-a con 0x7f68b009b6f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b01339d0 0x7f68b0127ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): picked mon.noname-b con 0x7f68b01339d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b012a5e0 con 0x7f68b009b6f0 2024-09-18T10:25:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b0129040 con 0x7f68b01339d0 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b006fa90 con 0x7f68b009b0b0 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b841f700 10 monclient(hunting): authenticate will time out at 1793.396386s 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 0x7f68b01332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b0127580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 0x7f68b01332a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37546/0 (socket says 172.21.15.142:37546) 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 -- 172.21.15.142:0/2386136608 learned_addr learned my addr 172.21.15.142:0/2386136608 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): get_auth_request con 0x7f68b009b0b0 auth_method 0 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b0127580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42536/0 (socket says 172.21.15.142:42536) 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): get_auth_request con 0x7f68b009b6f0 auth_method 0 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b01339d0 0x7f68b0127ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): get_auth_request con 0x7f68b01339d0 auth_method 0 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b69bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient(hunting): handle_auth_done global_id 75644 payload 824 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b01339d0 msgr2=0x7f68b0127ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68b01339d0 0x7f68b0127ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 msgr2=0x7f68b0127580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b0127580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 monclient: found mon.noname-c 2024-09-18T10:25:04.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a0009420 con 0x7f68b009b0b0 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b59ba700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68b009b6f0 0x7f68b0127580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 msgr2=0x7f68b01332a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68b61bb700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 0x7f68b01332a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00c5e0 tx=0x7f68ac004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac005d40 con 0x7f68b009b0b0 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68a67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.468+0000 7f68a67fc700 10 monclient: dump: 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.486 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac0070d0 con 0x7f68b009b0b0 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 5 monclient: authenticate success, global_id 75644 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 monclient: _renew_subs 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:04.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b012e240 con 0x7f68b009b0b0 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac007490 con 0x7f68b009b0b0 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b49b8700 4 set_mon_vals no callback set 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 10 monclient: dump: 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.488 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 monclient: _renew_subs 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b0128840 con 0x7f68b009b0b0 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 0x7f68b002ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b0064950 con 0x7f68b0128000 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 0x7f68b002ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient: get_auth_request con 0x7f68b0128000 auth_method 0 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f68ac01b3b0 con 0x7f68b009b0b0 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 10 monclient(hunting): handle_auth_done global_id 75647 payload 824 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b59ba700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68b012e760 con 0x7f68b0128000 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6894035ac0 0x7f6894037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f68ac048450 con 0x7f68b009b0b0 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b69bc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6894035ac0 0x7f6894037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.472+0000 7f68b69bc700 10 monclient: get_auth_request con 0x7f6894035ac0 auth_method 0 2024-09-18T10:25:04.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68b59ba700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 msgr2=0x7f68b002ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68b59ba700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 0x7f68b002ae80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f68b0098600 tx=0x7f68a0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0021070 con 0x7f68b0128000 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 10 monclient: dump: 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.492 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f68a000bea0 con 0x7f68b0128000 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 msgr2=0x7f68b002ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 0x7f68b002ae80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f68b0098600 tx=0x7f68a0016000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 msgr2=0x7f68b002ae80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68a67fc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b0128000 0x7f68b002ae80 secure :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f68b0098600 tx=0x7f68a0016000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.476+0000 7f68b69bc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6894035ac0 0x7f6894037f70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f68a8007fd0 tx=0x7f68a8002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b841f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:04.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b841f700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 0x7f68b005ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b841f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b01283d0 con 0x7f68b005e890 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 0x7f68b005ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient: get_auth_request con 0x7f68b005e890 auth_method 0 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 10 monclient(hunting): handle_auth_done global_id 75650 payload 824 2024-09-18T10:25:04.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.488+0000 7f68b59ba700 1 -- 172.21.15.142:0/2386136608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f68b012a7e0 con 0x7f68b005e890 2024-09-18T10:25:04.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b59ba700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 msgr2=0x7f68b005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b59ba700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 0x7f68b005ec90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f68a00030c0 tx=0x7f68a001e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0025e80 con 0x7f68b005e890 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 10 monclient: dump: 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.499 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f68a001b650 con 0x7f68b005e890 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 msgr2=0x7f68b005ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 0x7f68b005ec90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f68a00030c0 tx=0x7f68a001e070 comp rx=0 tx=0).stop 2024-09-18T10:25:04.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 msgr2=0x7f68b005ec90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68a67fc700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b005e890 0x7f68b005ec90 secure :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f68a00030c0 tx=0x7f68a001e070 comp rx=0 tx=0).stop 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:04.501 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.502 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.503 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:04.504 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.505 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.506 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:04.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:04.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.511 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.512 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:04.513 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6894035ac0 msgr2=0x7f6894037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6894035ac0 0x7f6894037f70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f68a8007fd0 tx=0x7f68a8002620 comp rx=0 tx=0).stop 2024-09-18T10:25:04.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 10 monclient: shutdown 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 msgr2=0x7f68b01332a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 --2- 172.21.15.142:0/2386136608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68b009b0b0 0x7f68b01332a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00c5e0 tx=0x7f68ac004a80 comp rx=0 tx=0).stop 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b69bc700 1 -- 172.21.15.142:0/2386136608 reap_dead start 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b69bc700 1 -- 172.21.15.142:0/2386136608 reap_dead start 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 shutdown_connections 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 >> 172.21.15.142:0/2386136608 conn(0x7f68b00a14e0 msgr2=0x7f68b0097d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 shutdown_connections 2024-09-18T10:25:04.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.492+0000 7f68b841f700 1 -- 172.21.15.142:0/2386136608 wait complete. 2024-09-18T10:25:04.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:04.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:25:04.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.616+0000 7f55b4b0b700 10 monclient: get_monmap_and_config 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.616+0000 7f55b4b0b700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.616+0000 7f55b4b0b700 10 monclient: monmap: 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.625 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:04.623198+0000 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:04.623198+0000 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:04.626 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 Processor -- start 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 -- start start 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient: init 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55b009abd0 0x7f55b00960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-c con 0x7f55b009abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55b009b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-b con 0x7f55b009c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b009dc90 con 0x7f55b009b740 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b009b500 con 0x7f55b009abd0 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b009afa0 con 0x7f55b009c5e0 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55b4b0b700 10 monclient(hunting): authenticate will time out at 1793.548739s 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37572/0 (socket says 172.21.15.142:37572) 2024-09-18T10:25:04.630 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 -- 172.21.15.142:0/225009200 learned_addr learned my addr 172.21.15.142:0/225009200 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): get_auth_request con 0x7f55b009c5e0 auth_method 0 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 1 --2- 172.21.15.142:0/225009200 >> v2:172.21.15.150:3300/0 conn(0x7f55b009abd0 0x7f55b00960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): get_auth_request con 0x7f55b009abd0 auth_method 0 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient(hunting): handle_auth_done global_id 75653 payload 274 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 -- 172.21.15.142:0/225009200 >> v2:172.21.15.150:3300/0 conn(0x7f55b009abd0 msgr2=0x7f55b00960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 --2- 172.21.15.142:0/225009200 >> v2:172.21.15.150:3300/0 conn(0x7f55b009abd0 0x7f55b00960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 -- 172.21.15.142:0/225009200 >> v1:172.21.15.150:6789/0 conn(0x7f55b009b740 legacy=0x7f55b009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 monclient: found mon.noname-b 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:04.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.620+0000 7f55affff700 1 -- 172.21.15.142:0/225009200 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b006f810 con 0x7f55b009c5e0 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55affff700 1 -- 172.21.15.142:0/225009200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 msgr2=0x7f55b0137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55affff700 1 --2- 172.21.15.142:0/225009200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009f20 tx=0x7f55a8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=354598730321a42f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 1 -- 172.21.15.142:0/225009200 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a800e070 con 0x7f55b009c5e0 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: dump: 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.639 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 1 -- 172.21.15.142:0/225009200 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a800bbe0 con 0x7f55b009c5e0 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 5 monclient: authenticate success, global_id 75653 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 1 -- 172.21.15.142:0/225009200 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a80092e0 con 0x7f55b009c5e0 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 4 set_mon_vals no callback set 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55ae7fc700 10 monclient: dump: 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.641 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: shutdown 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- 172.21.15.142:0/225009200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 msgr2=0x7f55b0137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/225009200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009f20 tx=0x7f55a8009840 comp rx=0 tx=0).stop 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- 172.21.15.142:0/225009200 shutdown_connections 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/225009200 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009c5e0 0x7f55b0137690 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/225009200 >> v2:172.21.15.150:3300/0 conn(0x7f55b009abd0 0x7f55b00960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- 172.21.15.142:0/225009200 >> 172.21.15.142:0/225009200 conn(0x7f55b0070ef0 msgr2=0x7f55b00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- 172.21.15.142:0/225009200 shutdown_connections 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- 172.21.15.142:0/225009200 wait complete. 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: monmap: 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 Processor -- start 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- start start 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: init 2024-09-18T10:25:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009abd0 0x7f55b0141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-c con 0x7f55b009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 0x7f55b0141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-a con 0x7f55b009b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b009c5e0 0x7f55b01421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): picked mon.noname-b con 0x7f55b009c5e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b009d240 con 0x7f55b009b740 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b006eb10 con 0x7f55b009c5e0 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b008f4d0 con 0x7f55b009abd0 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55b4b0b700 10 monclient(hunting): authenticate will time out at 1793.555777s 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 0x7f55b0141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 0x7f55b0141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42546/0 (socket says 172.21.15.142:42546) 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 1 -- 172.21.15.142:0/515347728 learned_addr learned my addr 172.21.15.142:0/515347728 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 10 monclient(hunting): get_auth_request con 0x7f55b009b740 auth_method 0 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55af7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009abd0 0x7f55b0141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.624+0000 7f55aeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55af7fe700 10 monclient(hunting): get_auth_request con 0x7f55b009abd0 auth_method 0 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55affff700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b009c5e0 0x7f55b01421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55affff700 10 monclient(hunting): get_auth_request con 0x7f55b009c5e0 auth_method 0 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55affff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55affff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient(hunting): handle_auth_done global_id 184158 payload 824 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009abd0 msgr2=0x7f55b0141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009abd0 0x7f55b0141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b009c5e0 msgr2=0x7f55b01421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b009c5e0 0x7f55b01421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 monclient: found mon.noname-a 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8009ae0 con 0x7f55b009b740 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55af7fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55b009abd0 0x7f55b0141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 msgr2=0x7f55b0141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55aeffd700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 0x7f55b0141ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55a400c5d0 tx=0x7f55a4007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a40117a0 con 0x7f55b009b740 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: dump: 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.648 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 5 monclient: authenticate success, global_id 184158 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a4011900 con 0x7f55b009b740 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 10 monclient: _renew_subs 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b008f9e0 con 0x7f55b009b740 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4011cc0 con 0x7f55b009b740 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55977fe700 10 monclient: dump: 2024-09-18T10:25:04.649 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 10 monclient: _renew_subs 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b0092250 con 0x7f55b009b740 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55adffb700 4 set_mon_vals no callback set 2024-09-18T10:25:04.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.628+0000 7f55b4b0b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f55a4012630 con 0x7f55b009b740 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 0x7f55b008a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b00648a0 con 0x7f55b0089ff0 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55977fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f559c035ac0 0x7f559c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f55a4048e00 con 0x7f55b009b740 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 0x7f55b008a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient: get_auth_request con 0x7f55b0089ff0 auth_method 0 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55affff700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f559c035ac0 0x7f559c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55affff700 10 monclient: get_auth_request con 0x7f559c035ac0 auth_method 0 2024-09-18T10:25:04.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 10 monclient(hunting): handle_auth_done global_id 174229 payload 824 2024-09-18T10:25:04.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55b0072a80 con 0x7f55b0089ff0 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55affff700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f559c035ac0 0x7f559c037f70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009ab0 tx=0x7f55a8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 msgr2=0x7f55b008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.632+0000 7f55af7fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 0x7f55b008a3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55a0011840 tx=0x7f55a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a000fad0 con 0x7f55b0089ff0 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:04.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 10 monclient: dump: 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f55a000fc30 con 0x7f55b0089ff0 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:04.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 msgr2=0x7f55b008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 0x7f55b008a3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55a0011840 tx=0x7f55a0008040 comp rx=0 tx=0).stop 2024-09-18T10:25:04.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 msgr2=0x7f55b008a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.636+0000 7f55977fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b0089ff0 0x7f55b008a3c0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55a0011840 tx=0x7f55a0008040 comp rx=0 tx=0).stop 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55b4b0b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 0x7f55b008d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55b4b0b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b00678a0 con 0x7f55b008a900 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 0x7f55b008d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient: get_auth_request con 0x7f55b008a900 auth_method 0 2024-09-18T10:25:04.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 10 monclient(hunting): handle_auth_done global_id 174232 payload 824 2024-09-18T10:25:04.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.648+0000 7f55af7fe700 1 -- 172.21.15.142:0/515347728 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f55b0090100 con 0x7f55b008a900 2024-09-18T10:25:04.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55af7fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 msgr2=0x7f55b008d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55af7fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 0x7f55b008d980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f55a001c000 tx=0x7f55a001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0006930 con 0x7f55b008a900 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 10 monclient: dump: 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.659 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f55a0015670 con 0x7f55b008a900 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 msgr2=0x7f55b008d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 0x7f55b008d980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f55a001c000 tx=0x7f55a001d000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 msgr2=0x7f55b008d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55977fe700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b008a900 0x7f55b008d980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 148, 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:04.661 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.662 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.663 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:04.664 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:04.665 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.666 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.667 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:04.668 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.669 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:04.670 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:04.671 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.672 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:04.673 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f559c035ac0 msgr2=0x7f559c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f559c035ac0 0x7f559c037f70 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009ab0 tx=0x7f55a8017000 comp rx=0 tx=0).stop 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 10 monclient: shutdown 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 msgr2=0x7f55b0141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 --2- 172.21.15.142:0/515347728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55b009b740 0x7f55b0141ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f55a400c5d0 tx=0x7f55a4007f60 comp rx=0 tx=0).stop 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55affff700 1 -- 172.21.15.142:0/515347728 reap_dead start 2024-09-18T10:25:04.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55affff700 1 -- 172.21.15.142:0/515347728 reap_dead start 2024-09-18T10:25:04.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 shutdown_connections 2024-09-18T10:25:04.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 >> 172.21.15.142:0/515347728 conn(0x7f55b0070ef0 msgr2=0x7f55b009de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 shutdown_connections 2024-09-18T10:25:04.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.652+0000 7f55b4b0b700 1 -- 172.21.15.142:0/515347728 wait complete. 2024-09-18T10:25:04.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:04.678 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-18T10:25:04.678 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:25:04.678 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-18T10:25:04.679 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:25:04.679 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:25:04.679 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:25:04.698 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:25:04.698 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:25:04.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: monmap: 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:04.801741+0000 2024-09-18T10:25:04.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:04.801741+0000 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:04.805 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:04.806 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:04.806 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 Processor -- start 2024-09-18T10:25:04.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 -- start start 2024-09-18T10:25:04.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: init 2024-09-18T10:25:04.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 0x7fa84012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa8400a1cf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa84012f810 0x7fa84009f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa84012f810 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 0x7fa8401320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa8400a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84009d990 con 0x7fa84012f810 2024-09-18T10:25:04.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84009dc70 con 0x7fa8400a1cf0 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8400a20c0 con 0x7fa8400a02d0 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.796+0000 7fa83f59e700 10 monclient(hunting): authenticate will time out at 1793.727984s 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83e59c700 1 -- >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 msgr2=0x7fa84012f030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83e59c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 0x7fa84012f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 0x7fa8401320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa84012f810 0x7fa84009f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 0x7fa8401320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37590/0 (socket says 172.21.15.142:37590) 2024-09-18T10:25:04.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 1 -- 172.21.15.142:0/3580218730 learned_addr learned my addr 172.21.15.142:0/3580218730 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): get_auth_request con 0x7fa84012f810 auth_method 0 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): get_auth_request con 0x7fa8400a02d0 auth_method 0 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient(hunting): handle_auth_done global_id 184161 payload 274 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 -- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 msgr2=0x7fa8401320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 0x7fa8401320e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 -- 172.21.15.142:0/3580218730 >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 msgr2=0x7fa84012f030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/3580218730 >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 0x7fa84012f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 monclient: found mon.noname-a 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 -- 172.21.15.142:0/3580218730 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84009f9c0 con 0x7fa84012f810 2024-09-18T10:25:04.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 -- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84012f810 msgr2=0x7fa84009f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84012f810 0x7fa84009f480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa840071e60 tx=0x7fa8300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db23c246a4cc239 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 1 -- 172.21.15.142:0/3580218730 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830011840 con 0x7fa84012f810 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: dump: 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.815 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83f59e700 5 monclient: authenticate success, global_id 184161 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 1 -- 172.21.15.142:0/3580218730 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8300119a0 con 0x7fa84012f810 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83f59e700 4 set_mon_vals no callback set 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 1 -- 172.21.15.142:0/3580218730 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830011d40 con 0x7fa84012f810 2024-09-18T10:25:04.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.800+0000 7fa83d59a700 10 monclient: dump: 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.817 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: shutdown 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84012f810 msgr2=0x7fa84009f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84012f810 0x7fa84009f480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa840071e60 tx=0x7fa8300091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- 172.21.15.142:0/3580218730 shutdown_connections 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8400a02d0 0x7fa8401320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- 172.21.15.142:0/3580218730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84012f810 0x7fa84009f480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- 172.21.15.142:0/3580218730 >> v2:172.21.15.150:3300/0 conn(0x7fa8400a1cf0 0x7fa84012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- 172.21.15.142:0/3580218730 >> 172.21.15.142:0/3580218730 conn(0x7fa840071820 msgr2=0x7fa8400a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- 172.21.15.142:0/3580218730 shutdown_connections 2024-09-18T10:25:04.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- 172.21.15.142:0/3580218730 wait complete. 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: build_initial_monmap 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: monmap: 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:04.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 Processor -- start 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- start start 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: init 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fa8400a02d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:04.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 0x7fa84009baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fa8400a1cf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa84012f810 0x7fa84009c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fa84012f810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84006dc40 con 0x7fa8400a02d0 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84009cdc0 con 0x7fa8400a1cf0 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa840132050 con 0x7fa84012f810 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83f59e700 10 monclient(hunting): authenticate will time out at 1793.734303s 2024-09-18T10:25:04.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83dd9b700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 msgr2=0x7fa84009baf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42568/0 (socket says 172.21.15.142:42568) 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 -- 172.21.15.142:0/1023728527 learned_addr learned my addr 172.21.15.142:0/1023728527 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): get_auth_request con 0x7fa8400a02d0 auth_method 0 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 0x7fa84009baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa84012f810 0x7fa84009c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): get_auth_request con 0x7fa84012f810 auth_method 0 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient(hunting): handle_auth_done global_id 184164 payload 824 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 -- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa84012f810 msgr2=0x7fa84009c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa84012f810 0x7fa84009c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 -- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 msgr2=0x7fa84009baf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 0x7fa84009baf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:25:04.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 monclient: found mon.noname-a 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.804+0000 7fa83e59c700 1 -- 172.21.15.142:0/1023728527 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830009420 con 0x7fa8400a02d0 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83e59c700 1 -- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 msgr2=0x7fa84009b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83e59c700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa82800b7e0 tx=0x7fa828009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828003e20 con 0x7fa8400a02d0 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: dump: 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 5 monclient: authenticate success, global_id 184164 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _renew_subs 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828006150 con 0x7fa8400a02d0 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84006bb30 con 0x7fa8400a02d0 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828006510 con 0x7fa8400a02d0 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83cd99700 4 set_mon_vals no callback set 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 10 monclient: dump: 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.829 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:04.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _renew_subs 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84006c6f0 con 0x7fa8400a02d0 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa8280103c0 con 0x7fa8400a02d0 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84006b0b0 con 0x7fa8400a02d0 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.808+0000 7fa8367fc700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 0x7fa820037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:04.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.812+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa828048720 con 0x7fa8400a02d0 2024-09-18T10:25:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.812+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 0x7fa820037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.812+0000 7fa83dd9b700 10 monclient: get_auth_request con 0x7fa820035a70 auth_method 0 2024-09-18T10:25:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.812+0000 7fa83dd9b700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 0x7fa820037f20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa8300091e0 tx=0x7fa830017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:04.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.816+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa84006b0b0 con 0x7fa8400a02d0 2024-09-18T10:25:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.816+0000 7fa8367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:04.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:04.816+0000 7fa8367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:05.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa83f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:05.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa83f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:05.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa83f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:05.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa84006b0b0 con 0x7fa8400a02d0 2024-09-18T10:25:05.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa8367fc700 1 -- 172.21.15.142:0/1023728527 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa84006b0b0 con 0x7fa8400a02d0 2024-09-18T10:25:05.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa8367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:05.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.044+0000 7fa8367fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:05.052 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:05.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 msgr2=0x7fa820037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:05.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 0x7fa820037f20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa8300091e0 tx=0x7fa830017000 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 10 monclient: shutdown 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 msgr2=0x7fa84009b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa82800b7e0 tx=0x7fa828009220 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 shutdown_connections 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa820035a70 0x7fa820037f20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa84012f810 0x7fa84009c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8400a1cf0 0x7fa84009baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 --2- 172.21.15.142:0/1023728527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8400a02d0 0x7fa84009b5b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 >> 172.21.15.142:0/1023728527 conn(0x7fa840071820 msgr2=0x7fa84012ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 shutdown_connections 2024-09-18T10:25:05.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:05.048+0000 7fa83f59e700 1 -- 172.21.15.142:0/1023728527 wait complete. 2024-09-18T10:25:08.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: get_monmap_and_config 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: build_initial_monmap 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: monmap: 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:08.176331+0000 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:08.176331+0000 2024-09-18T10:25:08.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 Processor -- start 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 -- start start 2024-09-18T10:25:08.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: init 2024-09-18T10:25:08.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:08.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f433009aae0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4330132570 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 0x7f4330138450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-b con 0x7f4330136070 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433009aeb0 con 0x7f433009aae0 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433006f6d0 con 0x7f4330132570 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43300955a0 con 0x7f4330136070 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f4334d6d700 10 monclient(hunting): authenticate will time out at 1797.101994s 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432effd700 1 -- >> v1:172.21.15.150:6789/0 conn(0x7f4330132570 legacy=0x7f4330134950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:08.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f433009aae0 0x7f4330096260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37718/0 (socket says 172.21.15.142:37718) 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432f7fe700 1 -- 172.21.15.142:0/2248512701 learned_addr learned my addr 172.21.15.142:0/2248512701 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 1 --2- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 0x7f4330138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): get_auth_request con 0x7f4330136070 auth_method 0 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:08.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient(hunting): handle_auth_done global_id 75665 payload 274 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 1 -- 172.21.15.142:0/2248512701 >> v1:172.21.15.150:6789/0 conn(0x7f4330132570 legacy=0x7f4330134950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 1 -- 172.21.15.142:0/2248512701 >> v1:172.21.15.142:6789/0 conn(0x7f433009aae0 legacy=0x7f4330096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 1 monclient: found mon.noname-b 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:08.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.172+0000 7f432ffff700 1 -- 172.21.15.142:0/2248512701 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330095b60 con 0x7f4330136070 2024-09-18T10:25:08.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432ffff700 1 -- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 msgr2=0x7f4330138450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:08.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432ffff700 1 --2- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 0x7f4330138450 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4328009910 tx=0x7f43280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24124b9c49fb5764 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:08.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 1 -- 172.21.15.142:0/2248512701 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432800ba70 con 0x7f4330136070 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: dump: 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:08.188 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 5 monclient: authenticate success, global_id 75665 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 1 -- 172.21.15.142:0/2248512701 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432800bbd0 con 0x7f4330136070 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 1 -- 172.21.15.142:0/2248512701 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328011420 con 0x7f4330136070 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 4 set_mon_vals no callback set 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f432e7fc700 10 monclient: dump: 2024-09-18T10:25:08.189 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 10 monclient: shutdown 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 -- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 msgr2=0x7f4330138450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:08.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 --2- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 0x7f4330138450 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4328009910 tx=0x7f43280091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 -- 172.21.15.142:0/2248512701 shutdown_connections 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 --2- 172.21.15.142:0/2248512701 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4330136070 0x7f4330138450 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 -- 172.21.15.142:0/2248512701 >> 172.21.15.142:0/2248512701 conn(0x7f43300a1fe0 msgr2=0x7f4330070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 -- 172.21.15.142:0/2248512701 shutdown_connections 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.176+0000 7f4334d6d700 1 -- 172.21.15.142:0/2248512701 wait complete. 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient: build_initial_monmap 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient: monmap: 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 Processor -- start 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 -- start start 2024-09-18T10:25:08.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient: init 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f433009aae0 0x7f433006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-c con 0x7f433009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-a con 0x7f4330132570 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 0x7f433006daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:08.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): picked mon.noname-b con 0x7f4330136070 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433006f8d0 con 0x7f4330132570 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4330098a70 con 0x7f4330136070 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4330127740 con 0x7f433009aae0 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f4334d6d700 10 monclient(hunting): authenticate will time out at 1797.110956s 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f433009aae0 0x7f433006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:42572/0 (socket says 172.21.15.142:42572) 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 -- 172.21.15.142:0/3305414447 learned_addr learned my addr 172.21.15.142:0/3305414447 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:08.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): get_auth_request con 0x7f4330132570 auth_method 0 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432ffff700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 msgr2=0x7f433006daa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): get_auth_request con 0x7f433009aae0 auth_method 0 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432ffff700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 0x7f433006daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:08.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient(hunting): handle_auth_done global_id 184167 payload 824 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f433009aae0 msgr2=0x7f433006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f433009aae0 0x7f433006d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 msgr2=0x7f433006daa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 0x7f433006daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 monclient: found mon.noname-a 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.180+0000 7f432effd700 1 -- 172.21.15.142:0/3305414447 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328009420 con 0x7f4330132570 2024-09-18T10:25:08.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f432effd700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 msgr2=0x7f433006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f432effd700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f432000bb80 tx=0x7f43200097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000cad0 con 0x7f4330132570 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: dump: 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:08.201 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 5 monclient: authenticate success, global_id 184167 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432000cc30 con 0x7f4330132570 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _renew_subs 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43301271b0 con 0x7f4330132570 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000ce20 con 0x7f4330132570 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:08.202 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f43177fe700 10 monclient: dump: 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _renew_subs 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:08.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f432dffb700 4 set_mon_vals no callback set 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4330126500 con 0x7f4330132570 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.184+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4330072a80 con 0x7f4330132570 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.188+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4320012020 con 0x7f4330132570 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.188+0000 7f43177fe700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 0x7f431c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.188+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4320048d20 con 0x7f4330132570 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.188+0000 7f432f7fe700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 0x7f431c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:08.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.188+0000 7f432f7fe700 10 monclient: get_auth_request con 0x7f431c035a70 auth_method 0 2024-09-18T10:25:08.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.192+0000 7f432f7fe700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 0x7f431c037f20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4324007880 tx=0x7f4324007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:08.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.192+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f43200073b0 con 0x7f4330132570 2024-09-18T10:25:08.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.192+0000 7f43177fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:08.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.192+0000 7f43177fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f43300678a0 con 0x7f4330132570 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f43177fe700 1 -- 172.21.15.142:0/3305414447 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4320014340 con 0x7f4330132570 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f43177fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f43177fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:08.424 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 msgr2=0x7f431c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 0x7f431c037f20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4324007880 tx=0x7f4324007f90 comp rx=0 tx=0).stop 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 10 monclient: shutdown 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 msgr2=0x7f433006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f432000bb80 tx=0x7f43200097f0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 shutdown_connections 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f431c035a70 0x7f431c037f20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4330136070 0x7f433006daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4330132570 0x7f433006d560 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 --2- 172.21.15.142:0/3305414447 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f433009aae0 0x7f433006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 >> 172.21.15.142:0/3305414447 conn(0x7f43300a1fe0 msgr2=0x7f4330096c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 shutdown_connections 2024-09-18T10:25:08.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:08.416+0000 7f4334d6d700 1 -- 172.21.15.142:0/3305414447 wait complete. 2024-09-18T10:25:11.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:11.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.540+0000 7f9e7e3a2700 10 monclient: get_monmap_and_config 2024-09-18T10:25:11.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.540+0000 7f9e7e3a2700 10 monclient: build_initial_monmap 2024-09-18T10:25:11.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.540+0000 7f9e7e3a2700 10 monclient: monmap: 2024-09-18T10:25:11.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:11.547201+0000 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:11.547201+0000 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:11.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 Processor -- start 2024-09-18T10:25:11.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 -- start start 2024-09-18T10:25:11.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient: init 2024-09-18T10:25:11.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:11.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e7806f1b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9e7809d9a0 0x7f9e78133790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-a con 0x7f9e7809d9a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e78133cd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e78098b30 con 0x7f9e7806f1b0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e77fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9e7806f1b0 0x7f9e7806f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37734/0 (socket says 172.21.15.142:37734) 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e77fff700 1 -- 172.21.15.142:0/4133986805 learned_addr learned my addr 172.21.15.142:0/4133986805 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7809dda0 con 0x7f9e78133cd0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7806f060 con 0x7f9e7809d9a0 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:11.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 1 --2- 172.21.15.142:0/4133986805 >> v2:172.21.15.142:3300/0 conn(0x7f9e7809d9a0 0x7f9e78133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7e3a2700 10 monclient(hunting): authenticate will time out at 1800.474856s 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e7c93f700 1 -- 172.21.15.142:0/4133986805 >> v1:172.21.15.150:6789/0 conn(0x7f9e78133cd0 legacy=0x7f9e781360b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): get_auth_request con 0x7f9e7809d9a0 auth_method 0 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient(hunting): handle_auth_done global_id 184170 payload 274 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 1 -- 172.21.15.142:0/4133986805 >> v1:172.21.15.150:6789/0 conn(0x7f9e78133cd0 legacy=0x7f9e781360b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 1 -- 172.21.15.142:0/4133986805 >> v1:172.21.15.142:6789/0 conn(0x7f9e7806f1b0 legacy=0x7f9e7806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:11.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:11.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 1 monclient: found mon.noname-a 2024-09-18T10:25:11.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:11.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:11.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.544+0000 7f9e777fe700 1 -- 172.21.15.142:0/4133986805 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78098150 con 0x7f9e7809d9a0 2024-09-18T10:25:11.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e777fe700 1 -- 172.21.15.142:0/4133986805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7809d9a0 msgr2=0x7f9e78133790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:11.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e777fe700 1 --2- 172.21.15.142:0/4133986805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7809d9a0 0x7f9e78133790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00aa90 tx=0x7f9e6c00a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfc93e960b713b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:11.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 1 -- 172.21.15.142:0/4133986805 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c00cd30 con 0x7f9e7809d9a0 2024-09-18T10:25:11.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:11.561 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: dump: 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:11.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 5 monclient: authenticate success, global_id 184170 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 1 -- 172.21.15.142:0/4133986805 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c00ce90 con 0x7f9e7809d9a0 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 4 set_mon_vals no callback set 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 1 -- 172.21.15.142:0/4133986805 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c018640 con 0x7f9e7809d9a0 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e76ffd700 10 monclient: dump: 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:11.563 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 10 monclient: shutdown 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7809d9a0 msgr2=0x7f9e78133790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:11.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4133986805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7809d9a0 0x7f9e78133790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00aa90 tx=0x7f9e6c00a330 comp rx=0 tx=0).stop 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 shutdown_connections 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4133986805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7809d9a0 0x7f9e78133790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 >> 172.21.15.142:0/4133986805 conn(0x7f9e780a1fa0 msgr2=0x7f9e780a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 shutdown_connections 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4133986805 wait complete. 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 10 monclient: build_initial_monmap 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 10 monclient: monmap: 2024-09-18T10:25:11.565 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:11.566 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 Processor -- start 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 1 -- start start 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.548+0000 7f9e7e3a2700 10 monclient: init 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:11.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-c con 0x7f9e7806f1b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 0x7f9e7811d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-b con 0x7f9e7809d9a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 0x7f9e7811d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:11.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): picked mon.noname-a con 0x7f9e78133cd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:11.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:11.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e781209d0 con 0x7f9e78133cd0 2024-09-18T10:25:11.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:11.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e78120f00 con 0x7f9e7809d9a0 2024-09-18T10:25:11.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e78097a10 con 0x7f9e7806f1b0 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7e3a2700 10 monclient(hunting): authenticate will time out at 1800.481161s 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e777fe700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 msgr2=0x7f9e7811d340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e777fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 0x7f9e7811d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:11.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 0x7f9e7811d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37634/0 (socket says 172.21.15.142:37634) 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 -- 172.21.15.142:0/4294770146 learned_addr learned my addr 172.21.15.142:0/4294770146 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): get_auth_request con 0x7f9e7806f1b0 auth_method 0 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:11.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): get_auth_request con 0x7f9e78133cd0 auth_method 0 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:11.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:11.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:11.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:11.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient(hunting): handle_auth_done global_id 75671 payload 824 2024-09-18T10:25:11.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 -- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 msgr2=0x7f9e7811d340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:11.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 0x7f9e7811d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 -- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 msgr2=0x7f9e7811d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 0x7f9e7811d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 monclient: found mon.noname-c 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:11.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e77fff700 1 -- 172.21.15.142:0/4294770146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c00a5d0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.552+0000 7f9e7c93f700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 0x7f9e7811d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e77fff700 1 -- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 msgr2=0x7f9e781327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e77fff700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b640 tx=0x7f9e68009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68011840 con 0x7f9e7806f1b0 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: dump: 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.581 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: _finish_auth 0 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e680119a0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 5 monclient: authenticate success, global_id 75671 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _renew_subs 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7811f2b0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:11.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e68011d60 con 0x7f9e7806f1b0 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 10 monclient: dump: 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.583 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _renew_subs 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e78140f10 con 0x7f9e7806f1b0 2024-09-18T10:25:11.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e767fc700 4 set_mon_vals no callback set 2024-09-18T10:25:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9e68010590 con 0x7f9e7806f1b0 2024-09-18T10:25:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:11.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e78120cb0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 0x7f9e64037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9e68048fa0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e777fe700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 0x7f9e64037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.556+0000 7f9e777fe700 10 monclient: get_auth_request con 0x7f9e64035a70 auth_method 0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.560+0000 7f9e777fe700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 0x7f9e64037f20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00aa90 tx=0x7f9e6c019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.564+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9e68079670 con 0x7f9e7806f1b0 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.564+0000 7f9e5ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:11.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.564+0000 7f9e5ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:11.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e7e3a2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e7e3a2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e7e3a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9e780696a0 con 0x7f9e7806f1b0 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e5ffff700 1 -- 172.21.15.142:0/4294770146 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f9e68015030 con 0x7f9e7806f1b0 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e5ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:11.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.788+0000 7f9e5ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:11.798 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 msgr2=0x7f9e64037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 0x7f9e64037f20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00aa90 tx=0x7f9e6c019000 comp rx=0 tx=0).stop 2024-09-18T10:25:11.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 10 monclient: shutdown 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 msgr2=0x7f9e781327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b640 tx=0x7f9e68009120 comp rx=0 tx=0).stop 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 shutdown_connections 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9e64035a70 0x7f9e64037f20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e78133cd0 0x7f9e7811d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e7809d9a0 0x7f9e7811d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 --2- 172.21.15.142:0/4294770146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e7806f1b0 0x7f9e781327a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 >> 172.21.15.142:0/4294770146 conn(0x7f9e780a1fa0 msgr2=0x7f9e7806da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 shutdown_connections 2024-09-18T10:25:11.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:11.792+0000 7f9e7e3a2700 1 -- 172.21.15.142:0/4294770146 wait complete. 2024-09-18T10:25:14.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: get_monmap_and_config 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: build_initial_monmap 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: monmap: 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:14.917611+0000 2024-09-18T10:25:14.920 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:14.917611+0000 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 Processor -- start 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 -- start start 2024-09-18T10:25:14.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: init 2024-09-18T10:25:14.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-c con 0x7f48d8097a30 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48d809c6f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48d809d200 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d809a7d0 con 0x7f48d809c6f0 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d809a4f0 con 0x7f48d809d200 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d809db60 con 0x7f48d8097a30 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dfffa700 10 monclient(hunting): authenticate will time out at 1803.843133s 2024-09-18T10:25:14.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dd595700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f48d809c6f0 0x7f48d809caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49090/0 (socket says 172.21.15.142:49090) 2024-09-18T10:25:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 learned_addr learned my addr 172.21.15.142:0/380324778 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48de597700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.150:6789/0 conn(0x7f48d809d200 legacy=0x7f48d8136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 msgr2=0x7f48d8097e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.912+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:14.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:14.916+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.142:6789/0 conn(0x7f48d809c6f0 legacy=0x7f48d809caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:15.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.116+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 msgr2=0x7f48d8097e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:15.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.116+0000 7f48de597700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.150:6789/0 conn(0x7f48d809d200 legacy=0x7f48d8136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:15.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.116+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:25:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.520+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 msgr2=0x7f48d8097e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.520+0000 7f48de597700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.150:6789/0 conn(0x7f48d809d200 legacy=0x7f48d8136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:15.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:15.520+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:25:16.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:16.324+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 msgr2=0x7f48d8097e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:16.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:16.324+0000 7f48de597700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.150:6789/0 conn(0x7f48d809d200 legacy=0x7f48d8136910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.150:6789/0 2024-09-18T10:25:16.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:16.324+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:25:17.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): tick 2024-09-18T10:25:17.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:25:17.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:25:17.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 msgr2=0x7f48d8097e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:17.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48d8097a30 0x7f48d8097e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.150:6789/0 conn(0x7f48d809d200 legacy=0x7f48d8136910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.142:6789/0 conn(0x7f48d809c6f0 legacy=0x7f48d809caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient: _add_conns ranks=[2,3] 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): picked mon.noname-b con 0x7f48bc003790 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0087f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc00b110 con 0x7f48bc0087f0 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc00b250 con 0x7f48bc003790 2024-09-18T10:25:17.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48cf7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 10 monclient(hunting): get_auth_request con 0x7f48bc0087f0 auth_method 0 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 10 monclient(hunting): get_auth_request con 0x7f48bc003790 auth_method 0 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:17.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:17.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:25:17.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:17.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:17.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:17.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:17.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:18.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 10 monclient(hunting): get_auth_request con 0x7f48bc003790 auth_method 0 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 10 monclient(hunting): get_auth_request con 0x7f48bc0087f0 auth_method 0 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:18.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:18.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.120+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 10 monclient(hunting): get_auth_request con 0x7f48bc0087f0 auth_method 0 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 10 monclient(hunting): get_auth_request con 0x7f48bc003790 auth_method 0 2024-09-18T10:25:18.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:18.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:18.524+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:25:19.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 10 monclient(hunting): get_auth_request con 0x7f48bc003790 auth_method 0 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 10 monclient(hunting): get_auth_request con 0x7f48bc0087f0 auth_method 0 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:25:19.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:25:19.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:19.328+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:25:20.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): tick 2024-09-18T10:25:20.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 monclient(hunting): continuing hunt 2024-09-18T10:25:20.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 msgr2=0x7f48bc003bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc003790 0x7f48bc003bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0087f0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48de597700 1 -- 172.21.15.142:0/380324778 reap_dead start 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0046b0 0x7f48bc004ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0046b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 0x7f48bc00abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): picked mon.noname-c con 0x7f48bc003bf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:20.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48bc005170 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d809a210 con 0x7f48bc005170 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.912+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc00b110 con 0x7f48bc0046b0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48cf7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48cf7fe700 1 -- 172.21.15.142:0/380324778 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc00b250 con 0x7f48bc003bf0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48cf7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48dd595700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48dd595700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.142:3300/0 conn(0x7f48bc0046b0 0x7f48bc004ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): get_auth_request con 0x7f48bc0046b0 auth_method 0 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:20.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient(hunting): handle_auth_done global_id 184176 payload 274 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 msgr2=0x7f48bc00abd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 0x7f48bc00abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> v1:172.21.15.142:6789/0 conn(0x7f48bc005170 legacy=0x7f48bc0075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient: _finish_hunting 0 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 monclient: found mon.noname-a 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:20.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d809a4f0 con 0x7f48bc0046b0 2024-09-18T10:25:20.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 msgr2=0x7f48bc004ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:20.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.916+0000 7f48ddd96700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48bc004ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f48bc003bc0 tx=0x7f48d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe1e88dd151592f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:20.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 1 -- 172.21.15.142:0/380324778 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d400e030 con 0x7f48bc0046b0 2024-09-18T10:25:20.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:20.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: dump: 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:20.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: _finish_auth 0 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 5 monclient: authenticate success, global_id 184176 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 1 -- 172.21.15.142:0/380324778 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d4005b80 con 0x7f48bc0046b0 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 4 set_mon_vals no callback set 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 1 -- 172.21.15.142:0/380324778 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40178d0 con 0x7f48bc0046b0 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dcd94700 10 monclient: dump: 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:20.930 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 10 monclient: shutdown 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 -- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 msgr2=0x7f48bc004ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48bc004ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f48bc003bc0 tx=0x7f48d400b040 comp rx=0 tx=0).stop 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 -- 172.21.15.142:0/380324778 shutdown_connections 2024-09-18T10:25:20.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 --2- 172.21.15.142:0/380324778 >> v2:172.21.15.150:3300/0 conn(0x7f48bc003bf0 0x7f48bc00abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 --2- 172.21.15.142:0/380324778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48bc004ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 -- 172.21.15.142:0/380324778 >> 172.21.15.142:0/380324778 conn(0x7f48d80a1470 msgr2=0x7f48d80a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:20.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 -- 172.21.15.142:0/380324778 shutdown_connections 2024-09-18T10:25:20.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 1 -- 172.21.15.142:0/380324778 wait complete. 2024-09-18T10:25:20.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 10 monclient: build_initial_monmap 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.920+0000 7f48dfffa700 10 monclient: monmap: 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:20.941 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 Processor -- start 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 -- start start 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient: init 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 0x7f48d813d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-b con 0x7f48bc003bf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48d813dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0046b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:20.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 0x7f48d813e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): picked mon.noname-c con 0x7f48bc005170 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d806f670 con 0x7f48bc0046b0 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d80a06b0 con 0x7f48bc003bf0 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8129cb0 con 0x7f48bc005170 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dfffa700 10 monclient(hunting): authenticate will time out at 1809.853588s 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48ddd96700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 msgr2=0x7f48d813d670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48ddd96700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 0x7f48d813d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48d813dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:20.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 0x7f48d813e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48d813dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38938/0 (socket says 172.21.15.142:38938) 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): get_auth_request con 0x7f48bc005170 auth_method 0 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 1 -- 172.21.15.142:0/1805969282 learned_addr learned my addr 172.21.15.142:0/1805969282 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): get_auth_request con 0x7f48bc0046b0 auth_method 0 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:20.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48dd595700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient(hunting): handle_auth_done global_id 75674 payload 824 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 -- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 msgr2=0x7f48d813d670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 0x7f48d813d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 -- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 msgr2=0x7f48d813dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48d813dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient: _finish_hunting 0 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 monclient: found mon.noname-c 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.924+0000 7f48de597700 1 -- 172.21.15.142:0/1805969282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4009b00 con 0x7f48bc005170 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48de597700 1 -- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 msgr2=0x7f48d813e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:20.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48de597700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 0x7f48d813e0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f48d809d5d0 tx=0x7f48d0009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d000cc30 con 0x7f48bc005170 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: dump: 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:20.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 5 monclient: authenticate success, global_id 75674 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d000cd90 con 0x7f48bc005170 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 10 monclient: _renew_subs 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d812a400 con 0x7f48bc005170 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0016430 con 0x7f48bc005170 2024-09-18T10:25:20.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 10 monclient: dump: 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:20.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cffff700 4 set_mon_vals no callback set 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 10 monclient: _renew_subs 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d812b890 con 0x7f48bc005170 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f48d0012020 con 0x7f48bc005170 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.928+0000 7f48cdffb700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 0x7f48bc039bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f48d0048d10 con 0x7f48bc005170 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48dfffa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48dfffa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48dfffa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48ddd96700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 0x7f48bc039bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d812a940 con 0x7f48bc005170 2024-09-18T10:25:20.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48ddd96700 10 monclient: get_auth_request con 0x7f48bc037700 auth_method 0 2024-09-18T10:25:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.932+0000 7f48ddd96700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 0x7f48bc039bb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005140 tx=0x7f48d401f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.940+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f48d00153e0 con 0x7f48bc005170 2024-09-18T10:25:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.940+0000 7f48cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:20.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:20.940+0000 7f48cdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:21.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48dfffa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:21.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48dfffa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:21.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48dfffa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f48d80696a0 con 0x7f48bc005170 2024-09-18T10:25:21.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48cdffb700 1 -- 172.21.15.142:0/1805969282 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f48d001d300 con 0x7f48bc005170 2024-09-18T10:25:21.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:21.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.168+0000 7f48cdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:21.179 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":152,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:21.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 msgr2=0x7f48bc039bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 0x7f48bc039bb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f48bc005140 tx=0x7f48d401f000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 10 monclient: shutdown 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 msgr2=0x7f48d813e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 0x7f48d813e0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f48d809d5d0 tx=0x7f48d0009660 comp rx=0 tx=0).stop 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 shutdown_connections 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f48bc037700 0x7f48bc039bb0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48bc005170 0x7f48d813e0f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48bc0046b0 0x7f48d813dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 --2- 172.21.15.142:0/1805969282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48bc003bf0 0x7f48d813d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 >> 172.21.15.142:0/1805969282 conn(0x7f48d80a1470 msgr2=0x7f48d809f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:21.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 shutdown_connections 2024-09-18T10:25:21.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.172+0000 7f48dfffa700 1 -- 172.21.15.142:0/1805969282 wait complete. 2024-09-18T10:25:21.193 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:25:21.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:25:21.193 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:25:21.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: get_monmap_and_config 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: build_initial_monmap 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: monmap: 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:21.301281+0000 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:21.301281+0000 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:21.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 Processor -- start 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 -- start start 2024-09-18T10:25:21.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: init 2024-09-18T10:25:21.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:21.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:25:21.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cf409abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:21.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 0x7f3cf409c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-b con 0x7f3cf4097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 0x7f3cf4137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-c con 0x7f3cf409db80 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40958c0 con 0x7f3cf409abd0 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf406fa60 con 0x7f3cf409db80 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4095ef0 con 0x7f3cf4097880 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cfa40d700 10 monclient(hunting): authenticate will time out at 1810.226929s 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf3fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3cf409abd0 0x7f3cf409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49106/0 (socket says 172.21.15.142:49106) 2024-09-18T10:25:21.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf3fff700 1 -- 172.21.15.142:0/3079009008 learned_addr learned my addr 172.21.15.142:0/3079009008 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf89aa700 1 -- 172.21.15.142:0/3079009008 >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 msgr2=0x7f3cf4137690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf89aa700 1 --2- 172.21.15.142:0/3079009008 >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 0x7f3cf4137690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 --2- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 0x7f3cf409c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): get_auth_request con 0x7f3cf4097880 auth_method 0 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_done global_id 75677 payload 274 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 -- 172.21.15.142:0/3079009008 >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 msgr2=0x7f3cf4137690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 --2- 172.21.15.142:0/3079009008 >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 0x7f3cf4137690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 -- 172.21.15.142:0/3079009008 >> v1:172.21.15.142:6789/0 conn(0x7f3cf409abd0 legacy=0x7f3cf409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:21.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:21.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 monclient: found mon.noname-b 2024-09-18T10:25:21.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:21.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.296+0000 7f3cf37fe700 1 -- 172.21.15.142:0/3079009008 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf409afa0 con 0x7f3cf4097880 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf37fe700 1 -- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 msgr2=0x7f3cf409c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf37fe700 1 --2- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 0x7f3cf409c5e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009e60 tx=0x7f3cdc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb5c9fe12ec6dd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 1 -- 172.21.15.142:0/3079009008 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00e070 con 0x7f3cf4097880 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: dump: 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 5 monclient: authenticate success, global_id 75677 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 1 -- 172.21.15.142:0/3079009008 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc012430 con 0x7f3cf4097880 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 4 set_mon_vals no callback set 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 1 -- 172.21.15.142:0/3079009008 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0127d0 con 0x7f3cf4097880 2024-09-18T10:25:21.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cf2ffd700 10 monclient: dump: 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 10 monclient: shutdown 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 -- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 msgr2=0x7f3cf409c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 0x7f3cf409c5e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009e60 tx=0x7f3cdc009780 comp rx=0 tx=0).stop 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 -- 172.21.15.142:0/3079009008 shutdown_connections 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/3079009008 >> v2:172.21.15.150:3300/0 conn(0x7f3cf409db80 0x7f3cf4137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/3079009008 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf4097880 0x7f3cf409c5e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 -- 172.21.15.142:0/3079009008 >> 172.21.15.142:0/3079009008 conn(0x7f3cf40a14e0 msgr2=0x7f3cf40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 -- 172.21.15.142:0/3079009008 shutdown_connections 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.300+0000 7f3cfa40d700 1 -- 172.21.15.142:0/3079009008 wait complete. 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient: build_initial_monmap 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient: monmap: 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:21.314 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 Processor -- start 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 -- start start 2024-09-18T10:25:21.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient: init 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cf4097880 0x7f3cf41417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-b con 0x7f3cf4097880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf409abd0 0x7f3cf4094310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-c con 0x7f3cf409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 0x7f3cf40989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): picked mon.noname-a con 0x7f3cf409db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40a0120 con 0x7f3cf409db80 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4091810 con 0x7f3cf4097880 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4142640 con 0x7f3cf409abd0 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cfa40d700 10 monclient(hunting): authenticate will time out at 1810.233847s 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf3fff700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cf4097880 msgr2=0x7f3cf41417d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf409abd0 0x7f3cf4094310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 0x7f3cf40989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf3fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cf4097880 0x7f3cf41417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:21.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 0x7f3cf40989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38944/0 (socket says 172.21.15.142:38944) 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 learned_addr learned my addr 172.21.15.142:0/519717652 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): get_auth_request con 0x7f3cf409abd0 auth_method 0 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): get_auth_request con 0x7f3cf409db80 auth_method 0 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient(hunting): handle_auth_done global_id 184182 payload 824 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf409abd0 msgr2=0x7f3cf4094310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cf409abd0 0x7f3cf4094310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cf4097880 msgr2=0x7f3cf41417d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cf4097880 0x7f3cf41417d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient: _finish_hunting 0 2024-09-18T10:25:21.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 monclient: found mon.noname-a 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.304+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc009a20 con 0x7f3cf409db80 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 msgr2=0x7f3cf40989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf89aa700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 0x7f3cf40989c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00b590 tx=0x7f3cec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec003dd0 con 0x7f3cf409db80 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: dump: 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.325 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: _finish_auth 0 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 5 monclient: authenticate success, global_id 184182 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec006150 con 0x7f3cf409db80 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 monclient: _renew_subs 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf409a520 con 0x7f3cf409db80 2024-09-18T10:25:21.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec006510 con 0x7f3cf409db80 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf27fc700 4 set_mon_vals no callback set 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 10 monclient: dump: 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.327 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 monclient: _renew_subs 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf40907e0 con 0x7f3cf409db80 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 0x7f3cf4098f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3cec01b3b0 con 0x7f3cf409db80 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4094c20 con 0x7f3cf4094850 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 0x7f3cf4098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient: get_auth_request con 0x7f3cf4094850 auth_method 0 2024-09-18T10:25:21.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cd4035a70 0x7f3cd4037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3cec0484a0 con 0x7f3cf409db80 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_done global_id 184185 payload 824 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.308+0000 7f3cf3fff700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cf4091d30 con 0x7f3cf4094850 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cf37fe700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cd4035a70 0x7f3cd4037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cf37fe700 10 monclient: get_auth_request con 0x7f3cd4035a70 auth_method 0 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cf3fff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 msgr2=0x7f3cf4098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cf3fff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 0x7f3cf4098f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40048c0 tx=0x7f3ce4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4005e00 con 0x7f3cf4094850 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 10 monclient: dump: 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.330 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cf37fe700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cd4035a70 0x7f3cd4037f20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009e60 tx=0x7f3cdc018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3ce40070d0 con 0x7f3cf4094850 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 msgr2=0x7f3cf4098f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 0x7f3cf4098f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40048c0 tx=0x7f3ce4007f90 comp rx=0 tx=0).stop 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 msgr2=0x7f3cf4098f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:21.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.312+0000 7f3cebfff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4094850 0x7f3cf4098f00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.320+0000 7f3cfa40d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:21.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.320+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 0x7f3cf4087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.320+0000 7f3cfa40d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.320+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4064950 con 0x7f3cf4099440 2024-09-18T10:25:21.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 0x7f3cf4087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient: get_auth_request con 0x7f3cf4099440 auth_method 0 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 10 monclient(hunting): handle_auth_done global_id 184188 payload 824 2024-09-18T10:25:21.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 1 -- 172.21.15.142:0/519717652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3cf4091980 con 0x7f3cf4099440 2024-09-18T10:25:21.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 msgr2=0x7f3cf4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cf3fff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 0x7f3cf4087fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4014000 tx=0x7f3ce4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4023070 con 0x7f3cf4099440 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cebfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.324+0000 7f3cebfff700 10 monclient: dump: 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.335 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f3ce4005bf0 con 0x7f3cf4099440 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 msgr2=0x7f3cf4087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 0x7f3cf4087fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4014000 tx=0x7f3ce4015000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 msgr2=0x7f3cf4087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cebfff700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf4099440 0x7f3cf4087fe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:21.336 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 152, 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.337 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.338 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.339 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:21.340 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.341 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:21.342 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.344 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.345 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.347 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.348 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.350 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:21.350 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cd4035a70 msgr2=0x7f3cd4037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cd4035a70 0x7f3cd4037f20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009e60 tx=0x7f3cdc018000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 10 monclient: shutdown 2024-09-18T10:25:21.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 msgr2=0x7f3cf40989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 --2- 172.21.15.142:0/519717652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cf409db80 0x7f3cf40989c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00b590 tx=0x7f3cec0090f0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 reap_dead start 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cf89aa700 1 -- 172.21.15.142:0/519717652 reap_dead start 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 shutdown_connections 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 >> 172.21.15.142:0/519717652 conn(0x7f3cf40a14e0 msgr2=0x7f3cf4096330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 shutdown_connections 2024-09-18T10:25:21.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.328+0000 7f3cfa40d700 1 -- 172.21.15.142:0/519717652 wait complete. 2024-09-18T10:25:21.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:25:21.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:25:21.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:25:21.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: get_monmap_and_config 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: build_initial_monmap 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: monmap: 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:21.465394+0000 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:21.465394+0000 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:21.468 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:21.469 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:21.469 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 Processor -- start 2024-09-18T10:25:21.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 -- start start 2024-09-18T10:25:21.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: init 2024-09-18T10:25:21.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f85d406e770 0x7f85d409b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-a con 0x7f85d406e770 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85d409b6e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-b con 0x7f85d412f2f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d406f2e0 con 0x7f85d409b6e0 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d409dc40 con 0x7f85d406e770 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d406eb40 con 0x7f85d412f2f0 2024-09-18T10:25:21.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85daabb700 10 monclient(hunting): authenticate will time out at 1810.390818s 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57948/0 (socket says 172.21.15.142:57948) 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 -- 172.21.15.142:0/4102722093 learned_addr learned my addr 172.21.15.142:0/4102722093 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): get_auth_request con 0x7f85d412f2f0 auth_method 0 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient(hunting): handle_auth_done global_id 75683 payload 274 2024-09-18T10:25:21.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 -- 172.21.15.142:0/4102722093 >> v2:172.21.15.142:3300/0 conn(0x7f85d406e770 msgr2=0x7f85d409b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 --2- 172.21.15.142:0/4102722093 >> v2:172.21.15.142:3300/0 conn(0x7f85d406e770 0x7f85d409b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 -- 172.21.15.142:0/4102722093 >> v1:172.21.15.142:6789/0 conn(0x7f85d409b6e0 legacy=0x7f85d412ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient: _finish_hunting 0 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 monclient: found mon.noname-b 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.460+0000 7f85da2ba700 1 -- 172.21.15.142:0/4102722093 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d40989c0 con 0x7f85d412f2f0 2024-09-18T10:25:21.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85da2ba700 1 -- 172.21.15.142:0/4102722093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 msgr2=0x7f85d4139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85da2ba700 1 --2- 172.21.15.142:0/4102722093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009f20 tx=0x7f85cc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed54374db2a86a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 1 -- 172.21.15.142:0/4102722093 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00e070 con 0x7f85d412f2f0 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: dump: 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.478 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: _finish_auth 0 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 1 -- 172.21.15.142:0/4102722093 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc00bc90 con 0x7f85d412f2f0 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 5 monclient: authenticate success, global_id 75683 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 1 -- 172.21.15.142:0/4102722093 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc012660 con 0x7f85d412f2f0 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 4 set_mon_vals no callback set 2024-09-18T10:25:21.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85d8ab7700 10 monclient: dump: 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 10 monclient: shutdown 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 -- 172.21.15.142:0/4102722093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 msgr2=0x7f85d4139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 --2- 172.21.15.142:0/4102722093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f85cc009f20 tx=0x7f85cc009840 comp rx=0 tx=0).stop 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 -- 172.21.15.142:0/4102722093 shutdown_connections 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 --2- 172.21.15.142:0/4102722093 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d412f2f0 0x7f85d4139200 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 --2- 172.21.15.142:0/4102722093 >> v2:172.21.15.142:3300/0 conn(0x7f85d406e770 0x7f85d409b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 -- 172.21.15.142:0/4102722093 >> 172.21.15.142:0/4102722093 conn(0x7f85d4070ef0 msgr2=0x7f85d40712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 -- 172.21.15.142:0/4102722093 shutdown_connections 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 1 -- 172.21.15.142:0/4102722093 wait complete. 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 10 monclient: build_initial_monmap 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.464+0000 7f85daabb700 10 monclient: monmap: 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:21.481 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 Processor -- start 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 -- start start 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient: init 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:21.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85d406e770 0x7f85d4131870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-a con 0x7f85d406e770 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 0x7f85d4132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-c con 0x7f85d409b6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85d412f2f0 0x7f85d4133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): picked mon.noname-b con 0x7f85d412f2f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4134070 con 0x7f85d406e770 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4134fd0 con 0x7f85d412f2f0 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4135590 con 0x7f85d409b6e0 2024-09-18T10:25:21.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85daabb700 10 monclient(hunting): authenticate will time out at 1810.397580s 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 0x7f85d4132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85d406e770 0x7f85d4131870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85d406e770 0x7f85d4131870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38974/0 (socket says 172.21.15.142:38974) 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 0x7f85d4132ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:57958/0 (socket says 172.21.15.142:57958) 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 1 -- 172.21.15.142:0/3535375568 learned_addr learned my addr 172.21.15.142:0/3535375568 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85da2ba700 1 -- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85d412f2f0 msgr2=0x7f85d4133410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.150:3300/0 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): get_auth_request con 0x7f85d406e770 auth_method 0 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): get_auth_request con 0x7f85d409b6e0 auth_method 0 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85da2ba700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85d412f2f0 0x7f85d4133410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient(hunting): handle_auth_done global_id 75686 payload 824 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85d412f2f0 msgr2=0x7f85d4133410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:21.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85d412f2f0 0x7f85d4133410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85d406e770 msgr2=0x7f85d4131870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85d406e770 0x7f85d4131870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient: _finish_hunting 0 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 monclient: found mon.noname-c 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.468+0000 7f85d92b8700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc009ae0 con 0x7f85d409b6e0 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d92b8700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 msgr2=0x7f85d4132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d92b8700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 0x7f85d4132ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f85c800c4e0 tx=0x7f85c8007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80117a0 con 0x7f85d409b6e0 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: dump: 2024-09-18T10:25:21.490 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:21.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 5 monclient: authenticate success, global_id 75686 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c8011920 con 0x7f85d409b6e0 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 monclient: _renew_subs 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d4135870 con 0x7f85d409b6e0 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011ce0 con 0x7f85d409b6e0 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 10 monclient: dump: 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 monclient: _renew_subs 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d3fff700 4 set_mon_vals no callback set 2024-09-18T10:25:21.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d4134b30 con 0x7f85d409b6e0 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 0x7f85d4096350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d40648a0 con 0x7f85d4133950 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 0x7f85d4096350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient: get_auth_request con 0x7f85d4133950 auth_method 0 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f85c8012630 con 0x7f85d409b6e0 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_done global_id 75689 payload 824 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d9ab9700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85d41322e0 con 0x7f85d4133950 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f85c0035a70 0x7f85c0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f85c8048ce0 con 0x7f85d409b6e0 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85da2ba700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f85c0035a70 0x7f85c0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.472+0000 7f85da2ba700 10 monclient: get_auth_request con 0x7f85c0035a70 auth_method 0 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d9ab9700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 msgr2=0x7f85d4096350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d9ab9700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 0x7f85d4096350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e900 tx=0x7f85c4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c40029f0 con 0x7f85d4133950 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 10 monclient: dump: 2024-09-18T10:25:21.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f85c4002b50 con 0x7f85d4133950 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:21.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 msgr2=0x7f85d4096350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 0x7f85d4096350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e900 tx=0x7f85c4009250 comp rx=0 tx=0).stop 2024-09-18T10:25:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 msgr2=0x7f85d4096350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85d1ffb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4133950 0x7f85d4096350 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f85c400e900 tx=0x7f85c4009250 comp rx=0 tx=0).stop 2024-09-18T10:25:21.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.476+0000 7f85da2ba700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f85c0035a70 0x7f85c0037f20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f85d40677f0 tx=0x7f85cc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.484+0000 7f85daabb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85daabb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 0x7f85d4096c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85daabb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d4133d20 con 0x7f85d4096890 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 0x7f85d4096c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient: get_auth_request con 0x7f85d4096890 auth_method 0 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:21.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 10 monclient(hunting): handle_auth_done global_id 75692 payload 824 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 1 -- 172.21.15.142:0/3535375568 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f85d4072a80 con 0x7f85d4096890 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 msgr2=0x7f85d4096c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d9ab9700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 0x7f85d4096c90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f85c4018000 tx=0x7f85c4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4006030 con 0x7f85d4096890 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 10 monclient: dump: 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:21.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f85c4006190 con 0x7f85d4096890 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 msgr2=0x7f85d4096c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 0x7f85d4096c90 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f85c4018000 tx=0x7f85c4016000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 msgr2=0x7f85d4096c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.488+0000 7f85d1ffb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d4096890 0x7f85d4096c90 secure :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f85c4018000 tx=0x7f85c4016000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 152, 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:21.502 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.503 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.504 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:21.505 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:21.506 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.507 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.508 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.509 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.510 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.511 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.512 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:21.513 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:21.514 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:21.515 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:21.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f85c0035a70 msgr2=0x7f85c0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f85c0035a70 0x7f85c0037f20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f85d40677f0 tx=0x7f85cc017000 comp rx=0 tx=0).stop 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 10 monclient: shutdown 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 msgr2=0x7f85d4132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 --2- 172.21.15.142:0/3535375568 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85d409b6e0 0x7f85d4132ed0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f85c800c4e0 tx=0x7f85c8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85da2ba700 1 -- 172.21.15.142:0/3535375568 reap_dead start 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85da2ba700 1 -- 172.21.15.142:0/3535375568 reap_dead start 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 shutdown_connections 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 >> 172.21.15.142:0/3535375568 conn(0x7f85d4070ef0 msgr2=0x7f85d4099140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 shutdown_connections 2024-09-18T10:25:21.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:21.492+0000 7f85daabb700 1 -- 172.21.15.142:0/3535375568 wait complete. 2024-09-18T10:25:21.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:25:21.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:25:41.519 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-18T10:25:41.519 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-18T10:25:41.519 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T10:25:41.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T10:25:41.523 INFO:tasks.ceph.mon.b:Started 2024-09-18T10:25:41.523 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:25:41.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: get_monmap_and_config 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: build_initial_monmap 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: monmap: 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:41.632701+0000 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:41.632701+0000 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:41.635 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:41.636 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 Processor -- start 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 -- start start 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: init 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb6009d870 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 0x7ffb600990d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-c con 0x7ffb600954b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb60134670 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb60069c70 con 0x7ffb60134670 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb600958b0 con 0x7ffb6009d870 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6006f9d0 con 0x7ffb600954b0 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb66f61700 10 monclient(hunting): authenticate will time out at 1830.558187s 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb654fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ffb60134670 0x7ffb60136a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:36988/0 (socket says 172.21.15.142:36988) 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 learned_addr learned my addr 172.21.15.142:0/743740948 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 0x7ffb600990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request con 0x7ffb600954b0 auth_method 0 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 1 -- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 msgr2=0x7ffb600990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 1 -- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 msgr2=0x7ffb600990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:25:41.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 0x7ffb600990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.628+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 0x7ffb600990d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 >> v1:172.21.15.142:6789/0 conn(0x7ffb60134670 legacy=0x7ffb60136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 1 -- 172.21.15.142:0/743740948 >> v1:172.21.15.150:6789/0 conn(0x7ffb6009d870 legacy=0x7ffb600989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943803562 0 0) 0x7ffb60069c70 con 0x7ffb60134670 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): my global_id is 184194 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50003570 con 0x7ffb60134670 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087204633 0 0) 0x7ffb50003570 con 0x7ffb60134670 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 msgr2=0x7ffb600990d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 --2- 172.21.15.142:0/743740948 >> v2:172.21.15.150:3300/0 conn(0x7ffb600954b0 0x7ffb600990d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 >> v1:172.21.15.150:6789/0 conn(0x7ffb6009d870 legacy=0x7ffb600989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6009dc40 con 0x7ffb60134670 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb66f61700 5 monclient: authenticate success, global_id 184194 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ffb58001be0 con 0x7ffb60134670 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb66f61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: dump: 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:41.649 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 >> v1:172.21.15.142:6789/0 conn(0x7ffb60134670 legacy=0x7ffb60136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50003c50 0x7ffb50004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): picked mon.b con 0x7ffb50003c50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb50004810 0x7ffb50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): picked mon.c con 0x7ffb50004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 0x7ffb5000be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): picked mon.a con 0x7ffb50009a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c480 con 0x7ffb50009a50 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c5f0 con 0x7ffb50003c50 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000c7b0 con 0x7ffb50004810 2024-09-18T10:25:41.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:41.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:25:41.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb66f61700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:41.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 0x7ffb5000be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50003c50 0x7ffb50004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): get_auth_request con 0x7ffb50009a50 auth_method 0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): get_auth_request con 0x7ffb50003c50 auth_method 0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb50004810 0x7ffb50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request con 0x7ffb50004810 auth_method 0 2024-09-18T10:25:41.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb64cfd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient(hunting): handle_auth_done global_id 184194 payload 293 2024-09-18T10:25:41.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb50004810 msgr2=0x7ffb50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb50004810 0x7ffb50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50003c50 msgr2=0x7ffb50004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50003c50 0x7ffb50004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 monclient: found mon.a 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50003570 con 0x7ffb50009a50 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.632+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 reap_dead start 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb654fe700 1 -- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 msgr2=0x7ffb5000be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb654fe700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 0x7ffb5000be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb58001850 tx=0x7ffb5800a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e331a9281605f21 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:41.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800c5d0 con 0x7ffb50009a50 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: dump: 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.656 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800c730 con 0x7ffb50009a50 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 1 -- 172.21.15.142:0/743740948 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5800caf0 con 0x7ffb50009a50 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 4 set_mon_vals no callback set 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:41.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb5f7fe700 10 monclient: dump: 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:41.658 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 10 monclient: shutdown 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 msgr2=0x7ffb5000be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 0x7ffb5000be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb58001850 tx=0x7ffb5800a220 comp rx=0 tx=0).stop 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- 172.21.15.142:0/743740948 shutdown_connections 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 --2- 172.21.15.142:0/743740948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb50009a50 0x7ffb5000be30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- 172.21.15.142:0/743740948 >> 172.21.15.142:0/743740948 conn(0x7ffb600a14e0 msgr2=0x7ffb600a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- 172.21.15.142:0/743740948 shutdown_connections 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- 172.21.15.142:0/743740948 wait complete. 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 10 monclient: build_initial_monmap 2024-09-18T10:25:41.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 10 monclient: monmap: 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 Processor -- start 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 1 -- start start 2024-09-18T10:25:41.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.636+0000 7ffb66f61700 10 monclient: init 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50009a50 0x7ffb601340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-b con 0x7ffb50009a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-c con 0x7ffb60094a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 0x7ffb60136580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): picked mon.noname-a con 0x7ffb60136120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:41.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6009f900 con 0x7ffb60136120 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb601393b0 con 0x7ffb50009a50 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb60139970 con 0x7ffb60094a20 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb66f61700 10 monclient(hunting): authenticate will time out at 1830.568694s 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:45198/0 (socket says 172.21.15.142:45198) 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 -- 172.21.15.142:0/482666807 learned_addr learned my addr 172.21.15.142:0/482666807 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request con 0x7ffb60094a20 auth_method 0 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 0x7ffb60136580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 10 monclient(hunting): get_auth_request con 0x7ffb60136120 auth_method 0 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb64cfd700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50009a50 0x7ffb601340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb64cfd700 10 monclient(hunting): get_auth_request con 0x7ffb50009a50 auth_method 0 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb64cfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:41.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb64cfd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb64cfd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient(hunting): handle_auth_done global_id 75695 payload 824 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 -- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50009a50 msgr2=0x7ffb601340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50009a50 0x7ffb601340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 -- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 msgr2=0x7ffb60136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 0x7ffb60136580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 monclient: found mon.noname-c 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 -- 172.21.15.142:0/482666807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb601390d0 con 0x7ffb60094a20 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb654fe700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 0x7ffb60136580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:41.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 -- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 msgr2=0x7ffb60135be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5ffff700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00fac0 tx=0x7ffb4c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.640+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00eb50 con 0x7ffb60094a20 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: dump: 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:41.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: _finish_auth 0 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 5 monclient: authenticate success, global_id 75695 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _renew_subs 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c00ecb0 con 0x7ffb60094a20 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb60137670 con 0x7ffb60094a20 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c013440 con 0x7ffb60094a20 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:41.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5e7fc700 4 set_mon_vals no callback set 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 10 monclient: dump: 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _renew_subs 2024-09-18T10:25:41.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb60094f90 con 0x7ffb60094a20 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb600696a0 con 0x7ffb60094a20 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ffb4c0138b0 con 0x7ffb60094a20 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 0x7ffb44037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ffb4c049780 con 0x7ffb60094a20 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb64cfd700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 0x7ffb44037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.644+0000 7ffb64cfd700 10 monclient: get_auth_request con 0x7ffb44035af0 auth_method 0 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.648+0000 7ffb64cfd700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 0x7ffb44037fa0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007a60 tx=0x7ffb54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.656+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ffb4c0116a0 con 0x7ffb60094a20 2024-09-18T10:25:41.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.656+0000 7ffb5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:41.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.656+0000 7ffb5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.876+0000 7ffb66f61700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.876+0000 7ffb66f61700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.876+0000 7ffb66f61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.876+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffb60095d10 con 0x7ffb60094a20 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.880+0000 7ffb5cff9700 1 -- 172.21.15.142:0/482666807 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7ffb4c0116a0 con 0x7ffb60094a20 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.880+0000 7ffb5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.880+0000 7ffb5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:41.887 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":152,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:41.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 msgr2=0x7ffb44037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 0x7ffb44037fa0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007a60 tx=0x7ffb54009040 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 10 monclient: shutdown 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 msgr2=0x7ffb60135be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00fac0 tx=0x7ffb4c008040 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 shutdown_connections 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ffb44035af0 0x7ffb44037fa0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb60136120 0x7ffb60136580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffb60094a20 0x7ffb60135be0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 --2- 172.21.15.142:0/482666807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb50009a50 0x7ffb601340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 >> 172.21.15.142:0/482666807 conn(0x7ffb600a14e0 msgr2=0x7ffb6009c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 shutdown_connections 2024-09-18T10:25:41.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:41.884+0000 7ffb66f61700 1 -- 172.21.15.142:0/482666807 wait complete. 2024-09-18T10:25:44.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.008+0000 7f413da17700 10 monclient: get_monmap_and_config 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.008+0000 7f413da17700 10 monclient: build_initial_monmap 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.008+0000 7f413da17700 10 monclient: monmap: 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:45.014872+0000 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:45.014872+0000 2024-09-18T10:25:45.017 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:45.018 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.008+0000 7f413da17700 1 Processor -- start 2024-09-18T10:25:45.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 -- start start 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient: init 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f413809cf50 0x7f413809d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): picked mon.noname-c con 0x7f413809cf50 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f413809e270 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): picked mon.noname-b con 0x7f413809d860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413809ef60 con 0x7f413809e270 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413806b8b0 con 0x7f413809cf50 2024-09-18T10:25:45.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4138092f20 con 0x7f413809d860 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient(hunting): authenticate will time out at 1833.940688s 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54298/0 (socket says 172.21.15.142:54298) 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 -- 172.21.15.142:0/761764599 learned_addr learned my addr 172.21.15.142:0/761764599 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:45.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): get_auth_request con 0x7f413809d860 auth_method 0 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f4136ffd700 1 --2- 172.21.15.142:0/761764599 >> v2:172.21.15.150:3300/0 conn(0x7f413809cf50 0x7f413809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f4136ffd700 10 monclient(hunting): get_auth_request con 0x7f413809cf50 auth_method 0 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f4136ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f4136ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:45.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f4136ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient(hunting): handle_auth_done global_id 75698 payload 274 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 -- 172.21.15.142:0/761764599 >> v2:172.21.15.150:3300/0 conn(0x7f413809cf50 msgr2=0x7f413809d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 --2- 172.21.15.142:0/761764599 >> v2:172.21.15.150:3300/0 conn(0x7f413809cf50 0x7f413809d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 -- 172.21.15.142:0/761764599 >> v1:172.21.15.150:6789/0 conn(0x7f413809e270 legacy=0x7f4138136800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 monclient: found mon.noname-b 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:45.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 -- 172.21.15.142:0/761764599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413809dc60 con 0x7f413809d860 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 -- 172.21.15.142:0/761764599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 msgr2=0x7f4138139f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41377fe700 1 --2- 172.21.15.142:0/761764599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4130009f20 tx=0x7f4130009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2377ff552858f540 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 1 -- 172.21.15.142:0/761764599 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413000e070 con 0x7f413809d860 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: dump: 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:45.031 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:45.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 5 monclient: authenticate success, global_id 75698 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 1 -- 172.21.15.142:0/761764599 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413000bc90 con 0x7f413809d860 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f413da17700 4 set_mon_vals no callback set 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 1 -- 172.21.15.142:0/761764599 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41300092e0 con 0x7f413809d860 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:45.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.012+0000 7f41367fc700 10 monclient: dump: 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:45.034 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: shutdown 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- 172.21.15.142:0/761764599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 msgr2=0x7f4138139f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- 172.21.15.142:0/761764599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4130009f20 tx=0x7f4130009840 comp rx=0 tx=0).stop 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- 172.21.15.142:0/761764599 shutdown_connections 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- 172.21.15.142:0/761764599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138139f30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- 172.21.15.142:0/761764599 >> v2:172.21.15.150:3300/0 conn(0x7f413809cf50 0x7f413809d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- 172.21.15.142:0/761764599 >> 172.21.15.142:0/761764599 conn(0x7f41380a2660 msgr2=0x7f41380a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- 172.21.15.142:0/761764599 shutdown_connections 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- 172.21.15.142:0/761764599 wait complete. 2024-09-18T10:25:45.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: build_initial_monmap 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: monmap: 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:45.036 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 Processor -- start 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- start start 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: init 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): picked mon.noname-a con 0x7f413809cf50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): picked mon.noname-c con 0x7f413809d860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413809e270 0x7f4138142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): picked mon.noname-b con 0x7f413809e270 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413808e820 con 0x7f413809cf50 2024-09-18T10:25:45.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4138092680 con 0x7f413809e270 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413809bad0 con 0x7f413809d860 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f413da17700 10 monclient(hunting): authenticate will time out at 1833.946331s 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36582/0 (socket says 172.21.15.142:36582) 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 -- 172.21.15.142:0/4289275009 learned_addr learned my addr 172.21.15.142:0/4289275009 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): get_auth_request con 0x7f413809cf50 auth_method 0 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:45.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): get_auth_request con 0x7f413809d860 auth_method 0 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f41377fe700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413809e270 0x7f4138142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f41377fe700 10 monclient(hunting): get_auth_request con 0x7f413809e270 auth_method 0 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f41377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f41377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f41377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient(hunting): handle_auth_done global_id 184200 payload 824 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 -- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 msgr2=0x7f4138141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 -- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413809e270 msgr2=0x7f4138142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413809e270 0x7f4138142350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 monclient: found mon.noname-a 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:45.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f4136ffd700 1 -- 172.21.15.142:0/4289275009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130009ae0 con 0x7f413809cf50 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.016+0000 7f412e7fc700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f4136ffd700 1 -- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 msgr2=0x7f41381418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f4136ffd700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f412800d890 tx=0x7f4128008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128012800 con 0x7f413809cf50 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: dump: 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:45.041 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 5 monclient: authenticate success, global_id 184200 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4128012980 con 0x7f413809cf50 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _renew_subs 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413808d5e0 con 0x7f413809cf50 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:45.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128012d40 con 0x7f413809cf50 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f41357fa700 4 set_mon_vals no callback set 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 10 monclient: dump: 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:45.043 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _renew_subs 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413808c5f0 con 0x7f413809cf50 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4128014640 con 0x7f413809cf50 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 0x7f4118037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41380696a0 con 0x7f413809cf50 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4128049610 con 0x7f413809cf50 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412e7fc700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 0x7f4118037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:45.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.020+0000 7f412e7fc700 10 monclient: get_auth_request con 0x7f4118035ac0 auth_method 0 2024-09-18T10:25:45.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.024+0000 7f412e7fc700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 0x7f4118037f70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f413809b9a0 tx=0x7f4120007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:45.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.028+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f412801eb40 con 0x7f413809cf50 2024-09-18T10:25:45.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.028+0000 7f412f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:45.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.028+0000 7f412f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f413da17700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f413da17700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f413da17700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41380679e0 con 0x7f413809cf50 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f412f7fe700 1 -- 172.21.15.142:0/4289275009 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f412802c610 con 0x7f413809cf50 2024-09-18T10:25:45.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f412f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:45.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.248+0000 7f412f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:45.258 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":152,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 msgr2=0x7f4118037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 0x7f4118037f70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f413809b9a0 tx=0x7f4120007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 10 monclient: shutdown 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 msgr2=0x7f41381418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f412800d890 tx=0x7f4128008040 comp rx=0 tx=0).stop 2024-09-18T10:25:45.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 shutdown_connections 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4118035ac0 0x7f4118037f70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413809e270 0x7f4138142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f413809d860 0x7f4138141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 --2- 172.21.15.142:0/4289275009 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f413809cf50 0x7f41381418d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 >> 172.21.15.142:0/4289275009 conn(0x7f41380a2660 msgr2=0x7f41380950a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 shutdown_connections 2024-09-18T10:25:45.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:45.252+0000 7f413da17700 1 -- 172.21.15.142:0/4289275009 wait complete. 2024-09-18T10:25:48.271 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:48.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.372+0000 7fdcb341c700 10 monclient: get_monmap_and_config 2024-09-18T10:25:48.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.372+0000 7fdcb341c700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.372+0000 7fdcb341c700 10 monclient: monmap: 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:48.379340+0000 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:48.379340+0000 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:48.382 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 Processor -- start 2024-09-18T10:25:48.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 -- start start 2024-09-18T10:25:48.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient: init 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdcac0a0180 0x7fdcac09ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-c con 0x7fdcac0a0180 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcac0a0550 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-b con 0x7fdcac0a1080 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac09a110 con 0x7fdcac0a0550 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcac09aa20 con 0x7fdcac0a0180 2024-09-18T10:25:48.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac09a6d0 con 0x7fdcac0a1080 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb341c700 10 monclient(hunting): authenticate will time out at 1837.304831s 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54320/0 (socket says 172.21.15.142:54320) 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 -- 172.21.15.142:0/3121197434 learned_addr learned my addr 172.21.15.142:0/3121197434 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): get_auth_request con 0x7fdcac0a1080 auth_method 0 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb241a700 1 --2- 172.21.15.142:0/3121197434 >> v2:172.21.15.150:3300/0 conn(0x7fdcac0a0180 0x7fdcac09ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb241a700 10 monclient(hunting): get_auth_request con 0x7fdcac0a0180 auth_method 0 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb241a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb241a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb241a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient(hunting): handle_auth_done global_id 75704 payload 274 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 -- 172.21.15.142:0/3121197434 >> v2:172.21.15.150:3300/0 conn(0x7fdcac0a0180 msgr2=0x7fdcac09ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 --2- 172.21.15.142:0/3121197434 >> v2:172.21.15.150:3300/0 conn(0x7fdcac0a0180 0x7fdcac09ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 -- 172.21.15.142:0/3121197434 >> v1:172.21.15.150:6789/0 conn(0x7fdcac0a0550 legacy=0x7fdcac0a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 monclient: found mon.noname-b 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.376+0000 7fdcb2c1b700 1 -- 172.21.15.142:0/3121197434 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac09a3f0 con 0x7fdcac0a1080 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb2c1b700 1 -- 172.21.15.142:0/3121197434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 msgr2=0x7fdcac139920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb2c1b700 1 --2- 172.21.15.142:0/3121197434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009f20 tx=0x7fdca4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92dc88a04dac0215 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 1 -- 172.21.15.142:0/3121197434 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400e070 con 0x7fdcac0a1080 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: dump: 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.391 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 5 monclient: authenticate success, global_id 75704 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 1 -- 172.21.15.142:0/3121197434 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca400bc90 con 0x7fdcac0a1080 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 4 set_mon_vals no callback set 2024-09-18T10:25:48.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 1 -- 172.21.15.142:0/3121197434 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca40092e0 con 0x7fdcac0a1080 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb1418700 10 monclient: dump: 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.393 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: shutdown 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- 172.21.15.142:0/3121197434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 msgr2=0x7fdcac139920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- 172.21.15.142:0/3121197434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009f20 tx=0x7fdca4009840 comp rx=0 tx=0).stop 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- 172.21.15.142:0/3121197434 shutdown_connections 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- 172.21.15.142:0/3121197434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a1080 0x7fdcac139920 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- 172.21.15.142:0/3121197434 >> v2:172.21.15.150:3300/0 conn(0x7fdcac0a0180 0x7fdcac09ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- 172.21.15.142:0/3121197434 >> 172.21.15.142:0/3121197434 conn(0x7fdcac0a5790 msgr2=0x7fdcac0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- 172.21.15.142:0/3121197434 shutdown_connections 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- 172.21.15.142:0/3121197434 wait complete. 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: monmap: 2024-09-18T10:25:48.394 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 Processor -- start 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- start start 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: init 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.395 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-a con 0x7fdcac0a0180 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a0550 0x7fdcac08d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-c con 0x7fdcac0a0550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 0x7fdcac08d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): picked mon.noname-b con 0x7fdcac0a1080 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac08e700 con 0x7fdcac0a0180 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0986e0 con 0x7fdcac0a1080 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac09f1b0 con 0x7fdcac0a0550 2024-09-18T10:25:48.396 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb341c700 10 monclient(hunting): authenticate will time out at 1837.311875s 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.380+0000 7fdcb241a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36584/0 (socket says 172.21.15.142:36584) 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 -- 172.21.15.142:0/993519793 learned_addr learned my addr 172.21.15.142:0/993519793 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): get_auth_request con 0x7fdcac0a0180 auth_method 0 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a0550 0x7fdcac08d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.397 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): get_auth_request con 0x7fdcac0a0550 auth_method 0 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 0x7fdcac08d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 10 monclient(hunting): get_auth_request con 0x7fdcac0a1080 auth_method 0 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb1c19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient(hunting): handle_auth_done global_id 184203 payload 824 2024-09-18T10:25:48.398 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 -- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a0550 msgr2=0x7fdcac08d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a0550 0x7fdcac08d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 -- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 msgr2=0x7fdcac08d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 0x7fdcac08d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 monclient: found mon.noname-a 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 -- 172.21.15.142:0/993519793 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009ae0 con 0x7fdcac0a0180 2024-09-18T10:25:48.399 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb2c1b700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 0x7fdcac08d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 -- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 msgr2=0x7fdcac08cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb241a700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdca800d8c0 tx=0x7fdca8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8012800 con 0x7fdcac0a0180 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: dump: 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.402 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8012960 con 0x7fdcac0a0180 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 5 monclient: authenticate success, global_id 184203 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 10 monclient: _renew_subs 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac0901a0 con 0x7fdcac0a0180 2024-09-18T10:25:48.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8012d00 con 0x7fdcac0a0180 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdca27fc700 10 monclient: dump: 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.404 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb0c17700 4 set_mon_vals no callback set 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 10 monclient: _renew_subs 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.384+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac08f240 con 0x7fdcac0a0180 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdca8014640 con 0x7fdcac0a0180 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdca27fc700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 0x7fdc90037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdca8049570 con 0x7fdcac0a0180 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb341c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb341c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb341c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac072a80 con 0x7fdcac0a0180 2024-09-18T10:25:48.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb1c19700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 0x7fdc90037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.388+0000 7fdcb1c19700 10 monclient: get_auth_request con 0x7fdc90035b10 auth_method 0 2024-09-18T10:25:48.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.392+0000 7fdcb1c19700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 0x7fdc90037fc0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdcac098af0 tx=0x7fdc9c007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.392+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdca8011b20 con 0x7fdcac0a0180 2024-09-18T10:25:48.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.392+0000 7fdca27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.392+0000 7fdca27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:48.624 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdcb341c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdcb341c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdcb341c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdcac0676a0 con 0x7fdcac0a0180 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdca27fc700 1 -- 172.21.15.142:0/993519793 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fdca8011340 con 0x7fdcac0a0180 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdca27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:48.625 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.616+0000 7fdca27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:48.626 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 msgr2=0x7fdc90037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 0x7fdc90037fc0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fdcac098af0 tx=0x7fdc9c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 10 monclient: shutdown 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 msgr2=0x7fdcac08cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdca800d8c0 tx=0x7fdca8008040 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 shutdown_connections 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc90035b10 0x7fdc90037fc0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcac0a1080 0x7fdcac08d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcac0a0550 0x7fdcac08d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 --2- 172.21.15.142:0/993519793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcac0a0180 0x7fdcac08cbc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 >> 172.21.15.142:0/993519793 conn(0x7fdcac0a5790 msgr2=0x7fdcac09e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 shutdown_connections 2024-09-18T10:25:48.629 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.620+0000 7fdcb341c700 1 -- 172.21.15.142:0/993519793 wait complete. 2024-09-18T10:25:48.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:25:48.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:25:48.639 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: get_monmap_and_config 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: monmap: 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:48.736678+0000 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:48.736678+0000 2024-09-18T10:25:48.739 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:48.740 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 Processor -- start 2024-09-18T10:25:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 -- start start 2024-09-18T10:25:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: init 2024-09-18T10:25:48.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdefc0999e0 0x7fdefc0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fdefc0999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 0x7fdefc0a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fdefc09c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdefc0a0c00 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdefc0a20c0 con 0x7fdefc0a0c00 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdefc09a000 con 0x7fdefc0999e0 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0a1c80 con 0x7fdefc09c410 2024-09-18T10:25:48.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefbfff700 10 monclient(hunting): authenticate will time out at 1837.662992s 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 0x7fdefc0a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefb7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdefc0a0c00 0x7fdefc138910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38246/0 (socket says 172.21.15.142:38246) 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefb7fe700 1 -- 172.21.15.142:0/834396547 learned_addr learned my addr 172.21.15.142:0/834396547 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request con 0x7fdefc09c410 auth_method 0 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 1 --2- 172.21.15.142:0/834396547 >> v2:172.21.15.150:3300/0 conn(0x7fdefc0999e0 0x7fdefc0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): get_auth_request con 0x7fdefc0999e0 auth_method 0 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_done global_id 75710 payload 274 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 -- 172.21.15.142:0/834396547 >> v2:172.21.15.150:3300/0 conn(0x7fdefc0999e0 msgr2=0x7fdefc0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/834396547 >> v2:172.21.15.150:3300/0 conn(0x7fdefc0999e0 0x7fdefc0a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 -- 172.21.15.142:0/834396547 >> v1:172.21.15.142:6789/0 conn(0x7fdefc0a0c00 legacy=0x7fdefc138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 monclient: found mon.noname-b 2024-09-18T10:25:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.732+0000 7fdefa7fc700 1 -- 172.21.15.142:0/834396547 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc09a580 con 0x7fdefc09c410 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefa7fc700 1 -- 172.21.15.142:0/834396547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 msgr2=0x7fdefc0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/834396547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 0x7fdefc0a06c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bebcf5d3d4727d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 1 -- 172.21.15.142:0/834396547 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee400b850 con 0x7fdefc09c410 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:48.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: dump: 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 1 -- 172.21.15.142:0/834396547 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee400b9b0 con 0x7fdefc09c410 2024-09-18T10:25:48.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 1 -- 172.21.15.142:0/834396547 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee400bd50 con 0x7fdefc09c410 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 5 monclient: authenticate success, global_id 75710 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 4 set_mon_vals no callback set 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdef9ffb700 10 monclient: dump: 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.760 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 10 monclient: shutdown 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 -- 172.21.15.142:0/834396547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 msgr2=0x7fdefc0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 --2- 172.21.15.142:0/834396547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 0x7fdefc0a06c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee40091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 -- 172.21.15.142:0/834396547 shutdown_connections 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 --2- 172.21.15.142:0/834396547 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc09c410 0x7fdefc0a06c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 --2- 172.21.15.142:0/834396547 >> v2:172.21.15.150:3300/0 conn(0x7fdefc0999e0 0x7fdefc0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 -- 172.21.15.142:0/834396547 >> 172.21.15.142:0/834396547 conn(0x7fdefc0a5790 msgr2=0x7fdefc0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 -- 172.21.15.142:0/834396547 shutdown_connections 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.736+0000 7fdefbfff700 1 -- 172.21.15.142:0/834396547 wait complete. 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient: monmap: 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 Processor -- start 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 -- start start 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient: init 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 0x7fdefc08e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-c con 0x7fdefc0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdefc09c410 0x7fdefc08e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-b con 0x7fdefc09c410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc0a0c00 0x7fdefc08eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): picked mon.noname-a con 0x7fdefc0a0c00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc08f8f0 con 0x7fdefc0a0c00 2024-09-18T10:25:48.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc092660 con 0x7fdefc09c410 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc09f480 con 0x7fdefc0999e0 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefbfff700 10 monclient(hunting): authenticate will time out at 1837.671068s 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 0x7fdefc08e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 0x7fdefc08e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54356/0 (socket says 172.21.15.142:54356) 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 -- 172.21.15.142:0/3151522931 learned_addr learned my addr 172.21.15.142:0/3151522931 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): get_auth_request con 0x7fdefc0999e0 auth_method 0 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc0a0c00 0x7fdefc08eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): get_auth_request con 0x7fdefc0a0c00 auth_method 0 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdefc09c410 0x7fdefc08e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request con 0x7fdefc09c410 auth_method 0 2024-09-18T10:25:48.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient(hunting): handle_auth_done global_id 75713 payload 824 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdefc09c410 msgr2=0x7fdefc08e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdefc09c410 0x7fdefc08e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc0a0c00 msgr2=0x7fdefc08eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc0a0c00 0x7fdefc08eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 monclient: found mon.noname-c 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.740+0000 7fdefaffd700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee4009420 con 0x7fdefc0999e0 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefaffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 msgr2=0x7fdefc08e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefaffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 0x7fdefc08e030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00ecb0 tx=0x7fdeec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00cb30 con 0x7fdefc0999e0 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: dump: 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 5 monclient: authenticate success, global_id 75713 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec00cc90 con 0x7fdefc0999e0 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 monclient: _renew_subs 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdefc08fd70 con 0x7fdefc0999e0 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec013470 con 0x7fdefc0999e0 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef2ffd700 10 monclient: dump: 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdef97fa700 4 set_mon_vals no callback set 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 monclient: _renew_subs 2024-09-18T10:25:48.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdefc0993b0 con 0x7fdefc0999e0 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 0x7fdefc06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc0647f0 con 0x7fdefc08eff0 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 0x7fdefc06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient: get_auth_request con 0x7fdefc08eff0 auth_method 0 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.744+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_done global_id 184209 payload 824 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefa7fc700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdefc091a70 con 0x7fdefc08eff0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdeec013860 con 0x7fdefc0999e0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdedc035ac0 0x7fdedc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdeec049300 con 0x7fdefc0999e0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefb7fe700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdedc035ac0 0x7fdedc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefb7fe700 10 monclient: get_auth_request con 0x7fdedc035ac0 auth_method 0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefa7fc700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 msgr2=0x7fdefc06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 0x7fdefc06dad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4011c80 con 0x7fdefc08eff0 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 10 monclient: dump: 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.771 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fdee4011de0 con 0x7fdefc08eff0 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 msgr2=0x7fdefc06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 0x7fdefc06dad0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee4016000 comp rx=0 tx=0).stop 2024-09-18T10:25:48.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 msgr2=0x7fdefc06dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:48.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdef2ffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc08eff0 0x7fdefc06dad0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.748+0000 7fdefb7fe700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdedc035ac0 0x7fdedc037f70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0901c0 tx=0x7fdef4005340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefbfff700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 0x7fdefc06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefbfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc072a80 con 0x7fdefc06e010 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 0x7fdefc06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient: get_auth_request con 0x7fdefc06e010 auth_method 0 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 10 monclient(hunting): handle_auth_done global_id 184212 payload 824 2024-09-18T10:25:48.775 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 1 -- 172.21.15.142:0/3151522931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdefc067800 con 0x7fdefc06e010 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 msgr2=0x7fdefc06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdefa7fc700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 0x7fdefc06e410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdee40030c0 tx=0x7fdee401f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee401cb90 con 0x7fdefc06e010 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 10 monclient: dump: 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.776 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fdee4011a70 con 0x7fdefc06e010 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 msgr2=0x7fdefc06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.760+0000 7fdef2ffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 0x7fdefc06e410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdee40030c0 tx=0x7fdee401f040 comp rx=0 tx=0).stop 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdef2ffd700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 msgr2=0x7fdefc06e410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdef2ffd700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdefc06e010 0x7fdefc06e410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.777 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:48.778 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:48.779 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:48.780 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:48.781 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.782 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.783 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:48.784 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.785 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.786 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.787 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.788 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:48.789 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdedc035ac0 msgr2=0x7fdedc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdedc035ac0 0x7fdedc037f70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0901c0 tx=0x7fdef4005340 comp rx=0 tx=0).stop 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 10 monclient: shutdown 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 msgr2=0x7fdefc08e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 --2- 172.21.15.142:0/3151522931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdefc0999e0 0x7fdefc08e030 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00ecb0 tx=0x7fdeec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefb7fe700 1 -- 172.21.15.142:0/3151522931 reap_dead start 2024-09-18T10:25:48.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefb7fe700 1 -- 172.21.15.142:0/3151522931 reap_dead start 2024-09-18T10:25:48.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 shutdown_connections 2024-09-18T10:25:48.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 >> 172.21.15.142:0/3151522931 conn(0x7fdefc0a5790 msgr2=0x7fdefc09d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 shutdown_connections 2024-09-18T10:25:48.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.764+0000 7fdefbfff700 1 -- 172.21.15.142:0/3151522931 wait complete. 2024-09-18T10:25:48.792 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:48.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:25:48.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:25:48.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: get_monmap_and_config 2024-09-18T10:25:48.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: monmap: 2024-09-18T10:25:48.902 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.902 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:48.900208+0000 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:48.900208+0000 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:48.903 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 Processor -- start 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 -- start start 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: init 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T10:25:48.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 0x7f7c0c0989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-b con 0x7f7c0c06f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c0c098f00 0x7f7c0c0992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-c con 0x7f7c0c098f00 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c0c133660 0x7f7c0c135a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c133660 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c09b030 con 0x7f7c0c133660 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c099830 con 0x7f7c0c098f00 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c09c330 con 0x7f7c0c06f3b0 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c13061700 10 monclient(hunting): authenticate will time out at 1837.825719s 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c0c133660 0x7f7c0c135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 0x7f7c0c0989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c0c133660 0x7f7c0c135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36626/0 (socket says 172.21.15.142:36626) 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 -- 172.21.15.142:0/1849732864 learned_addr learned my addr 172.21.15.142:0/1849732864 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): get_auth_request con 0x7f7c0c06f3b0 auth_method 0 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): get_auth_request con 0x7f7c0c133660 auth_method 0 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 1 --2- 172.21.15.142:0/1849732864 >> v2:172.21.15.150:3300/0 conn(0x7f7c0c098f00 0x7f7c0c0992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): get_auth_request con 0x7f7c0c098f00 auth_method 0 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient(hunting): handle_auth_done global_id 184215 payload 274 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 -- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 msgr2=0x7f7c0c0989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 0x7f7c0c0989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 -- 172.21.15.142:0/1849732864 >> v2:172.21.15.150:3300/0 conn(0x7f7c0c098f00 msgr2=0x7f7c0c0992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 --2- 172.21.15.142:0/1849732864 >> v2:172.21.15.150:3300/0 conn(0x7f7c0c098f00 0x7f7c0c0992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 monclient: found mon.noname-a 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c12860700 1 -- 172.21.15.142:0/1849732864 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c09c530 con 0x7f7c0c133660 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.895+0000 7f7c1205f700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 0x7f7c0c0989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c12860700 1 -- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 msgr2=0x7f7c0c135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c12860700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c135a40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c069270 tx=0x7f7c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7283e4c698337296 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 1 -- 172.21.15.142:0/1849732864 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400e070 con 0x7f7c0c133660 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: dump: 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.912 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 5 monclient: authenticate success, global_id 184215 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 1 -- 172.21.15.142:0/1849732864 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400bb80 con 0x7f7c0c133660 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 4 set_mon_vals no callback set 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 1 -- 172.21.15.142:0/1849732864 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040092e0 con 0x7f7c0c133660 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:48.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c1105d700 10 monclient: dump: 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 10 monclient: shutdown 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 msgr2=0x7f7c0c135a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c135a40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c069270 tx=0x7f7c04009840 comp rx=0 tx=0).stop 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- 172.21.15.142:0/1849732864 shutdown_connections 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c135a40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 --2- 172.21.15.142:0/1849732864 >> v2:172.21.15.150:3300/0 conn(0x7f7c0c098f00 0x7f7c0c0992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 --2- 172.21.15.142:0/1849732864 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c06f3b0 0x7f7c0c0989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- 172.21.15.142:0/1849732864 >> 172.21.15.142:0/1849732864 conn(0x7f7c0c070ef0 msgr2=0x7f7c0c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- 172.21.15.142:0/1849732864 shutdown_connections 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- 172.21.15.142:0/1849732864 wait complete. 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 10 monclient: build_initial_monmap 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 10 monclient: monmap: 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:48.922 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 Processor -- start 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 1 -- start start 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.899+0000 7f7c13061700 10 monclient: init 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c0c06f3b0 0x7f7c0c132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-b con 0x7f7c0c06f3b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 0x7f7c0c06a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-c con 0x7f7c0c098f00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c133660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c12d0e0 con 0x7f7c0c133660 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c12e440 con 0x7f7c0c06f3b0 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c06fa20 con 0x7f7c0c098f00 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c13061700 10 monclient(hunting): authenticate will time out at 1837.832439s 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c06a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36636/0 (socket says 172.21.15.142:36636) 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c0c06f3b0 0x7f7c0c132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 0x7f7c0c06a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 1 -- 172.21.15.142:0/2933039687 learned_addr learned my addr 172.21.15.142:0/2933039687 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): get_auth_request con 0x7f7c0c133660 auth_method 0 2024-09-18T10:25:48.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): get_auth_request con 0x7f7c0c06f3b0 auth_method 0 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): get_auth_request con 0x7f7c0c098f00 auth_method 0 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient(hunting): handle_auth_done global_id 75719 payload 824 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c0c06f3b0 msgr2=0x7f7c0c132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c0c06f3b0 0x7f7c0c132130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 msgr2=0x7f7c0c06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c06a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient: _finish_hunting 0 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 monclient: found mon.noname-c 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:48.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08009420 con 0x7f7c0c098f00 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1205f700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c0c06f3b0 0x7f7c0c132130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c12860700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c0c133660 0x7f7c0c06a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 msgr2=0x7f7c0c06a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.903+0000 7f7c1185e700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 0x7f7c0c06a360 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00ea80 tx=0x7f7bfc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00eec0 con 0x7f7c0c098f00 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: dump: 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.927 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 5 monclient: authenticate success, global_id 75719 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc00ca00 con 0x7f7c0c098f00 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 monclient: _renew_subs 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c04009ae0 con 0x7f7c0c098f00 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:48.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00cd80 con 0x7f7c0c098f00 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c03fff700 4 set_mon_vals no callback set 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 10 monclient: dump: 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.929 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 monclient: _renew_subs 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c12ee20 con 0x7f7c0c098f00 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 0x7f7c0c06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0648a0 con 0x7f7c0c12d270 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 0x7f7c0c06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient: get_auth_request con 0x7f7c0c12d270 auth_method 0 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 10 monclient(hunting): handle_auth_done global_id 75722 payload 824 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c1205f700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c0c12e2a0 con 0x7f7c0c12d270 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7bfc013680 con 0x7f7c0c098f00 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7bf0035ac0 0x7f7bf0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7bfc0491c0 con 0x7f7c0c098f00 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c12860700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7bf0035ac0 0x7f7bf0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.907+0000 7f7c12860700 10 monclient: get_auth_request con 0x7f7bf0035ac0 auth_method 0 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c1205f700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 msgr2=0x7f7c0c06ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c1205f700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 0x7f7c0c06ade0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800f4a0 tx=0x7f7c080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c08014860 con 0x7f7c0c12d270 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 10 monclient: dump: 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7c080149c0 con 0x7f7c0c12d270 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:48.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 msgr2=0x7f7c0c06ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 0x7f7c0c06ade0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7c0800f4a0 tx=0x7f7c080091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 msgr2=0x7f7c0c06ade0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c027fc700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c12d270 0x7f7c0c06ade0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.911+0000 7f7c12860700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7bf0035ac0 0x7f7bf0037f70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7c04012000 tx=0x7f7c04018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c13061700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c13061700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 0x7f7c0c1269b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c13061700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c12e640 con 0x7f7c0c1265b0 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 0x7f7c0c1269b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient: get_auth_request con 0x7f7c0c1265b0 auth_method 0 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:48.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.919+0000 7f7c1205f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c1205f700 10 monclient(hunting): handle_auth_done global_id 75725 payload 824 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c1205f700 1 -- 172.21.15.142:0/2933039687 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7c0c072a80 con 0x7f7c0c1265b0 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c1205f700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 msgr2=0x7f7c0c1269b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c1205f700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 0x7f7c0c1269b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7c0801b000 tx=0x7f7c0801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0801ca00 con 0x7f7c0c1265b0 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 10 monclient: dump: 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:48.936 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f7c08014650 con 0x7f7c0c1265b0 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 msgr2=0x7f7c0c1269b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 0x7f7c0c1269b0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7c0801b000 tx=0x7f7c0801c000 comp rx=0 tx=0).stop 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 msgr2=0x7f7c0c1269b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:48.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c027fc700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c1265b0 0x7f7c0c1269b0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:48.938 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.939 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.940 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:48.941 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:48.942 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.943 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.944 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.945 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:48.946 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:48.947 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.948 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:48.949 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7bf0035ac0 msgr2=0x7f7bf0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7bf0035ac0 0x7f7bf0037f70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7c04012000 tx=0x7f7c04018000 comp rx=0 tx=0).stop 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 10 monclient: shutdown 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 msgr2=0x7f7c0c06a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:48.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c12860700 1 -- 172.21.15.142:0/2933039687 reap_dead start 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.923+0000 7f7c13061700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 0x7f7c0c06a360 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00ea80 tx=0x7f7bfc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.927+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 shutdown_connections 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.927+0000 7f7c13061700 1 --2- 172.21.15.142:0/2933039687 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c0c098f00 0x7f7c0c06a360 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.927+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 >> 172.21.15.142:0/2933039687 conn(0x7f7c0c070ef0 msgr2=0x7f7c0c09f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.927+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 shutdown_connections 2024-09-18T10:25:48.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:48.927+0000 7f7c13061700 1 -- 172.21.15.142:0/2933039687 wait complete. 2024-09-18T10:25:48.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:48.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:25:48.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: get_monmap_and_config 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: build_initial_monmap 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: monmap: 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:49.058039+0000 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:49.058039+0000 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:49.061 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 Processor -- start 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 -- start start 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: init 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b2c09a6a0 0x7f0b2c06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c09a6a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b2c09aa70 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b2c09b580 0x7f0b2c13d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c09b580 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c09dae0 con 0x7f0b2c09aa70 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c099bc0 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0998e0 con 0x7f0b2c09b580 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31e58700 10 monclient(hunting): authenticate will time out at 1837.983510s 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0b2c09a6a0 0x7f0b2c06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b2bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0b2c09aa70 0x7f0b2c09ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38252/0 (socket says 172.21.15.142:38252) 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1242477702 learned_addr learned my addr 172.21.15.142:0/1242477702 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): get_auth_request con 0x7f0b2c09a6a0 auth_method 0 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31657700 1 --2- 172.21.15.142:0/1242477702 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c09b580 0x7f0b2c13d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31657700 10 monclient(hunting): get_auth_request con 0x7f0b2c09b580 auth_method 0 2024-09-18T10:25:49.067 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.051+0000 7f0b31657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 10 monclient(hunting): handle_auth_done global_id 184221 payload 274 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 -- 172.21.15.142:0/1242477702 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c09b580 msgr2=0x7f0b2c13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:49.068 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 --2- 172.21.15.142:0/1242477702 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c09b580 0x7f0b2c13d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 -- 172.21.15.142:0/1242477702 >> v1:172.21.15.142:6789/0 conn(0x7f0b2c09aa70 legacy=0x7f0b2c09ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 10 monclient: _finish_hunting 0 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 monclient: found mon.noname-a 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 -- 172.21.15.142:0/1242477702 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c09bed0 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 -- 172.21.15.142:0/1242477702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 msgr2=0x7f0b2c06f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b30e56700 1 --2- 172.21.15.142:0/1242477702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06f7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c099f00 tx=0x7f0b180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d904cd4a31e882 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 1 -- 172.21.15.142:0/1242477702 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18011840 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: dump: 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.078 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 1 -- 172.21.15.142:0/1242477702 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b180119a0 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31e58700 5 monclient: authenticate success, global_id 184221 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31e58700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31e58700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b31e58700 4 set_mon_vals no callback set 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 1 -- 172.21.15.142:0/1242477702 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18011d40 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.055+0000 7f0b2b7fe700 10 monclient: dump: 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.079 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 10 monclient: shutdown 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- 172.21.15.142:0/1242477702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 msgr2=0x7f0b2c06f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1242477702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06f7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c099f00 tx=0x7f0b180091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- 172.21.15.142:0/1242477702 shutdown_connections 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1242477702 >> v2:172.21.15.150:3300/0 conn(0x7f0b2c09b580 0x7f0b2c13d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1242477702 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06f7f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- 172.21.15.142:0/1242477702 >> 172.21.15.142:0/1242477702 conn(0x7f0b2c070ef0 msgr2=0x7f0b2c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- 172.21.15.142:0/1242477702 shutdown_connections 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- 172.21.15.142:0/1242477702 wait complete. 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 10 monclient: build_initial_monmap 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 10 monclient: monmap: 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.081 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 Processor -- start 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 1 -- start start 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.059+0000 7f0b31e58700 10 monclient: init 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c09a6a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b2c09aa70 0x7f0b2c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c09aa70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c09b580 0x7f0b2c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c09b580 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c094830 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c126df0 con 0x7f0b2c09b580 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0a0690 con 0x7f0b2c09aa70 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31e58700 10 monclient(hunting): authenticate will time out at 1837.992614s 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b2c09aa70 0x7f0b2c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36654/0 (socket says 172.21.15.142:36654) 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 -- 172.21.15.142:0/1158366825 learned_addr learned my addr 172.21.15.142:0/1158366825 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): get_auth_request con 0x7f0b2c09a6a0 auth_method 0 2024-09-18T10:25:49.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): get_auth_request con 0x7f0b2c09aa70 auth_method 0 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c09b580 0x7f0b2c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): get_auth_request con 0x7f0b2c09b580 auth_method 0 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b31657700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient(hunting): handle_auth_done global_id 184224 payload 824 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b2c09aa70 msgr2=0x7f0b2c06c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b2c09aa70 0x7f0b2c06c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c09b580 msgr2=0x7f0b2c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c09b580 0x7f0b2c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient: _finish_hunting 0 2024-09-18T10:25:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 monclient: found mon.noname-a 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18009420 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 msgr2=0x7f0b2c06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b30e56700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c071df0 tx=0x7f0b18016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18006af0 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.063+0000 7f0b28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: dump: 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: _finish_auth 0 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 5 monclient: authenticate success, global_id 184224 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1800bc20 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 monclient: _renew_subs 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c06e010 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1801a460 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 10 monclient: dump: 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.088 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2affd700 4 set_mon_vals no callback set 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 monclient: _renew_subs 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c1267c0 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0648a0 con 0x7f0b2c06cfc0 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b1801a850 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 msgr2=0x7f0b2c092800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 msgr2=0x7f0b2c092800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:25:49.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:49.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035af0 0x7f0b10037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0b1804a860 con 0x7f0b2c09a6a0 2024-09-18T10:25:49.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31657700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035af0 0x7f0b10037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.067+0000 7f0b31657700 10 monclient: get_auth_request con 0x7f0b10035af0 auth_method 0 2024-09-18T10:25:49.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.071+0000 7f0b31657700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035af0 0x7f0b10037fa0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009fd0 tx=0x7f0b2400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient: get_auth_request con 0x7f0b2c06cfc0 auth_method 0 2024-09-18T10:25:49.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_done global_id 194122 payload 824 2024-09-18T10:25:49.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b2c126bc0 con 0x7f0b2c06cfc0 2024-09-18T10:25:49.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 msgr2=0x7f0b2c092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000fb00 tx=0x7f0b20007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000db30 con 0x7f0b2c06cfc0 2024-09-18T10:25:49.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 10 monclient: dump: 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0b2000dc90 con 0x7f0b2c06cfc0 2024-09-18T10:25:49.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 msgr2=0x7f0b2c092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000fb00 tx=0x7f0b20007f90 comp rx=0 tx=0).stop 2024-09-18T10:25:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 msgr2=0x7f0b2c092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:49.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.271+0000 7f0b28ff9700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c06cfc0 0x7f0b2c092800 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000fb00 tx=0x7f0b20007f90 comp rx=0 tx=0).stop 2024-09-18T10:25:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.283+0000 7f0b31e58700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.283+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 0x7f0b2c093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.283+0000 7f0b31e58700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.283+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c072a80 con 0x7f0b2c092d40 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.283+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 0x7f0b2c093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient: get_auth_request con 0x7f0b2c092d40 auth_method 0 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 10 monclient(hunting): handle_auth_done global_id 194125 payload 824 2024-09-18T10:25:49.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b2c06d390 con 0x7f0b2c092d40 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 msgr2=0x7f0b2c093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b2bfff700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 0x7f0b2c093140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b20017000 tx=0x7f0b20018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2001cb80 con 0x7f0b2c092d40 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b28ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.287+0000 7f0b28ff9700 10 monclient: dump: 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.297 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.298 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0b2001cce0 con 0x7f0b2c092d40 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 msgr2=0x7f0b2c093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 0x7f0b2c093140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b20017000 tx=0x7f0b20018000 comp rx=0 tx=0).stop 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 msgr2=0x7f0b2c093140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b28ff9700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b2c092d40 0x7f0b2c093140 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:49.299 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:49.300 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:49.301 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:49.302 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:49.303 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.304 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:49.305 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:49.306 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.307 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.308 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:49.309 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:49.310 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:49.311 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:49.312 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:49.312 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035af0 msgr2=0x7f0b10037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b10035af0 0x7f0b10037fa0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0b24009fd0 tx=0x7f0b2400b040 comp rx=0 tx=0).stop 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 10 monclient: shutdown 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 msgr2=0x7f0b2c06c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31657700 1 -- 172.21.15.142:0/1158366825 reap_dead start 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c071df0 tx=0x7f0b18016000 comp rx=0 tx=0).stop 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 shutdown_connections 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 --2- 172.21.15.142:0/1158366825 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b2c09a6a0 0x7f0b2c06c000 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 >> 172.21.15.142:0/1158366825 conn(0x7f0b2c070ef0 msgr2=0x7f0b2c09f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 shutdown_connections 2024-09-18T10:25:49.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.291+0000 7f0b31e58700 1 -- 172.21.15.142:0/1158366825 wait complete. 2024-09-18T10:25:49.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:49.314 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:25:49.315 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: get_monmap_and_config 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: build_initial_monmap 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: monmap: 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:49.424938+0000 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:49.424938+0000 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:49.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 Processor -- start 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 -- start start 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: init 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:25:49.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe33009a7f0 0x7fe330098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-a con 0x7fe33009a7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 0x7fe33009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-b con 0x7fe330099220 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe33009c320 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33009abc0 con 0x7fe33009c320 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33009d950 con 0x7fe33009a7f0 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33009db90 con 0x7fe330099220 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe3353e9700 10 monclient(hunting): authenticate will time out at 1838.350619s 2024-09-18T10:25:49.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe33009a7f0 0x7fe330098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 0x7fe33009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 0x7fe33009bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54418/0 (socket says 172.21.15.142:54418) 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 1 -- 172.21.15.142:0/1954601550 learned_addr learned my addr 172.21.15.142:0/1954601550 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe33009a7f0 0x7fe330098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36668/0 (socket says 172.21.15.142:36668) 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): get_auth_request con 0x7fe330099220 auth_method 0 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): get_auth_request con 0x7fe33009a7f0 auth_method 0 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient(hunting): handle_auth_done global_id 184227 payload 274 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 -- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 msgr2=0x7fe33009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 --2- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 0x7fe33009bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 -- 172.21.15.142:0/1954601550 >> v1:172.21.15.150:6789/0 conn(0x7fe33009c320 legacy=0x7fe33013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 monclient: found mon.noname-a 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:49.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.419+0000 7fe32ffff700 1 -- 172.21.15.142:0/1954601550 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330099a30 con 0x7fe33009a7f0 2024-09-18T10:25:49.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32ffff700 1 -- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe33009a7f0 msgr2=0x7fe330098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32ffff700 1 --2- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe33009a7f0 0x7fe330098ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098e0 tx=0x7fe3200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ed801fb73988d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 1 -- 172.21.15.142:0/1954601550 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32000b850 con 0x7fe33009a7f0 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: dump: 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.439 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: _finish_auth 0 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 1 -- 172.21.15.142:0/1954601550 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32000b9b0 con 0x7fe33009a7f0 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 5 monclient: authenticate success, global_id 184227 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 4 set_mon_vals no callback set 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 1 -- 172.21.15.142:0/1954601550 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32000bd50 con 0x7fe33009a7f0 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:49.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe32effd700 10 monclient: dump: 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 10 monclient: shutdown 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 -- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe33009a7f0 msgr2=0x7fe330098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 --2- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe33009a7f0 0x7fe330098ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe3200098e0 tx=0x7fe3200091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 -- 172.21.15.142:0/1954601550 shutdown_connections 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 --2- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe330099220 0x7fe33009bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 --2- 172.21.15.142:0/1954601550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe33009a7f0 0x7fe330098ce0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 -- 172.21.15.142:0/1954601550 >> 172.21.15.142:0/1954601550 conn(0x7fe3300a1fe0 msgr2=0x7fe330070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 -- 172.21.15.142:0/1954601550 shutdown_connections 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.423+0000 7fe3353e9700 1 -- 172.21.15.142:0/1954601550 wait complete. 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient: build_initial_monmap 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient: monmap: 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:49.442 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 Processor -- start 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 -- start start 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient: init 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe330099220 0x7fe330093d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-b con 0x7fe330099220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 0x7fe33006e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-c con 0x7fe33009c320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): picked mon.noname-a con 0x7fe330146250 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33006f970 con 0x7fe330146250 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330097670 con 0x7fe330099220 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33006cb60 con 0x7fe33009c320 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe3353e9700 10 monclient(hunting): authenticate will time out at 1838.357945s 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 0x7fe33006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36682/0 (socket says 172.21.15.142:36682) 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 1 -- 172.21.15.142:0/943713202 learned_addr learned my addr 172.21.15.142:0/943713202 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): get_auth_request con 0x7fe330146250 auth_method 0 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): get_auth_request con 0x7fe33009c320 auth_method 0 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe330099220 0x7fe330093d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): get_auth_request con 0x7fe330099220 auth_method 0 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient(hunting): handle_auth_done global_id 75734 payload 824 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 -- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe330099220 msgr2=0x7fe330093d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe330099220 0x7fe330093d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 -- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 msgr2=0x7fe33006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe32f7fe700 1 -- 172.21.15.142:0/943713202 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320009420 con 0x7fe33009c320 2024-09-18T10:25:49.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.427+0000 7fe334be8700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:49.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe32f7fe700 1 -- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 msgr2=0x7fe33006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:49.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe32f7fe700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 0x7fe33006e930 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe324006fd0 tx=0x7fe324009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324013890 con 0x7fe33009c320 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: dump: 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.450 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: _finish_auth 0 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3240139f0 con 0x7fe33009c320 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 5 monclient: authenticate success, global_id 75734 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _renew_subs 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33006ce40 con 0x7fe33009c320 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe324013d70 con 0x7fe33009c320 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:49.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 10 monclient: dump: 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _renew_subs 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe32e7fc700 4 set_mon_vals no callback set 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33006bc00 con 0x7fe33009c320 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe3240125e0 con 0x7fe33009c320 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 0x7fe31c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe3240490b0 con 0x7fe33009c320 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.431+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33013a640 con 0x7fe33009c320 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.435+0000 7fe32ffff700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 0x7fe31c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.435+0000 7fe32ffff700 10 monclient: get_auth_request con 0x7fe31c035ac0 auth_method 0 2024-09-18T10:25:49.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.435+0000 7fe32ffff700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 0x7fe31c037f70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe33004d340 tx=0x7fe320016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:49.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.443+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe33013a640 con 0x7fe33009c320 2024-09-18T10:25:49.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.443+0000 7fe317fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:49.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.443+0000 7fe317fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:49.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.671+0000 7fe3353e9700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:25:49.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.671+0000 7fe3353e9700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:25:49.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.671+0000 7fe3353e9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:49.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.671+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fe33013a640 con 0x7fe33009c320 2024-09-18T10:25:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe317fff700 1 -- 172.21.15.142:0/943713202 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fe33013a640 con 0x7fe33009c320 2024-09-18T10:25:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe317fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:25:49.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe317fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 msgr2=0x7fe31c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 0x7fe31c037f70 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe33004d340 tx=0x7fe320016000 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 10 monclient: shutdown 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 msgr2=0x7fe33006e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 0x7fe33006e930 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe324006fd0 tx=0x7fe324009250 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 shutdown_connections 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe31c035ac0 0x7fe31c037f70 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe330146250 0x7fe33006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe33009c320 0x7fe33006e930 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 --2- 172.21.15.142:0/943713202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe330099220 0x7fe330093d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 >> 172.21.15.142:0/943713202 conn(0x7fe3300a1fe0 msgr2=0x7fe330139760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 shutdown_connections 2024-09-18T10:25:49.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:49.679+0000 7fe3353e9700 1 -- 172.21.15.142:0/943713202 wait complete. 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=2463019480,config=3140514009,health=2704546950,kv=4095092511,logm=1606861544}) 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=8,logm=51} crc {auth=1452192770,config=3140514009,health=517958292,kv=4261215878,logm=1392351856}) 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=2463019480,config=3140514009,health=2704546950,kv=4095092511,logm=1606861544}) 2024-09-18T10:25:49.693 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:49.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=7,logm=50} crc {auth=3967067926,config=3140514009,health=3179124566,kv=264744547,logm=325453031}) 2024-09-18T10:25:49.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:25:50.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153352136}) 2024-09-18T10:25:50.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942983963}) 2024-09-18T10:25:50.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153352136}) 2024-09-18T10:25:50.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.127+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259081495}) 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272702070}) 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277850216}) 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272702070}) 2024-09-18T10:25:50.139 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.131+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1637249184}) 2024-09-18T10:25:50.142 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.142 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741241085}) 2024-09-18T10:25:50.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001767015}) 2024-09-18T10:25:50.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741241085}) 2024-09-18T10:25:50.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1087906925}) 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727055767}) 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262669765}) 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727055767}) 2024-09-18T10:25:50.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.135+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802940496}) 2024-09-18T10:25:50.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446863730}) 2024-09-18T10:25:50.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725959516}) 2024-09-18T10:25:50.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446863730}) 2024-09-18T10:25:50.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621496595}) 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298882448}) 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263205563}) 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298882448}) 2024-09-18T10:25:50.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.139+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183228833}) 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255301574}) 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902739340}) 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255301574}) 2024-09-18T10:25:50.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846286343}) 2024-09-18T10:25:50.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416868473}) 2024-09-18T10:25:50.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015205040}) 2024-09-18T10:25:50.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416868473}) 2024-09-18T10:25:50.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810220540}) 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350267061}) 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583085425}) 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350267061}) 2024-09-18T10:25:50.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.143+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926265216}) 2024-09-18T10:25:50.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188703783}) 2024-09-18T10:25:50.154 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764245310}) 2024-09-18T10:25:50.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188703783}) 2024-09-18T10:25:50.155 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126517984}) 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869948483}) 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181280363}) 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869948483}) 2024-09-18T10:25:50.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278019570}) 2024-09-18T10:25:50.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017237627}) 2024-09-18T10:25:50.157 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3867009610}) 2024-09-18T10:25:50.158 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.158 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3017237627}) 2024-09-18T10:25:50.158 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.147+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958906534}) 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692122717}) 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=95321617}) 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1692122717}) 2024-09-18T10:25:50.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488984896}) 2024-09-18T10:25:50.160 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.160 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434068327}) 2024-09-18T10:25:50.160 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281171420}) 2024-09-18T10:25:50.160 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.161 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434068327}) 2024-09-18T10:25:50.161 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.151+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505137398}) 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821438364}) 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3857797225}) 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821438364}) 2024-09-18T10:25:50.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609286642}) 2024-09-18T10:25:50.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837220903}) 2024-09-18T10:25:50.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771757129}) 2024-09-18T10:25:50.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837220903}) 2024-09-18T10:25:50.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680630788}) 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961653416}) 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241038115}) 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3961653416}) 2024-09-18T10:25:50.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.155+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155231109}) 2024-09-18T10:25:50.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895990160}) 2024-09-18T10:25:50.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202058006}) 2024-09-18T10:25:50.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895990160}) 2024-09-18T10:25:50.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3434765246}) 2024-09-18T10:25:50.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771130003}) 2024-09-18T10:25:50.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491750039}) 2024-09-18T10:25:50.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771130003}) 2024-09-18T10:25:50.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3664825516}) 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55468086}) 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=381088868}) 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55468086}) 2024-09-18T10:25:50.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.159+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803985596}) 2024-09-18T10:25:50.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309696973}) 2024-09-18T10:25:50.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=6677643}) 2024-09-18T10:25:50.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309696973}) 2024-09-18T10:25:50.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316844685}) 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863516906}) 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155757225}) 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863516906}) 2024-09-18T10:25:50.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3851749232}) 2024-09-18T10:25:50.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=675092898}) 2024-09-18T10:25:50.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148563681}) 2024-09-18T10:25:50.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=675092898}) 2024-09-18T10:25:50.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.163+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3041621022}) 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973851515}) 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=226044014}) 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973851515}) 2024-09-18T10:25:50.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677459668}) 2024-09-18T10:25:50.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147272899}) 2024-09-18T10:25:50.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26612948}) 2024-09-18T10:25:50.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147272899}) 2024-09-18T10:25:50.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.167+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803713180}) 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377749383}) 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884894288}) 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377749383}) 2024-09-18T10:25:50.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443285210}) 2024-09-18T10:25:50.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888055966}) 2024-09-18T10:25:50.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030188630}) 2024-09-18T10:25:50.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888055966}) 2024-09-18T10:25:50.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497080165}) 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891861484}) 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087852546}) 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891861484}) 2024-09-18T10:25:50.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.171+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436765826}) 2024-09-18T10:25:50.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281846886}) 2024-09-18T10:25:50.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014432469}) 2024-09-18T10:25:50.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281846886}) 2024-09-18T10:25:50.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863268309}) 2024-09-18T10:25:50.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127493043}) 2024-09-18T10:25:50.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364009875}) 2024-09-18T10:25:50.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127493043}) 2024-09-18T10:25:50.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.175+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264577076}) 2024-09-18T10:25:50.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64555068}) 2024-09-18T10:25:50.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239786398}) 2024-09-18T10:25:50.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64555068}) 2024-09-18T10:25:50.187 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457747147}) 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308536958}) 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3571067446}) 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2308536958}) 2024-09-18T10:25:50.188 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1113595061}) 2024-09-18T10:25:50.189 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.189 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568323168}) 2024-09-18T10:25:50.189 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937418245}) 2024-09-18T10:25:50.190 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.190 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568323168}) 2024-09-18T10:25:50.190 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.179+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717634750}) 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399400999}) 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697221735}) 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399400999}) 2024-09-18T10:25:50.191 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201033678}) 2024-09-18T10:25:50.192 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.193 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074922425}) 2024-09-18T10:25:50.193 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609509814}) 2024-09-18T10:25:50.193 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.193 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074922425}) 2024-09-18T10:25:50.193 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.183+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343214162}) 2024-09-18T10:25:50.194 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.194 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533163865}) 2024-09-18T10:25:50.194 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264756532}) 2024-09-18T10:25:50.194 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.195 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533163865}) 2024-09-18T10:25:50.195 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547767977}) 2024-09-18T10:25:50.195 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.195 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852360964}) 2024-09-18T10:25:50.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468659717}) 2024-09-18T10:25:50.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2852360964}) 2024-09-18T10:25:50.196 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2177324256}) 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373936891}) 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=362375053}) 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373936891}) 2024-09-18T10:25:50.197 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604995105}) 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559382301}) 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024606172}) 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.187+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559382301}) 2024-09-18T10:25:50.198 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954532302}) 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687806437}) 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526282076}) 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1687806437}) 2024-09-18T10:25:50.199 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456054471}) 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953778333}) 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420242329}) 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953778333}) 2024-09-18T10:25:50.200 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178598298}) 2024-09-18T10:25:50.201 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.201 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477898242}) 2024-09-18T10:25:50.201 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2428815276}) 2024-09-18T10:25:50.201 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.201 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477898242}) 2024-09-18T10:25:50.202 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.191+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360034645}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563611402}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356769887}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563611402}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=359409525}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234488684}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195789143}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234488684}) 2024-09-18T10:25:50.203 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210612910}) 2024-09-18T10:25:50.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228432823}) 2024-09-18T10:25:50.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393210485}) 2024-09-18T10:25:50.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.204 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228432823}) 2024-09-18T10:25:50.205 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739736709}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981086138}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376085073}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981086138}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.195+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308446151}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764570565}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=938756218}) 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.206 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764570565}) 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390819561}) 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052112112}) 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505443639}) 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052112112}) 2024-09-18T10:25:50.207 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=405832276}) 2024-09-18T10:25:50.208 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.208 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165479699}) 2024-09-18T10:25:50.209 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197358751}) 2024-09-18T10:25:50.209 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.209 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165479699}) 2024-09-18T10:25:50.209 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123427626}) 2024-09-18T10:25:50.210 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.210 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539085099}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88629793}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539085099}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.199+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3816884479}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681713669}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1916382934}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681713669}) 2024-09-18T10:25:50.211 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318425763}) 2024-09-18T10:25:50.213 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035484493}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2358913055}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035484493}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982958932}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415702514}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36898389}) 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.214 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415702514}) 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402407057}) 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600177391}) 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73213442}) 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600177391}) 2024-09-18T10:25:50.215 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.203+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775026989}) 2024-09-18T10:25:50.217 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.217 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847938992}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159258678}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847938992}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1268733473}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462926551}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112471678}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462926551}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140064061}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026165110}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2847641276}) 2024-09-18T10:25:50.218 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026165110}) 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700135101}) 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611189532}) 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281186408}) 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611189532}) 2024-09-18T10:25:50.219 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.207+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1980189773}) 2024-09-18T10:25:50.228 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.228 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431993390}) 2024-09-18T10:25:50.228 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948495633}) 2024-09-18T10:25:50.228 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.229 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431993390}) 2024-09-18T10:25:50.229 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.219+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3800813782}) 2024-09-18T10:25:50.231 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.231 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888306366}) 2024-09-18T10:25:50.232 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259924794}) 2024-09-18T10:25:50.232 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.232 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888306366}) 2024-09-18T10:25:50.232 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.223+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939040673}) 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438975149}) 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177349374}) 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438975149}) 2024-09-18T10:25:50.694 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.683+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838194187}) 2024-09-18T10:25:50.703 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:25:50.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199674545}) 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083521607}) 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199674545}) 2024-09-18T10:25:50.717 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.707+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707357494}) 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 10 monclient: get_monmap_and_config 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 10 monclient: build_initial_monmap 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 10 monclient: monmap: 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:50.794405+0000 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:50.794405+0000 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.797 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 1 Processor -- start 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 1 -- start start 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.787+0000 7fa6a9900700 10 monclient: init 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T10:25:50.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6a40a2700 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6a4097ad0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a41374c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a409e290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a409b910 con 0x7fa6a4097ad0 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a40a2ad0 con 0x7fa6a40a2700 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a4098730 con 0x7fa6a409e290 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a9900700 10 monclient(hunting): authenticate will time out at 1839.721447s 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa69bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fa6a4097ad0 0x7fa6a4097160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38262/0 (socket says 172.21.15.142:38262) 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa69bfff700 1 -- 172.21.15.142:0/1377844099 learned_addr learned my addr 172.21.15.142:0/1377844099 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:50.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 1 --2- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a41374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): get_auth_request con 0x7fa6a409e290 auth_method 0 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient(hunting): handle_auth_done global_id 75737 payload 274 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 1 -- 172.21.15.142:0/1377844099 >> v1:172.21.15.150:6789/0 conn(0x7fa6a40a2700 legacy=0x7fa6a4096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 1 -- 172.21.15.142:0/1377844099 >> v1:172.21.15.142:6789/0 conn(0x7fa6a4097ad0 legacy=0x7fa6a4097160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 1 monclient: found mon.noname-b 2024-09-18T10:25:50.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:50.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.791+0000 7fa6a90ff700 1 -- 172.21.15.142:0/1377844099 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4097870 con 0x7fa6a409e290 2024-09-18T10:25:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a90ff700 1 -- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 msgr2=0x7fa6a41374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a90ff700 1 --2- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a41374c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009e90 tx=0x7fa69c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7788a7bc7b57b4d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 1 -- 172.21.15.142:0/1377844099 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c00e070 con 0x7fa6a409e290 2024-09-18T10:25:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:50.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: dump: 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:50.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 1 -- 172.21.15.142:0/1377844099 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c012430 con 0x7fa6a409e290 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 5 monclient: authenticate success, global_id 75737 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 1 -- 172.21.15.142:0/1377844099 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0127d0 con 0x7fa6a409e290 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 4 set_mon_vals no callback set 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa69b7fe700 10 monclient: dump: 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.807 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 10 monclient: shutdown 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 msgr2=0x7fa6a41374c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 --2- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a41374c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009e90 tx=0x7fa69c009780 comp rx=0 tx=0).stop 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- 172.21.15.142:0/1377844099 shutdown_connections 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 --2- 172.21.15.142:0/1377844099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a41374c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:50.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- 172.21.15.142:0/1377844099 >> 172.21.15.142:0/1377844099 conn(0x7fa6a4071700 msgr2=0x7fa6a40a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- 172.21.15.142:0/1377844099 shutdown_connections 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- 172.21.15.142:0/1377844099 wait complete. 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 10 monclient: build_initial_monmap 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 10 monclient: monmap: 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:50.809 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:50.810 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 Processor -- start 2024-09-18T10:25:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 1 -- start start 2024-09-18T10:25:50.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.795+0000 7fa6a9900700 10 monclient: init 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 0x7fa6a411ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-a con 0x7fa6a4097ad0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-c con 0x7fa6a409e290 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a40a2700 0x7fa6a411b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a40a2700 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a411cac0 con 0x7fa6a4097ad0 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a409a6a0 con 0x7fa6a40a2700 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a4097ec0 con 0x7fa6a409e290 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient(hunting): authenticate will time out at 1839.728226s 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54464/0 (socket says 172.21.15.142:54464) 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 -- 172.21.15.142:0/3968999932 learned_addr learned my addr 172.21.15.142:0/3968999932 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:50.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): get_auth_request con 0x7fa6a409e290 auth_method 0 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 0x7fa6a411ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 10 monclient(hunting): get_auth_request con 0x7fa6a4097ad0 auth_method 0 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a90ff700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a40a2700 0x7fa6a411b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:50.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a90ff700 10 monclient(hunting): get_auth_request con 0x7fa6a40a2700 auth_method 0 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a90ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a90ff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a90ff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient(hunting): handle_auth_done global_id 75740 payload 824 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 -- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a40a2700 msgr2=0x7fa6a411b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a40a2700 0x7fa6a411b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 -- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 msgr2=0x7fa6a411ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 0x7fa6a411ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 monclient: found mon.noname-c 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:50.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 -- 172.21.15.142:0/3968999932 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c009a20 con 0x7fa6a409e290 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a88fe700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 0x7fa6a411ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 -- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 msgr2=0x7fa6a411b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69bfff700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa69400b610 tx=0x7fa694009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694011840 con 0x7fa6a409e290 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: dump: 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:50.818 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: _finish_auth 0 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 5 monclient: authenticate success, global_id 75740 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6940119e0 con 0x7fa6a409e290 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _renew_subs 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694011da0 con 0x7fa6a409e290 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a411e070 con 0x7fa6a409e290 2024-09-18T10:25:50.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa698ff9700 10 monclient: dump: 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:50.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa69affd700 4 set_mon_vals no callback set 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _renew_subs 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.799+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a41375b0 con 0x7fa6a409e290 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a9900700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a9900700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a9900700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a4072a80 con 0x7fa6a409e290 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa694010590 con 0x7fa6a409e290 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa698ff9700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 0x7fa690037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa694048aa0 con 0x7fa6a409e290 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a88fe700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 0x7fa690037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:50.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.803+0000 7fa6a88fe700 10 monclient: get_auth_request con 0x7fa690035ac0 auth_method 0 2024-09-18T10:25:50.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.807+0000 7fa6a88fe700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 0x7fa690037f70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0007fd0 tx=0x7fa6a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:50.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.811+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa694018910 con 0x7fa6a409e290 2024-09-18T10:25:50.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.811+0000 7fa698ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:50.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:50.811+0000 7fa698ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:50.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497487908}) 2024-09-18T10:25:50.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293353911}) 2024-09-18T10:25:50.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497487908}) 2024-09-18T10:25:50.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2901658777}) 2024-09-18T10:25:50.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154620507}) 2024-09-18T10:25:50.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86960653}) 2024-09-18T10:25:50.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3154620507}) 2024-09-18T10:25:50.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.847+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449940017}) 2024-09-18T10:25:50.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105197238}) 2024-09-18T10:25:50.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268398890}) 2024-09-18T10:25:50.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105197238}) 2024-09-18T10:25:50.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288283077}) 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.851+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657300699}) 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533539037}) 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657300699}) 2024-09-18T10:25:50.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495066685}) 2024-09-18T10:25:50.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872359437}) 2024-09-18T10:25:50.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=753365271}) 2024-09-18T10:25:50.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872359437}) 2024-09-18T10:25:50.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.855+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197704606}) 2024-09-18T10:25:50.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613960376}) 2024-09-18T10:25:50.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211347647}) 2024-09-18T10:25:50.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:25:50.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613960376}) 2024-09-18T10:25:50.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:25:50.859+0000 7fe37895a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1045353676}) 2024-09-18T10:25:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa6a9900700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa6a9900700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa6a9900700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa6a4068360 con 0x7fa6a409e290 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa698ff9700 1 -- 172.21.15.142:0/3968999932 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa694018310 con 0x7fa6a409e290 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa698ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.031+0000 7fa698ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:51.041 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 msgr2=0x7fa690037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 0x7fa690037f70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0007fd0 tx=0x7fa6a0009040 comp rx=0 tx=0).stop 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 10 monclient: shutdown 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 msgr2=0x7fa6a411b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa69400b610 tx=0x7fa694009120 comp rx=0 tx=0).stop 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 shutdown_connections 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa690035ac0 0x7fa690037f70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a40a2700 0x7fa6a411b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6a409e290 0x7fa6a411b070 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 --2- 172.21.15.142:0/3968999932 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6a4097ad0 0x7fa6a411ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 >> 172.21.15.142:0/3968999932 conn(0x7fa6a4071700 msgr2=0x7fa6a4099d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 shutdown_connections 2024-09-18T10:25:51.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.035+0000 7fa6a9900700 1 -- 172.21.15.142:0/3968999932 wait complete. 2024-09-18T10:25:51.054 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:25:51.054 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:25:51.055 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:25:51.055 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:25:51.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: get_monmap_and_config 2024-09-18T10:25:51.159 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: monmap: 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:51.157227+0000 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:51.157227+0000 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:51.160 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 Processor -- start 2024-09-18T10:25:51.163 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 -- start start 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: init 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fe009a6a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6fe006f7f0 0x7f6fe009b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-a con 0x7f6fe006f7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6fe009b6c0 0x7f6fe013d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-c con 0x7f6fe009b6c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe006fbf0 con 0x7f6fe009a6a0 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.164 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe009dae0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe0099bc0 con 0x7f6fe009b6c0 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe71b3700 10 monclient(hunting): authenticate will time out at 1840.082936s 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6fe006f7f0 0x7f6fe009b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe61b1700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6fe009a6a0 0x7f6fe009aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38276/0 (socket says 172.21.15.142:38276) 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe61b1700 1 -- 172.21.15.142:0/3838143221 learned_addr learned my addr 172.21.15.142:0/3838143221 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request con 0x7f6fe006f7f0 auth_method 0 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 1 --2- 172.21.15.142:0/3838143221 >> v2:172.21.15.150:3300/0 conn(0x7f6fe009b6c0 0x7f6fe013d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): get_auth_request con 0x7f6fe009b6c0 auth_method 0 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_done global_id 184236 payload 274 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 -- 172.21.15.142:0/3838143221 >> v2:172.21.15.150:3300/0 conn(0x7f6fe009b6c0 msgr2=0x7f6fe013d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/3838143221 >> v2:172.21.15.150:3300/0 conn(0x7f6fe009b6c0 0x7f6fe013d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 -- 172.21.15.142:0/3838143221 >> v1:172.21.15.142:6789/0 conn(0x7f6fe009a6a0 legacy=0x7f6fe009aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 monclient: found mon.noname-a 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.151+0000 7f6fe59b0700 1 -- 172.21.15.142:0/3838143221 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe00998e0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe59b0700 1 -- 172.21.15.142:0/3838143221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006f7f0 msgr2=0x7f6fe009b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/3838143221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006f7f0 0x7f6fe009b180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009e60 tx=0x7f6fdc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2513496f9a0692 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 1 -- 172.21.15.142:0/3838143221 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc00e070 con 0x7f6fe006f7f0 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: dump: 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.174 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 5 monclient: authenticate success, global_id 184236 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 1 -- 172.21.15.142:0/3838143221 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc00bd30 con 0x7f6fe006f7f0 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 4 set_mon_vals no callback set 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 1 -- 172.21.15.142:0/3838143221 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc00bea0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.175 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe51af700 10 monclient: dump: 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.176 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 10 monclient: shutdown 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 -- 172.21.15.142:0/3838143221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006f7f0 msgr2=0x7f6fe009b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/3838143221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006f7f0 0x7f6fe009b180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009e60 tx=0x7f6fdc009780 comp rx=0 tx=0).stop 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 -- 172.21.15.142:0/3838143221 shutdown_connections 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/3838143221 >> v2:172.21.15.150:3300/0 conn(0x7f6fe009b6c0 0x7f6fe013d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/3838143221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006f7f0 0x7f6fe009b180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 -- 172.21.15.142:0/3838143221 >> 172.21.15.142:0/3838143221 conn(0x7f6fe0070ef0 msgr2=0x7f6fe00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 -- 172.21.15.142:0/3838143221 shutdown_connections 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 1 -- 172.21.15.142:0/3838143221 wait complete. 2024-09-18T10:25:51.177 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.155+0000 7f6fe71b3700 10 monclient: monmap: 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 Processor -- start 2024-09-18T10:25:51.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 -- start start 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient: init 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-c con 0x7f6fe006f7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 0x7f6fe006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-a con 0x7f6fe009a6a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fe009b6c0 0x7f6fe006ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): picked mon.noname-b con 0x7f6fe009b6c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe013ce90 con 0x7f6fe009a6a0 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0094830 con 0x7f6fe009b6c0 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00a03e0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe71b3700 10 monclient(hunting): authenticate will time out at 1840.089600s 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 0x7f6fe006c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 0x7f6fe006c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36708/0 (socket says 172.21.15.142:36708) 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54476/0 (socket says 172.21.15.142:54476) 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 1 -- 172.21.15.142:0/192296522 learned_addr learned my addr 172.21.15.142:0/192296522 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request con 0x7f6fe009a6a0 auth_method 0 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): get_auth_request con 0x7f6fe006f7f0 auth_method 0 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fe009b6c0 0x7f6fe006ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): get_auth_request con 0x7f6fe009b6c0 auth_method 0 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient(hunting): handle_auth_done global_id 75743 payload 824 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fe009b6c0 msgr2=0x7f6fe006ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fe009b6c0 0x7f6fe006ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 msgr2=0x7f6fe006c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 0x7f6fe006c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 monclient: found mon.noname-c 2024-09-18T10:25:51.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe61b1700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc009a20 con 0x7f6fe006f7f0 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe69b2700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fe009b6c0 0x7f6fe006ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.159+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe009a6a0 0x7f6fe006c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe61b1700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 msgr2=0x7f6fe006c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe61b1700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c560 tx=0x7f6fd0007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0005d40 con 0x7f6fe006f7f0 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: dump: 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.183 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 5 monclient: authenticate success, global_id 75743 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 monclient: _renew_subs 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe006e1b0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd00070d0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0007490 con 0x7f6fe006f7f0 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 10 monclient: dump: 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.185 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe49ae700 4 set_mon_vals no callback set 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 monclient: _renew_subs 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe01267c0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 0x7f6fe0092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe00648a0 con 0x7f6fe006d350 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 0x7f6fe0092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient: get_auth_request con 0x7f6fe006d350 auth_method 0 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_done global_id 184242 payload 824 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fe59b0700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fe013d8e0 con 0x7f6fe006d350 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6fd001b3b0 con 0x7f6fe006f7f0 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fc4035ac0 0x7f6fc4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.163+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6fd0048300 con 0x7f6fe006f7f0 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fe69b2700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fc4035ac0 0x7f6fc4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fe69b2700 10 monclient: get_auth_request con 0x7f6fc4035ac0 auth_method 0 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fe59b0700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 msgr2=0x7f6fe0092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 0x7f6fe0092800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00a0280 tx=0x7f6fdc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc022070 con 0x7f6fe006d350 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 10 monclient: dump: 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6fdc01d840 con 0x7f6fe006d350 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 msgr2=0x7f6fe0092800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 0x7f6fe0092800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00a0280 tx=0x7f6fdc018000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 msgr2=0x7f6fe0092800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fd67fc700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe006d350 0x7f6fe0092800 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00a0280 tx=0x7f6fdc018000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.167+0000 7f6fe69b2700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fc4035ac0 0x7f6fc4037f70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6fe01266f0 tx=0x7f6fd80091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe71b3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 0x7f6fe0093140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe71b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe0072a80 con 0x7f6fe0092d40 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 0x7f6fe0093140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient: get_auth_request con 0x7f6fe0092d40 auth_method 0 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 10 monclient(hunting): handle_auth_done global_id 184245 payload 824 2024-09-18T10:25:51.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.179+0000 7f6fe59b0700 1 -- 172.21.15.142:0/192296522 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6fe0126b30 con 0x7f6fe0092d40 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe59b0700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 msgr2=0x7f6fe0093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe59b0700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 0x7f6fe0093140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc012fd0 tx=0x7f6fdc01f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc01c620 con 0x7f6fe0092d40 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 10 monclient: dump: 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7f6fdc02d040 con 0x7f6fe0092d40 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 msgr2=0x7f6fe0093140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 0x7f6fe0093140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc012fd0 tx=0x7f6fdc01f070 comp rx=0 tx=0).stop 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 msgr2=0x7f6fe0093140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fd67fc700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6fe0092d40 0x7f6fe0093140 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc012fd0 tx=0x7f6fdc01f070 comp rx=0 tx=0).stop 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:51.195 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.196 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.197 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:51.198 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.199 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.200 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.201 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:51.204 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:51.205 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.206 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:51.207 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:51.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fc4035ac0 msgr2=0x7f6fc4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6fc4035ac0 0x7f6fc4037f70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6fe01266f0 tx=0x7f6fd80091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 10 monclient: shutdown 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe69b2700 1 -- 172.21.15.142:0/192296522 reap_dead start 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 msgr2=0x7f6fe006c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c560 tx=0x7f6fd0007f90 comp rx=0 tx=0).stop 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 shutdown_connections 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 --2- 172.21.15.142:0/192296522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6fe006f7f0 0x7f6fe006c390 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 >> 172.21.15.142:0/192296522 conn(0x7f6fe0070ef0 msgr2=0x7f6fe009c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 shutdown_connections 2024-09-18T10:25:51.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.183+0000 7f6fe71b3700 1 -- 172.21.15.142:0/192296522 wait complete. 2024-09-18T10:25:51.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:51.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:25:51.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:25:51.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.311+0000 7f99deeaa700 10 monclient: get_monmap_and_config 2024-09-18T10:25:51.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.311+0000 7f99deeaa700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.311+0000 7f99deeaa700 10 monclient: monmap: 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:51.319387+0000 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:51.319387+0000 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:51.322 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 Processor -- start 2024-09-18T10:25:51.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 -- start start 2024-09-18T10:25:51.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient: init 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99d809abd0 0x7f99d80960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-a con 0x7f99d809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99d809b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 0x7f99d8137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-c con 0x7f99d809c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d809dc90 con 0x7f99d809b740 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d809b500 con 0x7f99d809abd0 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d809afa0 con 0x7f99d809c5e0 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99deeaa700 10 monclient(hunting): authenticate will time out at 1840.244821s 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f99d809abd0 0x7f99d80960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99dd6a7700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f99d809b740 0x7f99d809bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:38280/0 (socket says 172.21.15.142:38280) 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99dd6a7700 1 -- 172.21.15.142:0/1818280735 learned_addr learned my addr 172.21.15.142:0/1818280735 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): get_auth_request con 0x7f99d809abd0 auth_method 0 2024-09-18T10:25:51.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99de6a9700 1 --2- 172.21.15.142:0/1818280735 >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 0x7f99d8137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient(hunting): handle_auth_done global_id 184248 payload 274 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 -- 172.21.15.142:0/1818280735 >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 msgr2=0x7f99d8137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 --2- 172.21.15.142:0/1818280735 >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 0x7f99d8137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 -- 172.21.15.142:0/1818280735 >> v1:172.21.15.142:6789/0 conn(0x7f99d809b740 legacy=0x7f99d809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 monclient: found mon.noname-a 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 -- 172.21.15.142:0/1818280735 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d806f810 con 0x7f99d809abd0 2024-09-18T10:25:51.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99de6a9700 10 monclient: get_auth_request con 0x7f99d809c5e0 auth_method 0 2024-09-18T10:25:51.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99de6a9700 1 --2- 172.21.15.142:0/1818280735 >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 0x7f99d8137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 -- 172.21.15.142:0/1818280735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 msgr2=0x7f99d80960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.315+0000 7f99ddea8700 1 --2- 172.21.15.142:0/1818280735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d80960e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99d8076de0 tx=0x7f99d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851027d23b0e4cbc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 1 -- 172.21.15.142:0/1818280735 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d400b970 con 0x7f99d809abd0 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: dump: 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 5 monclient: authenticate success, global_id 184248 2024-09-18T10:25:51.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 1 -- 172.21.15.142:0/1818280735 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d400bad0 con 0x7f99d809abd0 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 4 set_mon_vals no callback set 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 1 -- 172.21.15.142:0/1818280735 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d400be70 con 0x7f99d809abd0 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dcea6700 10 monclient: dump: 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.338 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: shutdown 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- 172.21.15.142:0/1818280735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 msgr2=0x7f99d80960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- 172.21.15.142:0/1818280735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d80960e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99d8076de0 tx=0x7f99d40091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- 172.21.15.142:0/1818280735 shutdown_connections 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- 172.21.15.142:0/1818280735 >> v2:172.21.15.150:3300/0 conn(0x7f99d809c5e0 0x7f99d8137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- 172.21.15.142:0/1818280735 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d80960e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- 172.21.15.142:0/1818280735 >> 172.21.15.142:0/1818280735 conn(0x7f99d8070ef0 msgr2=0x7f99d80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- 172.21.15.142:0/1818280735 shutdown_connections 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- 172.21.15.142:0/1818280735 wait complete. 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: monmap: 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:51.340 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 Processor -- start 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- start start 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: init 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-a con 0x7f99d809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d809b740 0x7f99d813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-b con 0x7f99d809b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 0x7f99d808bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): picked mon.noname-c con 0x7f99d809c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d809d240 con 0x7f99d809abd0 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80a0040 con 0x7f99d809b740 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d808f2f0 con 0x7f99d809c5e0 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99deeaa700 10 monclient(hunting): authenticate will time out at 1840.251560s 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d813f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36732/0 (socket says 172.21.15.142:36732) 2024-09-18T10:25:51.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 1 -- 172.21.15.142:0/2264829904 learned_addr learned my addr 172.21.15.142:0/2264829904 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): get_auth_request con 0x7f99d809abd0 auth_method 0 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 0x7f99d808bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): get_auth_request con 0x7f99d809c5e0 auth_method 0 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dd6a7700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d809b740 0x7f99d813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99de6a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.319+0000 7f99dd6a7700 10 monclient(hunting): get_auth_request con 0x7f99d809b740 auth_method 0 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99dd6a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99dd6a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99dd6a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 10 monclient(hunting): handle_auth_done global_id 75746 payload 824 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d809b740 msgr2=0x7f99d813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d809b740 0x7f99d813feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 msgr2=0x7f99d813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f99d809abd0 0x7f99d813f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 monclient: found mon.noname-c 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4009420 con 0x7f99d809c5e0 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99dd6a7700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99d809b740 0x7f99d813feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 msgr2=0x7f99d808bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99de6a9700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 0x7f99d808bc90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f99d000cf30 tx=0x7f99d0007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000ae90 con 0x7f99d809c5e0 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: dump: 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d000d650 con 0x7f99d809c5e0 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000da10 con 0x7f99d809c5e0 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cffff700 4 set_mon_vals no callback set 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99cdffb700 10 monclient: dump: 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.347 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 5 monclient: authenticate success, global_id 75746 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 10 monclient: _renew_subs 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d808f820 con 0x7f99d809c5e0 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 10 monclient: _renew_subs 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.323+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d808e300 con 0x7f99d809c5e0 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99deeaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99deeaa700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 0x7f99d808c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80648a0 con 0x7f99d808c1d0 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 0x7f99d808c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient: get_auth_request con 0x7f99d808c1d0 auth_method 0 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f99d001e3b0 con 0x7f99d809c5e0 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 10 monclient(hunting): handle_auth_done global_id 75749 payload 824 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f99d808f480 con 0x7f99d808c1d0 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99bc035ac0 0x7f99bc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f99d0049790 con 0x7f99d809c5e0 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99dd6a7700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99bc035ac0 0x7f99bc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99dd6a7700 10 monclient: get_auth_request con 0x7f99bc035ac0 auth_method 0 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 msgr2=0x7f99d808c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99ddea8700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 0x7f99d808c5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f99d8095e60 tx=0x7f99d4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4021070 con 0x7f99d808c1d0 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.327+0000 7f99cdffb700 10 monclient: dump: 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f99d4026460 con 0x7f99d808c1d0 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 msgr2=0x7f99d808c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 0x7f99d808c5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f99d8095e60 tx=0x7f99d4017000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 msgr2=0x7f99d808c5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99cdffb700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808c1d0 0x7f99d808c5a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.331+0000 7f99dd6a7700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99bc035ac0 0x7f99bc037f70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f99d809d1b0 tx=0x7f99c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99deeaa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99deeaa700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 0x7f99d8087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99deeaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d80904f0 con 0x7f99d808cae0 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 0x7f99d8087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient: get_auth_request con 0x7f99d808cae0 auth_method 0 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 10 monclient(hunting): handle_auth_done global_id 75752 payload 824 2024-09-18T10:25:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.339+0000 7f99ddea8700 1 -- 172.21.15.142:0/2264829904 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f99d8072a80 con 0x7f99d808cae0 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99ddea8700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 msgr2=0x7f99d8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99ddea8700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 0x7f99d8087fe0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f99d4025000 tx=0x7f99d4021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d401bdc0 con 0x7f99d808cae0 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 10 monclient: dump: 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f99d4011d50 con 0x7f99d808cae0 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 msgr2=0x7f99d8087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 0x7f99d8087fe0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f99d4025000 tx=0x7f99d4021000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 msgr2=0x7f99d8087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99cdffb700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d808cae0 0x7f99d8087fe0 secure :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f99d4025000 tx=0x7f99d4021000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:51.358 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.359 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.360 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.361 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.362 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:51.363 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.364 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.365 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.366 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.367 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.368 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.369 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.370 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:51.371 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:51.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99bc035ac0 msgr2=0x7f99bc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f99bc035ac0 0x7f99bc037f70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f99d809d1b0 tx=0x7f99c8009040 comp rx=0 tx=0).stop 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 10 monclient: shutdown 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99de6a9700 1 -- 172.21.15.142:0/2264829904 reap_dead start 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 msgr2=0x7f99d808bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 0x7f99d808bc90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f99d000cf30 tx=0x7f99d0007840 comp rx=0 tx=0).stop 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 shutdown_connections 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 --2- 172.21.15.142:0/2264829904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99d809c5e0 0x7f99d808bc90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 >> 172.21.15.142:0/2264829904 conn(0x7f99d8070ef0 msgr2=0x7f99d809de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 shutdown_connections 2024-09-18T10:25:51.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.343+0000 7f99deeaa700 1 -- 172.21.15.142:0/2264829904 wait complete. 2024-09-18T10:25:51.373 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:51.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:25:51.374 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: get_monmap_and_config 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: monmap: 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:51.488837+0000 2024-09-18T10:25:51.491 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:51.488837+0000 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:51.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 Processor -- start 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 -- start start 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: init 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 0x7f095c0a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f095c0999e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f095c09c410 0x7f095c0a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f095c09c410 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f095c0a0c00 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0a20c0 con 0x7f095c0a0c00 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c09a000 con 0x7f095c09c410 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0a1c80 con 0x7f095c0999e0 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095bfff700 10 monclient(hunting): authenticate will time out at 1840.415082s 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f095c09c410 0x7f095c0a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 0x7f095c0a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 0x7f095c0a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54532/0 (socket says 172.21.15.142:54532) 2024-09-18T10:25:51.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f095c09c410 0x7f095c0a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36740/0 (socket says 172.21.15.142:36740) 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 1 -- 172.21.15.142:0/1253342082 learned_addr learned my addr 172.21.15.142:0/1253342082 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): get_auth_request con 0x7f095c09c410 auth_method 0 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): get_auth_request con 0x7f095c0999e0 auth_method 0 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.483+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 10 monclient(hunting): handle_auth_done global_id 184254 payload 274 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 -- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 msgr2=0x7f095c0a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 --2- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 0x7f095c0a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 -- 172.21.15.142:0/1253342082 >> v1:172.21.15.150:6789/0 conn(0x7f095c0a0c00 legacy=0x7f095c138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 monclient: found mon.noname-a 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 -- 172.21.15.142:0/1253342082 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c09a580 con 0x7f095c09c410 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 -- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c09c410 msgr2=0x7f095c0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095a7fc700 1 --2- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c09c410 0x7f095c0a06c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09440099b0 tx=0x7f0944009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b872c2121e2aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 1 -- 172.21.15.142:0/1253342082 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944009ea0 con 0x7f095c09c410 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: dump: 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.500 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 5 monclient: authenticate success, global_id 184254 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 1 -- 172.21.15.142:0/1253342082 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094400ba70 con 0x7f095c09c410 2024-09-18T10:25:51.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 4 set_mon_vals no callback set 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 1 -- 172.21.15.142:0/1253342082 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094400be30 con 0x7f095c09c410 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f0959ffb700 10 monclient: dump: 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 10 monclient: shutdown 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 -- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c09c410 msgr2=0x7f095c0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 --2- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c09c410 0x7f095c0a06c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09440099b0 tx=0x7f0944009250 comp rx=0 tx=0).stop 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 -- 172.21.15.142:0/1253342082 shutdown_connections 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 --2- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c09c410 0x7f095c0a06c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 --2- 172.21.15.142:0/1253342082 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c0999e0 0x7f095c0a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 -- 172.21.15.142:0/1253342082 >> 172.21.15.142:0/1253342082 conn(0x7f095c0a5790 msgr2=0x7f095c0a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 -- 172.21.15.142:0/1253342082 shutdown_connections 2024-09-18T10:25:51.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.487+0000 7f095bfff700 1 -- 172.21.15.142:0/1253342082 wait complete. 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient: monmap: 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 Processor -- start 2024-09-18T10:25:51.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 -- start start 2024-09-18T10:25:51.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient: init 2024-09-18T10:25:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 0x7f095c08e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f095c0999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c09c410 0x7f095c08e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f095c09c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c0a0c00 0x7f095c08eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f095c0a0c00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0915b0 con 0x7f095c0999e0 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c092480 con 0x7f095c0a0c00 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c090370 con 0x7f095c09c410 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095bfff700 10 monclient(hunting): authenticate will time out at 1840.422951s 2024-09-18T10:25:51.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 0x7f095c08e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c09c410 0x7f095c08e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 0x7f095c08e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:36750/0 (socket says 172.21.15.142:36750) 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 -- 172.21.15.142:0/764770308 learned_addr learned my addr 172.21.15.142:0/764770308 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): get_auth_request con 0x7f095c0999e0 auth_method 0 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): get_auth_request con 0x7f095c09c410 auth_method 0 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c0a0c00 0x7f095c08eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 10 monclient(hunting): get_auth_request con 0x7f095c0a0c00 auth_method 0 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient(hunting): handle_auth_done global_id 184257 payload 824 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c09c410 msgr2=0x7f095c08e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f095c09c410 0x7f095c08e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c0a0c00 msgr2=0x7f095c08eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c0a0c00 0x7f095c08eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 monclient: found mon.noname-a 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095affd700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440094f0 con 0x7f095c0999e0 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.491+0000 7f095b7fe700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c0a0c00 0x7f095c08eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095affd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 msgr2=0x7f095c08e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095affd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 0x7f095c08e060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f094c00ec80 tx=0x7f094c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c00ca60 con 0x7f095c0999e0 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: dump: 2024-09-18T10:25:51.514 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c00cbc0 con 0x7f095c0999e0 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 5 monclient: authenticate success, global_id 184257 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 monclient: _renew_subs 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c0912d0 con 0x7f095c0999e0 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c013470 con 0x7f095c0999e0 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 10 monclient: dump: 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.516 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f09597fa700 4 set_mon_vals no callback set 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 monclient: _renew_subs 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c099410 con 0x7f095c0999e0 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 0x7f095c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0647f0 con 0x7f095c08f020 2024-09-18T10:25:51.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.495+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f094c0137e0 con 0x7f095c0999e0 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 0x7f095c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient: get_auth_request con 0x7f095c08f020 auth_method 0 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f093c035ac0 0x7f093c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 10 monclient(hunting): handle_auth_done global_id 194146 payload 824 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f095c072a80 con 0x7f095c08f020 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f094c049660 con 0x7f095c0999e0 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095b7fe700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f093c035ac0 0x7f093c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095b7fe700 10 monclient: get_auth_request con 0x7f093c035ac0 auth_method 0 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 msgr2=0x7f095c06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095a7fc700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 0x7f095c06dad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f095c0a56a0 tx=0x7f0944016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944021070 con 0x7f095c08f020 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f0952ffd700 10 monclient: dump: 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.519 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.499+0000 7f095b7fe700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f093c035ac0 0x7f093c037f70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0954007fd0 tx=0x7f0954009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f094401c450 con 0x7f095c08f020 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 msgr2=0x7f095c06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 0x7f095c06dad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f095c0a56a0 tx=0x7f0944016000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 msgr2=0x7f095c06dad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.503+0000 7f0952ffd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c08f020 0x7f095c06dad0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f095c0a56a0 tx=0x7f0944016000 comp rx=0 tx=0).stop 2024-09-18T10:25:51.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:25:51.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095bfff700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 0x7f095c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095bfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c090fb0 con 0x7f095c06e010 2024-09-18T10:25:51.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095a7fc700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 0x7f095c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095a7fc700 10 monclient: get_auth_request con 0x7f095c06e010 auth_method 0 2024-09-18T10:25:51.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.511+0000 7f095a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 10 monclient(hunting): handle_auth_done global_id 194149 payload 824 2024-09-18T10:25:51.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 1 -- 172.21.15.142:0/764770308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f095c067800 con 0x7f095c06e010 2024-09-18T10:25:51.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 msgr2=0x7f095c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f095a7fc700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 0x7f095c06e410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0944003080 tx=0x7f094401e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944025b30 con 0x7f095c06e010 2024-09-18T10:25:51.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 10 monclient: dump: 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.526 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f0944025c90 con 0x7f095c06e010 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.515+0000 7f0952ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 msgr2=0x7f095c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f0952ffd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 0x7f095c06e410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0944003080 tx=0x7f094401e070 comp rx=0 tx=0).stop 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f0952ffd700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 msgr2=0x7f095c06e410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:25:51.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f0952ffd700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f095c06e010 0x7f095c06e410 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0944003080 tx=0x7f094401e070 comp rx=0 tx=0).stop 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 154, 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 5, 2024-09-18T10:25:51.528 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.529 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.530 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.531 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.531 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:25:51.532 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:25:51.533 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.534 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:25:51.535 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.536 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.537 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:25:51.538 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:25:51.539 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:25:51.540 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:25:51.541 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:25:51.541 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:25:51.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f093c035ac0 msgr2=0x7f093c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f093c035ac0 0x7f093c037f70 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0954007fd0 tx=0x7f0954009040 comp rx=0 tx=0).stop 2024-09-18T10:25:51.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 10 monclient: shutdown 2024-09-18T10:25:51.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 msgr2=0x7f095c08e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 --2- 172.21.15.142:0/764770308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f095c0999e0 0x7f095c08e060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f094c00ec80 tx=0x7f094c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095b7fe700 1 -- 172.21.15.142:0/764770308 reap_dead start 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095b7fe700 1 -- 172.21.15.142:0/764770308 reap_dead start 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 shutdown_connections 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 >> 172.21.15.142:0/764770308 conn(0x7f095c0a5790 msgr2=0x7f095c09d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 shutdown_connections 2024-09-18T10:25:51.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.519+0000 7f095bfff700 1 -- 172.21.15.142:0/764770308 wait complete. 2024-09-18T10:25:51.543 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:25:51.543 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:25:51.544 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:25:51.544 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:25:51.544 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:25:51.544 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:25:51.544 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:25:51.562 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:25:51.562 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:25:51.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:51.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.655+0000 7f62db59e700 10 monclient: get_monmap_and_config 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.655+0000 7f62db59e700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.655+0000 7f62db59e700 10 monclient: monmap: 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:51.663560+0000 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:51.663560+0000 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:51.666 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:51.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:51.667 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:51.667 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:51.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 Processor -- start 2024-09-18T10:25:51.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 -- start start 2024-09-18T10:25:51.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient: init 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62dc06e860 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 0x7f62dc09c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-a con 0x7f62dc06f460 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-b con 0x7f62dc09ca30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc06f220 con 0x7f62dc06e860 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc099780 con 0x7f62dc06f460 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0998c0 con 0x7f62dc09ca30 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62db59e700 10 monclient(hunting): authenticate will time out at 1840.589036s 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62d1d9b700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 msgr2=0x7f62dc09c4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62d1d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 0x7f62dc09c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54542/0 (socket says 172.21.15.142:54542) 2024-09-18T10:25:51.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 -- 172.21.15.142:0/3595426965 learned_addr learned my addr 172.21.15.142:0/3595426965 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): get_auth_request con 0x7f62dc09ca30 auth_method 0 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient(hunting): handle_auth_done global_id 75761 payload 274 2024-09-18T10:25:51.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 -- 172.21.15.142:0/3595426965 >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 msgr2=0x7f62dc09c4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 --2- 172.21.15.142:0/3595426965 >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 0x7f62dc09c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 -- 172.21.15.142:0/3595426965 >> v1:172.21.15.150:6789/0 conn(0x7f62dc06e860 legacy=0x7f62dc09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 monclient: found mon.noname-b 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.659+0000 7f62dad9d700 1 -- 172.21.15.142:0/3595426965 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc09b770 con 0x7f62dc09ca30 2024-09-18T10:25:51.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62dad9d700 1 -- 172.21.15.142:0/3595426965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 msgr2=0x7f62dc1395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62dad9d700 1 --2- 172.21.15.142:0/3595426965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f62dc098a50 tx=0x7f62d4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b020a0291b6608b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 1 -- 172.21.15.142:0/3595426965 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d400e070 con 0x7f62dc09ca30 2024-09-18T10:25:51.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: dump: 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 5 monclient: authenticate success, global_id 75761 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 1 -- 172.21.15.142:0/3595426965 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d400bc60 con 0x7f62dc09ca30 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 4 set_mon_vals no callback set 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 1 -- 172.21.15.142:0/3595426965 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4007400 con 0x7f62dc09ca30 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d9d9b700 10 monclient: dump: 2024-09-18T10:25:51.681 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: shutdown 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- 172.21.15.142:0/3595426965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 msgr2=0x7f62dc1395c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- 172.21.15.142:0/3595426965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f62dc098a50 tx=0x7f62d4009840 comp rx=0 tx=0).stop 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- 172.21.15.142:0/3595426965 shutdown_connections 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- 172.21.15.142:0/3595426965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc09ca30 0x7f62dc1395c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- 172.21.15.142:0/3595426965 >> v2:172.21.15.142:3300/0 conn(0x7f62dc06f460 0x7f62dc09c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- 172.21.15.142:0/3595426965 >> 172.21.15.142:0/3595426965 conn(0x7f62dc0a1fe0 msgr2=0x7f62dc070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- 172.21.15.142:0/3595426965 shutdown_connections 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- 172.21.15.142:0/3595426965 wait complete. 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: build_initial_monmap 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: monmap: 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:51.683 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 Processor -- start 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- start start 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: init 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62dc06e860 0x7f62dc092840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-b con 0x7f62dc06e860 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-c con 0x7f62dc06f460 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 0x7f62dc08ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): picked mon.noname-a con 0x7f62dc09ca30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc09d230 con 0x7f62dc09ca30 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc097ad0 con 0x7f62dc06e860 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:51.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc091560 con 0x7f62dc06f460 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62db59e700 10 monclient(hunting): authenticate will time out at 1840.595347s 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62dad9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 msgr2=0x7f62dc08ff50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62dad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 0x7f62dc08ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:54546/0 (socket says 172.21.15.142:54546) 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 1 -- 172.21.15.142:0/537454152 learned_addr learned my addr 172.21.15.142:0/537454152 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): get_auth_request con 0x7f62dc06f460 auth_method 0 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62dc06e860 0x7f62dc092840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): get_auth_request con 0x7f62dc06e860 auth_method 0 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62d1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.663+0000 7f62da59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62da59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 10 monclient(hunting): handle_auth_done global_id 75764 payload 824 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 -- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62dc06e860 msgr2=0x7f62dc092840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62dc06e860 0x7f62dc092840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 -- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 msgr2=0x7f62dc08ff50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 0x7f62dc08ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 monclient: found mon.noname-c 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 -- 172.21.15.142:0/537454152 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009ae0 con 0x7f62dc06f460 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 -- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 msgr2=0x7f62dc08fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d1d9b700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f62c400b7e0 tx=0x7f62c4009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4003e20 con 0x7f62dc06f460 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: dump: 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:25:51.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c4006150 con 0x7f62dc06f460 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 5 monclient: authenticate success, global_id 75764 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 10 monclient: _renew_subs 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc091a00 con 0x7f62dc06f460 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4006510 con 0x7f62dc06f460 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d2ffd700 10 monclient: dump: 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:51.690 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 10 monclient: _renew_subs 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62d8d99700 4 set_mon_vals no callback set 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.667+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc092140 con 0x7f62dc06f460 2024-09-18T10:25:51.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62db59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f62c401c440 con 0x7f62dc06f460 2024-09-18T10:25:51.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62db59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62db59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc096620 con 0x7f62dc06f460 2024-09-18T10:25:51.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62d2ffd700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 0x7f62bc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f62c40486f0 con 0x7f62dc06f460 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62da59c700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 0x7f62bc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62da59c700 10 monclient: get_auth_request con 0x7f62bc035a70 auth_method 0 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.671+0000 7f62da59c700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 0x7f62bc037f20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f62cc003a30 tx=0x7f62cc007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.675+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f62c400dd60 con 0x7f62dc06f460 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.675+0000 7f62d2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:51.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.675+0000 7f62d2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f62dc08ce70 con 0x7f62dc06f460 2024-09-18T10:25:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62d2ffd700 1 -- 172.21.15.142:0/537454152 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f62c4017030 con 0x7f62dc06f460 2024-09-18T10:25:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62d2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62d2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:51.912 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 msgr2=0x7f62bc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 0x7f62bc037f20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f62cc003a30 tx=0x7f62cc007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 10 monclient: shutdown 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 msgr2=0x7f62dc08fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f62c400b7e0 tx=0x7f62c4009220 comp rx=0 tx=0).stop 2024-09-18T10:25:51.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 shutdown_connections 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f62bc035a70 0x7f62bc037f20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f62dc09ca30 0x7f62dc08ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f62dc06f460 0x7f62dc08fa10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.903+0000 7f62db59e700 1 --2- 172.21.15.142:0/537454152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62dc06e860 0x7f62dc092840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.907+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 >> 172.21.15.142:0/537454152 conn(0x7f62dc0a1fe0 msgr2=0x7f62dc089ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.907+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 shutdown_connections 2024-09-18T10:25:51.915 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:51.907+0000 7f62db59e700 1 -- 172.21.15.142:0/537454152 wait complete. 2024-09-18T10:25:54.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.027+0000 7fa5a5379700 10 monclient: get_monmap_and_config 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.027+0000 7fa5a5379700 10 monclient: build_initial_monmap 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.027+0000 7fa5a5379700 10 monclient: monmap: 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:55.035316+0000 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:55.035316+0000 2024-09-18T10:25:55.038 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 Processor -- start 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 -- start start 2024-09-18T10:25:55.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient: init 2024-09-18T10:25:55.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:55.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5a009aae0 0x7fa5a0096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-c con 0x7fa5a009aae0 addr v2:172.21.15.150:3300/0 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-b con 0x7fa5a00967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 0x7fa5a009c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-a con 0x7fa5a009c320 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a0095880 con 0x7fa5a009c320 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a009aeb0 con 0x7fa5a009aae0 2024-09-18T10:25:55.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a006f6d0 con 0x7fa5a00967a0 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a5379700 10 monclient(hunting): authenticate will time out at 1843.960776s 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a4b78700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 msgr2=0x7fa5a009c780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5a4b78700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 0x7fa5a009c780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50360/0 (socket says 172.21.15.142:50360) 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 -- 172.21.15.142:0/2582490692 learned_addr learned my addr 172.21.15.142:0/2582490692 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): get_auth_request con 0x7fa5a00967a0 auth_method 0 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:55.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 1 --2- 172.21.15.142:0/2582490692 >> v2:172.21.15.150:3300/0 conn(0x7fa5a009aae0 0x7fa5a0096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): get_auth_request con 0x7fa5a009aae0 auth_method 0 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:55.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa59ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient(hunting): handle_auth_done global_id 75767 payload 274 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 -- 172.21.15.142:0/2582490692 >> v2:172.21.15.150:3300/0 conn(0x7fa5a009aae0 msgr2=0x7fa5a0096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2582490692 >> v2:172.21.15.150:3300/0 conn(0x7fa5a009aae0 0x7fa5a0096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 -- 172.21.15.142:0/2582490692 >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 msgr2=0x7fa5a009c780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2582490692 >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 0x7fa5a009c780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 monclient: found mon.noname-b 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.031+0000 7fa5977fe700 1 -- 172.21.15.142:0/2582490692 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00955a0 con 0x7fa5a00967a0 2024-09-18T10:25:55.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 1 -- 172.21.15.142:0/2582490692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 msgr2=0x7fa5a009bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2582490692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0071c60 tx=0x7fa588009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0ad256cac371fdf server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 1 -- 172.21.15.142:0/2582490692 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588009e70 con 0x7fa5a00967a0 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: dump: 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:55.047 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 5 monclient: authenticate success, global_id 75767 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 1 -- 172.21.15.142:0/2582490692 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58800ba70 con 0x7fa5a00967a0 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 4 set_mon_vals no callback set 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 1 -- 172.21.15.142:0/2582490692 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58800be30 con 0x7fa5a00967a0 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:55.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59f7fe700 10 monclient: dump: 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:55.049 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: shutdown 2024-09-18T10:25:55.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:55.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- 172.21.15.142:0/2582490692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 msgr2=0x7fa5a009bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:55.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2582490692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0071c60 tx=0x7fa588009250 comp rx=0 tx=0).stop 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- 172.21.15.142:0/2582490692 shutdown_connections 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2582490692 >> v2:172.21.15.142:3300/0 conn(0x7fa5a009c320 0x7fa5a009c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2582490692 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a00967a0 0x7fa5a009bde0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2582490692 >> v2:172.21.15.150:3300/0 conn(0x7fa5a009aae0 0x7fa5a0096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- 172.21.15.142:0/2582490692 >> 172.21.15.142:0/2582490692 conn(0x7fa5a00a1fe0 msgr2=0x7fa5a0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- 172.21.15.142:0/2582490692 shutdown_connections 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- 172.21.15.142:0/2582490692 wait complete. 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: build_initial_monmap 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: monmap: 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:55.055 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 Processor -- start 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- start start 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: init 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 0x7fa5a01357c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-a con 0x7fa5a00967a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-c con 0x7fa5a009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a009c320 0x7fa5a008bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): picked mon.noname-b con 0x7fa5a009c320 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a0135d00 con 0x7fa5a00967a0 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a006f8d0 con 0x7fa5a009c320 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a006ed60 con 0x7fa5a009aae0 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 msgr2=0x7fa5a01357c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:55.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5a5379700 10 monclient(hunting): authenticate will time out at 1843.967631s 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa59ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 0x7fa5a01357c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50366/0 (socket says 172.21.15.142:50366) 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 1 -- 172.21.15.142:0/2373177873 learned_addr learned my addr 172.21.15.142:0/2373177873 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 10 monclient(hunting): get_auth_request con 0x7fa5a009aae0 auth_method 0 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.035+0000 7fa5977fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a009c320 0x7fa5a008bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:55.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): get_auth_request con 0x7fa5a009c320 auth_method 0 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a4b78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient(hunting): handle_auth_done global_id 75770 payload 824 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 -- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a009c320 msgr2=0x7fa5a008bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a009c320 0x7fa5a008bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 -- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 msgr2=0x7fa5a01357c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 0x7fa5a01357c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient: _finish_hunting 0 2024-09-18T10:25:55.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 monclient: found mon.noname-c 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 -- 172.21.15.142:0/2373177873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5880094f0 con 0x7fa5a009aae0 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 -- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 msgr2=0x7fa5a00940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5977fe700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58801f070 con 0x7fa5a009aae0 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: dump: 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.060 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: _finish_auth 0 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 5 monclient: authenticate success, global_id 75770 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 10 monclient: _renew_subs 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588006b80 con 0x7fa5a009aae0 2024-09-18T10:25:55.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a008f2f0 con 0x7fa5a009aae0 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588011bf0 con 0x7fa5a009aae0 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59cff9700 10 monclient: dump: 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:55.062 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa59effd700 4 set_mon_vals no callback set 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 10 monclient: _renew_subs 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.039+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a008df10 con 0x7fa5a009aae0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa5a5379700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa5a5379700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa5a5379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a008d880 con 0x7fa5a009aae0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa588023070 con 0x7fa5a009aae0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa59cff9700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 0x7fa580037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa58804a120 con 0x7fa5a009aae0 2024-09-18T10:25:55.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa59ffff700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 0x7fa580037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:55.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.043+0000 7fa59ffff700 10 monclient: get_auth_request con 0x7fa580035a70 auth_method 0 2024-09-18T10:25:55.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.047+0000 7fa59ffff700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 0x7fa580037f20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa590009510 tx=0x7fa5900091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:55.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.051+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa5a008d880 con 0x7fa5a009aae0 2024-09-18T10:25:55.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.051+0000 7fa59cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:55.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.051+0000 7fa59cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa5a5379700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa5a5379700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa5a5379700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa5a009c0b0 con 0x7fa5a009aae0 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa59cff9700 1 -- 172.21.15.142:0/2373177873 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa5a009c0b0 con 0x7fa5a009aae0 2024-09-18T10:25:55.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa59cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:55.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.275+0000 7fa59cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:55.284 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:55.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 msgr2=0x7fa580037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 0x7fa580037f20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa590009510 tx=0x7fa5900091a0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 10 monclient: shutdown 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 msgr2=0x7fa5a00940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa588006fd0 tx=0x7fa588017000 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 shutdown_connections 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa580035a70 0x7fa580037f20 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5a009c320 0x7fa5a008bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5a009aae0 0x7fa5a00940f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 --2- 172.21.15.142:0/2373177873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5a00967a0 0x7fa5a01357c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:55.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 >> 172.21.15.142:0/2373177873 conn(0x7fa5a00a1fe0 msgr2=0x7fa5a006c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:55.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 shutdown_connections 2024-09-18T10:25:55.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:55.279+0000 7fa5a5379700 1 -- 172.21.15.142:0/2373177873 wait complete. 2024-09-18T10:25:58.304 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: get_monmap_and_config 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: build_initial_monmap 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: monmap: 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:58.414 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:25:58.411964+0000 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:25:58.411964+0000 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:25:58.415 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 Processor -- start 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 -- start start 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: init 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f606c0a15e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 0x7f606c135470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a20c0 addr v2:172.21.15.142:3300/0 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): picked mon.noname-b con 0x7f606c1359b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:58.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c096a50 con 0x7f606c0a15e0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f606c06d990 con 0x7f606c0a20c0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c06d440 con 0x7f606c1359b0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f607368f700 10 monclient(hunting): authenticate will time out at 1847.337438s 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6070c2a700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 msgr2=0x7f606c135470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6070c2a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 0x7f606c135470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50368/0 (socket says 172.21.15.142:50368) 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 -- 172.21.15.142:0/4078770313 learned_addr learned my addr 172.21.15.142:0/4078770313 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): get_auth_request con 0x7f606c1359b0 auth_method 0 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:58.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient(hunting): handle_auth_done global_id 75773 payload 274 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 -- 172.21.15.142:0/4078770313 >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 msgr2=0x7f606c135470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 --2- 172.21.15.142:0/4078770313 >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 0x7f606c135470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 -- 172.21.15.142:0/4078770313 >> v1:172.21.15.150:6789/0 conn(0x7f606c0a15e0 legacy=0x7f606c0a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient: _finish_hunting 0 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 monclient: found mon.noname-b 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:25:58.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.407+0000 7f6071c2c700 1 -- 172.21.15.142:0/4078770313 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c06d6b0 con 0x7f606c1359b0 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f6071c2c700 1 -- 172.21.15.142:0/4078770313 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 msgr2=0x7f606c137d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f6071c2c700 1 --2- 172.21.15.142:0/4078770313 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0813253221ccdd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 1 -- 172.21.15.142:0/4078770313 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606400bb50 con 0x7f606c1359b0 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: dump: 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:58.423 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: _finish_auth 0 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 1 -- 172.21.15.142:0/4078770313 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606400bcb0 con 0x7f606c1359b0 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 5 monclient: authenticate success, global_id 75773 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 1 -- 172.21.15.142:0/4078770313 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064011670 con 0x7f606c1359b0 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 10 monclient: get_monmap_and_config success 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 4 set_mon_vals no callback set 2024-09-18T10:25:58.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f606bfff700 10 monclient: dump: 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 10 monclient: shutdown 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 -- 172.21.15.142:0/4078770313 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 msgr2=0x7f606c137d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 --2- 172.21.15.142:0/4078770313 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f60640099b0 tx=0x7f6064009250 comp rx=0 tx=0).stop 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 -- 172.21.15.142:0/4078770313 shutdown_connections 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 --2- 172.21.15.142:0/4078770313 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c1359b0 0x7f606c137d90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 --2- 172.21.15.142:0/4078770313 >> v2:172.21.15.142:3300/0 conn(0x7f606c0a20c0 0x7f606c135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 -- 172.21.15.142:0/4078770313 >> 172.21.15.142:0/4078770313 conn(0x7f606c071460 msgr2=0x7f606c0a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 -- 172.21.15.142:0/4078770313 shutdown_connections 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 1 -- 172.21.15.142:0/4078770313 wait complete. 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 10 monclient: build_initial_monmap 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.411+0000 7f607368f700 10 monclient: monmap: 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:25:58.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 Processor -- start 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 -- start start 2024-09-18T10:25:58.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient: init 2024-09-18T10:25:58.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient: _reopen_session rank -1 2024-09-18T10:25:58.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:25:58.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f606c0a15e0 0x7f606c09d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): picked mon.noname-b con 0x7f606c0a15e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): picked mon.noname-c con 0x7f606c0a20c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 0x7f606c097680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): picked mon.noname-a con 0x7f606c1359b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c09b450 con 0x7f606c1359b0 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c09dcf0 con 0x7f606c0a15e0 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f606c09cbb0 con 0x7f606c0a20c0 2024-09-18T10:25:58.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): _renew_subs 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607368f700 10 monclient(hunting): authenticate will time out at 1847.346015s 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6071c2c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 msgr2=0x7f606c097680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6071c2c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 0x7f606c097680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50378/0 (socket says 172.21.15.142:50378) 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 -- 172.21.15.142:0/482462843 learned_addr learned my addr 172.21.15.142:0/482462843 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): get_auth_request con 0x7f606c0a20c0 auth_method 0 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f606c0a15e0 0x7f606c09d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): get_auth_request con 0x7f606c0a15e0 auth_method 0 2024-09-18T10:25:58.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f607142b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient(hunting): handle_auth_done global_id 75776 payload 824 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 -- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f606c0a15e0 msgr2=0x7f606c09d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f606c0a15e0 0x7f606c09d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 -- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 msgr2=0x7f606c097680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:25:58.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 0x7f606c097680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient: _finish_hunting 0 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 monclient: found mon.noname-c 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.415+0000 7f6070c2a700 1 -- 172.21.15.142:0/482462843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60640094f0 con 0x7f606c0a20c0 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f6070c2a700 1 -- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 msgr2=0x7f606c097140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f6070c2a700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f605c00bbb0 tx=0x7f605c0097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f6070c2a700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f605c00bbb0 tx=0x7f605c0097f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f605c00cad0 mon_map magic: 0 v1 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c00cad0 con 0x7f606c0a20c0 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: dump: 2024-09-18T10:25:58.435 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: _finish_auth 0 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605c00cc30 con 0x7f606c0a20c0 2024-09-18T10:25:58.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 5 monclient: authenticate success, global_id 75776 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _renew_subs 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c099150 con 0x7f606c0a20c0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f606b7fe700 4 set_mon_vals no callback set 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _renew_subs 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c098c40 con 0x7f606c0a20c0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c0696a0 con 0x7f606c0a20c0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c00cde0 con 0x7f606c0a20c0 2024-09-18T10:25:58.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 10 monclient: dump: 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:25:58.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f605c012020 con 0x7f606c0a20c0 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.419+0000 7f60697fa700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 0x7f6058037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.423+0000 7f607142b700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 0x7f6058037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.423+0000 7f607142b700 10 monclient: get_auth_request con 0x7f6058035a70 auth_method 0 2024-09-18T10:25:58.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.423+0000 7f607142b700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 0x7f6058037f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6060003a30 tx=0x7f6060007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:25:58.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:58.431+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f605c048dd0 con 0x7f606c0a20c0 2024-09-18T10:25:59.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.043+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f605c01d900 con 0x7f606c0a20c0 2024-09-18T10:25:59.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.043+0000 7f60697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:25:59.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.043+0000 7f60697fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:25:59.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f607368f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f607368f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f607368f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f606c09df20 con 0x7f606c0a20c0 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f60697fa700 1 -- 172.21.15.142:0/482462843 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7f605c014340 con 0x7f606c0a20c0 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f60697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:25:59.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.271+0000 7f60697fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:25:59.281 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:25:59.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 msgr2=0x7f6058037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 0x7f6058037f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6060003a30 tx=0x7f6060007ea0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 10 monclient: shutdown 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 msgr2=0x7f606c097140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f605c00bbb0 tx=0x7f605c0097f0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 shutdown_connections 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6058035a70 0x7f6058037f20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f606c1359b0 0x7f606c097680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f606c0a20c0 0x7f606c097140 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 --2- 172.21.15.142:0/482462843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f606c0a15e0 0x7f606c09d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:25:59.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 >> 172.21.15.142:0/482462843 conn(0x7f606c071460 msgr2=0x7f606c12e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:25:59.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 shutdown_connections 2024-09-18T10:25:59.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:25:59.275+0000 7f607368f700 1 -- 172.21.15.142:0/482462843 wait complete. 2024-09-18T10:26:02.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:02.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: get_monmap_and_config 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: build_initial_monmap 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: monmap: 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:02.408392+0000 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:02.408392+0000 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:02.411 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:02.412 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:02.412 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:02.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 Processor -- start 2024-09-18T10:26:02.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 -- start start 2024-09-18T10:26:02.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: init 2024-09-18T10:26:02.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-c con 0x7f0d6009d920 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-b con 0x7f0d600953c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-a con 0x7f0d600957d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6006fba0 con 0x7f0d600957d0 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6009b0b0 con 0x7f0d6009d920 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:02.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6009a7e0 con 0x7f0d600953c0 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d65700700 10 monclient(hunting): authenticate will time out at 1851.333910s 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d64eff700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 msgr2=0x7f0d600971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d64eff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33190/0 (socket says 172.21.15.142:33190) 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 learned_addr learned my addr 172.21.15.142:0/3213278854 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600953c0 auth_method 0 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:02.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d6009d920 auth_method 0 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:02.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.407+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:02.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d64eff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 msgr2=0x7f0d600971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:02.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d64eff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:02.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:02.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600953c0 auth_method 0 2024-09-18T10:26:02.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d6009d920 auth_method 0 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:02.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:02.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:02.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:02.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:02.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:02.611+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d64eff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 msgr2=0x7f0d600971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d64eff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600953c0 auth_method 0 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:03.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d6009d920 auth_method 0 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:03.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:03.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.015+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d64eff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 msgr2=0x7f0d600971f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d64eff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600953c0 auth_method 0 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:03.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d6009d920 auth_method 0 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:03.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:03.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:03.819+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:05.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient(hunting): tick 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 msgr2=0x7f0d60096cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d600953c0 0x7f0d60096cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 msgr2=0x7f0d6006f660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.150:3300/0 conn(0x7f0d6009d920 0x7f0d6006f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 -- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 msgr2=0x7f0d600971f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 --2- 172.21.15.142:0/3213278854 >> v2:172.21.15.142:3300/0 conn(0x7f0d600957d0 0x7f0d600971f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient: _add_conns ranks=[1,0] 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d400036e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:05.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d40007400 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:05.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 -- 172.21.15.142:0/3213278854 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6009aaf0 con 0x7f0d40007400 2024-09-18T10:26:05.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 1 -- 172.21.15.142:0/3213278854 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4000c130 con 0x7f0d400036e0 2024-09-18T10:26:05.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> v1:172.21.15.142:6789/0 conn(0x7f0d40007400 legacy=0x7f0d400098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:26:05.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.407+0000 7f0d5dffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.411+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> v1:172.21.15.150:6789/0 conn(0x7f0d400036e0 legacy=0x7f0d40005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.411+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:26:05.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.611+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/3213278854 >> v1:172.21.15.142:6789/0 conn(0x7f0d40007400 legacy=0x7f0d400098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:26:05.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161073846 0 0) 0x7f0d4000c130 con 0x7f0d400036e0 2024-09-18T10:26:05.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:05.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:05.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient(hunting): my global_id is 194164 2024-09-18T10:26:05.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c003570 con 0x7f0d400036e0 2024-09-18T10:26:05.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486482215 0 0) 0x7f0d4c003570 con 0x7f0d400036e0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 >> v1:172.21.15.142:6789/0 conn(0x7f0d40007400 legacy=0x7f0d400098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: _finish_hunting 0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4000d350 con 0x7f0d400036e0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: _finish_auth 0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f0d5000d480 con 0x7f0d400036e0 2024-09-18T10:26:05.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d65700700 5 monclient: authenticate success, global_id 194164 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: dump: 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:05.750 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 >> v1:172.21.15.150:6789/0 conn(0x7f0d400036e0 legacy=0x7f0d40005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d64eff700 1 -- 172.21.15.142:0/3213278854 reap_dead start 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 0x7f0d4c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.739+0000 7f0d5effd700 10 monclient(hunting): picked mon.b con 0x7f0d4c003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): picked mon.c con 0x7f0d4c004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:05.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 0x7f0d4c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): picked mon.a con 0x7f0d4c0053c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c00c2a0 con 0x7f0d4c0053c0 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c00c410 con 0x7f0d4c003a60 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c00c5d0 con 0x7f0d4c004810 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5effd700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:26:05.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d65700700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d64eff700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 msgr2=0x7f0d4c00ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d64eff700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 0x7f0d4c00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 0x7f0d4c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d4c003a60 auth_method 0 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d4c004810 auth_method 0 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:05.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:05.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_done global_id 194164 payload 293 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 msgr2=0x7f0d4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 msgr2=0x7f0d4c00ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 0x7f0d4c00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 monclient: found mon.b 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c003570 con 0x7f0d4c003a60 2024-09-18T10:26:05.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 10 monclient: _finish_auth 0 2024-09-18T10:26:05.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:05.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d4c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:26:05.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 msgr2=0x7f0d4c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.743+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 0x7f0d4c003e30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000d5e0 tx=0x7f0d50008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f687cace400c1a35 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50013d20 con 0x7f0d4c003a60 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: dump: 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d50013e80 con 0x7f0d4c003a60 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 4 set_mon_vals no callback set 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 1 -- 172.21.15.142:0/3213278854 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d500166b0 con 0x7f0d4c003a60 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d5effd700 10 monclient: dump: 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 10 monclient: shutdown 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 -- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 msgr2=0x7f0d4c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 0x7f0d4c003e30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000d5e0 tx=0x7f0d50008040 comp rx=0 tx=0).stop 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 -- 172.21.15.142:0/3213278854 shutdown_connections 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c0053c0 0x7f0d4c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d4c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.747+0000 7f0d65700700 1 --2- 172.21.15.142:0/3213278854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c003a60 0x7f0d4c003e30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 -- 172.21.15.142:0/3213278854 >> 172.21.15.142:0/3213278854 conn(0x7f0d600a1fe0 msgr2=0x7f0d60070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 -- 172.21.15.142:0/3213278854 shutdown_connections 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 -- 172.21.15.142:0/3213278854 wait complete. 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient: build_initial_monmap 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient: monmap: 2024-09-18T10:26:05.767 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:05.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:05.769 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 Processor -- start 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 -- start start 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient: init 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 0x7f0d60140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-a con 0x7f0d4c003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:05.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-c con 0x7f0d4c004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 0x7f0d6008c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d65700700 10 monclient(hunting): picked mon.noname-b con 0x7f0d4c0053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6008e0b0 con 0x7f0d4c003a60 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.751+0000 7f0d5ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 msgr2=0x7f0d60140c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6008fc70 con 0x7f0d4c0053c0 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d60141ac0 con 0x7f0d4c004810 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:05.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d65700700 10 monclient(hunting): authenticate will time out at 1854.680062s 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 0x7f0d60140c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33234/0 (socket says 172.21.15.142:33234) 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/1625077855 learned_addr learned my addr 172.21.15.142:0/1625077855 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d4c004810 auth_method 0 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 0x7f0d6008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): get_auth_request con 0x7f0d4c0053c0 auth_method 0 2024-09-18T10:26:05.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_done global_id 75779 payload 824 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 msgr2=0x7f0d6008c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 0x7f0d6008c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 msgr2=0x7f0d60140c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:05.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 0x7f0d60140c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 monclient: found mon.noname-c 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/1625077855 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6008f990 con 0x7f0d4c004810 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d64eff700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 0x7f0d6008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 -- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 msgr2=0x7f0d600944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:05.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.755+0000 7f0d5f7fe700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0d48013b00 tx=0x7f0d480136c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:05.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48018070 con 0x7f0d4c004810 2024-09-18T10:26:05.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:05.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: dump: 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: _finish_auth 0 2024-09-18T10:26:05.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 5 monclient: authenticate success, global_id 75779 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4800e5d0 con 0x7f0d4c004810 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _renew_subs 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4800e970 con 0x7f0d4c004810 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6008ff50 con 0x7f0d4c004810 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d5e7fc700 4 set_mon_vals no callback set 2024-09-18T10:26:05.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 10 monclient: dump: 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:05.780 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _renew_subs 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6008dc00 con 0x7f0d4c004810 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60072a80 con 0x7f0d4c004810 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0d4800ece0 con 0x7f0d4c004810 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 0x7f0d40038830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0d48049e20 con 0x7f0d4c004810 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 0x7f0d40038830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.759+0000 7f0d5ffff700 10 monclient: get_auth_request con 0x7f0d40036380 auth_method 0 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.763+0000 7f0d5ffff700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 0x7f0d40038830 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000d310 tx=0x7f0d50020000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.767+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0d4800f370 con 0x7f0d4c004810 2024-09-18T10:26:05.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.767+0000 7f0d57fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:05.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.767+0000 7f0d57fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:06.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d65700700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:26:06.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d65700700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:06.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d65700700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.000 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0d600678a0 con 0x7f0d4c004810 2024-09-18T10:26:06.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d57fff700 1 -- 172.21.15.142:0/1625077855 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f0d48021030 con 0x7f0d4c004810 2024-09-18T10:26:06.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d57fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:06.001 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.995+0000 7f0d57fff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:06.001 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":156,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:26:06.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 msgr2=0x7f0d40038830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 0x7f0d40038830 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000d310 tx=0x7f0d50020000 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 10 monclient: shutdown 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 msgr2=0x7f0d600944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0d48013b00 tx=0x7f0d480136c0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 shutdown_connections 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0d40036380 0x7f0d40038830 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d4c0053c0 0x7f0d6008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0d4c004810 0x7f0d600944b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 --2- 172.21.15.142:0/1625077855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d4c003a60 0x7f0d60140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 >> 172.21.15.142:0/1625077855 conn(0x7f0d600a1fe0 msgr2=0x7f0d600987d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 shutdown_connections 2024-09-18T10:26:06.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:05.999+0000 7f0d65700700 1 -- 172.21.15.142:0/1625077855 wait complete. 2024-09-18T10:26:06.014 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:26:06.015 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:26:06.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:26:06.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.111+0000 7fd352ad7700 10 monclient: get_monmap_and_config 2024-09-18T10:26:06.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.111+0000 7fd352ad7700 10 monclient: build_initial_monmap 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.111+0000 7fd352ad7700 10 monclient: monmap: 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:06.115319+0000 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:06.115319+0000 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:06.118 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 Processor -- start 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 -- start start 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient: init 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-b con 0x7fd34c09abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd34c09b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 0x7fd34c137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-c con 0x7fd34c09c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34c09dc90 con 0x7fd34c09b740 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd34c09b500 con 0x7fd34c09c5e0 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c09afa0 con 0x7fd34c09abd0 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd352ad7700 10 monclient(hunting): authenticate will time out at 1855.040721s 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33236/0 (socket says 172.21.15.142:33236) 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 -- 172.21.15.142:0/1464921003 learned_addr learned my addr 172.21.15.142:0/1464921003 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): get_auth_request con 0x7fd34c09abd0 auth_method 0 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 1 --2- 172.21.15.142:0/1464921003 >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 0x7fd34c137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 10 monclient(hunting): get_auth_request con 0x7fd34c09c5e0 auth_method 0 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient(hunting): handle_auth_done global_id 75782 payload 274 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 -- 172.21.15.142:0/1464921003 >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 msgr2=0x7fd34c137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 --2- 172.21.15.142:0/1464921003 >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 0x7fd34c137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 -- 172.21.15.142:0/1464921003 >> v1:172.21.15.150:6789/0 conn(0x7fd34c09b740 legacy=0x7fd34c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient: _finish_hunting 0 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 monclient: found mon.noname-b 2024-09-18T10:26:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 -- 172.21.15.142:0/1464921003 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34c06f810 con 0x7fd34c09abd0 2024-09-18T10:26:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd3522d6700 1 --2- 172.21.15.142:0/1464921003 >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 0x7fd34c137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:26:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 -- 172.21.15.142:0/1464921003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 msgr2=0x7fd34c0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.115+0000 7fd351ad5700 1 --2- 172.21.15.142:0/1464921003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd3480099b0 tx=0x7fd348009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd7244277c2593c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 1 -- 172.21.15.142:0/1464921003 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34800ba70 con 0x7fd34c09abd0 2024-09-18T10:26:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: dump: 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: _finish_auth 0 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 5 monclient: authenticate success, global_id 75782 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 1 -- 172.21.15.142:0/1464921003 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34800bbd0 con 0x7fd34c09abd0 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 4 set_mon_vals no callback set 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 1 -- 172.21.15.142:0/1464921003 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011420 con 0x7fd34c09abd0 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd350ad3700 10 monclient: dump: 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.133 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: shutdown 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- 172.21.15.142:0/1464921003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 msgr2=0x7fd34c0960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- 172.21.15.142:0/1464921003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd3480099b0 tx=0x7fd348009250 comp rx=0 tx=0).stop 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- 172.21.15.142:0/1464921003 shutdown_connections 2024-09-18T10:26:06.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- 172.21.15.142:0/1464921003 >> v2:172.21.15.150:3300/0 conn(0x7fd34c09c5e0 0x7fd34c137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- 172.21.15.142:0/1464921003 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09abd0 0x7fd34c0960e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- 172.21.15.142:0/1464921003 >> 172.21.15.142:0/1464921003 conn(0x7fd34c070ef0 msgr2=0x7fd34c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- 172.21.15.142:0/1464921003 shutdown_connections 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- 172.21.15.142:0/1464921003 wait complete. 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: build_initial_monmap 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: monmap: 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:06.135 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 Processor -- start 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- start start 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: init 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd34c09abd0 0x7fd34c13f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-b con 0x7fd34c09abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 0x7fd34c13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-c con 0x7fd34c09b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd34c09c5e0 0x7fd34c1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): picked mon.noname-a con 0x7fd34c09c5e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c08dbf0 con 0x7fd34c09c5e0 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c09d240 con 0x7fd34c09abd0 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c09fdc0 con 0x7fd34c09b740 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd352ad7700 10 monclient(hunting): authenticate will time out at 1855.047033s 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3522d6700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd34c09c5e0 msgr2=0x7fd34c1401c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3522d6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd34c09c5e0 0x7fd34c1401c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 0x7fd34c13fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 0x7fd34c13fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33240/0 (socket says 172.21.15.142:33240) 2024-09-18T10:26:06.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 -- 172.21.15.142:0/141741229 learned_addr learned my addr 172.21.15.142:0/141741229 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): get_auth_request con 0x7fd34c09b740 auth_method 0 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd34c09abd0 0x7fd34c13f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): get_auth_request con 0x7fd34c09abd0 auth_method 0 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient(hunting): handle_auth_done global_id 75785 payload 824 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd34c09abd0 msgr2=0x7fd34c13f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd34c09abd0 0x7fd34c13f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd34c09c5e0 msgr2=0x7fd34c1401c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd34c09c5e0 0x7fd34c1401c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient: _finish_hunting 0 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 monclient: found mon.noname-c 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.119+0000 7fd3512d4700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480094f0 con 0x7fd34c09b740 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd3512d4700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 msgr2=0x7fd34c13fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd3512d4700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 0x7fd34c13fc80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c600 tx=0x7fd33c007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c0117a0 con 0x7fd34c09b740 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: dump: 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.140 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: _finish_auth 0 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33c011900 con 0x7fd34c09b740 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 5 monclient: authenticate success, global_id 75785 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 monclient: _renew_subs 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c090480 con 0x7fd34c09b740 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c011cc0 con 0x7fd34c09b740 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 10 monclient: dump: 2024-09-18T10:26:06.141 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 monclient: _renew_subs 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd343fff700 4 set_mon_vals no callback set 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c08f130 con 0x7fd34c09b740 2024-09-18T10:26:06.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd33c012630 con 0x7fd34c09b740 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 0x7fd34c0883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.123+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c0648a0 con 0x7fd34c087fe0 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd330035a70 0x7fd330037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 0x7fd34c0883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd33c048d60 con 0x7fd34c09b740 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient: get_auth_request con 0x7fd34c087fe0 auth_method 0 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd3522d6700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd330035a70 0x7fd330037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd3522d6700 10 monclient: get_auth_request con 0x7fd330035a70 auth_method 0 2024-09-18T10:26:06.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 10 monclient(hunting): handle_auth_done global_id 75788 payload 824 2024-09-18T10:26:06.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd34c08f5a0 con 0x7fd34c087fe0 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 msgr2=0x7fd34c0883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd351ad5700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 0x7fd34c0883b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd348015000 tx=0x7fd348016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd3522d6700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd330035a70 0x7fd330037f20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3440094d0 tx=0x7fd344009730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348020070 con 0x7fd34c087fe0 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.127+0000 7fd341ffb700 10 monclient: dump: 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.145 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd348000ca0 con 0x7fd34c087fe0 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 msgr2=0x7fd34c0883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 0x7fd34c0883b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd348015000 tx=0x7fd348016000 comp rx=0 tx=0).stop 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 msgr2=0x7fd34c0883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:26:06.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.131+0000 7fd341ffb700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c087fe0 0x7fd34c0883b0 secure :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd348015000 tx=0x7fd348016000 comp rx=0 tx=0).stop 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd352ad7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd352ad7700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 0x7fd34c08bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd352ad7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c072a80 con 0x7fd34c0888f0 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 0x7fd34c08bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient: get_auth_request con 0x7fd34c0888f0 auth_method 0 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 10 monclient(hunting): handle_auth_done global_id 75791 payload 824 2024-09-18T10:26:06.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.143+0000 7fd351ad5700 1 -- 172.21.15.142:0/141741229 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd34c0008d0 con 0x7fd34c0888f0 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd351ad5700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 msgr2=0x7fd34c08bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd351ad5700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 0x7fd34c08bb20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd3480030c0 tx=0x7fd34801d050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348037070 con 0x7fd34c0888f0 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:06.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 10 monclient: dump: 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.151 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7fd348033430 con 0x7fd34c0888f0 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 msgr2=0x7fd34c08bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 0x7fd34c08bb20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd3480030c0 tx=0x7fd34801d050 comp rx=0 tx=0).stop 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 msgr2=0x7fd34c08bb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:26:06.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd341ffb700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c0888f0 0x7fd34c08bb20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 156, 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.153 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.154 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.155 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:26:06.156 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.157 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.158 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.159 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:26:06.160 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:26:06.161 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.162 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.163 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.164 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:26:06.165 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd330035a70 msgr2=0x7fd330037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd330035a70 0x7fd330037f20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd3440094d0 tx=0x7fd344009730 comp rx=0 tx=0).stop 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 10 monclient: shutdown 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 msgr2=0x7fd34c13fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 --2- 172.21.15.142:0/141741229 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd34c09b740 0x7fd34c13fc80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00c600 tx=0x7fd33c007f60 comp rx=0 tx=0).stop 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd3522d6700 1 -- 172.21.15.142:0/141741229 reap_dead start 2024-09-18T10:26:06.167 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd3522d6700 1 -- 172.21.15.142:0/141741229 reap_dead start 2024-09-18T10:26:06.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 shutdown_connections 2024-09-18T10:26:06.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 >> 172.21.15.142:0/141741229 conn(0x7fd34c070ef0 msgr2=0x7fd34c09de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:06.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 shutdown_connections 2024-09-18T10:26:06.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.147+0000 7fd352ad7700 1 -- 172.21.15.142:0/141741229 wait complete. 2024-09-18T10:26:06.168 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:26:06.168 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:26:06.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:26:06.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.271+0000 7fcdabfff700 10 monclient: get_monmap_and_config 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.271+0000 7fcdabfff700 10 monclient: build_initial_monmap 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.271+0000 7fcdabfff700 10 monclient: monmap: 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:06.274976+0000 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:06.274976+0000 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:06.280 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:06.281 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:06.281 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:06.281 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:06.281 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 Processor -- start 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 -- start start 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient: init 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-18T10:26:06.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdac09bb70 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdac130800 0x7fcdac09e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-c con 0x7fcdac130800 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 0x7fcdac137440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-a con 0x7fcdac135060 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdac12fd30 con 0x7fcdac09bb70 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac130c20 con 0x7fcdac135060 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdac09bf40 con 0x7fcdac130800 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:06.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdabfff700 10 monclient(hunting): authenticate will time out at 1855.201042s 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdab7fe700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 msgr2=0x7fcdac137440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcdab7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 0x7fcdac137440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdac130800 0x7fcdac09e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcdac130800 0x7fcdac09e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:52564/0 (socket says 172.21.15.142:52564) 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 -- 172.21.15.142:0/3083186773 learned_addr learned my addr 172.21.15.142:0/3083186773 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): get_auth_request con 0x7fcdac130800 auth_method 0 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient(hunting): handle_auth_done global_id 194176 payload 274 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 -- 172.21.15.142:0/3083186773 >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 msgr2=0x7fcdac137440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 --2- 172.21.15.142:0/3083186773 >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 0x7fcdac137440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 -- 172.21.15.142:0/3083186773 >> v1:172.21.15.150:6789/0 conn(0x7fcdac09bb70 legacy=0x7fcdac1300f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient: _finish_hunting 0 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 monclient: found mon.noname-c 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.275+0000 7fcda27fc700 1 -- 172.21.15.142:0/3083186773 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac09d960 con 0x7fcdac130800 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcda27fc700 1 -- 172.21.15.142:0/3083186773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 msgr2=0x7fcdac09e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcda27fc700 1 --2- 172.21.15.142:0/3083186773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac09e810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdac06ba30 tx=0x7fcd94009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb8cfa5c9b0b75f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 1 -- 172.21.15.142:0/3083186773 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400ba70 con 0x7fcdac130800 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:06.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: dump: 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 1 -- 172.21.15.142:0/3083186773 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9400bbd0 con 0x7fcdac130800 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 1 -- 172.21.15.142:0/3083186773 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011420 con 0x7fcdac130800 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdaa7fc700 10 monclient: dump: 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.290 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 5 monclient: authenticate success, global_id 194176 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 4 set_mon_vals no callback set 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 10 monclient: shutdown 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- 172.21.15.142:0/3083186773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 msgr2=0x7fcdac09e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 --2- 172.21.15.142:0/3083186773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac09e810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdac06ba30 tx=0x7fcd94009250 comp rx=0 tx=0).stop 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- 172.21.15.142:0/3083186773 shutdown_connections 2024-09-18T10:26:06.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 --2- 172.21.15.142:0/3083186773 >> v2:172.21.15.142:3300/0 conn(0x7fcdac135060 0x7fcdac137440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 --2- 172.21.15.142:0/3083186773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac09e810 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- 172.21.15.142:0/3083186773 >> 172.21.15.142:0/3083186773 conn(0x7fcdac06a2e0 msgr2=0x7fcdac0a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- 172.21.15.142:0/3083186773 shutdown_connections 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- 172.21.15.142:0/3083186773 wait complete. 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 10 monclient: build_initial_monmap 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 10 monclient: monmap: 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 Processor -- start 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 1 -- start start 2024-09-18T10:26:06.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.279+0000 7fcdabfff700 10 monclient: init 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-c con 0x7fcdac09bb70 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac095820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-b con 0x7fcdac130800 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdac135060 0x7fcdac06dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): picked mon.noname-a con 0x7fcdac135060 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac09d0a0 con 0x7fcdac135060 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac09ae70 con 0x7fcdac130800 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac06f830 con 0x7fcdac09bb70 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdabfff700 10 monclient(hunting): authenticate will time out at 1855.208269s 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdab7fe700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdac135060 msgr2=0x7fcdac06dd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdab7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdac135060 0x7fcdac06dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:33270/0 (socket says 172.21.15.142:33270) 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 -- 172.21.15.142:0/1049945842 learned_addr learned my addr 172.21.15.142:0/1049945842 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): get_auth_request con 0x7fcdac09bb70 auth_method 0 2024-09-18T10:26:06.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac095820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): get_auth_request con 0x7fcdac130800 auth_method 0 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient(hunting): handle_auth_done global_id 75794 payload 824 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 msgr2=0x7fcdac095820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac130800 0x7fcdac095820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdac135060 msgr2=0x7fcdac06dd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdac135060 0x7fcdac06dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient: _finish_hunting 0 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 monclient: found mon.noname-c 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd940094f0 con 0x7fcdac09bb70 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 msgr2=0x7fcdac0952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.283+0000 7fcdaaffd700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00b5e0 tx=0x7fcd9c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00f720 con 0x7fcdac09bb70 2024-09-18T10:26:06.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: dump: 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: _finish_auth 0 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c00f880 con 0x7fcdac09bb70 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00fc40 con 0x7fcdac09bb70 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda9ffb700 4 set_mon_vals no callback set 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 10 monclient: dump: 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 5 monclient: authenticate success, global_id 75794 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 monclient: _renew_subs 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac132560 con 0x7fcdac09bb70 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 monclient: _renew_subs 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac095d60 con 0x7fcdac09bb70 2024-09-18T10:26:06.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcd9c011610 con 0x7fcdac09bb70 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 0x7fcdac06e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac064830 con 0x7fcdac133710 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd8c035a70 0x7fcd8c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fcd9c048aa0 con 0x7fcdac09bb70 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 0x7fcdac06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient: get_auth_request con 0x7fcdac133710 auth_method 0 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdab7fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd8c035a70 0x7fcd8c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcdab7fe700 10 monclient: get_auth_request con 0x7fcd8c035a70 auth_method 0 2024-09-18T10:26:06.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 10 monclient(hunting): handle_auth_done global_id 194182 payload 824 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.287+0000 7fcda27fc700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdac04c430 con 0x7fcdac133710 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda27fc700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 msgr2=0x7fcdac06e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda27fc700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 0x7fcdac06e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdac068bf0 tx=0x7fcd94016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcdab7fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd8c035a70 0x7fcd8c037f20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcdac069120 tx=0x7fcda4009370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94021070 con 0x7fcdac133710 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 10 monclient: dump: 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.306 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcd9401c660 con 0x7fcdac133710 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 msgr2=0x7fcdac06e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 0x7fcdac06e280 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdac068bf0 tx=0x7fcd94016000 comp rx=0 tx=0).stop 2024-09-18T10:26:06.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 msgr2=0x7fcdac06e280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:26:06.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.291+0000 7fcda37fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac133710 0x7fcdac06e280 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdac068bf0 tx=0x7fcd94016000 comp rx=0 tx=0).stop 2024-09-18T10:26:06.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.299+0000 7fcdabfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:26:06.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.299+0000 7fcdabfff700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 0x7fcdac06ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:06.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.299+0000 7fcdabfff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:06.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.299+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1326f0 con 0x7fcdac06e7c0 2024-09-18T10:26:06.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 0x7fcdac06ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient: get_auth_request con 0x7fcdac06e7c0 auth_method 0 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 10 monclient(hunting): handle_auth_done global_id 194185 payload 824 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 1 -- 172.21.15.142:0/1049945842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcdac06f100 con 0x7fcdac06e7c0 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 msgr2=0x7fcdac06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda27fc700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 0x7fcdac06ebc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcdac1312d0 tx=0x7fcd9401e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94033460 con 0x7fcdac06e7c0 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:06.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.303+0000 7fcda37fe700 10 monclient: dump: 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:06.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7fcd9401b440 con 0x7fcdac06e7c0 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 msgr2=0x7fcdac06ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 0x7fcdac06ebc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcdac1312d0 tx=0x7fcd9401e070 comp rx=0 tx=0).stop 2024-09-18T10:26:06.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 msgr2=0x7fcdac06ebc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcda37fe700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdac06e7c0 0x7fcdac06ebc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 156, 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:26:06.313 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.314 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.315 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:26:06.316 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:26:06.317 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.318 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.319 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.320 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:26:06.321 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:26:06.322 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:26:06.323 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:26:06.324 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:26:06.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd8c035a70 msgr2=0x7fcd8c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd8c035a70 0x7fcd8c037f20 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcdac069120 tx=0x7fcda4009370 comp rx=0 tx=0).stop 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 10 monclient: shutdown 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 msgr2=0x7fcdac0952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdab7fe700 1 -- 172.21.15.142:0/1049945842 reap_dead start 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00b5e0 tx=0x7fcd9c009120 comp rx=0 tx=0).stop 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 shutdown_connections 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 --2- 172.21.15.142:0/1049945842 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdac09bb70 0x7fcdac0952e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 >> 172.21.15.142:0/1049945842 conn(0x7fcdac06a2e0 msgr2=0x7fcdac070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 shutdown_connections 2024-09-18T10:26:06.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:06.307+0000 7fcdabfff700 1 -- 172.21.15.142:0/1049945842 wait complete. 2024-09-18T10:26:06.327 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:26:06.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:26:26.327 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:26:26.328 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:26:26.328 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:26:26.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:26:26.332 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:26:26.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:26:26.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:26.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 10 monclient: get_monmap_and_config 2024-09-18T10:26:26.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 10 monclient: build_initial_monmap 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 10 monclient: monmap: 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:26.430609+0000 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:26.430609+0000 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:26.436 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 1 Processor -- start 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 1 -- start start 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.427+0000 7f641a151700 10 monclient: init 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): picked mon.noname-a con 0x7f641409aae0 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6414097c80 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f64140980c0 0x7f641409c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): picked mon.noname-c con 0x7f64140980c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641409d4d0 con 0x7f6414097c80 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641409aeb0 con 0x7f641409aae0 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641409d8f0 con 0x7f64140980c0 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641a151700 10 monclient(hunting): authenticate will time out at 1875.356218s 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641894e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6414097c80 0x7f641409bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:59140/0 (socket says 172.21.15.142:59140) 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641894e700 1 -- 172.21.15.142:0/369003013 learned_addr learned my addr 172.21.15.142:0/369003013 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 10 monclient(hunting): get_auth_request con 0x7f641409aae0 auth_method 0 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:26.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 --2- 172.21.15.142:0/369003013 >> v2:172.21.15.150:3300/0 conn(0x7f64140980c0 0x7f641409c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): get_auth_request con 0x7f64140980c0 auth_method 0 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 1 -- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 msgr2=0x7f641412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 1 -- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 msgr2=0x7f641412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 1 --2- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f641914f700 1 --2- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:26.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient(hunting): handle_auth_done global_id 194188 payload 274 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 -- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 msgr2=0x7f641412feb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 --2- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 -- 172.21.15.142:0/369003013 >> v1:172.21.15.142:6789/0 conn(0x7f6414097c80 legacy=0x7f641409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient: _finish_hunting 0 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 monclient: found mon.noname-c 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:26:26.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 -- 172.21.15.142:0/369003013 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414095460 con 0x7f64140980c0 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 -- 172.21.15.142:0/369003013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64140980c0 msgr2=0x7f641409c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f6419950700 1 --2- 172.21.15.142:0/369003013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64140980c0 0x7f641409c4f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f641409b7e0 tx=0x7f640c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd7bdc47a79c0cc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f640bfff700 1 -- 172.21.15.142:0/369003013 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c00e070 con 0x7f64140980c0 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f640bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f640bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.431+0000 7f640bfff700 10 monclient: dump: 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:26.445 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 10 monclient: _finish_auth 0 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 1 -- 172.21.15.142:0/369003013 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c00be40 con 0x7f64140980c0 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 1 -- 172.21.15.142:0/369003013 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c012660 con 0x7f64140980c0 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f640bfff700 10 monclient: dump: 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:26.446 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 5 monclient: authenticate success, global_id 194188 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 4 set_mon_vals no callback set 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: shutdown 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:26.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- 172.21.15.142:0/369003013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64140980c0 msgr2=0x7f641409c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- 172.21.15.142:0/369003013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64140980c0 0x7f641409c4f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f641409b7e0 tx=0x7f640c009780 comp rx=0 tx=0).stop 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- 172.21.15.142:0/369003013 shutdown_connections 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- 172.21.15.142:0/369003013 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64140980c0 0x7f641409c4f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- 172.21.15.142:0/369003013 >> v2:172.21.15.142:3300/0 conn(0x7f641409aae0 0x7f641412feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- 172.21.15.142:0/369003013 >> 172.21.15.142:0/369003013 conn(0x7f64140a13f0 msgr2=0x7f64140a17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- 172.21.15.142:0/369003013 shutdown_connections 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- 172.21.15.142:0/369003013 wait complete. 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: build_initial_monmap 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: monmap: 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:26.448 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 Processor -- start 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- start start 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: init 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): picked mon.noname-c con 0x7f6414097c80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:26.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): picked mon.noname-a con 0x7f64140980c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 0x7f641406b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): picked mon.noname-b con 0x7f641409aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641406c9c0 con 0x7f64140980c0 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641406e740 con 0x7f641409aae0 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6414097270 con 0x7f6414097c80 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641a151700 10 monclient(hunting): authenticate will time out at 1875.363319s 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35790/0 (socket says 172.21.15.142:35790) 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:44218/0 (socket says 172.21.15.142:44218) 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 -- 172.21.15.142:0/3807913092 learned_addr learned my addr 172.21.15.142:0/3807913092 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): get_auth_request con 0x7f6414097c80 auth_method 0 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 10 monclient(hunting): get_auth_request con 0x7f64140980c0 auth_method 0 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 msgr2=0x7f641406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:26.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 msgr2=0x7f641406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641894e700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f6419950700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 0x7f641406b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f6419950700 10 monclient(hunting): get_auth_request con 0x7f641409aae0 auth_method 0 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f6419950700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f6419950700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f6419950700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient(hunting): handle_auth_done global_id 75797 payload 824 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 msgr2=0x7f641406b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 0x7f641406b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 msgr2=0x7f641406ae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:26.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 10 monclient: _finish_hunting 0 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.435+0000 7f641914f700 1 monclient: found mon.noname-c 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641914f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641914f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641914f700 1 -- 172.21.15.142:0/3807913092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009a20 con 0x7f6414097c80 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f6419950700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 0x7f641406b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641914f700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 msgr2=0x7f641406f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641914f700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f641000d890 tx=0x7f6410008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641000b810 con 0x7f6414097c80 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: dump: 2024-09-18T10:26:26.453 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: _finish_auth 0 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 5 monclient: authenticate success, global_id 75797 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 10 monclient: _renew_subs 2024-09-18T10:26:26.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641000b990 con 0x7f6414097c80 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641406d100 con 0x7f6414097c80 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641000bd50 con 0x7f6414097c80 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f640b7fe700 4 set_mon_vals no callback set 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f64097fa700 10 monclient: dump: 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:26.455 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 10 monclient: _renew_subs 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:26.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.439+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641406d500 con 0x7f6414097c80 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641a151700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641a151700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641a151700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6414072a80 con 0x7f6414097c80 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6410014640 con 0x7f6414097c80 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f64097fa700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 0x7f63f8037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6410049700 con 0x7f6414097c80 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641894e700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 0x7f63f8037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.443+0000 7f641894e700 10 monclient: get_auth_request con 0x7f63f8035af0 auth_method 0 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.447+0000 7f641894e700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 0x7f63f8037fa0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6404007fd0 tx=0x7f6404009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:26.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.451+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f641001c360 con 0x7f6414097c80 2024-09-18T10:26:26.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.451+0000 7f64097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:26.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.451+0000 7f64097fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f641a151700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f641a151700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f641a151700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f64140678a0 con 0x7f6414097c80 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f64097fa700 1 -- 172.21.15.142:0/3807913092 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f6410011340 con 0x7f6414097c80 2024-09-18T10:26:26.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f64097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:26.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.671+0000 7f64097fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:26.676 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":156,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:26:26.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 msgr2=0x7f63f8037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 0x7f63f8037fa0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6404007fd0 tx=0x7f6404009040 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 10 monclient: shutdown 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 msgr2=0x7f641406f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f641000d890 tx=0x7f6410008040 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 shutdown_connections 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f63f8035af0 0x7f63f8037fa0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641409aae0 0x7f641406b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64140980c0 0x7f641406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 --2- 172.21.15.142:0/3807913092 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6414097c80 0x7f641406f450 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 >> 172.21.15.142:0/3807913092 conn(0x7f64140a13f0 msgr2=0x7f641409f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:26.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 shutdown_connections 2024-09-18T10:26:26.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:26.675+0000 7f641a151700 1 -- 172.21.15.142:0/3807913092 wait complete. 2024-09-18T10:26:29.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: get_monmap_and_config 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: build_initial_monmap 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: monmap: 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:29.804814+0000 2024-09-18T10:26:29.807 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:29.804814+0000 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:29.808 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 Processor -- start 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 -- start start 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: init 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-a con 0x7f782009a830 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-b con 0x7f782009ac00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:29.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f782009bed0 0x7f782009c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-c con 0x7f782009bed0 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782009afd0 con 0x7f782009a830 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7820099d50 con 0x7f782009bed0 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782009dca0 con 0x7f782009ac00 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f78276ba700 10 monclient(hunting): authenticate will time out at 1878.730457s 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35836/0 (socket says 172.21.15.142:35836) 2024-09-18T10:26:29.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 -- 172.21.15.142:0/2821301875 learned_addr learned my addr 172.21.15.142:0/2821301875 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): get_auth_request con 0x7f782009ac00 auth_method 0 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 10 monclient(hunting): get_auth_request con 0x7f782009a830 auth_method 0 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825c57700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.150:3300/0 conn(0x7f782009bed0 0x7f782009c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 1 -- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 msgr2=0x7f7820098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 1 -- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 msgr2=0x7f7820098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825c57700 10 monclient(hunting): get_auth_request con 0x7f782009bed0 auth_method 0 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825c57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825c57700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825c57700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7825456700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient(hunting): handle_auth_done global_id 75800 payload 274 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 -- 172.21.15.142:0/2821301875 >> v2:172.21.15.150:3300/0 conn(0x7f782009bed0 msgr2=0x7f782009c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.150:3300/0 conn(0x7f782009bed0 0x7f782009c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 -- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 msgr2=0x7f7820098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient: _finish_hunting 0 2024-09-18T10:26:29.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 monclient: found mon.noname-b 2024-09-18T10:26:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:26:29.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.803+0000 7f7824c55700 1 -- 172.21.15.142:0/2821301875 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7820099a70 con 0x7f782009ac00 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7824c55700 1 -- 172.21.15.142:0/2821301875 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 msgr2=0x7f7820099110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7824c55700 1 --2- 172.21.15.142:0/2821301875 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7820067860 tx=0x7f781c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6afd10ca6faa6d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 1 -- 172.21.15.142:0/2821301875 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c00e070 con 0x7f782009ac00 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:26:29.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: dump: 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: _finish_auth 0 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:29.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 5 monclient: authenticate success, global_id 75800 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 1 -- 172.21.15.142:0/2821301875 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c00bc60 con 0x7f782009ac00 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 4 set_mon_vals no callback set 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 1 -- 172.21.15.142:0/2821301875 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c0092e0 con 0x7f782009ac00 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f7817fff700 10 monclient: dump: 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.823 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 10 monclient: shutdown 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 -- 172.21.15.142:0/2821301875 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 msgr2=0x7f7820099110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 --2- 172.21.15.142:0/2821301875 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7820067860 tx=0x7f781c009840 comp rx=0 tx=0).stop 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 -- 172.21.15.142:0/2821301875 shutdown_connections 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.150:3300/0 conn(0x7f782009bed0 0x7f782009c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 --2- 172.21.15.142:0/2821301875 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009ac00 0x7f7820099110 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 --2- 172.21.15.142:0/2821301875 >> v2:172.21.15.142:3300/0 conn(0x7f782009a830 0x7f7820098bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 -- 172.21.15.142:0/2821301875 >> 172.21.15.142:0/2821301875 conn(0x7f78200a14e0 msgr2=0x7f78200a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 -- 172.21.15.142:0/2821301875 shutdown_connections 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 1 -- 172.21.15.142:0/2821301875 wait complete. 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 10 monclient: build_initial_monmap 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.807+0000 7f78276ba700 10 monclient: monmap: 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.825 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 Processor -- start 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 -- start start 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient: init 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-a con 0x7f782009a830 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782009ac00 0x7f782006c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-b con 0x7f782009ac00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): picked mon.noname-c con 0x7f782009bed0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782006e880 con 0x7f782009a830 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7820126920 con 0x7f782009ac00 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7820126c00 con 0x7f782009bed0 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f78276ba700 10 monclient(hunting): authenticate will time out at 1878.737531s 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35848/0 (socket says 172.21.15.142:35848) 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 -- 172.21.15.142:0/3726301662 learned_addr learned my addr 172.21.15.142:0/3726301662 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:29.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): get_auth_request con 0x7f782009bed0 auth_method 0 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 10 monclient(hunting): get_auth_request con 0x7f782009a830 auth_method 0 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782009ac00 0x7f782006c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): get_auth_request con 0x7f782009ac00 auth_method 0 2024-09-18T10:26:29.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 msgr2=0x7f7820146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 msgr2=0x7f7820146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825456700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7824c55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient(hunting): handle_auth_done global_id 75803 payload 824 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782009ac00 msgr2=0x7f782006c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782009ac00 0x7f782006c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 msgr2=0x7f7820146050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient: _finish_hunting 0 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 monclient: found mon.noname-c 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.811+0000 7f7825c57700 1 -- 172.21.15.142:0/3726301662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c009ae0 con 0x7f782009bed0 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f7825c57700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 msgr2=0x7f782006c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f7825c57700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f781800d890 tx=0x7f7818008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818012800 con 0x7f782009bed0 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: dump: 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:29.830 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: _finish_auth 0 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7818012980 con 0x7f782009bed0 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 5 monclient: authenticate success, global_id 75803 2024-09-18T10:26:29.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _renew_subs 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7820126ee0 con 0x7f782009bed0 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818012d40 con 0x7f782009bed0 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 10 monclient: dump: 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:29.832 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78177fe700 4 set_mon_vals no callback set 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _renew_subs 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7820139bc0 con 0x7f782009bed0 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7818014640 con 0x7f782009bed0 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 0x7f7804037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f78180495b0 con 0x7f782009bed0 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:29.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7820139db0 con 0x7f782009bed0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f7825456700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 0x7f7804037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.815+0000 7f7825456700 10 monclient: get_auth_request con 0x7f7804035ac0 auth_method 0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.819+0000 7f7825456700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 0x7f7804037f70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7810006f30 tx=0x7f7810008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.823+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f781801e460 con 0x7f782009bed0 2024-09-18T10:26:29.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.823+0000 7f78157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:29.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:29.823+0000 7f78157fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.047+0000 7f78276ba700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:26:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.047+0000 7f78276ba700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.047+0000 7f78276ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:30.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.047+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78200696a0 con 0x7f782009bed0 2024-09-18T10:26:30.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78157fa700 1 -- 172.21.15.142:0/3726301662 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f781802c610 con 0x7f782009bed0 2024-09-18T10:26:30.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78157fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:30.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78157fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:30.054 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":156,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 msgr2=0x7f7804037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 0x7f7804037f70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7810006f30 tx=0x7f7810008040 comp rx=0 tx=0).stop 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 10 monclient: shutdown 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 msgr2=0x7f782006c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f781800d890 tx=0x7f7818008040 comp rx=0 tx=0).stop 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 shutdown_connections 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7804035ac0 0x7f7804037f70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f782009bed0 0x7f782006c6b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782009ac00 0x7f782006c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:30.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 --2- 172.21.15.142:0/3726301662 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f782009a830 0x7f7820146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:30.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 >> 172.21.15.142:0/3726301662 conn(0x7f78200a14e0 msgr2=0x7f782009f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:30.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 shutdown_connections 2024-09-18T10:26:30.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:30.051+0000 7f78276ba700 1 -- 172.21.15.142:0/3726301662 wait complete. 2024-09-18T10:26:33.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: get_monmap_and_config 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: build_initial_monmap 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: monmap: 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:33.178079+0000 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:33.178079+0000 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:33.181 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 Processor -- start 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 -- start start 2024-09-18T10:26:33.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: init 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-a con 0x7feee009a7f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feee0099220 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feee009c4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee009abc0 con 0x7feee009c4f0 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee009d950 con 0x7feee0099220 2024-09-18T10:26:33.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feee009a7f0 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feee48d2700 10 monclient(hunting): authenticate will time out at 1882.103589s 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7feee009c4f0 0x7feee013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:42438/0 (socket says 172.21.15.142:42438) 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 learned_addr learned my addr 172.21.15.142:0/862436775 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feee009a7f0 auth_method 0 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.175+0000 7feedf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:33.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feee009c4f0 legacy=0x7feee013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:33.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.179+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feee0099220 legacy=0x7feee009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:33.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:33.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feee009a7f0 auth_method 0 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:33.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.383+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:33.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:33.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feee009a7f0 auth_method 0 2024-09-18T10:26:33.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:33.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:33.787+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:34.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feee009a7f0 auth_method 0 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:34.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:34.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): tick 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 msgr2=0x7feee0098ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feee009a7f0 0x7feee0098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feee0099220 legacy=0x7feee009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feee009c4f0 legacy=0x7feee013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient: _add_conns ranks=[2,4] 2024-09-18T10:26:36.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-b con 0x7feecc003860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-c con 0x7feecc006310 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feecc006310 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc008e60 con 0x7feecc003860 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.174+0000 7feedd7fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc006310 auth_method 0 2024-09-18T10:26:36.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc003860 auth_method 0 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:36.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc003860 auth_method 0 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc006310 auth_method 0 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc003860 auth_method 0 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc006310 auth_method 0 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:36.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:37.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc003860 auth_method 0 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc006310 auth_method 0 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:37.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:37.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:37.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:39.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): tick 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 msgr2=0x7feecc005c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc003860 0x7feecc005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 msgr2=0x7feecc0087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc006310 0x7feecc0087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 reap_dead start 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feecc003c70 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:39.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-b con 0x7feecc007e00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-c con 0x7feecc008840 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee0099a30 con 0x7feecc003c70 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feecc008840 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feecc008e60 con 0x7feecc007e00 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedd7fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc007e00 auth_method 0 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc008840 auth_method 0 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:39.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feecc003c70 legacy=0x7feecc0066e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc007e00 auth_method 0 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:39.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc008840 auth_method 0 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc007e00 auth_method 0 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc008840 auth_method 0 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:39.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:39.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feecc007e00 auth_method 0 2024-09-18T10:26:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:40.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc008840 auth_method 0 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:40.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:40.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:40.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:42.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): tick 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 msgr2=0x7feecc008270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feecc007e00 0x7feecc008270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 msgr2=0x7feecc00d2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.150:3300/0 conn(0x7feecc008840 0x7feecc00d2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feecc003c70 legacy=0x7feecc0066e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient: _add_conns ranks=[3,1] 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-a con 0x7feecc00dc70 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feecc010dd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:42.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc00eb90 con 0x7feecc010dd0 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feecc00dc70 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedd7fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc00dc70 auth_method 0 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:42.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:42.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.178+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feecc010dd0 legacy=0x7feecc0131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc00dc70 auth_method 0 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.382+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:42.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:42.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.786+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:42.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.786+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:42.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:42.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:42.786+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feecc00dc70 auth_method 0 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:43.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:43.590+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:45.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): tick 2024-09-18T10:26:45.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:45.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 msgr2=0x7feecc00e070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc00dc70 0x7feecc00e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feecc010dd0 legacy=0x7feecc0131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 reap_dead start 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feecc008270 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feecc00e040 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:45.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): picked mon.noname-a con 0x7feecc005c60 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc0158e0 con 0x7feecc008270 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feecc00e410 con 0x7feecc00e040 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 1 -- 172.21.15.142:0/862436775 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feecc005c60 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedd7fa700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc005c60 auth_method 0 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:45.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:45.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.178+0000 7feedf7fe700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feecc008270 legacy=0x7feecc0149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:45.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.182+0000 7feedeffd700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feecc00e040 legacy=0x7feecc0150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc005c60 auth_method 0 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:45.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.382+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc005c60 auth_method 0 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:45.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:45.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:45.786+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feecc005c60 auth_method 0 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:46.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:46.590+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.718+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363927141 0 0) 0x7feecc00eb90 con 0x7feecc00e040 2024-09-18T10:26:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.718+0000 7feede7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.718+0000 7feede7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.718+0000 7feede7fc700 10 monclient(hunting): my global_id is 194200 2024-09-18T10:26:47.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.718+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec8003570 con 0x7feecc00e040 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813547899 0 0) 0x7feec8003570 con 0x7feecc00e040 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 msgr2=0x7feecc0076f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 --2- 172.21.15.142:0/862436775 >> v2:172.21.15.142:3300/0 conn(0x7feecc005c60 0x7feecc0076f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.142:6789/0 conn(0x7feecc008270 legacy=0x7feecc0149d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:47.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc00d2f0 con 0x7feecc00e040 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: _finish_auth 0 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feee48d2700 5 monclient: authenticate success, global_id 194200 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7feed0001a10 con 0x7feecc00e040 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feee48d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: dump: 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.733 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 >> v1:172.21.15.150:6789/0 conn(0x7feecc00e040 legacy=0x7feecc0150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feec8003a60 0x7feec8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): picked mon.c con 0x7feec8003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:47.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feec8004810 0x7feec8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): picked mon.a con 0x7feec8004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feec800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): picked mon.b con 0x7feec80052c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee009db90 con 0x7feec8004810 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec800c250 con 0x7feec80052c0 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec800c410 con 0x7feec8003a60 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feede7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feee48d2700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:47.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feec8003a60 0x7feec8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feec8003a60 auth_method 0 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feec800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedeffd700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feec8004810 0x7feec8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feec80052c0 auth_method 0 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:47.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.722+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feec8004810 auth_method 0 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedeffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient(hunting): handle_auth_done global_id 194200 payload 293 2024-09-18T10:26:47.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feec8003a60 msgr2=0x7feec8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:47.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feec8003a60 0x7feec8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feec8004810 msgr2=0x7feec8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:47.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feec8004810 0x7feec8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient: _finish_hunting 0 2024-09-18T10:26:47.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 monclient: found mon.b 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec8003570 con 0x7feec80052c0 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 10 monclient: _finish_auth 0 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedf7fe700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feec8003a60 0x7feec8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:26:47.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 reap_dead start 2024-09-18T10:26:47.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 msgr2=0x7feec800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:47.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feedffff700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feec800ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feee006fcb0 tx=0x7feed8004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23b50d52f827faeb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:47.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8001630 con 0x7feec80052c0 2024-09-18T10:26:47.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:47.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:47.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: dump: 2024-09-18T10:26:47.748 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:47.748 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:47.748 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:47.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feee48d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed800d040 con 0x7feec80052c0 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feee48d2700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feee48d2700 4 set_mon_vals no callback set 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 1 -- 172.21.15.142:0/862436775 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80019e0 con 0x7feec80052c0 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:47.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.726+0000 7feede7fc700 10 monclient: dump: 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.751 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 10 monclient: shutdown 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 msgr2=0x7feec800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feec800ba60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feee006fcb0 tx=0x7feed8004a60 comp rx=0 tx=0).stop 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- 172.21.15.142:0/862436775 shutdown_connections 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 --2- 172.21.15.142:0/862436775 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feec800ba60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- 172.21.15.142:0/862436775 >> 172.21.15.142:0/862436775 conn(0x7feee00a1fe0 msgr2=0x7feee0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:47.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- 172.21.15.142:0/862436775 shutdown_connections 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- 172.21.15.142:0/862436775 wait complete. 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 10 monclient: build_initial_monmap 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 10 monclient: monmap: 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:47.754 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 Processor -- start 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 1 -- start start 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.730+0000 7feee48d2700 10 monclient: init 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-b con 0x7feec80052c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-a con 0x7feee0138c20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feee006b760 0x7feee006bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): picked mon.noname-c con 0x7feee006b760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0138ff0 con 0x7feee0138c20 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee006f970 con 0x7feec80052c0 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee006ecd0 con 0x7feee006b760 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feee48d2700 10 monclient(hunting): authenticate will time out at 1896.660867s 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feee006b760 0x7feee006bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:42826/0 (socket says 172.21.15.142:42826) 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 -- 172.21.15.142:0/427286886 learned_addr learned my addr 172.21.15.142:0/427286886 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:47.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): get_auth_request con 0x7feec80052c0 auth_method 0 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): get_auth_request con 0x7feee006b760 auth_method 0 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 10 monclient(hunting): get_auth_request con 0x7feee0138c20 auth_method 0 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:47.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 msgr2=0x7feee006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 msgr2=0x7feee006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedeffd700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient(hunting): handle_auth_done global_id 194203 payload 824 2024-09-18T10:26:47.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feee006b760 msgr2=0x7feee006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feee006b760 0x7feee006bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 msgr2=0x7feee006b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 monclient: found mon.noname-b 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.734+0000 7feedf7fe700 1 -- 172.21.15.142:0/427286886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee006e9f0 con 0x7feec80052c0 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feedf7fe700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 msgr2=0x7feee01386e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:47.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feedf7fe700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feed4018980 tx=0x7feed4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4017cd0 con 0x7feec80052c0 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: dump: 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:47.760 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: _finish_auth 0 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 5 monclient: authenticate success, global_id 194203 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed401e040 con 0x7feec80052c0 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _renew_subs 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0146990 con 0x7feec80052c0 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4024480 con 0x7feec80052c0 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:47.761 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 10 monclient: dump: 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feeddffb700 4 set_mon_vals no callback set 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _renew_subs 2024-09-18T10:26:47.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee013af80 con 0x7feec80052c0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7feed4024870 con 0x7feec80052c0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee0091f50 con 0x7feec80052c0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 0x7feecc03a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7feed404abd0 con 0x7feec80052c0 2024-09-18T10:26:47.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feedeffd700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 0x7feecc03a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:47.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.738+0000 7feedeffd700 10 monclient: get_auth_request con 0x7feecc0381a0 auth_method 0 2024-09-18T10:26:47.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.742+0000 7feedeffd700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 0x7feecc03a650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7feed0007de0 tx=0x7feed001b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:47.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.750+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7feed402d400 con 0x7feec80052c0 2024-09-18T10:26:47.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.750+0000 7feec77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:47.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.750+0000 7feec77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:47.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.970+0000 7feee48d2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:26:47.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:47.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:26:47.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feee0072a80 con 0x7feec80052c0 2024-09-18T10:26:47.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feec77fe700 1 -- 172.21.15.142:0/427286886 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7feed4015bb0 con 0x7feec80052c0 2024-09-18T10:26:47.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feec77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:47.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feec77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:47.980 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:26:47.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 msgr2=0x7feecc03a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 0x7feecc03a650 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7feed0007de0 tx=0x7feed001b040 comp rx=0 tx=0).stop 2024-09-18T10:26:47.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 10 monclient: shutdown 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 msgr2=0x7feee01386e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feed4018980 tx=0x7feed4008040 comp rx=0 tx=0).stop 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 shutdown_connections 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feecc0381a0 0x7feecc03a650 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feee006b760 0x7feee006bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feee0138c20 0x7feee006b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 --2- 172.21.15.142:0/427286886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec80052c0 0x7feee01386e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 >> 172.21.15.142:0/427286886 conn(0x7feee00a1fe0 msgr2=0x7feee0094c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 shutdown_connections 2024-09-18T10:26:47.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:47.974+0000 7feee48d2700 1 -- 172.21.15.142:0/427286886 wait complete. 2024-09-18T10:26:50.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:51.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: get_monmap_and_config 2024-09-18T10:26:51.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: build_initial_monmap 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: monmap: 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:51.106101+0000 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:51.106101+0000 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:51.109 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:51.110 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 Processor -- start 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 -- start start 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: init 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3cd009a7f0 0x7f3cd0098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-c con 0x7f3cd009a7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:51.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cd0099220 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cd009c4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd009abc0 con 0x7f3cd009c4f0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd009d950 con 0x7f3cd0099220 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd009db90 con 0x7f3cd009a7f0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccffff700 10 monclient(hunting): authenticate will time out at 1900.031643s 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccf7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3cd009c4f0 0x7f3cd013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40000/0 (socket says 172.21.15.142:40000) 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.102+0000 7f3ccf7fe700 1 -- 172.21.15.142:0/1394591710 learned_addr learned my addr 172.21.15.142:0/1394591710 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 --2- 172.21.15.142:0/1394591710 >> v2:172.21.15.150:3300/0 conn(0x7f3cd009a7f0 0x7f3cd0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): get_auth_request con 0x7f3cd009a7f0 auth_method 0 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:51.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient(hunting): handle_auth_done global_id 194206 payload 274 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 -- 172.21.15.142:0/1394591710 >> v1:172.21.15.150:6789/0 conn(0x7f3cd0099220 legacy=0x7f3cd009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 -- 172.21.15.142:0/1394591710 >> v1:172.21.15.142:6789/0 conn(0x7f3cd009c4f0 legacy=0x7f3cd013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient: _finish_hunting 0 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 monclient: found mon.noname-c 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:26:51.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 -- 172.21.15.142:0/1394591710 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0099a30 con 0x7f3cd009a7f0 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 -- 172.21.15.142:0/1394591710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009a7f0 msgr2=0x7f3cd0098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cceffd700 1 --2- 172.21.15.142:0/1394591710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009a7f0 0x7f3cd0098ce0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80098e0 tx=0x7f3cb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd9810b3de8463d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 1 -- 172.21.15.142:0/1394591710 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb800ba70 con 0x7f3cd009a7f0 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: dump: 2024-09-18T10:26:51.119 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: _finish_auth 0 2024-09-18T10:26:51.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 1 -- 172.21.15.142:0/1394591710 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb800bbd0 con 0x7f3cd009a7f0 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3ccffff700 5 monclient: authenticate success, global_id 194206 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3ccffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3ccffff700 10 monclient: get_monmap_and_config success 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3ccffff700 4 set_mon_vals no callback set 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 1 -- 172.21.15.142:0/1394591710 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8011420 con 0x7f3cd009a7f0 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.106+0000 7f3cce7fc700 10 monclient: dump: 2024-09-18T10:26:51.121 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: shutdown 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:51.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- 172.21.15.142:0/1394591710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009a7f0 msgr2=0x7f3cd0098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1394591710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009a7f0 0x7f3cd0098ce0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80098e0 tx=0x7f3cb80091a0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- 172.21.15.142:0/1394591710 shutdown_connections 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1394591710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009a7f0 0x7f3cd0098ce0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- 172.21.15.142:0/1394591710 >> 172.21.15.142:0/1394591710 conn(0x7f3cd00a1fe0 msgr2=0x7f3cd0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- 172.21.15.142:0/1394591710 shutdown_connections 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- 172.21.15.142:0/1394591710 wait complete. 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: build_initial_monmap 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: monmap: 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:26:51.123 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 Processor -- start 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- start start 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: init 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:51.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-c con 0x7f3cd0099220 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-a con 0x7f3cd009a7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009c4f0 0x7f3cd006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): picked mon.noname-b con 0x7f3cd009c4f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd006b500 con 0x7f3cd009a7f0 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd006c7c0 con 0x7f3cd009c4f0 2024-09-18T10:26:51.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd006c9f0 con 0x7f3cd0099220 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccffff700 10 monclient(hunting): authenticate will time out at 1900.038466s 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:58602/0 (socket says 172.21.15.142:58602) 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 -- 172.21.15.142:0/1164372407 learned_addr learned my addr 172.21.15.142:0/1164372407 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 10 monclient(hunting): get_auth_request con 0x7f3cd009a7f0 auth_method 0 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:51824/0 (socket says 172.21.15.142:51824) 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:51.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): get_auth_request con 0x7f3cd0099220 auth_method 0 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009c4f0 0x7f3cd006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 msgr2=0x7f3cd006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 msgr2=0x7f3cd006a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:26:51.127 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): get_auth_request con 0x7f3cd009c4f0 auth_method 0 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cc67fc700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3ccf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient(hunting): handle_auth_done global_id 75809 payload 824 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009c4f0 msgr2=0x7f3cd006e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:51.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009c4f0 0x7f3cd006e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 msgr2=0x7f3cd006a780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient: _finish_hunting 0 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 monclient: found mon.noname-c 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:26:51.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.110+0000 7f3cceffd700 1 -- 172.21.15.142:0/1164372407 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8009420 con 0x7f3cd0099220 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cceffd700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 msgr2=0x7f3cd00938e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cceffd700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3cd006fc80 tx=0x7f3cb8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8000cc0 con 0x7f3cd0099220 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: dump: 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:51.132 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: _finish_auth 0 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:26:51.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 5 monclient: authenticate success, global_id 75809 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb801a670 con 0x7f3cd0099220 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _renew_subs 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd00943d0 con 0x7f3cd0099220 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb801aa30 con 0x7f3cd0099220 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3cc77fe700 10 monclient: dump: 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:51.134 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccdffb700 4 set_mon_vals no callback set 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _renew_subs 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:51.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd006ae60 con 0x7f3cd0099220 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.114+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd006cc30 con 0x7f3cd0099220 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.118+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3cb8025440 con 0x7f3cd0099220 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.118+0000 7f3cc77fe700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 0x7f3cb0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.118+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3cb804aae0 con 0x7f3cd0099220 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.118+0000 7f3cc67fc700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 0x7f3cb0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.118+0000 7f3cc67fc700 10 monclient: get_auth_request con 0x7f3cb0035ac0 auth_method 0 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.122+0000 7f3cc67fc700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 0x7f3cb0037f70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0094060 tx=0x7f3cc0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:26:51.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.126+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f3cb801b440 con 0x7f3cd0099220 2024-09-18T10:26:51.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.126+0000 7f3cc77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:26:51.137 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.126+0000 7f3cc77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:26:51.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3ccffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:26:51.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3ccffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:51.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3ccffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:26:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3cd006ae60 con 0x7f3cd0099220 2024-09-18T10:26:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3cc77fe700 1 -- 172.21.15.142:0/1164372407 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f3cd006ae60 con 0x7f3cd0099220 2024-09-18T10:26:51.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3cc77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:26:51.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.346+0000 7f3cc77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:26:51.354 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:26:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 msgr2=0x7f3cb0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 0x7f3cb0037f70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0094060 tx=0x7f3cc0009040 comp rx=0 tx=0).stop 2024-09-18T10:26:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 10 monclient: shutdown 2024-09-18T10:26:51.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 msgr2=0x7f3cd00938e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3cd006fc80 tx=0x7f3cb8016000 comp rx=0 tx=0).stop 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 shutdown_connections 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3cb0035ac0 0x7f3cb0037f70 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd009c4f0 0x7f3cd006e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3cd009a7f0 0x7f3cd006a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 --2- 172.21.15.142:0/1164372407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3cd0099220 0x7f3cd00938e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 >> 172.21.15.142:0/1164372407 conn(0x7f3cd00a1fe0 msgr2=0x7f3cd01397d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 shutdown_connections 2024-09-18T10:26:51.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:51.350+0000 7f3ccffff700 1 -- 172.21.15.142:0/1164372407 wait complete. 2024-09-18T10:26:54.368 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: get_monmap_and_config 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: build_initial_monmap 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: monmap: 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:26:54.477969+0000 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:26:54.477969+0000 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:26:54.481 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 Processor -- start 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 -- start start 2024-09-18T10:26:54.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: init 2024-09-18T10:26:54.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: _reopen_session rank -1 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa93809aae0 addr v1:172.21.15.150:6789/0 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-c con 0x7fa938132570 addr v2:172.21.15.150:3300/0 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-b con 0x7fa938134e90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93809aeb0 con 0x7fa93809aae0 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806f6d0 con 0x7fa938132570 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380955a0 con 0x7fa938134e90 2024-09-18T10:26:54.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93d610700 10 monclient(hunting): authenticate will time out at 1903.403458s 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39426/0 (socket says 172.21.15.142:39426) 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 learned_addr learned my addr 172.21.15.142:0/2770561171 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa938134e90 auth_method 0 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa93ce0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.474+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:54.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:54.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:54.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:54.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:54.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:54.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.150:6789/0 conn(0x7fa93809aae0 legacy=0x7fa938096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:54.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:54.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa938134e90 auth_method 0 2024-09-18T10:26:54.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:54.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:54.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa938132570 auth_method 0 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:54.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:54.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:55.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:55.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa938134e90 auth_method 0 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa938132570 auth_method 0 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:55.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:55.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:55.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:55.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:55.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:55.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa938134e90 auth_method 0 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa938132570 auth_method 0 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:55.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:55.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:55.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:26:57.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): tick 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 monclient(hunting): continuing hunt 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 msgr2=0x7fa938137270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa938134e90 0x7fa938137270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 msgr2=0x7fa938134950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa938132570 0x7fa938134950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.150:6789/0 conn(0x7fa93809aae0 legacy=0x7fa938096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient: _add_conns ranks=[3,0] 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a con 0x7fa918003770 addr v2:172.21.15.142:3300/0 2024-09-18T10:26:57.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9180062d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa938095b60 con 0x7fa9180062d0 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806f6d0 con 0x7fa918003770 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa935ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa918003770 auth_method 0 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:57.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:57.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:26:57.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.478+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa9180062d0 legacy=0x7fa918008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:26:57.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:57.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa918003770 auth_method 0 2024-09-18T10:26:57.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:57.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:57.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:57.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:57.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:57.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:57.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:57.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:26:58.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa918003770 auth_method 0 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:58.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa918003770 auth_method 0 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:26:58.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:26:58.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): tick 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 monclient(hunting): continuing hunt 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa918003770 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa9180062d0 legacy=0x7fa918008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:00.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 reap_dead start 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-c con 0x7fa91800a190 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-b con 0x7fa91800a600 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a con 0x7fa9180088e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806f6d0 con 0x7fa9180088e0 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9380955a0 con 0x7fa91800a190 2024-09-18T10:27:00.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa918009590 con 0x7fa91800a600 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa935ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa91800a600 auth_method 0 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:00.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa91800a190 auth_method 0 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:00.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:00.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:00.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa91800a600 auth_method 0 2024-09-18T10:27:00.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa91800a190 auth_method 0 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:00.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:00.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:01.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa91800a600 auth_method 0 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa91800a190 auth_method 0 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:01.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:01.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa91800a600 auth_method 0 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa91800a190 auth_method 0 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:01.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa9377fe700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:01.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:01.890+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:03.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient(hunting): tick 2024-09-18T10:27:03.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 monclient(hunting): continuing hunt 2024-09-18T10:27:03.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:27:03.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 msgr2=0x7fa91800a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a600 0x7fa91800a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa91800a190 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient: _add_conns ranks=[0,1] 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa918009970 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa918010a40 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91800b010 con 0x7fa918009970 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa918015620 con 0x7fa918010a40 2024-09-18T10:27:03.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa935ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:03.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa918009970 legacy=0x7fa91800f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:03.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:03.478+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.150:6789/0 conn(0x7fa918010a40 legacy=0x7fa918012e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): tick 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 monclient(hunting): continuing hunt 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.150:6789/0 conn(0x7fa918010a40 legacy=0x7fa918012e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa918009970 legacy=0x7fa91800f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 reap_dead start 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a con 0x7fa9180088e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa91800a9d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): picked mon.noname-c con 0x7fa9180160f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9180167c0 con 0x7fa91800a9d0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806f6d0 con 0x7fa9180088e0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9380955a0 con 0x7fa9180160f0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa935ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180160f0 auth_method 0 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.478+0000 7fa9377fe700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa91800a9d0 legacy=0x7fa9180159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:06.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:06.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:06.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:06.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:06.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180160f0 auth_method 0 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 msgr2=0x7fa918005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:06.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:06.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:06.682+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.150:3300/0 conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180160f0 auth_method 0 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa937fff700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.093 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_done global_id 194212 payload 274 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 msgr2=0x7fa918008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> v1:172.21.15.142:6789/0 conn(0x7fa91800a9d0 legacy=0x7fa9180159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient: _finish_hunting 0 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 monclient: found mon.noname-c 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:27:07.094 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9180164e0 con 0x7fa9180160f0 2024-09-18T10:27:07.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 msgr2=0x7fa918005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:07.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.086+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa918005be0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9380676a0 tx=0x7fa930008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f874e60ba3a6829 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 1 -- 172.21.15.142:0/2770561171 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930018b20 con 0x7fa9180160f0 2024-09-18T10:27:07.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: dump: 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.096 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: _finish_auth 0 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 1 -- 172.21.15.142:0/2770561171 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93000c040 con 0x7fa9180160f0 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 1 -- 172.21.15.142:0/2770561171 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9300223a0 con 0x7fa9180160f0 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 5 monclient: authenticate success, global_id 194212 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 4 set_mon_vals no callback set 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa936ffd700 10 monclient: dump: 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.098 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 10 monclient: shutdown 2024-09-18T10:27:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 -- 172.21.15.142:0/2770561171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 msgr2=0x7fa918005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa918005be0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9380676a0 tx=0x7fa930008040 comp rx=0 tx=0).stop 2024-09-18T10:27:07.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 -- 172.21.15.142:0/2770561171 shutdown_connections 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 --2- 172.21.15.142:0/2770561171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa918005be0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 --2- 172.21.15.142:0/2770561171 >> v2:172.21.15.142:3300/0 conn(0x7fa9180088e0 0x7fa918008f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 -- 172.21.15.142:0/2770561171 >> 172.21.15.142:0/2770561171 conn(0x7fa9380a1fe0 msgr2=0x7fa938070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 -- 172.21.15.142:0/2770561171 shutdown_connections 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 1 -- 172.21.15.142:0/2770561171 wait complete. 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 10 monclient: build_initial_monmap 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.090+0000 7fa93d610700 10 monclient: monmap: 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:07.103 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 Processor -- start 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 -- start start 2024-09-18T10:27:07.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient: init 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-a con 0x7fa9180088e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 0x7fa9380988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-c con 0x7fa91800a9d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa938098e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): picked mon.noname-b con 0x7fa9180160f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93812dce0 con 0x7fa9180088e0 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93809f540 con 0x7fa9180160f0 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93812d330 con 0x7fa91800a9d0 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93d610700 10 monclient(hunting): authenticate will time out at 1916.022320s 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 0x7fa9380988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52508/0 (socket says 172.21.15.142:52508) 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 -- 172.21.15.142:0/3786642683 learned_addr learned my addr 172.21.15.142:0/3786642683 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:07.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 10 monclient(hunting): get_auth_request con 0x7fa9180088e0 auth_method 0 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): get_auth_request con 0x7fa91800a9d0 auth_method 0 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa938098e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request con 0x7fa9180160f0 auth_method 0 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 msgr2=0x7fa938131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 msgr2=0x7fa938131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa937fff700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa93ce0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient(hunting): handle_auth_done global_id 75812 payload 824 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 msgr2=0x7fa938098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa938098e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.108 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 msgr2=0x7fa938131f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient: _finish_hunting 0 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 monclient: found mon.noname-c 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.109 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.094+0000 7fa9377fe700 1 -- 172.21.15.142:0/3786642683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93000e040 con 0x7fa91800a9d0 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa9377fe700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 msgr2=0x7fa9380988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa9377fe700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 0x7fa9380988d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa918009f70 tx=0x7fa920007380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92000eb00 con 0x7fa91800a9d0 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: dump: 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.112 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: _finish_auth 0 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa920015040 con 0x7fa91800a9d0 2024-09-18T10:27:07.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 5 monclient: authenticate success, global_id 75812 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 10 monclient: _renew_subs 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92000eed0 con 0x7fa91800a9d0 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9381314f0 con 0x7fa91800a9d0 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa9367fc700 4 set_mon_vals no callback set 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa92ffff700 10 monclient: dump: 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.114 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 10 monclient: _renew_subs 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93812d020 con 0x7fa91800a9d0 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.098+0000 7fa93d610700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:07.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa93d610700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa93d610700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93812d4c0 con 0x7fa91800a9d0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa92001b690 con 0x7fa91800a9d0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa92ffff700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 0x7fa91803b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa92004aaa0 con 0x7fa91800a9d0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa937fff700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 0x7fa91803b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.102+0000 7fa937fff700 10 monclient: get_auth_request con 0x7fa918039200 auth_method 0 2024-09-18T10:27:07.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.106+0000 7fa937fff700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 0x7fa91803b6b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa928001070 tx=0x7fa928009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.110+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa920025460 con 0x7fa91800a9d0 2024-09-18T10:27:07.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.110+0000 7fa92ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:07.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.110+0000 7fa92ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:07.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa93812be60 con 0x7fa91800a9d0 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa92ffff700 1 -- 172.21.15.142:0/3786642683 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fa93812be60 con 0x7fa91800a9d0 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa92ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:07.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa92ffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:07.336 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:07.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 msgr2=0x7fa91803b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 0x7fa91803b6b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa928001070 tx=0x7fa928009040 comp rx=0 tx=0).stop 2024-09-18T10:27:07.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 10 monclient: shutdown 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 msgr2=0x7fa9380988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 0x7fa9380988d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa918009f70 tx=0x7fa920007380 comp rx=0 tx=0).stop 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 shutdown_connections 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa918039200 0x7fa91803b6b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9180160f0 0x7fa938098e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa91800a9d0 0x7fa9380988d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 --2- 172.21.15.142:0/3786642683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9180088e0 0x7fa938131f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 >> 172.21.15.142:0/3786642683 conn(0x7fa9380a1fe0 msgr2=0x7fa938096860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 shutdown_connections 2024-09-18T10:27:07.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.330+0000 7fa93d610700 1 -- 172.21.15.142:0/3786642683 wait complete. 2024-09-18T10:27:07.349 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:27:07.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:27:07.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: build_initial_monmap 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: monmap: 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:07.457708+0000 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:07.457708+0000 2024-09-18T10:27:07.460 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:07.461 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 Processor -- start 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 -- start start 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: init 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff29c09abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff29c09b740 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff29c09c5e0 0x7ff29c137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff29c09c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29c09dc90 con 0x7ff29c09b740 2024-09-18T10:27:07.465 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29c09b500 con 0x7ff29c09abd0 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff29c09afa0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29b59e700 10 monclient(hunting): authenticate will time out at 1916.383387s 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52524/0 (socket says 172.21.15.142:52524) 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 -- 172.21.15.142:0/1819061113 learned_addr learned my addr 172.21.15.142:0/1819061113 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 10 monclient(hunting): get_auth_request con 0x7ff29c09abd0 auth_method 0 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/1819061113 >> v2:172.21.15.150:3300/0 conn(0x7ff29c09c5e0 0x7ff29c137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): get_auth_request con 0x7ff29c09c5e0 auth_method 0 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 -- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 msgr2=0x7ff29c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 -- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 msgr2=0x7ff29c0960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 --2- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29a59c700 1 --2- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.454+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff291d9b700 1 -- 172.21.15.142:0/1819061113 >> v1:172.21.15.150:6789/0 conn(0x7ff29c09b740 legacy=0x7ff29c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_done global_id 194218 payload 274 2024-09-18T10:27:07.467 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 1 -- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 msgr2=0x7ff29c0960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 1 -- 172.21.15.142:0/1819061113 >> v1:172.21.15.150:6789/0 conn(0x7ff29c09b740 legacy=0x7ff29c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 1 monclient: found mon.noname-c 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:27:07.468 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.458+0000 7ff29ad9d700 1 -- 172.21.15.142:0/1819061113 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29c06f810 con 0x7ff29c09c5e0 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29ad9d700 1 -- 172.21.15.142:0/1819061113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09c5e0 msgr2=0x7ff29c137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/1819061113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09c5e0 0x7ff29c137690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff29c095e60 tx=0x7ff294009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec1f67645309504 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 1 -- 172.21.15.142:0/1819061113 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29400e070 con 0x7ff29c09c5e0 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: dump: 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.472 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: _finish_auth 0 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 5 monclient: authenticate success, global_id 194218 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 1 -- 172.21.15.142:0/1819061113 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29400be40 con 0x7ff29c09c5e0 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:07.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 4 set_mon_vals no callback set 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 1 -- 172.21.15.142:0/1819061113 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294012660 con 0x7ff29c09c5e0 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff299d9b700 10 monclient: dump: 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.474 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 10 monclient: shutdown 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- 172.21.15.142:0/1819061113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09c5e0 msgr2=0x7ff29c137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 --2- 172.21.15.142:0/1819061113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09c5e0 0x7ff29c137690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff29c095e60 tx=0x7ff294009780 comp rx=0 tx=0).stop 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- 172.21.15.142:0/1819061113 shutdown_connections 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 --2- 172.21.15.142:0/1819061113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09c5e0 0x7ff29c137690 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 --2- 172.21.15.142:0/1819061113 >> v2:172.21.15.142:3300/0 conn(0x7ff29c09abd0 0x7ff29c0960e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- 172.21.15.142:0/1819061113 >> 172.21.15.142:0/1819061113 conn(0x7ff29c070ef0 msgr2=0x7ff29c0712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- 172.21.15.142:0/1819061113 shutdown_connections 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- 172.21.15.142:0/1819061113 wait complete. 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 10 monclient: build_initial_monmap 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 10 monclient: monmap: 2024-09-18T10:27:07.475 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 Processor -- start 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 1 -- start start 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.462+0000 7ff29b59e700 10 monclient: init 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:07.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 0x7ff29c08b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff29c09abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09b740 0x7ff29c08bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff29c09b740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff29c09c5e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c094210 con 0x7ff29c09abd0 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c08d740 con 0x7ff29c09b740 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c06eaf0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient(hunting): authenticate will time out at 1916.392260s 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 0x7ff29c08b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46856/0 (socket says 172.21.15.142:46856) 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 learned_addr learned my addr 172.21.15.142:0/148524709 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): get_auth_request con 0x7ff29c09c5e0 auth_method 0 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 10 monclient(hunting): get_auth_request con 0x7ff29c09abd0 auth_method 0 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09b740 0x7ff29c08bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): get_auth_request con 0x7ff29c09b740 auth_method 0 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 msgr2=0x7ff29c08b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 msgr2=0x7ff29c08b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 0x7ff29c08b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:07.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 0x7ff29c08b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff291d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient(hunting): handle_auth_done global_id 75815 payload 824 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09b740 msgr2=0x7ff29c08bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff29c09b740 0x7ff29c08bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 msgr2=0x7ff29c08b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c09abd0 0x7ff29c08b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient: _finish_hunting 0 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 monclient: found mon.noname-c 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294009a20 con 0x7ff29c09c5e0 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 msgr2=0x7ff29c08c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29ad9d700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff2940094c0 tx=0x7ff294017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294012e80 con 0x7ff29c09c5e0 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 10 monclient: dump: 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.483 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 10 monclient: _finish_auth 0 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29401b600 con 0x7ff29c09c5e0 2024-09-18T10:27:07.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 5 monclient: authenticate success, global_id 75815 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient: _renew_subs 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.466+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29c09d240 con 0x7ff29c09c5e0 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29401b9c0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 10 monclient: dump: 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:07.485 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff298d99700 4 set_mon_vals no callback set 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 10 monclient: _renew_subs 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29c08d1c0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c0648a0 con 0x7ff29c08dd90 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 10 monclient: get_auth_request con 0x7ff29c08dd90 auth_method 0 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff2940263f0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff27c035ac0 0x7ff27c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff29404acf0 con 0x7ff29c09c5e0 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff291d9b700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff27c035ac0 0x7ff27c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.470+0000 7ff291d9b700 10 monclient: get_auth_request con 0x7ff27c035ac0 auth_method 0 2024-09-18T10:27:07.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.474+0000 7ff291d9b700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff27c035ac0 0x7ff27c037f70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff29c08ce50 tx=0x7ff28c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 10 monclient: get_auth_request con 0x7ff29c08dd90 auth_method 0 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:07.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:07.674+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:08.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient: get_auth_request con 0x7ff29c08dd90 auth_method 0 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 10 monclient(hunting): handle_auth_done global_id 204099 payload 824 2024-09-18T10:27:08.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff29c08ef90 con 0x7ff29c08dd90 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.078+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff284009060 tx=0x7ff284008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284016a20 con 0x7ff29c08dd90 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 10 monclient: dump: 2024-09-18T10:27:08.086 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.087 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff28400c040 con 0x7ff29c08dd90 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff284009060 tx=0x7ff284008040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 msgr2=0x7ff29c08c680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.082+0000 7ff292ffd700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c08dd90 0x7ff29c08c680 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff284009060 tx=0x7ff284008040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29b59e700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 0x7ff29c0883e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29c072a80 con 0x7ff29c087fe0 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 0x7ff29c0883e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient: get_auth_request con 0x7ff29c087fe0 auth_method 0 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 10 monclient(hunting): handle_auth_done global_id 204102 payload 824 2024-09-18T10:27:08.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff29c0678a0 con 0x7ff29c087fe0 2024-09-18T10:27:08.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 msgr2=0x7ff29c0883e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.094+0000 7ff29a59c700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 0x7ff29c0883e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff28401f000 tx=0x7ff284022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284022940 con 0x7ff29c087fe0 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 10 monclient: dump: 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.102 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7ff284022aa0 con 0x7ff29c087fe0 2024-09-18T10:27:08.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 msgr2=0x7ff29c0883e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 0x7ff29c0883e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff28401f000 tx=0x7ff284022000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 msgr2=0x7ff29c0883e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff292ffd700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff29c087fe0 0x7ff29c0883e0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff28401f000 tx=0x7ff284022000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:08.104 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.105 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.106 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:08.107 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.108 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.109 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.110 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.111 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.112 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.113 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.114 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.115 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:08.116 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff27c035ac0 msgr2=0x7ff27c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff27c035ac0 0x7ff27c037f70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff29c08ce50 tx=0x7ff28c009040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 10 monclient: shutdown 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29ad9d700 1 -- 172.21.15.142:0/148524709 reap_dead start 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 msgr2=0x7ff29c08c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff2940094c0 tx=0x7ff294017000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 shutdown_connections 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 --2- 172.21.15.142:0/148524709 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff29c09c5e0 0x7ff29c08c140 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 >> 172.21.15.142:0/148524709 conn(0x7ff29c070ef0 msgr2=0x7ff29c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 shutdown_connections 2024-09-18T10:27:08.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.098+0000 7ff29b59e700 1 -- 172.21.15.142:0/148524709 wait complete. 2024-09-18T10:27:08.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:08.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:27:08.120 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.226+0000 7f3a8ac61700 10 monclient: get_monmap_and_config 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.226+0000 7f3a8ac61700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.226+0000 7f3a8ac61700 10 monclient: monmap: 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:08.231679+0000 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:08.231679+0000 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.234 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:08.235 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 Processor -- start 2024-09-18T10:27:08.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 -- start start 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient: init 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3a8409c2c0 0x7f3a840999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-c con 0x7f3a8409c2c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a840a0110 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a840a0c20 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a840a6620 con 0x7f3a840a0110 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a840a1c70 con 0x7f3a840a0c20 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8409ab70 con 0x7f3a8409c2c0 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8ac61700 10 monclient(hunting): authenticate will time out at 1917.157592s 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8945e700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3a840a0110 0x7f3a840a0510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51294/0 (socket says 172.21.15.142:51294) 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a8945e700 1 -- 172.21.15.142:0/2032132850 learned_addr learned my addr 172.21.15.142:0/2032132850 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 1 --2- 172.21.15.142:0/2032132850 >> v2:172.21.15.150:3300/0 conn(0x7f3a8409c2c0 0x7f3a840999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): get_auth_request con 0x7f3a8409c2c0 auth_method 0 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.238 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.230+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8945e700 1 -- 172.21.15.142:0/2032132850 >> v1:172.21.15.142:6789/0 conn(0x7f3a840a0110 legacy=0x7f3a840a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 1 -- 172.21.15.142:0/2032132850 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315543371 0 0) 0x7f3a800017d0 con 0x7f3a840a0110 2024-09-18T10:27:08.239 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient(hunting): my global_id is 204105 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 1 -- 172.21.15.142:0/2032132850 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a70003570 con 0x7f3a840a0110 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8a460700 1 -- 172.21.15.142:0/2032132850 >> v1:172.21.15.150:6789/0 conn(0x7f3a840a0c20 legacy=0x7f3a841398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_done global_id 194224 payload 274 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 -- 172.21.15.142:0/2032132850 >> v1:172.21.15.150:6789/0 conn(0x7f3a840a0c20 legacy=0x7f3a841398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 -- 172.21.15.142:0/2032132850 >> v1:172.21.15.142:6789/0 conn(0x7f3a840a0110 legacy=0x7f3a840a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 monclient: found mon.noname-c 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:27:08.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 -- 172.21.15.142:0/2032132850 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8409c690 con 0x7f3a8409c2c0 2024-09-18T10:27:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 -- 172.21.15.142:0/2032132850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8409c2c0 msgr2=0x7f3a840999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a89c5f700 1 --2- 172.21.15.142:0/2032132850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8409c2c0 0x7f3a840999e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a8409acb0 tx=0x7f3a780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aecb19b474efb41 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 1 -- 172.21.15.142:0/2032132850 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78009ea0 con 0x7f3a8409c2c0 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: dump: 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.242 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8ac61700 5 monclient: authenticate success, global_id 194224 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8ac61700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 1 -- 172.21.15.142:0/2032132850 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7800ba70 con 0x7f3a8409c2c0 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8ac61700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:08.243 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a8ac61700 4 set_mon_vals no callback set 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 1 -- 172.21.15.142:0/2032132850 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7800be30 con 0x7f3a8409c2c0 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.234+0000 7f3a88c5d700 10 monclient: dump: 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.244 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.245 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: shutdown 2024-09-18T10:27:08.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- 172.21.15.142:0/2032132850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8409c2c0 msgr2=0x7f3a840999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/2032132850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8409c2c0 0x7f3a840999e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a8409acb0 tx=0x7f3a780091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- 172.21.15.142:0/2032132850 shutdown_connections 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/2032132850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8409c2c0 0x7f3a840999e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- 172.21.15.142:0/2032132850 >> 172.21.15.142:0/2032132850 conn(0x7f3a840a5720 msgr2=0x7f3a840a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- 172.21.15.142:0/2032132850 shutdown_connections 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- 172.21.15.142:0/2032132850 wait complete. 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: monmap: 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:08.251 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 Processor -- start 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- start start 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: init 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 0x7f3a84141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-c con 0x7f3a8409c2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:08.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 0x7f3a84141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-a con 0x7f3a840a0110 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840a0c20 0x7f3a841423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): picked mon.noname-b con 0x7f3a840a0c20 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a840a2c40 con 0x7f3a840a0110 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84091d50 con 0x7f3a840a0c20 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84092ac0 con 0x7f3a8409c2c0 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8ac61700 10 monclient(hunting): authenticate will time out at 1917.167580s 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 0x7f3a84141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8945e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 0x7f3a84141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 0x7f3a84141960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46860/0 (socket says 172.21.15.142:46860) 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8945e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 0x7f3a84141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52578/0 (socket says 172.21.15.142:52578) 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 1 -- 172.21.15.142:0/4189131470 learned_addr learned my addr 172.21.15.142:0/4189131470 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 10 monclient(hunting): get_auth_request con 0x7f3a8409c2c0 auth_method 0 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a89c5f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.238+0000 7f3a8945e700 10 monclient(hunting): get_auth_request con 0x7f3a840a0110 auth_method 0 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840a0c20 0x7f3a841423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): get_auth_request con 0x7f3a840a0c20 auth_method 0 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8a460700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient(hunting): handle_auth_done global_id 75818 payload 824 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840a0c20 msgr2=0x7f3a841423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840a0c20 0x7f3a841423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 msgr2=0x7f3a84141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 0x7f3a84141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 monclient: found mon.noname-c 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a78009420 con 0x7f3a8409c2c0 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8945e700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a840a0110 0x7f3a84141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 msgr2=0x7f3a84141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a89c5f700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 0x7f3a84141960 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800b3c0 tx=0x7f3a78016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7800bdf0 con 0x7f3a8409c2c0 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: dump: 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.257 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7801f070 con 0x7f3a8409c2c0 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 5 monclient: authenticate success, global_id 75818 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 10 monclient: _renew_subs 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a84091790 con 0x7f3a8409c2c0 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a780266f0 con 0x7f3a8409c2c0 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a75ffb700 10 monclient: dump: 2024-09-18T10:27:08.258 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a77fff700 4 set_mon_vals no callback set 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 10 monclient: _renew_subs 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.242+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8408fc60 con 0x7f3a8409c2c0 2024-09-18T10:27:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8ac61700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 0x7f3a8405e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8408f6c0 con 0x7f3a8408f2f0 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3a78026a60 con 0x7f3a8409c2c0 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 0x7f3a8405e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient: get_auth_request con 0x7f3a8408f2f0 auth_method 0 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a75ffb700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3a68035b10 0x7f3a68037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f3a78021070 con 0x7f3a8409c2c0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8a460700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3a68035b10 0x7f3a68037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 10 monclient(hunting): handle_auth_done global_id 75821 payload 824 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8945e700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a84072a80 con 0x7f3a8408f2f0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.246+0000 7f3a8a460700 10 monclient: get_auth_request con 0x7f3a68035b10 auth_method 0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a8945e700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 msgr2=0x7f3a8405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a8945e700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 0x7f3a8405e890 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3a800019e0 tx=0x7f3a80009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a800074f0 con 0x7f3a8408f2f0 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 10 monclient: dump: 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.263 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a8a460700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3a68035b10 0x7f3a68037fc0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3a84099690 tx=0x7f3a7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3a80007650 con 0x7f3a8408f2f0 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 msgr2=0x7f3a8405e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 0x7f3a8405e890 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3a800019e0 tx=0x7f3a80009040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 msgr2=0x7f3a8405e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.250+0000 7f3a75ffb700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8408f2f0 0x7f3a8405e890 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8ac61700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 0x7f3a84029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8ac61700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84064750 con 0x7f3a84091920 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 0x7f3a84029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient: get_auth_request con 0x7f3a84091920 auth_method 0 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 10 monclient(hunting): handle_auth_done global_id 75824 payload 824 2024-09-18T10:27:08.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.262+0000 7f3a8945e700 1 -- 172.21.15.142:0/4189131470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3a8406e190 con 0x7f3a84091920 2024-09-18T10:27:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8945e700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 msgr2=0x7f3a84029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8945e700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 0x7f3a84029d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3a80018000 tx=0x7f3a80016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8001f860 con 0x7f3a84091920 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 10 monclient: dump: 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f3a800072e0 con 0x7f3a84091920 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:08.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 msgr2=0x7f3a84029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 0x7f3a84029d00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3a80018000 tx=0x7f3a80016000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 msgr2=0x7f3a84029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a75ffb700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a84091920 0x7f3a84029d00 secure :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3a80018000 tx=0x7f3a80016000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:08.273 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.274 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.275 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:08.276 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.277 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:08.278 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.279 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.280 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.281 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.282 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.283 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.284 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:27:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.285 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.286 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.287 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:08.287 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:08.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3a68035b10 msgr2=0x7f3a68037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3a68035b10 0x7f3a68037fc0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f3a84099690 tx=0x7f3a7c009040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 10 monclient: shutdown 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 msgr2=0x7f3a84141960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 --2- 172.21.15.142:0/4189131470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a8409c2c0 0x7f3a84141960 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800b3c0 tx=0x7f3a78016000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8a460700 1 -- 172.21.15.142:0/4189131470 reap_dead start 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8a460700 1 -- 172.21.15.142:0/4189131470 reap_dead start 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 shutdown_connections 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 >> 172.21.15.142:0/4189131470 conn(0x7f3a840a5720 msgr2=0x7f3a8409e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 shutdown_connections 2024-09-18T10:27:08.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.266+0000 7f3a8ac61700 1 -- 172.21.15.142:0/4189131470 wait complete. 2024-09-18T10:27:08.290 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:08.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:27:08.290 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:27:08.405 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.398+0000 7fbb7d305700 10 monclient: get_monmap_and_config 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.398+0000 7fbb7d305700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.398+0000 7fbb7d305700 10 monclient: monmap: 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:08.403239+0000 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:08.403239+0000 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:08.406 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:08.407 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:08.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 Processor -- start 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 -- start start 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient: init 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb7809abd0 0x7fbb7809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-a con 0x7fbb7809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb7809db80 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb78137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-b con 0x7fbb78097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb780958c0 con 0x7fbb7809db80 2024-09-18T10:27:08.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7806fa60 con 0x7fbb7809abd0 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78095ef0 con 0x7fbb78097880 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb7d305700 10 monclient(hunting): authenticate will time out at 1917.329465s 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb7809abd0 0x7fbb7809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb78137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb7809abd0 0x7fbb7809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52584/0 (socket says 172.21.15.142:52584) 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 -- 172.21.15.142:0/4225229563 learned_addr learned my addr 172.21.15.142:0/4225229563 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): get_auth_request con 0x7fbb7809abd0 auth_method 0 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb767fc700 1 --1- 172.21.15.142:0/4225229563 >> v1:172.21.15.142:6789/0 conn(0x7fbb7809db80 0x7fbb7809c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51308/0 (socket says 172.21.15.142:51308) 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): get_auth_request con 0x7fbb78097880 auth_method 0 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_done global_id 204111 payload 274 2024-09-18T10:27:08.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 -- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 msgr2=0x7fbb78137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb78137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 -- 172.21.15.142:0/4225229563 >> v1:172.21.15.142:6789/0 conn(0x7fbb7809db80 legacy=0x7fbb7809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 monclient: found mon.noname-a 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb76ffd700 1 -- 172.21.15.142:0/4225229563 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7809afa0 con 0x7fbb7809abd0 2024-09-18T10:27:08.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.402+0000 7fbb777fe700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb78137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb76ffd700 1 -- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 msgr2=0x7fbb7809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb76ffd700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7809bed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb600099b0 tx=0x7fbb60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380afdfbd3261ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 1 -- 172.21.15.142:0/4225229563 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60009ea0 con 0x7fbb7809abd0 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: dump: 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.418 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 1 -- 172.21.15.142:0/4225229563 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6000ba70 con 0x7fbb7809abd0 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 5 monclient: authenticate success, global_id 204111 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 1 -- 172.21.15.142:0/4225229563 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000be30 con 0x7fbb7809abd0 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 4 set_mon_vals no callback set 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb75ffb700 10 monclient: dump: 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 10 monclient: shutdown 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 -- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 msgr2=0x7fbb7809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7809bed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbb600099b0 tx=0x7fbb60009250 comp rx=0 tx=0).stop 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 -- 172.21.15.142:0/4225229563 shutdown_connections 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb78137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 --2- 172.21.15.142:0/4225229563 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7809bed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 -- 172.21.15.142:0/4225229563 >> 172.21.15.142:0/4225229563 conn(0x7fbb780a14e0 msgr2=0x7fbb780a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 -- 172.21.15.142:0/4225229563 shutdown_connections 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 1 -- 172.21.15.142:0/4225229563 wait complete. 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.406+0000 7fbb7d305700 10 monclient: monmap: 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 Processor -- start 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 -- start start 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient: init 2024-09-18T10:27:08.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb7813f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-c con 0x7fbb78097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7813fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-a con 0x7fbb7809abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7809db80 0x7fbb7808bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): picked mon.noname-b con 0x7fbb7809db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7809ffc0 con 0x7fbb7809abd0 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7808f7d0 con 0x7fbb7809db80 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7808dc60 con 0x7fbb78097880 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb7d305700 10 monclient(hunting): authenticate will time out at 1917.338287s 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb7813f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7813fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb7813f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46904/0 (socket says 172.21.15.142:46904) 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 -- 172.21.15.142:0/496778640 learned_addr learned my addr 172.21.15.142:0/496778640 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7813fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52586/0 (socket says 172.21.15.142:52586) 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): get_auth_request con 0x7fbb78097880 auth_method 0 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): get_auth_request con 0x7fbb7809abd0 auth_method 0 2024-09-18T10:27:08.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7809db80 0x7fbb7808bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): get_auth_request con 0x7fbb7809db80 auth_method 0 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient(hunting): handle_auth_done global_id 75830 payload 824 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7809db80 msgr2=0x7fbb7808bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7809db80 0x7fbb7808bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 msgr2=0x7fbb7813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb7809abd0 0x7fbb7813fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 monclient: found mon.noname-c 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.410+0000 7fbb76ffd700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb70009420 con 0x7fbb78097880 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb76ffd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 msgr2=0x7fbb7813f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb76ffd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb7813f8e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbb78095a50 tx=0x7fbb60016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60006b60 con 0x7fbb78097880 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:08.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: dump: 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 5 monclient: authenticate success, global_id 75830 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60020070 con 0x7fbb78097880 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 monclient: _renew_subs 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb600094f0 con 0x7fbb78097880 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60011d80 con 0x7fbb78097880 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb757fa700 4 set_mon_vals no callback set 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 10 monclient: dump: 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.432 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 monclient: _renew_subs 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7808e320 con 0x7fbb78097880 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 0x7fbb7808c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78064950 con 0x7fbb78140360 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbb60022030 con 0x7fbb78097880 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 0x7fbb7808c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbb58035ac0 0x7fbb58037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient: get_auth_request con 0x7fbb78140360 auth_method 0 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fbb6004acc0 con 0x7fbb78097880 2024-09-18T10:27:08.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.414+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb777fe700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbb58035ac0 0x7fbb58037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb777fe700 10 monclient: get_auth_request con 0x7fbb58035ac0 auth_method 0 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb767fc700 10 monclient(hunting): handle_auth_done global_id 194236 payload 824 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb767fc700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb780696a0 con 0x7fbb78140360 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb777fe700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbb58035ac0 0x7fbb58037f70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009ec0 tx=0x7fbb700091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb767fc700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 msgr2=0x7fbb7808c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb767fc700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 0x7fbb7808c140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb680053c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68006020 con 0x7fbb78140360 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb6effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:08.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.418+0000 7fbb6effd700 10 monclient: dump: 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbb68006180 con 0x7fbb78140360 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 msgr2=0x7fbb7808c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 0x7fbb7808c140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb680053c0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 msgr2=0x7fbb7808c140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.422+0000 7fbb6effd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb78140360 0x7fbb7808c140 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb680053c0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb7d305700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:08.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb7d305700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 0x7fbb7808ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb7d305700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb78097d50 con 0x7fbb7808c680 2024-09-18T10:27:08.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 0x7fbb7808ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient: get_auth_request con 0x7fbb7808c680 auth_method 0 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 10 monclient(hunting): handle_auth_done global_id 194239 payload 824 2024-09-18T10:27:08.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.430+0000 7fbb767fc700 1 -- 172.21.15.142:0/496778640 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb7808cfc0 con 0x7fbb7808c680 2024-09-18T10:27:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb767fc700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 msgr2=0x7fbb7808ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb767fc700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 0x7fbb7808ca80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb68018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6801ca70 con 0x7fbb7808c680 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 10 monclient: dump: 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fbb6801cbd0 con 0x7fbb7808c680 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 msgr2=0x7fbb7808ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 0x7fbb7808ca80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb68018000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 msgr2=0x7fbb7808ca80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb6effd700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb7808c680 0x7fbb7808ca80 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbb680079a0 tx=0x7fbb68018000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:08.442 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.443 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.444 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:08.445 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:08.446 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:08.447 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.448 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:08.449 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:08.452 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.453 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:08.454 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:08.455 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:08.455 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbb58035ac0 msgr2=0x7fbb58037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbb58035ac0 0x7fbb58037f70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbb70009ec0 tx=0x7fbb700091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 10 monclient: shutdown 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 msgr2=0x7fbb7813f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 --2- 172.21.15.142:0/496778640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb78097880 0x7fbb7813f8e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbb78095a50 tx=0x7fbb60016000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb777fe700 1 -- 172.21.15.142:0/496778640 reap_dead start 2024-09-18T10:27:08.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb777fe700 1 -- 172.21.15.142:0/496778640 reap_dead start 2024-09-18T10:27:08.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 shutdown_connections 2024-09-18T10:27:08.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 >> 172.21.15.142:0/496778640 conn(0x7fbb780a14e0 msgr2=0x7fbb78096360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 shutdown_connections 2024-09-18T10:27:08.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.434+0000 7fbb7d305700 1 -- 172.21.15.142:0/496778640 wait complete. 2024-09-18T10:27:08.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:08.458 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:27:08.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:27:08.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: get_monmap_and_config 2024-09-18T10:27:08.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: monmap: 2024-09-18T10:27:08.564 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.564 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:08.561684+0000 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:08.561684+0000 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:08.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 Processor -- start 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 -- start start 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: init 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f25ec09cdd0 0x7f25ec09e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-c con 0x7f25ec09cdd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f25ec0a02e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25ec0a0bf0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:08.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ec09d1a0 con 0x7f25ec0a0bf0 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ec09e980 con 0x7f25ec09cdd0 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec09d780 con 0x7f25ec0a02e0 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25f0ea6700 10 monclient(hunting): authenticate will time out at 1917.487629s 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46914/0 (socket says 172.21.15.142:46914) 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 1 -- 172.21.15.142:0/834559111 learned_addr learned my addr 172.21.15.142:0/834559111 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 10 monclient(hunting): get_auth_request con 0x7f25ec0a02e0 auth_method 0 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.558+0000 7f25ea59c700 1 --2- 172.21.15.142:0/834559111 >> v2:172.21.15.150:3300/0 conn(0x7f25ec09cdd0 0x7f25ec09e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25ea59c700 10 monclient(hunting): get_auth_request con 0x7f25ec09cdd0 auth_method 0 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25ea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25ea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25ea59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient(hunting): handle_auth_done global_id 75833 payload 274 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 -- 172.21.15.142:0/834559111 >> v2:172.21.15.150:3300/0 conn(0x7f25ec09cdd0 msgr2=0x7f25ec09e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/834559111 >> v2:172.21.15.150:3300/0 conn(0x7f25ec09cdd0 0x7f25ec09e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 -- 172.21.15.142:0/834559111 >> v1:172.21.15.150:6789/0 conn(0x7f25ec0a0bf0 legacy=0x7f25ec138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:08.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 monclient: found mon.noname-b 2024-09-18T10:27:08.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:08.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 -- 172.21.15.142:0/834559111 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec09dd40 con 0x7f25ec0a02e0 2024-09-18T10:27:08.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 -- 172.21.15.142:0/834559111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 msgr2=0x7f25ec0a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/834559111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f25d40099b0 tx=0x7f25d4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf80312e94761b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 1 -- 172.21.15.142:0/834559111 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d400ba70 con 0x7f25ec0a02e0 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: dump: 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.578 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 5 monclient: authenticate success, global_id 75833 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 1 -- 172.21.15.142:0/834559111 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d400bbd0 con 0x7f25ec0a02e0 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 4 set_mon_vals no callback set 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 1 -- 172.21.15.142:0/834559111 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d4011420 con 0x7f25ec0a02e0 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:08.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25e9d9b700 10 monclient: dump: 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 10 monclient: shutdown 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 -- 172.21.15.142:0/834559111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 msgr2=0x7f25ec0a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/834559111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f25d40099b0 tx=0x7f25d4009250 comp rx=0 tx=0).stop 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 -- 172.21.15.142:0/834559111 shutdown_connections 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/834559111 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a02e0 0x7f25ec0a06b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/834559111 >> v2:172.21.15.150:3300/0 conn(0x7f25ec09cdd0 0x7f25ec09e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.562+0000 7f25f0ea6700 1 -- 172.21.15.142:0/834559111 >> 172.21.15.142:0/834559111 conn(0x7f25ec071680 msgr2=0x7f25ec0a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- 172.21.15.142:0/834559111 shutdown_connections 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- 172.21.15.142:0/834559111 wait complete. 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient: build_initial_monmap 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient: monmap: 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:08.581 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 Processor -- start 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- start start 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient: init 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-a con 0x7f25ec09cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:08.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 0x7f25ec142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f25ec0a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a0bf0 0x7f25ec08cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): picked mon.noname-c con 0x7f25ec0a0bf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec090b10 con 0x7f25ec09cdd0 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec0907a0 con 0x7f25ec0a02e0 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec08f560 con 0x7f25ec0a0bf0 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25f0ea6700 10 monclient(hunting): authenticate will time out at 1917.494109s 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a0bf0 0x7f25ec08cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52600/0 (socket says 172.21.15.142:52600) 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 -- 172.21.15.142:0/911227928 learned_addr learned my addr 172.21.15.142:0/911227928 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): get_auth_request con 0x7f25ec09cdd0 auth_method 0 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): get_auth_request con 0x7f25ec0a0bf0 auth_method 0 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 0x7f25ec142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 10 monclient(hunting): get_auth_request con 0x7f25ec0a02e0 auth_method 0 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient(hunting): handle_auth_done global_id 204117 payload 824 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 -- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a0bf0 msgr2=0x7f25ec08cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a0bf0 0x7f25ec08cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 -- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 msgr2=0x7f25ec142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 0x7f25ec142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient: _finish_hunting 0 2024-09-18T10:27:08.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 monclient: found mon.noname-a 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25ea59c700 1 -- 172.21.15.142:0/911227928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40094f0 con 0x7f25ec09cdd0 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.566+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 0x7f25ec142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25ea59c700 1 -- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 msgr2=0x7f25ec141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25ea59c700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00d890 tx=0x7f25dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc012800 con 0x7f25ec09cdd0 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: dump: 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.586 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 5 monclient: authenticate success, global_id 204117 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc012980 con 0x7f25ec09cdd0 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:08.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _renew_subs 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc012d40 con 0x7f25ec09cdd0 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ec0904c0 con 0x7f25ec09cdd0 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e8d99700 4 set_mon_vals no callback set 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 10 monclient: dump: 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:08.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _renew_subs 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ec091750 con 0x7f25ec09cdd0 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f25dc014640 con 0x7f25ec09cdd0 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ec091970 con 0x7f25ec09cdd0 2024-09-18T10:27:08.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 0x7f25cc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f25dc0496d0 con 0x7f25ec09cdd0 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 0x7f25cc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.570+0000 7f25e1d9b700 10 monclient: get_auth_request con 0x7f25cc035ac0 auth_method 0 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.574+0000 7f25e1d9b700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 0x7f25cc037f70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f25d40099b0 tx=0x7f25d4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.582+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f25dc011570 con 0x7f25ec09cdd0 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.582+0000 7f25e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:08.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.582+0000 7f25e2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25f0ea6700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:27:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25f0ea6700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:27:08.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25f0ea6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:08.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f25ec0696a0 con 0x7f25ec09cdd0 2024-09-18T10:27:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25e2ffd700 1 -- 172.21.15.142:0/911227928 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f25dc01dbb0 con 0x7f25ec09cdd0 2024-09-18T10:27:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:27:08.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f25e2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:08.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=7,logm=51} crc {auth=3520507105,config=3140514009,health=749566279,kv=4261215878,logm=3751322529}) 2024-09-18T10:27:08.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=4062302704,config=3140514009,health=66923700,kv=4261215878,logm=3813221408}) 2024-09-18T10:27:08.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=7,logm=51} crc {auth=3520507105,config=3140514009,health=749566279,kv=4261215878,logm=3751322529}) 2024-09-18T10:27:08.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.806+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=3783470321,config=3140514009,health=749566279,kv=4261215878,logm=161373960}) 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 msgr2=0x7f25cc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 0x7f25cc037f70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f25d40099b0 tx=0x7f25d4016000 comp rx=0 tx=0).stop 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 10 monclient: shutdown 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 msgr2=0x7f25ec141cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00d890 tx=0x7f25dc008040 comp rx=0 tx=0).stop 2024-09-18T10:27:08.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 shutdown_connections 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f25cc035ac0 0x7f25cc037f70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25ec0a0bf0 0x7f25ec08cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25ec0a02e0 0x7f25ec142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 --2- 172.21.15.142:0/911227928 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25ec09cdd0 0x7f25ec141cd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 >> 172.21.15.142:0/911227928 conn(0x7f25ec071680 msgr2=0x7f25ec09eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 shutdown_connections 2024-09-18T10:27:08.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f25f0ea6700 1 -- 172.21.15.142:0/911227928 wait complete. 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090583344}) 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610186680}) 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090583344}) 2024-09-18T10:27:08.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.810+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318313011}) 2024-09-18T10:27:08.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106923750}) 2024-09-18T10:27:08.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3143711200}) 2024-09-18T10:27:08.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106923750}) 2024-09-18T10:27:08.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.814+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110396993}) 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279868422}) 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093070362}) 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279868422}) 2024-09-18T10:27:08.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160805006}) 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178132780}) 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133932059}) 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1178132780}) 2024-09-18T10:27:08.825 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.818+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562408706}) 2024-09-18T10:27:08.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159706266}) 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2040514626}) 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159706266}) 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071579078}) 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723759372}) 2024-09-18T10:27:08.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=94661292}) 2024-09-18T10:27:08.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723759372}) 2024-09-18T10:27:08.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519662899}) 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061462098}) 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1486554296}) 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061462098}) 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841775768}) 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333263475}) 2024-09-18T10:27:08.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423800274}) 2024-09-18T10:27:08.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333263475}) 2024-09-18T10:27:08.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.822+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1161955208}) 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140462795}) 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1004612584}) 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140462795}) 2024-09-18T10:27:08.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464143934}) 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810879025}) 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612130815}) 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810879025}) 2024-09-18T10:27:08.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2397474306}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2196442616}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420938369}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2196442616}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883035527}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482380490}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=783920969}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482380490}) 2024-09-18T10:27:08.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.826+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212782740}) 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87096869}) 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544808957}) 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87096869}) 2024-09-18T10:27:08.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1053017306}) 2024-09-18T10:27:08.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859577951}) 2024-09-18T10:27:08.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119886757}) 2024-09-18T10:27:08.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859577951}) 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3839002269}) 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935710998}) 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2018633738}) 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935710998}) 2024-09-18T10:27:08.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.830+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042020633}) 2024-09-18T10:27:08.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60026270}) 2024-09-18T10:27:08.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079240190}) 2024-09-18T10:27:08.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60026270}) 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285841073}) 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844016874}) 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017185726}) 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844016874}) 2024-09-18T10:27:08.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078955408}) 2024-09-18T10:27:08.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615598255}) 2024-09-18T10:27:08.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022273046}) 2024-09-18T10:27:08.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615598255}) 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040571573}) 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491093498}) 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168994807}) 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491093498}) 2024-09-18T10:27:08.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.834+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986352849}) 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078768811}) 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804597668}) 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078768811}) 2024-09-18T10:27:08.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2999998145}) 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839850176}) 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500525576}) 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839850176}) 2024-09-18T10:27:08.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1093231843}) 2024-09-18T10:27:08.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52708961}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631676155}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52708961}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177457118}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113811234}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139124608}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113811234}) 2024-09-18T10:27:08.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.838+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71604823}) 2024-09-18T10:27:08.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936155632}) 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4231069941}) 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1936155632}) 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571186047}) 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976374398}) 2024-09-18T10:27:08.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478552939}) 2024-09-18T10:27:08.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976374398}) 2024-09-18T10:27:08.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123020068}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259448605}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=493438148}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259448605}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3200678986}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799488334}) 2024-09-18T10:27:08.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=522762501}) 2024-09-18T10:27:08.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799488334}) 2024-09-18T10:27:08.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.842+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=577496883}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800964241}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533937529}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3800964241}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007016781}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=631158133}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062633941}) 2024-09-18T10:27:08.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=631158133}) 2024-09-18T10:27:08.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587414707}) 2024-09-18T10:27:08.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096521496}) 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615088707}) 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096521496}) 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.846+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036370987}) 2024-09-18T10:27:08.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194875775}) 2024-09-18T10:27:08.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617472194}) 2024-09-18T10:27:08.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=194875775}) 2024-09-18T10:27:08.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110972223}) 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=634725176}) 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909101333}) 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=634725176}) 2024-09-18T10:27:08.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057702376}) 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77544686}) 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252967455}) 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77544686}) 2024-09-18T10:27:08.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711325990}) 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515766822}) 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017294065}) 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515766822}) 2024-09-18T10:27:08.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.850+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1539454241}) 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740163955}) 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628727419}) 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740163955}) 2024-09-18T10:27:08.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2103900787}) 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988604387}) 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367963928}) 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988604387}) 2024-09-18T10:27:08.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.854+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961942537}) 2024-09-18T10:27:08.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984527406}) 2024-09-18T10:27:08.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237562042}) 2024-09-18T10:27:08.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984527406}) 2024-09-18T10:27:08.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262752033}) 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501103714}) 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140773398}) 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501103714}) 2024-09-18T10:27:08.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.858+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=996391232}) 2024-09-18T10:27:08.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206205979}) 2024-09-18T10:27:08.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179923629}) 2024-09-18T10:27:08.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206205979}) 2024-09-18T10:27:08.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2833744970}) 2024-09-18T10:27:08.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252034531}) 2024-09-18T10:27:08.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191510445}) 2024-09-18T10:27:08.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252034531}) 2024-09-18T10:27:08.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.862+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162638234}) 2024-09-18T10:27:08.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151564318}) 2024-09-18T10:27:08.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245977455}) 2024-09-18T10:27:08.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151564318}) 2024-09-18T10:27:08.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186549446}) 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241632488}) 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=126461837}) 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241632488}) 2024-09-18T10:27:08.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.866+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2545843507}) 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951571387}) 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246866395}) 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951571387}) 2024-09-18T10:27:08.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.870+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874952750}) 2024-09-18T10:27:08.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554360896}) 2024-09-18T10:27:08.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080011795}) 2024-09-18T10:27:08.880 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554360896}) 2024-09-18T10:27:08.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.874+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282842845}) 2024-09-18T10:27:08.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979588798}) 2024-09-18T10:27:08.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785751264}) 2024-09-18T10:27:08.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979588798}) 2024-09-18T10:27:08.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.878+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953205990}) 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113823302}) 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578450630}) 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113823302}) 2024-09-18T10:27:08.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.894+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065686414}) 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260226734}) 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512620631}) 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260226734}) 2024-09-18T10:27:08.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710540736}) 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114394312}) 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2947655956}) 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114394312}) 2024-09-18T10:27:08.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.898+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543378454}) 2024-09-18T10:27:08.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990540788}) 2024-09-18T10:27:08.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360801897}) 2024-09-18T10:27:08.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990540788}) 2024-09-18T10:27:08.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=676298711}) 2024-09-18T10:27:08.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056240274}) 2024-09-18T10:27:08.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514513200}) 2024-09-18T10:27:08.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056240274}) 2024-09-18T10:27:08.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260286776}) 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488683713}) 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2464639876}) 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488683713}) 2024-09-18T10:27:08.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.902+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610818189}) 2024-09-18T10:27:08.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41530426}) 2024-09-18T10:27:08.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2313200824}) 2024-09-18T10:27:08.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41530426}) 2024-09-18T10:27:08.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972122693}) 2024-09-18T10:27:08.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998177309}) 2024-09-18T10:27:08.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2426012541}) 2024-09-18T10:27:08.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998177309}) 2024-09-18T10:27:08.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.906+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386333578}) 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075986933}) 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056334940}) 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075986933}) 2024-09-18T10:27:08.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3418038434}) 2024-09-18T10:27:08.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287152263}) 2024-09-18T10:27:08.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904527402}) 2024-09-18T10:27:08.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287152263}) 2024-09-18T10:27:08.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132818682}) 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2326812722}) 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900788316}) 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2326812722}) 2024-09-18T10:27:08.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.910+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961751205}) 2024-09-18T10:27:08.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070834506}) 2024-09-18T10:27:08.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247201636}) 2024-09-18T10:27:08.919 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3070834506}) 2024-09-18T10:27:08.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019020995}) 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526705010}) 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721835280}) 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526705010}) 2024-09-18T10:27:08.921 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.914+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352008753}) 2024-09-18T10:27:08.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130062088}) 2024-09-18T10:27:08.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070401348}) 2024-09-18T10:27:08.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130062088}) 2024-09-18T10:27:08.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021234287}) 2024-09-18T10:27:08.923 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985330688}) 2024-09-18T10:27:08.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3689721426}) 2024-09-18T10:27:08.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985330688}) 2024-09-18T10:27:08.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045289013}) 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940742742}) 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3620324717}) 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940742742}) 2024-09-18T10:27:08.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.918+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088909358}) 2024-09-18T10:27:08.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188597734}) 2024-09-18T10:27:08.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929350265}) 2024-09-18T10:27:08.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188597734}) 2024-09-18T10:27:08.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758458294}) 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563514789}) 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155177652}) 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563514789}) 2024-09-18T10:27:08.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877403970}) 2024-09-18T10:27:08.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236462297}) 2024-09-18T10:27:08.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191701458}) 2024-09-18T10:27:08.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236462297}) 2024-09-18T10:27:08.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.922+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3087482413}) 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446154599}) 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3736102199}) 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446154599}) 2024-09-18T10:27:08.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638507961}) 2024-09-18T10:27:08.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635650883}) 2024-09-18T10:27:08.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=469044250}) 2024-09-18T10:27:08.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635650883}) 2024-09-18T10:27:08.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362839503}) 2024-09-18T10:27:08.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594399345}) 2024-09-18T10:27:08.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425824750}) 2024-09-18T10:27:08.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594399345}) 2024-09-18T10:27:08.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.926+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279619376}) 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361502044}) 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936019926}) 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361502044}) 2024-09-18T10:27:08.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578034680}) 2024-09-18T10:27:08.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103424146}) 2024-09-18T10:27:08.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422779130}) 2024-09-18T10:27:08.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103424146}) 2024-09-18T10:27:08.937 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=255728375}) 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395886895}) 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=228063096}) 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1395886895}) 2024-09-18T10:27:08.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.930+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094065599}) 2024-09-18T10:27:08.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=211692676}) 2024-09-18T10:27:08.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177047922}) 2024-09-18T10:27:08.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.939 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=211692676}) 2024-09-18T10:27:08.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673425002}) 2024-09-18T10:27:08.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.940 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386899197}) 2024-09-18T10:27:08.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909157050}) 2024-09-18T10:27:08.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386899197}) 2024-09-18T10:27:08.941 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063639296}) 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918369715}) 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341632799}) 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918369715}) 2024-09-18T10:27:08.942 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.934+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275191661}) 2024-09-18T10:27:08.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673040308}) 2024-09-18T10:27:08.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3352461132}) 2024-09-18T10:27:08.943 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673040308}) 2024-09-18T10:27:08.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681634567}) 2024-09-18T10:27:08.944 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133917120}) 2024-09-18T10:27:08.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968903100}) 2024-09-18T10:27:08.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133917120}) 2024-09-18T10:27:08.945 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404770641}) 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110544458}) 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.938+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384220685}) 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110544458}) 2024-09-18T10:27:08.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1036093912}) 2024-09-18T10:27:08.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085197560}) 2024-09-18T10:27:08.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312465003}) 2024-09-18T10:27:08.947 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085197560}) 2024-09-18T10:27:08.948 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581766497}) 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099330528}) 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492343864}) 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099330528}) 2024-09-18T10:27:08.949 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.942+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195798230}) 2024-09-18T10:27:08.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334909538}) 2024-09-18T10:27:08.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409623125}) 2024-09-18T10:27:08.950 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334909538}) 2024-09-18T10:27:08.951 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.946+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945261816}) 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706719919}) 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822108752}) 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706719919}) 2024-09-18T10:27:08.959 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626763293}) 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3493277874}) 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563757647}) 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3493277874}) 2024-09-18T10:27:08.961 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.954+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164081389}) 2024-09-18T10:27:08.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300192984}) 2024-09-18T10:27:08.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448883518}) 2024-09-18T10:27:08.962 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300192984}) 2024-09-18T10:27:08.963 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4046304952}) 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678887418}) 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813252823}) 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678887418}) 2024-09-18T10:27:08.964 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63137283}) 2024-09-18T10:27:08.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.965 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817269674}) 2024-09-18T10:27:08.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032638971}) 2024-09-18T10:27:08.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817269674}) 2024-09-18T10:27:08.966 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.958+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643891354}) 2024-09-18T10:27:08.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3238928674}) 2024-09-18T10:27:08.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3408620565}) 2024-09-18T10:27:08.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.968 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3238928674}) 2024-09-18T10:27:08.969 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130158686}) 2024-09-18T10:27:08.970 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121540544}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2150166652}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121540544}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.962+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275636063}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854828419}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984719183}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854828419}) 2024-09-18T10:27:08.971 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772910791}) 2024-09-18T10:27:08.972 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937172104}) 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3493635669}) 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937172104}) 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447215320}) 2024-09-18T10:27:08.973 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738660208}) 2024-09-18T10:27:08.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4185696664}) 2024-09-18T10:27:08.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738660208}) 2024-09-18T10:27:08.974 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.966+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171827807}) 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2} crc {logm=1250916624,mdsmap=1531151339,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2248175583,mon_config_key=1147460298}) 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=2} crc {logm=2927730223,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=3764429256,mon_config_key=620546692,monmap=4028308432}) 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2} crc {logm=1250916624,mdsmap=1531151339,mgr=255263467,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2248175583,mon_config_key=1147460298}) 2024-09-18T10:27:08.975 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.970+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=2} crc {logm=1922558890,mdsmap=2147943554,mgr=87021513,mgr_metadata=2738435100,mgrstat=4160224561,mon_config_key=620546692,monmap=4028308432}) 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=2166527243,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1605346506}) 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=5,osd_snap=2,osdmap=55} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=500417744,osd_snap=454132730,osdmap=557503576}) 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=54} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=2166527243,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1605346506}) 2024-09-18T10:27:08.985 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:27:08.978+0000 7f0b1ec79700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mon_config_key=3242667245,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=464678694}) 2024-09-18T10:27:09.828 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:27:09.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:09.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: get_monmap_and_config 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: build_initial_monmap 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: monmap: 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:09.936217+0000 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:09.936217+0000 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:09.939 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:09.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 Processor -- start 2024-09-18T10:27:09.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 -- start start 2024-09-18T10:27:09.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: init 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8d80999e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-b con 0x7fd8d809b160 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8d8137910 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d80a1c80 con 0x7fd8d8137910 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d8099db0 con 0x7fd8d80999e0 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d809a9f0 con 0x7fd8d809b160 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:09.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8dc8b3700 10 monclient(hunting): authenticate will time out at 1918.861843s 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46930/0 (socket says 172.21.15.142:46930) 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/2130522316 learned_addr learned my addr 172.21.15.142:0/2130522316 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): get_auth_request con 0x7fd8d809b160 auth_method 0 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_done global_id 75839 payload 274 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/2130522316 >> v1:172.21.15.150:6789/0 conn(0x7fd8d80999e0 legacy=0x7fd8d80a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/2130522316 >> v1:172.21.15.142:6789/0 conn(0x7fd8d8137910 legacy=0x7fd8d8139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:09.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:27:09.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 monclient: found mon.noname-b 2024-09-18T10:27:09.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:09.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:09.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.934+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/2130522316 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d80a1ec0 con 0x7fd8d809b160 2024-09-18T10:27:09.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/2130522316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 msgr2=0x7fd8d80a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:09.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d5d9b700 1 --2- 172.21.15.142:0/2130522316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0099e0 tx=0x7fd8cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30d3ed420fac1076 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 1 -- 172.21.15.142:0/2130522316 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00bb50 con 0x7fd8d809b160 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: dump: 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:09.948 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: _finish_auth 0 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 5 monclient: authenticate success, global_id 75839 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 1 -- 172.21.15.142:0/2130522316 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc00bcb0 con 0x7fd8d809b160 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:09.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 1 -- 172.21.15.142:0/2130522316 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc011670 con 0x7fd8d809b160 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 4 set_mon_vals no callback set 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8d559a700 10 monclient: dump: 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:09.950 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: shutdown 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/2130522316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 msgr2=0x7fd8d80a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/2130522316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc0099e0 tx=0x7fd8cc009250 comp rx=0 tx=0).stop 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/2130522316 shutdown_connections 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/2130522316 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d809b160 0x7fd8d80a0890 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/2130522316 >> 172.21.15.142:0/2130522316 conn(0x7fd8d80a5790 msgr2=0x7fd8d80a5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:09.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/2130522316 shutdown_connections 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/2130522316 wait complete. 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: build_initial_monmap 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: monmap: 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:09.952 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 Processor -- start 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- start start 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: init 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8d80999e0 0x7fd8d8141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:09.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-b con 0x7fd8d80999e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-a con 0x7fd8d809b160 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d8137910 0x7fd8d8142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): picked mon.noname-c con 0x7fd8d8137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d812fbb0 con 0x7fd8d809b160 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d8130b60 con 0x7fd8d80999e0 2024-09-18T10:27:09.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d812de10 con 0x7fd8d8137910 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.938+0000 7fd8dc8b3700 10 monclient(hunting): authenticate will time out at 1918.867891s 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52606/0 (socket says 172.21.15.142:52606) 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d8137910 0x7fd8d8142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:09.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/958666526 learned_addr learned my addr 172.21.15.142:0/958666526 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): get_auth_request con 0x7fd8d809b160 auth_method 0 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): get_auth_request con 0x7fd8d8137910 auth_method 0 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:09.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8d80999e0 0x7fd8d8141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): get_auth_request con 0x7fd8d80999e0 auth_method 0 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:09.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient(hunting): handle_auth_done global_id 204120 payload 824 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d8137910 msgr2=0x7fd8d8142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d8137910 0x7fd8d8142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8d80999e0 msgr2=0x7fd8d8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8d80999e0 0x7fd8d8141d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 monclient: found mon.noname-a 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:09.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/958666526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc0094f0 con 0x7fd8d809b160 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 -- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 msgr2=0x7fd8d81422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d5d9b700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8069fc0 tx=0x7fd8cc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc011c30 con 0x7fd8d809b160 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: dump: 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:09.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: _finish_auth 0 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 5 monclient: authenticate success, global_id 204120 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 10 monclient: _renew_subs 2024-09-18T10:27:09.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d812e540 con 0x7fd8d809b160 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc020070 con 0x7fd8d809b160 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc01b450 con 0x7fd8d809b160 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8c67fc700 10 monclient: dump: 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:09.963 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8d4d99700 4 set_mon_vals no callback set 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 10 monclient: _renew_subs 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.942+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d812f8d0 con 0x7fd8d809b160 2024-09-18T10:27:09.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8dc8b3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8dc8b3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8dc8b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d812f8d0 con 0x7fd8d809b160 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd8cc022030 con 0x7fd8d809b160 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8c67fc700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 0x7fd8b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd8cc04ac20 con 0x7fd8d809b160 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8d659c700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 0x7fd8b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.946+0000 7fd8d659c700 10 monclient: get_auth_request con 0x7fd8b8035ac0 auth_method 0 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.950+0000 7fd8d659c700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 0x7fd8b8037f70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0007fd0 tx=0x7fd8d00026a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.954+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd8cc01c7b0 con 0x7fd8d809b160 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.954+0000 7fd8c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:09.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:09.954+0000 7fd8c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:10.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8dc8b3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8dc8b3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8dc8b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd8d812f8d0 con 0x7fd8d809b160 2024-09-18T10:27:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8c67fc700 1 -- 172.21.15.142:0/958666526 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fd8cc01a840 con 0x7fd8d809b160 2024-09-18T10:27:10.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:10.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.174+0000 7fd8c67fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:10.181 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:10.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 msgr2=0x7fd8b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 0x7fd8b8037f70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0007fd0 tx=0x7fd8d00026a0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 10 monclient: shutdown 2024-09-18T10:27:10.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 msgr2=0x7fd8d81422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8069fc0 tx=0x7fd8cc016000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 shutdown_connections 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8b8035ac0 0x7fd8b8037f70 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8d8137910 0x7fd8d8142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8d809b160 0x7fd8d81422c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 --2- 172.21.15.142:0/958666526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8d80999e0 0x7fd8d8141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 >> 172.21.15.142:0/958666526 conn(0x7fd8d80a5790 msgr2=0x7fd8d809e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 shutdown_connections 2024-09-18T10:27:10.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.178+0000 7fd8dc8b3700 1 -- 172.21.15.142:0/958666526 wait complete. 2024-09-18T10:27:10.195 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:27:10.195 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:27:10.195 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:27:10.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:27:10.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.294+0000 7f0c892c5700 10 monclient: get_monmap_and_config 2024-09-18T10:27:10.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.294+0000 7f0c892c5700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.294+0000 7f0c892c5700 10 monclient: monmap: 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:10.299233+0000 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:10.299233+0000 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:10.302 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 Processor -- start 2024-09-18T10:27:10.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 -- start start 2024-09-18T10:27:10.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient: init 2024-09-18T10:27:10.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0c8409abd0 0x7f0c8409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-a con 0x7f0c8409abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c8409db80 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c84097880 0x7f0c84137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-b con 0x7f0c84097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840958c0 con 0x7f0c8409db80 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8406fa60 con 0x7f0c8409abd0 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84095ef0 con 0x7f0c84097880 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c892c5700 10 monclient(hunting): authenticate will time out at 1919.224852s 2024-09-18T10:27:10.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0c8409abd0 0x7f0c8409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c827fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0c8409db80 0x7f0c8409c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51312/0 (socket says 172.21.15.142:51312) 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c827fc700 1 -- 172.21.15.142:0/60785544 learned_addr learned my addr 172.21.15.142:0/60785544 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request con 0x7f0c8409abd0 auth_method 0 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c84097880 0x7f0c84137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): get_auth_request con 0x7f0c84097880 auth_method 0 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_done global_id 204123 payload 274 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 -- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c84097880 msgr2=0x7f0c84137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c84097880 0x7f0c84137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 -- 172.21.15.142:0/60785544 >> v1:172.21.15.142:6789/0 conn(0x7f0c8409db80 legacy=0x7f0c8409c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 monclient: found mon.noname-a 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 -- 172.21.15.142:0/60785544 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8409afa0 con 0x7f0c8409abd0 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 -- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 msgr2=0x7f0c8409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.298+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c8409bed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009e60 tx=0x7f0c74009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec19c926f166de97 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 1 -- 172.21.15.142:0/60785544 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400e070 con 0x7f0c8409abd0 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:10.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: dump: 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 5 monclient: authenticate success, global_id 204123 2024-09-18T10:27:10.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 1 -- 172.21.15.142:0/60785544 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7400bd70 con 0x7f0c8409abd0 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 1 -- 172.21.15.142:0/60785544 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74012660 con 0x7f0c8409abd0 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c81ffb700 10 monclient: dump: 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.313 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 4 set_mon_vals no callback set 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: shutdown 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 msgr2=0x7f0c8409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c8409bed0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009e60 tx=0x7f0c74009780 comp rx=0 tx=0).stop 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- 172.21.15.142:0/60785544 shutdown_connections 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c84097880 0x7f0c84137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- 172.21.15.142:0/60785544 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c8409bed0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- 172.21.15.142:0/60785544 >> 172.21.15.142:0/60785544 conn(0x7f0c840a14e0 msgr2=0x7f0c840a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- 172.21.15.142:0/60785544 shutdown_connections 2024-09-18T10:27:10.314 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- 172.21.15.142:0/60785544 wait complete. 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: monmap: 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:10.315 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:10.316 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:10.316 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 Processor -- start 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- start start 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: init 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c84097880 0x7f0c84141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-b con 0x7f0c84097880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c84141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-a con 0x7f0c8409abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 0x7f0c841421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): picked mon.noname-c con 0x7f0c8409db80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84097c90 con 0x7f0c8409abd0 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840a0020 con 0x7f0c84097880 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840927b0 con 0x7f0c8409db80 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c892c5700 10 monclient(hunting): authenticate will time out at 1919.230441s 2024-09-18T10:27:10.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 0x7f0c841421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c84141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 0x7f0c841421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46946/0 (socket says 172.21.15.142:46946) 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 learned_addr learned my addr 172.21.15.142:0/3624178599 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): get_auth_request con 0x7f0c8409db80 auth_method 0 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): get_auth_request con 0x7f0c8409abd0 auth_method 0 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c84097880 0x7f0c84141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c827fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request con 0x7f0c84097880 auth_method 0 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient(hunting): handle_auth_done global_id 75848 payload 824 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c84097880 msgr2=0x7f0c84141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c84097880 0x7f0c84141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 msgr2=0x7f0c84141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8409abd0 0x7f0c84141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 monclient: found mon.noname-c 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74009a20 con 0x7f0c8409db80 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.302+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c84097880 0x7f0c84141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 msgr2=0x7f0c841421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c837fe700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 0x7f0c841421e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00ea50 tx=0x7f0c7c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c0096b0 con 0x7f0c8409db80 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:10.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: dump: 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c00ca60 con 0x7f0c8409db80 2024-09-18T10:27:10.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 5 monclient: authenticate success, global_id 75848 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 monclient: _renew_subs 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c84091460 con 0x7f0c8409db80 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c00ce20 con 0x7f0c8409db80 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 10 monclient: dump: 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.327 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c817fa700 4 set_mon_vals no callback set 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 monclient: _renew_subs 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c84090610 con 0x7f0c8409db80 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 0x7f0c8408a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84064950 con 0x7f0c84089ff0 2024-09-18T10:27:10.328 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 0x7f0c8408a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient: get_auth_request con 0x7f0c84089ff0 auth_method 0 2024-09-18T10:27:10.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.329 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0c7c013680 con 0x7f0c8409db80 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_done global_id 204129 payload 824 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c82ffd700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c840918e0 con 0x7f0c84089ff0 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0c64035ac0 0x7f0c64037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0c7c049290 con 0x7f0c8409db80 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.306+0000 7f0c827fc700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0c64035ac0 0x7f0c64037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c827fc700 10 monclient: get_auth_request con 0x7f0c64035ac0 auth_method 0 2024-09-18T10:27:10.330 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c82ffd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 msgr2=0x7f0c8408a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 0x7f0c8408a3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009e60 tx=0x7f0c74018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74012e30 con 0x7f0c84089ff0 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 10 monclient: dump: 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.331 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0c74027440 con 0x7f0c84089ff0 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 msgr2=0x7f0c8408a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 0x7f0c8408a3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009e60 tx=0x7f0c74018000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 msgr2=0x7f0c8408a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c7affd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c84089ff0 0x7f0c8408a3c0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009e60 tx=0x7f0c74018000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.310+0000 7f0c827fc700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0c64035ac0 0x7f0c64037f70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007880 tx=0x7f0c6c007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c892c5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:10.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c892c5700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 0x7f0c8408ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c892c5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840696a0 con 0x7f0c8408a900 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 0x7f0c8408ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient: get_auth_request con 0x7f0c8408a900 auth_method 0 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 10 monclient(hunting): handle_auth_done global_id 204132 payload 824 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.322+0000 7f0c82ffd700 1 -- 172.21.15.142:0/3624178599 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0c84091ad0 con 0x7f0c8408a900 2024-09-18T10:27:10.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c82ffd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 msgr2=0x7f0c8408ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c82ffd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 0x7f0c8408ded0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c840a01e0 tx=0x7f0c74020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7401cd70 con 0x7f0c8408a900 2024-09-18T10:27:10.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 10 monclient: dump: 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.338 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f0c74011390 con 0x7f0c8408a900 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 msgr2=0x7f0c8408ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 0x7f0c8408ded0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c840a01e0 tx=0x7f0c74020040 comp rx=0 tx=0).stop 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 msgr2=0x7f0c8408ded0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c7affd700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0c8408a900 0x7f0c8408ded0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c840a01e0 tx=0x7f0c74020040 comp rx=0 tx=0).stop 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:10.340 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.341 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:10.342 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:10.343 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.344 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.345 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:10.346 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.347 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.348 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.349 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.350 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.351 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.352 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.353 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.354 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:10.355 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:10.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0c64035ac0 msgr2=0x7f0c64037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0c64035ac0 0x7f0c64037f70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007880 tx=0x7f0c6c007f90 comp rx=0 tx=0).stop 2024-09-18T10:27:10.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 10 monclient: shutdown 2024-09-18T10:27:10.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 msgr2=0x7f0c841421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 --2- 172.21.15.142:0/3624178599 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0c8409db80 0x7f0c841421e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00ea50 tx=0x7f0c7c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 reap_dead start 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c837fe700 1 -- 172.21.15.142:0/3624178599 reap_dead start 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 shutdown_connections 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 >> 172.21.15.142:0/3624178599 conn(0x7f0c840a14e0 msgr2=0x7f0c84096360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 shutdown_connections 2024-09-18T10:27:10.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.326+0000 7f0c892c5700 1 -- 172.21.15.142:0/3624178599 wait complete. 2024-09-18T10:27:10.358 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:10.358 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:27:10.359 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:27:10.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: get_monmap_and_config 2024-09-18T10:27:10.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: monmap: 2024-09-18T10:27:10.470 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:10.468018+0000 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:10.468018+0000 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:10.471 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 Processor -- start 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 -- start start 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: init 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36d80a0100 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36d806edb0 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f36d81360e0 0x7f36d81384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-c con 0x7f36d81360e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d806d090 con 0x7f36d806edb0 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d809ee20 con 0x7f36d80a0100 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d80a0e50 con 0x7f36d81360e0 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36dca17700 10 monclient(hunting): authenticate will time out at 1919.393520s 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d6ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f36d806edb0 0x7f36d806f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:51326/0 (socket says 172.21.15.142:51326) 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 learned_addr learned my addr 172.21.15.142:0/2785660917 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2785660917 >> v2:172.21.15.150:3300/0 conn(0x7f36d81360e0 0x7f36d81384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d7fff700 10 monclient(hunting): get_auth_request con 0x7f36d81360e0 auth_method 0 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.466+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 >> v1:172.21.15.142:6789/0 conn(0x7f36d806edb0 legacy=0x7f36d806f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922774509 0 0) 0x7f36d806d090 con 0x7f36d806edb0 2024-09-18T10:27:10.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): my global_id is 204135 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4003570 con 0x7f36d806edb0 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 1 -- 172.21.15.142:0/2785660917 >> v1:172.21.15.150:6789/0 conn(0x7f36d80a0100 legacy=0x7f36d809e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.0 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 37389498 0 0) 0x7f36c4003570 con 0x7f36d806edb0 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 >> v2:172.21.15.150:3300/0 conn(0x7f36d81360e0 msgr2=0x7f36d81384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 --2- 172.21.15.142:0/2785660917 >> v2:172.21.15.150:3300/0 conn(0x7f36d81360e0 0x7f36d81384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 >> v1:172.21.15.150:6789/0 conn(0x7f36d80a0100 legacy=0x7f36d809e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 monclient: found mon.noname-a-legacy 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.142:6789/0 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d806f8c0 con 0x7f36d806edb0 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36dca17700 5 monclient: authenticate success, global_id 204135 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36dca17700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.0 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f36c0001860 con 0x7f36d806edb0 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: dump: 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.488 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] but i'm connected to v1:172.21.15.142:6789/0, reconnecting 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 >> v1:172.21.15.142:6789/0 conn(0x7f36d806edb0 legacy=0x7f36d806f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4003c50 0x7f36c4004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): picked mon.b con 0x7f36c4003c50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 0x7f36c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): picked mon.c con 0x7f36c4004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36c4009a50 0x7f36c400be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): picked mon.a con 0x7f36c4009a50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c400c3c0 con 0x7f36c4009a50 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c400c530 con 0x7f36c4003c50 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c400c6f0 con 0x7f36c4004810 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d67fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36dca17700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 0x7f36c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): get_auth_request con 0x7f36c4004810 auth_method 0 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4003c50 0x7f36c4004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): get_auth_request con 0x7f36c4003c50 auth_method 0 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36c4009a50 0x7f36c400be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:10.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): get_auth_request con 0x7f36c4009a50 auth_method 0 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_done global_id 204135 payload 293 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4003c50 msgr2=0x7f36c4004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4003c50 0x7f36c4004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36c4009a50 msgr2=0x7f36c400be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36c4009a50 0x7f36c400be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 monclient: found mon.c 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4003570 con 0x7f36c4004810 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36c4009a50 0x7f36c400be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.470+0000 7f36d7fff700 1 -- 172.21.15.142:0/2785660917 reap_dead start 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 msgr2=0x7f36c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 0x7f36c4004c80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f36c000c5c0 tx=0x7f36c0007b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a89be7a084ad052 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000ce50 con 0x7f36c4004810 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: dump: 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0005920 con 0x7f36c4004810 2024-09-18T10:27:10.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 4 set_mon_vals no callback set 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 1 -- 172.21.15.142:0/2785660917 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0005ce0 con 0x7f36c4004810 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d67fc700 10 monclient: dump: 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.495 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: shutdown 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 msgr2=0x7f36c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 0x7f36c4004c80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f36c000c5c0 tx=0x7f36c0007b00 comp rx=0 tx=0).stop 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- 172.21.15.142:0/2785660917 shutdown_connections 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 --2- 172.21.15.142:0/2785660917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36c4004810 0x7f36c4004c80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- 172.21.15.142:0/2785660917 >> 172.21.15.142:0/2785660917 conn(0x7f36d8096a50 msgr2=0x7f36d8071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- 172.21.15.142:0/2785660917 shutdown_connections 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- 172.21.15.142:0/2785660917 wait complete. 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: monmap: 2024-09-18T10:27:10.496 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 Processor -- start 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- start start 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: init 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4004810 0x7f36d8135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-b con 0x7f36c4004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 0x7f36d811ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-c con 0x7f36d8135580 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36d811f110 0x7f36d811f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): picked mon.noname-a con 0x7f36d811f110 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8135950 con 0x7f36d811f110 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d806fad0 con 0x7f36c4004810 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d80a2530 con 0x7f36d8135580 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36dca17700 10 monclient(hunting): authenticate will time out at 1919.402833s 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 0x7f36d811ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 0x7f36d811ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46960/0 (socket says 172.21.15.142:46960) 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2587043633 learned_addr learned my addr 172.21.15.142:0/2587043633 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): get_auth_request con 0x7f36d8135580 auth_method 0 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4004810 0x7f36d8135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): get_auth_request con 0x7f36c4004810 auth_method 0 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36d811f110 0x7f36d811f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 10 monclient(hunting): get_auth_request con 0x7f36d811f110 auth_method 0 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient(hunting): handle_auth_done global_id 75851 payload 824 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4004810 msgr2=0x7f36d8135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36c4004810 0x7f36d8135040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36d811f110 msgr2=0x7f36d811f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36d811f110 0x7f36d811f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 monclient: found mon.noname-c 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8122160 con 0x7f36d8135580 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.474+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36d811f110 0x7f36d811f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d6ffd700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 msgr2=0x7f36d811ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d6ffd700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 0x7f36d811ebd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f36c400c880 tx=0x7f36c0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c00072e0 con 0x7f36d8135580 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: dump: 2024-09-18T10:27:10.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0007440 con 0x7f36d8135580 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 5 monclient: authenticate success, global_id 75851 2024-09-18T10:27:10.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 monclient: _renew_subs 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d8120b40 con 0x7f36d8135580 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0007800 con 0x7f36d8135580 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36cf7fe700 10 monclient: dump: 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.503 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d5ffb700 4 set_mon_vals no callback set 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 monclient: _renew_subs 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d8120630 con 0x7f36d8135580 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 0x7f36d809a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81228c0 con 0x7f36d811faa0 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 0x7f36d809a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d77fe700 10 monclient: get_auth_request con 0x7f36d811faa0 auth_method 0 2024-09-18T10:27:10.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.478+0000 7f36d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f36c0004a40 con 0x7f36d8135580 2024-09-18T10:27:10.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 10 monclient(hunting): handle_auth_done global_id 75854 payload 824 2024-09-18T10:27:10.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36d81223a0 con 0x7f36d811faa0 2024-09-18T10:27:10.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f36b8035af0 0x7f36b8037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f36c0049a40 con 0x7f36d8135580 2024-09-18T10:27:10.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f36b8035af0 0x7f36b8037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d7fff700 10 monclient: get_auth_request con 0x7f36b8035af0 auth_method 0 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 msgr2=0x7f36d809a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 0x7f36d809a950 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d0001920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0012d40 con 0x7f36d811faa0 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 10 monclient: dump: 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.510 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f36d0012ea0 con 0x7f36d811faa0 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 msgr2=0x7f36d809a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 0x7f36d809a950 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d0001920 comp rx=0 tx=0).stop 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 msgr2=0x7f36d809a950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.482+0000 7f36cf7fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d811faa0 0x7f36d809a950 secure :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d0001920 comp rx=0 tx=0).stop 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.486+0000 7f36d7fff700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f36b8035af0 0x7f36b8037fa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f36d809e430 tx=0x7f36c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.511 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36dca17700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36dca17700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 0x7f36d809b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36dca17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8064810 con 0x7f36d809ae90 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 0x7f36d809b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient: get_auth_request con 0x7f36d809ae90 auth_method 0 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 10 monclient(hunting): handle_auth_done global_id 75857 payload 824 2024-09-18T10:27:10.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.494+0000 7f36d77fe700 1 -- 172.21.15.142:0/2587043633 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f36d8072a80 con 0x7f36d809ae90 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36d77fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 msgr2=0x7f36d809b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36d77fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 0x7f36d809b290 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0015ca0 con 0x7f36d809ae90 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 10 monclient: dump: 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f36d0015e00 con 0x7f36d809ae90 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 msgr2=0x7f36d809b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 0x7f36d809b290 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d001c000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 msgr2=0x7f36d809b290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36cf7fe700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d809ae90 0x7f36d809b290 secure :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f36d0007670 tx=0x7f36d001c000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.514 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:10.515 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.516 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:10.517 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:10.518 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.519 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.520 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.521 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.522 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.523 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.524 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:27:10.525 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.526 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.527 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.527 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.527 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:10.527 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f36b8035af0 msgr2=0x7f36b8037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f36b8035af0 0x7f36b8037fa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f36d809e430 tx=0x7f36c8008040 comp rx=0 tx=0).stop 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 10 monclient: shutdown 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 msgr2=0x7f36d811ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 --2- 172.21.15.142:0/2587043633 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36d8135580 0x7f36d811ebd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f36c400c880 tx=0x7f36c0015000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36d7fff700 1 -- 172.21.15.142:0/2587043633 reap_dead start 2024-09-18T10:27:10.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36d7fff700 1 -- 172.21.15.142:0/2587043633 reap_dead start 2024-09-18T10:27:10.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 shutdown_connections 2024-09-18T10:27:10.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 >> 172.21.15.142:0/2587043633 conn(0x7f36d8096a50 msgr2=0x7f36d8069ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 shutdown_connections 2024-09-18T10:27:10.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.498+0000 7f36dca17700 1 -- 172.21.15.142:0/2587043633 wait complete. 2024-09-18T10:27:10.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:10.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:27:10.529 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:27:10.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: get_monmap_and_config 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: monmap: 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:10.637194+0000 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:10.637194+0000 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:10.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:10.641 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:10.641 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:10.641 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 Processor -- start 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 -- start start 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: init 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1c740999e0 0x7f1c740a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-c con 0x7f1c740999e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c7409c410 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-b con 0x7f1c74137910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c740a20c0 con 0x7f1c7409c410 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7409a000 con 0x7f1c740999e0 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740a1c80 con 0x7f1c74137910 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a0ab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a8ac700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.634+0000 7f1c7a0ab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46982/0 (socket says 172.21.15.142:46982) 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a8ac700 10 monclient(hunting): authenticate will time out at 1919.564013s 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2935410681 learned_addr learned my addr 172.21.15.142:0/2935410681 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): get_auth_request con 0x7f1c74137910 auth_method 0 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 1 --2- 172.21.15.142:0/2935410681 >> v2:172.21.15.150:3300/0 conn(0x7f1c740999e0 0x7f1c740a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): get_auth_request con 0x7f1c740999e0 auth_method 0 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_done global_id 75860 payload 274 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2935410681 >> v2:172.21.15.150:3300/0 conn(0x7f1c740999e0 msgr2=0x7f1c740a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 --2- 172.21.15.142:0/2935410681 >> v2:172.21.15.150:3300/0 conn(0x7f1c740999e0 0x7f1c740a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2935410681 >> v1:172.21.15.150:6789/0 conn(0x7f1c7409c410 legacy=0x7f1c740a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 monclient: found mon.noname-b 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2935410681 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7409a580 con 0x7f1c74137910 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2935410681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 msgr2=0x7f1c74139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a0ab700 1 --2- 172.21.15.142:0/2935410681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c56dab5232db3f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 1 -- 172.21.15.142:0/2935410681 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c00e070 con 0x7f1c74137910 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: dump: 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a8ac700 5 monclient: authenticate success, global_id 75860 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 1 -- 172.21.15.142:0/2935410681 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c00bbe0 con 0x7f1c74137910 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a8ac700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a8ac700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c7a8ac700 4 set_mon_vals no callback set 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 1 -- 172.21.15.142:0/2935410681 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c0092e0 con 0x7f1c74137910 2024-09-18T10:27:10.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.638+0000 7f1c788a8700 10 monclient: dump: 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: shutdown 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2935410681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 msgr2=0x7f1c74139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2935410681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c009840 comp rx=0 tx=0).stop 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2935410681 shutdown_connections 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2935410681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c74137910 0x7f1c74139cf0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2935410681 >> v2:172.21.15.150:3300/0 conn(0x7f1c740999e0 0x7f1c740a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2935410681 >> 172.21.15.142:0/2935410681 conn(0x7f1c740a5790 msgr2=0x7f1c740a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2935410681 shutdown_connections 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2935410681 wait complete. 2024-09-18T10:27:10.654 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: monmap: 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:10.655 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 Processor -- start 2024-09-18T10:27:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- start start 2024-09-18T10:27:10.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: init 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c740999e0 0x7f1c74095d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-a con 0x7f1c740999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-c con 0x7f1c7409c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74137910 0x7f1c7406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): picked mon.noname-b con 0x7f1c74137910 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:10.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7408daf0 con 0x7f1c740999e0 2024-09-18T10:27:10.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7408d6d0 con 0x7f1c74137910 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7408ce30 con 0x7f1c7409c410 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a8ac700 10 monclient(hunting): authenticate will time out at 1919.570530s 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c740999e0 0x7f1c74095d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c740999e0 0x7f1c74095d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:52670/0 (socket says 172.21.15.142:52670) 2024-09-18T10:27:10.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:46992/0 (socket says 172.21.15.142:46992) 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 1 -- 172.21.15.142:0/2319472096 learned_addr learned my addr 172.21.15.142:0/2319472096 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): get_auth_request con 0x7f1c740999e0 auth_method 0 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): get_auth_request con 0x7f1c7409c410 auth_method 0 2024-09-18T10:27:10.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74137910 0x7f1c7406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): get_auth_request con 0x7f1c74137910 auth_method 0 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c7a0ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient(hunting): handle_auth_done global_id 75863 payload 824 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74137910 msgr2=0x7f1c7406d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74137910 0x7f1c7406d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c740999e0 msgr2=0x7f1c74095d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c740999e0 0x7f1c74095d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 monclient: found mon.noname-c 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.642+0000 7f1c790a9700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c009ae0 con 0x7f1c7409c410 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c790a9700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 msgr2=0x7f1c7406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c790a9700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c4e0 tx=0x7f1c64007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400ce90 con 0x7f1c7409c410 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: dump: 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.669 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 5 monclient: authenticate success, global_id 75863 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c640117a0 con 0x7f1c7409c410 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 monclient: _renew_subs 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64011b80 con 0x7f1c7409c410 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7408d110 con 0x7f1c7409c410 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c6bfff700 4 set_mon_vals no callback set 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 10 monclient: dump: 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 monclient: _renew_subs 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c74095240 con 0x7f1c7409c410 2024-09-18T10:27:10.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 0x7f1c740978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740647f0 con 0x7f1c7406dc60 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 0x7f1c740978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient: get_auth_request con 0x7f1c7406dc60 auth_method 0 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1c6401c460 con 0x7f1c7409c410 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.646+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1c58035ac0 0x7f1c58037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c798aa700 10 monclient(hunting): handle_auth_done global_id 194269 payload 824 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1c64048b90 con 0x7f1c7409c410 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c798aa700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c7406e030 con 0x7f1c7406dc60 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c7a0ab700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1c58035ac0 0x7f1c58037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c7a0ab700 10 monclient: get_auth_request con 0x7f1c58035ac0 auth_method 0 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c798aa700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 msgr2=0x7f1c740978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c798aa700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 0x7f1c740978e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c70011c20 tx=0x7f1c700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7000fde0 con 0x7f1c7406dc60 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 10 monclient: dump: 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.673 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c7a0ab700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1c58035ac0 0x7f1c58037f70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1c7001d480 con 0x7f1c7406dc60 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 msgr2=0x7f1c740978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 0x7f1c740978e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c70011c20 tx=0x7f1c700091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 msgr2=0x7f1c740978e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.650+0000 7f1c69ffb700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c7406dc60 0x7f1c740978e0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1c70011c20 tx=0x7f1c700091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c7a8ac700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 0x7f1c74098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c7a8ac700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74136b80 con 0x7f1c74097e20 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 0x7f1c74098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient: get_auth_request con 0x7f1c74097e20 auth_method 0 2024-09-18T10:27:10.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 10 monclient(hunting): handle_auth_done global_id 194272 payload 824 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.662+0000 7f1c798aa700 1 -- 172.21.15.142:0/2319472096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1c74072a80 con 0x7f1c74097e20 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c798aa700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 msgr2=0x7f1c74098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c798aa700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 0x7f1c74098220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c7001c000 tx=0x7f1c7001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7001dea0 con 0x7f1c74097e20 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:10.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 10 monclient: dump: 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f1c7001c3e0 con 0x7f1c74097e20 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:10.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 msgr2=0x7f1c74098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 0x7f1c74098220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c7001c000 tx=0x7f1c7001d000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 msgr2=0x7f1c74098220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.666+0000 7f1c69ffb700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c74097e20 0x7f1c74098220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 178, 2024-09-18T10:27:10.680 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.681 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.682 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:10.683 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:10.684 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:10.686 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.688 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:10.689 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.690 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:10.691 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:10.692 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1c58035ac0 msgr2=0x7f1c58037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1c58035ac0 0x7f1c58037f70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009f20 tx=0x7f1c6c017000 comp rx=0 tx=0).stop 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 10 monclient: shutdown 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 msgr2=0x7f1c7406d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c4e0 tx=0x7f1c64007ea0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a0ab700 1 -- 172.21.15.142:0/2319472096 reap_dead start 2024-09-18T10:27:10.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 shutdown_connections 2024-09-18T10:27:10.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 --2- 172.21.15.142:0/2319472096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c7409c410 0x7f1c7406d1e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 >> 172.21.15.142:0/2319472096 conn(0x7f1c740a5790 msgr2=0x7f1c74096590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 shutdown_connections 2024-09-18T10:27:10.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.670+0000 7f1c7a8ac700 1 -- 172.21.15.142:0/2319472096 wait complete. 2024-09-18T10:27:10.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:10.694 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:27:10.694 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:27:10.695 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:27:10.695 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:27:10.695 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:27:10.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:27:10.712 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:27:10.712 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:27:10.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.854+0000 7f326674e700 10 monclient: get_monmap_and_config 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.854+0000 7f326674e700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.854+0000 7f326674e700 10 monclient: monmap: 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:10.859229+0000 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:10.859229+0000 2024-09-18T10:27:10.862 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:10.863 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 Processor -- start 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 -- start start 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient: init 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32600a5f90 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:10.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): picked mon.noname-b con 0x7f32600990b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f326009e360 0x7f3260139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): picked mon.noname-c con 0x7f326009e360 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326009d980 con 0x7f32600a5f90 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326009d780 con 0x7f326009e360 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326009fc70 con 0x7f32600990b0 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f326674e700 10 monclient(hunting): authenticate will time out at 1919.784976s 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f325ffff700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f32600a5f90 legacy=0x7f32600a0270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:10.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:47000/0 (socket says 172.21.15.142:47000) 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 -- 172.21.15.142:0/1588902586 learned_addr learned my addr 172.21.15.142:0/1588902586 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): get_auth_request con 0x7f32600990b0 auth_method 0 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3264ceb700 1 --2- 172.21.15.142:0/1588902586 >> v2:172.21.15.150:3300/0 conn(0x7f326009e360 0x7f3260139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3264ceb700 10 monclient(hunting): get_auth_request con 0x7f326009e360 auth_method 0 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3264ceb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3264ceb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3264ceb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient(hunting): handle_auth_done global_id 75866 payload 274 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 -- 172.21.15.142:0/1588902586 >> v2:172.21.15.150:3300/0 conn(0x7f326009e360 msgr2=0x7f3260139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 --2- 172.21.15.142:0/1588902586 >> v2:172.21.15.150:3300/0 conn(0x7f326009e360 0x7f3260139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 -- 172.21.15.142:0/1588902586 >> v1:172.21.15.142:6789/0 conn(0x7f32600a5f90 legacy=0x7f32600a0270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 monclient: found mon.noname-b 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.858+0000 7f3257fff700 1 -- 172.21.15.142:0/1588902586 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32600a6360 con 0x7f32600990b0 2024-09-18T10:27:10.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 1 -- 172.21.15.142:0/1588902586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 msgr2=0x7f32600a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 1 --2- 172.21.15.142:0/1588902586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32480099b0 tx=0x7f3248009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50d9b19944d03c67 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 1 -- 172.21.15.142:0/1588902586 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324800ba70 con 0x7f32600990b0 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: dump: 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.878 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 5 monclient: authenticate success, global_id 75866 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 1 -- 172.21.15.142:0/1588902586 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324800bbd0 con 0x7f32600990b0 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 1 -- 172.21.15.142:0/1588902586 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248011420 con 0x7f32600990b0 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 4 set_mon_vals no callback set 2024-09-18T10:27:10.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325f7fe700 10 monclient: dump: 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: shutdown 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- 172.21.15.142:0/1588902586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 msgr2=0x7f32600a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- 172.21.15.142:0/1588902586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f32480099b0 tx=0x7f3248009250 comp rx=0 tx=0).stop 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- 172.21.15.142:0/1588902586 shutdown_connections 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- 172.21.15.142:0/1588902586 >> v2:172.21.15.150:3300/0 conn(0x7f326009e360 0x7f3260139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- 172.21.15.142:0/1588902586 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600990b0 0x7f32600a0980 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- 172.21.15.142:0/1588902586 >> 172.21.15.142:0/1588902586 conn(0x7f3260071680 msgr2=0x7f32600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- 172.21.15.142:0/1588902586 shutdown_connections 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- 172.21.15.142:0/1588902586 wait complete. 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: build_initial_monmap 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: monmap: 2024-09-18T10:27:10.881 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 Processor -- start 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- start start 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: init 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:10.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 0x7f3260141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): picked mon.noname-a con 0x7f32600990b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326009e360 0x7f3260142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): picked mon.noname-b con 0x7f326009e360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): picked mon.noname-c con 0x7f32600a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32600907f0 con 0x7f32600990b0 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326008f5b0 con 0x7f326009e360 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3260090510 con 0x7f32600a5f90 2024-09-18T10:27:10.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f326674e700 10 monclient(hunting): authenticate will time out at 1919.791399s 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 msgr2=0x7f3260141d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f325ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 0x7f3260141d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:47016/0 (socket says 172.21.15.142:47016) 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 1 -- 172.21.15.142:0/3453545199 learned_addr learned my addr 172.21.15.142:0/3453545199 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): get_auth_request con 0x7f32600a5f90 auth_method 0 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3264ceb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326009e360 0x7f3260142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 10 monclient(hunting): get_auth_request con 0x7f326009e360 auth_method 0 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.862+0000 7f3257fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3257fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 10 monclient(hunting): handle_auth_done global_id 75869 payload 824 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 -- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326009e360 msgr2=0x7f3260142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326009e360 0x7f3260142290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 -- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 msgr2=0x7f3260141d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 0x7f3260141d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 10 monclient: _finish_hunting 0 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 monclient: found mon.noname-c 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 -- 172.21.15.142:0/3453545199 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480094f0 con 0x7f32600a5f90 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 -- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 msgr2=0x7f326008d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f3264ceb700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3258009fa0 tx=0x7f3258009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325800ee80 con 0x7f32600a5f90 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: dump: 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.886 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: _finish_auth 0 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3258014420 con 0x7f32600a5f90 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 5 monclient: authenticate success, global_id 75869 2024-09-18T10:27:10.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580147c0 con 0x7f32600a5f90 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 10 monclient: _renew_subs 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3260090ad0 con 0x7f32600a5f90 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325effd700 4 set_mon_vals no callback set 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f325cff9700 10 monclient: dump: 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:10.888 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 10 monclient: _renew_subs 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.866+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32601427d0 con 0x7f32600a5f90 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f3258014b70 con 0x7f32600a5f90 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f325cff9700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 0x7f3240037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f32580496b0 con 0x7f32600a5f90 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f326674e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f326674e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f326674e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32600696a0 con 0x7f32600a5f90 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f325ffff700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 0x7f3240037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.870+0000 7f325ffff700 10 monclient: get_auth_request con 0x7f3240035a70 auth_method 0 2024-09-18T10:27:10.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.874+0000 7f325ffff700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 0x7f3240037f20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3250006fd0 tx=0x7f3250006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:10.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.878+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f325801bb00 con 0x7f32600a5f90 2024-09-18T10:27:10.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.878+0000 7f325cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:10.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:10.878+0000 7f325cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:11.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f326674e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:11.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f326674e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:11.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f326674e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:11.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f326009cc60 con 0x7f32600a5f90 2024-09-18T10:27:11.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f325cff9700 1 -- 172.21.15.142:0/3453545199 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f325802c620 con 0x7f32600a5f90 2024-09-18T10:27:11.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f325cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:11.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.106+0000 7f325cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:11.113 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:11.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 msgr2=0x7f3240037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:11.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 0x7f3240037f20 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3250006fd0 tx=0x7f3250006e10 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 10 monclient: shutdown 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 msgr2=0x7f326008d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3258009fa0 tx=0x7f3258009900 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 shutdown_connections 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3240035a70 0x7f3240037f20 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32600a5f90 0x7f326008d3c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f326009e360 0x7f3260142290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 --2- 172.21.15.142:0/3453545199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32600990b0 0x7f3260141d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 >> 172.21.15.142:0/3453545199 conn(0x7f3260071680 msgr2=0x7f326009f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 shutdown_connections 2024-09-18T10:27:11.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:11.110+0000 7f326674e700 1 -- 172.21.15.142:0/3453545199 wait complete. 2024-09-18T10:27:14.128 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: get_monmap_and_config 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: build_initial_monmap 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: monmap: 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:14.237518+0000 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:14.237518+0000 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:14.240 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:14.241 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 Processor -- start 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 -- start start 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: init 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2d2809a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d28099220 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d2809c4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:14.244 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2809abc0 con 0x7f2d2809c4f0 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2809d950 con 0x7f2d28099220 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2809db90 con 0x7f2d2809a7f0 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d27fff700 10 monclient(hunting): authenticate will time out at 1923.163019s 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d277fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f2d2809c4f0 legacy=0x7f2d2813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60280/0 (socket says 172.21.15.142:60280) 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 learned_addr learned my addr 172.21.15.142:0/3402325165 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request con 0x7f2d2809a7f0 auth_method 0 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:14.245 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_done global_id 75872 payload 274 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> v1:172.21.15.150:6789/0 conn(0x7f2d28099220 legacy=0x7f2d2809bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> v1:172.21.15.142:6789/0 conn(0x7f2d2809c4f0 legacy=0x7f2d2813e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 monclient: found mon.noname-b 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.234+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28099a30 con 0x7f2d2809a7f0 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.238+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 msgr2=0x7f2d28098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:14.246 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.238+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 msgr2=0x7f2d28098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:14.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.238+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f2d1c0099e0 tx=0x7f2d1c009250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:14.247 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.238+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:14.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient: get_auth_request con 0x7f2d2809a7f0 auth_method 0 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 0 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 msgr2=0x7f2d28098ce0 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d2809a7f0 0x7f2d28098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient: _add_conns ranks=[4,3] 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 --2- 172.21.15.142:0/3402325165 >> v2:172.21.15.150:3300/0 conn(0x7f2d10003550 0x7f2d100059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f2d10003550 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:14.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 --2- 172.21.15.142:0/3402325165 >> v2:172.21.15.142:3300/0 conn(0x7f2d100062f0 0x7f2d100087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f2d100062f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d10009070 con 0x7f2d100062f0 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d100091e0 con 0x7f2d10003550 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d25ffb700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d267fc700 1 -- 172.21.15.142:0/3402325165 >> v2:172.21.15.142:3300/0 conn(0x7f2d100062f0 msgr2=0x7f2d100087a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d267fc700 1 --2- 172.21.15.142:0/3402325165 >> v2:172.21.15.142:3300/0 conn(0x7f2d100062f0 0x7f2d100087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> v2:172.21.15.150:3300/0 conn(0x7f2d10003550 0x7f2d100059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request con 0x7f2d10003550 auth_method 0 2024-09-18T10:27:14.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_done global_id 75872 payload 293 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> v2:172.21.15.142:3300/0 conn(0x7f2d100062f0 msgr2=0x7f2d100087a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> v2:172.21.15.142:3300/0 conn(0x7f2d100062f0 0x7f2d100087a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 monclient: found mon.noname-c 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:27:14.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.442+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10009410 con 0x7f2d10003550 2024-09-18T10:27:14.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3402325165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10003550 msgr2=0x7f2d100059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:14.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10003550 0x7f2d100059c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009340 tx=0x7f2d1c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb52be8b1d6f963a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:14.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c017070 con 0x7f2d10003550 2024-09-18T10:27:14.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:14.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: dump: 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:14.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:14.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c01b410 con 0x7f2d10003550 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 5 monclient: authenticate success, global_id 75872 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 1 -- 172.21.15.142:0/3402325165 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c01b7b0 con 0x7f2d10003550 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 4 set_mon_vals no callback set 2024-09-18T10:27:14.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d25ffb700 10 monclient: dump: 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.457 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 10 monclient: shutdown 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 -- 172.21.15.142:0/3402325165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10003550 msgr2=0x7f2d100059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10003550 0x7f2d100059c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009340 tx=0x7f2d1c008040 comp rx=0 tx=0).stop 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d277fe700 1 -- 172.21.15.142:0/3402325165 reap_dead start 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 -- 172.21.15.142:0/3402325165 shutdown_connections 2024-09-18T10:27:14.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3402325165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d10003550 0x7f2d100059c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 -- 172.21.15.142:0/3402325165 >> 172.21.15.142:0/3402325165 conn(0x7f2d280a1fe0 msgr2=0x7f2d28070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 -- 172.21.15.142:0/3402325165 shutdown_connections 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 1 -- 172.21.15.142:0/3402325165 wait complete. 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 10 monclient: build_initial_monmap 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.446+0000 7f2d27fff700 10 monclient: monmap: 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:14.459 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:14.460 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:14.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:14.461 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 Processor -- start 2024-09-18T10:27:14.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 -- start start 2024-09-18T10:27:14.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient: init 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-c con 0x7f2d10003550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 0x7f2d2806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-a con 0x7f2d2806e670 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:14.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28092a90 0x7f2d28092ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): picked mon.noname-b con 0x7f2d28092a90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:14.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.472 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2806efc0 con 0x7f2d2806e670 2024-09-18T10:27:14.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2806f970 con 0x7f2d28092a90 2024-09-18T10:27:14.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:14.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28097800 con 0x7f2d10003550 2024-09-18T10:27:14.473 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:14.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d27fff700 10 monclient(hunting): authenticate will time out at 1923.377180s 2024-09-18T10:27:14.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d267fc700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 msgr2=0x7f2d2806ea80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:14.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d267fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 0x7f2d2806ea80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:14.474 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.475 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60298/0 (socket says 172.21.15.142:60298) 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3451608469 learned_addr learned my addr 172.21.15.142:0/3451608469 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request con 0x7f2d10003550 auth_method 0 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:14.476 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28092a90 0x7f2d28092ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:14.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:14.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:14.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): get_auth_request con 0x7f2d28092a90 auth_method 0 2024-09-18T10:27:14.477 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:14.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient(hunting): handle_auth_done global_id 75875 payload 824 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28092a90 msgr2=0x7f2d28092ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28092a90 0x7f2d28092ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 msgr2=0x7f2d2806ea80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 0x7f2d2806ea80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:27:14.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 monclient: found mon.noname-c 2024-09-18T10:27:14.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:14.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.450+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3451608469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2806a4b0 con 0x7f2d10003550 2024-09-18T10:27:14.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d26ffd700 1 -- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 msgr2=0x7f2d281460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:14.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d26ffd700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b950 tx=0x7f2d1c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:14.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c024430 con 0x7f2d10003550 2024-09-18T10:27:14.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:14.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:14.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: dump: 2024-09-18T10:27:14.481 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.482 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:14.483 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:14.483 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:14.483 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: _finish_auth 0 2024-09-18T10:27:14.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:14.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 5 monclient: authenticate success, global_id 75875 2024-09-18T10:27:14.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _renew_subs 2024-09-18T10:27:14.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0245b0 con 0x7f2d10003550 2024-09-18T10:27:14.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2806b890 con 0x7f2d10003550 2024-09-18T10:27:14.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:14.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c024970 con 0x7f2d10003550 2024-09-18T10:27:14.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:14.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:14.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 10 monclient: dump: 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:14.486 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d257fa700 4 set_mon_vals no callback set 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _renew_subs 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2806a930 con 0x7f2d10003550 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2d1c022040 con 0x7f2d10003550 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 0x7f2d14037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2d1c049df0 con 0x7f2d10003550 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d267fc700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 0x7f2d14037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:14.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d267fc700 10 monclient: get_auth_request con 0x7f2d14035aa0 auth_method 0 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.454+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2806a930 con 0x7f2d10003550 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.458+0000 7f2d267fc700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 0x7f2d14037f50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2d2809c360 tx=0x7f2d20007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.466+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2d1c020080 con 0x7f2d10003550 2024-09-18T10:27:14.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.466+0000 7f2d0effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:14.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.466+0000 7f2d0effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:14.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:14.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:14.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:14.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2d2806a930 con 0x7f2d10003550 2024-09-18T10:27:14.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d0effd700 1 -- 172.21.15.142:0/3451608469 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2d1c029030 con 0x7f2d10003550 2024-09-18T10:27:14.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d0effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:14.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d0effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:14.696 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 msgr2=0x7f2d14037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 0x7f2d14037f50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2d2809c360 tx=0x7f2d20007e80 comp rx=0 tx=0).stop 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 10 monclient: shutdown 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 msgr2=0x7f2d281460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b950 tx=0x7f2d1c019000 comp rx=0 tx=0).stop 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 shutdown_connections 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2d14035aa0 0x7f2d14037f50 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d28092a90 0x7f2d28092ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d2806e670 0x7f2d2806ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 --2- 172.21.15.142:0/3451608469 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d10003550 0x7f2d281460e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:14.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 >> 172.21.15.142:0/3451608469 conn(0x7f2d280a1fe0 msgr2=0x7f2d28139b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:14.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 shutdown_connections 2024-09-18T10:27:14.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:14.690+0000 7f2d27fff700 1 -- 172.21.15.142:0/3451608469 wait complete. 2024-09-18T10:27:17.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:17.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: get_monmap_and_config 2024-09-18T10:27:17.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: build_initial_monmap 2024-09-18T10:27:17.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: monmap: 2024-09-18T10:27:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:17.818615+0000 2024-09-18T10:27:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:17.818615+0000 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:17.826 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:17.827 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:17.827 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 1 Processor -- start 2024-09-18T10:27:17.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 1 -- start start 2024-09-18T10:27:17.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: init 2024-09-18T10:27:17.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:17.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.814+0000 7fdb3f585700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:27:17.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb3809a7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:17.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb380993f0 0x7fdb3813b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-c con 0x7fdb380993f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 0x7fdb3813df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-a con 0x7fdb3813bb30 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3809abc0 con 0x7fdb3809a7f0 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb38099810 con 0x7fdb3813bb30 2024-09-18T10:27:17.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3809d950 con 0x7fdb380993f0 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3f585700 10 monclient(hunting): authenticate will time out at 1926.744134s 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3ed84700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 msgr2=0x7fdb3813df10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3ed84700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 0x7fdb3813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:17.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb380993f0 0x7fdb3813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb380993f0 0x7fdb3813b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:40280/0 (socket says 172.21.15.142:40280) 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 -- 172.21.15.142:0/418007778 learned_addr learned my addr 172.21.15.142:0/418007778 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): get_auth_request con 0x7fdb380993f0 auth_method 0 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3e583700 1 --1- 172.21.15.142:0/418007778 >> v1:172.21.15.150:6789/0 conn(0x7fdb3809a7f0 0x7fdb38098ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:44930/0 (socket says 172.21.15.142:44930) 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:17.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:17.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:17.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3e583700 1 -- 172.21.15.142:0/418007778 >> v1:172.21.15.150:6789/0 conn(0x7fdb3809a7f0 legacy=0x7fdb38098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:17.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient(hunting): handle_auth_done global_id 194284 payload 274 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 -- 172.21.15.142:0/418007778 >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 msgr2=0x7fdb3813df10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 --2- 172.21.15.142:0/418007778 >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 0x7fdb3813df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 -- 172.21.15.142:0/418007778 >> v1:172.21.15.150:6789/0 conn(0x7fdb3809a7f0 legacy=0x7fdb38098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient: _finish_hunting 0 2024-09-18T10:27:17.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 monclient: found mon.noname-c 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.818+0000 7fdb3dd82700 1 -- 172.21.15.142:0/418007778 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3809db90 con 0x7fdb380993f0 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3dd82700 1 -- 172.21.15.142:0/418007778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb380993f0 msgr2=0x7fdb3813b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3dd82700 1 --2- 172.21.15.142:0/418007778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb380993f0 0x7fdb3813b5f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb3809d6b0 tx=0x7fdb28009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abbe848aa2e7ece4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 1 -- 172.21.15.142:0/418007778 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800bba0 con 0x7fdb380993f0 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: dump: 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:17.841 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: _finish_auth 0 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 1 -- 172.21.15.142:0/418007778 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2800bd00 con 0x7fdb380993f0 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 1 -- 172.21.15.142:0/418007778 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28011670 con 0x7fdb380993f0 2024-09-18T10:27:17.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3d581700 10 monclient: dump: 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 5 monclient: authenticate success, global_id 194284 2024-09-18T10:27:17.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 4 set_mon_vals no callback set 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 10 monclient: shutdown 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 -- 172.21.15.142:0/418007778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb380993f0 msgr2=0x7fdb3813b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 --2- 172.21.15.142:0/418007778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb380993f0 0x7fdb3813b5f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb3809d6b0 tx=0x7fdb28009250 comp rx=0 tx=0).stop 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 -- 172.21.15.142:0/418007778 shutdown_connections 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 --2- 172.21.15.142:0/418007778 >> v2:172.21.15.142:3300/0 conn(0x7fdb3813bb30 0x7fdb3813df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 --2- 172.21.15.142:0/418007778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb380993f0 0x7fdb3813b5f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 -- 172.21.15.142:0/418007778 >> 172.21.15.142:0/418007778 conn(0x7fdb380a1fe0 msgr2=0x7fdb38070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 -- 172.21.15.142:0/418007778 shutdown_connections 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.822+0000 7fdb3f585700 1 -- 172.21.15.142:0/418007778 wait complete. 2024-09-18T10:27:17.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient: build_initial_monmap 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient: monmap: 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 Processor -- start 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 -- start start 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient: init 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-c con 0x7fdb380993f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 0x7fdb3806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-a con 0x7fdb3809a7f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3813bb30 0x7fdb3806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): picked mon.noname-b con 0x7fdb3813bb30 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:17.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3806f970 con 0x7fdb3809a7f0 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3806d830 con 0x7fdb3813bb30 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3806bf30 con 0x7fdb380993f0 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3f585700 10 monclient(hunting): authenticate will time out at 1926.753538s 2024-09-18T10:27:17.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3dd82700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 msgr2=0x7fdb3806a2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60304/0 (socket says 172.21.15.142:60304) 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3dd82700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 0x7fdb3806a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 -- 172.21.15.142:0/423581296 learned_addr learned my addr 172.21.15.142:0/423581296 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): get_auth_request con 0x7fdb380993f0 auth_method 0 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:17.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3813bb30 0x7fdb3806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): get_auth_request con 0x7fdb3813bb30 auth_method 0 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3ed84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient(hunting): handle_auth_done global_id 75878 payload 824 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 -- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3813bb30 msgr2=0x7fdb3806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:17.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3813bb30 0x7fdb3806a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 -- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 msgr2=0x7fdb3806a2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 0x7fdb3806a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient: _finish_hunting 0 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 monclient: found mon.noname-c 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.826+0000 7fdb3e583700 1 -- 172.21.15.142:0/423581296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280094f0 con 0x7fdb380993f0 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3e583700 1 -- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 msgr2=0x7fdb3806e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3e583700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400cf60 tx=0x7fdb34001db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400a960 con 0x7fdb380993f0 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: dump: 2024-09-18T10:27:17.850 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 5 monclient: authenticate success, global_id 75878 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _renew_subs 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:17.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3806ba20 con 0x7fdb380993f0 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _renew_subs 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3806e9f0 con 0x7fdb380993f0 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3400aac0 con 0x7fdb380993f0 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400ae80 con 0x7fdb380993f0 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3cd80700 4 set_mon_vals no callback set 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 10 monclient: dump: 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:17.852 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb200050b0 con 0x7fdb380993f0 2024-09-18T10:27:17.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdb34007300 con 0x7fdb380993f0 2024-09-18T10:27:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 0x7fdb1c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdb340489f0 con 0x7fdb380993f0 2024-09-18T10:27:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3dd82700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 0x7fdb1c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.830+0000 7fdb3dd82700 10 monclient: get_auth_request con 0x7fdb1c035a70 auth_method 0 2024-09-18T10:27:17.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.834+0000 7fdb3dd82700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 0x7fdb1c037f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdb3809d6b0 tx=0x7fdb28017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:17.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.842+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdb34010ae0 con 0x7fdb380993f0 2024-09-18T10:27:17.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.842+0000 7fdb2e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:17.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:17.842+0000 7fdb2e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:18.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.066+0000 7fdb3f585700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:18.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.066+0000 7fdb3f585700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:18.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.066+0000 7fdb3f585700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:18.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.066+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdb200058e0 con 0x7fdb380993f0 2024-09-18T10:27:18.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb2e7fc700 1 -- 172.21.15.142:0/423581296 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fdb34017020 con 0x7fdb380993f0 2024-09-18T10:27:18.074 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb2e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:18.075 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb2e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:18.075 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:18.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 msgr2=0x7fdb1c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:18.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 0x7fdb1c037f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdb3809d6b0 tx=0x7fdb28017000 comp rx=0 tx=0).stop 2024-09-18T10:27:18.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 10 monclient: shutdown 2024-09-18T10:27:18.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 msgr2=0x7fdb3806e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdb3400cf60 tx=0x7fdb34001db0 comp rx=0 tx=0).stop 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 shutdown_connections 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdb1c035a70 0x7fdb1c037f20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb3813bb30 0x7fdb3806a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb3809a7f0 0x7fdb3806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 --2- 172.21.15.142:0/423581296 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb380993f0 0x7fdb3806e4b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 >> 172.21.15.142:0/423581296 conn(0x7fdb380a1fe0 msgr2=0x7fdb38139800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 shutdown_connections 2024-09-18T10:27:18.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:18.070+0000 7fdb3f585700 1 -- 172.21.15.142:0/423581296 wait complete. 2024-09-18T10:27:21.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: get_monmap_and_config 2024-09-18T10:27:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: build_initial_monmap 2024-09-18T10:27:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: monmap: 2024-09-18T10:27:21.206 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:21.201212+0000 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:21.201212+0000 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:21.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 Processor -- start 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 -- start start 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: init 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-a con 0x7f37f809aae0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37f80967a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-b con 0x7f37f809c4f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f8095880 con 0x7f37f80967a0 2024-09-18T10:27:21.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f809aeb0 con 0x7f37f809aae0 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f806f6d0 con 0x7f37f809c4f0 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fe5c2700 10 monclient(hunting): authenticate will time out at 1930.126726s 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fcdbf700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f37f80967a0 legacy=0x7f37f809bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fd5c0700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 msgr2=0x7f37f8096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fd5c0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:60318/0 (socket says 172.21.15.142:60318) 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 learned_addr learned my addr 172.21.15.142:0/138699044 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:21.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 10 monclient(hunting): get_auth_request con 0x7f37f809c4f0 auth_method 0 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:21.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.198+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fcdbf700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.142:6789/0 conn(0x7f37f80967a0 legacy=0x7f37f809bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 msgr2=0x7f37f8096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 10 monclient(hunting): get_auth_request con 0x7f37f809c4f0 auth_method 0 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:21.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:21.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:21.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:21.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:21.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.402+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fcdbf700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.142:6789/0 conn(0x7f37f80967a0 legacy=0x7f37f809bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 msgr2=0x7f37f8096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 10 monclient(hunting): get_auth_request con 0x7f37f809c4f0 auth_method 0 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:21.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:21.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:21.806+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fcdbf700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.142:6789/0 conn(0x7f37f80967a0 legacy=0x7f37f809bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 msgr2=0x7f37f8096260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 10 monclient(hunting): get_auth_request con 0x7f37f809c4f0 auth_method 0 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:22.615 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:22.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:22.609+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): tick 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 monclient(hunting): continuing hunt 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 msgr2=0x7f37f81375a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f809c4f0 0x7f37f81375a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 msgr2=0x7f37f8096260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.142:3300/0 conn(0x7f37f809aae0 0x7f37f8096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.142:6789/0 conn(0x7f37f80967a0 legacy=0x7f37f809bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient: _add_conns ranks=[4,1] 2024-09-18T10:27:24.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): picked mon.noname-c con 0x7f37dc0036f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37dc006250 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 -- 172.21.15.142:0/138699044 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f80955a0 con 0x7f37dc006250 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 1 -- 172.21.15.142:0/138699044 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f809aeb0 con 0x7f37dc0036f0 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37eeffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request con 0x7f37dc0036f0 auth_method 0 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:24.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:24.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:24.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:24.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.201+0000 7f37fcdbf700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.150:6789/0 conn(0x7f37dc006250 legacy=0x7f37dc008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request con 0x7f37dc0036f0 auth_method 0 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:24.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:24.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:24.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.405+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request con 0x7f37dc0036f0 auth_method 0 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:24.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.809+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:27:24.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799863787 0 0) 0x7f37f80955a0 con 0x7f37dc006250 2024-09-18T10:27:24.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:24.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): my global_id is 194290 2024-09-18T10:27:24.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e4003570 con 0x7f37dc006250 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 364188780 0 0) 0x7f37e4003570 con 0x7f37dc006250 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 msgr2=0x7f37dc005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 --2- 172.21.15.142:0/138699044 >> v2:172.21.15.150:3300/0 conn(0x7f37dc0036f0 0x7f37dc005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: _finish_hunting 0 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f8095880 con 0x7f37dc006250 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: _finish_auth 0 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f37f4004310 con 0x7f37dc006250 2024-09-18T10:27:24.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fe5c2700 5 monclient: authenticate success, global_id 194290 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fe5c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: dump: 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:24.949 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 >> v1:172.21.15.150:6789/0 conn(0x7f37dc006250 legacy=0x7f37dc008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 reap_dead start 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 0x7f37e4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): picked mon.b con 0x7f37e4003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4004810 0x7f37e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): picked mon.c con 0x7f37e4004810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37e400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): picked mon.a con 0x7f37e40052e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f809aeb0 con 0x7f37e40052e0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c250 con 0x7f37e4003a60 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e400c410 con 0x7f37e4004810 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37effff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fe5c2700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fddc1700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 msgr2=0x7f37e400ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fddc1700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37e400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fcdbf700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4004810 0x7f37e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 0x7f37e4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request con 0x7f37e4003a60 auth_method 0 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fd5c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fd5c0700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fcdbf700 10 monclient(hunting): get_auth_request con 0x7f37e4004810 auth_method 0 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fcdbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fcdbf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.937+0000 7f37fcdbf700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_done global_id 194290 payload 293 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4004810 msgr2=0x7f37e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:24.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4004810 0x7f37e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 msgr2=0x7f37e400ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37e400ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient: _finish_hunting 0 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 monclient: found mon.b 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e4003570 con 0x7f37e4003a60 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 10 monclient: _finish_auth 0 2024-09-18T10:27:24.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 msgr2=0x7f37e4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 0x7f37e4003e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009b60 tx=0x7f37e8003de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1703966a7d02538b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e80056d0 con 0x7f37e4003a60 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: dump: 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:24.958 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fe5c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e800f070 con 0x7f37e4003a60 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fe5c2700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37fe5c2700 4 set_mon_vals no callback set 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 1 -- 172.21.15.142:0/138699044 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e80153f0 con 0x7f37e4003a60 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.941+0000 7f37effff700 10 monclient: dump: 2024-09-18T10:27:24.959 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 10 monclient: shutdown 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 msgr2=0x7f37e4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:24.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 0x7f37e4003e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009b60 tx=0x7f37e8003de0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- 172.21.15.142:0/138699044 shutdown_connections 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37e400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4004810 0x7f37e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/138699044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4003a60 0x7f37e4003e50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- 172.21.15.142:0/138699044 >> 172.21.15.142:0/138699044 conn(0x7f37f80a1fe0 msgr2=0x7f37f8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- 172.21.15.142:0/138699044 shutdown_connections 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- 172.21.15.142:0/138699044 wait complete. 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 10 monclient: build_initial_monmap 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 10 monclient: monmap: 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:24.961 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:24.962 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 Processor -- start 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 1 -- start start 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.945+0000 7f37fe5c2700 10 monclient: init 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-c con 0x7f37e4003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4004810 0x7f37f8140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-b con 0x7f37e4004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37f80988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): picked mon.noname-a con 0x7f37e40052e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:24.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f812be70 con 0x7f37e40052e0 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f806f940 con 0x7f37e4004810 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f806e900 con 0x7f37e4003a60 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fe5c2700 10 monclient(hunting): authenticate will time out at 1933.876744s 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fddc1700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 msgr2=0x7f37f80988d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fddc1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37f80988d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:24.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37242/0 (socket says 172.21.15.142:37242) 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 -- 172.21.15.142:0/2735271128 learned_addr learned my addr 172.21.15.142:0/2735271128 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request con 0x7f37e4003a60 auth_method 0 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4004810 0x7f37f8140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): get_auth_request con 0x7f37e4004810 auth_method 0 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:24.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fcdbf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient(hunting): handle_auth_done global_id 75881 payload 824 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 -- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4004810 msgr2=0x7f37f8140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4004810 0x7f37f8140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 -- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 msgr2=0x7f37f80988d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37f80988d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient: _finish_hunting 0 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 monclient: found mon.noname-c 2024-09-18T10:27:24.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:24.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:24.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.949+0000 7f37fd5c0700 1 -- 172.21.15.142:0/2735271128 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f812bb90 con 0x7f37e4003a60 2024-09-18T10:27:24.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.953+0000 7f37fd5c0700 1 -- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 msgr2=0x7f37f812f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:24.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.953+0000 7f37fd5c0700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003a80 tx=0x7f37e801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8015470 con 0x7f37e4003a60 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: dump: 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:24.968 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: _finish_auth 0 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e80058e0 con 0x7f37e4003a60 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 5 monclient: authenticate success, global_id 75881 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _renew_subs 2024-09-18T10:27:24.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f809a190 con 0x7f37e4003a60 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8005ca0 con 0x7f37e4003a60 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 10 monclient: dump: 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.970 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ef7fe700 4 set_mon_vals no callback set 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _renew_subs 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:24.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f812ffe0 con 0x7f37e4003a60 2024-09-18T10:27:24.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f37e801e660 con 0x7f37e4003a60 2024-09-18T10:27:24.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:24.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:24.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:24.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f8072a80 con 0x7f37e4003a60 2024-09-18T10:27:24.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 0x7f37dc038610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:24.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f37e804a940 con 0x7f37e4003a60 2024-09-18T10:27:24.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fcdbf700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 0x7f37dc038610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:24.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.961+0000 7f37fcdbf700 10 monclient: get_auth_request con 0x7f37dc036160 auth_method 0 2024-09-18T10:27:24.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.965+0000 7f37fcdbf700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 0x7f37dc038610 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f37f812ac50 tx=0x7f37f4007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:24.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.973+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f37e8020440 con 0x7f37e4003a60 2024-09-18T10:27:24.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.973+0000 7f37ed7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:24.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:24.973+0000 7f37ed7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:25.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37fe5c2700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:25.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37fe5c2700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:25.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37fe5c2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f37f812acc0 con 0x7f37e4003a60 2024-09-18T10:27:25.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37ed7fa700 1 -- 172.21.15.142:0/2735271128 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f37e802f070 con 0x7f37e4003a60 2024-09-18T10:27:25.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37ed7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:25.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.189+0000 7f37ed7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:25.198 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":180,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 msgr2=0x7f37dc038610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 0x7f37dc038610 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f37f812ac50 tx=0x7f37f4007c20 comp rx=0 tx=0).stop 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 10 monclient: shutdown 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 msgr2=0x7f37f812f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003a80 tx=0x7f37e801b000 comp rx=0 tx=0).stop 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 shutdown_connections 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37dc036160 0x7f37dc038610 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e40052e0 0x7f37f80988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f37e4004810 0x7f37f8140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 --2- 172.21.15.142:0/2735271128 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37e4003a60 0x7f37f812f880 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 >> 172.21.15.142:0/2735271128 conn(0x7f37f80a1fe0 msgr2=0x7f37f8132970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:25.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 shutdown_connections 2024-09-18T10:27:25.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.193+0000 7f37fe5c2700 1 -- 172.21.15.142:0/2735271128 wait complete. 2024-09-18T10:27:25.211 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:27:25.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:27:25.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:27:25.317 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.309+0000 7ff34b59e700 10 monclient: get_monmap_and_config 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.309+0000 7ff34b59e700 10 monclient: build_initial_monmap 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.309+0000 7ff34b59e700 10 monclient: monmap: 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:25.315235+0000 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:25.315235+0000 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:25.318 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:25.319 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.320 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 Processor -- start 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 -- start start 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient: init 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff34c09c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff34c134880 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff34c138380 0x7ff34c13a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff34c138380 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0a6620 con 0x7ff34c134880 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0a1c70 con 0x7ff34c09c2c0 2024-09-18T10:27:25.321 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c09ab90 con 0x7ff34c138380 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34b59e700 10 monclient(hunting): authenticate will time out at 1934.241049s 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34ad9d700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7ff34c138380 msgr2=0x7ff34c13a760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34ad9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff34c138380 0x7ff34c13a760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff349d9b700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7ff34c134880 legacy=0x7ff34c136c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34a59c700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff34c09c2c0 0x7ff34c0999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:38050/0 (socket says 172.21.15.142:38050) 2024-09-18T10:27:25.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 learned_addr learned my addr 172.21.15.142:0/1666335237 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:25.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.313+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 >> v1:172.21.15.150:6789/0 conn(0x7ff34c09c2c0 legacy=0x7ff34c0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485116892 0 0) 0x7ff34c0a1c70 con 0x7ff34c09c2c0 2024-09-18T10:27:25.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): my global_id is 194296 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff334003570 con 0x7ff34c09c2c0 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2289279269 0 0) 0x7ff334003570 con 0x7ff34c09c2c0 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 >> v2:172.21.15.142:3300/0 conn(0x7ff34c138380 msgr2=0x7ff34c13a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 --2- 172.21.15.142:0/1666335237 >> v2:172.21.15.142:3300/0 conn(0x7ff34c138380 0x7ff34c13a760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 >> v1:172.21.15.142:6789/0 conn(0x7ff34c134880 legacy=0x7ff34c136c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: _finish_hunting 0 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c09c690 con 0x7ff34c09c2c0 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: _finish_auth 0 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:25.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34b59e700 5 monclient: authenticate success, global_id 194296 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7ff33c0017a0 con 0x7ff34c09c2c0 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: dump: 2024-09-18T10:27:25.339 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.341 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:27:25.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 >> v1:172.21.15.150:6789/0 conn(0x7ff34c09c2c0 legacy=0x7ff34c0999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff334003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): picked mon.c con 0x7ff334003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004810 0x7ff334004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): picked mon.a con 0x7ff334004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340053c0 0x7ff33400baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): picked mon.b con 0x7ff3340053c0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400c350 con 0x7ff334004810 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400c4c0 con 0x7ff3340053c0 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400c680 con 0x7ff334003a60 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34b59e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff349d9b700 1 -- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004810 msgr2=0x7ff334004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:25.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff334003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff349d9b700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004810 0x7ff334004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): get_auth_request con 0x7ff334003a60 auth_method 0 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340053c0 0x7ff33400baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): get_auth_request con 0x7ff3340053c0 auth_method 0 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient(hunting): handle_auth_done global_id 194296 payload 293 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340053c0 msgr2=0x7ff33400baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340053c0 0x7ff33400baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004810 msgr2=0x7ff334004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004810 0x7ff334004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 monclient: found mon.c 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:25.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff334003570 con 0x7ff334003a60 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34ad9d700 1 -- 172.21.15.142:0/1666335237 reap_dead start 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 10 monclient: _finish_auth 0 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 -- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 msgr2=0x7ff334003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34a59c700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff334003e50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0014d0 tx=0x7ff33c007b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab3fb6851ad38e4b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.317+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c00ce20 con 0x7ff334003a60 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: dump: 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.346 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c005920 con 0x7ff334003a60 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 4 set_mon_vals no callback set 2024-09-18T10:27:25.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 1 -- 172.21.15.142:0/1666335237 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c005ce0 con 0x7ff334003a60 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34959a700 10 monclient: dump: 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: shutdown 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 msgr2=0x7ff334003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff334003e50 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0014d0 tx=0x7ff33c007b20 comp rx=0 tx=0).stop 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- 172.21.15.142:0/1666335237 shutdown_connections 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 --2- 172.21.15.142:0/1666335237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff334003e50 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- 172.21.15.142:0/1666335237 >> 172.21.15.142:0/1666335237 conn(0x7ff34c0a5720 msgr2=0x7ff34c0a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- 172.21.15.142:0/1666335237 shutdown_connections 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- 172.21.15.142:0/1666335237 wait complete. 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: build_initial_monmap 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: monmap: 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:25.349 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 Processor -- start 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- start start 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: init 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:27:25.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-c con 0x7ff334003a60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c141d20 0x7ff34c1420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-a con 0x7ff34c141d20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c06d5e0 0x7ff34c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): picked mon.noname-b con 0x7ff34c06d5e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c12bec0 con 0x7ff34c141d20 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c06f070 con 0x7ff34c06d5e0 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c12cad0 con 0x7ff334003a60 2024-09-18T10:27:25.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34b59e700 10 monclient(hunting): authenticate will time out at 1934.251296s 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff349d9b700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c141d20 msgr2=0x7ff34c1420f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff349d9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c141d20 0x7ff34c1420f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37264/0 (socket says 172.21.15.142:37264) 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 -- 172.21.15.142:0/176670970 learned_addr learned my addr 172.21.15.142:0/176670970 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): get_auth_request con 0x7ff334003a60 auth_method 0 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c06d5e0 0x7ff34c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 10 monclient(hunting): get_auth_request con 0x7ff34c06d5e0 auth_method 0 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient(hunting): handle_auth_done global_id 75884 payload 824 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c06d5e0 msgr2=0x7ff34c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c06d5e0 0x7ff34c06da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c141d20 msgr2=0x7ff34c1420f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff34c141d20 0x7ff34c1420f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient: _finish_hunting 0 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 monclient: found mon.noname-c 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34a59c700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c06f560 con 0x7ff334003a60 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.321+0000 7ff34ad9d700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff34c06d5e0 0x7ff34c06da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34a59c700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 msgr2=0x7ff34c1417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34a59c700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00cb80 tx=0x7ff33c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c001190 con 0x7ff334003a60 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: dump: 2024-09-18T10:27:25.354 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: _finish_auth 0 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 5 monclient: authenticate success, global_id 75884 2024-09-18T10:27:25.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c0074f0 con 0x7ff334003a60 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 10 monclient: _renew_subs 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c142630 con 0x7ff334003a60 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c01e430 con 0x7ff334003a60 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff348d99700 4 set_mon_vals no callback set 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff33a7fc700 10 monclient: dump: 2024-09-18T10:27:25.356 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 10 monclient: _renew_subs 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c12b800 con 0x7ff334003a60 2024-09-18T10:27:25.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.325+0000 7ff34b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff34b59e700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 0x7ff34c06e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c064750 con 0x7ff34c06df90 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 0x7ff34c06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient: get_auth_request con 0x7ff34c06df90 auth_method 0 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff33c01e7a0 con 0x7ff334003a60 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff32c035a70 0x7ff32c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff33c049f80 con 0x7ff334003a60 2024-09-18T10:27:25.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 10 monclient(hunting): handle_auth_done global_id 75887 payload 824 2024-09-18T10:27:25.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff34c12bc30 con 0x7ff34c06df90 2024-09-18T10:27:25.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff34ad9d700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff32c035a70 0x7ff32c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff34ad9d700 10 monclient: get_auth_request con 0x7ff32c035a70 auth_method 0 2024-09-18T10:27:25.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 msgr2=0x7ff34c06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff349d9b700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 0x7ff34c06e360 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0a1b00 tx=0x7ff340009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340006150 con 0x7ff34c06df90 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.329+0000 7ff33a7fc700 10 monclient: dump: 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.362 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff3400062b0 con 0x7ff34c06df90 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 msgr2=0x7ff34c06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 0x7ff34c06e360 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0a1b00 tx=0x7ff340009220 comp rx=0 tx=0).stop 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 msgr2=0x7ff34c06e360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff33a7fc700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06df90 0x7ff34c06e360 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.333+0000 7ff34ad9d700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff32c035a70 0x7ff32c037f20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff34c09a840 tx=0x7ff3440095d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff34b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff34b59e700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 0x7ff34c06ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff34b59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c12b9f0 con 0x7ff34c06aa20 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff349d9b700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 0x7ff34c06ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff349d9b700 10 monclient: get_auth_request con 0x7ff34c06aa20 auth_method 0 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff349d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff349d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.341+0000 7ff349d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 10 monclient(hunting): handle_auth_done global_id 75890 payload 824 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 1 -- 172.21.15.142:0/176670970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff34c072a80 con 0x7ff34c06aa20 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 msgr2=0x7ff34c06ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff349d9b700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 0x7ff34c06ae20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff340013000 tx=0x7ff340014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340020920 con 0x7ff34c06aa20 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 10 monclient: dump: 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.365 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7ff340003d80 con 0x7ff34c06aa20 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 msgr2=0x7ff34c06ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 0x7ff34c06ae20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff340013000 tx=0x7ff340014000 comp rx=0 tx=0).stop 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 msgr2=0x7ff34c06ae20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.345+0000 7ff33a7fc700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34c06aa20 0x7ff34c06ae20 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.366 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 180, 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.367 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.368 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.369 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:25.370 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.371 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:25.372 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.373 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.374 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.375 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.377 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.378 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:25.379 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:25.381 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:25.382 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.383 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:25.384 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff32c035a70 msgr2=0x7ff32c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff32c035a70 0x7ff32c037f20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff34c09a840 tx=0x7ff3440095d0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 10 monclient: shutdown 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 msgr2=0x7ff34c1417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34ad9d700 1 -- 172.21.15.142:0/176670970 reap_dead start 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00cb80 tx=0x7ff33c015000 comp rx=0 tx=0).stop 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 shutdown_connections 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 --2- 172.21.15.142:0/176670970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff334003a60 0x7ff34c1417e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 >> 172.21.15.142:0/176670970 conn(0x7ff34c0a5720 msgr2=0x7ff34c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 shutdown_connections 2024-09-18T10:27:25.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.349+0000 7ff34b59e700 1 -- 172.21.15.142:0/176670970 wait complete. 2024-09-18T10:27:25.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:27:25.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:27:25.387 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:27:25.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 10 monclient: get_monmap_and_config 2024-09-18T10:27:25.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 10 monclient: build_initial_monmap 2024-09-18T10:27:25.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 10 monclient: monmap: 2024-09-18T10:27:25.501 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:25.501 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.501 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:25.497579+0000 2024-09-18T10:27:25.501 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:25.497579+0000 2024-09-18T10:27:25.501 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:25.502 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:25.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 1 Processor -- start 2024-09-18T10:27:25.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 1 -- start start 2024-09-18T10:27:25.516 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.493+0000 7feb1f65f700 10 monclient: init 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-b con 0x7feb1806f410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb18096a50 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb18097530 0x7feb18136630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-c con 0x7feb18097530 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb180983f0 con 0x7feb18096a50 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb18068cf0 con 0x7feb18097530 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1806f7e0 con 0x7feb1806f410 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1f65f700 10 monclient(hunting): authenticate will time out at 1934.424790s 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37284/0 (socket says 172.21.15.142:37284) 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 -- 172.21.15.142:0/1583401273 learned_addr learned my addr 172.21.15.142:0/1583401273 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): get_auth_request con 0x7feb1806f410 auth_method 0 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1dbfc700 1 --2- 172.21.15.142:0/1583401273 >> v2:172.21.15.150:3300/0 conn(0x7feb18097530 0x7feb18136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1dbfc700 10 monclient(hunting): get_auth_request con 0x7feb18097530 auth_method 0 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1dbfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1dbfc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1dbfc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_done global_id 75893 payload 274 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 -- 172.21.15.142:0/1583401273 >> v2:172.21.15.150:3300/0 conn(0x7feb18097530 msgr2=0x7feb18136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/1583401273 >> v2:172.21.15.150:3300/0 conn(0x7feb18097530 0x7feb18136630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 -- 172.21.15.142:0/1583401273 >> v1:172.21.15.150:6789/0 conn(0x7feb18096a50 legacy=0x7feb18096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient: _finish_hunting 0 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 monclient: found mon.noname-b 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 -- 172.21.15.142:0/1583401273 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1809bf00 con 0x7feb1806f410 2024-09-18T10:27:25.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 -- 172.21.15.142:0/1583401273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 msgr2=0x7feb18099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.497+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/1583401273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7feb1809c0f0 tx=0x7feb14009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937198c5c005de05 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 1 -- 172.21.15.142:0/1583401273 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1400ba40 con 0x7feb1806f410 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: dump: 2024-09-18T10:27:25.521 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.522 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: _finish_auth 0 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 1 -- 172.21.15.142:0/1583401273 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1400bba0 con 0x7feb1806f410 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 5 monclient: authenticate success, global_id 75893 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:25.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 4 set_mon_vals no callback set 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 1 -- 172.21.15.142:0/1583401273 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14011420 con 0x7feb1806f410 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb0ffff700 10 monclient: dump: 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.524 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: shutdown 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- 172.21.15.142:0/1583401273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 msgr2=0x7feb18099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- 172.21.15.142:0/1583401273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7feb1809c0f0 tx=0x7feb14009250 comp rx=0 tx=0).stop 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- 172.21.15.142:0/1583401273 shutdown_connections 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- 172.21.15.142:0/1583401273 >> v2:172.21.15.150:3300/0 conn(0x7feb18097530 0x7feb18136630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- 172.21.15.142:0/1583401273 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb1806f410 0x7feb18099300 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- 172.21.15.142:0/1583401273 >> 172.21.15.142:0/1583401273 conn(0x7feb18071740 msgr2=0x7feb180a1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- 172.21.15.142:0/1583401273 shutdown_connections 2024-09-18T10:27:25.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- 172.21.15.142:0/1583401273 wait complete. 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: build_initial_monmap 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: monmap: 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:25.526 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 Processor -- start 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- start start 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: init 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb1806f410 0x7feb1809c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-a con 0x7feb1806f410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18096a50 0x7feb1813fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-b con 0x7feb18096a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): picked mon.noname-c con 0x7feb18097530 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1809c190 con 0x7feb1806f410 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18099870 con 0x7feb18096a50 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb18120c40 con 0x7feb18097530 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1d3fb700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb1806f410 msgr2=0x7feb1809c8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1f65f700 10 monclient(hunting): authenticate will time out at 1934.430975s 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1d3fb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb1806f410 0x7feb1809c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:25.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37294/0 (socket says 172.21.15.142:37294) 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 learned_addr learned my addr 172.21.15.142:0/3779471392 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): get_auth_request con 0x7feb18097530 auth_method 0 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18096a50 0x7feb1813fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 10 monclient(hunting): get_auth_request con 0x7feb18096a50 auth_method 0 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient(hunting): handle_auth_done global_id 75896 payload 824 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18096a50 msgr2=0x7feb1813fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18096a50 0x7feb1813fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb1806f410 msgr2=0x7feb1809c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb1806f410 0x7feb1809c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient: _finish_hunting 0 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 monclient: found mon.noname-c 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140094f0 con 0x7feb18097530 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.501+0000 7feb1cbfa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb18096a50 0x7feb1813fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:25.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.505+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 msgr2=0x7feb1813bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.505+0000 7feb1dbfc700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb10009fa0 tx=0x7feb10009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.505+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:27:25.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10014420 con 0x7feb18097530 2024-09-18T10:27:25.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: dump: 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.533 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10014580 con 0x7feb18097530 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 5 monclient: authenticate success, global_id 75896 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 monclient: _renew_subs 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb18121180 con 0x7feb18097530 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10014940 con 0x7feb18097530 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 10 monclient: dump: 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.536 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.537 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.537 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0f7fe700 4 set_mon_vals no callback set 2024-09-18T10:27:25.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 monclient: _renew_subs 2024-09-18T10:27:25.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1813c060 con 0x7feb18097530 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 0x7feb1813c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb180647a0 con 0x7feb1811fca0 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7feb10014cf0 con 0x7feb18097530 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 0x7feb1813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient: get_auth_request con 0x7feb1811fca0 auth_method 0 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feafc035ac0 0x7feafc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7feb10049720 con 0x7feb18097530 2024-09-18T10:27:25.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_done global_id 194308 payload 824 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1cbfa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feafc035ac0 0x7feafc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1d3fb700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb1813c280 con 0x7feb1811fca0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.509+0000 7feb1cbfa700 10 monclient: get_auth_request con 0x7feafc035ac0 auth_method 0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb1d3fb700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 msgr2=0x7feb1813c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 0x7feb1813c470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb14013000 tx=0x7feb14016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb1cbfa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feafc035ac0 0x7feafc037f70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb08007fd0 tx=0x7feb08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14011e10 con 0x7feb1811fca0 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb0d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.513+0000 7feb0d7fa700 10 monclient: dump: 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.542 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7feb1401b660 con 0x7feb1811fca0 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 msgr2=0x7feb1813c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 0x7feb1813c470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb14013000 tx=0x7feb14016000 comp rx=0 tx=0).stop 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 msgr2=0x7feb1813c470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.517+0000 7feb0d7fa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1811fca0 0x7feb1813c470 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb14013000 tx=0x7feb14016000 comp rx=0 tx=0).stop 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1f65f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1f65f700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 0x7feb1813cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1f65f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1813d2f0 con 0x7feb1813c9b0 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 0x7feb1813cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient: get_auth_request con 0x7feb1813c9b0 auth_method 0 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:25.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 10 monclient(hunting): handle_auth_done global_id 194311 payload 824 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.525+0000 7feb1d3fb700 1 -- 172.21.15.142:0/3779471392 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb180696a0 con 0x7feb1813c9b0 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1d3fb700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 msgr2=0x7feb1813cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1d3fb700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 0x7feb1813cdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb14009ef0 tx=0x7feb1401e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14031420 con 0x7feb1813c9b0 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 10 monclient: dump: 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:25.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7feb1401b450 con 0x7feb1813c9b0 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 msgr2=0x7feb1813cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 0x7feb1813cdb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb14009ef0 tx=0x7feb1401e040 comp rx=0 tx=0).stop 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 msgr2=0x7feb1813cdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb0d7fa700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb1813c9b0 0x7feb1813cdb0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feb14009ef0 tx=0x7feb1401e040 comp rx=0 tx=0).stop 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 180, 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:25.547 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.548 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.549 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:25.550 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:25.551 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.552 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.553 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.554 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:25.555 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.556 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.557 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:25.558 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:25.559 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:25.560 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feafc035ac0 msgr2=0x7feafc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feafc035ac0 0x7feafc037f70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb08007fd0 tx=0x7feb08009040 comp rx=0 tx=0).stop 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 10 monclient: shutdown 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 msgr2=0x7feb1813bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb10009fa0 tx=0x7feb10009900 comp rx=0 tx=0).stop 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1dbfc700 1 -- 172.21.15.142:0/3779471392 reap_dead start 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 shutdown_connections 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 --2- 172.21.15.142:0/3779471392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb18097530 0x7feb1813bb20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:25.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.529+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 >> 172.21.15.142:0/3779471392 conn(0x7feb18071740 msgr2=0x7feb180a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:25.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.533+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 shutdown_connections 2024-09-18T10:27:25.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:25.533+0000 7feb1f65f700 1 -- 172.21.15.142:0/3779471392 wait complete. 2024-09-18T10:27:25.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:27:25.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:27:45.567 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:27:45.567 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:27:45.567 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:27:45.567 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:27:45.570 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:27:45.570 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:27:45.571 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 10 monclient: get_monmap_and_config 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 10 monclient: build_initial_monmap 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 10 monclient: monmap: 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:45.670859+0000 2024-09-18T10:27:45.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:45.670859+0000 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 1 Processor -- start 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 1 -- start start 2024-09-18T10:27:45.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.665+0000 7fd56c82c700 10 monclient: init 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd56809d870 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 0x7fd568134130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-c con 0x7fd56812f510 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:45.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 0x7fd568136680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-b con 0x7fd56812f930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56809c440 con 0x7fd56809d870 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd56809a6c0 con 0x7fd56812f510 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56809dc40 con 0x7fd56812f930 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient(hunting): authenticate will time out at 1954.596877s 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 0x7fd568136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56659c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd56809d870 0x7fd56812ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:35014/0 (socket says 172.21.15.142:35014) 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56659c700 1 -- 172.21.15.142:0/3972410800 learned_addr learned my addr 172.21.15.142:0/3972410800 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:45.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): get_auth_request con 0x7fd56812f930 auth_method 0 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 1 --2- 172.21.15.142:0/3972410800 >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 0x7fd568134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 10 monclient(hunting): get_auth_request con 0x7fd56812f510 auth_method 0 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient(hunting): handle_auth_done global_id 75899 payload 274 2024-09-18T10:27:45.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 -- 172.21.15.142:0/3972410800 >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 msgr2=0x7fd568134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 --2- 172.21.15.142:0/3972410800 >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 0x7fd568134130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 -- 172.21.15.142:0/3972410800 >> v1:172.21.15.142:6789/0 conn(0x7fd56809d870 legacy=0x7fd56812ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 monclient: found mon.noname-b 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 -- 172.21.15.142:0/3972410800 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56809a9a0 con 0x7fd56812f930 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd565d9b700 1 --2- 172.21.15.142:0/3972410800 >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 0x7fd568134130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 -- 172.21.15.142:0/3972410800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 msgr2=0x7fd568136680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd566d9d700 1 --2- 172.21.15.142:0/3972410800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 0x7fd568136680 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d890a6829f4b47 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 1 -- 172.21.15.142:0/3972410800 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c009e80 con 0x7fd56812f930 2024-09-18T10:27:45.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: dump: 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: _finish_auth 0 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 5 monclient: authenticate success, global_id 75899 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 1 -- 172.21.15.142:0/3972410800 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00ba70 con 0x7fd56812f930 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56c82c700 4 set_mon_vals no callback set 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 1 -- 172.21.15.142:0/3972410800 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00bdf0 con 0x7fd56812f930 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.669+0000 7fd56559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56559a700 10 monclient: dump: 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:45.690 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: shutdown 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- 172.21.15.142:0/3972410800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 msgr2=0x7fd568136680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- 172.21.15.142:0/3972410800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 0x7fd568136680 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- 172.21.15.142:0/3972410800 shutdown_connections 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- 172.21.15.142:0/3972410800 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56812f930 0x7fd568136680 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- 172.21.15.142:0/3972410800 >> v2:172.21.15.150:3300/0 conn(0x7fd56812f510 0x7fd568134130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- 172.21.15.142:0/3972410800 >> 172.21.15.142:0/3972410800 conn(0x7fd568071820 msgr2=0x7fd5680a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- 172.21.15.142:0/3972410800 shutdown_connections 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- 172.21.15.142:0/3972410800 wait complete. 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: build_initial_monmap 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: monmap: 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:45.692 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 Processor -- start 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- start start 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: init 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-c con 0x7fd56809d870 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:45.693 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56812f510 0x7fd568094810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-b con 0x7fd56812f510 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): picked mon.noname-a con 0x7fd56812f930 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56812e690 con 0x7fd56812f930 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568099e30 con 0x7fd56812f510 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56806f2c0 con 0x7fd56809d870 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56c82c700 10 monclient(hunting): authenticate will time out at 1954.602554s 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.694 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39386/0 (socket says 172.21.15.142:39386) 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35036/0 (socket says 172.21.15.142:35036) 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 -- 172.21.15.142:0/2439095712 learned_addr learned my addr 172.21.15.142:0/2439095712 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): get_auth_request con 0x7fd56809d870 auth_method 0 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 10 monclient(hunting): get_auth_request con 0x7fd56812f930 auth_method 0 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:45.695 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 msgr2=0x7fd56806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 msgr2=0x7fd56806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd565d9b700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56812f510 0x7fd568094810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd566d9d700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd565d9b700 10 monclient(hunting): get_auth_request con 0x7fd56812f510 auth_method 0 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd565d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd565d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd565d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient(hunting): handle_auth_done global_id 75902 payload 824 2024-09-18T10:27:45.696 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56812f510 msgr2=0x7fd568094810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56812f510 0x7fd568094810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 msgr2=0x7fd56806d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient: _finish_hunting 0 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 monclient: found mon.noname-c 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.673+0000 7fd56659c700 1 -- 172.21.15.142:0/2439095712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c009420 con 0x7fd56809d870 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56659c700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 msgr2=0x7fd5680942d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56659c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd56000bc30 tx=0x7fd560009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56000cbc0 con 0x7fd56809d870 2024-09-18T10:27:45.697 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: dump: 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:45.698 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: _finish_auth 0 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 5 monclient: authenticate success, global_id 75902 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56000cd40 con 0x7fd56809d870 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _renew_subs 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd568128180 con 0x7fd56809d870 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560014690 con 0x7fd56809d870 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd557fff700 4 set_mon_vals no callback set 2024-09-18T10:27:45.699 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 10 monclient: dump: 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _renew_subs 2024-09-18T10:27:45.700 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56806ebd0 con 0x7fd56809d870 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd560014ac0 con 0x7fd56809d870 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 0x7fd548037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd560048e60 con 0x7fd56809d870 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd565d9b700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 0x7fd548037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd565d9b700 10 monclient: get_auth_request con 0x7fd548035ac0 auth_method 0 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.701 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.677+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5680696a0 con 0x7fd56809d870 2024-09-18T10:27:45.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.681+0000 7fd565d9b700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 0x7fd548037f70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd558006f00 tx=0x7fd55800b500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:45.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.685+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd560019070 con 0x7fd56809d870 2024-09-18T10:27:45.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.685+0000 7fd5567fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:45.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.685+0000 7fd5567fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:45.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd568067950 con 0x7fd56809d870 2024-09-18T10:27:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd5567fc700 1 -- 172.21.15.142:0/2439095712 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fd5600073c0 con 0x7fd56809d870 2024-09-18T10:27:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd5567fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:45.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd5567fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:45.919 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":180,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:45.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 msgr2=0x7fd548037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 0x7fd548037f70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd558006f00 tx=0x7fd55800b500 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 10 monclient: shutdown 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 msgr2=0x7fd5680942d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd56000bc30 tx=0x7fd560009800 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 shutdown_connections 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd548035ac0 0x7fd548037f70 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd56812f930 0x7fd56806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56812f510 0x7fd568094810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 --2- 172.21.15.142:0/2439095712 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd56809d870 0x7fd5680942d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 >> 172.21.15.142:0/2439095712 conn(0x7fd568071820 msgr2=0x7fd5680a0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 shutdown_connections 2024-09-18T10:27:45.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:45.913+0000 7fd56c82c700 1 -- 172.21.15.142:0/2439095712 wait complete. 2024-09-18T10:27:48.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.037+0000 7f94649ed700 10 monclient: get_monmap_and_config 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.037+0000 7f94649ed700 10 monclient: build_initial_monmap 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.037+0000 7f94649ed700 10 monclient: monmap: 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:49.043675+0000 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:49.043675+0000 2024-09-18T10:27:49.046 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:49.047 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 Processor -- start 2024-09-18T10:27:49.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 -- start start 2024-09-18T10:27:49.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient: init 2024-09-18T10:27:49.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-b con 0x7f9460070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f946009a380 0x7f9460071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-c con 0x7f946009a380 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f946009df70 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9460071d40 con 0x7f946009df70 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946006fc50 con 0x7f946009a380 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946009e720 con 0x7f9460070ef0 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:49.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f94649ed700 10 monclient(hunting): authenticate will time out at 1957.969657s 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39444/0 (socket says 172.21.15.142:39444) 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 -- 172.21.15.142:0/1678905533 learned_addr learned my addr 172.21.15.142:0/1678905533 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): get_auth_request con 0x7f9460070ef0 auth_method 0 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 1 --2- 172.21.15.142:0/1678905533 >> v2:172.21.15.150:3300/0 conn(0x7f946009a380 0x7f9460071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): get_auth_request con 0x7f946009a380 auth_method 0 2024-09-18T10:27:49.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient(hunting): handle_auth_done global_id 75905 payload 274 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 -- 172.21.15.142:0/1678905533 >> v2:172.21.15.150:3300/0 conn(0x7f946009a380 msgr2=0x7f9460071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 --2- 172.21.15.142:0/1678905533 >> v2:172.21.15.150:3300/0 conn(0x7f946009a380 0x7f9460071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 -- 172.21.15.142:0/1678905533 >> v1:172.21.15.142:6789/0 conn(0x7f946009df70 legacy=0x7f94601386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 monclient: found mon.noname-b 2024-09-18T10:27:49.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:49.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.041+0000 7f945ed9d700 1 -- 172.21.15.142:0/1678905533 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946009e9e0 con 0x7f9460070ef0 2024-09-18T10:27:49.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945ed9d700 1 -- 172.21.15.142:0/1678905533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 msgr2=0x7f94600712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:49.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945ed9d700 1 --2- 172.21.15.142:0/1678905533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f946006acb0 tx=0x7f9448009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5653ebabe609a50 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 1 -- 172.21.15.142:0/1678905533 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448009e70 con 0x7f9460070ef0 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: dump: 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:49.058 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 1 -- 172.21.15.142:0/1678905533 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944800ba70 con 0x7f9460070ef0 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 5 monclient: authenticate success, global_id 75905 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 1 -- 172.21.15.142:0/1678905533 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944800be10 con 0x7f9460070ef0 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 4 set_mon_vals no callback set 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f945dd9b700 10 monclient: dump: 2024-09-18T10:27:49.059 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 10 monclient: shutdown 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:49.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 -- 172.21.15.142:0/1678905533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 msgr2=0x7f94600712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 --2- 172.21.15.142:0/1678905533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f946006acb0 tx=0x7f9448009250 comp rx=0 tx=0).stop 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 -- 172.21.15.142:0/1678905533 shutdown_connections 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 --2- 172.21.15.142:0/1678905533 >> v2:172.21.15.150:3300/0 conn(0x7f946009a380 0x7f9460071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 --2- 172.21.15.142:0/1678905533 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f94600712c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 -- 172.21.15.142:0/1678905533 >> 172.21.15.142:0/1678905533 conn(0x7f946006a2e0 msgr2=0x7f94600a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 -- 172.21.15.142:0/1678905533 shutdown_connections 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 1 -- 172.21.15.142:0/1678905533 wait complete. 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 10 monclient: build_initial_monmap 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.045+0000 7f94649ed700 10 monclient: monmap: 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:49.061 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 Processor -- start 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 -- start start 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient: init 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-c con 0x7f9460070ef0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:49.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946009a380 0x7f9460141e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-b con 0x7f946009a380 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): picked mon.noname-a con 0x7f946009df70 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946009e470 con 0x7f946009df70 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94600a0a60 con 0x7f946009a380 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9460090c70 con 0x7f9460070ef0 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f94649ed700 10 monclient(hunting): authenticate will time out at 1957.976835s 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:39446/0 (socket says 172.21.15.142:39446) 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 -- 172.21.15.142:0/1998605105 learned_addr learned my addr 172.21.15.142:0/1998605105 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): get_auth_request con 0x7f9460070ef0 auth_method 0 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 10 monclient(hunting): get_auth_request con 0x7f946009df70 auth_method 0 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:49.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946009a380 0x7f9460141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 msgr2=0x7f946008d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 msgr2=0x7f946008d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): get_auth_request con 0x7f946009a380 auth_method 0 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945f59e700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:49.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient(hunting): handle_auth_done global_id 75908 payload 824 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946009a380 msgr2=0x7f9460141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946009a380 0x7f9460141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 msgr2=0x7f946008d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 -- 172.21.15.142:0/1998605105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94480094f0 con 0x7f9460070ef0 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 msgr2=0x7f9460141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:49.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.049+0000 7f945ed9d700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f946009d690 tx=0x7f9448017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448000d10 con 0x7f9460070ef0 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: dump: 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:49.069 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: _finish_auth 0 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 5 monclient: authenticate success, global_id 75908 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9448011e20 con 0x7f9460070ef0 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _renew_subs 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946008eef0 con 0x7f9460070ef0 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944801a8a0 con 0x7f9460070ef0 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:49.070 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 10 monclient: dump: 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f945d59a700 4 set_mon_vals no callback set 2024-09-18T10:27:49.071 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _renew_subs 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9460090a40 con 0x7f9460070ef0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f944801ac50 con 0x7f9460070ef0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9460072a80 con 0x7f9460070ef0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 0x7f9440037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f944804a620 con 0x7f9460070ef0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f945e59c700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 0x7f9440037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.053+0000 7f945e59c700 10 monclient: get_auth_request con 0x7f9440035b10 auth_method 0 2024-09-18T10:27:49.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.057+0000 7f945e59c700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 0x7f9440037fc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f945000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:49.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.065+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f944802c3a0 con 0x7f9460070ef0 2024-09-18T10:27:49.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.065+0000 7f9456ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:49.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.065+0000 7f9456ffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:49.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.281+0000 7f94649ed700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.281+0000 7f94649ed700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.281+0000 7f94649ed700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.281+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f946008eef0 con 0x7f9460070ef0 2024-09-18T10:27:49.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f9456ffd700 1 -- 172.21.15.142:0/1998605105 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f946008eef0 con 0x7f9460070ef0 2024-09-18T10:27:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f9456ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:49.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f9456ffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:49.291 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":180,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 msgr2=0x7f9440037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 0x7f9440037fc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f945000b040 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 10 monclient: shutdown 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 msgr2=0x7f9460141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f946009d690 tx=0x7f9448017000 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 shutdown_connections 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9440035b10 0x7f9440037fc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f946009df70 0x7f946008d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f946009a380 0x7f9460141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 --2- 172.21.15.142:0/1998605105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9460070ef0 0x7f9460141900 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:49.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 >> 172.21.15.142:0/1998605105 conn(0x7f946006a2e0 msgr2=0x7f946009a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 shutdown_connections 2024-09-18T10:27:49.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:49.285+0000 7f94649ed700 1 -- 172.21.15.142:0/1998605105 wait complete. 2024-09-18T10:27:52.308 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:52.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.413+0000 7f2411c81700 10 monclient: get_monmap_and_config 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.413+0000 7f2411c81700 10 monclient: build_initial_monmap 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.413+0000 7f2411c81700 10 monclient: monmap: 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:52.419443+0000 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:52.419443+0000 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:52.422 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:52.423 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:52.423 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 Processor -- start 2024-09-18T10:27:52.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 -- start start 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient: init 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-a con 0x7f240c09aae0 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-b con 0x7f240c0967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f240c09c320 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240c095880 con 0x7f240c09c320 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240c09aeb0 con 0x7f240c09aae0 2024-09-18T10:27:52.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c06f6d0 con 0x7f240c0967a0 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2411c81700 10 monclient(hunting): authenticate will time out at 1961.345350s 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:43994/0 (socket says 172.21.15.142:43994) 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 -- 172.21.15.142:0/3659823767 learned_addr learned my addr 172.21.15.142:0/3659823767 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): get_auth_request con 0x7f240c0967a0 auth_method 0 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:52.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 10 monclient(hunting): get_auth_request con 0x7f240c09aae0 auth_method 0 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 1 -- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 msgr2=0x7f240c096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 1 -- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 msgr2=0x7f240c096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 1 --2- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f2410c7f700 1 --2- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient(hunting): handle_auth_done global_id 75911 payload 274 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 -- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 msgr2=0x7f240c096260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:52.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 --2- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 -- 172.21.15.142:0/3659823767 >> v1:172.21.15.150:6789/0 conn(0x7f240c09c320 legacy=0x7f240c136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 monclient: found mon.noname-b 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.417+0000 7f240bfff700 1 -- 172.21.15.142:0/3659823767 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c0955a0 con 0x7f240c0967a0 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240bfff700 1 -- 172.21.15.142:0/3659823767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 msgr2=0x7f240c09bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240bfff700 1 --2- 172.21.15.142:0/3659823767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0099b0 tx=0x7f23fc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7626e6808f342415 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 1 -- 172.21.15.142:0/3659823767 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00ba70 con 0x7f240c0967a0 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:52.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: dump: 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:27:52.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 5 monclient: authenticate success, global_id 75911 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 1 -- 172.21.15.142:0/3659823767 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc00bbd0 con 0x7f240c0967a0 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 4 set_mon_vals no callback set 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 1 -- 172.21.15.142:0/3659823767 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc011420 con 0x7f240c0967a0 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f240b7fe700 10 monclient: dump: 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.435 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 10 monclient: shutdown 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- 172.21.15.142:0/3659823767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 msgr2=0x7f240c09bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 --2- 172.21.15.142:0/3659823767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0099b0 tx=0x7f23fc009250 comp rx=0 tx=0).stop 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- 172.21.15.142:0/3659823767 shutdown_connections 2024-09-18T10:27:52.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 --2- 172.21.15.142:0/3659823767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c0967a0 0x7f240c09bde0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 --2- 172.21.15.142:0/3659823767 >> v2:172.21.15.142:3300/0 conn(0x7f240c09aae0 0x7f240c096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- 172.21.15.142:0/3659823767 >> 172.21.15.142:0/3659823767 conn(0x7f240c0a1fe0 msgr2=0x7f240c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- 172.21.15.142:0/3659823767 shutdown_connections 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- 172.21.15.142:0/3659823767 wait complete. 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 10 monclient: build_initial_monmap 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 10 monclient: monmap: 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.437 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 Processor -- start 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 1 -- start start 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.421+0000 7f2411c81700 10 monclient: init 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-a con 0x7f240c0967a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f240c09aae0 0x7f240c141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-b con 0x7f240c09aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:52.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 0x7f240c08dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): picked mon.noname-c con 0x7f240c09c320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c06f8d0 con 0x7f240c0967a0 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c097750 con 0x7f240c09aae0 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240c08fc60 con 0x7f240c09c320 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411c81700 10 monclient(hunting): authenticate will time out at 1961.353180s 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53926/0 (socket says 172.21.15.142:53926) 2024-09-18T10:27:52.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 -- 172.21.15.142:0/2580969503 learned_addr learned my addr 172.21.15.142:0/2580969503 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 10 monclient(hunting): get_auth_request con 0x7f240c0967a0 auth_method 0 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 0x7f240c08dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): get_auth_request con 0x7f240c09c320 auth_method 0 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:52.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 msgr2=0x7f240c1418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 msgr2=0x7f240c1418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f240c09aae0 0x7f240c141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): get_auth_request con 0x7f240c09aae0 auth_method 0 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:52.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2410c7f700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f240bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient(hunting): handle_auth_done global_id 75914 payload 824 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f240c09aae0 msgr2=0x7f240c141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f240c09aae0 0x7f240c141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 msgr2=0x7f240c1418a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient: _finish_hunting 0 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 monclient: found mon.noname-c 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.425+0000 7f2411480700 1 -- 172.21.15.142:0/2580969503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0094f0 con 0x7f240c09c320 2024-09-18T10:27:52.444 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411480700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 msgr2=0x7f240c08dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411480700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 0x7f240c08dd00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f240400c490 tx=0x7f2404007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24040117a0 con 0x7f240c09c320 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: dump: 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.445 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: _finish_auth 0 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2404011920 con 0x7f240c09c320 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 5 monclient: authenticate success, global_id 75914 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _renew_subs 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404011ce0 con 0x7f240c09c320 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.446 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f240affd700 4 set_mon_vals no callback set 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c08f750 con 0x7f240c09c320 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 10 monclient: dump: 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:52.447 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _renew_subs 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c090320 con 0x7f240c09c320 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2404012630 con 0x7f240c09c320 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c072a80 con 0x7f240c09c320 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 0x7f23f0037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2404048c00 con 0x7f240c09c320 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2410c7f700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 0x7f23f0037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:52.448 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.429+0000 7f2410c7f700 10 monclient: get_auth_request con 0x7f23f0035ac0 auth_method 0 2024-09-18T10:27:52.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.433+0000 7f2410c7f700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 0x7f23f0037f70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f23f40094d0 tx=0x7f23f400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:52.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.437+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2404002010 con 0x7f240c09c320 2024-09-18T10:27:52.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.437+0000 7f2408ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:52.449 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.437+0000 7f2408ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:52.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2411c81700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2411c81700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2411c81700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f240c0678a0 con 0x7f240c09c320 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2408ff9700 1 -- 172.21.15.142:0/2580969503 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f24040102e0 con 0x7f240c09c320 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2408ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:52.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.661+0000 7f2408ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:52.670 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":180,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 msgr2=0x7f23f0037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 0x7f23f0037f70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f23f40094d0 tx=0x7f23f400b040 comp rx=0 tx=0).stop 2024-09-18T10:27:52.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 10 monclient: shutdown 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 msgr2=0x7f240c08dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 0x7f240c08dd00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f240400c490 tx=0x7f2404007ea0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 shutdown_connections 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f23f0035ac0 0x7f23f0037f70 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f240c09c320 0x7f240c08dd00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f240c09aae0 0x7f240c141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 --2- 172.21.15.142:0/2580969503 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f240c0967a0 0x7f240c1418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 >> 172.21.15.142:0/2580969503 conn(0x7f240c0a1fe0 msgr2=0x7f240c09f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 shutdown_connections 2024-09-18T10:27:52.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:52.665+0000 7f2411c81700 1 -- 172.21.15.142:0/2580969503 wait complete. 2024-09-18T10:27:55.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:55.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: get_monmap_and_config 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: build_initial_monmap 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: monmap: 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:55.794505+0000 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:55.794505+0000 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:55.797 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:55.798 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:55.798 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:55.798 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:55.798 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 Processor -- start 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 -- start start 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: init 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-b con 0x7f38a009d920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38a00953c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f38a00957d0 0x7f38a00973c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-c con 0x7f38a00957d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:55.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a009b0b0 con 0x7f38a00953c0 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a006fba0 con 0x7f38a00957d0 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a009a7e0 con 0x7f38a009d920 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.789+0000 7f38a7003700 10 monclient(hunting): authenticate will time out at 1964.719985s 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44012/0 (socket says 172.21.15.142:44012) 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 -- 172.21.15.142:0/2384616162 learned_addr learned my addr 172.21.15.142:0/2384616162 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): get_auth_request con 0x7f38a009d920 auth_method 0 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:55.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 1 --2- 172.21.15.142:0/2384616162 >> v2:172.21.15.150:3300/0 conn(0x7f38a00957d0 0x7f38a00973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): get_auth_request con 0x7f38a00957d0 auth_method 0 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6802700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient(hunting): handle_auth_done global_id 75917 payload 274 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 -- 172.21.15.142:0/2384616162 >> v2:172.21.15.150:3300/0 conn(0x7f38a00957d0 msgr2=0x7f38a00973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:55.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 --2- 172.21.15.142:0/2384616162 >> v2:172.21.15.150:3300/0 conn(0x7f38a00957d0 0x7f38a00973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 -- 172.21.15.142:0/2384616162 >> v1:172.21.15.150:6789/0 conn(0x7f38a00953c0 legacy=0x7f38a0096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient: _finish_hunting 0 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 monclient: found mon.noname-b 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:55.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 -- 172.21.15.142:0/2384616162 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0097900 con 0x7f38a009d920 2024-09-18T10:27:55.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 -- 172.21.15.142:0/2384616162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 msgr2=0x7f38a006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:55.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.793+0000 7f38a6001700 1 --2- 172.21.15.142:0/2384616162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f389000bc50 tx=0x7f389000b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397d98f96396a2e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:55.808 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 1 -- 172.21.15.142:0/2384616162 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389000de00 con 0x7f38a009d920 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: dump: 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:55.809 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: _finish_auth 0 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 1 -- 172.21.15.142:0/2384616162 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3890010470 con 0x7f38a009d920 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 5 monclient: authenticate success, global_id 75917 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 1 -- 172.21.15.142:0/2384616162 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890010830 con 0x7f38a009d920 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 4 set_mon_vals no callback set 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:55.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a4fff700 10 monclient: dump: 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 10 monclient: shutdown 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:55.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- 172.21.15.142:0/2384616162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 msgr2=0x7f38a006f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 --2- 172.21.15.142:0/2384616162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f389000bc50 tx=0x7f389000b4f0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- 172.21.15.142:0/2384616162 shutdown_connections 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 --2- 172.21.15.142:0/2384616162 >> v2:172.21.15.150:3300/0 conn(0x7f38a00957d0 0x7f38a00973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 --2- 172.21.15.142:0/2384616162 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a009d920 0x7f38a006f660 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- 172.21.15.142:0/2384616162 >> 172.21.15.142:0/2384616162 conn(0x7f38a00a1fe0 msgr2=0x7f38a0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- 172.21.15.142:0/2384616162 shutdown_connections 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- 172.21.15.142:0/2384616162 wait complete. 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 10 monclient: build_initial_monmap 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 10 monclient: monmap: 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:55.812 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 Processor -- start 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 1 -- start start 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.797+0000 7f38a7003700 10 monclient: init 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-a con 0x7f38a00953c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 0x7f38a008a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-c con 0x7f38a00957d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 0x7f38a008a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): picked mon.noname-b con 0x7f38a009d920 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a008ba60 con 0x7f38a00953c0 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a00a0a40 con 0x7f38a009d920 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a008c0b0 con 0x7f38a00957d0 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:55.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient(hunting): authenticate will time out at 1964.728385s 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 0x7f38a008a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53938/0 (socket says 172.21.15.142:53938) 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 -- 172.21.15.142:0/2269267819 learned_addr learned my addr 172.21.15.142:0/2269267819 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 10 monclient(hunting): get_auth_request con 0x7f38a00953c0 auth_method 0 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): get_auth_request con 0x7f38a00957d0 auth_method 0 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:55.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 msgr2=0x7f38a0089b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 msgr2=0x7f38a0089b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6001700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 0x7f38a008a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 10 monclient(hunting): get_auth_request con 0x7f38a009d920 auth_method 0 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient(hunting): handle_auth_done global_id 75920 payload 824 2024-09-18T10:27:55.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 msgr2=0x7f38a008a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 0x7f38a008a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 msgr2=0x7f38a0089b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient: _finish_hunting 0 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 monclient: found mon.noname-c 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 -- 172.21.15.142:0/2269267819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389000b790 con 0x7f38a00957d0 2024-09-18T10:27:55.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a6802700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 0x7f38a008a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:27:55.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 msgr2=0x7f38a008a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:55.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a5800700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 0x7f38a008a0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c4e0 tx=0x7f389c007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:55.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0117a0 con 0x7f38a00957d0 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: dump: 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:55.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 5 monclient: authenticate success, global_id 75920 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _renew_subs 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a008cfa0 con 0x7f38a00957d0 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c011920 con 0x7f38a00957d0 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c011ce0 con 0x7f38a00957d0 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:55.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f3895ffb700 10 monclient: dump: 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _renew_subs 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:55.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a008e230 con 0x7f38a00957d0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.801+0000 7f38977fe700 4 set_mon_vals no callback set 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f389c012630 con 0x7f38a00957d0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f3895ffb700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 0x7f3884037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a7003700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a7003700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a7003700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f389c048da0 con 0x7f38a00957d0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a0095080 con 0x7f38a00957d0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a6001700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 0x7f3884037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.805+0000 7f38a6001700 10 monclient: get_auth_request con 0x7f3884035ac0 auth_method 0 2024-09-18T10:27:55.823 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.809+0000 7f38a6001700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 0x7f3884037f70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f389000b760 tx=0x7f389001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:55.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.813+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f389c010900 con 0x7f38a00957d0 2024-09-18T10:27:55.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.817+0000 7f3895ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:55.824 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:55.817+0000 7f3895ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:56.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.033+0000 7f38a7003700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:56.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.033+0000 7f38a7003700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:56.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.033+0000 7f38a7003700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:56.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.033+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f38a0072a80 con 0x7f38a00957d0 2024-09-18T10:27:56.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f3895ffb700 1 -- 172.21.15.142:0/2269267819 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f389c02a610 con 0x7f38a00957d0 2024-09-18T10:27:56.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f3895ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:56.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f3895ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:56.042 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":180,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:56.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 msgr2=0x7f3884037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:56.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 0x7f3884037f70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f389000b760 tx=0x7f389001a000 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 10 monclient: shutdown 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 msgr2=0x7f38a008a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 0x7f38a008a0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c4e0 tx=0x7f389c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 shutdown_connections 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f3884035ac0 0x7f3884037f70 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a009d920 0x7f38a008a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f38a00957d0 0x7f38a008a0c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 --2- 172.21.15.142:0/2269267819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38a00953c0 0x7f38a0089b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 >> 172.21.15.142:0/2269267819 conn(0x7f38a00a1fe0 msgr2=0x7f38a009f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 shutdown_connections 2024-09-18T10:27:56.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:56.037+0000 7f38a7003700 1 -- 172.21.15.142:0/2269267819 wait complete. 2024-09-18T10:27:59.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:27:59.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.157+0000 7f2e06e34700 10 monclient: get_monmap_and_config 2024-09-18T10:27:59.165 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.157+0000 7f2e06e34700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.157+0000 7f2e06e34700 10 monclient: monmap: 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:59.163195+0000 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:59.163195+0000 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:59.166 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:59.167 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.168 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 Processor -- start 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 -- start start 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient: init 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 0x7f2e001323c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-c con 0x7f2e000676a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:59.169 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e000985c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e000989c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e00067a70 con 0x7f2e000985c0 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0012f610 con 0x7f2e000989c0 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0009e5d0 con 0x7f2e000676a0 2024-09-18T10:27:59.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06e34700 10 monclient(hunting): authenticate will time out at 1968.089102s 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05631700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2e000985c0 0x7f2e0012e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49900/0 (socket says 172.21.15.142:49900) 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05631700 1 -- 172.21.15.142:0/2100134430 learned_addr learned my addr 172.21.15.142:0/2100134430 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 0x7f2e001323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 10 monclient(hunting): get_auth_request con 0x7f2e000676a0 auth_method 0 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 msgr2=0x7f2e001323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 msgr2=0x7f2e001323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 0x7f2e001323c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:27:59.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 0x7f2e001323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:27:59.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e05631700 1 -- 172.21.15.142:0/2100134430 >> v1:172.21.15.142:6789/0 conn(0x7f2e000985c0 legacy=0x7f2e0012e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.161+0000 7f2e06633700 1 -- 172.21.15.142:0/2100134430 >> v1:172.21.15.150:6789/0 conn(0x7f2e000989c0 legacy=0x7f2e0012ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977866365 0 0) 0x7f2e0012f610 con 0x7f2e000989c0 2024-09-18T10:27:59.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): my global_id is 194335 2024-09-18T10:27:59.179 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dec003570 con 0x7f2e000989c0 2024-09-18T10:27:59.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125352616 0 0) 0x7f2dec003570 con 0x7f2e000989c0 2024-09-18T10:27:59.181 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 msgr2=0x7f2e001323c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 --2- 172.21.15.142:0/2100134430 >> v2:172.21.15.150:3300/0 conn(0x7f2e000676a0 0x7f2e001323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 >> v1:172.21.15.142:6789/0 conn(0x7f2e000985c0 legacy=0x7f2e0012e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e00096af0 con 0x7f2e000989c0 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f2df8001b70 con 0x7f2e000989c0 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.182 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: dump: 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.183 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 >> v1:172.21.15.150:6789/0 conn(0x7f2e000989c0 legacy=0x7f2e0012ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2dec004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): picked mon.b con 0x7f2dec003c50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dec004810 0x7f2dec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): picked mon.a con 0x7f2dec004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2dec009a50 0x7f2dec00be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): picked mon.c con 0x7f2dec009a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0009e5d0 con 0x7f2dec004810 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec00c4b0 con 0x7f2dec003c50 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.184 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec00c670 con 0x7f2dec009a50 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e04e30700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dec004810 0x7f2dec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2dec004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): get_auth_request con 0x7f2dec004810 auth_method 0 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): get_auth_request con 0x7f2dec003c50 auth_method 0 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:59.185 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2dec009a50 0x7f2dec00be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): get_auth_request con 0x7f2dec009a50 auth_method 0 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.186 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient(hunting): handle_auth_done global_id 194335 payload 293 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2dec009a50 msgr2=0x7f2dec00be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2dec009a50 0x7f2dec00be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dec004810 msgr2=0x7f2dec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2dec004810 0x7f2dec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 monclient: found mon.b 2024-09-18T10:27:59.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec003570 con 0x7f2dec003c50 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e05e32700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06633700 1 -- 172.21.15.142:0/2100134430 reap_dead start 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06e34700 5 monclient: authenticate success, global_id 194335 2024-09-18T10:27:59.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.173+0000 7f2e06e34700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.177+0000 7f2e05e32700 1 -- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 msgr2=0x7f2dec004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.177+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2dec004060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00bb40 tx=0x7f2dfc008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4200cf36fe465533 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc018070 con 0x7f2dec003c50 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: dump: 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.193 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc00d7e0 con 0x7f2dec003c50 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:59.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 4 set_mon_vals no callback set 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 1 -- 172.21.15.142:0/2100134430 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc00dba0 con 0x7f2dec003c50 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e04e30700 10 monclient: dump: 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.195 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 10 monclient: shutdown 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 -- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 msgr2=0x7f2dec004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2dec004060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00bb40 tx=0x7f2dfc008040 comp rx=0 tx=0).stop 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 -- 172.21.15.142:0/2100134430 shutdown_connections 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2100134430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2dec004060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 -- 172.21.15.142:0/2100134430 >> 172.21.15.142:0/2100134430 conn(0x7f2e00071ae0 msgr2=0x7f2e000a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 -- 172.21.15.142:0/2100134430 shutdown_connections 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 1 -- 172.21.15.142:0/2100134430 wait complete. 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.181+0000 7f2e06e34700 10 monclient: monmap: 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:59.197 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 Processor -- start 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 -- start start 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient: init 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2e001415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-b con 0x7f2dec003c50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-c con 0x7f2e00141ae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e0012f040 0x7f2e001186f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): picked mon.noname-a con 0x7f2e0012f040 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0009dfb0 con 0x7f2e0012f040 2024-09-18T10:27:59.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e00131190 con 0x7f2dec003c50 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0011a0f0 con 0x7f2e00141ae0 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06e34700 10 monclient(hunting): authenticate will time out at 1968.112946s 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44034/0 (socket says 172.21.15.142:44034) 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 -- 172.21.15.142:0/2837628873 learned_addr learned my addr 172.21.15.142:0/2837628873 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): get_auth_request con 0x7f2e00141ae0 auth_method 0 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.199 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e0012f040 0x7f2e001186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): get_auth_request con 0x7f2e0012f040 auth_method 0 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e06633700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient(hunting): handle_auth_done global_id 75923 payload 824 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 -- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 msgr2=0x7f2e001415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.200 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2e001415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 -- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e0012f040 msgr2=0x7f2e001186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e0012f040 0x7f2e001186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 monclient: found mon.noname-c 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.201 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 -- 172.21.15.142:0/2837628873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0011bc20 con 0x7f2e00141ae0 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 -- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 msgr2=0x7f2e0012eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.185+0000 7f2e05631700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2df0001e90 tx=0x7f2df0007e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0012770 con 0x7f2e00141ae0 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: dump: 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.203 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 5 monclient: authenticate success, global_id 75923 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df00128d0 con 0x7f2e00141ae0 2024-09-18T10:27:59.204 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _renew_subs 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0011bf00 con 0x7f2e00141ae0 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0012c70 con 0x7f2e00141ae0 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 10 monclient: dump: 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.205 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df7fff700 4 set_mon_vals no callback set 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _renew_subs 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0011acc0 con 0x7f2e00141ae0 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2df0013600 con 0x7f2e00141ae0 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.206 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e00072a80 con 0x7f2e00141ae0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 0x7f2de4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2df00490d0 con 0x7f2e00141ae0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 0x7f2de4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.189+0000 7f2e05e32700 10 monclient: get_auth_request con 0x7f2de4035ac0 auth_method 0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.193+0000 7f2e05e32700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 0x7f2de4037f70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00b2b0 tx=0x7f2dfc01a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.201+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2df0011a90 con 0x7f2e00141ae0 2024-09-18T10:27:59.207 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.201+0000 7f2df5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.201+0000 7f2df5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:59.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:27:59.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2e00068360 con 0x7f2e00141ae0 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2df5ffb700 1 -- 172.21.15.142:0/2837628873 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f2df00112b0 con 0x7f2e00141ae0 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2df5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2df5ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:59.431 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:27:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 msgr2=0x7f2de4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 0x7f2de4037f70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00b2b0 tx=0x7f2dfc01a000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 10 monclient: shutdown 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 msgr2=0x7f2e0012eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2df0001e90 tx=0x7f2df0007e20 comp rx=0 tx=0).stop 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 shutdown_connections 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2de4035ac0 0x7f2de4037f70 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e0012f040 0x7f2e001186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e00141ae0 0x7f2e0012eb00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 --2- 172.21.15.142:0/2837628873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2dec003c50 0x7f2e001415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 >> 172.21.15.142:0/2837628873 conn(0x7f2e00071ae0 msgr2=0x7f2e0009e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 shutdown_connections 2024-09-18T10:27:59.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.425+0000 7f2e06e34700 1 -- 172.21.15.142:0/2837628873 wait complete. 2024-09-18T10:27:59.445 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:27:59.445 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:27:59.445 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:27:59.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: get_monmap_and_config 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: monmap: 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:59.546427+0000 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:59.546427+0000 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:59.549 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:59.550 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:59.550 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:59.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 Processor -- start 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 -- start start 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: init 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32f809abd0 0x7f32f809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-c con 0x7f32f809abd0 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32f809db80 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f8137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-b con 0x7f32f8097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80958c0 con 0x7f32f809db80 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f806fa60 con 0x7f32f809abd0 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8095ef0 con 0x7f32f8097880 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.541+0000 7f32fec6a700 10 monclient(hunting): authenticate will time out at 1968.471988s 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f8137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f7fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f32f809db80 0x7f32f809c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49904/0 (socket says 172.21.15.142:49904) 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f7fff700 1 -- 172.21.15.142:0/4205700576 learned_addr learned my addr 172.21.15.142:0/4205700576 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): get_auth_request con 0x7f32f8097880 auth_method 0 2024-09-18T10:27:59.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 1 --2- 172.21.15.142:0/4205700576 >> v2:172.21.15.150:3300/0 conn(0x7f32f809abd0 0x7f32f809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): get_auth_request con 0x7f32f809abd0 auth_method 0 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient(hunting): handle_auth_done global_id 75926 payload 274 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 -- 172.21.15.142:0/4205700576 >> v2:172.21.15.150:3300/0 conn(0x7f32f809abd0 msgr2=0x7f32f809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 --2- 172.21.15.142:0/4205700576 >> v2:172.21.15.150:3300/0 conn(0x7f32f809abd0 0x7f32f809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 -- 172.21.15.142:0/4205700576 >> v1:172.21.15.142:6789/0 conn(0x7f32f809db80 legacy=0x7f32f809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 monclient: found mon.noname-b 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 -- 172.21.15.142:0/4205700576 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f809afa0 con 0x7f32f8097880 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 -- 172.21.15.142:0/4205700576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 msgr2=0x7f32f8137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fd207700 1 --2- 172.21.15.142:0/4205700576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f8137690 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098e0 tx=0x7f32e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248d9ecb8932b21e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 1 -- 172.21.15.142:0/4205700576 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e800b850 con 0x7f32f8097880 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: dump: 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.562 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fec6a700 5 monclient: authenticate success, global_id 75926 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fec6a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 1 -- 172.21.15.142:0/4205700576 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e800b9b0 con 0x7f32f8097880 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.563 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fec6a700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32fec6a700 4 set_mon_vals no callback set 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 1 -- 172.21.15.142:0/4205700576 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e800bd50 con 0x7f32f8097880 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.545+0000 7f32f77fe700 10 monclient: dump: 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.564 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: shutdown 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- 172.21.15.142:0/4205700576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 msgr2=0x7f32f8137690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- 172.21.15.142:0/4205700576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f8137690 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098e0 tx=0x7f32e80091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- 172.21.15.142:0/4205700576 shutdown_connections 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- 172.21.15.142:0/4205700576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f8137690 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- 172.21.15.142:0/4205700576 >> v2:172.21.15.150:3300/0 conn(0x7f32f809abd0 0x7f32f809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- 172.21.15.142:0/4205700576 >> 172.21.15.142:0/4205700576 conn(0x7f32f80a14e0 msgr2=0x7f32f80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- 172.21.15.142:0/4205700576 shutdown_connections 2024-09-18T10:27:59.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- 172.21.15.142:0/4205700576 wait complete. 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: monmap: 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 Processor -- start 2024-09-18T10:27:59.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- start start 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: init 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f813f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-c con 0x7f32f8097880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32f809abd0 0x7f32f813fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-b con 0x7f32f809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f809db80 0x7f32f81401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): picked mon.noname-a con 0x7f32f809db80 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f808f6e0 con 0x7f32f809db80 2024-09-18T10:27:59.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8097c90 con 0x7f32f809abd0 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f809fbe0 con 0x7f32f8097880 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fec6a700 10 monclient(hunting): authenticate will time out at 1968.479768s 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f813f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f813f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44056/0 (socket says 172.21.15.142:44056) 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 1 -- 172.21.15.142:0/273202641 learned_addr learned my addr 172.21.15.142:0/273202641 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 10 monclient(hunting): get_auth_request con 0x7f32f8097880 auth_method 0 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fca06700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.549+0000 7f32fd207700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f809db80 0x7f32f81401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): get_auth_request con 0x7f32f809db80 auth_method 0 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32f809abd0 0x7f32f813fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): get_auth_request con 0x7f32f809abd0 auth_method 0 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient(hunting): handle_auth_done global_id 75929 payload 824 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32f809abd0 msgr2=0x7f32f813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32f809abd0 0x7f32f813fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f809db80 msgr2=0x7f32f81401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f809db80 0x7f32f81401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 monclient: found mon.noname-c 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e8009420 con 0x7f32f8097880 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fd207700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f809db80 0x7f32f81401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 msgr2=0x7f32f813f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fca06700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f813f740 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f32f000ecb0 tx=0x7f32f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f000ca60 con 0x7f32f8097880 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: dump: 2024-09-18T10:27:59.571 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f000cbc0 con 0x7f32f8097880 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 5 monclient: authenticate success, global_id 75929 2024-09-18T10:27:59.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f0013470 con 0x7f32f8097880 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f4ff9700 10 monclient: dump: 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.573 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32f67fc700 4 set_mon_vals no callback set 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 10 monclient: _renew_subs 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80903e0 con 0x7f32f8097880 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 10 monclient: _renew_subs 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.553+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f808d560 con 0x7f32f8097880 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fec6a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fec6a700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 0x7f32f80883b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8064950 con 0x7f32f8087fe0 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 0x7f32f80883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient: get_auth_request con 0x7f32f8087fe0 auth_method 0 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f32f00138a0 con 0x7f32f8097880 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 10 monclient(hunting): handle_auth_done global_id 214104 payload 824 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f4ff9700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32d8035ac0 0x7f32d8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32f80696a0 con 0x7f32f8087fe0 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f32f00495f0 con 0x7f32f8097880 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fd207700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32d8035ac0 0x7f32d8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32fd207700 10 monclient: get_auth_request con 0x7f32d8035ac0 auth_method 0 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 msgr2=0x7f32f80883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.557+0000 7f32f7fff700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 0x7f32f80883b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f32f808efc0 tx=0x7f32e0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000de80 con 0x7f32f8087fe0 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 10 monclient: dump: 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f32e00070d0 con 0x7f32f8087fe0 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 msgr2=0x7f32f80883b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 0x7f32f80883b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f32f808efc0 tx=0x7f32e0009040 comp rx=0 tx=0).stop 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 msgr2=0x7f32f80883b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32f4ff9700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f8087fe0 0x7f32f80883b0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f32f808efc0 tx=0x7f32e0009040 comp rx=0 tx=0).stop 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.561+0000 7f32fd207700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32d8035ac0 0x7f32d8037f70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f32e80030c0 tx=0x7f32e8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32fec6a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:59.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32fec6a700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 0x7f32f808bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32fec6a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f808f490 con 0x7f32f80888f0 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 0x7f32f808bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient: get_auth_request con 0x7f32f80888f0 auth_method 0 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 10 monclient(hunting): handle_auth_done global_id 214107 payload 824 2024-09-18T10:27:59.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.569+0000 7f32f7fff700 1 -- 172.21.15.142:0/273202641 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f32f8067950 con 0x7f32f80888f0 2024-09-18T10:27:59.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f7fff700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 msgr2=0x7f32f808bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f7fff700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 0x7f32f808bab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f32e0017000 tx=0x7f32e0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0022410 con 0x7f32f80888f0 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 10 monclient: dump: 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.583 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f32e0022570 con 0x7f32f80888f0 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 msgr2=0x7f32f808bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 0x7f32f808bab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f32e0017000 tx=0x7f32e0018000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 msgr2=0x7f32f808bab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32f4ff9700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32f80888f0 0x7f32f808bab0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f32e0017000 tx=0x7f32e0018000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:27:59.585 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.586 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.587 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:59.588 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.589 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:59.590 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.591 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.592 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.593 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:59.594 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:59.595 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.596 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.597 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.597 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.597 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:59.597 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32d8035ac0 msgr2=0x7f32d8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f32d8035ac0 0x7f32d8037f70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f32e80030c0 tx=0x7f32e8016000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 10 monclient: shutdown 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 msgr2=0x7f32f813f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 --2- 172.21.15.142:0/273202641 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f32f8097880 0x7f32f813f740 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f32f000ecb0 tx=0x7f32f00091a0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fd207700 1 -- 172.21.15.142:0/273202641 reap_dead start 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fd207700 1 -- 172.21.15.142:0/273202641 reap_dead start 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 shutdown_connections 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 >> 172.21.15.142:0/273202641 conn(0x7f32f80a14e0 msgr2=0x7f32f8096360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 shutdown_connections 2024-09-18T10:27:59.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.573+0000 7f32fec6a700 1 -- 172.21.15.142:0/273202641 wait complete. 2024-09-18T10:27:59.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:59.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:27:59.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:27:59.709 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.701+0000 7f283b030700 10 monclient: get_monmap_and_config 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.701+0000 7f283b030700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.701+0000 7f283b030700 10 monclient: monmap: 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:59.707237+0000 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:59.707237+0000 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:59.710 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:59.711 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 Processor -- start 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 -- start start 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient: init 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f283409cdd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2834134a50 0x7f2834136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): picked mon.noname-a con 0x7f2834134a50 addr v2:172.21.15.142:3300/0 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2834137370 addr v1:172.21.15.142:6789/0 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283409d1a0 con 0x7f2834137370 2024-09-18T10:27:59.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283409d780 con 0x7f283409cdd0 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283409dd40 con 0x7f2834134a50 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f283b030700 10 monclient(hunting): authenticate will time out at 1968.633655s 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2834134a50 0x7f2834136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2834134a50 0x7f2834136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53986/0 (socket says 172.21.15.142:53986) 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f28395cd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2834137370 0x7f2834139750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49908/0 (socket says 172.21.15.142:49908) 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 -- 172.21.15.142:0/502745254 learned_addr learned my addr 172.21.15.142:0/502745254 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): get_auth_request con 0x7f2834134a50 auth_method 0 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.714 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient(hunting): handle_auth_done global_id 214110 payload 274 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 -- 172.21.15.142:0/502745254 >> v1:172.21.15.150:6789/0 conn(0x7f283409cdd0 legacy=0x7f283409e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 -- 172.21.15.142:0/502745254 >> v1:172.21.15.142:6789/0 conn(0x7f2834137370 legacy=0x7f2834139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 monclient: found mon.noname-a 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:59.715 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.705+0000 7f2833fff700 1 -- 172.21.15.142:0/502745254 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28340a6710 con 0x7f2834134a50 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f2833fff700 1 -- 172.21.15.142:0/502745254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2834134a50 msgr2=0x7f2834136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f2833fff700 1 --2- 172.21.15.142:0/502745254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2834134a50 0x7f2834136e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f281c0099b0 tx=0x7f281c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8793585b96d0b38f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 1 -- 172.21.15.142:0/502745254 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c00bb50 con 0x7f2834134a50 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: dump: 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.717 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 1 -- 172.21.15.142:0/502745254 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c00bcb0 con 0x7f2834134a50 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 1 -- 172.21.15.142:0/502745254 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c011670 con 0x7f2834134a50 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 5 monclient: authenticate success, global_id 214110 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:59.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 4 set_mon_vals no callback set 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f28337fe700 10 monclient: dump: 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.719 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 10 monclient: shutdown 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 -- 172.21.15.142:0/502745254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2834134a50 msgr2=0x7f2834136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 --2- 172.21.15.142:0/502745254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2834134a50 0x7f2834136e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f281c0099b0 tx=0x7f281c009250 comp rx=0 tx=0).stop 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 -- 172.21.15.142:0/502745254 shutdown_connections 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 --2- 172.21.15.142:0/502745254 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2834134a50 0x7f2834136e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 -- 172.21.15.142:0/502745254 >> 172.21.15.142:0/502745254 conn(0x7f2834071680 msgr2=0x7f28340a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 -- 172.21.15.142:0/502745254 shutdown_connections 2024-09-18T10:27:59.720 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 1 -- 172.21.15.142:0/502745254 wait complete. 2024-09-18T10:27:59.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.709+0000 7f283b030700 10 monclient: monmap: 2024-09-18T10:27:59.725 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.725 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 Processor -- start 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 -- start start 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient: init 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): picked mon.noname-a con 0x7f283409cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834134a50 0x7f2834133a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): picked mon.noname-b con 0x7f2834134a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2834137370 0x7f2834133f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): picked mon.noname-c con 0x7f2834137370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283406da70 con 0x7f283409cdd0 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283406ecf0 con 0x7f2834134a50 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283406bd50 con 0x7f2834137370 2024-09-18T10:27:59.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f283b030700 10 monclient(hunting): authenticate will time out at 1968.641871s 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:53998/0 (socket says 172.21.15.142:53998) 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 -- 172.21.15.142:0/1649239417 learned_addr learned my addr 172.21.15.142:0/1649239417 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): get_auth_request con 0x7f283409cdd0 auth_method 0 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2834137370 0x7f2834133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834134a50 0x7f2834133a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): get_auth_request con 0x7f2834137370 auth_method 0 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): get_auth_request con 0x7f2834134a50 auth_method 0 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f28395cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient(hunting): handle_auth_done global_id 214113 payload 824 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2834137370 msgr2=0x7f2834133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2834137370 0x7f2834133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834134a50 msgr2=0x7f2834133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834134a50 0x7f2834133a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 monclient: found mon.noname-a 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:27:59.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.713+0000 7f2838dcc700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0094f0 con 0x7f283409cdd0 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2838dcc700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 msgr2=0x7f2834139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2838dcc700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f282400c5d0 tx=0x7f2824007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28240117a0 con 0x7f283409cdd0 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: dump: 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.731 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2824011900 con 0x7f283409cdd0 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 5 monclient: authenticate success, global_id 214113 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 monclient: _renew_subs 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283406d790 con 0x7f283409cdd0 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.732 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824011cc0 con 0x7f283409cdd0 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 10 monclient: dump: 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.733 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2832ffd700 4 set_mon_vals no callback set 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 monclient: _renew_subs 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283406de00 con 0x7f283409cdd0 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 0x7f28341344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2824012630 con 0x7f283409cdd0 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28340648c0 con 0x7f283406c830 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 0x7f28341344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient: get_auth_request con 0x7f283406c830 auth_method 0 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.734 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2814035ac0 0x7f2814037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2824048c20 con 0x7f283409cdd0 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 10 monclient(hunting): handle_auth_done global_id 75935 payload 824 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f2833fff700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f283406dc30 con 0x7f283406c830 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f28395cd700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2814035ac0 0x7f2814037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.717+0000 7f28395cd700 10 monclient: get_auth_request con 0x7f2814035ac0 auth_method 0 2024-09-18T10:27:59.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2833fff700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 msgr2=0x7f28341344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2833fff700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 0x7f28341344d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f281c015000 tx=0x7f281c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c021070 con 0x7f283406c830 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 10 monclient: dump: 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.737 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f28395cd700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2814035ac0 0x7f2814037f70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f282c006e00 tx=0x7f282c005340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f281c01c450 con 0x7f283406c830 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 msgr2=0x7f28341344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 0x7f28341344d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f281c015000 tx=0x7f281c016000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 msgr2=0x7f28341344d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.721+0000 7f2830ff9700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406c830 0x7f28341344d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f283b030700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f283b030700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 0x7f283406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f283b030700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283406c170 con 0x7f283406abb0 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 0x7f283406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient: get_auth_request con 0x7f283406abb0 auth_method 0 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 10 monclient(hunting): handle_auth_done global_id 75938 payload 824 2024-09-18T10:27:59.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.733+0000 7f2833fff700 1 -- 172.21.15.142:0/1649239417 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f28340696a0 con 0x7f283406abb0 2024-09-18T10:27:59.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2833fff700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 msgr2=0x7f283406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2833fff700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 0x7f283406afb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f28340721d0 tx=0x7f281c021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c01b650 con 0x7f283406abb0 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 10 monclient: dump: 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.743 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f281c01b7b0 con 0x7f283406abb0 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:59.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 msgr2=0x7f283406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 0x7f283406afb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f28340721d0 tx=0x7f281c021000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 msgr2=0x7f283406afb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f2830ff9700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f283406abb0 0x7f283406afb0 secure :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f28340721d0 tx=0x7f281c021000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:27:59.745 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.746 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.747 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:59.748 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.749 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:59.750 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.751 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.752 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.753 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:59.754 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.755 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:27:59.756 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:27:59.757 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:59.758 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:59.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2814035ac0 msgr2=0x7f2814037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2814035ac0 0x7f2814037f70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f282c006e00 tx=0x7f282c005340 comp rx=0 tx=0).stop 2024-09-18T10:27:59.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 10 monclient: shutdown 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 msgr2=0x7f2834139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f282400c5d0 tx=0x7f2824007f60 comp rx=0 tx=0).stop 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f28395cd700 1 -- 172.21.15.142:0/1649239417 reap_dead start 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 shutdown_connections 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 --2- 172.21.15.142:0/1649239417 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f283409cdd0 0x7f2834139f60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 >> 172.21.15.142:0/1649239417 conn(0x7f2834071680 msgr2=0x7f28341364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 shutdown_connections 2024-09-18T10:27:59.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.737+0000 7f283b030700 1 -- 172.21.15.142:0/1649239417 wait complete. 2024-09-18T10:27:59.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:59.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:27:59.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:27:59.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: get_monmap_and_config 2024-09-18T10:27:59.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.866 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: monmap: 2024-09-18T10:27:59.866 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.866 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:27:59.864093+0000 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:27:59.864093+0000 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:27:59.867 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 Processor -- start 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 -- start start 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: init 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-b con 0x7fe0a80a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0a80990b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0a809e360 0x7fe0a8139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-c con 0x7fe0a809e360 addr v2:172.21.15.150:3300/0 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a809d980 con 0x7fe0a80990b0 2024-09-18T10:27:59.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a809d780 con 0x7fe0a809e360 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a809fc70 con 0x7fe0a80a5f90 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0af36b700 10 monclient(hunting): authenticate will time out at 1968.790160s 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44092/0 (socket says 172.21.15.142:44092) 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 -- 172.21.15.142:0/2814158853 learned_addr learned my addr 172.21.15.142:0/2814158853 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): get_auth_request con 0x7fe0a80a5f90 auth_method 0 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 1 --2- 172.21.15.142:0/2814158853 >> v2:172.21.15.150:3300/0 conn(0x7fe0a809e360 0x7fe0a8139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): get_auth_request con 0x7fe0a809e360 auth_method 0 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient(hunting): handle_auth_done global_id 75941 payload 274 2024-09-18T10:27:59.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 -- 172.21.15.142:0/2814158853 >> v2:172.21.15.150:3300/0 conn(0x7fe0a809e360 msgr2=0x7fe0a8139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 --2- 172.21.15.142:0/2814158853 >> v2:172.21.15.150:3300/0 conn(0x7fe0a809e360 0x7fe0a8139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 -- 172.21.15.142:0/2814158853 >> v1:172.21.15.150:6789/0 conn(0x7fe0a80990b0 legacy=0x7fe0a80a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 monclient: found mon.noname-b 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.861+0000 7fe0ad107700 1 -- 172.21.15.142:0/2814158853 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80a6360 con 0x7fe0a80a5f90 2024-09-18T10:27:59.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0ad107700 1 -- 172.21.15.142:0/2814158853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 msgr2=0x7fe0a80a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0ad107700 1 --2- 172.21.15.142:0/2814158853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40099b0 tx=0x7fe0a4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32dc2f7ddff3ebe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 1 -- 172.21.15.142:0/2814158853 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4009ea0 con 0x7fe0a80a5f90 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: dump: 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.879 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 1 -- 172.21.15.142:0/2814158853 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a400ba70 con 0x7fe0a80a5f90 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 5 monclient: authenticate success, global_id 75941 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 10 monclient: get_monmap_and_config success 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 4 set_mon_vals no callback set 2024-09-18T10:27:59.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 1 -- 172.21.15.142:0/2814158853 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400be30 con 0x7fe0a80a5f90 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe09bfff700 10 monclient: dump: 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.881 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 10 monclient: shutdown 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 -- 172.21.15.142:0/2814158853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 msgr2=0x7fe0a80a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 --2- 172.21.15.142:0/2814158853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40099b0 tx=0x7fe0a4009250 comp rx=0 tx=0).stop 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 -- 172.21.15.142:0/2814158853 shutdown_connections 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 --2- 172.21.15.142:0/2814158853 >> v2:172.21.15.150:3300/0 conn(0x7fe0a809e360 0x7fe0a8139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 --2- 172.21.15.142:0/2814158853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a80a5f90 0x7fe0a80a0270 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 -- 172.21.15.142:0/2814158853 >> 172.21.15.142:0/2814158853 conn(0x7fe0a8071680 msgr2=0x7fe0a80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 -- 172.21.15.142:0/2814158853 shutdown_connections 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 1 -- 172.21.15.142:0/2814158853 wait complete. 2024-09-18T10:27:59.882 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 10 monclient: build_initial_monmap 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.865+0000 7fe0af36b700 10 monclient: monmap: 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 Processor -- start 2024-09-18T10:27:59.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 -- start start 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient: init 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient: _reopen_session rank -1 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 0x7fe0a8141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-a con 0x7fe0a80990b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 0x7fe0a8142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-c con 0x7fe0a809e360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a80a5f90 0x7fe0a8142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): picked mon.noname-b con 0x7fe0a80a5f90 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a80984b0 con 0x7fe0a80990b0 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a809c120 con 0x7fe0a80a5f90 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a808edd0 con 0x7fe0a809e360 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): _renew_subs 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0af36b700 10 monclient(hunting): authenticate will time out at 1968.797078s 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 0x7fe0a8141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 0x7fe0a8142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 0x7fe0a8141ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54004/0 (socket says 172.21.15.142:54004) 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 1 -- 172.21.15.142:0/3306765970 learned_addr learned my addr 172.21.15.142:0/3306765970 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): get_auth_request con 0x7fe0a80990b0 auth_method 0 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): get_auth_request con 0x7fe0a809e360 auth_method 0 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a80a5f90 0x7fe0a8142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): get_auth_request con 0x7fe0a80a5f90 auth_method 0 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad908700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient(hunting): handle_auth_done global_id 75944 payload 824 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a80a5f90 msgr2=0x7fe0a8142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a80a5f90 0x7fe0a8142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 msgr2=0x7fe0a8141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 0x7fe0a8141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient: _finish_hunting 0 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 monclient: found mon.noname-c 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:27:59.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ac906700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40094f0 con 0x7fe0a809e360 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.869+0000 7fe0ad107700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe0a80990b0 0x7fe0a8141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ac906700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 msgr2=0x7fe0a8142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ac906700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 0x7fe0a8142000 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00c4e0 tx=0x7fe09c007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00ce90 con 0x7fe0a809e360 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: dump: 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.888 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: _finish_auth 0 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c0117a0 con 0x7fe0a809e360 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 5 monclient: authenticate success, global_id 75944 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 monclient: _renew_subs 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a808e8c0 con 0x7fe0a809e360 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c011b80 con 0x7fe0a809e360 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 10 monclient: dump: 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.890 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe09b7fe700 4 set_mon_vals no callback set 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 monclient: _renew_subs 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:27:59.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a8091660 con 0x7fe0a809e360 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 0x7fe0a805e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a808e450 con 0x7fe0a808e080 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe09c01c460 con 0x7fe0a809e360 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 0x7fe0a805e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient: get_auth_request con 0x7fe0a808e080 auth_method 0 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe08c035ac0 0x7fe08c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe09c048ba0 con 0x7fe0a809e360 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 10 monclient(hunting): handle_auth_done global_id 194353 payload 824 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad107700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0a8090500 con 0x7fe0a808e080 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad908700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe08c035ac0 0x7fe08c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.873+0000 7fe0ad908700 10 monclient: get_auth_request con 0x7fe08c035ac0 auth_method 0 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0ad107700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 msgr2=0x7fe0a805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0ad107700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 0x7fe0a805e890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40030a0 tx=0x7fe0a4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4021070 con 0x7fe0a808e080 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0ad908700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe08c035ac0 0x7fe08c037f70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009fd0 tx=0x7fe0a000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 10 monclient: dump: 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.896 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe0a4000cf0 con 0x7fe0a808e080 2024-09-18T10:27:59.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:27:59.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:27:59.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 msgr2=0x7fe0a805e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 0x7fe0a805e890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40030a0 tx=0x7fe0a4016000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 msgr2=0x7fe0a805e890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.877+0000 7fe0997fa700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a808e080 0x7fe0a805e890 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40030a0 tx=0x7fe0a4016000 comp rx=0 tx=0).stop 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.885+0000 7fe0af36b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.885+0000 7fe0af36b700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 0x7fe0a8029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.885+0000 7fe0af36b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.885+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8064950 con 0x7fe0a805edd0 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 0x7fe0a8029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient: get_auth_request con 0x7fe0a805edd0 auth_method 0 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:27:59.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:27:59.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 10 monclient(hunting): handle_auth_done global_id 194356 payload 824 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 1 -- 172.21.15.142:0/3306765970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe0a806db00 con 0x7fe0a805edd0 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 msgr2=0x7fe0a8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0ad107700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 0x7fe0a8029d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40094c0 tx=0x7fe0a401e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4032480 con 0x7fe0a805edd0 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0997fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.889+0000 7fe0997fa700 10 monclient: dump: 2024-09-18T10:27:59.900 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7fe0a40325e0 con 0x7fe0a805edd0 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:27:59.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 msgr2=0x7fe0a8029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 0x7fe0a8029d00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40094c0 tx=0x7fe0a401e070 comp rx=0 tx=0).stop 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 msgr2=0x7fe0a8029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0997fa700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a805edd0 0x7fe0a8029d00 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40094c0 tx=0x7fe0a401e070 comp rx=0 tx=0).stop 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:59.902 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.903 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.904 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:27:59.905 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:27:59.906 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:27:59.907 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.908 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.909 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:27:59.910 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:27:59.911 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.912 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.913 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:27:59.914 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe08c035ac0 msgr2=0x7fe08c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe08c035ac0 0x7fe08c037f70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009fd0 tx=0x7fe0a000b040 comp rx=0 tx=0).stop 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 10 monclient: shutdown 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 msgr2=0x7fe0a8142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 --2- 172.21.15.142:0/3306765970 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe0a809e360 0x7fe0a8142000 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00c4e0 tx=0x7fe09c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0ad908700 1 -- 172.21.15.142:0/3306765970 reap_dead start 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0ad908700 1 -- 172.21.15.142:0/3306765970 reap_dead start 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 shutdown_connections 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 >> 172.21.15.142:0/3306765970 conn(0x7fe0a8071680 msgr2=0x7fe0a806ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:27:59.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 shutdown_connections 2024-09-18T10:27:59.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:27:59.893+0000 7fe0af36b700 1 -- 172.21.15.142:0/3306765970 wait complete. 2024-09-18T10:27:59.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:27:59.918 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:27:59.918 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: get_monmap_and_config 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: build_initial_monmap 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: monmap: 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:00.027894+0000 2024-09-18T10:28:00.030 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:00.027894+0000 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:00.031 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 Processor -- start 2024-09-18T10:28:00.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 -- start start 2024-09-18T10:28:00.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: init 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba440a5f90 0x7fba440a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-c con 0x7fba440a5f90 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fba440990b0 0x7fba440a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-a con 0x7fba440990b0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba4409e360 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4409d980 con 0x7fba4409e360 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4409d780 con 0x7fba440990b0 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:00.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4409fc70 con 0x7fba440a5f90 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba49ea1700 10 monclient(hunting): authenticate will time out at 1968.953374s 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fba440990b0 0x7fba440a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba43fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fba4409e360 0x7fba44138a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49912/0 (socket says 172.21.15.142:49912) 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba43fff700 1 -- 172.21.15.142:0/1676790246 learned_addr learned my addr 172.21.15.142:0/1676790246 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): get_auth_request con 0x7fba440990b0 auth_method 0 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:00.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba437fe700 1 --2- 172.21.15.142:0/1676790246 >> v2:172.21.15.150:3300/0 conn(0x7fba440a5f90 0x7fba440a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba437fe700 10 monclient(hunting): get_auth_request con 0x7fba440a5f90 auth_method 0 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba437fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient(hunting): handle_auth_done global_id 214119 payload 274 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 -- 172.21.15.142:0/1676790246 >> v2:172.21.15.150:3300/0 conn(0x7fba440a5f90 msgr2=0x7fba440a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 --2- 172.21.15.142:0/1676790246 >> v2:172.21.15.150:3300/0 conn(0x7fba440a5f90 0x7fba440a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 -- 172.21.15.142:0/1676790246 >> v1:172.21.15.142:6789/0 conn(0x7fba4409e360 legacy=0x7fba44138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 monclient: found mon.noname-a 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:00.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.025+0000 7fba42ffd700 1 -- 172.21.15.142:0/1676790246 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440a6360 con 0x7fba440990b0 2024-09-18T10:28:00.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba42ffd700 1 -- 172.21.15.142:0/1676790246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba440990b0 msgr2=0x7fba440a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:00.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba42ffd700 1 --2- 172.21.15.142:0/1676790246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba440990b0 0x7fba440a07b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba440a1e60 tx=0x7fba2c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb30d3d81888d9e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:00.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 1 -- 172.21.15.142:0/1676790246 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00b890 con 0x7fba440990b0 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: dump: 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:00.043 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 5 monclient: authenticate success, global_id 214119 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 1 -- 172.21.15.142:0/1676790246 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c00b9f0 con 0x7fba440990b0 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 4 set_mon_vals no callback set 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 1 -- 172.21.15.142:0/1676790246 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00bd90 con 0x7fba440990b0 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:00.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba427fc700 10 monclient: dump: 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: shutdown 2024-09-18T10:28:00.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- 172.21.15.142:0/1676790246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba440990b0 msgr2=0x7fba440a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- 172.21.15.142:0/1676790246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba440990b0 0x7fba440a07b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba440a1e60 tx=0x7fba2c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- 172.21.15.142:0/1676790246 shutdown_connections 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- 172.21.15.142:0/1676790246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba440990b0 0x7fba440a07b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- 172.21.15.142:0/1676790246 >> v2:172.21.15.150:3300/0 conn(0x7fba440a5f90 0x7fba440a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- 172.21.15.142:0/1676790246 >> 172.21.15.142:0/1676790246 conn(0x7fba44071680 msgr2=0x7fba440a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- 172.21.15.142:0/1676790246 shutdown_connections 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- 172.21.15.142:0/1676790246 wait complete. 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: build_initial_monmap 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: monmap: 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:00.046 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 Processor -- start 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- start start 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: init 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba440990b0 0x7fba44141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.047 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-b con 0x7fba440990b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-a con 0x7fba4409e360 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba440a5f90 0x7fba4408cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): picked mon.noname-c con 0x7fba440a5f90 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440984b0 con 0x7fba4409e360 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440a4220 con 0x7fba440990b0 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4408ee80 con 0x7fba440a5f90 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:00.048 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.029+0000 7fba49ea1700 10 monclient(hunting): authenticate will time out at 1968.959907s 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54026/0 (socket says 172.21.15.142:54026) 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 -- 172.21.15.142:0/3898763136 learned_addr learned my addr 172.21.15.142:0/3898763136 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba440a5f90 0x7fba4408cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): get_auth_request con 0x7fba4409e360 auth_method 0 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): get_auth_request con 0x7fba440a5f90 auth_method 0 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:00.049 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba440990b0 0x7fba44141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): get_auth_request con 0x7fba440990b0 auth_method 0 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:00.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient(hunting): handle_auth_done global_id 214122 payload 824 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 -- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba440a5f90 msgr2=0x7fba4408cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba440a5f90 0x7fba4408cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 -- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba440990b0 msgr2=0x7fba44141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba440990b0 0x7fba44141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 monclient: found mon.noname-a 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 -- 172.21.15.142:0/3898763136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c009420 con 0x7fba4409e360 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 -- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 msgr2=0x7fba441421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba42ffd700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba44068f40 tx=0x7fba2c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:00.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00b890 con 0x7fba4409e360 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: dump: 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:00.052 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 5 monclient: authenticate success, global_id 214122 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c011d50 con 0x7fba4409e360 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 10 monclient: _renew_subs 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4408e970 con 0x7fba4409e360 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01a6c0 con 0x7fba4409e360 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba3b7fe700 10 monclient: dump: 2024-09-18T10:28:00.053 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba41ffb700 4 set_mon_vals no callback set 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 10 monclient: _renew_subs 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.033+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44090550 con 0x7fba4409e360 2024-09-18T10:28:00.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba49ea1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:00.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba49ea1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:00.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba49ea1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba440696a0 con 0x7fba4409e360 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fba2c01aaf0 con 0x7fba4409e360 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba3b7fe700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 0x7fba24037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fba2c04a830 con 0x7fba4409e360 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba437fe700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 0x7fba24037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.037+0000 7fba437fe700 10 monclient: get_auth_request con 0x7fba24035b10 auth_method 0 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.041+0000 7fba437fe700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 0x7fba24037fc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009fd0 tx=0x7fba3c009880 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.045+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fba2c022520 con 0x7fba4409e360 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.045+0000 7fba3b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:00.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.045+0000 7fba3b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:00.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.261+0000 7fba49ea1700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:28:00.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.261+0000 7fba49ea1700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:28:00.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.261+0000 7fba49ea1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:00.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.261+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fba44067950 con 0x7fba4409e360 2024-09-18T10:28:00.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.265+0000 7fba3b7fe700 1 -- 172.21.15.142:0/3898763136 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fba2c029020 con 0x7fba4409e360 2024-09-18T10:28:00.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.265+0000 7fba3b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:28:00.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.265+0000 7fba3b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:00.272 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.273 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=8,logm=49} crc {auth=352052156,config=3140514009,health=1939983743,kv=4261215878,logm=4115996183}) 2024-09-18T10:28:00.273 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=1102703426,config=3140514009,health=536113986,kv=4261215878,logm=3195560112}) 2024-09-18T10:28:00.273 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.273 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=8,logm=49} crc {auth=352052156,config=3140514009,health=1939983743,kv=4261215878,logm=4115996183}) 2024-09-18T10:28:00.273 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.265+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=1093192839,config=3140514009,health=536113986,kv=4261215878,logm=2984975970}) 2024-09-18T10:28:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 msgr2=0x7fba24037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 0x7fba24037fc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009fd0 tx=0x7fba3c009880 comp rx=0 tx=0).stop 2024-09-18T10:28:00.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 10 monclient: shutdown 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 msgr2=0x7fba441421a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba44068f40 tx=0x7fba2c017000 comp rx=0 tx=0).stop 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 shutdown_connections 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fba24035b10 0x7fba24037fc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba440a5f90 0x7fba4408cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba4409e360 0x7fba441421a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 --2- 172.21.15.142:0/3898763136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba440990b0 0x7fba44141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:00.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 >> 172.21.15.142:0/3898763136 conn(0x7fba44071680 msgr2=0x7fba4409e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:00.277 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.277 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368619207}) 2024-09-18T10:28:00.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 shutdown_connections 2024-09-18T10:28:00.277 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135087460}) 2024-09-18T10:28:00.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:00.269+0000 7fba49ea1700 1 -- 172.21.15.142:0/3898763136 wait complete. 2024-09-18T10:28:00.278 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.278 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3368619207}) 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.269+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890211481}) 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390280383}) 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778450121}) 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390280383}) 2024-09-18T10:28:00.279 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1373634462}) 2024-09-18T10:28:00.281 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.281 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155886852}) 2024-09-18T10:28:00.281 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355795381}) 2024-09-18T10:28:00.281 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.282 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155886852}) 2024-09-18T10:28:00.282 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.273+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161866267}) 2024-09-18T10:28:00.284 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.284 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970291129}) 2024-09-18T10:28:00.284 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1605288305}) 2024-09-18T10:28:00.284 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.285 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970291129}) 2024-09-18T10:28:00.285 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.277+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522789137}) 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587771187}) 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780258680}) 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2587771187}) 2024-09-18T10:28:00.286 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201906719}) 2024-09-18T10:28:00.287 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.287 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191983408}) 2024-09-18T10:28:00.287 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3249094301}) 2024-09-18T10:28:00.287 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.287 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191983408}) 2024-09-18T10:28:00.288 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693292588}) 2024-09-18T10:28:00.288 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.288 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484993574}) 2024-09-18T10:28:00.288 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785669570}) 2024-09-18T10:28:00.289 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.289 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484993574}) 2024-09-18T10:28:00.289 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3528367869}) 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063413524}) 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208775268}) 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063413524}) 2024-09-18T10:28:00.290 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.281+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2631014240}) 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114947912}) 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34443305}) 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114947912}) 2024-09-18T10:28:00.291 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870628476}) 2024-09-18T10:28:00.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180113032}) 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606714453}) 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180113032}) 2024-09-18T10:28:00.293 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635558674}) 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884210403}) 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068393384}) 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884210403}) 2024-09-18T10:28:00.294 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.285+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266066896}) 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363897437}) 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922089336}) 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363897437}) 2024-09-18T10:28:00.295 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396542030}) 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214684214}) 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498520587}) 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214684214}) 2024-09-18T10:28:00.296 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1995399342}) 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693562912}) 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2821206299}) 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693562912}) 2024-09-18T10:28:00.297 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.289+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345207899}) 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402348436}) 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3197245609}) 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402348436}) 2024-09-18T10:28:00.298 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3961785919}) 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838388093}) 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=115064792}) 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838388093}) 2024-09-18T10:28:00.299 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393331677}) 2024-09-18T10:28:00.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398131011}) 2024-09-18T10:28:00.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219380438}) 2024-09-18T10:28:00.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.300 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398131011}) 2024-09-18T10:28:00.301 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890185575}) 2024-09-18T10:28:00.302 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854503176}) 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3517553114}) 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854503176}) 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.293+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057246808}) 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.303 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905395304}) 2024-09-18T10:28:00.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904739304}) 2024-09-18T10:28:00.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905395304}) 2024-09-18T10:28:00.304 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255391436}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890963723}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330086432}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890963723}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229542741}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875671768}) 2024-09-18T10:28:00.306 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410468065}) 2024-09-18T10:28:00.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875671768}) 2024-09-18T10:28:00.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.297+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777566198}) 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862831165}) 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311763169}) 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862831165}) 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106894613}) 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.308 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243765537}) 2024-09-18T10:28:00.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412862622}) 2024-09-18T10:28:00.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243765537}) 2024-09-18T10:28:00.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721321737}) 2024-09-18T10:28:00.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016696766}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279130578}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016696766}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.301+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3997053591}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939936746}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374890526}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939936746}) 2024-09-18T10:28:00.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485224111}) 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825783110}) 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086015080}) 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3825783110}) 2024-09-18T10:28:00.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549568156}) 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372105901}) 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244117327}) 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372105901}) 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.305+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082551403}) 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314983862}) 2024-09-18T10:28:00.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196740111}) 2024-09-18T10:28:00.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1314983862}) 2024-09-18T10:28:00.315 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240727945}) 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342317279}) 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491141271}) 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342317279}) 2024-09-18T10:28:00.316 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.309+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35395959}) 2024-09-18T10:28:00.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778602616}) 2024-09-18T10:28:00.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763916235}) 2024-09-18T10:28:00.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778602616}) 2024-09-18T10:28:00.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=687998832}) 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779268526}) 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168565416}) 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779268526}) 2024-09-18T10:28:00.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.317+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890926579}) 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574157003}) 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763195556}) 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574157003}) 2024-09-18T10:28:00.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178150545}) 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432197804}) 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1780041163}) 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432197804}) 2024-09-18T10:28:00.329 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.321+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3510088641}) 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737343289}) 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4294747326}) 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737343289}) 2024-09-18T10:28:00.335 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.329+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197452289}) 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311780452}) 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770535482}) 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311780452}) 2024-09-18T10:28:00.348 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.341+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1001018976}) 2024-09-18T10:28:00.676 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.677 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732780586}) 2024-09-18T10:28:00.677 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281178436}) 2024-09-18T10:28:00.677 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.677 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732780586}) 2024-09-18T10:28:00.677 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.669+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214266598}) 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222710463}) 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460482269}) 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222710463}) 2024-09-18T10:28:00.684 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.677+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2412695928}) 2024-09-18T10:28:00.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639677476}) 2024-09-18T10:28:00.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3837563002}) 2024-09-18T10:28:00.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=639677476}) 2024-09-18T10:28:00.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252842194}) 2024-09-18T10:28:00.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925878219}) 2024-09-18T10:28:00.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771064174}) 2024-09-18T10:28:00.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925878219}) 2024-09-18T10:28:00.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.841+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349434781}) 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133527466}) 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420028047}) 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133527466}) 2024-09-18T10:28:00.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.845+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296871668}) 2024-09-18T10:28:00.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937475784}) 2024-09-18T10:28:00.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645194974}) 2024-09-18T10:28:00.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937475784}) 2024-09-18T10:28:00.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947556899}) 2024-09-18T10:28:00.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638126848}) 2024-09-18T10:28:00.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306034465}) 2024-09-18T10:28:00.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638126848}) 2024-09-18T10:28:00.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.849+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103496953}) 2024-09-18T10:28:00.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555958294}) 2024-09-18T10:28:00.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=591530197}) 2024-09-18T10:28:00.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555958294}) 2024-09-18T10:28:00.859 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3408766489}) 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074562236}) 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764987374}) 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2074562236}) 2024-09-18T10:28:00.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.853+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666061543}) 2024-09-18T10:28:00.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193924527}) 2024-09-18T10:28:00.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339143411}) 2024-09-18T10:28:00.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193924527}) 2024-09-18T10:28:00.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225419007}) 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229321469}) 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1557081129}) 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229321469}) 2024-09-18T10:28:00.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106987566}) 2024-09-18T10:28:00.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494587428}) 2024-09-18T10:28:00.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130333233}) 2024-09-18T10:28:00.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494587428}) 2024-09-18T10:28:00.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.857+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861368175}) 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652608924}) 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115849700}) 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2652608924}) 2024-09-18T10:28:00.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2599042984}) 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242388639}) 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136512204}) 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242388639}) 2024-09-18T10:28:00.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657851669}) 2024-09-18T10:28:00.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795889897}) 2024-09-18T10:28:00.869 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134149946}) 2024-09-18T10:28:00.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=795889897}) 2024-09-18T10:28:00.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.861+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113379998}) 2024-09-18T10:28:00.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.870 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576741218}) 2024-09-18T10:28:00.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3853459076}) 2024-09-18T10:28:00.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3576741218}) 2024-09-18T10:28:00.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141633655}) 2024-09-18T10:28:00.871 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679643307}) 2024-09-18T10:28:00.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254708252}) 2024-09-18T10:28:00.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679643307}) 2024-09-18T10:28:00.872 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2931975857}) 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874460638}) 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=361582545}) 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874460638}) 2024-09-18T10:28:00.873 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.865+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417178457}) 2024-09-18T10:28:00.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.874 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876194057}) 2024-09-18T10:28:00.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1436975566}) 2024-09-18T10:28:00.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876194057}) 2024-09-18T10:28:00.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768598779}) 2024-09-18T10:28:00.875 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411614617}) 2024-09-18T10:28:00.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3690791605}) 2024-09-18T10:28:00.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411614617}) 2024-09-18T10:28:00.876 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213122286}) 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210602681}) 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923810748}) 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210602681}) 2024-09-18T10:28:00.877 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.869+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753236322}) 2024-09-18T10:28:00.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.878 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32267119}) 2024-09-18T10:28:00.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1837925119}) 2024-09-18T10:28:00.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32267119}) 2024-09-18T10:28:00.879 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=41112794}) 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901969257}) 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148229274}) 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901969257}) 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27224171}) 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.881 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100583967}) 2024-09-18T10:28:00.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159591411}) 2024-09-18T10:28:00.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100583967}) 2024-09-18T10:28:00.882 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.873+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1691357141}) 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179069676}) 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541203552}) 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179069676}) 2024-09-18T10:28:00.883 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102040756}) 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964410504}) 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4068682256}) 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964410504}) 2024-09-18T10:28:00.884 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.877+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2833847159}) 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091360982}) 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478447484}) 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091360982}) 2024-09-18T10:28:00.886 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820633412}) 2024-09-18T10:28:00.887 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338133881}) 2024-09-18T10:28:00.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149308406}) 2024-09-18T10:28:00.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338133881}) 2024-09-18T10:28:00.888 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472982454}) 2024-09-18T10:28:00.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030875761}) 2024-09-18T10:28:00.889 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3899522856}) 2024-09-18T10:28:00.890 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.890 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1030875761}) 2024-09-18T10:28:00.890 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.881+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622697373}) 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785396733}) 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504388695}) 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785396733}) 2024-09-18T10:28:00.891 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226909774}) 2024-09-18T10:28:00.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.892 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677767286}) 2024-09-18T10:28:00.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898424165}) 2024-09-18T10:28:00.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677767286}) 2024-09-18T10:28:00.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.885+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563979478}) 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583690438}) 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159432650}) 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583690438}) 2024-09-18T10:28:00.894 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1823308120}) 2024-09-18T10:28:00.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753501039}) 2024-09-18T10:28:00.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154613430}) 2024-09-18T10:28:00.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.895 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3753501039}) 2024-09-18T10:28:00.896 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430809986}) 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212561109}) 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225026401}) 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2212561109}) 2024-09-18T10:28:00.897 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.889+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124054093}) 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486810778}) 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101373113}) 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486810778}) 2024-09-18T10:28:00.898 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966035796}) 2024-09-18T10:28:00.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.899 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011912756}) 2024-09-18T10:28:00.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853871164}) 2024-09-18T10:28:00.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011912756}) 2024-09-18T10:28:00.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751469014}) 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344294284}) 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667446412}) 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=344294284}) 2024-09-18T10:28:00.901 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.893+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788594704}) 2024-09-18T10:28:00.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947951042}) 2024-09-18T10:28:00.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501269649}) 2024-09-18T10:28:00.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.902 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947951042}) 2024-09-18T10:28:00.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403817772}) 2024-09-18T10:28:00.903 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439663824}) 2024-09-18T10:28:00.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612599722}) 2024-09-18T10:28:00.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439663824}) 2024-09-18T10:28:00.904 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848679524}) 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382441877}) 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472862456}) 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382441877}) 2024-09-18T10:28:00.905 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.897+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701921319}) 2024-09-18T10:28:00.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977429967}) 2024-09-18T10:28:00.906 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807219513}) 2024-09-18T10:28:00.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977429967}) 2024-09-18T10:28:00.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592965440}) 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020757854}) 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3393285347}) 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020757854}) 2024-09-18T10:28:00.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423283439}) 2024-09-18T10:28:00.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671266356}) 2024-09-18T10:28:00.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2091367544}) 2024-09-18T10:28:00.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671266356}) 2024-09-18T10:28:00.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.901+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013587680}) 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121812007}) 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1995970244}) 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121812007}) 2024-09-18T10:28:00.911 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185816155}) 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514981454}) 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756272374}) 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514981454}) 2024-09-18T10:28:00.912 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745827260}) 2024-09-18T10:28:00.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.913 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172808383}) 2024-09-18T10:28:00.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023009134}) 2024-09-18T10:28:00.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172808383}) 2024-09-18T10:28:00.914 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.905+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124409087}) 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2423807968}) 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=506186624}) 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2423807968}) 2024-09-18T10:28:00.915 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2330015617}) 2024-09-18T10:28:00.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400813304}) 2024-09-18T10:28:00.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779802811}) 2024-09-18T10:28:00.916 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400813304}) 2024-09-18T10:28:00.917 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.909+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3949110413}) 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902180694}) 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=288271368}) 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902180694}) 2024-09-18T10:28:00.918 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178595643}) 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389699772}) 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136177810}) 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389699772}) 2024-09-18T10:28:00.920 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.913+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520902552}) 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503158404}) 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355384692}) 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503158404}) 2024-09-18T10:28:00.922 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2517110595}) 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547352567}) 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=682374374}) 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547352567}) 2024-09-18T10:28:00.924 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.917+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=488089506}) 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223672431}) 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91267378}) 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223672431}) 2024-09-18T10:28:00.926 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395224028}) 2024-09-18T10:28:00.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.927 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647125360}) 2024-09-18T10:28:00.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57498149}) 2024-09-18T10:28:00.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647125360}) 2024-09-18T10:28:00.928 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1485882156}) 2024-09-18T10:28:00.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348197608}) 2024-09-18T10:28:00.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1245512510}) 2024-09-18T10:28:00.929 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348197608}) 2024-09-18T10:28:00.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.921+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436955832}) 2024-09-18T10:28:00.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.930 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892656007}) 2024-09-18T10:28:00.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2923584402}) 2024-09-18T10:28:00.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892656007}) 2024-09-18T10:28:00.931 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578967889}) 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305227038}) 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56461965}) 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305227038}) 2024-09-18T10:28:00.932 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781840438}) 2024-09-18T10:28:00.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.933 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806672066}) 2024-09-18T10:28:00.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872686023}) 2024-09-18T10:28:00.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806672066}) 2024-09-18T10:28:00.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.925+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=309747564}) 2024-09-18T10:28:00.934 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202190738}) 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679665110}) 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202190738}) 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3240236643}) 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528274020}) 2024-09-18T10:28:00.935 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195172570}) 2024-09-18T10:28:00.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528274020}) 2024-09-18T10:28:00.936 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.929+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1000565510}) 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=92136361,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663136,mgr_metadata=2738435100,mgrstat=2556900986,mon_config_key=2207406588,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1306083370}) 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=39} crc {logm=3504697222,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=154145421,mon_config_key=620546692,monmap=3140514009,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=3710981144}) 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=37} crc {logm=92136361,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663136,mgr_metadata=2738435100,mgrstat=2556900986,mon_config_key=2207406588,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1306083370}) 2024-09-18T10:28:00.938 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.933+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=38} crc {logm=2546589547,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=57299912,mon_config_key=620546692,monmap=3505902565,osd_metadata=2539426001,osd_pg_creating=69081760,osd_snap=454132730,osdmap=4212746949}) 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=952025566}) 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=13} crc {osdmap=3464279230}) 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=952025566}) 2024-09-18T10:28:00.946 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:00.941+0000 7f5d3195d700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=14} crc {osdmap=3783972964}) 2024-09-18T10:28:01.294 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:28:01.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: monmap: 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.404 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:01.401793+0000 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:01.401793+0000 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:01.405 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 Processor -- start 2024-09-18T10:28:01.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 -- start start 2024-09-18T10:28:01.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: init 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194012f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f19400a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 0x7f194009f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f194012f810 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19400a02d0 0x7f19401320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f19400a02d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194009d990 con 0x7f19400a02d0 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194009dc70 con 0x7f194012f810 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400a20c0 con 0x7f19400a1cf0 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.397+0000 7f193f59e700 10 monclient(hunting): authenticate will time out at 1970.327995s 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194012f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19400a02d0 0x7f19401320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f19400a02d0 0x7f19401320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54038/0 (socket says 172.21.15.142:54038) 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194012f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:44130/0 (socket says 172.21.15.142:44130) 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 -- 172.21.15.142:0/835586796 learned_addr learned my addr 172.21.15.142:0/835586796 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): get_auth_request con 0x7f19400a02d0 auth_method 0 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): get_auth_request con 0x7f19400a1cf0 auth_method 0 2024-09-18T10:28:01.412 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 1 --2- 172.21.15.142:0/835586796 >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 0x7f194009f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 10 monclient(hunting): get_auth_request con 0x7f194012f810 auth_method 0 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient(hunting): handle_auth_done global_id 214125 payload 274 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 -- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 msgr2=0x7f194012f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 --2- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194012f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 -- 172.21.15.142:0/835586796 >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 msgr2=0x7f194009f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 --2- 172.21.15.142:0/835586796 >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 0x7f194009f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 monclient: found mon.noname-a 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 -- 172.21.15.142:0/835586796 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194009f9c0 con 0x7f19400a02d0 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193dd9b700 1 --2- 172.21.15.142:0/835586796 >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 0x7f194009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 -- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19400a02d0 msgr2=0x7f19401320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193ed9d700 1 --2- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19400a02d0 0x7f19401320e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19400a6930 tx=0x7f1938009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e4e2f556500028 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 1 -- 172.21.15.142:0/835586796 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800e070 con 0x7f19400a02d0 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: dump: 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.415 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 1 -- 172.21.15.142:0/835586796 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193800bc30 con 0x7f19400a02d0 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193f59e700 5 monclient: authenticate success, global_id 214125 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 1 -- 172.21.15.142:0/835586796 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380092e0 con 0x7f19400a02d0 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193f59e700 4 set_mon_vals no callback set 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.401+0000 7f193d59a700 10 monclient: dump: 2024-09-18T10:28:01.416 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: shutdown 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19400a02d0 msgr2=0x7f19401320e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19400a02d0 0x7f19401320e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19400a6930 tx=0x7f1938009840 comp rx=0 tx=0).stop 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- 172.21.15.142:0/835586796 shutdown_connections 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f19400a02d0 0x7f19401320e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- 172.21.15.142:0/835586796 >> v2:172.21.15.150:3300/0 conn(0x7f194012f810 0x7f194009f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- 172.21.15.142:0/835586796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194012f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- 172.21.15.142:0/835586796 >> 172.21.15.142:0/835586796 conn(0x7f1940071820 msgr2=0x7f19400a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- 172.21.15.142:0/835586796 shutdown_connections 2024-09-18T10:28:01.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- 172.21.15.142:0/835586796 wait complete. 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: monmap: 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.426 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 Processor -- start 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- start start 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: init 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19400a02d0 0x7f194006f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f19400a02d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f19400a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:01.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f194012f810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194006cc00 con 0x7f194012f810 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194006d300 con 0x7f19400a02d0 2024-09-18T10:28:01.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194006c920 con 0x7f19400a1cf0 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193f59e700 10 monclient(hunting): authenticate will time out at 1970.334885s 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54052/0 (socket says 172.21.15.142:54052) 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 -- 172.21.15.142:0/1363697207 learned_addr learned my addr 172.21.15.142:0/1363697207 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): get_auth_request con 0x7f194012f810 auth_method 0 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): get_auth_request con 0x7f19400a1cf0 auth_method 0 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19400a02d0 0x7f194006f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): get_auth_request con 0x7f19400a02d0 auth_method 0 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient(hunting): handle_auth_done global_id 214128 payload 824 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 -- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 msgr2=0x7f194006b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194006b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 -- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19400a02d0 msgr2=0x7f194006f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19400a02d0 0x7f194006f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 monclient: found mon.noname-a 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.405+0000 7f193ed9d700 1 -- 172.21.15.142:0/1363697207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938009ae0 con 0x7f194012f810 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193ed9d700 1 -- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 msgr2=0x7f194009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193ed9d700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1938012000 tx=0x7f1938018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938008c10 con 0x7f194012f810 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: dump: 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.432 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 5 monclient: authenticate success, global_id 214128 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938008db0 con 0x7f194012f810 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380266b0 con 0x7f194012f810 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 10 monclient: dump: 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193cd99700 4 set_mon_vals no callback set 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _renew_subs 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194006cee0 con 0x7f194012f810 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _renew_subs 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194009b420 con 0x7f194012f810 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1940072a80 con 0x7f194012f810 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.409+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1938003040 con 0x7f194012f810 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.413+0000 7f192e7fc700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.413+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f193804acf0 con 0x7f194012f810 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.413+0000 7f193e59c700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.413+0000 7f193e59c700 10 monclient: get_auth_request con 0x7f1920035ac0 auth_method 0 2024-09-18T10:28:01.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.413+0000 7f193e59c700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1930009db0 tx=0x7f19300091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.421+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f193802f5d0 con 0x7f194012f810 2024-09-18T10:28:01.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.421+0000 7f192e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.421+0000 7f192e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:01.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f19400a1a90 con 0x7f194012f810 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f192e7fc700 1 -- 172.21.15.142:0/1363697207 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f193802b030 con 0x7f194012f810 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f192e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f192e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:01.647 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:01.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 msgr2=0x7f1920037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1930009db0 tx=0x7f19300091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 10 monclient: shutdown 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 msgr2=0x7f194009aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.641+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1938012000 tx=0x7f1938018000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 shutdown_connections 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f194012f810 0x7f194009aee0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f19400a1cf0 0x7f194006b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 --2- 172.21.15.142:0/1363697207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19400a02d0 0x7f194006f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 >> 172.21.15.142:0/1363697207 conn(0x7f1940071820 msgr2=0x7f194012ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 shutdown_connections 2024-09-18T10:28:01.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.645+0000 7f193f59e700 1 -- 172.21.15.142:0/1363697207 wait complete. 2024-09-18T10:28:01.665 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:28:01.665 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:28:01.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:28:01.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: get_monmap_and_config 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: monmap: 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:01.769877+0000 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:01.769877+0000 2024-09-18T10:28:01.772 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:01.773 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 1 Processor -- start 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 1 -- start start 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: init 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca0009cf70 0x7fca0009e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient(hunting): picked mon.noname-a con 0x7fca0009cf70 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca000a0270 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:01.776 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca000a0da0 0x7fca00139a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 10 monclient(hunting): picked mon.noname-c con 0x7fca000a0da0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.765+0000 7fca07865700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca000a1df0 con 0x7fca000a0270 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0009dfb0 con 0x7fca0009cf70 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca000a1650 con 0x7fca000a0da0 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 10 monclient(hunting): authenticate will time out at 1970.696044s 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca0009cf70 0x7fca0009e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fca0009cf70 0x7fca0009e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54054/0 (socket says 172.21.15.142:54054) 2024-09-18T10:28:01.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 -- 172.21.15.142:0/3012947543 learned_addr learned my addr 172.21.15.142:0/3012947543 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): get_auth_request con 0x7fca0009cf70 auth_method 0 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05e02700 1 --2- 172.21.15.142:0/3012947543 >> v2:172.21.15.150:3300/0 conn(0x7fca000a0da0 0x7fca00139a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05e02700 10 monclient(hunting): get_auth_request con 0x7fca000a0da0 auth_method 0 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05e02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05e02700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05e02700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient(hunting): handle_auth_done global_id 214131 payload 274 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 -- 172.21.15.142:0/3012947543 >> v2:172.21.15.150:3300/0 conn(0x7fca000a0da0 msgr2=0x7fca00139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 --2- 172.21.15.142:0/3012947543 >> v2:172.21.15.150:3300/0 conn(0x7fca000a0da0 0x7fca00139a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 -- 172.21.15.142:0/3012947543 >> v1:172.21.15.150:6789/0 conn(0x7fca000a0270 legacy=0x7fca000a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 monclient: found mon.noname-a 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.779 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 -- 172.21.15.142:0/3012947543 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0009d340 con 0x7fca0009cf70 2024-09-18T10:28:01.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 -- 172.21.15.142:0/3012947543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0009cf70 msgr2=0x7fca0009e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca05601700 1 --2- 172.21.15.142:0/3012947543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0009cf70 0x7fca0009e3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fca000a33c0 tx=0x7fc9fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ae6523f82db0a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 1 -- 172.21.15.142:0/3012947543 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00e070 con 0x7fca0009cf70 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: dump: 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.783 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 5 monclient: authenticate success, global_id 214131 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 1 -- 172.21.15.142:0/3012947543 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc00bc60 con 0x7fca0009cf70 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 1 -- 172.21.15.142:0/3012947543 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0092e0 con 0x7fca0009cf70 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fca07865700 4 set_mon_vals no callback set 2024-09-18T10:28:01.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.769+0000 7fc9f7fff700 10 monclient: dump: 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: shutdown 2024-09-18T10:28:01.785 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- 172.21.15.142:0/3012947543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0009cf70 msgr2=0x7fca0009e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- 172.21.15.142:0/3012947543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0009cf70 0x7fca0009e3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fca000a33c0 tx=0x7fc9fc009840 comp rx=0 tx=0).stop 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- 172.21.15.142:0/3012947543 shutdown_connections 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- 172.21.15.142:0/3012947543 >> v2:172.21.15.150:3300/0 conn(0x7fca000a0da0 0x7fca00139a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- 172.21.15.142:0/3012947543 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0009cf70 0x7fca0009e3d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- 172.21.15.142:0/3012947543 >> 172.21.15.142:0/3012947543 conn(0x7fca000a5880 msgr2=0x7fca000a5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- 172.21.15.142:0/3012947543 shutdown_connections 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- 172.21.15.142:0/3012947543 wait complete. 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: monmap: 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:01.786 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 Processor -- start 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- start start 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: init 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0009cf70 0x7fca00141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.787 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): picked mon.noname-b con 0x7fca0009cf70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): picked mon.noname-a con 0x7fca000a0270 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca000a0da0 0x7fca0008d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): picked mon.noname-c con 0x7fca000a0da0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000986e0 con 0x7fca000a0270 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca00090a80 con 0x7fca0009cf70 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0008ec70 con 0x7fca000a0da0 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca07865700 10 monclient(hunting): authenticate will time out at 1970.703703s 2024-09-18T10:28:01.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54070/0 (socket says 172.21.15.142:54070) 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 1 -- 172.21.15.142:0/1398374561 learned_addr learned my addr 172.21.15.142:0/1398374561 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): get_auth_request con 0x7fca000a0270 auth_method 0 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca05e02700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca000a0da0 0x7fca0008d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca05e02700 10 monclient(hunting): get_auth_request con 0x7fca000a0da0 auth_method 0 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca05e02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca05e02700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca05e02700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.773+0000 7fca04e00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0009cf70 0x7fca00141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05e02700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05e02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05e02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 10 monclient(hunting): get_auth_request con 0x7fca0009cf70 auth_method 0 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 10 monclient(hunting): handle_auth_done global_id 214134 payload 824 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca000a0da0 msgr2=0x7fca0008d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fca000a0da0 0x7fca0008d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0009cf70 msgr2=0x7fca00141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0009cf70 0x7fca00141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 monclient: found mon.noname-a 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009ae0 con 0x7fca000a0270 2024-09-18T10:28:01.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca0009cf70 0x7fca00141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 msgr2=0x7fca00142230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca04e00700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000c5d0 tx=0x7fc9f0007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f00117a0 con 0x7fca000a0270 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: dump: 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.794 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f0011900 con 0x7fca000a0270 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 5 monclient: authenticate success, global_id 214134 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 10 monclient: _renew_subs 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.795 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca000907a0 con 0x7fca000a0270 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0011cc0 con 0x7fca000a0270 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f57fa700 10 monclient: dump: 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.796 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fc9f6ffd700 4 set_mon_vals no callback set 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 10 monclient: _renew_subs 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.777+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca00142770 con 0x7fca000a0270 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca07865700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca07865700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 0x7fca0008e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca00064850 con 0x7fca0008de40 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc9f0012630 con 0x7fca000a0270 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 0x7fca0008e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient: get_auth_request con 0x7fca0008de40 auth_method 0 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9e4035ac0 0x7fc9e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc9f0048ce0 con 0x7fca000a0270 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 10 monclient(hunting): handle_auth_done global_id 214137 payload 824 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fca0008f9e0 con 0x7fca0008de40 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05e02700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9e4035ac0 0x7fc9e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05e02700 10 monclient: get_auth_request con 0x7fc9e4035ac0 auth_method 0 2024-09-18T10:28:01.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 msgr2=0x7fca0008e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 0x7fca0008e210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca000a1c90 tx=0x7fc9fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc012a60 con 0x7fca0008de40 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 10 monclient: dump: 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fc9fc026400 con 0x7fca0008de40 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.781+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 msgr2=0x7fca0008e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.785+0000 7fc9f57fa700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 0x7fca0008e210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca000a1c90 tx=0x7fc9fc017000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.785+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 msgr2=0x7fca0008e210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.785+0000 7fc9f57fa700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca0008de40 0x7fca0008e210 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca000a1c90 tx=0x7fc9fc017000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.785+0000 7fca05e02700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9e4035ac0 0x7fc9e4037f70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fca00091110 tx=0x7fc9f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca07865700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca07865700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 0x7fca0002a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca07865700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0006d2d0 con 0x7fca00029d00 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 0x7fca0002a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient: get_auth_request con 0x7fca00029d00 auth_method 0 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 10 monclient(hunting): handle_auth_done global_id 214140 payload 824 2024-09-18T10:28:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.793+0000 7fca05601700 1 -- 172.21.15.142:0/1398374561 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fca000696a0 con 0x7fca00029d00 2024-09-18T10:28:01.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca05601700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 msgr2=0x7fca0002a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca05601700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 0x7fca0002a100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc022000 tx=0x7fc9fc01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc02d070 con 0x7fca00029d00 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 10 monclient: dump: 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7fc9fc01ee00 con 0x7fca00029d00 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:01.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 msgr2=0x7fca0002a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 0x7fca0002a100 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc022000 tx=0x7fc9fc01f000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 msgr2=0x7fca0002a100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fc9f57fa700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca00029d00 0x7fca0002a100 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc022000 tx=0x7fc9fc01f000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:01.806 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.807 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.808 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.811 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.812 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.813 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.814 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.815 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:01.816 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.817 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.818 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:01.818 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9e4035ac0 msgr2=0x7fc9e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc9e4035ac0 0x7fc9e4037f70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fca00091110 tx=0x7fc9f8008040 comp rx=0 tx=0).stop 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 10 monclient: shutdown 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 msgr2=0x7fca00142230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca05e02700 1 -- 172.21.15.142:0/1398374561 reap_dead start 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000c5d0 tx=0x7fc9f0007f60 comp rx=0 tx=0).stop 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 shutdown_connections 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 --2- 172.21.15.142:0/1398374561 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fca000a0270 0x7fca00142230 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 >> 172.21.15.142:0/1398374561 conn(0x7fca000a5880 msgr2=0x7fca000a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 shutdown_connections 2024-09-18T10:28:01.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.797+0000 7fca07865700 1 -- 172.21.15.142:0/1398374561 wait complete. 2024-09-18T10:28:01.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:01.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:28:01.821 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.921+0000 7f692da01700 10 monclient: get_monmap_and_config 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.921+0000 7f692da01700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.921+0000 7f692da01700 10 monclient: monmap: 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:01.926642+0000 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:01.926642+0000 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:01.929 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:01.930 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:01.930 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:01.930 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:01.930 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 Processor -- start 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 -- start start 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient: init 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f692809c700 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6928097b10 0x7f692809d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): picked mon.noname-c con 0x7f6928097b10 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:01.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6928137750 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:01.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69280970a0 con 0x7f692809c700 2024-09-18T10:28:01.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6928097f10 con 0x7f6928137750 2024-09-18T10:28:01.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692809e240 con 0x7f6928097b10 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f692da01700 10 monclient(hunting): authenticate will time out at 1970.852975s 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f6926ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f692809c700 0x7f692809cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:49926/0 (socket says 172.21.15.142:49926) 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f6926ffd700 1 -- 172.21.15.142:0/392723731 learned_addr learned my addr 172.21.15.142:0/392723731 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 1 --2- 172.21.15.142:0/392723731 >> v2:172.21.15.150:3300/0 conn(0x7f6928097b10 0x7f692809d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): get_auth_request con 0x7f6928097b10 auth_method 0 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient(hunting): handle_auth_done global_id 194377 payload 274 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 1 -- 172.21.15.142:0/392723731 >> v1:172.21.15.150:6789/0 conn(0x7f6928137750 legacy=0x7f6928139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 1 -- 172.21.15.142:0/392723731 >> v1:172.21.15.142:6789/0 conn(0x7f692809c700 legacy=0x7f692809cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 1 monclient: found mon.noname-c 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:01.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.925+0000 7f69267fc700 1 -- 172.21.15.142:0/392723731 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928096dc0 con 0x7f6928097b10 2024-09-18T10:28:01.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f69267fc700 1 -- 172.21.15.142:0/392723731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 msgr2=0x7f692809d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f69267fc700 1 --2- 172.21.15.142:0/392723731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f692809d1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6928097830 tx=0x7f691c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ebc4864bb7b0a90 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 1 -- 172.21.15.142:0/392723731 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00b9a0 con 0x7f6928097b10 2024-09-18T10:28:01.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: dump: 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 5 monclient: authenticate success, global_id 194377 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 1 -- 172.21.15.142:0/392723731 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c00bb00 con 0x7f6928097b10 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 4 set_mon_vals no callback set 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 1 -- 172.21.15.142:0/392723731 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00bea0 con 0x7f6928097b10 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f6925ffb700 10 monclient: dump: 2024-09-18T10:28:01.939 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 10 monclient: shutdown 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- 172.21.15.142:0/392723731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 msgr2=0x7f692809d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 --2- 172.21.15.142:0/392723731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f692809d1e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6928097830 tx=0x7f691c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- 172.21.15.142:0/392723731 shutdown_connections 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 --2- 172.21.15.142:0/392723731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f692809d1e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- 172.21.15.142:0/392723731 >> 172.21.15.142:0/392723731 conn(0x7f6928071920 msgr2=0x7f69280a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- 172.21.15.142:0/392723731 shutdown_connections 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- 172.21.15.142:0/392723731 wait complete. 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 10 monclient: build_initial_monmap 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 10 monclient: monmap: 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:01.941 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 Processor -- start 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 1 -- start start 2024-09-18T10:28:01.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.929+0000 7f692da01700 10 monclient: init 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f69281365f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): picked mon.noname-b con 0x7f6928097b10 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 0x7f692811f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): picked mon.noname-a con 0x7f692809c700 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:01.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928137750 0x7f692811f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): picked mon.noname-c con 0x7f6928137750 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692809e470 con 0x7f692809c700 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692809bd70 con 0x7f6928097b10 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6928122a70 con 0x7f6928137750 2024-09-18T10:28:01.949 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient(hunting): authenticate will time out at 1970.860322s 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 0x7f692811f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 0x7f692811f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:54086/0 (socket says 172.21.15.142:54086) 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 -- 172.21.15.142:0/1658562952 learned_addr learned my addr 172.21.15.142:0/1658562952 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): get_auth_request con 0x7f692809c700 auth_method 0 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928137750 0x7f692811f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f69281365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.950 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): get_auth_request con 0x7f6928137750 auth_method 0 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): get_auth_request con 0x7f6928097b10 auth_method 0 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.951 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient(hunting): handle_auth_done global_id 214143 payload 824 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928137750 msgr2=0x7f692811f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928137750 0x7f692811f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 msgr2=0x7f69281365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6928097b10 0x7f69281365f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient: _finish_hunting 0 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 monclient: found mon.noname-a 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c009420 con 0x7f692809c700 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 msgr2=0x7f692811f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f69267fc700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 0x7f692811f140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f69280a2a50 tx=0x7f691c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c006af0 con 0x7f692809c700 2024-09-18T10:28:01.952 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 10 monclient: dump: 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 10 monclient: _finish_auth 0 2024-09-18T10:28:01.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 5 monclient: authenticate success, global_id 214143 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c020070 con 0x7f692809c700 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient: _renew_subs 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.933+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69281223f0 con 0x7f692809c700 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c01b450 con 0x7f692809c700 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 10 monclient: dump: 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.954 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 10 monclient: _renew_subs 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6928120c40 con 0x7f692809c700 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6924ff9700 4 set_mon_vals no callback set 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 0x7f692811fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6928136f00 con 0x7f6928136b30 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f691c022030 con 0x7f692809c700 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 0x7f692811fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient: get_auth_request con 0x7f6928136b30 auth_method 0 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6914035ac0 0x7f6914037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 10 monclient(hunting): handle_auth_done global_id 75962 payload 824 2024-09-18T10:28:01.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f6926ffd700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6928064680 con 0x7f6928136b30 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f691c04acf0 con 0x7f692809c700 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f69277fe700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6914035ac0 0x7f6914037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.937+0000 7f69277fe700 10 monclient: get_auth_request con 0x7f6914035ac0 auth_method 0 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f6926ffd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 msgr2=0x7f692811fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f6926ffd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 0x7f692811fbc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f691800eac0 tx=0x7f6918007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691800cc40 con 0x7f6928136b30 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 10 monclient: dump: 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.957 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f691800cda0 con 0x7f6928136b30 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 msgr2=0x7f692811fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 0x7f692811fbc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f691800eac0 tx=0x7f6918007f90 comp rx=0 tx=0).stop 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 msgr2=0x7f692811fbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f690effd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928136b30 0x7f692811fbc0 secure :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f691800eac0 tx=0x7f6918007f90 comp rx=0 tx=0).stop 2024-09-18T10:28:01.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.941+0000 7f69277fe700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6914035ac0 0x7f6914037f70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6920007fd0 tx=0x7f6920002620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f692da01700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f692da01700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 0x7f692811b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f692da01700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6928120c00 con 0x7f6928120100 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 0x7f692811b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient: get_auth_request con 0x7f6928120100 auth_method 0 2024-09-18T10:28:01.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 10 monclient(hunting): handle_auth_done global_id 75965 payload 824 2024-09-18T10:28:01.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.953+0000 7f6926ffd700 1 -- 172.21.15.142:0/1658562952 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6928122ea0 con 0x7f6928120100 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f6926ffd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 msgr2=0x7f692811b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f6926ffd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 0x7f692811b390 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6918007360 tx=0x7f6918016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918013c90 con 0x7f6928120100 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 10 monclient: dump: 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:01.963 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f6918013df0 con 0x7f6928120100 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:01.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 msgr2=0x7f692811b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 0x7f692811b390 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6918007360 tx=0x7f6918016000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 msgr2=0x7f692811b390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f690effd700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6928120100 0x7f692811b390 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:01.965 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.966 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.967 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:01.968 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:01.969 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:01.970 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.971 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.973 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:01.974 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:01.975 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:01.976 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.977 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:01.978 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:01.979 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:01.979 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:01.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6914035ac0 msgr2=0x7f6914037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6914035ac0 0x7f6914037f70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6920007fd0 tx=0x7f6920002620 comp rx=0 tx=0).stop 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 10 monclient: shutdown 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 msgr2=0x7f692811f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 --2- 172.21.15.142:0/1658562952 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f692809c700 0x7f692811f140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f69280a2a50 tx=0x7f691c016000 comp rx=0 tx=0).stop 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f69277fe700 1 -- 172.21.15.142:0/1658562952 reap_dead start 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f69277fe700 1 -- 172.21.15.142:0/1658562952 reap_dead start 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 shutdown_connections 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 >> 172.21.15.142:0/1658562952 conn(0x7f6928071920 msgr2=0x7f692809a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 shutdown_connections 2024-09-18T10:28:01.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:01.957+0000 7f692da01700 1 -- 172.21.15.142:0/1658562952 wait complete. 2024-09-18T10:28:01.981 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:01.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:28:01.982 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: get_monmap_and_config 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: build_initial_monmap 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: monmap: 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.091 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:02.088798+0000 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:02.088798+0000 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:02.092 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 Processor -- start 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 -- start start 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: init 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-18T10:28:02.095 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f168409c2c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-b con 0x7f1684134880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16841371a0 0x7f1684139580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-c con 0x7f16841371a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16840a6620 con 0x7f168409c2c0 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168409ab70 con 0x7f16841371a0 2024-09-18T10:28:02.096 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16840a1c70 con 0x7f1684134880 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1688ea7700 10 monclient(hunting): authenticate will time out at 1971.015786s 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53546/0 (socket says 172.21.15.142:53546) 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.085+0000 7f16837fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f168409c2c0 0x7f16840999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33916/0 (socket says 172.21.15.142:33916) 2024-09-18T10:28:02.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 -- 172.21.15.142:0/2274635810 learned_addr learned my addr 172.21.15.142:0/2274635810 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): get_auth_request con 0x7f1684134880 auth_method 0 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 1 --2- 172.21.15.142:0/2274635810 >> v2:172.21.15.150:3300/0 conn(0x7f16841371a0 0x7f1684139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): get_auth_request con 0x7f16841371a0 auth_method 0 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1683fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient(hunting): handle_auth_done global_id 75968 payload 274 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 -- 172.21.15.142:0/2274635810 >> v2:172.21.15.150:3300/0 conn(0x7f16841371a0 msgr2=0x7f1684139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 --2- 172.21.15.142:0/2274635810 >> v2:172.21.15.150:3300/0 conn(0x7f16841371a0 0x7f1684139580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 -- 172.21.15.142:0/2274635810 >> v1:172.21.15.142:6789/0 conn(0x7f168409c2c0 legacy=0x7f16840999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 monclient: found mon.noname-b 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 -- 172.21.15.142:0/2274635810 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168409c690 con 0x7f1684134880 2024-09-18T10:28:02.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 -- 172.21.15.142:0/2274635810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 msgr2=0x7f1684136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1682ffd700 1 --2- 172.21.15.142:0/2274635810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f166c0099b0 tx=0x7f166c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1484a3604df1f218 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 1 -- 172.21.15.142:0/2274635810 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c009ea0 con 0x7f1684134880 2024-09-18T10:28:02.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: dump: 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.112 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1688ea7700 5 monclient: authenticate success, global_id 75968 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1688ea7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 1 -- 172.21.15.142:0/2274635810 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f166c00ba70 con 0x7f1684134880 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1688ea7700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f1688ea7700 4 set_mon_vals no callback set 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 1 -- 172.21.15.142:0/2274635810 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c00be30 con 0x7f1684134880 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.089+0000 7f16827fc700 10 monclient: dump: 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.113 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: shutdown 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- 172.21.15.142:0/2274635810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 msgr2=0x7f1684136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- 172.21.15.142:0/2274635810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f166c0099b0 tx=0x7f166c009250 comp rx=0 tx=0).stop 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- 172.21.15.142:0/2274635810 shutdown_connections 2024-09-18T10:28:02.114 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- 172.21.15.142:0/2274635810 >> v2:172.21.15.150:3300/0 conn(0x7f16841371a0 0x7f1684139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- 172.21.15.142:0/2274635810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1684134880 0x7f1684136c60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- 172.21.15.142:0/2274635810 >> 172.21.15.142:0/2274635810 conn(0x7f16840a5720 msgr2=0x7f16840a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- 172.21.15.142:0/2274635810 shutdown_connections 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- 172.21.15.142:0/2274635810 wait complete. 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: build_initial_monmap 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: monmap: 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.115 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 Processor -- start 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- start start 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: init 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 0x7f168406c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-c con 0x7f168409c2c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1684134880 0x7f168406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-a con 0x7f1684134880 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16841371a0 0x7f168406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): picked mon.noname-b con 0x7f16841371a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16840a45f0 con 0x7f1684134880 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168412a990 con 0x7f16841371a0 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168412af50 con 0x7f168409c2c0 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1688ea7700 10 monclient(hunting): authenticate will time out at 1971.023859s 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f16837fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 0x7f168406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.093+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1684134880 0x7f168406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1684134880 0x7f168406ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45400/0 (socket says 172.21.15.142:45400) 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 1 -- 172.21.15.142:0/3762503523 learned_addr learned my addr 172.21.15.142:0/3762503523 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:02.117 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): get_auth_request con 0x7f168409c2c0 auth_method 0 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): get_auth_request con 0x7f1684134880 auth_method 0 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16841371a0 0x7f168406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 10 monclient(hunting): get_auth_request con 0x7f16841371a0 auth_method 0 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient(hunting): handle_auth_done global_id 75971 payload 824 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16841371a0 msgr2=0x7f168406d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16841371a0 0x7f168406d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1684134880 msgr2=0x7f168406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1684134880 0x7f168406ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient: _finish_hunting 0 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 monclient: found mon.noname-c 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:02.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c0094f0 con 0x7f168409c2c0 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1683fff700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16841371a0 0x7f168406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 msgr2=0x7f168406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f16837fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 0x7f168406c910 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f167400c530 tx=0x7f1674007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674005c80 con 0x7f168409c2c0 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: dump: 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.120 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 5 monclient: authenticate success, global_id 75971 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1674005e00 con 0x7f168409c2c0 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 10 monclient: _renew_subs 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168412ac70 con 0x7f168409c2c0 2024-09-18T10:28:02.121 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674007320 con 0x7f168409c2c0 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f167b7fe700 10 monclient: dump: 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.122 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1681ffb700 4 set_mon_vals no callback set 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 10 monclient: _renew_subs 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.097+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168412b7d0 con 0x7f168409c2c0 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f167401b3b0 con 0x7f168409c2c0 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1688ea7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1688ea7700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 0x7f168409db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684064750 con 0x7f168406f590 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f167b7fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1664035b10 0x7f1664037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 0x7f168409db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient: get_auth_request con 0x7f168406f590 auth_method 0 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1674048470 con 0x7f168409c2c0 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1683fff700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1664035b10 0x7f1664037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1683fff700 10 monclient: get_auth_request con 0x7f1664035b10 auth_method 0 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 10 monclient(hunting): handle_auth_done global_id 194389 payload 824 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.101+0000 7f1682ffd700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1684072a80 con 0x7f168406f590 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f1682ffd700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 msgr2=0x7f168409db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f1682ffd700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 0x7f168409db10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f166c0030a0 tx=0x7f166c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f1683fff700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1664035b10 0x7f1664037fc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f168406ea40 tx=0x7f167c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c021070 con 0x7f168406f590 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 10 monclient: dump: 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.125 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.126 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f166c01c450 con 0x7f168406f590 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 msgr2=0x7f168409db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 0x7f168409db10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f166c0030a0 tx=0x7f166c016000 comp rx=0 tx=0).stop 2024-09-18T10:28:02.128 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 msgr2=0x7f168409db10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.105+0000 7f167b7fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168406f590 0x7f168409db10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.113+0000 7f1688ea7700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.113+0000 7f1688ea7700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 0x7f168409e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.113+0000 7f1688ea7700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.113+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1684067750 con 0x7f168409e050 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 0x7f168409e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient: get_auth_request con 0x7f168409e050 auth_method 0 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 10 monclient(hunting): handle_auth_done global_id 194392 payload 824 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 1 -- 172.21.15.142:0/3762503523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f168409e990 con 0x7f168409e050 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 msgr2=0x7f168409e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f1682ffd700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 0x7f168409e450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f166c0094c0 tx=0x7f166c01e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c033440 con 0x7f168409e050 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f167b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f167b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.117+0000 7f167b7fe700 10 monclient: dump: 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f166c011d50 con 0x7f168409e050 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 msgr2=0x7f168409e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 0x7f168409e450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f166c0094c0 tx=0x7f166c01e070 comp rx=0 tx=0).stop 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 msgr2=0x7f168409e450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:02.131 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f167b7fe700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168409e050 0x7f168409e450 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 184, 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:02.132 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:02.133 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.134 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:02.135 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:02.136 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.137 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:02.138 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.139 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.140 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.142 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:02.143 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:02.144 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:02.144 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:02.144 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:02.144 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:02.144 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:02.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1664035b10 msgr2=0x7f1664037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1664035b10 0x7f1664037fc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f168406ea40 tx=0x7f167c00b040 comp rx=0 tx=0).stop 2024-09-18T10:28:02.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 10 monclient: shutdown 2024-09-18T10:28:02.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:02.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 msgr2=0x7f168406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 --2- 172.21.15.142:0/3762503523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f168409c2c0 0x7f168406c910 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f167400c530 tx=0x7f1674007ed0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1683fff700 1 -- 172.21.15.142:0/3762503523 reap_dead start 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1683fff700 1 -- 172.21.15.142:0/3762503523 reap_dead start 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 shutdown_connections 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 >> 172.21.15.142:0/3762503523 conn(0x7f16840a5720 msgr2=0x7f16840a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 shutdown_connections 2024-09-18T10:28:02.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.121+0000 7f1688ea7700 1 -- 172.21.15.142:0/3762503523 wait complete. 2024-09-18T10:28:02.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:02.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-18T10:28:02.147 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:28:02.147 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-18T10:28:02.147 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:28:02.147 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:28:02.147 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:28:02.163 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:28:02.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:28:02.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: get_monmap_and_config 2024-09-18T10:28:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: build_initial_monmap 2024-09-18T10:28:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: monmap: 2024-09-18T10:28:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:02.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:02.272024+0000 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:02.272024+0000 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:02.275 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 Processor -- start 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 -- start start 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: init 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 0x7f9604095940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-a con 0x7f960409d710 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:02.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-b con 0x7f9604098860 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96040991a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9604099b60 con 0x7f96040991a0 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960409dae0 con 0x7f960409d710 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960409af70 con 0x7f9604098860 2024-09-18T10:28:02.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f960b2b6700 10 monclient(hunting): authenticate will time out at 1971.197533s 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9609853700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f96040991a0 legacy=0x7f9604135510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9609052700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 msgr2=0x7f9604095940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9609052700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 0x7f9604095940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53564/0 (socket says 172.21.15.142:53564) 2024-09-18T10:28:02.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 -- 172.21.15.142:0/545724422 learned_addr learned my addr 172.21.15.142:0/545724422 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): get_auth_request con 0x7f9604098860 auth_method 0 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient(hunting): handle_auth_done global_id 75974 payload 274 2024-09-18T10:28:02.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 -- 172.21.15.142:0/545724422 >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 msgr2=0x7f9604095940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 --2- 172.21.15.142:0/545724422 >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 0x7f9604095940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 -- 172.21.15.142:0/545724422 >> v1:172.21.15.142:6789/0 conn(0x7f96040991a0 legacy=0x7f9604135510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient: _finish_hunting 0 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 monclient: found mon.noname-b 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.269+0000 7f9608851700 1 -- 172.21.15.142:0/545724422 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040998b0 con 0x7f9604098860 2024-09-18T10:28:02.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f9608851700 1 -- 172.21.15.142:0/545724422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 msgr2=0x7f9604098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f9608851700 1 --2- 172.21.15.142:0/545724422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9600009f20 tx=0x7f9600009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c362cd9a09e4cac server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 1 -- 172.21.15.142:0/545724422 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960000e070 con 0x7f9604098860 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: dump: 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.288 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: _finish_auth 0 2024-09-18T10:28:02.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 1 -- 172.21.15.142:0/545724422 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960000bc90 con 0x7f9604098860 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 5 monclient: authenticate success, global_id 75974 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 1 -- 172.21.15.142:0/545724422 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9600012660 con 0x7f9604098860 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f95fbfff700 10 monclient: dump: 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.290 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 4 set_mon_vals no callback set 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 10 monclient: shutdown 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 -- 172.21.15.142:0/545724422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 msgr2=0x7f9604098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 --2- 172.21.15.142:0/545724422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9600009f20 tx=0x7f9600009840 comp rx=0 tx=0).stop 2024-09-18T10:28:02.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 -- 172.21.15.142:0/545724422 shutdown_connections 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 --2- 172.21.15.142:0/545724422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9604098860 0x7f9604098c60 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 --2- 172.21.15.142:0/545724422 >> v2:172.21.15.142:3300/0 conn(0x7f960409d710 0x7f9604095940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 -- 172.21.15.142:0/545724422 >> 172.21.15.142:0/545724422 conn(0x7f9604071940 msgr2=0x7f96040a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 -- 172.21.15.142:0/545724422 shutdown_connections 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 1 -- 172.21.15.142:0/545724422 wait complete. 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 10 monclient: build_initial_monmap 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.273+0000 7f960b2b6700 10 monclient: monmap: 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 Processor -- start 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 -- start start 2024-09-18T10:28:02.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient: init 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 0x7f9604135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-a con 0x7f9604098860 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96040991a0 0x7f9604135cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-b con 0x7f96040991a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609052700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 msgr2=0x7f9604135770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): picked mon.noname-c con 0x7f960409d710 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604096330 con 0x7f9604098860 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609052700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 0x7f9604135770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604091920 con 0x7f96040991a0 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9604091640 con 0x7f960409d710 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f960b2b6700 10 monclient(hunting): authenticate will time out at 1971.205653s 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53568/0 (socket says 172.21.15.142:53568) 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 -- 172.21.15.142:0/389502739 learned_addr learned my addr 172.21.15.142:0/389502739 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): get_auth_request con 0x7f960409d710 auth_method 0 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96040991a0 0x7f9604135cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): get_auth_request con 0x7f96040991a0 auth_method 0 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9608851700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient(hunting): handle_auth_done global_id 75977 payload 824 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 -- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96040991a0 msgr2=0x7f9604135cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96040991a0 0x7f9604135cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 -- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 msgr2=0x7f9604135770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 0x7f9604135770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient: _finish_hunting 0 2024-09-18T10:28:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 monclient: found mon.noname-c 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.277+0000 7f9609853700 1 -- 172.21.15.142:0/389502739 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9600009ae0 con 0x7f960409d710 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f9609853700 1 -- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 msgr2=0x7f960408dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f9609853700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00b610 tx=0x7f95fc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc003e20 con 0x7f960409d710 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: dump: 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.302 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: _finish_auth 0 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 5 monclient: authenticate success, global_id 75977 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95fc006150 con 0x7f960409d710 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _renew_subs 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9604090460 con 0x7f960409d710 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:02.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95fc006510 con 0x7f960409d710 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 10 monclient: dump: 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:02.304 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95fb7fe700 4 set_mon_vals no callback set 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _renew_subs 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96041361f0 con 0x7f960409d710 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f95fc01b3b0 con 0x7f960409d710 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f95fc048470 con 0x7f960409d710 2024-09-18T10:28:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9604072a80 con 0x7f960409d710 2024-09-18T10:28:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f9609052700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.281+0000 7f9609052700 10 monclient: get_auth_request con 0x7f95e8035a70 auth_method 0 2024-09-18T10:28:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.285+0000 7f9609052700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f95f4006fd0 tx=0x7f95f4006e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:02.289+0000 7f9609853700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00b610 tx=0x7f95fc0090f0 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f95fc017070 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-18T10:28:03.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.121+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f95fc017070 con 0x7f960409d710 2024-09-18T10:28:03.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.121+0000 7f95f97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:03.124 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.121+0000 7f95f97fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:03.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:03.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:03.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:03.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f96040678a0 con 0x7f960409d710 2024-09-18T10:28:03.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f95f97fa700 1 -- 172.21.15.142:0/389502739 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f95fc018840 con 0x7f960409d710 2024-09-18T10:28:03.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f95f97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:03.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f95f97fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:03.351 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:03.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 msgr2=0x7f95e8037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f95f4006fd0 tx=0x7f95f4006e10 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 10 monclient: shutdown 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 msgr2=0x7f960408dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00b610 tx=0x7f95fc0090f0 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 shutdown_connections 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f95e8035a70 0x7f95e8037f20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f960409d710 0x7f960408dd80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96040991a0 0x7f9604135cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 --2- 172.21.15.142:0/389502739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9604098860 0x7f9604135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 >> 172.21.15.142:0/389502739 conn(0x7f9604071940 msgr2=0x7f960409c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:03.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 shutdown_connections 2024-09-18T10:28:03.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:03.345+0000 7f960b2b6700 1 -- 172.21.15.142:0/389502739 wait complete. 2024-09-18T10:28:06.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:06.478 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: get_monmap_and_config 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: build_initial_monmap 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: monmap: 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:06.476493+0000 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:06.476493+0000 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:06.479 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:06.480 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:06.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 Processor -- start 2024-09-18T10:28:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 -- start start 2024-09-18T10:28:06.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: init 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9e809a830 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-b con 0x7ff9e809ac00 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9e813b6e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e8099820 con 0x7ff9e809a830 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e8099d50 con 0x7ff9e813b6e0 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e809afd0 con 0x7ff9e809ac00 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ed470700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7ff9e809a830 legacy=0x7ff9e8098bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:06.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ef6d4700 10 monclient(hunting): authenticate will time out at 1975.402080s 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53582/0 (socket says 172.21.15.142:53582) 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 -- 172.21.15.142:0/739571643 learned_addr learned my addr 172.21.15.142:0/739571643 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): get_auth_request con 0x7ff9e809ac00 auth_method 0 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient(hunting): handle_auth_done global_id 75980 payload 274 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 -- 172.21.15.142:0/739571643 >> v1:172.21.15.150:6789/0 conn(0x7ff9e813b6e0 legacy=0x7ff9e813dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:06.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 -- 172.21.15.142:0/739571643 >> v1:172.21.15.142:6789/0 conn(0x7ff9e809a830 legacy=0x7ff9e8098bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient: _finish_hunting 0 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 monclient: found mon.noname-b 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.473+0000 7ff9ecc6f700 1 -- 172.21.15.142:0/739571643 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e809dcf0 con 0x7ff9e809ac00 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ecc6f700 1 -- 172.21.15.142:0/739571643 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 msgr2=0x7ff9e80992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ecc6f700 1 --2- 172.21.15.142:0/739571643 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80099e0 tx=0x7ff9d8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e9a481fcb819a4b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 1 -- 172.21.15.142:0/739571643 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d800bb50 con 0x7ff9e809ac00 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: dump: 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:06.487 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: _finish_auth 0 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 5 monclient: authenticate success, global_id 75980 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 1 -- 172.21.15.142:0/739571643 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d800bcb0 con 0x7ff9e809ac00 2024-09-18T10:28:06.488 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 4 set_mon_vals no callback set 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 1 -- 172.21.15.142:0/739571643 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8011650 con 0x7ff9e809ac00 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9dffff700 10 monclient: dump: 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:06.489 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 10 monclient: shutdown 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/739571643 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 msgr2=0x7ff9e80992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/739571643 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80099e0 tx=0x7ff9d8009250 comp rx=0 tx=0).stop 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/739571643 shutdown_connections 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/739571643 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e809ac00 0x7ff9e80992e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/739571643 >> 172.21.15.142:0/739571643 conn(0x7ff9e80a14e0 msgr2=0x7ff9e80a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:06.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/739571643 shutdown_connections 2024-09-18T10:28:06.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/739571643 wait complete. 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 10 monclient: build_initial_monmap 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 10 monclient: monmap: 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 Processor -- start 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 1 -- start start 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.477+0000 7ff9ef6d4700 10 monclient: init 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e809a830 0x7ff9e813dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-b con 0x7ff9e809a830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 0x7ff9e80907f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-a con 0x7ff9e809ac00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): picked mon.noname-c con 0x7ff9e813b6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8093350 con 0x7ff9e809ac00 2024-09-18T10:28:06.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e80983d0 con 0x7ff9e809a830 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8091aa0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ef6d4700 10 monclient(hunting): authenticate will time out at 1975.408721s 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ecc6f700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 msgr2=0x7ff9e80907f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ecc6f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 0x7ff9e80907f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53584/0 (socket says 172.21.15.142:53584) 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 -- 172.21.15.142:0/84335705 learned_addr learned my addr 172.21.15.142:0/84335705 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): get_auth_request con 0x7ff9e813b6e0 auth_method 0 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:06.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e809a830 0x7ff9e813dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): get_auth_request con 0x7ff9e809a830 auth_method 0 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9ed470700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient(hunting): handle_auth_done global_id 75983 payload 824 2024-09-18T10:28:06.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 -- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e809a830 msgr2=0x7ff9e813dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e809a830 0x7ff9e813dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 -- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 msgr2=0x7ff9e80907f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 0x7ff9e80907f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient: _finish_hunting 0 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 monclient: found mon.noname-c 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 -- 172.21.15.142:0/84335705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80094f0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 -- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 msgr2=0x7ff9e8090d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.481+0000 7ff9edc71700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c5b0 tx=0x7ff9e4007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40117a0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: dump: 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: _finish_auth 0 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 5 monclient: authenticate success, global_id 75983 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4011900 con 0x7ff9e813b6e0 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _renew_subs 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8092df0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4011cc0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9df7fe700 4 set_mon_vals no callback set 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 10 monclient: dump: 2024-09-18T10:28:06.503 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _renew_subs 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e813e630 con 0x7ff9e813b6e0 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:06.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e80696a0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff9e4012630 con 0x7ff9e813b6e0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 0x7ff9cc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff9e4048bc0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ed470700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 0x7ff9cc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.485+0000 7ff9ed470700 10 monclient: get_auth_request con 0x7ff9cc035a70 auth_method 0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.489+0000 7ff9ed470700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 0x7ff9cc037f20 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009fd0 tx=0x7ff9e0009760 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:06.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.497+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff9e4011e20 con 0x7ff9e813b6e0 2024-09-18T10:28:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.497+0000 7ff9dd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:06.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.497+0000 7ff9dd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:06.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:06.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:06.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:06.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9e8067950 con 0x7ff9e813b6e0 2024-09-18T10:28:06.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9dd7fa700 1 -- 172.21.15.142:0/84335705 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7ff9e40102e0 con 0x7ff9e813b6e0 2024-09-18T10:28:06.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9dd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:06.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9dd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:06.727 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:06.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 msgr2=0x7ff9cc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:06.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 0x7ff9cc037f20 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009fd0 tx=0x7ff9e0009760 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 10 monclient: shutdown 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 msgr2=0x7ff9e8090d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c5b0 tx=0x7ff9e4007f60 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 shutdown_connections 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff9cc035a70 0x7ff9cc037f20 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff9e813b6e0 0x7ff9e8090d30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff9e809ac00 0x7ff9e80907f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 --2- 172.21.15.142:0/84335705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9e809a830 0x7ff9e813dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 >> 172.21.15.142:0/84335705 conn(0x7ff9e80a14e0 msgr2=0x7ff9e8135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:06.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 shutdown_connections 2024-09-18T10:28:06.731 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:06.721+0000 7ff9ef6d4700 1 -- 172.21.15.142:0/84335705 wait complete. 2024-09-18T10:28:09.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: get_monmap_and_config 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: build_initial_monmap 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: monmap: 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:09.852656+0000 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:09.852656+0000 2024-09-18T10:28:09.855 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:09.856 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 Processor -- start 2024-09-18T10:28:09.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 -- start start 2024-09-18T10:28:09.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: init 2024-09-18T10:28:09.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feac409a830 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feac409ac00 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): picked mon.noname-b con 0x7feac40992e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4099d50 con 0x7feac409ac00 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac409dca0 con 0x7feac409a830 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac4099a70 con 0x7feac40992e0 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feaca012700 10 monclient(hunting): authenticate will time out at 1978.778290s 2024-09-18T10:28:09.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac2ffd700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7feac409ac00 legacy=0x7feac413b6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53598/0 (socket says 172.21.15.142:53598) 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 -- 172.21.15.142:0/4219087525 learned_addr learned my addr 172.21.15.142:0/4219087525 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): get_auth_request con 0x7feac40992e0 auth_method 0 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient(hunting): handle_auth_done global_id 75986 payload 274 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 -- 172.21.15.142:0/4219087525 >> v1:172.21.15.150:6789/0 conn(0x7feac409a830 legacy=0x7feac4098bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 -- 172.21.15.142:0/4219087525 >> v1:172.21.15.142:6789/0 conn(0x7feac409ac00 legacy=0x7feac413b6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:09.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 monclient: found mon.noname-b 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.849+0000 7feac3fff700 1 -- 172.21.15.142:0/4219087525 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac409a030 con 0x7feac40992e0 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac3fff700 1 -- 172.21.15.142:0/4219087525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 msgr2=0x7feac413ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:09.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac3fff700 1 --2- 172.21.15.142:0/4219087525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feabc0099e0 tx=0x7feabc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a1a0edca161ef1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 1 -- 172.21.15.142:0/4219087525 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00bb50 con 0x7feac40992e0 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: dump: 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.867 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 5 monclient: authenticate success, global_id 75986 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 1 -- 172.21.15.142:0/4219087525 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc00bcb0 con 0x7feac40992e0 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 4 set_mon_vals no callback set 2024-09-18T10:28:09.868 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 1 -- 172.21.15.142:0/4219087525 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc011650 con 0x7feac40992e0 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feac27fc700 10 monclient: dump: 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:09.869 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 10 monclient: shutdown 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 -- 172.21.15.142:0/4219087525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 msgr2=0x7feac413ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 --2- 172.21.15.142:0/4219087525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feabc0099e0 tx=0x7feabc009250 comp rx=0 tx=0).stop 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 -- 172.21.15.142:0/4219087525 shutdown_connections 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 --2- 172.21.15.142:0/4219087525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac40992e0 0x7feac413ee10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 -- 172.21.15.142:0/4219087525 >> 172.21.15.142:0/4219087525 conn(0x7feac40a14e0 msgr2=0x7feac40a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 -- 172.21.15.142:0/4219087525 shutdown_connections 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 1 -- 172.21.15.142:0/4219087525 wait complete. 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 10 monclient: build_initial_monmap 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.853+0000 7feaca012700 10 monclient: monmap: 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:09.870 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 Processor -- start 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 -- start start 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient: init 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:28:09.871 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac40992e0 0x7feac413c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): picked mon.noname-b con 0x7feac40992e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): picked mon.noname-c con 0x7feac409a830 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 0x7feac413d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): picked mon.noname-a con 0x7feac409ac00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac406b410 con 0x7feac409ac00 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac406ad50 con 0x7feac40992e0 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac4094830 con 0x7feac409a830 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:09.872 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feaca012700 10 monclient(hunting): authenticate will time out at 1978.784965s 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac3fff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 msgr2=0x7feac413d0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 0x7feac413d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:53608/0 (socket says 172.21.15.142:53608) 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 -- 172.21.15.142:0/3547273044 learned_addr learned my addr 172.21.15.142:0/3547273044 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): get_auth_request con 0x7feac409a830 auth_method 0 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac40992e0 0x7feac413c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:09.873 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): get_auth_request con 0x7feac40992e0 auth_method 0 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient(hunting): handle_auth_done global_id 75989 payload 824 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 -- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac40992e0 msgr2=0x7feac413c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac40992e0 0x7feac413c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:09.874 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 -- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 msgr2=0x7feac413d0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 0x7feac413d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 monclient: found mon.noname-c 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.857+0000 7feac2ffd700 1 -- 172.21.15.142:0/3547273044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc0094f0 con 0x7feac409a830 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feac2ffd700 1 -- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 msgr2=0x7feac413cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feac2ffd700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feab400bb80 tx=0x7feab40097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400cb00 con 0x7feac409a830 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: dump: 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:09.875 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 5 monclient: authenticate success, global_id 75989 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _renew_subs 2024-09-18T10:28:09.876 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab400cc60 con 0x7feac409a830 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac40a0680 con 0x7feac409a830 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab400ce10 con 0x7feac409a830 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 10 monclient: dump: 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:09.877 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _renew_subs 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac406f140 con 0x7feac409a830 2024-09-18T10:28:09.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feac1ffb700 4 set_mon_vals no callback set 2024-09-18T10:28:09.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:09.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:09.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:09.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac40696a0 con 0x7feac409a830 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7feab4012020 con 0x7feac409a830 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 0x7feab0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7feab4048d50 con 0x7feac409a830 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feac37fe700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 0x7feab0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.861+0000 7feac37fe700 10 monclient: get_auth_request con 0x7feab0035a70 auth_method 0 2024-09-18T10:28:09.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.865+0000 7feac37fe700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 0x7feab0037f20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7feab8003a30 tx=0x7feab8007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:09.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.873+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7feab401d900 con 0x7feac409a830 2024-09-18T10:28:09.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.873+0000 7feaab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:09.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:09.873+0000 7feaab7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:10.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.089+0000 7feaca012700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:10.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.089+0000 7feaca012700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:10.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.089+0000 7feaca012700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:10.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.089+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feac406b090 con 0x7feac409a830 2024-09-18T10:28:10.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaab7fe700 1 -- 172.21.15.142:0/3547273044 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7feac406b090 con 0x7feac409a830 2024-09-18T10:28:10.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:10.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaab7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:10.099 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:10.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 msgr2=0x7feab0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:10.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 0x7feab0037f20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7feab8003a30 tx=0x7feab8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 10 monclient: shutdown 2024-09-18T10:28:10.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 msgr2=0x7feac413cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feab400bb80 tx=0x7feab40097b0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 shutdown_connections 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feab0035a70 0x7feab0037f20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feac409ac00 0x7feac413d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feac409a830 0x7feac413cba0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 --2- 172.21.15.142:0/3547273044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feac40992e0 0x7feac413c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 >> 172.21.15.142:0/3547273044 conn(0x7feac40a14e0 msgr2=0x7feac409dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 shutdown_connections 2024-09-18T10:28:10.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:10.093+0000 7feaca012700 1 -- 172.21.15.142:0/3547273044 wait complete. 2024-09-18T10:28:13.113 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:13.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 10 monclient: get_monmap_and_config 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 10 monclient: build_initial_monmap 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 10 monclient: monmap: 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:13.222388+0000 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:13.222388+0000 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:13.225 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:13.226 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:13.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 1 Processor -- start 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 1 -- start start 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.217+0000 7f79b9221700 10 monclient: init 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79b409cdd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-b con 0x7f79b4134a50 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79b4137370 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b409d1a0 con 0x7f79b409cdd0 2024-09-18T10:28:13.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b409d780 con 0x7f79b4137370 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b409dd40 con 0x7f79b4134a50 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b9221700 10 monclient(hunting): authenticate will time out at 1982.148326s 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b2d9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f79b409cdd0 legacy=0x7f79b409e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42300/0 (socket says 172.21.15.142:42300) 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 learned_addr learned my addr 172.21.15.142:0/2649969045 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79b4134a50 auth_method 0 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:13.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:13.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:13.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:13.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.221+0000 7f79b359e700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.150:6789/0 conn(0x7f79b4137370 legacy=0x7f79b4139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:13.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.142:6789/0 conn(0x7f79b409cdd0 legacy=0x7f79b409e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79b4134a50 auth_method 0 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:13.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:28:13.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.142:6789/0 conn(0x7f79b409cdd0 legacy=0x7f79b409e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79b4134a50 auth_method 0 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:13.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:13.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:28:14.638 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.142:6789/0 conn(0x7f79b409cdd0 legacy=0x7f79b409e440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79b4134a50 auth_method 0 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:14.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:14.633+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): tick 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 monclient(hunting): continuing hunt 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 msgr2=0x7f79b4136e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b4134a50 0x7f79b4136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:16.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.150:6789/0 conn(0x7f79b4137370 legacy=0x7f79b4139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 -- 172.21.15.142:0/2649969045 >> v1:172.21.15.142:6789/0 conn(0x7f79b409cdd0 legacy=0x7f79b409e440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient: _add_conns ranks=[3,4] 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 0x7f79a0005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): picked mon.noname-a con 0x7f79a00037a0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): picked mon.noname-c con 0x7f79a00062e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 -- 172.21.15.142:0/2649969045 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b409dd40 con 0x7f79a00037a0 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 1 -- 172.21.15.142:0/2649969045 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79a0008e50 con 0x7f79a00062e0 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b0d99700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 msgr2=0x7f79a0005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:16.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b2d9d700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 0x7f79a0005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79a00062e0 auth_method 0 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 msgr2=0x7f79a0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 msgr2=0x7f79a0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:16.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.221+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:16.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 msgr2=0x7f79a0005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:16.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b2d9d700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 0x7f79a0005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79a00062e0 auth_method 0 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 msgr2=0x7f79a0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 msgr2=0x7f79a0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:16.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.425+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:28:16.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b2d9d700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 msgr2=0x7f79a0005bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:16.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b2d9d700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 0x7f79a0005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:28:16.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.150:3300/0 conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79a00062e0 auth_method 0 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:16.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient(hunting): handle_auth_done global_id 194404 payload 274 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 msgr2=0x7f79a0005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> v2:172.21.15.142:3300/0 conn(0x7f79a00037a0 0x7f79a0005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient: _finish_hunting 0 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 monclient: found mon.noname-c 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:16.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b40a6710 con 0x7f79a00062e0 2024-09-18T10:28:16.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79a00062e0 msgr2=0x7f79a0008790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:16.837 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.829+0000 7f79b259c700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79a00062e0 0x7f79a0008790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f79a00062b0 tx=0x7f79a4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa89af1dc06ec8a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:16.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 1 -- 172.21.15.142:0/2649969045 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4018b70 con 0x7f79a00062e0 2024-09-18T10:28:16.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: dump: 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: _finish_auth 0 2024-09-18T10:28:16.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 1 -- 172.21.15.142:0/2649969045 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a400c040 con 0x7f79a00062e0 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 1 -- 172.21.15.142:0/2649969045 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4021440 con 0x7f79a00062e0 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b1d9b700 10 monclient: dump: 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:16.840 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:16.841 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:16.841 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:16.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 5 monclient: authenticate success, global_id 194404 2024-09-18T10:28:16.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:16.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 4 set_mon_vals no callback set 2024-09-18T10:28:16.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 10 monclient: shutdown 2024-09-18T10:28:16.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:16.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 -- 172.21.15.142:0/2649969045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79a00062e0 msgr2=0x7f79a0008790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:16.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79a00062e0 0x7f79a0008790 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f79a00062b0 tx=0x7f79a4008040 comp rx=0 tx=0).stop 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b359e700 1 -- 172.21.15.142:0/2649969045 reap_dead start 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 -- 172.21.15.142:0/2649969045 shutdown_connections 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 --2- 172.21.15.142:0/2649969045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79a00062e0 0x7f79a0008790 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 -- 172.21.15.142:0/2649969045 >> 172.21.15.142:0/2649969045 conn(0x7f79b4071680 msgr2=0x7f79b40a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 -- 172.21.15.142:0/2649969045 shutdown_connections 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 1 -- 172.21.15.142:0/2649969045 wait complete. 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 10 monclient: build_initial_monmap 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.833+0000 7f79b9221700 10 monclient: monmap: 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:16.843 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 Processor -- start 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 -- start start 2024-09-18T10:28:16.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient: init 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 0x7f79b406f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-a con 0x7f79a00062e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-c con 0x7f79b406b780 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:16.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 0x7f79b409b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): picked mon.noname-b con 0x7f79b409b370 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b406c9c0 con 0x7f79a00062e0 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b2d9d700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 msgr2=0x7f79b406f630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b406ac20 con 0x7f79b409b370 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b413a810 con 0x7f79b406b780 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b9221700 10 monclient(hunting): authenticate will time out at 1985.765462s 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b2d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 0x7f79b406f630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:16.848 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42338/0 (socket says 172.21.15.142:42338) 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 -- 172.21.15.142:0/3693692650 learned_addr learned my addr 172.21.15.142:0/3693692650 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): get_auth_request con 0x7f79b406b780 auth_method 0 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 0x7f79b409b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 10 monclient(hunting): get_auth_request con 0x7f79b409b370 auth_method 0 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:16.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient(hunting): handle_auth_done global_id 75992 payload 824 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 -- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 msgr2=0x7f79b409b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 0x7f79b409b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 -- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 msgr2=0x7f79b406f630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 0x7f79b406f630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient: _finish_hunting 0 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 monclient: found mon.noname-c 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b259c700 1 -- 172.21.15.142:0/3693692650 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a40107e0 con 0x7f79b406b780 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.837+0000 7f79b359e700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 0x7f79b409b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b259c700 1 -- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 msgr2=0x7f79b409ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b259c700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006fd0 tx=0x7f79a4021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:16.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4025050 con 0x7f79b406b780 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: dump: 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:16.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: _finish_auth 0 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4026040 con 0x7f79b406b780 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 5 monclient: authenticate success, global_id 75992 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _renew_subs 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a402c620 con 0x7f79b406b780 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b406b310 con 0x7f79b406b780 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 10 monclient: dump: 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:16.852 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b159a700 4 set_mon_vals no callback set 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _renew_subs 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:16.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b406ebd0 con 0x7f79b406b780 2024-09-18T10:28:16.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:16.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:16.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b406ccf0 con 0x7f79b406b780 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.841+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f79a4016400 con 0x7f79b406b780 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.845+0000 7f799affd700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 0x7f79a0038d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.845+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f79a404b5c0 con 0x7f79b406b780 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.845+0000 7f79b2d9d700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 0x7f79a0038d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.845+0000 7f79b2d9d700 10 monclient: get_auth_request con 0x7f79a0036890 auth_method 0 2024-09-18T10:28:16.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.845+0000 7f79b2d9d700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 0x7f79a0038d40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f79a80039d0 tx=0x7f79a80072d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:16.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.849+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f79a402a5f0 con 0x7f79b406b780 2024-09-18T10:28:16.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.849+0000 7f799affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:16.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:16.849+0000 7f799affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:17.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f79b9221700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f79b9221700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f79b9221700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f79b4138ea0 con 0x7f79b406b780 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f799affd700 1 -- 172.21.15.142:0/3693692650 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7f79a4031070 con 0x7f79b406b780 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f799affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:17.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.069+0000 7f799affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:17.078 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 msgr2=0x7f79a0038d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 0x7f79a0038d40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f79a80039d0 tx=0x7f79a80072d0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 10 monclient: shutdown 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 msgr2=0x7f79b409ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006fd0 tx=0x7f79a4021000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 shutdown_connections 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f79a0036890 0x7f79a0038d40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79b409b370 0x7f79b409b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f79b406b780 0x7f79b409ae30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 --2- 172.21.15.142:0/3693692650 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f79a00062e0 0x7f79b406f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 >> 172.21.15.142:0/3693692650 conn(0x7f79b4071680 msgr2=0x7f79b4133f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:17.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 shutdown_connections 2024-09-18T10:28:17.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.073+0000 7f79b9221700 1 -- 172.21.15.142:0/3693692650 wait complete. 2024-09-18T10:28:17.090 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:28:17.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-18T10:28:17.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.185+0000 7f4234a5e700 10 monclient: get_monmap_and_config 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.185+0000 7f4234a5e700 10 monclient: build_initial_monmap 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.185+0000 7f4234a5e700 10 monclient: monmap: 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:17.191331+0000 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:17.191331+0000 2024-09-18T10:28:17.195 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 Processor -- start 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 -- start start 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient: init 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:17.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f423009abd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f423009b740 0x7f4230134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-a con 0x7f423009b740 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4230134bb0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423009dc90 con 0x7f423009abd0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423009afa0 con 0x7f4230134bb0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423009b500 con 0x7f423009b740 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f4234a5e700 10 monclient(hunting): authenticate will time out at 1986.116710s 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422effd700 1 -- >> v2:172.21.15.142:3300/0 conn(0x7f423009b740 msgr2=0x7f4230134670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422f7fe700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f423009abd0 legacy=0x7f42300960e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:17.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f423009b740 0x7f4230134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:17.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422ffff700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4230134bb0 0x7f4230136f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:43092/0 (socket says 172.21.15.142:43092) 2024-09-18T10:28:17.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 learned_addr learned my addr 172.21.15.142:0/2875302225 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:17.208 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 >> v1:172.21.15.150:6789/0 conn(0x7f4230134bb0 legacy=0x7f4230136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496168808 0 0) 0x7f423009afa0 con 0x7f4230134bb0 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422e7fc700 10 monclient(hunting): my global_id is 194410 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.189+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218003570 con 0x7f4230134bb0 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244241876 0 0) 0x7f4218003570 con 0x7f4230134bb0 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 >> v2:172.21.15.142:3300/0 conn(0x7f423009b740 msgr2=0x7f4230134670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 --2- 172.21.15.142:0/2875302225 >> v2:172.21.15.142:3300/0 conn(0x7f423009b740 0x7f4230134670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 >> v1:172.21.15.142:6789/0 conn(0x7f423009abd0 legacy=0x7f42300960e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: _finish_hunting 0 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:28:17.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423006f810 con 0x7f4230134bb0 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 5 monclient: authenticate success, global_id 194410 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7f4228001b20 con 0x7f4230134bb0 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: dump: 2024-09-18T10:28:17.210 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 >> v1:172.21.15.150:6789/0 conn(0x7f4230134bb0 legacy=0x7f4230136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:17.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4218003a60 0x7f4218003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): picked mon.a con 0x7f4218003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218004810 0x7f4218004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): picked mon.b con 0x7f4218004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 0x7f421800fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): picked mon.c con 0x7f4218005410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218010780 con 0x7f4218003a60 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180108f0 con 0x7f4218004810 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4218010ab0 con 0x7f4218005410 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:17.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422f7fe700 1 -- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4218003a60 msgr2=0x7f4218003e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422f7fe700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4218003a60 0x7f4218003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 0x7f421800fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): get_auth_request con 0x7f4218005410 auth_method 0 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422effd700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218004810 0x7f4218004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422effd700 10 monclient(hunting): get_auth_request con 0x7f4218004810 auth_method 0 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422effd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient(hunting): handle_auth_done global_id 194410 payload 293 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218004810 msgr2=0x7f4218004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218004810 0x7f4218004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4218003a60 msgr2=0x7f4218003e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4218003a60 0x7f4218003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 monclient: found mon.c 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:17.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218003570 con 0x7f4218005410 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 10 monclient: _finish_auth 0 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 reap_dead start 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 -- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 msgr2=0x7f421800fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422ffff700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 0x7f421800fed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4228001d00 tx=0x7f422800a240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cebfbee1e5ec2d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800c5a0 con 0x7f4218005410 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: dump: 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.215 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422800c700 con 0x7f4218005410 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f4234a5e700 4 set_mon_vals no callback set 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 1 -- 172.21.15.142:0/2875302225 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422800cac0 con 0x7f4218005410 2024-09-18T10:28:17.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.193+0000 7f422e7fc700 10 monclient: dump: 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: shutdown 2024-09-18T10:28:17.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:17.218 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 msgr2=0x7f421800fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.220 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 0x7f421800fed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4228001d00 tx=0x7f422800a240 comp rx=0 tx=0).stop 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- 172.21.15.142:0/2875302225 shutdown_connections 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 --2- 172.21.15.142:0/2875302225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4218005410 0x7f421800fed0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- 172.21.15.142:0/2875302225 >> 172.21.15.142:0/2875302225 conn(0x7f4230070ef0 msgr2=0x7f42300712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- 172.21.15.142:0/2875302225 shutdown_connections 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- 172.21.15.142:0/2875302225 wait complete. 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: build_initial_monmap 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: monmap: 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:17.221 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 Processor -- start 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- start start 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: init 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218005410 0x7f42300942d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-b con 0x7f4218005410 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.222 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-c con 0x7f4230094810 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4230094be0 0x7f423006c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): picked mon.noname-a con 0x7f4230094be0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42301306a0 con 0x7f4230094be0 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423009d240 con 0x7f4218005410 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423006e5d0 con 0x7f4230094810 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f4234a5e700 10 monclient(hunting): authenticate will time out at 1986.125494s 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422ffff700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4230094be0 msgr2=0x7f423006c8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4230094be0 0x7f423006c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.223 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42364/0 (socket says 172.21.15.142:42364) 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 -- 172.21.15.142:0/3378351914 learned_addr learned my addr 172.21.15.142:0/3378351914 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): get_auth_request con 0x7f4230094810 auth_method 0 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218005410 0x7f42300942d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient(hunting): handle_auth_done global_id 75995 payload 824 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218005410 msgr2=0x7f42300942d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.224 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218005410 0x7f42300942d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4230094be0 msgr2=0x7f423006c8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4230094be0 0x7f423006c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 monclient: found mon.noname-c 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422effd700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230126a00 con 0x7f4230094810 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422f7fe700 10 monclient: get_auth_request con 0x7f4218005410 auth_method 0 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.197+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4218005410 0x7f42300942d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422effd700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 msgr2=0x7f423006c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422effd700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f422400d800 tx=0x7f4224008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.225 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240050d0 con 0x7f4230094810 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: dump: 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.226 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422400f040 con 0x7f4230094810 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 5 monclient: authenticate success, global_id 75995 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240054c0 con 0x7f4230094810 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 monclient: _renew_subs 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422dffb700 4 set_mon_vals no callback set 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4230126ce0 con 0x7f4230094810 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 10 monclient: dump: 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.227 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 monclient: _renew_subs 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423006ea50 con 0x7f4230094810 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:17.228 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 0x7f423006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42300648a0 con 0x7f423006ce10 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f422401c400 con 0x7f4230094810 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 0x7f423006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient: get_auth_request con 0x7f423006ce10 auth_method 0 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f421c035aa0 0x7f421c037f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4224048a00 con 0x7f4230094810 2024-09-18T10:28:17.229 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 10 monclient(hunting): handle_auth_done global_id 75998 payload 824 2024-09-18T10:28:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422f7fe700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4230127050 con 0x7f423006ce10 2024-09-18T10:28:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422ffff700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f421c035aa0 0x7f421c037f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.230 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.201+0000 7f422ffff700 10 monclient: get_auth_request con 0x7f421c035aa0 auth_method 0 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f422f7fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 msgr2=0x7f423006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 0x7f423006d1e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f42200039d0 tx=0x7f4220008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220003e70 con 0x7f423006ce10 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 10 monclient: dump: 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.233 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f422ffff700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f421c035aa0 0x7f421c037f50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4218010c40 tx=0x7f4228016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4220006150 con 0x7f423006ce10 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 msgr2=0x7f423006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 0x7f423006d1e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f42200039d0 tx=0x7f4220008040 comp rx=0 tx=0).stop 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 msgr2=0x7f423006d1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.205+0000 7f42177fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f423006ce10 0x7f423006d1e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.234 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f4234a5e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f4234a5e700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 0x7f4230132c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f4234a5e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4230072a80 con 0x7f4230132870 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 0x7f4230132c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f422f7fe700 10 monclient: get_auth_request con 0x7f4230132870 auth_method 0 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f422f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f422f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.213+0000 7f422f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 10 monclient(hunting): handle_auth_done global_id 76001 payload 824 2024-09-18T10:28:17.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 1 -- 172.21.15.142:0/3378351914 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f423006d720 con 0x7f4230132870 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 msgr2=0x7f4230132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f422f7fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 0x7f4230132c70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4220003fd0 tx=0x7f4220017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220024070 con 0x7f4230132870 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 10 monclient: dump: 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.236 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f4220003c60 con 0x7f4230132870 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 msgr2=0x7f4230132c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 0x7f4230132c70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4220003fd0 tx=0x7f4220017000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 msgr2=0x7f4230132c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:17.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.217+0000 7f42177fe700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230132870 0x7f4230132c70 secure :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4220003fd0 tx=0x7f4220017000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 186, 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:17.238 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.239 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.240 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:17.241 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.242 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.243 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.244 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:17.245 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.246 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:17.247 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.248 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.249 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 7 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:17.250 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f421c035aa0 msgr2=0x7f421c037f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f421c035aa0 0x7f421c037f50 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4218010c40 tx=0x7f4228016000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 10 monclient: shutdown 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f422ffff700 1 -- 172.21.15.142:0/3378351914 reap_dead start 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 msgr2=0x7f423006c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f422400d800 tx=0x7f4224008040 comp rx=0 tx=0).stop 2024-09-18T10:28:17.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 shutdown_connections 2024-09-18T10:28:17.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 --2- 172.21.15.142:0/3378351914 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4230094810 0x7f423006c390 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 >> 172.21.15.142:0/3378351914 conn(0x7f4230070ef0 msgr2=0x7f42300996f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:17.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 shutdown_connections 2024-09-18T10:28:17.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.221+0000 7f4234a5e700 1 -- 172.21.15.142:0/3378351914 wait complete. 2024-09-18T10:28:17.253 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:28:17.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:28:17.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: get_monmap_and_config 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: build_initial_monmap 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: monmap: 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:17.358605+0000 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:17.358605+0000 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:17.361 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:17.362 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:17.362 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:17.362 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:17.362 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 1 Processor -- start 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 1 -- start start 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: init 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.353+0000 7f6374970700 10 monclient(hunting): picked mon.noname-b con 0x7f63700975d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f637006f8f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6370135520 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6370098f50 con 0x7f6370135520 2024-09-18T10:28:17.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63700979a0 con 0x7f637006f8f0 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6370099610 con 0x7f63700975d0 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient(hunting): authenticate will time out at 1986.284064s 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636ed9d700 1 -- >> v1:172.21.15.142:6789/0 conn(0x7f6370135520 legacy=0x7f6370137900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.142:6789/0 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42396/0 (socket says 172.21.15.142:42396) 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 -- 172.21.15.142:0/3935409291 learned_addr learned my addr 172.21.15.142:0/3935409291 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): get_auth_request con 0x7f63700975d0 auth_method 0 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient(hunting): handle_auth_done global_id 76004 payload 274 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 -- 172.21.15.142:0/3935409291 >> v1:172.21.15.150:6789/0 conn(0x7f637006f8f0 legacy=0x7f637009c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 -- 172.21.15.142:0/3935409291 >> v1:172.21.15.142:6789/0 conn(0x7f6370135520 legacy=0x7f6370137900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 monclient: found mon.noname-b 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 -- 172.21.15.142:0/3935409291 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637009d080 con 0x7f63700975d0 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 -- 172.21.15.142:0/3935409291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 msgr2=0x7f637009bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636e59c700 1 --2- 172.21.15.142:0/3935409291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f63640099e0 tx=0x7f6364009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc491f05d806dea7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 1 -- 172.21.15.142:0/3935409291 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636400baf0 con 0x7f63700975d0 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: dump: 2024-09-18T10:28:17.368 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: _finish_auth 0 2024-09-18T10:28:17.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 5 monclient: authenticate success, global_id 76004 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 1 -- 172.21.15.142:0/3935409291 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636400bc50 con 0x7f63700975d0 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 1 -- 172.21.15.142:0/3935409291 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364011420 con 0x7f63700975d0 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f6374970700 4 set_mon_vals no callback set 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.357+0000 7f636d59a700 10 monclient: dump: 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.370 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: shutdown 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- 172.21.15.142:0/3935409291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 msgr2=0x7f637009bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 --2- 172.21.15.142:0/3935409291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f63640099e0 tx=0x7f6364009250 comp rx=0 tx=0).stop 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- 172.21.15.142:0/3935409291 shutdown_connections 2024-09-18T10:28:17.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 --2- 172.21.15.142:0/3935409291 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f63700975d0 0x7f637009bd70 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- 172.21.15.142:0/3935409291 >> 172.21.15.142:0/3935409291 conn(0x7f6370071ae0 msgr2=0x7f63700a1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- 172.21.15.142:0/3935409291 shutdown_connections 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- 172.21.15.142:0/3935409291 wait complete. 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: build_initial_monmap 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: monmap: 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:17.372 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:17.373 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 Processor -- start 2024-09-18T10:28:17.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- start start 2024-09-18T10:28:17.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: init 2024-09-18T10:28:17.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f637006f8f0 0x7f6370134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): picked mon.noname-a con 0x7f637006f8f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63700975d0 0x7f6370093f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): picked mon.noname-b con 0x7f63700975d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 0x7f637008bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): picked mon.noname-c con 0x7f6370135520 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637009b780 con 0x7f637006f8f0 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637009fed0 con 0x7f63700975d0 2024-09-18T10:28:17.378 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637008dd60 con 0x7f6370135520 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636e59c700 1 -- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f637006f8f0 msgr2=0x7f6370134740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3300/0 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f6374970700 10 monclient(hunting): authenticate will time out at 1986.291029s 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f637006f8f0 0x7f6370134740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 0x7f637008bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 0x7f637008bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:42400/0 (socket says 172.21.15.142:42400) 2024-09-18T10:28:17.379 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 learned_addr learned my addr 172.21.15.142:0/4125712513 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): get_auth_request con 0x7f6370135520 auth_method 0 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636dd9b700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63700975d0 0x7f6370093f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636dd9b700 10 monclient(hunting): get_auth_request con 0x7f63700975d0 auth_method 0 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.380 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient(hunting): handle_auth_done global_id 76007 payload 824 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63700975d0 msgr2=0x7f6370093f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63700975d0 0x7f6370093f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f637006f8f0 msgr2=0x7f6370134740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f637006f8f0 0x7f6370134740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.361+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640094f0 con 0x7f6370135520 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 msgr2=0x7f637008bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636ed9d700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 0x7f637008bb90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f636800bf80 tx=0x7f6368009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368016430 con 0x7f6370135520 2024-09-18T10:28:17.381 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: dump: 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63680165b0 con 0x7f6370135520 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 5 monclient: authenticate success, global_id 76007 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 monclient: _renew_subs 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63701352f0 con 0x7f6370135520 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6368016760 con 0x7f6370135520 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 10 monclient: dump: 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.383 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636cd99700 4 set_mon_vals no callback set 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 monclient: _renew_subs 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637008d5f0 con 0x7f6370135520 2024-09-18T10:28:17.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 0x7f637008c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63700648a0 con 0x7f6370094460 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 0x7f637008c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient: get_auth_request con 0x7f6370094460 auth_method 0 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6368012020 con 0x7f6370135520 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.365+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6350035ac0 0x7f6350037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636e59c700 10 monclient(hunting): handle_auth_done global_id 194416 payload 824 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636e59c700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f637008f490 con 0x7f6370094460 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6368049090 con 0x7f6370135520 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636dd9b700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6350035ac0 0x7f6350037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636dd9b700 10 monclient: get_auth_request con 0x7f6350035ac0 auth_method 0 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636e59c700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 msgr2=0x7f637008c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636e59c700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 0x7f637008c0d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6364015000 tx=0x7f6364016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636400baf0 con 0x7f6370094460 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 10 monclient: dump: 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.386 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f636dd9b700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6350035ac0 0x7f6350037f70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6360003870 tx=0x7f6360009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f636401c450 con 0x7f6370094460 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.369+0000 7f635e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.373+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 msgr2=0x7f637008c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.373+0000 7f635e7fc700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 0x7f637008c0d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6364015000 tx=0x7f6364016000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.373+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 msgr2=0x7f637008c0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:17.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.373+0000 7f635e7fc700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6370094460 0x7f637008c0d0 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6364015000 tx=0x7f6364016000 comp rx=0 tx=0).stop 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f6374970700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f6374970700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 0x7f6370087fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f6374970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6370094830 con 0x7f637008c610 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 0x7f6370087fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient: get_auth_request con 0x7f637008c610 auth_method 0 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:17.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:17.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:17.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:17.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:17.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 10 monclient(hunting): handle_auth_done global_id 194419 payload 824 2024-09-18T10:28:17.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.381+0000 7f636e59c700 1 -- 172.21.15.142:0/4125712513 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f637008ca10 con 0x7f637008c610 2024-09-18T10:28:17.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f636e59c700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 msgr2=0x7f6370087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:17.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f636e59c700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 0x7f6370087fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63640099e0 tx=0x7f636401f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:17.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364026aa0 con 0x7f637008c610 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 10 monclient: dump: 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:17.392 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4399 (secure 0 0 0) 0x7f636402c070 con 0x7f637008c610 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 msgr2=0x7f6370087fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 0x7f6370087fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63640099e0 tx=0x7f636401f040 comp rx=0 tx=0).stop 2024-09-18T10:28:17.393 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 msgr2=0x7f6370087fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 186, 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:17.394 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.395 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.396 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:17.397 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:17.398 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:17.399 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.400 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.402 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:17.403 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:17.404 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:17.405 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:17.406 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:17.406 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:17.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f635e7fc700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f637008c610 0x7f6370087fe0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63640099e0 tx=0x7f636401f040 comp rx=0 tx=0).stop 2024-09-18T10:28:17.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6350035ac0 msgr2=0x7f6350037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6350035ac0 0x7f6350037f70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6360003870 tx=0x7f6360009040 comp rx=0 tx=0).stop 2024-09-18T10:28:17.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 10 monclient: shutdown 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 msgr2=0x7f637008bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 --2- 172.21.15.142:0/4125712513 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6370135520 0x7f637008bb90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f636800bf80 tx=0x7f6368009730 comp rx=0 tx=0).stop 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 reap_dead start 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f636ed9d700 1 -- 172.21.15.142:0/4125712513 reap_dead start 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 shutdown_connections 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 >> 172.21.15.142:0/4125712513 conn(0x7f6370071ae0 msgr2=0x7f637009a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 shutdown_connections 2024-09-18T10:28:17.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:17.385+0000 7f6374970700 1 -- 172.21.15.142:0/4125712513 wait complete. 2024-09-18T10:28:17.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:28:17.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:28:37.409 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-18T10:28:37.409 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-18T10:28:37.409 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T10:28:37.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T10:28:37.412 INFO:tasks.ceph.mon.a:Started 2024-09-18T10:28:37.412 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:28:37.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: get_monmap_and_config 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: build_initial_monmap 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: monmap: 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:37.516942+0000 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:37.516942+0000 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:37.520 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.521 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:37.521 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:37.521 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:37.521 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:37.522 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 Processor -- start 2024-09-18T10:28:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 -- start start 2024-09-18T10:28:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: init 2024-09-18T10:28:37.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:37.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-a con 0x7f839009bed0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f839006f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f839009c7e0 0x7f83901386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-c con 0x7f839009c7e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8390098eb0 con 0x7f839009bed0 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839006f600 con 0x7f839009c7e0 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839006f770 con 0x7f839006f230 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8397ea6700 10 monclient(hunting): authenticate will time out at 2006.442456s 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49524/0 (socket says 172.21.15.142:49524) 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38258/0 (socket says 172.21.15.142:38258) 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 -- 172.21.15.142:0/2369391723 learned_addr learned my addr 172.21.15.142:0/2369391723 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): get_auth_request con 0x7f839006f230 auth_method 0 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 10 monclient(hunting): get_auth_request con 0x7f839009bed0 auth_method 0 2024-09-18T10:28:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.150:3300/0 conn(0x7f839009c7e0 0x7f83901386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): get_auth_request con 0x7f839009c7e0 auth_method 0 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 -- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 msgr2=0x7f83900991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 -- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 msgr2=0x7f83900991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8396443700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395c42700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient(hunting): handle_auth_done global_id 76010 payload 274 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 -- 172.21.15.142:0/2369391723 >> v2:172.21.15.150:3300/0 conn(0x7f839009c7e0 msgr2=0x7f83901386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.150:3300/0 conn(0x7f839009c7e0 0x7f83901386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 -- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 msgr2=0x7f83900991d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient: _finish_hunting 0 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 monclient: found mon.noname-b 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.512+0000 7f8395441700 1 -- 172.21.15.142:0/2369391723 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390098ad0 con 0x7f839006f230 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8395441700 1 -- 172.21.15.142:0/2369391723 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 msgr2=0x7f839009c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8395441700 1 --2- 172.21.15.142:0/2369391723 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f839006a8a0 tx=0x7f8384009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce2e3bfd4213321c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 1 -- 172.21.15.142:0/2369391723 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384009e70 con 0x7f839006f230 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: dump: 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: _finish_auth 0 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 5 monclient: authenticate success, global_id 76010 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 1 -- 172.21.15.142:0/2369391723 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400ba70 con 0x7f839006f230 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 1 -- 172.21.15.142:0/2369391723 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838400be30 con 0x7f839006f230 2024-09-18T10:28:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8394c40700 10 monclient: dump: 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 4 set_mon_vals no callback set 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 10 monclient: shutdown 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 -- 172.21.15.142:0/2369391723 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 msgr2=0x7f839009c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 --2- 172.21.15.142:0/2369391723 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f839006a8a0 tx=0x7f8384009250 comp rx=0 tx=0).stop 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 -- 172.21.15.142:0/2369391723 shutdown_connections 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.150:3300/0 conn(0x7f839009c7e0 0x7f83901386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 --2- 172.21.15.142:0/2369391723 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f839009c2a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 --2- 172.21.15.142:0/2369391723 >> v2:172.21.15.142:3300/0 conn(0x7f839009bed0 0x7f83900991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 -- 172.21.15.142:0/2369391723 >> 172.21.15.142:0/2369391723 conn(0x7f8390071820 msgr2=0x7f83900a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 -- 172.21.15.142:0/2369391723 shutdown_connections 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 1 -- 172.21.15.142:0/2369391723 wait complete. 2024-09-18T10:28:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 10 monclient: build_initial_monmap 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.516+0000 7f8397ea6700 10 monclient: monmap: 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 Processor -- start 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 -- start start 2024-09-18T10:28:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient: init 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f8390141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-c con 0x7f839006f230 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f839009bed0 0x7f8390141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-b con 0x7f839009bed0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): picked mon.noname-a con 0x7f839009c7e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839008fe50 con 0x7f839009c7e0 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839009d550 con 0x7f839009bed0 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83900a1050 con 0x7f839006f230 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8397ea6700 10 monclient(hunting): authenticate will time out at 2006.449331s 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38264/0 (socket says 172.21.15.142:38264) 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 -- 172.21.15.142:0/1519573819 learned_addr learned my addr 172.21.15.142:0/1519573819 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 10 monclient(hunting): get_auth_request con 0x7f839009c7e0 auth_method 0 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f8390141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): get_auth_request con 0x7f839006f230 auth_method 0 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 msgr2=0x7f839008dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 msgr2=0x7f839008dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8396443700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f839009bed0 0x7f8390141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): get_auth_request con 0x7f839009bed0 auth_method 0 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395441700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient(hunting): handle_auth_done global_id 76013 payload 824 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f839009bed0 msgr2=0x7f8390141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f839009bed0 0x7f8390141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 msgr2=0x7f839008dd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient: _finish_hunting 0 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 monclient: found mon.noname-c 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.520+0000 7f8395c42700 1 -- 172.21.15.142:0/1519573819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840094f0 con 0x7f839006f230 2024-09-18T10:28:37.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8395c42700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 msgr2=0x7f8390141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8395c42700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f8390141990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f838c00d8c0 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c012800 con 0x7f839006f230 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: dump: 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c012980 con 0x7f839006f230 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 5 monclient: authenticate success, global_id 76013 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _renew_subs 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83900910c0 con 0x7f839006f230 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c012d40 con 0x7f839006f230 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 10 monclient: dump: 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8383fff700 4 set_mon_vals no callback set 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _renew_subs 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8390091460 con 0x7f839006f230 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8390090510 con 0x7f839006f230 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f838c014640 con 0x7f839006f230 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.524+0000 7f8381ffb700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 0x7f8374037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.528+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f838c0494a0 con 0x7f839006f230 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.528+0000 7f8395441700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 0x7f8374037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.528+0000 7f8395441700 10 monclient: get_auth_request con 0x7f8374035ac0 auth_method 0 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.528+0000 7f8395441700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 0x7f8374037f70 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f839006a8a0 tx=0x7f8384017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.536+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f838c00f310 con 0x7f839006f230 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.536+0000 7f8381ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:37.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.536+0000 7f8381ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:37.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.756+0000 7f8397ea6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:37.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.756+0000 7f8397ea6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:37.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.756+0000 7f8397ea6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:37.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.756+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f83900696a0 con 0x7f839006f230 2024-09-18T10:28:37.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8381ffb700 1 -- 172.21.15.142:0/1519573819 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f838c011340 con 0x7f839006f230 2024-09-18T10:28:37.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8381ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:37.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8381ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:37.766 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 msgr2=0x7f8374037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 0x7f8374037f70 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f839006a8a0 tx=0x7f8384017000 comp rx=0 tx=0).stop 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 10 monclient: shutdown 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 msgr2=0x7f8390141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f8390141990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f838c00d8c0 tx=0x7f838c008040 comp rx=0 tx=0).stop 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 shutdown_connections 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8374035ac0 0x7f8374037f70 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f839009c7e0 0x7f839008dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f839009bed0 0x7f8390141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 --2- 172.21.15.142:0/1519573819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f839006f230 0x7f8390141990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:37.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 >> 172.21.15.142:0/1519573819 conn(0x7f8390071820 msgr2=0x7f839009f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:37.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 shutdown_connections 2024-09-18T10:28:37.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:37.760+0000 7f8397ea6700 1 -- 172.21.15.142:0/1519573819 wait complete. 2024-09-18T10:28:40.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.888+0000 7fd726a38700 10 monclient: get_monmap_and_config 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.888+0000 7fd726a38700 10 monclient: build_initial_monmap 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient: monmap: 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:40.895739+0000 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:40.895739+0000 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.898 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 Processor -- start 2024-09-18T10:28:40.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 -- start start 2024-09-18T10:28:40.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient: init 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-a con 0x7fd7200a2700 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd720097ad0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd72009e290 0x7fd7201364b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-c con 0x7fd72009e290 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72009b910 con 0x7fd720097ad0 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:40.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd720098730 con 0x7fd7200a2700 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7200a2ad0 con 0x7fd72009e290 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726a38700 10 monclient(hunting): authenticate will time out at 2009.822976s 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38274/0 (socket says 172.21.15.142:38274) 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 -- 172.21.15.142:0/3668298846 learned_addr learned my addr 172.21.15.142:0/3668298846 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 10 monclient(hunting): get_auth_request con 0x7fd7200a2700 auth_method 0 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725235700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd720097ad0 0x7fd720096f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47306/0 (socket says 172.21.15.142:47306) 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:40.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 -- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 msgr2=0x7fd720096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 -- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 msgr2=0x7fd720096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 --2- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd725a36700 1 --2- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 1 --2- 172.21.15.142:0/3668298846 >> v2:172.21.15.150:3300/0 conn(0x7fd72009e290 0x7fd7201364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): get_auth_request con 0x7fd72009e290 auth_method 0 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:40.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.892+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 10 monclient(hunting): handle_auth_done global_id 194428 payload 274 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 -- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 msgr2=0x7fd720096a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 --2- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 -- 172.21.15.142:0/3668298846 >> v1:172.21.15.142:6789/0 conn(0x7fd720097ad0 legacy=0x7fd720096f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 10 monclient: _finish_hunting 0 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 monclient: found mon.noname-c 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:40.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 -- 172.21.15.142:0/3668298846 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72009d9c0 con 0x7fd72009e290 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 -- 172.21.15.142:0/3668298846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72009e290 msgr2=0x7fd7201364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726237700 1 --2- 172.21.15.142:0/3668298846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72009e290 0x7fd7201364b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd72009dfe0 tx=0x7fd718009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=821ee7856d6a4c75 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 1 -- 172.21.15.142:0/3668298846 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71800e070 con 0x7fd72009e290 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: dump: 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.909 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: _finish_auth 0 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 1 -- 172.21.15.142:0/3668298846 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71800be40 con 0x7fd72009e290 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 1 -- 172.21.15.142:0/3668298846 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd718012660 con 0x7fd72009e290 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:40.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd724a34700 10 monclient: dump: 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726a38700 5 monclient: authenticate success, global_id 194428 2024-09-18T10:28:40.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726a38700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.896+0000 7fd726a38700 4 set_mon_vals no callback set 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: shutdown 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- 172.21.15.142:0/3668298846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72009e290 msgr2=0x7fd7201364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- 172.21.15.142:0/3668298846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72009e290 0x7fd7201364b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd72009dfe0 tx=0x7fd718009780 comp rx=0 tx=0).stop 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- 172.21.15.142:0/3668298846 shutdown_connections 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- 172.21.15.142:0/3668298846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd72009e290 0x7fd7201364b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- 172.21.15.142:0/3668298846 >> v2:172.21.15.142:3300/0 conn(0x7fd7200a2700 0x7fd720096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- 172.21.15.142:0/3668298846 >> 172.21.15.142:0/3668298846 conn(0x7fd720071700 msgr2=0x7fd7200a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- 172.21.15.142:0/3668298846 shutdown_connections 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- 172.21.15.142:0/3668298846 wait complete. 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: build_initial_monmap 2024-09-18T10:28:40.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: monmap: 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 Processor -- start 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- start start 2024-09-18T10:28:40.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: init 2024-09-18T10:28:40.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:40.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:28:40.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-c con 0x7fd720097ad0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:40.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-a con 0x7fd72009e290 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7200a2700 0x7fd72013ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): picked mon.noname-b con 0x7fd7200a2700 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72011e890 con 0x7fd72009e290 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72011e2d0 con 0x7fd7200a2700 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.900+0000 7fd726a38700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72011dff0 con 0x7fd720097ad0 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient(hunting): authenticate will time out at 2009.832066s 2024-09-18T10:28:40.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:49568/0 (socket says 172.21.15.142:49568) 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 -- 172.21.15.142:0/1945333804 learned_addr learned my addr 172.21.15.142:0/1945333804 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): get_auth_request con 0x7fd720097ad0 auth_method 0 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 10 monclient(hunting): get_auth_request con 0x7fd72009e290 auth_method 0 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:40.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 msgr2=0x7fd720138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 msgr2=0x7fd720138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7200a2700 0x7fd72013ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725235700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): get_auth_request con 0x7fd7200a2700 auth_method 0 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726237700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:40.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient(hunting): handle_auth_done global_id 76016 payload 824 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7200a2700 msgr2=0x7fd72013ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7200a2700 0x7fd72013ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 msgr2=0x7fd720138f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient: _finish_hunting 0 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 monclient: found mon.noname-c 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 -- 172.21.15.142:0/1945333804 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd718009a20 con 0x7fd720097ad0 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 msgr2=0x7fd7201398a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd725a36700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd71c010e60 tx=0x7fd71c00b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c019440 con 0x7fd720097ad0 2024-09-18T10:28:40.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c0195a0 con 0x7fd720097ad0 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 5 monclient: authenticate success, global_id 76016 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient: _renew_subs 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72011e5b0 con 0x7fd720097ad0 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c019960 con 0x7fd720097ad0 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd715ffb700 10 monclient: dump: 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:40.924 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient: _renew_subs 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd7177fe700 4 set_mon_vals no callback set 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.904+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd720139490 con 0x7fd720097ad0 2024-09-18T10:28:40.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd726a38700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd726a38700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd726a38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd720072a80 con 0x7fd720097ad0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd71c019d50 con 0x7fd720097ad0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd715ffb700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 0x7fd704037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd71c04bae0 con 0x7fd720097ad0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd725235700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 0x7fd704037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.908+0000 7fd725235700 10 monclient: get_auth_request con 0x7fd704035ac0 auth_method 0 2024-09-18T10:28:40.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.912+0000 7fd725235700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 0x7fd704037f70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd710007fd0 tx=0x7fd710009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:40.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.916+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd71c0183a0 con 0x7fd720097ad0 2024-09-18T10:28:40.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.916+0000 7fd715ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:40.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:40.916+0000 7fd715ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:41.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd726a38700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd726a38700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd726a38700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd720068360 con 0x7fd720097ad0 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd715ffb700 1 -- 172.21.15.142:0/1945333804 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fd71c020310 con 0x7fd720097ad0 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd715ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:41.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.140+0000 7fd715ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:41.149 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 msgr2=0x7fd704037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 0x7fd704037f70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fd710007fd0 tx=0x7fd710009040 comp rx=0 tx=0).stop 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 10 monclient: shutdown 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 msgr2=0x7fd7201398a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd71c010e60 tx=0x7fd71c00b650 comp rx=0 tx=0).stop 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 shutdown_connections 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd704035ac0 0x7fd704037f70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd7200a2700 0x7fd72013ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:41.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd72009e290 0x7fd720138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:41.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 --2- 172.21.15.142:0/1945333804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd720097ad0 0x7fd7201398a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:41.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 >> 172.21.15.142:0/1945333804 conn(0x7fd720071700 msgr2=0x7fd7200a07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:41.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 shutdown_connections 2024-09-18T10:28:41.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:41.144+0000 7fd726a38700 1 -- 172.21.15.142:0/1945333804 wait complete. 2024-09-18T10:28:44.162 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:44.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: get_monmap_and_config 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: monmap: 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:44.274304+0000 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:44.274304+0000 2024-09-18T10:28:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:44.278 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 Processor -- start 2024-09-18T10:28:44.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 -- start start 2024-09-18T10:28:44.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: init 2024-09-18T10:28:44.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:44.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb3d806e860 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3d806f460 0x7fb3d809c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb3d806f460 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3d809c860 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d806f220 con 0x7fb3d809c860 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d8099780 con 0x7fb3d806e860 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:44.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d80998c0 con 0x7fb3d806f460 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d7fff700 10 monclient(hunting): authenticate will time out at 2013.199762s 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.268+0000 7fb3d6ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb3d809c860 0x7fb3d81385b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33178/0 (socket says 172.21.15.142:33178) 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d77fe700 1 -- 172.21.15.142:0/3839900970 learned_addr learned my addr 172.21.15.142:0/3839900970 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb3d806e860 auth_method 0 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 1 -- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 msgr2=0x7fb3d809bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:28:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 1 -- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 msgr2=0x7fb3d809bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3839900970 >> v2:172.21.15.150:3300/0 conn(0x7fb3d806f460 0x7fb3d809c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): get_auth_request con 0x7fb3d806f460 auth_method 0 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_done global_id 194434 payload 274 2024-09-18T10:28:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 msgr2=0x7fb3d809bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3839900970 >> v1:172.21.15.142:6789/0 conn(0x7fb3d809c860 legacy=0x7fb3d81385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 monclient: found mon.noname-c 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3839900970 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d809b770 con 0x7fb3d806f460 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3839900970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d806f460 msgr2=0x7fb3d809c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.272+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3839900970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d806f460 0x7fb3d809c320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8098a50 tx=0x7fb3c80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5520a5495005c1aa server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 1 -- 172.21.15.142:0/3839900970 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800b8c0 con 0x7fb3d806f460 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: dump: 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 5 monclient: authenticate success, global_id 194434 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 1 -- 172.21.15.142:0/3839900970 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c800ba20 con 0x7fb3d806f460 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 1 -- 172.21.15.142:0/3839900970 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800bdc0 con 0x7fb3d806f460 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 4 set_mon_vals no callback set 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d5ffb700 10 monclient: dump: 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: shutdown 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3839900970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d806f460 msgr2=0x7fb3d809c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3839900970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d806f460 0x7fb3d809c320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8098a50 tx=0x7fb3c80091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3839900970 shutdown_connections 2024-09-18T10:28:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3839900970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d806f460 0x7fb3d809c320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3839900970 >> v2:172.21.15.142:3300/0 conn(0x7fb3d806e860 0x7fb3d809bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3839900970 >> 172.21.15.142:0/3839900970 conn(0x7fb3d80a1fe0 msgr2=0x7fb3d8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3839900970 shutdown_connections 2024-09-18T10:28:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3839900970 wait complete. 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: build_initial_monmap 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: monmap: 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:44.296 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 Processor -- start 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- start start 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: init 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb3d806e860 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 0x7fb3d8141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb3d809c860 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3d81420f0 0x7fb3d8090090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb3d81420f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80a0380 con 0x7fb3d806e860 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8093900 con 0x7fb3d809c860 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d81424e0 con 0x7fb3d81420f0 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d7fff700 10 monclient(hunting): authenticate will time out at 2013.206899s 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41356/0 (socket says 172.21.15.142:41356) 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 -- 172.21.15.142:0/3412411748 learned_addr learned my addr 172.21.15.142:0/3412411748 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 10 monclient(hunting): get_auth_request con 0x7fb3d806e860 auth_method 0 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3d81420f0 0x7fb3d8090090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 0x7fb3d8141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): get_auth_request con 0x7fb3d809c860 auth_method 0 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:44.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): get_auth_request con 0x7fb3d81420f0 auth_method 0 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 msgr2=0x7fb3d8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 msgr2=0x7fb3d8141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:44.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient(hunting): handle_auth_done global_id 194437 payload 824 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3d81420f0 msgr2=0x7fb3d8090090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3d81420f0 0x7fb3d8090090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 msgr2=0x7fb3d8141670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:44.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient: _finish_hunting 0 2024-09-18T10:28:44.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 monclient: found mon.noname-b 2024-09-18T10:28:44.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:44.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.276+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3412411748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c8009420 con 0x7fb3d809c860 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d67fc700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 msgr2=0x7fb3d8141bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d67fc700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 0x7fb3d8141bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8015000 tx=0x7fb3c8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8011bb0 con 0x7fb3d809c860 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: dump: 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:44.306 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: _finish_auth 0 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8020070 con 0x7fb3d809c860 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 5 monclient: authenticate success, global_id 194437 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 10 monclient: _renew_subs 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d8093620 con 0x7fb3d809c860 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:44.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c801b450 con 0x7fb3d809c860 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3ceffd700 10 monclient: dump: 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:44.308 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d57fa700 4 set_mon_vals no callback set 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 10 monclient: _renew_subs 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.280+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d8091af0 con 0x7fb3d809c860 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb3c8022030 con 0x7fb3d809c860 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d7fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d8072a80 con 0x7fb3d809c860 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3ceffd700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 0x7fb3b8037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb3c804aac0 con 0x7fb3d809c860 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 0x7fb3b8037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:44.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.284+0000 7fb3d6ffd700 10 monclient: get_auth_request con 0x7fb3b8035ac0 auth_method 0 2024-09-18T10:28:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.288+0000 7fb3d6ffd700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 0x7fb3b8037f70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0006f00 tx=0x7fb3c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.296+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb3c801c450 con 0x7fb3d809c860 2024-09-18T10:28:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.296+0000 7fb3ceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:44.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.296+0000 7fb3ceffd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:44.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:44.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:44.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:44.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb3d8072a80 con 0x7fb3d809c860 2024-09-18T10:28:44.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3ceffd700 1 -- 172.21.15.142:0/3412411748 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fb3c801ab30 con 0x7fb3d809c860 2024-09-18T10:28:44.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3ceffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:44.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3ceffd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:44.528 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 msgr2=0x7fb3b8037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 0x7fb3b8037f70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0006f00 tx=0x7fb3c0008040 comp rx=0 tx=0).stop 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.520+0000 7fb3d7fff700 10 monclient: shutdown 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 msgr2=0x7fb3d8141bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 0x7fb3d8141bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8015000 tx=0x7fb3c8016000 comp rx=0 tx=0).stop 2024-09-18T10:28:44.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 shutdown_connections 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb3b8035ac0 0x7fb3b8037f70 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb3d81420f0 0x7fb3d8090090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3d809c860 0x7fb3d8141bb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 --2- 172.21.15.142:0/3412411748 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb3d806e860 0x7fb3d8141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 >> 172.21.15.142:0/3412411748 conn(0x7fb3d80a1fe0 msgr2=0x7fb3d809dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 shutdown_connections 2024-09-18T10:28:44.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:44.524+0000 7fb3d7fff700 1 -- 172.21.15.142:0/3412411748 wait complete. 2024-09-18T10:28:47.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.644+0000 7f05ce1ee700 10 monclient: get_monmap_and_config 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.644+0000 7f05ce1ee700 10 monclient: build_initial_monmap 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.644+0000 7f05ce1ee700 10 monclient: monmap: 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:47.650809+0000 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:47.650809+0000 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:47.653 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:47.654 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 Processor -- start 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 -- start start 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient: init 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05c809d920 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-b con 0x7f05c806f660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f05c80971f0 0x7f05c8136590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-c con 0x7f05c80971f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c809b0b0 con 0x7f05c809d920 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c8095ad0 con 0x7f05c80971f0 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c809a7e0 con 0x7f05c806f660 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05ce1ee700 10 monclient(hunting): authenticate will time out at 2016.576990s 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52078/0 (socket says 172.21.15.142:52078) 2024-09-18T10:28:47.658 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 -- 172.21.15.142:0/1808407955 learned_addr learned my addr 172.21.15.142:0/1808407955 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): get_auth_request con 0x7f05c806f660 auth_method 0 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 1 --2- 172.21.15.142:0/1808407955 >> v2:172.21.15.150:3300/0 conn(0x7f05c80971f0 0x7f05c8136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): get_auth_request con 0x7f05c80971f0 auth_method 0 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:47.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cd9ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_done global_id 76022 payload 274 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 -- 172.21.15.142:0/1808407955 >> v2:172.21.15.150:3300/0 conn(0x7f05c80971f0 msgr2=0x7f05c8136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 --2- 172.21.15.142:0/1808407955 >> v2:172.21.15.150:3300/0 conn(0x7f05c80971f0 0x7f05c8136590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 -- 172.21.15.142:0/1808407955 >> v1:172.21.15.150:6789/0 conn(0x7f05c809d920 legacy=0x7f05c80953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient: _finish_hunting 0 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 monclient: found mon.noname-b 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.648+0000 7f05cc9eb700 1 -- 172.21.15.142:0/1808407955 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c809aaf0 con 0x7f05c806f660 2024-09-18T10:28:47.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05cc9eb700 1 -- 172.21.15.142:0/1808407955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 msgr2=0x7f05c8096cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:47.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05cc9eb700 1 --2- 172.21.15.142:0/1808407955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f05b800bc50 tx=0x7f05b800b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa6adbe2716b32f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:47.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 1 -- 172.21.15.142:0/1808407955 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b800de00 con 0x7f05c806f660 2024-09-18T10:28:47.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:47.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: dump: 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: _finish_auth 0 2024-09-18T10:28:47.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 5 monclient: authenticate success, global_id 76022 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 1 -- 172.21.15.142:0/1808407955 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8010470 con 0x7f05c806f660 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 1 -- 172.21.15.142:0/1808407955 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8010830 con 0x7f05c806f660 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 4 set_mon_vals no callback set 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05bffff700 10 monclient: dump: 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:47.667 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 10 monclient: shutdown 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:47.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 -- 172.21.15.142:0/1808407955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 msgr2=0x7f05c8096cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/1808407955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f05b800bc50 tx=0x7f05b800b4f0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 -- 172.21.15.142:0/1808407955 shutdown_connections 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/1808407955 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8096cb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/1808407955 >> v2:172.21.15.150:3300/0 conn(0x7f05c80971f0 0x7f05c8136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 -- 172.21.15.142:0/1808407955 >> 172.21.15.142:0/1808407955 conn(0x7f05c80a1fe0 msgr2=0x7f05c8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 -- 172.21.15.142:0/1808407955 shutdown_connections 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 -- 172.21.15.142:0/1808407955 wait complete. 2024-09-18T10:28:47.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 10 monclient: build_initial_monmap 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 10 monclient: monmap: 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.652+0000 7f05ce1ee700 1 Processor -- start 2024-09-18T10:28:47.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 -- start start 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient: init 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-c con 0x7f05c806f660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c80971f0 0x7f05c808a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-b con 0x7f05c80971f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:47.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): picked mon.noname-a con 0x7f05c809d920 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c808d410 con 0x7f05c809d920 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80a0a50 con 0x7f05c80971f0 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c808ce50 con 0x7f05c806f660 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:47.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05ce1ee700 10 monclient(hunting): authenticate will time out at 2016.585038s 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52088/0 (socket says 172.21.15.142:52088) 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 -- 172.21.15.142:0/3617882716 learned_addr learned my addr 172.21.15.142:0/3617882716 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41368/0 (socket says 172.21.15.142:41368) 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): get_auth_request con 0x7f05c806f660 auth_method 0 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:47.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 10 monclient(hunting): get_auth_request con 0x7f05c809d920 auth_method 0 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c80971f0 0x7f05c808a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 msgr2=0x7f05c808a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 msgr2=0x7f05c808a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): get_auth_request con 0x7f05c80971f0 auth_method 0 2024-09-18T10:28:47.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd9ed700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cc9eb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient(hunting): handle_auth_done global_id 76025 payload 824 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c80971f0 msgr2=0x7f05c808a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c80971f0 0x7f05c808a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 msgr2=0x7f05c808a540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient: _finish_hunting 0 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 monclient: found mon.noname-c 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.656+0000 7f05cd1ec700 1 -- 172.21.15.142:0/3617882716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b800b790 con 0x7f05c806f660 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05cd1ec700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 msgr2=0x7f05c8089ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05cd1ec700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c5d0 tx=0x7f05c4007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c40117a0 con 0x7f05c806f660 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:47.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: dump: 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: _finish_auth 0 2024-09-18T10:28:47.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c4011900 con 0x7f05c806f660 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 5 monclient: authenticate success, global_id 76025 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _renew_subs 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c808d130 con 0x7f05c806f660 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4011cc0 con 0x7f05c806f660 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 10 monclient: dump: 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:47.681 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bf7fe700 4 set_mon_vals no callback set 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _renew_subs 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c8095000 con 0x7f05c806f660 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:47.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c8072a80 con 0x7f05c806f660 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f05c4012630 con 0x7f05c806f660 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 0x7f05ac037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f05c4048cf0 con 0x7f05c806f660 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05cc9eb700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 0x7f05ac037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.660+0000 7f05cc9eb700 10 monclient: get_auth_request con 0x7f05ac035b10 auth_method 0 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.664+0000 7f05cc9eb700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 0x7f05ac037fc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f05b800bc50 tx=0x7f05b801a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.668+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f05c401c460 con 0x7f05c806f660 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.668+0000 7f05bd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:47.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.668+0000 7f05bd7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:47.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.892+0000 7f05ce1ee700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:47.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.892+0000 7f05ce1ee700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:47.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.892+0000 7f05ce1ee700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:47.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.892+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f05c80678a0 con 0x7f05c806f660 2024-09-18T10:28:47.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05bd7fa700 1 -- 172.21.15.142:0/3617882716 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f05c4010750 con 0x7f05c806f660 2024-09-18T10:28:47.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05bd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:47.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05bd7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:47.902 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:47.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 msgr2=0x7f05ac037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 0x7f05ac037fc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f05b800bc50 tx=0x7f05b801a000 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 10 monclient: shutdown 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 msgr2=0x7f05c8089ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c5d0 tx=0x7f05c4007f60 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 shutdown_connections 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f05ac035b10 0x7f05ac037fc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05c809d920 0x7f05c808a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f05c80971f0 0x7f05c808a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 --2- 172.21.15.142:0/3617882716 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f05c806f660 0x7f05c8089ac0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 >> 172.21.15.142:0/3617882716 conn(0x7f05c80a1fe0 msgr2=0x7f05c809f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:47.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 shutdown_connections 2024-09-18T10:28:47.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:47.896+0000 7f05ce1ee700 1 -- 172.21.15.142:0/3617882716 wait complete. 2024-09-18T10:28:50.921 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:51.031 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: get_monmap_and_config 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: monmap: 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:51.029665+0000 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:51.029665+0000 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:51.032 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:51.033 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:51.033 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 Processor -- start 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 -- start start 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: init 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c09aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a2c0967a0 0x7f7a2c09bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c0967a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7a2c09c320 0x7f7a2c09c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c09c320 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c095880 con 0x7f7a2c09c320 2024-09-18T10:28:51.037 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c09aeb0 con 0x7f7a2c0967a0 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c06f6d0 con 0x7f7a2c09aae0 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a31cc4700 10 monclient(hunting): authenticate will time out at 2019.955116s 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52120/0 (socket says 172.21.15.142:52120) 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 1 -- 172.21.15.142:0/3959905495 learned_addr learned my addr 172.21.15.142:0/3959905495 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request con 0x7f7a2c09aae0 auth_method 0 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7a2c09c320 0x7f7a2c09c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.038 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/3959905495 >> v2:172.21.15.150:3300/0 conn(0x7f7a2c0967a0 0x7f7a2c09bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): get_auth_request con 0x7f7a2c09c320 auth_method 0 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request con 0x7f7a2c0967a0 auth_method 0 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 10 monclient(hunting): handle_auth_done global_id 224097 payload 274 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 1 -- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 msgr2=0x7f7a2c096260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.024+0000 7f7a314c3700 1 -- 172.21.15.142:0/3959905495 >> v2:172.21.15.150:3300/0 conn(0x7f7a2c0967a0 msgr2=0x7f7a2c09bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 1 --2- 172.21.15.142:0/3959905495 >> v2:172.21.15.150:3300/0 conn(0x7f7a2c0967a0 0x7f7a2c09bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 1 monclient: found mon.noname-a 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 1 -- 172.21.15.142:0/3959905495 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c0955a0 con 0x7f7a2c09c320 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 1 -- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09c320 msgr2=0x7f7a2c09c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a314c3700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09c320 0x7f7a2c09c780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c067590 tx=0x7f7a240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a0106026a688b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 1 -- 172.21.15.142:0/3959905495 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011840 con 0x7f7a2c09c320 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: dump: 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.041 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 5 monclient: authenticate success, global_id 224097 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 1 -- 172.21.15.142:0/3959905495 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a240119c0 con 0x7f7a2c09c320 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 4 set_mon_vals no callback set 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 1 -- 172.21.15.142:0/3959905495 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011d80 con 0x7f7a2c09c320 2024-09-18T10:28:51.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a2b7fe700 10 monclient: dump: 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.043 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 10 monclient: shutdown 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 -- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09c320 msgr2=0x7f7a2c09c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09c320 0x7f7a2c09c780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c067590 tx=0x7f7a240091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 -- 172.21.15.142:0/3959905495 shutdown_connections 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09c320 0x7f7a2c09c780 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/3959905495 >> v2:172.21.15.150:3300/0 conn(0x7f7a2c0967a0 0x7f7a2c09bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/3959905495 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09aae0 0x7f7a2c096260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 -- 172.21.15.142:0/3959905495 >> 172.21.15.142:0/3959905495 conn(0x7f7a2c0a1fe0 msgr2=0x7f7a2c070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 -- 172.21.15.142:0/3959905495 shutdown_connections 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.028+0000 7f7a31cc4700 1 -- 172.21.15.142:0/3959905495 wait complete. 2024-09-18T10:28:51.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient: monmap: 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 Processor -- start 2024-09-18T10:28:51.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 -- start start 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient: init 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a2c06ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0967a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-a con 0x7f7a2c09aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a2c135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c09c320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c06c2a0 con 0x7f7a2c09aae0 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0a07d0 con 0x7f7a2c0967a0 2024-09-18T10:28:51.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c06c7d0 con 0x7f7a2c09c320 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a31cc4700 10 monclient(hunting): authenticate will time out at 2019.961299s 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41394/0 (socket says 172.21.15.142:41394) 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 learned_addr learned my addr 172.21.15.142:0/4032385041 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request con 0x7f7a2c09aae0 auth_method 0 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a2c135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): get_auth_request con 0x7f7a2c09c320 auth_method 0 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a2c06ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request con 0x7f7a2c0967a0 auth_method 0 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_done global_id 224100 payload 824 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 msgr2=0x7f7a2c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.055 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a2c135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 msgr2=0x7f7a2c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a2c06ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 monclient: found mon.noname-a 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200094f0 con 0x7f7a2c09aae0 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.032+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a2c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 msgr2=0x7f7a2c134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009e70 tx=0x7f7a1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 msgr2=0x7f7a2c134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 msgr2=0x7f7a2c134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:28:51.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009e70 tx=0x7f7a1c00b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009e70 tx=0x7f7a1c00b040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 msgr2=0x7f7a2c134be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a2c09aae0 0x7f7a2c134be0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a18005500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): picked mon.noname-b con 0x7f7a2c0967a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 0x7f7a18008330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): picked mon.noname-a con 0x7f7a18005eb0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a1800aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): picked mon.noname-c con 0x7f7a2c09c320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1800b870 con 0x7f7a18005eb0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1800b9b0 con 0x7f7a2c0967a0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1800bb70 con 0x7f7a2c09c320 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a28ff9700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 0x7f7a18008330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a1800aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request con 0x7f7a18005eb0 auth_method 0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): get_auth_request con 0x7f7a2c09c320 auth_method 0 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a18005500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request con 0x7f7a2c0967a0 auth_method 0 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a314c3700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:28:51.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a30cc2700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient(hunting): handle_auth_done global_id 224100 payload 293 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 msgr2=0x7f7a1800aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a1800aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 msgr2=0x7f7a18005500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a18005500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 monclient: found mon.noname-a 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1800bd90 con 0x7f7a18005eb0 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 msgr2=0x7f7a18008330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.036+0000 7f7a2bfff700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 0x7f7a18008330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009e40 tx=0x7f7a1c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c023070 con 0x7f7a18005eb0 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: dump: 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.061 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 5 monclient: authenticate success, global_id 224100 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _renew_subs 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c00f040 con 0x7f7a18005eb0 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a24009420 con 0x7f7a18005eb0 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c014720 con 0x7f7a18005eb0 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a2affd700 4 set_mon_vals no callback set 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:51.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 10 monclient: dump: 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _renew_subs 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c06bf30 con 0x7f7a18005eb0 2024-09-18T10:28:51.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7a1c003020 con 0x7f7a18005eb0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 0x7f7a1803ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7a1c04acb0 con 0x7f7a18005eb0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c072a80 con 0x7f7a18005eb0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 0x7f7a1803ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.040+0000 7f7a30cc2700 10 monclient: get_auth_request con 0x7f7a2c09aae0 auth_method 0 2024-09-18T10:28:51.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.044+0000 7f7a30cc2700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 0x7f7a1803ff40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7a20006d40 tx=0x7f7a20008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.048+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7a1c018020 con 0x7f7a18005eb0 2024-09-18T10:28:51.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.048+0000 7f7a28ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.048+0000 7f7a28ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:51.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:51.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:51.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a2c06aff0 con 0x7f7a18005eb0 2024-09-18T10:28:51.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a28ff9700 1 -- 172.21.15.142:0/4032385041 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f7a1c01db00 con 0x7f7a18005eb0 2024-09-18T10:28:51.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a28ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:51.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a28ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:51.283 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:51.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 msgr2=0x7f7a1803ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 0x7f7a1803ff40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7a20006d40 tx=0x7f7a20008040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 10 monclient: shutdown 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 msgr2=0x7f7a18008330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 0x7f7a18008330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009e40 tx=0x7f7a1c01a000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.276+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 shutdown_connections 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7a2c09aae0 0x7f7a1803ff40 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a2c09c320 0x7f7a1800aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a18005eb0 0x7f7a18008330 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 --2- 172.21.15.142:0/4032385041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a2c0967a0 0x7f7a18005500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 >> 172.21.15.142:0/4032385041 conn(0x7f7a2c0a1fe0 msgr2=0x7f7a2c098a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 shutdown_connections 2024-09-18T10:28:51.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.280+0000 7f7a31cc4700 1 -- 172.21.15.142:0/4032385041 wait complete. 2024-09-18T10:28:51.297 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:28:51.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:28:51.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: get_monmap_and_config 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: monmap: 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:51.403910+0000 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:51.403910+0000 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.406 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:51.407 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:51.407 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:51.407 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:51.407 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:51.407 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 Processor -- start 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 -- start start 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: init 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12400999e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f124009c410 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 0x7f124013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-b con 0x7f1240138920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400a20c0 con 0x7f12400999e0 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12400a1c80 con 0x7f124009c410 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124009a000 con 0x7f1240138920 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f1244d0c700 10 monclient(hunting): authenticate will time out at 2020.329456s 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f12400999e0 0x7f12400a0180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33192/0 (socket says 172.21.15.142:33192) 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123f7fe700 1 -- 172.21.15.142:0/1246484361 learned_addr learned my addr 172.21.15.142:0/1246484361 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 1 --2- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 0x7f124013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): get_auth_request con 0x7f1240138920 auth_method 0 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.410 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient(hunting): handle_auth_done global_id 76034 payload 274 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 1 -- 172.21.15.142:0/1246484361 >> v1:172.21.15.150:6789/0 conn(0x7f124009c410 legacy=0x7f12400a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 1 -- 172.21.15.142:0/1246484361 >> v1:172.21.15.142:6789/0 conn(0x7f12400999e0 legacy=0x7f12400a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 1 monclient: found mon.noname-b 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.411 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.400+0000 7f123ffff700 1 -- 172.21.15.142:0/1246484361 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124009a580 con 0x7f1240138920 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123ffff700 1 -- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 msgr2=0x7f124013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123ffff700 1 --2- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 0x7f124013ad00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1238009910 tx=0x7f12380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e470d641c66ba73 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 1 -- 172.21.15.142:0/1246484361 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123800ba70 con 0x7f1240138920 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: dump: 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.413 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 5 monclient: authenticate success, global_id 76034 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 1 -- 172.21.15.142:0/1246484361 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123800bbd0 con 0x7f1240138920 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 4 set_mon_vals no callback set 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 1 -- 172.21.15.142:0/1246484361 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1238011420 con 0x7f1240138920 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f123e7fc700 10 monclient: dump: 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.415 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 10 monclient: shutdown 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 msgr2=0x7f124013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 --2- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 0x7f124013ad00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1238009910 tx=0x7f12380091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- 172.21.15.142:0/1246484361 shutdown_connections 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 --2- 172.21.15.142:0/1246484361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1240138920 0x7f124013ad00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- 172.21.15.142:0/1246484361 >> 172.21.15.142:0/1246484361 conn(0x7f12400a5790 msgr2=0x7f12400a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- 172.21.15.142:0/1246484361 shutdown_connections 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- 172.21.15.142:0/1246484361 wait complete. 2024-09-18T10:28:51.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 10 monclient: monmap: 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 Processor -- start 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 1 -- start start 2024-09-18T10:28:51.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.404+0000 7f1244d0c700 10 monclient: init 2024-09-18T10:28:51.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:28:51.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400999e0 0x7f124006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-a con 0x7f12400999e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 0x7f124006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-c con 0x7f124009c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1240138920 0x7f124006c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): picked mon.noname-b con 0x7f1240138920 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124006d410 con 0x7f12400999e0 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124009f090 con 0x7f1240138920 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124006ecf0 con 0x7f124009c410 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient(hunting): authenticate will time out at 2020.336468s 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400999e0 0x7f124006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 0x7f124006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400999e0 0x7f124006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41408/0 (socket says 172.21.15.142:41408) 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 1 -- 172.21.15.142:0/963940541 learned_addr learned my addr 172.21.15.142:0/963940541 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): get_auth_request con 0x7f12400999e0 auth_method 0 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): get_auth_request con 0x7f124009c410 auth_method 0 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1240138920 0x7f124006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 10 monclient(hunting): get_auth_request con 0x7f1240138920 auth_method 0 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient(hunting): handle_auth_done global_id 76037 payload 824 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1240138920 msgr2=0x7f124006c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1240138920 0x7f124006c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400999e0 msgr2=0x7f124006ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400999e0 0x7f124006ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 monclient: found mon.noname-c 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1238009420 con 0x7f124009c410 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123ffff700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1240138920 0x7f124006c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 msgr2=0x7f124006bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f123effd700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 0x7f124006bfc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f123000c4e0 tx=0x7f1230007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12300117a0 con 0x7f124009c410 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:51.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 10 monclient: dump: 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1230011920 con 0x7f124009c410 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 5 monclient: authenticate success, global_id 76037 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient: _renew_subs 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.408+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124006f000 con 0x7f124009c410 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230011ce0 con 0x7f124009c410 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 10 monclient: dump: 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.427 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123d7fa700 4 set_mon_vals no callback set 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 10 monclient: _renew_subs 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124006e700 con 0x7f124009c410 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f1230012630 con 0x7f124009c410 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 0x7f12400a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12400647f0 con 0x7f1240132400 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1224035ac0 0x7f1224037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 0x7f12400a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient: get_auth_request con 0x7f1240132400 auth_method 0 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f1230048ce0 con 0x7f124009c410 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 10 monclient(hunting): handle_auth_done global_id 224106 payload 824 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123f7fe700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1240072a80 con 0x7f1240132400 2024-09-18T10:28:51.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123ffff700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1224035ac0 0x7f1224037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.412+0000 7f123ffff700 10 monclient: get_auth_request con 0x7f1224035ac0 auth_method 0 2024-09-18T10:28:51.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f123f7fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 msgr2=0x7f12400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f123f7fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 0x7f12400a3670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f122800efd0 tx=0x7f122800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1228014480 con 0x7f1240132400 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 10 monclient: dump: 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.431 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f122800f040 con 0x7f1240132400 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 msgr2=0x7f12400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 0x7f12400a3670 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f122800efd0 tx=0x7f122800b040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 msgr2=0x7f12400a3670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f123ffff700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1224035ac0 0x7f1224037f70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1238009910 tx=0x7f1238016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.416+0000 7f12377fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1240132400 0x7f12400a3670 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f122800efd0 tx=0x7f122800b040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f1244d0c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:51.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f1244d0c700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 0x7f12400a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f1244d0c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124006d7c0 con 0x7f12400a3bb0 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 0x7f12400a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient: get_auth_request con 0x7f12400a3bb0 auth_method 0 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 10 monclient(hunting): handle_auth_done global_id 224109 payload 824 2024-09-18T10:28:51.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 1 -- 172.21.15.142:0/963940541 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f124006e8f0 con 0x7f12400a3bb0 2024-09-18T10:28:51.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 msgr2=0x7f12400a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f123f7fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 0x7f12400a3fb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1228019000 tx=0x7f122801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122801ddd0 con 0x7f12400a3bb0 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f12377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f12377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.428+0000 7f12377fe700 10 monclient: dump: 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f1228024070 con 0x7f12400a3bb0 2024-09-18T10:28:51.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:51.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:51.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 msgr2=0x7f12400a3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 0x7f12400a3fb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1228019000 tx=0x7f122801a000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 msgr2=0x7f12400a3fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f12377fe700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12400a3bb0 0x7f12400a3fb0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1228019000 tx=0x7f122801a000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:51.440 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.441 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.442 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:51.443 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.444 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.445 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.446 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.447 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:51.448 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.449 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.450 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:51.451 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:51.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1224035ac0 msgr2=0x7f1224037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1224035ac0 0x7f1224037f70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1238009910 tx=0x7f1238016000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 10 monclient: shutdown 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 msgr2=0x7f124006bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 --2- 172.21.15.142:0/963940541 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f124009c410 0x7f124006bfc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f123000c4e0 tx=0x7f1230007ea0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f123ffff700 1 -- 172.21.15.142:0/963940541 reap_dead start 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f123ffff700 1 -- 172.21.15.142:0/963940541 reap_dead start 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 shutdown_connections 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 >> 172.21.15.142:0/963940541 conn(0x7f12400a5790 msgr2=0x7f1240134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 shutdown_connections 2024-09-18T10:28:51.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.432+0000 7f1244d0c700 1 -- 172.21.15.142:0/963940541 wait complete. 2024-09-18T10:28:51.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:51.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:28:51.454 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:28:51.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.556+0000 7f57d5970700 10 monclient: get_monmap_and_config 2024-09-18T10:28:51.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.556+0000 7f57d5970700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.556+0000 7f57d5970700 10 monclient: monmap: 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:51.563447+0000 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:51.563447+0000 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:51.566 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 Processor -- start 2024-09-18T10:28:51.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 -- start start 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient: init 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f57d006f3b0 0x7f57d00989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-c con 0x7f57d006f3b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57d0098f00 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d0099320 0x7f57d0136d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-b con 0x7f57d0099320 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d009c330 con 0x7f57d0098f00 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d009b380 con 0x7f57d006f3b0 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d009b030 con 0x7f57d0099320 2024-09-18T10:28:51.569 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d5970700 10 monclient(hunting): authenticate will time out at 2020.488944s 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57c7fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f57d0098f00 0x7f57d0133660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33196/0 (socket says 172.21.15.142:33196) 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57c7fff700 1 -- 172.21.15.142:0/1468875404 learned_addr learned my addr 172.21.15.142:0/1468875404 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d0099320 0x7f57d0136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 --2- 172.21.15.142:0/1468875404 >> v2:172.21.15.150:3300/0 conn(0x7f57d006f3b0 0x7f57d00989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): get_auth_request con 0x7f57d0099320 auth_method 0 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): get_auth_request con 0x7f57d006f3b0 auth_method 0 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient(hunting): handle_auth_done global_id 194455 payload 274 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 -- 172.21.15.142:0/1468875404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d0099320 msgr2=0x7f57d0136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d0099320 0x7f57d0136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 -- 172.21.15.142:0/1468875404 >> v1:172.21.15.142:6789/0 conn(0x7f57d0098f00 legacy=0x7f57d0133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 monclient: found mon.noname-c 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:51.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.560+0000 7f57d496e700 1 -- 172.21.15.142:0/1468875404 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d009c0f0 con 0x7f57d006f3b0 2024-09-18T10:28:51.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d496e700 1 -- 172.21.15.142:0/1468875404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d006f3b0 msgr2=0x7f57d00989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d496e700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d006f3b0 0x7f57d00989c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0098e0 tx=0x7f57cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54ebda8abd03aae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 1 -- 172.21.15.142:0/1468875404 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00b850 con 0x7f57d006f3b0 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: dump: 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.574 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 1 -- 172.21.15.142:0/1468875404 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc00b9b0 con 0x7f57d006f3b0 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 1 -- 172.21.15.142:0/1468875404 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00bd50 con 0x7f57d006f3b0 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57c77fe700 10 monclient: dump: 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.575 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 5 monclient: authenticate success, global_id 194455 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 4 set_mon_vals no callback set 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 10 monclient: shutdown 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 -- 172.21.15.142:0/1468875404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d006f3b0 msgr2=0x7f57d00989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d006f3b0 0x7f57d00989c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0098e0 tx=0x7f57cc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 -- 172.21.15.142:0/1468875404 shutdown_connections 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d0099320 0x7f57d0136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 --2- 172.21.15.142:0/1468875404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d006f3b0 0x7f57d00989c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 -- 172.21.15.142:0/1468875404 >> 172.21.15.142:0/1468875404 conn(0x7f57d0070ef0 msgr2=0x7f57d00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 -- 172.21.15.142:0/1468875404 shutdown_connections 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 1 -- 172.21.15.142:0/1468875404 wait complete. 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.564+0000 7f57d5970700 10 monclient: monmap: 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:51.577 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:51.578 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 Processor -- start 2024-09-18T10:28:51.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 -- start start 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient: init 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 0x7f57d008cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-c con 0x7f57d006f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d0098f00 0x7f57d008d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-b con 0x7f57d0098f00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 0x7f57d008d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): picked mon.noname-a con 0x7f57d0099320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0090390 con 0x7f57d0099320 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.584 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d006fa20 con 0x7f57d0098f00 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0097290 con 0x7f57d006f3b0 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d5970700 10 monclient(hunting): authenticate will time out at 2020.497932s 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 0x7f57d008cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 0x7f57d008cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52186/0 (socket says 172.21.15.142:52186) 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 1 -- 172.21.15.142:0/1223420931 learned_addr learned my addr 172.21.15.142:0/1223420931 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): get_auth_request con 0x7f57d006f3b0 auth_method 0 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 0x7f57d008d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): get_auth_request con 0x7f57d0099320 auth_method 0 2024-09-18T10:28:51.585 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d0098f00 0x7f57d008d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): get_auth_request con 0x7f57d0098f00 auth_method 0 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.586 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient(hunting): handle_auth_done global_id 224112 payload 824 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 msgr2=0x7f57d008cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 0x7f57d008cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d0098f00 msgr2=0x7f57d008d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57d0098f00 0x7f57d008d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 monclient: found mon.noname-a 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d516f700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc009420 con 0x7f57d0099320 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.568+0000 7f57d496e700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d006f3b0 0x7f57d008cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:51.587 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d516f700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 msgr2=0x7f57d008d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d516f700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 0x7f57d008d5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009e50 tx=0x7f57c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c800ccb0 con 0x7f57d0099320 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: dump: 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.588 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 5 monclient: authenticate success, global_id 224112 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c800f040 con 0x7f57d0099320 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 monclient: _renew_subs 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c80146d0 con 0x7f57d0099320 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d008f170 con 0x7f57d0099320 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:51.589 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c6ffd700 4 set_mon_vals no callback set 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 10 monclient: dump: 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 monclient: _renew_subs 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:51.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d009d3f0 con 0x7f57d0099320 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f57c8014a40 con 0x7f57d0099320 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f57bc035ac0 0x7f57bc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 0x7f57d013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00648a0 con 0x7f57d009caa0 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f57c8049be0 con 0x7f57d0099320 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.572+0000 7f57c7fff700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 0x7f57d013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient: get_auth_request con 0x7f57d009caa0 auth_method 0 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57d496e700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f57bc035ac0 0x7f57bc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57d496e700 10 monclient: get_auth_request con 0x7f57bc035ac0 auth_method 0 2024-09-18T10:28:51.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 10 monclient(hunting): handle_auth_done global_id 76046 payload 824 2024-09-18T10:28:51.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57d008eee0 con 0x7f57d009caa0 2024-09-18T10:28:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 msgr2=0x7f57d013f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c7fff700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 0x7f57d013f740 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57c000eac0 tx=0x7f57c0007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57d496e700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f57bc035ac0 0x7f57bc037f70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00a010 tx=0x7f57cc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c000cc40 con 0x7f57d009caa0 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 10 monclient: dump: 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.594 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f57c000cda0 con 0x7f57d009caa0 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 msgr2=0x7f57d013f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 0x7f57d013f740 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57c000eac0 tx=0x7f57c0007f90 comp rx=0 tx=0).stop 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 msgr2=0x7f57d013f740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.576+0000 7f57c4ff9700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d009caa0 0x7f57d013f740 secure :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57c000eac0 tx=0x7f57c0007f90 comp rx=0 tx=0).stop 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57d5970700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57d5970700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 0x7f57d0140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57d5970700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d008f360 con 0x7f57d013fc80 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 0x7f57d0140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient: get_auth_request con 0x7f57d013fc80 auth_method 0 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 10 monclient(hunting): handle_auth_done global_id 76049 payload 824 2024-09-18T10:28:51.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.584+0000 7f57c7fff700 1 -- 172.21.15.142:0/1223420931 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f57d01405c0 con 0x7f57d013fc80 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c7fff700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 msgr2=0x7f57d0140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c7fff700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 0x7f57d0140080 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57c0007360 tx=0x7f57c0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c0013e70 con 0x7f57d013fc80 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 10 monclient: dump: 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.597 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f57c00213f0 con 0x7f57d013fc80 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 msgr2=0x7f57d0140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 0x7f57d0140080 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57c0007360 tx=0x7f57c0016000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 msgr2=0x7f57d0140080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57c4ff9700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f57d013fc80 0x7f57d0140080 secure :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57c0007360 tx=0x7f57c0016000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:51.599 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.600 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.601 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:51.602 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.603 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:51.604 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.605 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:51.606 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:51.607 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.608 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.609 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.610 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:51.611 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f57bc035ac0 msgr2=0x7f57bc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f57bc035ac0 0x7f57bc037f70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00a010 tx=0x7f57cc016000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 10 monclient: shutdown 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 msgr2=0x7f57d008d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d516f700 1 -- 172.21.15.142:0/1223420931 reap_dead start 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 0x7f57d008d5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009e50 tx=0x7f57c800b040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 shutdown_connections 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 --2- 172.21.15.142:0/1223420931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f57d0099320 0x7f57d008d5a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 >> 172.21.15.142:0/1223420931 conn(0x7f57d0070ef0 msgr2=0x7f57d00a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 shutdown_connections 2024-09-18T10:28:51.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.588+0000 7f57d5970700 1 -- 172.21.15.142:0/1223420931 wait complete. 2024-09-18T10:28:51.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:51.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:28:51.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 10 monclient: get_monmap_and_config 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 10 monclient: monmap: 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:51.722820+0000 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:51.722820+0000 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.725 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:51.726 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:51.726 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:51.726 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:51.726 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:51.726 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 1 Processor -- start 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 1 -- start start 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.716+0000 7f9aacdf9700 10 monclient: init 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9aa809c2c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-b con 0x7f9aa8134880 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9aa81371a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80a6620 con 0x7f9aa809c2c0 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80a1c70 con 0x7f9aa81371a0 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa809ab70 con 0x7f9aa8134880 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient(hunting): authenticate will time out at 2020.648408s 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52202/0 (socket says 172.21.15.142:52202) 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/461044807 learned_addr learned my addr 172.21.15.142:0/461044807 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): get_auth_request con 0x7f9aa8134880 auth_method 0 2024-09-18T10:28:51.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa77fe700 1 --1- 172.21.15.142:0/461044807 >> v1:172.21.15.142:6789/0 conn(0x7f9aa809c2c0 0x7f9aa80999e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33202/0 (socket says 172.21.15.142:33202) 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_done global_id 76052 payload 274 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/461044807 >> v1:172.21.15.150:6789/0 conn(0x7f9aa81371a0 legacy=0x7f9aa8139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/461044807 >> v1:172.21.15.142:6789/0 conn(0x7f9aa809c2c0 legacy=0x7f9aa80999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 monclient: found mon.noname-b 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/461044807 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa809c690 con 0x7f9aa8134880 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/461044807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 msgr2=0x7f9aa8136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa6ffd700 1 --2- 172.21.15.142:0/461044807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009f50 tx=0x7f9a98009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9d50c4c9ce4759a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 1 -- 172.21.15.142:0/461044807 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9800e070 con 0x7f9aa8134880 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: dump: 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.735 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 5 monclient: authenticate success, global_id 76052 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 1 -- 172.21.15.142:0/461044807 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9800bcc0 con 0x7f9aa8134880 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aacdf9700 4 set_mon_vals no callback set 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 1 -- 172.21.15.142:0/461044807 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98012660 con 0x7f9aa8134880 2024-09-18T10:28:51.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.720+0000 7f9aa67fc700 10 monclient: dump: 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: shutdown 2024-09-18T10:28:51.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- 172.21.15.142:0/461044807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 msgr2=0x7f9aa8136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/461044807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009f50 tx=0x7f9a98009840 comp rx=0 tx=0).stop 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- 172.21.15.142:0/461044807 shutdown_connections 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/461044807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa8134880 0x7f9aa8136c60 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- 172.21.15.142:0/461044807 >> 172.21.15.142:0/461044807 conn(0x7f9aa80a5720 msgr2=0x7f9aa80a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- 172.21.15.142:0/461044807 shutdown_connections 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- 172.21.15.142:0/461044807 wait complete. 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: monmap: 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:51.738 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 Processor -- start 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- start start 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: init 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa809c2c0 0x7f9aa806bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-a con 0x7f9aa809c2c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 0x7f9aa806c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-c con 0x7f9aa81371a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa806d600 0x7f9aa806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): picked mon.noname-b con 0x7f9aa806d600 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa806f040 con 0x7f9aa809c2c0 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa806dab0 con 0x7f9aa806d600 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa806f7b0 con 0x7f9aa81371a0 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aacdf9700 10 monclient(hunting): authenticate will time out at 2020.654979s 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 0x7f9aa806c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa809c2c0 0x7f9aa806bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 0x7f9aa806c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52212/0 (socket says 172.21.15.142:52212) 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/775472246 learned_addr learned my addr 172.21.15.142:0/775472246 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): get_auth_request con 0x7f9aa81371a0 auth_method 0 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): get_auth_request con 0x7f9aa809c2c0 auth_method 0 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa806d600 0x7f9aa806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa7fff700 10 monclient(hunting): get_auth_request con 0x7f9aa806d600 auth_method 0 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient(hunting): handle_auth_done global_id 76055 payload 824 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa806d600 msgr2=0x7f9aa806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa806d600 0x7f9aa806c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa809c2c0 msgr2=0x7f9aa806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa809c2c0 0x7f9aa806bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 monclient: found mon.noname-c 2024-09-18T10:28:51.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.724+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98009ae0 con 0x7f9aa81371a0 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aa6ffd700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 msgr2=0x7f9aa806c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aa6ffd700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 0x7f9aa806c160 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9a98003000 tx=0x7f9a98017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98022070 con 0x7f9aa81371a0 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-18T10:28:51.746 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a980126c0 con 0x7f9aa81371a0 2024-09-18T10:28:51.747 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 5 monclient: authenticate success, global_id 76055 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 monclient: _renew_subs 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa806ea10 con 0x7f9aa81371a0 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98012e70 con 0x7f9aa81371a0 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.748 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aa5ffb700 4 set_mon_vals no callback set 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 monclient: _renew_subs 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa806cbe0 con 0x7f9aa81371a0 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 0x7f9aa809ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.749 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa8064750 con 0x7f9aa809cb00 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.728+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9a9801e640 con 0x7f9aa81371a0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9a8f7fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a94035b10 0x7f9a94037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9a9804a0b0 con 0x7f9aa81371a0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 0x7f9aa809ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient: get_auth_request con 0x7f9aa809cb00 auth_method 0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a94035b10 0x7f9a94037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa7fff700 10 monclient: get_auth_request con 0x7f9a94035b10 auth_method 0 2024-09-18T10:28:51.750 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_done global_id 194464 payload 824 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aa806ec30 con 0x7f9aa809cb00 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa7fff700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a94035b10 0x7f9a94037fc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9aa806a3e0 tx=0x7f9aa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 msgr2=0x7f9aa809ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.732+0000 7f9aa77fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 0x7f9aa809ced0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00e740 tx=0x7f9a9c007ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c005260 con 0x7f9aa809cb00 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.751 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9a9c0053c0 con 0x7f9aa809cb00 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 msgr2=0x7f9aa809ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 0x7f9aa809ced0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00e740 tx=0x7f9a9c007ed0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 msgr2=0x7f9aa809ced0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.736+0000 7f9a8f7fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809cb00 0x7f9aa809ced0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00e740 tx=0x7f9a9c007ed0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aacdf9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:51.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 0x7f9aa814bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aacdf9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80a4790 con 0x7f9aa809d410 2024-09-18T10:28:51.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 0x7f9aa814bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient: get_auth_request con 0x7f9aa809d410 auth_method 0 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 10 monclient(hunting): handle_auth_done global_id 194467 payload 824 2024-09-18T10:28:51.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 1 -- 172.21.15.142:0/775472246 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9aa80a6620 con 0x7f9aa809d410 2024-09-18T10:28:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 msgr2=0x7f9aa814bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.748+0000 7f9aa77fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 0x7f9aa814bef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c017000 tx=0x7f9a9c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c013e30 con 0x7f9aa809d410 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 10 monclient: dump: 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.758 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f9a9c01c400 con 0x7f9aa809d410 2024-09-18T10:28:51.759 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 msgr2=0x7f9aa814bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 0x7f9aa814bef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c017000 tx=0x7f9a9c018000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 msgr2=0x7f9aa814bef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9a8f7fe700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa809d410 0x7f9aa814bef0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c017000 tx=0x7f9a9c018000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:51.760 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 1, 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.761 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.762 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:51.763 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:51.764 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.765 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.766 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.767 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:51.768 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.769 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.770 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:51.771 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:51.772 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a94035b10 msgr2=0x7f9a94037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9a94035b10 0x7f9a94037fc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9aa806a3e0 tx=0x7f9aa0009040 comp rx=0 tx=0).stop 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 10 monclient: shutdown 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 msgr2=0x7f9aa806c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 --2- 172.21.15.142:0/775472246 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa81371a0 0x7f9aa806c160 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9a98003000 tx=0x7f9a98017000 comp rx=0 tx=0).stop 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aa7fff700 1 -- 172.21.15.142:0/775472246 reap_dead start 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aa7fff700 1 -- 172.21.15.142:0/775472246 reap_dead start 2024-09-18T10:28:51.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 shutdown_connections 2024-09-18T10:28:51.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 >> 172.21.15.142:0/775472246 conn(0x7f9aa80a5720 msgr2=0x7f9aa80a3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 shutdown_connections 2024-09-18T10:28:51.774 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.752+0000 7f9aacdf9700 1 -- 172.21.15.142:0/775472246 wait complete. 2024-09-18T10:28:51.776 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:51.776 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:28:51.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:28:51.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.876+0000 7fec8f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:28:51.883 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.876+0000 7fec8f59e700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.876+0000 7fec8f59e700 10 monclient: monmap: 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:51.881405+0000 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:51.881405+0000 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:51.884 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 Processor -- start 2024-09-18T10:28:51.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 -- start start 2024-09-18T10:28:51.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient: init 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec9009c700 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec9009cad0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec90138550 0x7fec9013a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fec90138550 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec90097140 con 0x7fec9009c700 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec90097ec0 con 0x7fec9009cad0 2024-09-18T10:28:51.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec9009d5d0 con 0x7fec90138550 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8f59e700 10 monclient(hunting): authenticate will time out at 2020.809644s 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8e59c700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fec9009c700 0x7fec900977b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:33210/0 (socket says 172.21.15.142:33210) 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8e59c700 1 -- 172.21.15.142:0/285959232 learned_addr learned my addr 172.21.15.142:0/285959232 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 1 --2- 172.21.15.142:0/285959232 >> v2:172.21.15.150:3300/0 conn(0x7fec90138550 0x7fec9013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): get_auth_request con 0x7fec90138550 auth_method 0 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.880+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8e59c700 1 -- 172.21.15.142:0/285959232 >> v1:172.21.15.142:6789/0 conn(0x7fec9009c700 legacy=0x7fec900977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8dd9b700 1 -- 172.21.15.142:0/285959232 >> v1:172.21.15.150:6789/0 conn(0x7fec9009cad0 legacy=0x7fec9009cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.890 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_done global_id 194470 payload 274 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 -- 172.21.15.142:0/285959232 >> v1:172.21.15.150:6789/0 conn(0x7fec9009cad0 legacy=0x7fec9009cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 -- 172.21.15.142:0/285959232 >> v1:172.21.15.142:6789/0 conn(0x7fec9009c700 legacy=0x7fec900977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:51.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 -- 172.21.15.142:0/285959232 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9006f380 con 0x7fec90138550 2024-09-18T10:28:51.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 -- 172.21.15.142:0/285959232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec90138550 msgr2=0x7fec9013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8ed9d700 1 --2- 172.21.15.142:0/285959232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec90138550 0x7fec9013a930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec840099e0 tx=0x7fec840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8703bf51f226dc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 1 -- 172.21.15.142:0/285959232 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8400bba0 con 0x7fec90138550 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: dump: 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.894 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8f59e700 5 monclient: authenticate success, global_id 194470 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 1 -- 172.21.15.142:0/285959232 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8400bd20 con 0x7fec90138550 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8f59e700 4 set_mon_vals no callback set 2024-09-18T10:28:51.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 1 -- 172.21.15.142:0/285959232 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84011670 con 0x7fec90138550 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.884+0000 7fec8d59a700 10 monclient: dump: 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.896 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.897 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.897 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.897 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: shutdown 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- 172.21.15.142:0/285959232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec90138550 msgr2=0x7fec9013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 --2- 172.21.15.142:0/285959232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec90138550 0x7fec9013a930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec840099e0 tx=0x7fec840091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- 172.21.15.142:0/285959232 shutdown_connections 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 --2- 172.21.15.142:0/285959232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec90138550 0x7fec9013a930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- 172.21.15.142:0/285959232 >> 172.21.15.142:0/285959232 conn(0x7fec900718c0 msgr2=0x7fec900a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- 172.21.15.142:0/285959232 shutdown_connections 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- 172.21.15.142:0/285959232 wait complete. 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: build_initial_monmap 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: monmap: 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:51.900 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 Processor -- start 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- start start 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: init 2024-09-18T10:28:51.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 0x7fec9011ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fec9009c700 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec9009cad0 0x7fec9011f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fec9009cad0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fec90138550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec901377d0 con 0x7fec9009c700 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90120c00 con 0x7fec9009cad0 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec90120680 con 0x7fec90138550 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8f59e700 10 monclient(hunting): authenticate will time out at 2020.819076s 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 0x7fec9011ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:52224/0 (socket says 172.21.15.142:52224) 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 0x7fec9011ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:41466/0 (socket says 172.21.15.142:41466) 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 -- 172.21.15.142:0/3969714049 learned_addr learned my addr 172.21.15.142:0/3969714049 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): get_auth_request con 0x7fec90138550 auth_method 0 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec9009cad0 0x7fec9011f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): get_auth_request con 0x7fec9009c700 auth_method 0 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): get_auth_request con 0x7fec9009cad0 auth_method 0 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:51.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient(hunting): handle_auth_done global_id 76058 payload 824 2024-09-18T10:28:51.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 -- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec9009cad0 msgr2=0x7fec9011f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec9009cad0 0x7fec9011f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 -- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 msgr2=0x7fec9011ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 0x7fec9011ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 monclient: found mon.noname-c 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.888+0000 7fec8ed9d700 1 -- 172.21.15.142:0/3969714049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84009420 con 0x7fec90138550 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8ed9d700 1 -- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 msgr2=0x7fec9011f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:51.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8ed9d700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fec9009e2b0 tx=0x7fec84017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8401f070 con 0x7fec90138550 2024-09-18T10:28:51.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:51.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: dump: 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:51.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 5 monclient: authenticate success, global_id 76058 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8400bba0 con 0x7fec90138550 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84006e50 con 0x7fec90138550 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 10 monclient: _renew_subs 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9009e2e0 con 0x7fec90138550 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8cd99700 4 set_mon_vals no callback set 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 10 monclient: dump: 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.910 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 10 monclient: _renew_subs 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90122140 con 0x7fec90138550 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.892+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fec84023070 con 0x7fec90138550 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec7e7fc700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 0x7fec70037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec901382f0 con 0x7fec90138550 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fec8404a550 con 0x7fec90138550 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8e59c700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 0x7fec70037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8e59c700 10 monclient: get_auth_request con 0x7fec70035ac0 auth_method 0 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.896+0000 7fec8e59c700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 0x7fec70037f70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fec88005c50 tx=0x7fec88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:51.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.904+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fec8402d360 con 0x7fec90138550 2024-09-18T10:28:51.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.904+0000 7fec7e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:51.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:51.904+0000 7fec7e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:52.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.132+0000 7fec8f59e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:28:52.138 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.132+0000 7fec8f59e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:28:52.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.132+0000 7fec8f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:52.139 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.132+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fec90072a80 con 0x7fec90138550 2024-09-18T10:28:52.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec7e7fc700 1 -- 172.21.15.142:0/3969714049 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fec84015030 con 0x7fec90138550 2024-09-18T10:28:52.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec7e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:28:52.142 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.142 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=8,logm=52} crc {auth=1245054703,config=3140514010,health=3837640981,kv=4261215878,logm=2680336007}) 2024-09-18T10:28:52.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec7e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:52.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=7,logm=53} crc {auth=255911709,config=3140514009,health=2282731851,kv=264744547,logm=3610305929}) 2024-09-18T10:28:52.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=10,kv=8,logm=52} crc {auth=1245054703,config=3140514010,health=3837640981,kv=4261215878,logm=2680336007}) 2024-09-18T10:28:52.143 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=10,kv=8,logm=54} crc {auth=3846989989,config=3140514009,health=2970426933,kv=4261215878,logm=1252410856}) 2024-09-18T10:28:52.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.145 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477220382}) 2024-09-18T10:28:52.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786447857}) 2024-09-18T10:28:52.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2477220382}) 2024-09-18T10:28:52.146 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2555545326}) 2024-09-18T10:28:52.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 msgr2=0x7fec70037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:52.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 0x7fec70037f70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fec88005c50 tx=0x7fec88009040 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 10 monclient: shutdown 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 msgr2=0x7fec9011f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fec9009e2b0 tx=0x7fec84017000 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 shutdown_connections 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec70035ac0 0x7fec70037f70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec90138550 0x7fec9011f7a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec9009cad0 0x7fec9011f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 --2- 172.21.15.142:0/3969714049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec9009c700 0x7fec9011ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 >> 172.21.15.142:0/3969714049 conn(0x7fec900718c0 msgr2=0x7fec9009bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 shutdown_connections 2024-09-18T10:28:52.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:52.136+0000 7fec8f59e700 1 -- 172.21.15.142:0/3969714049 wait complete. 2024-09-18T10:28:52.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203230401}) 2024-09-18T10:28:52.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396017404}) 2024-09-18T10:28:52.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203230401}) 2024-09-18T10:28:52.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669425986}) 2024-09-18T10:28:52.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=359431528}) 2024-09-18T10:28:52.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716788937}) 2024-09-18T10:28:52.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=359431528}) 2024-09-18T10:28:52.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.144+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197286054}) 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579232795}) 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294434219}) 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579232795}) 2024-09-18T10:28:52.156 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.148+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106237435}) 2024-09-18T10:28:52.158 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626018521}) 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064710668}) 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626018521}) 2024-09-18T10:28:52.159 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2699600789}) 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674128559}) 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3041637214}) 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674128559}) 2024-09-18T10:28:52.162 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.152+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374048741}) 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937319206}) 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471216462}) 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937319206}) 2024-09-18T10:28:52.163 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226191616}) 2024-09-18T10:28:52.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088491313}) 2024-09-18T10:28:52.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392074809}) 2024-09-18T10:28:52.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.164 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088491313}) 2024-09-18T10:28:52.165 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205085681}) 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409330525}) 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.156+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812043275}) 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409330525}) 2024-09-18T10:28:52.166 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557566539}) 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768472783}) 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=85613130}) 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768472783}) 2024-09-18T10:28:52.167 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164847845}) 2024-09-18T10:28:52.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400043280}) 2024-09-18T10:28:52.168 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3623348118}) 2024-09-18T10:28:52.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400043280}) 2024-09-18T10:28:52.169 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.160+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036476417}) 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056122165}) 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2577191612}) 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056122165}) 2024-09-18T10:28:52.170 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518278176}) 2024-09-18T10:28:52.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605336398}) 2024-09-18T10:28:52.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302694908}) 2024-09-18T10:28:52.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.171 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605336398}) 2024-09-18T10:28:52.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874183090}) 2024-09-18T10:28:52.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283744064}) 2024-09-18T10:28:52.172 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248061706}) 2024-09-18T10:28:52.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283744064}) 2024-09-18T10:28:52.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.164+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215505895}) 2024-09-18T10:28:52.173 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255421523}) 2024-09-18T10:28:52.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1439069426}) 2024-09-18T10:28:52.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255421523}) 2024-09-18T10:28:52.174 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3040640586}) 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292996404}) 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872371661}) 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292996404}) 2024-09-18T10:28:52.175 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2523048077}) 2024-09-18T10:28:52.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304861375}) 2024-09-18T10:28:52.176 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2588373304}) 2024-09-18T10:28:52.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304861375}) 2024-09-18T10:28:52.177 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.168+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1066604891}) 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050850832}) 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=511759356}) 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050850832}) 2024-09-18T10:28:52.178 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122961805}) 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492198477}) 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243233267}) 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492198477}) 2024-09-18T10:28:52.179 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=363519671}) 2024-09-18T10:28:52.180 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985772080}) 2024-09-18T10:28:52.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926309583}) 2024-09-18T10:28:52.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985772080}) 2024-09-18T10:28:52.181 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.172+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000649036}) 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689876570}) 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025808903}) 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689876570}) 2024-09-18T10:28:52.182 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904630148}) 2024-09-18T10:28:52.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772210527}) 2024-09-18T10:28:52.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2402577237}) 2024-09-18T10:28:52.183 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772210527}) 2024-09-18T10:28:52.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881282133}) 2024-09-18T10:28:52.184 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182806346}) 2024-09-18T10:28:52.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2456177935}) 2024-09-18T10:28:52.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182806346}) 2024-09-18T10:28:52.185 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.176+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375222918}) 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295045977}) 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271147882}) 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295045977}) 2024-09-18T10:28:52.186 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.180+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829724876}) 2024-09-18T10:28:52.505 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.506 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519180483}) 2024-09-18T10:28:52.506 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2608296561}) 2024-09-18T10:28:52.506 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.506 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519180483}) 2024-09-18T10:28:52.506 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.496+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466122245}) 2024-09-18T10:28:52.516 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.516 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994401885}) 2024-09-18T10:28:52.516 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869434300}) 2024-09-18T10:28:52.516 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.516 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994401885}) 2024-09-18T10:28:52.517 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.508+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233349439}) 2024-09-18T10:28:52.524 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.524 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229428136}) 2024-09-18T10:28:52.524 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3402498525}) 2024-09-18T10:28:52.525 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.525 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229428136}) 2024-09-18T10:28:52.525 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.516+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118523057}) 2024-09-18T10:28:52.532 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.532 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3579227585}) 2024-09-18T10:28:52.533 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=961857455}) 2024-09-18T10:28:52.533 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.533 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3579227585}) 2024-09-18T10:28:52.533 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.524+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703344711}) 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665327305}) 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860590826}) 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665327305}) 2024-09-18T10:28:52.893 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.884+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014062990}) 2024-09-18T10:28:52.909 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485416255}) 2024-09-18T10:28:52.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673114756}) 2024-09-18T10:28:52.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:52.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485416255}) 2024-09-18T10:28:52.910 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:52.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585449549}) 2024-09-18T10:28:53.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766766117}) 2024-09-18T10:28:53.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764483006}) 2024-09-18T10:28:53.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.120 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766766117}) 2024-09-18T10:28:53.121 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.112+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855659443}) 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525643920}) 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1257186009}) 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525643920}) 2024-09-18T10:28:53.123 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2555998621}) 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325081410}) 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957163489}) 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325081410}) 2024-09-18T10:28:53.125 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.116+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337216317}) 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354772792}) 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1720935236}) 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354772792}) 2024-09-18T10:28:53.127 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960085388}) 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013173678}) 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070758107}) 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013173678}) 2024-09-18T10:28:53.129 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.120+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192251689}) 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137899959}) 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422102099}) 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137899959}) 2024-09-18T10:28:53.131 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676436092}) 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307335883}) 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208227785}) 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307335883}) 2024-09-18T10:28:53.133 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.124+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703284612}) 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800861738}) 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937749055}) 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800861738}) 2024-09-18T10:28:53.134 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168256353}) 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607127673}) 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740891143}) 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607127673}) 2024-09-18T10:28:53.135 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668263999}) 2024-09-18T10:28:53.136 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.136 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985641701}) 2024-09-18T10:28:53.136 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097899965}) 2024-09-18T10:28:53.136 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.137 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985641701}) 2024-09-18T10:28:53.137 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151744154}) 2024-09-18T10:28:53.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325699673}) 2024-09-18T10:28:53.147 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2283589057}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1325699673}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.128+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3916582830}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614622418}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1516088333}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2614622418}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3020611452}) 2024-09-18T10:28:53.148 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868324628}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=952339704}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868324628}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3262564748}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249320760}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059168673}) 2024-09-18T10:28:53.149 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249320760}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469728285}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127950439}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748305507}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127950439}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.132+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184953970}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898275475}) 2024-09-18T10:28:53.150 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839334953}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898275475}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783042984}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493758382}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1585410779}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493758382}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1671409134}) 2024-09-18T10:28:53.151 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207731646}) 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4156896648}) 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207731646}) 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3741841260}) 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484532568}) 2024-09-18T10:28:53.152 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605101277}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484532568}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.136+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874120218}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830972286}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3324126585}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830972286}) 2024-09-18T10:28:53.153 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.140+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693463607}) 2024-09-18T10:28:53.159 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:28:53.159 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: get_monmap_and_config 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: monmap: 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:53.259953+0000 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:53.259953+0000 2024-09-18T10:28:53.262 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:53.263 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 Processor -- start 2024-09-18T10:28:53.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 -- start start 2024-09-18T10:28:53.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: init 2024-09-18T10:28:53.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-18T10:28:53.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f573409c690 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:53.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 0x7f573409ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-b con 0x7f573409ca60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5734133930 0x7f5734135d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5734133930 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340980e0 con 0x7f573409c690 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5734097d70 con 0x7f5734133930 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5734097370 con 0x7f573409ca60 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5733fff700 10 monclient(hunting): authenticate will time out at 2022.187381s 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5732ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f573409c690 0x7f5734096c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37560/0 (socket says 172.21.15.142:37560) 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f5732ffd700 1 -- 172.21.15.142:0/8922438 learned_addr learned my addr 172.21.15.142:0/8922438 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:53.267 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 1 --2- 172.21.15.142:0/8922438 >> v2:172.21.15.142:3300/0 conn(0x7f5734133930 0x7f5734135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 0x7f573409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): get_auth_request con 0x7f5734133930 auth_method 0 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): get_auth_request con 0x7f573409ca60 auth_method 0 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.256+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 10 monclient(hunting): handle_auth_done global_id 224121 payload 274 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 -- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 msgr2=0x7f573409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 0x7f573409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 -- 172.21.15.142:0/8922438 >> v1:172.21.15.142:6789/0 conn(0x7f573409c690 legacy=0x7f5734096c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 -- 172.21.15.142:0/8922438 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734097a90 con 0x7f5734133930 2024-09-18T10:28:53.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57327fc700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 0x7f573409ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:53.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 -- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5734133930 msgr2=0x7f5734135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f57337fe700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5734133930 0x7f5734135d10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f572c009e90 tx=0x7f572c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc2a40700366085 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 1 -- 172.21.15.142:0/8922438 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00e070 con 0x7f5734133930 2024-09-18T10:28:53.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: dump: 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 5 monclient: authenticate success, global_id 224121 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 1 -- 172.21.15.142:0/8922438 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c00be00 con 0x7f5734133930 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 4 set_mon_vals no callback set 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 1 -- 172.21.15.142:0/8922438 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c012660 con 0x7f5734133930 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5731ffb700 10 monclient: dump: 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.275 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 10 monclient: shutdown 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 -- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5734133930 msgr2=0x7f5734135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5734133930 0x7f5734135d10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f572c009e90 tx=0x7f572c009780 comp rx=0 tx=0).stop 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 -- 172.21.15.142:0/8922438 shutdown_connections 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f573409ca60 0x7f573409ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 --2- 172.21.15.142:0/8922438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5734133930 0x7f5734135d10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 -- 172.21.15.142:0/8922438 >> 172.21.15.142:0/8922438 conn(0x7f5734071700 msgr2=0x7f57340a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 -- 172.21.15.142:0/8922438 shutdown_connections 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.260+0000 7f5733fff700 1 -- 172.21.15.142:0/8922438 wait complete. 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient: monmap: 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:53.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 Processor -- start 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 -- start start 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient: init 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 0x7f57341325e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-a con 0x7f573409c690 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:53.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573409ca60 0x7f573412ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-b con 0x7f573409ca60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5734133930 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734131680 con 0x7f573409c690 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734132b20 con 0x7f573409ca60 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5734131cd0 con 0x7f5734133930 2024-09-18T10:28:53.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5733fff700 10 monclient(hunting): authenticate will time out at 2022.193920s 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 0x7f57341325e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58260/0 (socket says 172.21.15.142:58260) 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 -- 172.21.15.142:0/1754249382 learned_addr learned my addr 172.21.15.142:0/1754249382 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): get_auth_request con 0x7f5734133930 auth_method 0 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 0x7f57341325e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48202/0 (socket says 172.21.15.142:48202) 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): get_auth_request con 0x7f573409c690 auth_method 0 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573409ca60 0x7f573412ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f5732ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): get_auth_request con 0x7f573409ca60 auth_method 0 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57327fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient(hunting): handle_auth_done global_id 76064 payload 824 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 -- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573409ca60 msgr2=0x7f573412ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573409ca60 0x7f573412ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 -- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 msgr2=0x7f57341325e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 0x7f57341325e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 monclient: found mon.noname-c 2024-09-18T10:28:53.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.264+0000 7f57337fe700 1 -- 172.21.15.142:0/1754249382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728009420 con 0x7f5734133930 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f57337fe700 1 -- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 msgr2=0x7f573412efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f57337fe700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f57340a2950 tx=0x7f572c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c012de0 con 0x7f5734133930 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: dump: 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.285 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572c021070 con 0x7f5734133930 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 5 monclient: authenticate success, global_id 76064 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _renew_subs 2024-09-18T10:28:53.286 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c009a20 con 0x7f5734133930 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c026680 con 0x7f5734133930 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f571affd700 10 monclient: dump: 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.287 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _renew_subs 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57340a14b0 con 0x7f5734133930 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f57317fa700 4 set_mon_vals no callback set 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.268+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573409b2e0 con 0x7f5734133930 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.272+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f572c023030 con 0x7f5734133930 2024-09-18T10:28:53.288 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.272+0000 7f571affd700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 0x7f5720037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.272+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f572c04b120 con 0x7f5734133930 2024-09-18T10:28:53.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.272+0000 7f5732ffd700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 0x7f5720037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.272+0000 7f5732ffd700 10 monclient: get_auth_request con 0x7f5720035b10 auth_method 0 2024-09-18T10:28:53.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.276+0000 7f5732ffd700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 0x7f5720037fc0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5724007880 tx=0x7f5724007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.284+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f572c01bb80 con 0x7f5734133930 2024-09-18T10:28:53.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.284+0000 7f571affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.284+0000 7f571affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562504130}) 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169688114}) 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562504130}) 2024-09-18T10:28:53.307 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871184230}) 2024-09-18T10:28:53.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798812111}) 2024-09-18T10:28:53.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=125102535}) 2024-09-18T10:28:53.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.309 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798812111}) 2024-09-18T10:28:53.310 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.300+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290177394}) 2024-09-18T10:28:53.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991386788}) 2024-09-18T10:28:53.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691865964}) 2024-09-18T10:28:53.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.311 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991386788}) 2024-09-18T10:28:53.312 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482808492}) 2024-09-18T10:28:53.313 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356459230}) 2024-09-18T10:28:53.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963355828}) 2024-09-18T10:28:53.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356459230}) 2024-09-18T10:28:53.314 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.304+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014925952}) 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333800002}) 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725187911}) 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333800002}) 2024-09-18T10:28:53.317 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.308+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559535094}) 2024-09-18T10:28:53.319 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.319 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703366621}) 2024-09-18T10:28:53.319 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=810925033}) 2024-09-18T10:28:53.319 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.319 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703366621}) 2024-09-18T10:28:53.320 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3968061201}) 2024-09-18T10:28:53.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376626285}) 2024-09-18T10:28:53.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586751787}) 2024-09-18T10:28:53.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.321 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376626285}) 2024-09-18T10:28:53.322 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.312+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4119473491}) 2024-09-18T10:28:53.322 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851477245}) 2024-09-18T10:28:53.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101517305}) 2024-09-18T10:28:53.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851477245}) 2024-09-18T10:28:53.323 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3311673930}) 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105044272}) 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259209113}) 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105044272}) 2024-09-18T10:28:53.324 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635073361}) 2024-09-18T10:28:53.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341397157}) 2024-09-18T10:28:53.325 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315719627}) 2024-09-18T10:28:53.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341397157}) 2024-09-18T10:28:53.326 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.316+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495577665}) 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835205824}) 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1797140227}) 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1835205824}) 2024-09-18T10:28:53.327 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.320+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2300239008}) 2024-09-18T10:28:53.462 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.462 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993404457}) 2024-09-18T10:28:53.463 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101779110}) 2024-09-18T10:28:53.463 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.463 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993404457}) 2024-09-18T10:28:53.463 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.456+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804715030}) 2024-09-18T10:28:53.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.508+0000 7f5733fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.508+0000 7f5733fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.508+0000 7f5733fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.508+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5734072a80 con 0x7f5734133930 2024-09-18T10:28:53.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f571affd700 1 -- 172.21.15.142:0/1754249382 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f572c01b3a0 con 0x7f5734133930 2024-09-18T10:28:53.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f571affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:53.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f571affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:53.519 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236321009}) 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257897579}) 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236321009}) 2024-09-18T10:28:53.520 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.512+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4090967138}) 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 msgr2=0x7f5720037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 0x7f5720037fc0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5724007880 tx=0x7f5724007f90 comp rx=0 tx=0).stop 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 10 monclient: shutdown 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 msgr2=0x7f573412efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f57340a2950 tx=0x7f572c018000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.521 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 shutdown_connections 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5720035b10 0x7f5720037fc0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5734133930 0x7f573412efa0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f573409ca60 0x7f573412ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 --2- 172.21.15.142:0/1754249382 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f573409c690 0x7f57341325e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 >> 172.21.15.142:0/1754249382 conn(0x7f5734071700 msgr2=0x7f573409f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 shutdown_connections 2024-09-18T10:28:53.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.512+0000 7f5733fff700 1 -- 172.21.15.142:0/1754249382 wait complete. 2024-09-18T10:28:53.532 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:28:53.532 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-18T10:28:53.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:28:53.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778299648}) 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071645662}) 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778299648}) 2024-09-18T10:28:53.587 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.580+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937646075}) 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382671153}) 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428160437}) 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382671153}) 2024-09-18T10:28:53.590 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042500150}) 2024-09-18T10:28:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.593 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124418068}) 2024-09-18T10:28:53.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2063298028}) 2024-09-18T10:28:53.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124418068}) 2024-09-18T10:28:53.594 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.584+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360211009}) 2024-09-18T10:28:53.595 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795166814}) 2024-09-18T10:28:53.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326433638}) 2024-09-18T10:28:53.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795166814}) 2024-09-18T10:28:53.596 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2584732122}) 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001637159}) 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409025597}) 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001637159}) 2024-09-18T10:28:53.598 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.588+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1104043609}) 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535211600}) 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960279811}) 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535211600}) 2024-09-18T10:28:53.599 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605859005}) 2024-09-18T10:28:53.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.600 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1883955463}) 2024-09-18T10:28:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2460147163}) 2024-09-18T10:28:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1883955463}) 2024-09-18T10:28:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378939122}) 2024-09-18T10:28:53.601 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494103984}) 2024-09-18T10:28:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2610105792}) 2024-09-18T10:28:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494103984}) 2024-09-18T10:28:53.602 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.592+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011988886}) 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27504985}) 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1299851042}) 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27504985}) 2024-09-18T10:28:53.603 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781364143}) 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450641707}) 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454800511}) 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450641707}) 2024-09-18T10:28:53.604 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095435797}) 2024-09-18T10:28:53.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785434851}) 2024-09-18T10:28:53.605 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1452833099}) 2024-09-18T10:28:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785434851}) 2024-09-18T10:28:53.606 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.596+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131258631}) 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462054417}) 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537597201}) 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462054417}) 2024-09-18T10:28:53.607 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244754285}) 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505480545}) 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993567779}) 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505480545}) 2024-09-18T10:28:53.608 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.600+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102986889}) 2024-09-18T10:28:53.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: get_monmap_and_config 2024-09-18T10:28:53.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.639 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: monmap: 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:53.637630+0000 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:53.637630+0000 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:53.640 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 Processor -- start 2024-09-18T10:28:53.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 -- start start 2024-09-18T10:28:53.645 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: init 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9f500999e0 0x7f9f500a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-a con 0x7f9f500999e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9f5009c410 0x7f9f500a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-c con 0x7f9f5009c410 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f500a0c00 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f500a20c0 con 0x7f9f500a0c00 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f500a1040 con 0x7f9f500999e0 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5009a000 con 0x7f9f5009c410 2024-09-18T10:28:53.646 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f56d37700 10 monclient(hunting): authenticate will time out at 2022.563038s 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9f500999e0 0x7f9f500a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9f500999e0 0x7f9f500a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48210/0 (socket says 172.21.15.142:48210) 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 -- 172.21.15.142:0/2085045197 learned_addr learned my addr 172.21.15.142:0/2085045197 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): get_auth_request con 0x7f9f500999e0 auth_method 0 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55534700 1 --2- 172.21.15.142:0/2085045197 >> v2:172.21.15.150:3300/0 conn(0x7f9f5009c410 0x7f9f500a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.647 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55534700 10 monclient(hunting): get_auth_request con 0x7f9f5009c410 auth_method 0 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55534700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55534700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55534700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient(hunting): handle_auth_done global_id 224127 payload 274 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 -- 172.21.15.142:0/2085045197 >> v2:172.21.15.150:3300/0 conn(0x7f9f5009c410 msgr2=0x7f9f500a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 --2- 172.21.15.142:0/2085045197 >> v2:172.21.15.150:3300/0 conn(0x7f9f5009c410 0x7f9f500a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 -- 172.21.15.142:0/2085045197 >> v1:172.21.15.150:6789/0 conn(0x7f9f500a0c00 legacy=0x7f9f50138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.632+0000 7f9f55d35700 1 -- 172.21.15.142:0/2085045197 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f500a1c80 con 0x7f9f500999e0 2024-09-18T10:28:53.648 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f55d35700 1 -- 172.21.15.142:0/2085045197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500999e0 msgr2=0x7f9f500a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f55d35700 1 --2- 172.21.15.142:0/2085045197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500999e0 0x7f9f500a0180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f5006fbb0 tx=0x7f9f4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f788dda717005751 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 1 -- 172.21.15.142:0/2085045197 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c00ba30 con 0x7f9f500999e0 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: dump: 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.649 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 5 monclient: authenticate success, global_id 224127 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 1 -- 172.21.15.142:0/2085045197 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c00bb90 con 0x7f9f500999e0 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 4 set_mon_vals no callback set 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 1 -- 172.21.15.142:0/2085045197 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c011420 con 0x7f9f500999e0 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.650 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f54d33700 10 monclient: dump: 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.651 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 10 monclient: shutdown 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 -- 172.21.15.142:0/2085045197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500999e0 msgr2=0x7f9f500a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 --2- 172.21.15.142:0/2085045197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500999e0 0x7f9f500a0180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f5006fbb0 tx=0x7f9f4c009250 comp rx=0 tx=0).stop 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 -- 172.21.15.142:0/2085045197 shutdown_connections 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 --2- 172.21.15.142:0/2085045197 >> v2:172.21.15.150:3300/0 conn(0x7f9f5009c410 0x7f9f500a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 --2- 172.21.15.142:0/2085045197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500999e0 0x7f9f500a0180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 -- 172.21.15.142:0/2085045197 >> 172.21.15.142:0/2085045197 conn(0x7f9f500a5790 msgr2=0x7f9f500a5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 -- 172.21.15.142:0/2085045197 shutdown_connections 2024-09-18T10:28:53.652 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 1 -- 172.21.15.142:0/2085045197 wait complete. 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.636+0000 7f9f56d37700 10 monclient: monmap: 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:53.653 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 Processor -- start 2024-09-18T10:28:53.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 -- start start 2024-09-18T10:28:53.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient: init 2024-09-18T10:28:53.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f500999e0 0x7f9f50141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-b con 0x7f9f500999e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f5009c410 0x7f9f50142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-c con 0x7f9f5009c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): picked mon.noname-a con 0x7f9f500a0c00 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500994c0 con 0x7f9f500a0c00 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500a4e80 con 0x7f9f500999e0 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f50091650 con 0x7f9f5009c410 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56d37700 10 monclient(hunting): authenticate will time out at 2022.569361s 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f5009c410 0x7f9f50142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48222/0 (socket says 172.21.15.142:48222) 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f5009c410 0x7f9f50142140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58268/0 (socket says 172.21.15.142:58268) 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f500999e0 0x7f9f50141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 learned_addr learned my addr 172.21.15.142:0/3443652297 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): get_auth_request con 0x7f9f500a0c00 auth_method 0 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.661 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): get_auth_request con 0x7f9f5009c410 auth_method 0 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): get_auth_request con 0x7f9f500999e0 auth_method 0 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55534700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient(hunting): handle_auth_done global_id 224130 payload 824 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f5009c410 msgr2=0x7f9f50142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9f5009c410 0x7f9f50142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f500999e0 msgr2=0x7f9f50141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f500999e0 0x7f9f50141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.640+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0094f0 con 0x7f9f500a0c00 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 msgr2=0x7f9f5008ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56536700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c930 tx=0x7f9f480026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48012840 con 0x7f9f500a0c00 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: dump: 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.664 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f480129c0 con 0x7f9f500a0c00 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 5 monclient: authenticate success, global_id 224130 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 monclient: _renew_subs 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f500913d0 con 0x7f9f500a0c00 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48012d80 con 0x7f9f500a0c00 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f47fff700 4 set_mon_vals no callback set 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 10 monclient: dump: 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.666 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 monclient: _renew_subs 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f50142680 con 0x7f9f500a0c00 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 0x7f9f5008e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f500647f0 con 0x7f9f50092570 2024-09-18T10:28:53.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9f48011590 con 0x7f9f500a0c00 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 0x7f9f5008e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient: get_auth_request con 0x7f9f50092570 auth_method 0 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f34035ac0 0x7f9f34037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9f48048c20 con 0x7f9f500a0c00 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 10 monclient(hunting): handle_auth_done global_id 224133 payload 824 2024-09-18T10:28:53.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55d35700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f50090620 con 0x7f9f50092570 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55534700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f34035ac0 0x7f9f34037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.644+0000 7f9f55534700 10 monclient: get_auth_request con 0x7f9f34035ac0 auth_method 0 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f55d35700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 msgr2=0x7f9f5008e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f55d35700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 0x7f9f5008e2f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f5009fda0 tx=0x7f9f4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c021070 con 0x7f9f50092570 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 10 monclient: dump: 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.669 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f9f4c006c70 con 0x7f9f50092570 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f55534700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f34035ac0 0x7f9f34037f70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9f40006fd0 tx=0x7f9f40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 msgr2=0x7f9f5008e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 0x7f9f5008e2f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f5009fda0 tx=0x7f9f4c016000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 msgr2=0x7f9f5008e2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.670 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.648+0000 7f9f45ffb700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f50092570 0x7f9f5008e2f0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9f5009fda0 tx=0x7f9f4c016000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f56d37700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:53.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f56d37700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 0x7f9f5008ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f56d37700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f50072a80 con 0x7f9f5008e830 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 0x7f9f5008ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient: get_auth_request con 0x7f9f5008e830 auth_method 0 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.673 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 10 monclient(hunting): handle_auth_done global_id 224136 payload 824 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.656+0000 7f9f55d35700 1 -- 172.21.15.142:0/3443652297 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f50091790 con 0x7f9f5008e830 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f55d35700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 msgr2=0x7f9f5008ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f55d35700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 0x7f9f5008ec30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009fd0 tx=0x7f9f4c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c029040 con 0x7f9f5008e830 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 10 monclient: dump: 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.674 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4410 (secure 0 0 0) 0x7f9f4c025a00 con 0x7f9f5008e830 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 msgr2=0x7f9f5008ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 0x7f9f5008ec30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009fd0 tx=0x7f9f4c021000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 msgr2=0x7f9f5008ec30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f45ffb700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f5008e830 0x7f9f5008ec30 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009fd0 tx=0x7f9f4c021000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:53.676 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.677 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.678 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:53.679 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.680 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.681 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.682 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.683 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.684 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:53.685 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:53.686 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.687 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.688 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:53.688 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:53.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f34035ac0 msgr2=0x7f9f34037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9f34035ac0 0x7f9f34037f70 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9f40006fd0 tx=0x7f9f40008040 comp rx=0 tx=0).stop 2024-09-18T10:28:53.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 10 monclient: shutdown 2024-09-18T10:28:53.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56536700 1 -- 172.21.15.142:0/3443652297 reap_dead start 2024-09-18T10:28:53.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 msgr2=0x7f9f5008ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f4800c930 tx=0x7f9f480026e0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 shutdown_connections 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 --2- 172.21.15.142:0/3443652297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f500a0c00 0x7f9f5008ddb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 >> 172.21.15.142:0/3443652297 conn(0x7f9f500a5790 msgr2=0x7f9f5009d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 shutdown_connections 2024-09-18T10:28:53.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.660+0000 7f9f56d37700 1 -- 172.21.15.142:0/3443652297 wait complete. 2024-09-18T10:28:53.690 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:53.690 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:28:53.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:28:53.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: get_monmap_and_config 2024-09-18T10:28:53.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: monmap: 2024-09-18T10:28:53.798 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:53.796077+0000 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:53.796077+0000 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:53.799 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 Processor -- start 2024-09-18T10:28:53.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 -- start start 2024-09-18T10:28:53.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: init 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 0x7f3818098cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-c con 0x7f381809a800 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f381809abd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f381813b6e0 0x7f381813dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-a con 0x7f381813b6e0 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381809da40 con 0x7f381809abd0 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3818099a40 con 0x7f381813b6e0 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381809b8e0 con 0x7f381809a800 2024-09-18T10:28:53.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381f9a9700 10 monclient(hunting): authenticate will time out at 2022.721544s 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f381813b6e0 0x7f381813dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f381813b6e0 0x7f381813dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48264/0 (socket says 172.21.15.142:48264) 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 -- 172.21.15.142:0/2143138922 learned_addr learned my addr 172.21.15.142:0/2143138922 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): get_auth_request con 0x7f381813b6e0 auth_method 0 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381d745700 1 --2- 172.21.15.142:0/2143138922 >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 0x7f3818098cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient(hunting): handle_auth_done global_id 224139 payload 274 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 -- 172.21.15.142:0/2143138922 >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 msgr2=0x7f3818098cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 --2- 172.21.15.142:0/2143138922 >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 0x7f3818098cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 -- 172.21.15.142:0/2143138922 >> v1:172.21.15.150:6789/0 conn(0x7f381809abd0 legacy=0x7f3818099230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381df46700 1 -- 172.21.15.142:0/2143138922 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381809dc80 con 0x7f381813b6e0 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381d745700 10 monclient: get_auth_request con 0x7f381809a800 auth_method 0 2024-09-18T10:28:53.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.792+0000 7f381d745700 1 --2- 172.21.15.142:0/2143138922 >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 0x7f3818098cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:28:53.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381df46700 1 -- 172.21.15.142:0/2143138922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f381813b6e0 msgr2=0x7f381813dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381df46700 1 --2- 172.21.15.142:0/2143138922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f381813b6e0 0x7f381813dac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f38180a21c0 tx=0x7f3810009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2730f5d8e1f58594 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 1 -- 172.21.15.142:0/2143138922 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381000e070 con 0x7f381813b6e0 2024-09-18T10:28:53.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: dump: 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 5 monclient: authenticate success, global_id 224139 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 1 -- 172.21.15.142:0/2143138922 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381000bc60 con 0x7f381813b6e0 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 1 -- 172.21.15.142:0/2143138922 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810012430 con 0x7f381813b6e0 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 4 set_mon_vals no callback set 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f380ffff700 10 monclient: dump: 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.812 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 10 monclient: shutdown 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- 172.21.15.142:0/2143138922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f381813b6e0 msgr2=0x7f381813dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 --2- 172.21.15.142:0/2143138922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f381813b6e0 0x7f381813dac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f38180a21c0 tx=0x7f3810009840 comp rx=0 tx=0).stop 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- 172.21.15.142:0/2143138922 shutdown_connections 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 --2- 172.21.15.142:0/2143138922 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f381813b6e0 0x7f381813dac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 --2- 172.21.15.142:0/2143138922 >> v2:172.21.15.150:3300/0 conn(0x7f381809a800 0x7f3818098cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- 172.21.15.142:0/2143138922 >> 172.21.15.142:0/2143138922 conn(0x7f3818071b20 msgr2=0x7f38180a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- 172.21.15.142:0/2143138922 shutdown_connections 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- 172.21.15.142:0/2143138922 wait complete. 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 10 monclient: monmap: 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:53.814 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 Processor -- start 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 1 -- start start 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.796+0000 7f381f9a9700 10 monclient: init 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 0x7f38180940c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-c con 0x7f381809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f381813b6e0 0x7f3818137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-b con 0x7f381813b6e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): picked mon.noname-a con 0x7f3818094600 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:53.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3818090900 con 0x7f3818094600 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381813b1a0 con 0x7f381813b6e0 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3818090b50 con 0x7f381809abd0 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381f9a9700 10 monclient(hunting): authenticate will time out at 2022.728459s 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 0x7f38180940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 0x7f38180940c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58274/0 (socket says 172.21.15.142:58274) 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48270/0 (socket says 172.21.15.142:48270) 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 1 -- 172.21.15.142:0/237238886 learned_addr learned my addr 172.21.15.142:0/237238886 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): get_auth_request con 0x7f381809abd0 auth_method 0 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): get_auth_request con 0x7f3818094600 auth_method 0 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f381813b6e0 0x7f3818137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): get_auth_request con 0x7f381813b6e0 auth_method 0 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381cf44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient(hunting): handle_auth_done global_id 224142 payload 824 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 msgr2=0x7f38180940c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 0x7f38180940c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f381813b6e0 msgr2=0x7f3818137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f381813b6e0 0x7f3818137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810009ae0 con 0x7f3818094600 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381d745700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809abd0 0x7f38180940c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 msgr2=0x7f3818138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.800+0000 7f381df46700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3810003000 tx=0x7f3810017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810012c40 con 0x7f3818094600 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: dump: 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 5 monclient: authenticate success, global_id 224142 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 monclient: _renew_subs 2024-09-18T10:28:53.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3810012da0 con 0x7f3818094600 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3818090f70 con 0x7f3818094600 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381001d670 con 0x7f3818094600 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 10 monclient: dump: 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.821 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380f7fe700 4 set_mon_vals no callback set 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 monclient: _renew_subs 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.822 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3818139730 con 0x7f3818094600 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 0x7f38181388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3818064950 con 0x7f3818071b20 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 0x7f38181388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient: get_auth_request con 0x7f3818071b20 auth_method 0 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.825 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f381001d9e0 con 0x7f3818094600 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 10 monclient(hunting): handle_auth_done global_id 76073 payload 824 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37fc035ac0 0x7f37fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381d745700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3818093800 con 0x7f3818071b20 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f381004a700 con 0x7f3818094600 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381cf44700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37fc035ac0 0x7f37fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.804+0000 7f381cf44700 10 monclient: get_auth_request con 0x7f37fc035ac0 auth_method 0 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f381d745700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 msgr2=0x7f38181388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f381d745700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 0x7f38181388a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f380800aef0 tx=0x7f3808008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808013440 con 0x7f3818071b20 2024-09-18T10:28:53.826 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 10 monclient: dump: 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.827 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f380800c040 con 0x7f3818071b20 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 msgr2=0x7f38181388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 0x7f38181388a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f380800aef0 tx=0x7f3808008040 comp rx=0 tx=0).stop 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 msgr2=0x7f38181388a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f380d7fa700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3818071b20 0x7f38181388a0 secure :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f380800aef0 tx=0x7f3808008040 comp rx=0 tx=0).stop 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.808+0000 7f381cf44700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37fc035ac0 0x7f37fc037f70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f38181395e0 tx=0x7f3814007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381f9a9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381f9a9700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 0x7f381809c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381f9a9700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3818139920 con 0x7f381809bdd0 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 0x7f381809c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.828 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient: get_auth_request con 0x7f381809bdd0 auth_method 0 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.829 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 10 monclient(hunting): handle_auth_done global_id 76076 payload 824 2024-09-18T10:28:53.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.816+0000 7f381d745700 1 -- 172.21.15.142:0/237238886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f38180696a0 con 0x7f381809bdd0 2024-09-18T10:28:53.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381d745700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 msgr2=0x7f381809c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381d745700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 0x7f381809c1d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f380800aef0 tx=0x7f380801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808023040 con 0x7f381809bdd0 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 10 monclient: dump: 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.831 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7f3808013e80 con 0x7f381809bdd0 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 msgr2=0x7f381809c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 0x7f381809c1d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f380800aef0 tx=0x7f380801d000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 msgr2=0x7f381809c1d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f380d7fa700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f381809bdd0 0x7f381809c1d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:53.832 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.833 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.834 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:53.835 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.836 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.837 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.838 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.839 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.840 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.841 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.842 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.843 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 6 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.844 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:53.845 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:53.846 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37fc035ac0 msgr2=0x7f37fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f37fc035ac0 0x7f37fc037f70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f38181395e0 tx=0x7f3814007ea0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 10 monclient: shutdown 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 msgr2=0x7f3818138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381df46700 1 -- 172.21.15.142:0/237238886 reap_dead start 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.820+0000 7f381f9a9700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3810003000 tx=0x7f3810017000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.824+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 shutdown_connections 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.824+0000 7f381f9a9700 1 --2- 172.21.15.142:0/237238886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3818094600 0x7f3818138360 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.824+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 >> 172.21.15.142:0/237238886 conn(0x7f381809a800 msgr2=0x7f3818136820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.824+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 shutdown_connections 2024-09-18T10:28:53.847 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.824+0000 7f381f9a9700 1 -- 172.21.15.142:0/237238886 wait complete. 2024-09-18T10:28:53.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:53.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:28:53.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009785794}) 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775003354}) 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1009785794}) 2024-09-18T10:28:53.900 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.892+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232459302}) 2024-09-18T10:28:53.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170713004}) 2024-09-18T10:28:53.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017801834}) 2024-09-18T10:28:53.907 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170713004}) 2024-09-18T10:28:53.908 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.900+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627956007}) 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899235413}) 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226952291}) 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899235413}) 2024-09-18T10:28:53.925 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:28:53.916+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376502383}) 2024-09-18T10:28:53.954 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: get_monmap_and_config 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: monmap: 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:53.952453+0000 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:53.952453+0000 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:53.955 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:53.956 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 Processor -- start 2024-09-18T10:28:53.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 -- start start 2024-09-18T10:28:53.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: init 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f193c09c700 0x7f193c096df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): picked mon.noname-a con 0x7f193c09c700 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 0x7f193c09cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): picked mon.noname-c con 0x7f193c098b30 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f193c09d010 addr v1:172.21.15.142:6789/0 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c098100 con 0x7f193c09d010 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c098f00 con 0x7f193c09c700 2024-09-18T10:28:53.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c097330 con 0x7f193c098b30 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194329e700 10 monclient(hunting): authenticate will time out at 2022.879330s 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f193c09c700 0x7f193c096df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1942a9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f193c09d010 0x7f193c12e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:37568/0 (socket says 172.21.15.142:37568) 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1942a9d700 1 -- 172.21.15.142:0/837303776 learned_addr learned my addr 172.21.15.142:0/837303776 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): get_auth_request con 0x7f193c09c700 auth_method 0 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1941a9b700 1 --2- 172.21.15.142:0/837303776 >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 0x7f193c09cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1941a9b700 10 monclient(hunting): get_auth_request con 0x7f193c098b30 auth_method 0 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1941a9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1941a9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f1941a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient(hunting): handle_auth_done global_id 224145 payload 274 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 1 -- 172.21.15.142:0/837303776 >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 msgr2=0x7f193c09cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 1 --2- 172.21.15.142:0/837303776 >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 0x7f193c09cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 1 -- 172.21.15.142:0/837303776 >> v1:172.21.15.142:6789/0 conn(0x7f193c09d010 legacy=0x7f193c12e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.948+0000 7f194229c700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 1 monclient: found mon.noname-a 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:53.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 1 -- 172.21.15.142:0/837303776 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c09e450 con 0x7f193c09c700 2024-09-18T10:28:53.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f1941a9b700 1 --2- 172.21.15.142:0/837303776 >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 0x7f193c09cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 1 -- 172.21.15.142:0/837303776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c09c700 msgr2=0x7f193c096df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194229c700 1 --2- 172.21.15.142:0/837303776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c09c700 0x7f193c096df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f193c096b70 tx=0x7f19380091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b558ba3e52f698c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 1 -- 172.21.15.142:0/837303776 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800b8c0 con 0x7f193c09c700 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: dump: 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 5 monclient: authenticate success, global_id 224145 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 1 -- 172.21.15.142:0/837303776 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193800ba20 con 0x7f193c09c700 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:53.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 4 set_mon_vals no callback set 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 1 -- 172.21.15.142:0/837303776 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193800bdc0 con 0x7f193c09c700 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194129a700 10 monclient: dump: 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.968 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 10 monclient: shutdown 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 -- 172.21.15.142:0/837303776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c09c700 msgr2=0x7f193c096df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 --2- 172.21.15.142:0/837303776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c09c700 0x7f193c096df0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f193c096b70 tx=0x7f19380091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 -- 172.21.15.142:0/837303776 shutdown_connections 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 --2- 172.21.15.142:0/837303776 >> v2:172.21.15.150:3300/0 conn(0x7f193c098b30 0x7f193c09cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 --2- 172.21.15.142:0/837303776 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c09c700 0x7f193c096df0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 -- 172.21.15.142:0/837303776 >> 172.21.15.142:0/837303776 conn(0x7f193c0716a0 msgr2=0x7f193c0a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:53.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 -- 172.21.15.142:0/837303776 shutdown_connections 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.952+0000 7f194329e700 1 -- 172.21.15.142:0/837303776 wait complete. 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient: build_initial_monmap 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient: monmap: 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 Processor -- start 2024-09-18T10:28:53.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 -- start start 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient: init 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c098b30 0x7f193c13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): picked mon.noname-a con 0x7f193c098b30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 0x7f193c13b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): picked mon.noname-c con 0x7f193c09c700 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c09d010 0x7f193c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): picked mon.noname-b con 0x7f193c09d010 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c120660 con 0x7f193c098b30 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c097600 con 0x7f193c09d010 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c11e110 con 0x7f193c09c700 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194329e700 10 monclient(hunting): authenticate will time out at 2022.885642s 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 0x7f193c13b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 0x7f193c13b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:58296/0 (socket says 172.21.15.142:58296) 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 -- 172.21.15.142:0/1324414892 learned_addr learned my addr 172.21.15.142:0/1324414892 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c098b30 0x7f193c13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): get_auth_request con 0x7f193c09c700 auth_method 0 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): get_auth_request con 0x7f193c098b30 auth_method 0 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c09d010 0x7f193c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): get_auth_request con 0x7f193c09d010 auth_method 0 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1942a9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient(hunting): handle_auth_done global_id 76079 payload 824 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c09d010 msgr2=0x7f193c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c09d010 0x7f193c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c098b30 msgr2=0x7f193c13bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c098b30 0x7f193c13bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient: _finish_hunting 0 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 monclient: found mon.noname-c 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f1941a9b700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938009420 con 0x7f193c09c700 2024-09-18T10:28:53.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.956+0000 7f194229c700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f193c098b30 0x7f193c13bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f1941a9b700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 msgr2=0x7f193c13b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f1941a9b700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 0x7f193c13b2e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d860 tx=0x7f192c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c012800 con 0x7f193c09c700 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.978 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 5 monclient: authenticate success, global_id 76079 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 monclient: _renew_subs 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c012960 con 0x7f193c09c700 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c120940 con 0x7f193c09c700 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:53.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c012d00 con 0x7f193c09c700 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f1940a99700 4 set_mon_vals no callback set 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.980 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 monclient: _renew_subs 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c13b9c0 con 0x7f193c09c700 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 0x7f193c029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c064680 con 0x7f193c11f4b0 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f192c013690 con 0x7f193c09c700 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 0x7f193c029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient: get_auth_request con 0x7f193c11f4b0 auth_method 0 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f192c049160 con 0x7f193c09c700 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 10 monclient(hunting): handle_auth_done global_id 194494 payload 824 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f194229c700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f193c06e2b0 con 0x7f193c11f4b0 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f1942a9d700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.960+0000 7f1942a9d700 10 monclient: get_auth_request con 0x7f1920035ac0 auth_method 0 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f194229c700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 msgr2=0x7f193c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f194229c700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 0x7f193c029d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f193c096b70 tx=0x7f1938016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938021070 con 0x7f193c11f4b0 2024-09-18T10:28:53.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f1942a9d700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f193c11f190 tx=0x7f1934005340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f1938000cf0 con 0x7f193c11f4b0 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 msgr2=0x7f193c029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 0x7f193c029d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f193c096b70 tx=0x7f1938016000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 msgr2=0x7f193c029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.964+0000 7f19327fc700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c11f4b0 0x7f193c029d00 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f193c096b70 tx=0x7f1938016000 comp rx=0 tx=0).stop 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194329e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194329e700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 0x7f193c02a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194329e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c120ad0 con 0x7f193c13c590 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 0x7f193c02a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient: get_auth_request con 0x7f193c13c590 auth_method 0 2024-09-18T10:28:53.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 10 monclient(hunting): handle_auth_done global_id 194497 payload 824 2024-09-18T10:28:53.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.976+0000 7f194229c700 1 -- 172.21.15.142:0/1324414892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f193c1207d0 con 0x7f193c13c590 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194229c700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 msgr2=0x7f193c02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194229c700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 0x7f193c02a240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19380093a0 tx=0x7f193801e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938025da0 con 0x7f193c13c590 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 10 monclient: dump: 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:53.987 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:53.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4578 (secure 0 0 0) 0x7f193801b440 con 0x7f193c13c590 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 msgr2=0x7f193c02a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 0x7f193c02a240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19380093a0 tx=0x7f193801e070 comp rx=0 tx=0).stop 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 msgr2=0x7f193c02a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f19327fc700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c13c590 0x7f193c02a240 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19380093a0 tx=0x7f193801e070 comp rx=0 tx=0).stop 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:53.989 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 190, 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 3, 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.990 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.991 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:28:53.992 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:28:53.993 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:28:53.994 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.995 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.996 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.997 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.998 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:53.999 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:54.000 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:28:54.001 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:28:54.002 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 msgr2=0x7f1920037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f1920035ac0 0x7f1920037f70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f193c11f190 tx=0x7f1934005340 comp rx=0 tx=0).stop 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 10 monclient: shutdown 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 msgr2=0x7f193c13b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 --2- 172.21.15.142:0/1324414892 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f193c09c700 0x7f193c13b2e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f192c00d860 tx=0x7f192c008040 comp rx=0 tx=0).stop 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f1942a9d700 1 -- 172.21.15.142:0/1324414892 reap_dead start 2024-09-18T10:28:54.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f1942a9d700 1 -- 172.21.15.142:0/1324414892 reap_dead start 2024-09-18T10:28:54.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 shutdown_connections 2024-09-18T10:28:54.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 >> 172.21.15.142:0/1324414892 conn(0x7f193c0716a0 msgr2=0x7f193c09b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:54.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 shutdown_connections 2024-09-18T10:28:54.004 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:53.980+0000 7f194329e700 1 -- 172.21.15.142:0/1324414892 wait complete. 2024-09-18T10:28:54.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:28:54.004 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-18T10:28:54.004 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-18T10:28:54.004 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-18T10:28:54.005 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:28:54.005 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:28:54.005 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:28:54.028 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:28:54.028 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-18T10:28:54.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:54.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.120+0000 7fb53d021700 10 monclient: get_monmap_and_config 2024-09-18T10:28:54.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.120+0000 7fb53d021700 10 monclient: build_initial_monmap 2024-09-18T10:28:54.129 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.120+0000 7fb53d021700 10 monclient: monmap: 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:54.127245+0000 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:54.127245+0000 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:54.130 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 Processor -- start 2024-09-18T10:28:54.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 -- start start 2024-09-18T10:28:54.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient: init 2024-09-18T10:28:54.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:54.133 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb53806a1b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5380699b0 0x7fb5381374a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-c con 0x7fb5380699b0 addr v2:172.21.15.150:3300/0 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 0x7fb538139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-b con 0x7fb5381379e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53806dc00 con 0x7fb53806a1b0 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb538069e30 con 0x7fb5380699b0 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53806b770 con 0x7fb5381379e0 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53d021700 10 monclient(hunting): authenticate will time out at 2023.053219s 2024-09-18T10:28:54.134 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53759e700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 msgr2=0x7fb538139dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb53759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 0x7fb538139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb5380699b0 0x7fb5381374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb536d9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb53806a1b0 0x7fb53806a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.142:39748/0 (socket says 172.21.15.142:39748) 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb536d9d700 1 -- 172.21.15.142:0/2071206594 learned_addr learned my addr 172.21.15.142:0/2071206594 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): get_auth_request con 0x7fb5380699b0 auth_method 0 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:54.135 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient(hunting): handle_auth_done global_id 194500 payload 274 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 -- 172.21.15.142:0/2071206594 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 msgr2=0x7fb538139dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 --2- 172.21.15.142:0/2071206594 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 0x7fb538139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 -- 172.21.15.142:0/2071206594 >> v1:172.21.15.150:6789/0 conn(0x7fb53806a1b0 legacy=0x7fb53806a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 monclient: found mon.noname-c 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.136 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.124+0000 7fb52ffff700 1 -- 172.21.15.142:0/2071206594 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53806d5b0 con 0x7fb5380699b0 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb52ffff700 1 -- 172.21.15.142:0/2071206594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5380699b0 msgr2=0x7fb5381374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb52ffff700 1 --2- 172.21.15.142:0/2071206594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5380699b0 0x7fb5381374a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5200098e0 tx=0x7fb5200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=490bacab16bcbbf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 1 -- 172.21.15.142:0/2071206594 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000ba70 con 0x7fb5380699b0 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: dump: 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.141 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: _finish_auth 0 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 1 -- 172.21.15.142:0/2071206594 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52000bbf0 con 0x7fb5380699b0 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 5 monclient: authenticate success, global_id 194500 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 1 -- 172.21.15.142:0/2071206594 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520011420 con 0x7fb5380699b0 2024-09-18T10:28:54.142 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 4 set_mon_vals no callback set 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53659c700 10 monclient: dump: 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:54.143 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 10 monclient: shutdown 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 -- 172.21.15.142:0/2071206594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5380699b0 msgr2=0x7fb5381374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 --2- 172.21.15.142:0/2071206594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5380699b0 0x7fb5381374a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb5200098e0 tx=0x7fb5200091a0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 -- 172.21.15.142:0/2071206594 shutdown_connections 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 --2- 172.21.15.142:0/2071206594 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5381379e0 0x7fb538139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 --2- 172.21.15.142:0/2071206594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5380699b0 0x7fb5381374a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 -- 172.21.15.142:0/2071206594 >> 172.21.15.142:0/2071206594 conn(0x7fb5380717f0 msgr2=0x7fb5380a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 -- 172.21.15.142:0/2071206594 shutdown_connections 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 -- 172.21.15.142:0/2071206594 wait complete. 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 10 monclient: build_initial_monmap 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 10 monclient: monmap: 2024-09-18T10:28:54.144 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.128+0000 7fb53d021700 1 Processor -- start 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 -- start start 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient: init 2024-09-18T10:28:54.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 0x7fb53811a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-c con 0x7fb5380699b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 0x7fb53811acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-b con 0x7fb53806a1b0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): picked mon.noname-a con 0x7fb5381379e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53811c350 con 0x7fb5381379e0 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb536d9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 msgr2=0x7fb53811a780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:28:54.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53806c900 con 0x7fb53806a1b0 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5381332e0 con 0x7fb5380699b0 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb536d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 0x7fb53811a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53d021700 10 monclient(hunting): authenticate will time out at 2023.061055s 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48338/0 (socket says 172.21.15.142:48338) 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 1 -- 172.21.15.142:0/1127496373 learned_addr learned my addr 172.21.15.142:0/1127496373 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 0x7fb53811acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): get_auth_request con 0x7fb5381379e0 auth_method 0 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:54.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): get_auth_request con 0x7fb53806a1b0 auth_method 0 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient(hunting): handle_auth_done global_id 194503 payload 824 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 -- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 msgr2=0x7fb53811a780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 0x7fb53811a780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 -- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 msgr2=0x7fb53811b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 monclient: found mon.noname-b 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb52ffff700 1 -- 172.21.15.142:0/1127496373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb520009420 con 0x7fb53806a1b0 2024-09-18T10:28:54.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.132+0000 7fb53759e700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:28:54.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52ffff700 1 -- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 msgr2=0x7fb53811acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:54.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52ffff700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 0x7fb53811acc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5200093f0 tx=0x7fb520016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52001b440 con 0x7fb53806a1b0 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: dump: 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:54.152 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: _finish_auth 0 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52001b5c0 con 0x7fb53806a1b0 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 5 monclient: authenticate success, global_id 194503 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52001b980 con 0x7fb53806a1b0 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _renew_subs 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb535d9b700 4 set_mon_vals no callback set 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53811d010 con 0x7fb53806a1b0 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:54.153 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 10 monclient: dump: 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _renew_subs 2024-09-18T10:28:54.154 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb538132e60 con 0x7fb53806a1b0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5380696a0 con 0x7fb53806a1b0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fb52002e440 con 0x7fb53806a1b0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.136+0000 7fb52effd700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 0x7fb518037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.140+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fb52004aec0 con 0x7fb53806a1b0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.140+0000 7fb536d9d700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 0x7fb518037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.140+0000 7fb536d9d700 10 monclient: get_auth_request con 0x7fb518035ac0 auth_method 0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.140+0000 7fb536d9d700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 0x7fb518037f70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb528004860 tx=0x7fb528007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:54.155 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.140+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 DispatchQueue::entry inject delay of 0.002000 2024-09-18T10:28:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.148+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fb520006ad0 con 0x7fb53806a1b0 2024-09-18T10:28:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.148+0000 7fb52effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:54.156 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.148+0000 7fb52effd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:54.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb53d021700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:54.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb53d021700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:54.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb53d021700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:28:54.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb5380696a0 con 0x7fb53806a1b0 2024-09-18T10:28:54.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb52effd700 1 -- 172.21.15.142:0/1127496373 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fb520023390 con 0x7fb53806a1b0 2024-09-18T10:28:54.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb52effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:54.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.376+0000 7fb52effd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:54.386 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:54.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 msgr2=0x7fb518037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 0x7fb518037f70 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb528004860 tx=0x7fb528007c10 comp rx=0 tx=0).stop 2024-09-18T10:28:54.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 10 monclient: shutdown 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 msgr2=0x7fb53811acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 0x7fb53811acc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5200093f0 tx=0x7fb520016000 comp rx=0 tx=0).stop 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 shutdown_connections 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fb518035ac0 0x7fb518037f70 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5381379e0 0x7fb53811b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb53806a1b0 0x7fb53811acc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 --2- 172.21.15.142:0/1127496373 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb5380699b0 0x7fb53811a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 >> 172.21.15.142:0/1127496373 conn(0x7fb5380717f0 msgr2=0x7fb5380a1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 shutdown_connections 2024-09-18T10:28:54.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:54.380+0000 7fb53d021700 1 -- 172.21.15.142:0/1127496373 wait complete. 2024-09-18T10:28:57.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:28:57.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.504+0000 7f7d24aaa700 10 monclient: get_monmap_and_config 2024-09-18T10:28:57.512 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.504+0000 7f7d24aaa700 10 monclient: build_initial_monmap 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.504+0000 7f7d24aaa700 10 monclient: monmap: 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:28:57.510453+0000 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:28:57.510453+0000 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:28:57.513 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.504+0000 7f7d24aaa700 1 Processor -- start 2024-09-18T10:28:57.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 -- start start 2024-09-18T10:28:57.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient: init 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d2009d920 addr v1:172.21.15.150:6789/0 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d2006f660 0x7f7d20096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-a con 0x7f7d2006f660 addr v2:172.21.15.142:3300/0 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 0x7f7d20136590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d200971f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2009b0b0 con 0x7f7d2009d920 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d20095ad0 con 0x7f7d2006f660 2024-09-18T10:28:57.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2009a7e0 con 0x7f7d200971f0 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d24aaa700 10 monclient(hunting): authenticate will time out at 2026.437381s 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 msgr2=0x7f7d20136590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 0x7f7d20136590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d2006f660 0x7f7d20096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7d2006f660 0x7f7d20096cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48344/0 (socket says 172.21.15.142:48344) 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 -- 172.21.15.142:0/226586747 learned_addr learned my addr 172.21.15.142:0/226586747 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): get_auth_request con 0x7f7d2006f660 auth_method 0 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:57.518 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient(hunting): handle_auth_done global_id 224154 payload 274 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 -- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 msgr2=0x7f7d20136590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 --2- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 0x7f7d20136590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 -- 172.21.15.142:0/226586747 >> v1:172.21.15.150:6789/0 conn(0x7f7d2009d920 legacy=0x7f7d200953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient: _finish_hunting 0 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 monclient: found mon.noname-a 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.519 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.508+0000 7f7d1effd700 1 -- 172.21.15.142:0/226586747 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2009aaf0 con 0x7f7d2006f660 2024-09-18T10:28:57.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1effd700 1 -- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 msgr2=0x7f7d20096cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1effd700 1 --2- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d20096cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d140099b0 tx=0x7f7d14009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84705215427acc76 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 1 -- 172.21.15.142:0/226586747 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400bb50 con 0x7f7d2006f660 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: dump: 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.524 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 1 -- 172.21.15.142:0/226586747 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1400bcb0 con 0x7f7d2006f660 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 5 monclient: authenticate success, global_id 224154 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: get_monmap_and_config success 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 1 -- 172.21.15.142:0/226586747 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14011670 con 0x7f7d2006f660 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:57.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 4 set_mon_vals no callback set 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d1e7fc700 10 monclient: dump: 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: shutdown 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 msgr2=0x7f7d20096cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d20096cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d140099b0 tx=0x7f7d14009250 comp rx=0 tx=0).stop 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- 172.21.15.142:0/226586747 shutdown_connections 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d200971f0 0x7f7d20136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/226586747 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d20096cb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- 172.21.15.142:0/226586747 >> 172.21.15.142:0/226586747 conn(0x7f7d200a1fe0 msgr2=0x7f7d20070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- 172.21.15.142:0/226586747 shutdown_connections 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- 172.21.15.142:0/226586747 wait complete. 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: build_initial_monmap 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: monmap: 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:28:57.527 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 Processor -- start 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 1 -- start start 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: init 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.512+0000 7f7d24aaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:28:57.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-a con 0x7f7d2006f660 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d200971f0 0x7f7d2013dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-b con 0x7f7d200971f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 0x7f7d20089b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): picked mon.noname-c con 0x7f7d2009d920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d200a0bc0 con 0x7f7d2006f660 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2008e310 con 0x7f7d200971f0 2024-09-18T10:28:57.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2008d0f0 con 0x7f7d2009d920 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient(hunting): authenticate will time out at 2026.444151s 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 msgr2=0x7f7d20089b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 0x7f7d20089b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48358/0 (socket says 172.21.15.142:48358) 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 -- 172.21.15.142:0/2612315411 learned_addr learned my addr 172.21.15.142:0/2612315411 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): get_auth_request con 0x7f7d2006f660 auth_method 0 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:57.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1effd700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d200971f0 0x7f7d2013dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1effd700 10 monclient(hunting): get_auth_request con 0x7f7d200971f0 auth_method 0 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient(hunting): handle_auth_done global_id 224157 payload 824 2024-09-18T10:28:57.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 -- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 msgr2=0x7f7d20089b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 0x7f7d20089b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 -- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d200971f0 msgr2=0x7f7d2013dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d200971f0 0x7f7d2013dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 monclient: found mon.noname-a 2024-09-18T10:28:57.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:28:57.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 -- 172.21.15.142:0/2612315411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d140094f0 con 0x7f7d2006f660 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 -- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 msgr2=0x7f7d2013d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1f7fe700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c5d0 tx=0x7f7d10007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d100117a0 con 0x7f7d2006f660 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: dump: 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:57.535 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: _finish_auth 0 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 5 monclient: authenticate success, global_id 224157 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d10011900 con 0x7f7d2006f660 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:28:57.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient: _renew_subs 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d10011cc0 con 0x7f7d2006f660 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2008d3d0 con 0x7f7d2006f660 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d077fe700 10 monclient: dump: 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:28:57.537 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d1dffb700 4 set_mon_vals no callback set 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient: _renew_subs 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.516+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2008c200 con 0x7f7d2006f660 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7d10012630 con 0x7f7d2006f660 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d24aaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d24aaa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d24aaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d077fe700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 0x7f7d0c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:28:57.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d20072a80 con 0x7f7d2006f660 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7d10048da0 con 0x7f7d2006f660 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d1effd700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 0x7f7d0c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.520+0000 7f7d1effd700 10 monclient: get_auth_request con 0x7f7d0c035a70 auth_method 0 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.524+0000 7f7d1effd700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 0x7f7d0c037f20 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7d140099b0 tx=0x7f7d14016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.528+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7d100108e0 con 0x7f7d2006f660 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.528+0000 7f7d077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:28:57.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.528+0000 7f7d077fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:28:57.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d24aaa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:28:57.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d24aaa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:57.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d24aaa700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:28:57.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7d200678a0 con 0x7f7d2006f660 2024-09-18T10:28:57.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d077fe700 1 -- 172.21.15.142:0/2612315411 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f7d100102e0 con 0x7f7d2006f660 2024-09-18T10:28:57.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:28:57.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.760+0000 7f7d077fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:28:57.769 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:28:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 msgr2=0x7f7d0c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 0x7f7d0c037f20 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7d140099b0 tx=0x7f7d14016000 comp rx=0 tx=0).stop 2024-09-18T10:28:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 10 monclient: shutdown 2024-09-18T10:28:57.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 msgr2=0x7f7d2013d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c5d0 tx=0x7f7d10007f60 comp rx=0 tx=0).stop 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 shutdown_connections 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7d0c035a70 0x7f7d0c037f20 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7d2009d920 0x7f7d20089b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d200971f0 0x7f7d2013dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 --2- 172.21.15.142:0/2612315411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d2006f660 0x7f7d2013d7d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 >> 172.21.15.142:0/2612315411 conn(0x7f7d200a1fe0 msgr2=0x7f7d2009f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 shutdown_connections 2024-09-18T10:28:57.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:28:57.764+0000 7f7d24aaa700 1 -- 172.21.15.142:0/2612315411 wait complete. 2024-09-18T10:29:00.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: get_monmap_and_config 2024-09-18T10:29:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: build_initial_monmap 2024-09-18T10:29:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: monmap: 2024-09-18T10:29:00.894 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:00.892354+0000 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:00.892354+0000 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:00.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 Processor -- start 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 -- start start 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: init 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa19009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): picked mon.noname-b con 0x7fa19009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa19009db80 0x7fa19009c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): picked mon.noname-a con 0x7fa19009db80 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 0x7fa190096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): picked mon.noname-c con 0x7fa19009c950 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa19006fa60 con 0x7fa19009db80 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1900958c0 con 0x7fa19009c950 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa190095ef0 con 0x7fa19009abd0 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa196231700 10 monclient(hunting): authenticate will time out at 2029.817950s 2024-09-18T10:29:00.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18f7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 msgr2=0x7fa19009bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa19009bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa19009db80 0x7fa19009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa19009db80 0x7fa19009c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48368/0 (socket says 172.21.15.142:48368) 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 -- 172.21.15.142:0/3336165770 learned_addr learned my addr 172.21.15.142:0/3336165770 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): get_auth_request con 0x7fa19009db80 auth_method 0 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:00.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18ffff700 1 --2- 172.21.15.142:0/3336165770 >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 0x7fa190096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient(hunting): handle_auth_done global_id 224160 payload 274 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 -- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 msgr2=0x7fa19009bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 --2- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa19009bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 -- 172.21.15.142:0/3336165770 >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 msgr2=0x7fa190096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 --2- 172.21.15.142:0/3336165770 >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 0x7fa190096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 monclient: found mon.noname-a 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18effd700 1 -- 172.21.15.142:0/3336165770 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19009afa0 con 0x7fa19009db80 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18ffff700 10 monclient: get_auth_request con 0x7fa19009c950 auth_method 0 2024-09-18T10:29:00.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.888+0000 7fa18ffff700 1 --2- 172.21.15.142:0/3336165770 >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 0x7fa190096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:29:00.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18effd700 1 -- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009db80 msgr2=0x7fa19009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18effd700 1 --2- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009db80 0x7fa19009c410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa19006fdb0 tx=0x7fa180009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4964cd6a4c41f085 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 1 -- 172.21.15.142:0/3336165770 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18000ba70 con 0x7fa19009db80 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: dump: 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:00.906 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 5 monclient: authenticate success, global_id 224160 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 1 -- 172.21.15.142:0/3336165770 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18000bbd0 con 0x7fa19009db80 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 4 set_mon_vals no callback set 2024-09-18T10:29:00.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 1 -- 172.21.15.142:0/3336165770 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180011420 con 0x7fa19009db80 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa18e7fc700 10 monclient: dump: 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:00.908 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 10 monclient: shutdown 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 -- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009db80 msgr2=0x7fa19009c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 --2- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009db80 0x7fa19009c410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa19006fdb0 tx=0x7fa180009250 comp rx=0 tx=0).stop 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 -- 172.21.15.142:0/3336165770 shutdown_connections 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 --2- 172.21.15.142:0/3336165770 >> v2:172.21.15.150:3300/0 conn(0x7fa19009c950 0x7fa190096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 --2- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009db80 0x7fa19009c410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 --2- 172.21.15.142:0/3336165770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa19009bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 -- 172.21.15.142:0/3336165770 >> 172.21.15.142:0/3336165770 conn(0x7fa1900a14e0 msgr2=0x7fa1900a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 -- 172.21.15.142:0/3336165770 shutdown_connections 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 -- 172.21.15.142:0/3336165770 wait complete. 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 10 monclient: build_initial_monmap 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 10 monclient: monmap: 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:00.910 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.892+0000 7fa196231700 1 Processor -- start 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 -- start start 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient: init 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa190141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): picked mon.noname-c con 0x7fa19009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): picked mon.noname-a con 0x7fa19009c950 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa19009db80 0x7fa1900989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): picked mon.noname-b con 0x7fa19009db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:00.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18f7fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 msgr2=0x7fa190141990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190090250 con 0x7fa19009c950 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190097c90 con 0x7fa19009db80 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900976e0 con 0x7fa19009abd0 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa190141990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa196231700 10 monclient(hunting): authenticate will time out at 2029.824849s 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48382/0 (socket says 172.21.15.142:48382) 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 -- 172.21.15.142:0/2727465625 learned_addr learned my addr 172.21.15.142:0/2727465625 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:00.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): get_auth_request con 0x7fa19009c950 auth_method 0 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa19009db80 0x7fa1900989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): get_auth_request con 0x7fa19009db80 auth_method 0 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:00.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient(hunting): handle_auth_done global_id 224163 payload 824 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 -- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 msgr2=0x7fa190141990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa190141990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 -- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa19009db80 msgr2=0x7fa1900989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa19009db80 0x7fa1900989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 monclient: found mon.noname-a 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.896+0000 7fa18effd700 1 -- 172.21.15.142:0/2727465625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1800094f0 con 0x7fa19009c950 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa18effd700 1 -- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 msgr2=0x7fa190141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:00.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa18effd700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b3c0 tx=0x7fa180017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:00.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180006a90 con 0x7fa19009c950 2024-09-18T10:29:00.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:00.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: dump: 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:00.917 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 5 monclient: authenticate success, global_id 224163 2024-09-18T10:29:00.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18001a630 con 0x7fa19009c950 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _renew_subs 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190091340 con 0x7fa19009c950 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18001a9d0 con 0x7fa19009c950 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 10 monclient: dump: 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:00.921 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _renew_subs 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1900903e0 con 0x7fa19009c950 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa18dffb700 4 set_mon_vals no callback set 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa190091580 con 0x7fa19009c950 2024-09-18T10:29:00.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fa1800253f0 con 0x7fa19009c950 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 0x7fa17c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fa18004a9d0 con 0x7fa19009c950 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa18f7fe700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 0x7fa17c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.900+0000 7fa18f7fe700 10 monclient: get_auth_request con 0x7fa17c035a70 auth_method 0 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.904+0000 7fa18f7fe700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 0x7fa17c037f20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa184006fd0 tx=0x7fa184006db0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.912+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fa1800256a0 con 0x7fa19009c950 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.912+0000 7fa1777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:00.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:00.912+0000 7fa1777fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:01.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa196231700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:01.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa196231700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:01.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa196231700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:01.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa190091860 con 0x7fa19009c950 2024-09-18T10:29:01.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa1777fe700 1 -- 172.21.15.142:0/2727465625 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1541 (secure 0 0 0) 0x7fa180015070 con 0x7fa19009c950 2024-09-18T10:29:01.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa1777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:01.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.140+0000 7fa1777fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:01.149 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:01.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 msgr2=0x7fa17c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:01.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 0x7fa17c037f20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa184006fd0 tx=0x7fa184006db0 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 10 monclient: shutdown 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 msgr2=0x7fa190141ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b3c0 tx=0x7fa180017000 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 shutdown_connections 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fa17c035a70 0x7fa17c037f20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa19009db80 0x7fa1900989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa19009c950 0x7fa190141ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 --2- 172.21.15.142:0/2727465625 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa19009abd0 0x7fa190141990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 >> 172.21.15.142:0/2727465625 conn(0x7fa1900a14e0 msgr2=0x7fa19006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 shutdown_connections 2024-09-18T10:29:01.152 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:01.144+0000 7fa196231700 1 -- 172.21.15.142:0/2727465625 wait complete. 2024-09-18T10:29:04.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:04.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.268+0000 7fcd422cf700 10 monclient: get_monmap_and_config 2024-09-18T10:29:04.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.268+0000 7fcd422cf700 10 monclient: build_initial_monmap 2024-09-18T10:29:04.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.268+0000 7fcd422cf700 10 monclient: monmap: 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:04.274385+0000 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:04.274385+0000 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:04.277 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:04.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.268+0000 7fcd422cf700 1 Processor -- start 2024-09-18T10:29:04.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 -- start start 2024-09-18T10:29:04.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient: init 2024-09-18T10:29:04.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-c con 0x7fcd3c0a1cf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd3c12f810 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-a con 0x7fcd3c0a02d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3c09fb90 con 0x7fcd3c12f810 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3c09d990 con 0x7fcd3c0a02d0 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3c09dc70 con 0x7fcd3c0a1cf0 2024-09-18T10:29:04.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd422cf700 10 monclient(hunting): authenticate will time out at 2033.200729s 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd40acc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fcd3c12f810 0x7fcd3c09f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:39954/0 (socket says 172.21.15.142:39954) 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45604/0 (socket says 172.21.15.142:45604) 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 learned_addr learned my addr 172.21.15.142:0/682613228 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a02d0 auth_method 0 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a1cf0 auth_method 0 2024-09-18T10:29:04.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:04.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.272+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 >> v1:172.21.15.142:6789/0 conn(0x7fcd3c12f810 legacy=0x7fcd3c09f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:04.480 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a02d0 auth_method 0 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a1cf0 auth_method 0 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.481 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.476+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a02d0 auth_method 0 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:04.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a1cf0 auth_method 0 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:29:04.888 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:04.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:04.889 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:04.880+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a02d0 auth_method 0 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:05.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 10 monclient(hunting): get_auth_request con 0x7fcd3c0a1cf0 auth_method 0 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:05.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:05.684+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:29:07.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): tick 2024-09-18T10:29:07.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 monclient(hunting): continuing hunt 2024-09-18T10:29:07.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): _reopen_session rank -1 2024-09-18T10:29:07.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 msgr2=0x7fcd3c12f030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:07.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.150:3300/0 conn(0x7fcd3c0a1cf0 0x7fcd3c12f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 -- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 msgr2=0x7fcd3c139420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 --2- 172.21.15.142:0/682613228 >> v2:172.21.15.142:3300/0 conn(0x7fcd3c0a02d0 0x7fcd3c139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 -- 172.21.15.142:0/682613228 >> v1:172.21.15.142:6789/0 conn(0x7fcd3c12f810 legacy=0x7fcd3c09f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient: _add_conns ranks=[2,1] 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): picked mon.noname-b con 0x7fcd20003760 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd200062b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 -- 172.21.15.142:0/682613228 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3c0a20c0 con 0x7fcd200062b0 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 1 -- 172.21.15.142:0/682613228 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c09d990 con 0x7fcd20003760 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd32ffd700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:07.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 msgr2=0x7fcd20005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:07.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:07.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.272+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 >> v1:172.21.15.150:6789/0 conn(0x7fcd200062b0 legacy=0x7fcd20008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:07.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.476+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 msgr2=0x7fcd20005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:07.482 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.476+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-18T10:29:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.880+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 msgr2=0x7fcd20005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:07.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:07.880+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-18T10:29:08.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.684+0000 7fcd412cd700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 msgr2=0x7fcd20005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:08.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.684+0000 7fcd412cd700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-18T10:29:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.952+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943060904 0 0) 0x7fcd3c0a20c0 con 0x7fcd200062b0 2024-09-18T10:29:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.952+0000 7fcd33fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.952+0000 7fcd33fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:08.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.952+0000 7fcd33fff700 10 monclient(hunting): my global_id is 194512 2024-09-18T10:29:08.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.952+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28003570 con 0x7fcd200062b0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2027545335 0 0) 0x7fcd28003570 con 0x7fcd200062b0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 msgr2=0x7fcd20005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd20003760 0x7fcd20005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2000b0b0 con 0x7fcd200062b0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: _finish_auth 0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fcd2c001760 con 0x7fcd200062b0 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:08.965 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: dump: 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:29:08.966 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 >> v1:172.21.15.150:6789/0 conn(0x7fcd200062b0 legacy=0x7fcd20008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28003a60 0x7fcd28003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): picked mon.a con 0x7fcd28003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 reap_dead start 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 0x7fcd28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): picked mon.b con 0x7fcd28004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd2800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): picked mon.c con 0x7fcd28005310 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c09d990 con 0x7fcd28003a60 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800c2d0 con 0x7fcd28004810 2024-09-18T10:29:08.967 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800c490 con 0x7fcd28005310 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd33fff700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd41ace700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 msgr2=0x7fcd2800ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd41ace700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd2800ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 0x7fcd28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): get_auth_request con 0x7fcd28004810 auth_method 0 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:08.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient(hunting): handle_auth_done global_id 194512 payload 293 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 msgr2=0x7fcd2800ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd2800ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28003a60 msgr2=0x7fcd28003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28003a60 0x7fcd28003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient: _finish_hunting 0 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 monclient: found mon.b 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28003570 con 0x7fcd28004810 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 10 monclient: _finish_auth 0 2024-09-18T10:29:08.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd40acc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd422cf700 5 monclient: authenticate success, global_id 194512 2024-09-18T10:29:08.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.956+0000 7fcd422cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:08.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd40acc700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 msgr2=0x7fcd28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:08.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd40acc700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 0x7fcd28004c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c001570 tx=0x7fcd2c007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=802a2c13a0c10725 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:08.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c00cd10 con 0x7fcd28004810 2024-09-18T10:29:08.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:08.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: dump: 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:08.976 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c00ce90 con 0x7fcd28004810 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 4 set_mon_vals no callback set 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 1 -- 172.21.15.142:0/682613228 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c005b70 con 0x7fcd28004810 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd33fff700 10 monclient: dump: 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:08.977 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 10 monclient: shutdown 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 -- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 msgr2=0x7fcd28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 0x7fcd28004c80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c001570 tx=0x7fcd2c007a00 comp rx=0 tx=0).stop 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 -- 172.21.15.142:0/682613228 shutdown_connections 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd2800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28004810 0x7fcd28004c80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 --2- 172.21.15.142:0/682613228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28003a60 0x7fcd28003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 -- 172.21.15.142:0/682613228 >> 172.21.15.142:0/682613228 conn(0x7fcd3c071820 msgr2=0x7fcd3c0a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 -- 172.21.15.142:0/682613228 shutdown_connections 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.960+0000 7fcd422cf700 1 -- 172.21.15.142:0/682613228 wait complete. 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient: build_initial_monmap 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient: monmap: 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:08.979 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 Processor -- start 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 -- start start 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient: init 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 0x7fcd3c141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-b con 0x7fcd28003a60 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-a con 0x7fcd28004810 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd3c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): picked mon.noname-c con 0x7fcd28005310 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:08.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c06f500 con 0x7fcd28004810 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c141f00 con 0x7fcd28003a60 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c06d770 con 0x7fcd28005310 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd422cf700 10 monclient(hunting): authenticate will time out at 2037.894541s 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd41ace700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 msgr2=0x7fcd3c06bd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd41ace700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd3c06bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45654/0 (socket says 172.21.15.142:45654) 2024-09-18T10:29:08.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 -- 172.21.15.142:0/4210025411 learned_addr learned my addr 172.21.15.142:0/4210025411 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): get_auth_request con 0x7fcd28004810 auth_method 0 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 0x7fcd3c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 10 monclient(hunting): get_auth_request con 0x7fcd28003a60 auth_method 0 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient(hunting): handle_auth_done global_id 224166 payload 824 2024-09-18T10:29:08.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 -- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 msgr2=0x7fcd3c06bd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd3c06bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 -- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 msgr2=0x7fcd3c141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 0x7fcd3c141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient: _finish_hunting 0 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 monclient: found mon.noname-a 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd40acc700 1 -- 172.21.15.142:0/4210025411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c06e2c0 con 0x7fcd28004810 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.964+0000 7fcd412cd700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 0x7fcd3c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd40acc700 1 -- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 msgr2=0x7fcd3c134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:08.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd40acc700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c460 tx=0x7fcd2c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c001720 con 0x7fcd28004810 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 10 monclient: dump: 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.986 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 10 monclient: _finish_auth 0 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.968+0000 7fcd422cf700 5 monclient: authenticate success, global_id 224166 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _renew_subs 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3c06f220 con 0x7fcd28004810 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _renew_subs 2024-09-18T10:29:08.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3c1415d0 con 0x7fcd28004810 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3c072a80 con 0x7fcd28004810 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c007c90 con 0x7fcd28004810 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c004e60 con 0x7fcd28004810 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd317fa700 10 monclient: dump: 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:08.988 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.972+0000 7fcd337fe700 4 set_mon_vals no callback set 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.976+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fcd2c007070 con 0x7fcd28004810 2024-09-18T10:29:08.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.976+0000 7fcd317fa700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 0x7fcd200387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:08.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.976+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fcd2c049b20 con 0x7fcd28004810 2024-09-18T10:29:08.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.976+0000 7fcd412cd700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 0x7fcd200387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:08.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.976+0000 7fcd412cd700 10 monclient: get_auth_request con 0x7fcd200362f0 auth_method 0 2024-09-18T10:29:08.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.980+0000 7fcd412cd700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 0x7fcd200387a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009880 tx=0x7fcd3800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:08.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.984+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fcd2c0053a0 con 0x7fcd28004810 2024-09-18T10:29:08.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.984+0000 7fcd317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:08.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:08.984+0000 7fcd317fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:09.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.204+0000 7fcd422cf700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.204+0000 7fcd422cf700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.204+0000 7fcd422cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.204+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd3c0a1a90 con 0x7fcd28004810 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd317fa700 1 -- 172.21.15.142:0/4210025411 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fcd2c023030 con 0x7fcd28004810 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd317fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd317fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:09.214 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 msgr2=0x7fcd200387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 0x7fcd200387a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009880 tx=0x7fcd3800b040 comp rx=0 tx=0).stop 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 10 monclient: shutdown 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 msgr2=0x7fcd3c134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c460 tx=0x7fcd2c015000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 shutdown_connections 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fcd200362f0 0x7fcd200387a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcd28005310 0x7fcd3c06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcd28004810 0x7fcd3c134fd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 --2- 172.21.15.142:0/4210025411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28003a60 0x7fcd3c141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 >> 172.21.15.142:0/4210025411 conn(0x7fcd3c071820 msgr2=0x7fcd3c09e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 shutdown_connections 2024-09-18T10:29:09.217 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.208+0000 7fcd422cf700 1 -- 172.21.15.142:0/4210025411 wait complete. 2024-09-18T10:29:09.228 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-18T10:29:09.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-18T10:29:09.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:29:09.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: get_monmap_and_config 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: build_initial_monmap 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: monmap: 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:09.333606+0000 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:09.333606+0000 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:09.336 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:09.337 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:09.337 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:09.337 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 Processor -- start 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 -- start start 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: init 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f459809abd0 0x7f45980960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-a con 0x7f459809abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 0x7f459809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-c con 0x7f459809b740 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 0x7f459809c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-b con 0x7f459809c410 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459809b500 con 0x7f459809abd0 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459809dc90 con 0x7f459809b740 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459809afa0 con 0x7f459809c410 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.328+0000 7f459dae5700 10 monclient(hunting): authenticate will time out at 2038.259926s 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459d2e4700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 msgr2=0x7f459809c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459d2e4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 0x7f459809c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:09.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f459809abd0 0x7f45980960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f459809abd0 0x7f45980960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45658/0 (socket says 172.21.15.142:45658) 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 -- 172.21.15.142:0/739935396 learned_addr learned my addr 172.21.15.142:0/739935396 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): get_auth_request con 0x7f459809abd0 auth_method 0 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 1 --2- 172.21.15.142:0/739935396 >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 0x7f459809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): get_auth_request con 0x7f459809b740 auth_method 0 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient(hunting): handle_auth_done global_id 224169 payload 274 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 -- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 msgr2=0x7f459809c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 --2- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 0x7f459809c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 -- 172.21.15.142:0/739935396 >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 msgr2=0x7f459809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 --2- 172.21.15.142:0/739935396 >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 0x7f459809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient: _finish_hunting 0 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 monclient: found mon.noname-a 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:09.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 -- 172.21.15.142:0/739935396 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459806f810 con 0x7f459809abd0 2024-09-18T10:29:09.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458ffff700 1 --2- 172.21.15.142:0/739935396 >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 0x7f459809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 -- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809abd0 msgr2=0x7f45980960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459cae3700 1 --2- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809abd0 0x7f45980960e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f459806a310 tx=0x7f4594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24258a29a890a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 1 -- 172.21.15.142:0/739935396 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594009e70 con 0x7f459809abd0 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: dump: 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.350 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 1 -- 172.21.15.142:0/739935396 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459400ba70 con 0x7f459809abd0 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459dae5700 5 monclient: authenticate success, global_id 224169 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459dae5700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f459dae5700 4 set_mon_vals no callback set 2024-09-18T10:29:09.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 1 -- 172.21.15.142:0/739935396 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400be30 con 0x7f459809abd0 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.332+0000 7f458f7fe700 10 monclient: dump: 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.352 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: shutdown 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809abd0 msgr2=0x7f45980960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809abd0 0x7f45980960e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f459806a310 tx=0x7f4594009250 comp rx=0 tx=0).stop 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- 172.21.15.142:0/739935396 shutdown_connections 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809c410 0x7f459809c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- 172.21.15.142:0/739935396 >> v2:172.21.15.150:3300/0 conn(0x7f459809b740 0x7f459809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- 172.21.15.142:0/739935396 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809abd0 0x7f45980960e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- 172.21.15.142:0/739935396 >> 172.21.15.142:0/739935396 conn(0x7f4598070ef0 msgr2=0x7f45980712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- 172.21.15.142:0/739935396 shutdown_connections 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- 172.21.15.142:0/739935396 wait complete. 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: build_initial_monmap 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: monmap: 2024-09-18T10:29:09.353 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 Processor -- start 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- start start 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: init 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:09.354 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f459809abd0 0x7f4598135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-b con 0x7f459809abd0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809b740 0x7f4598136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-c con 0x7f459809b740 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): picked mon.noname-a con 0x7f459809c410 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4598099620 con 0x7f459809c410 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459808d5b0 con 0x7f459809abd0 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459808d2d0 con 0x7f459809b740 2024-09-18T10:29:09.355 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459dae5700 10 monclient(hunting): authenticate will time out at 2038.266505s 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f458ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809b740 msgr2=0x7f4598136040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f458ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809b740 0x7f4598136040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45660/0 (socket says 172.21.15.142:45660) 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 learned_addr learned my addr 172.21.15.142:0/3692845085 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): get_auth_request con 0x7f459809c410 auth_method 0 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.356 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f459809abd0 0x7f4598135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): get_auth_request con 0x7f459809abd0 auth_method 0 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient(hunting): handle_auth_done global_id 224172 payload 824 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809b740 msgr2=0x7f4598136040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f459809b740 0x7f4598136040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f459809abd0 msgr2=0x7f4598135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f459809abd0 0x7f4598135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.357 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient: _finish_hunting 0 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 monclient: found mon.noname-a 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.336+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588009420 con 0x7f459809c410 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 msgr2=0x7f4598089a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459d2e4700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f459000bee0 tx=0x7f45900096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000cda0 con 0x7f459809c410 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: dump: 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.358 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4590016430 con 0x7f459809c410 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 5 monclient: authenticate success, global_id 224172 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900165e0 con 0x7f459809c410 2024-09-18T10:29:09.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 10 monclient: dump: 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 monclient: _renew_subs 2024-09-18T10:29:09.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45940094f0 con 0x7f459809c410 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458effd700 4 set_mon_vals no callback set 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 monclient: _renew_subs 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459808b4c0 con 0x7f459809c410 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 0x7f4598029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45980648a0 con 0x7f459808a730 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 0x7f4598029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient: get_auth_request con 0x7f459808a730 auth_method 0 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.361 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4590012020 con 0x7f459809c410 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.340+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f459cae3700 10 monclient(hunting): handle_auth_done global_id 224175 payload 824 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f459cae3700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f459806dad0 con 0x7f459808a730 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4584035a70 0x7f4584037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4590048e90 con 0x7f459809c410 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458ffff700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4584035a70 0x7f4584037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458ffff700 10 monclient: get_auth_request con 0x7f4584035a70 auth_method 0 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f459cae3700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 msgr2=0x7f4598029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f459cae3700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 0x7f4598029d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4594016000 tx=0x7f4594017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400bdf0 con 0x7f459808a730 2024-09-18T10:29:09.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 10 monclient: dump: 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4594020040 con 0x7f459808a730 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 msgr2=0x7f4598029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 0x7f4598029d00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4594016000 tx=0x7f4594017000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 msgr2=0x7f4598029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458cff9700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808a730 0x7f4598029d00 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4594016000 tx=0x7f4594017000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.344+0000 7f458ffff700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4584035a70 0x7f4584037f20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f45880096a0 tx=0x7f45880091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459dae5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459dae5700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 0x7f459802a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459dae5700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4598072a80 con 0x7f459808ab00 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 0x7f459802a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient: get_auth_request con 0x7f459808ab00 auth_method 0 2024-09-18T10:29:09.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 10 monclient(hunting): handle_auth_done global_id 224178 payload 824 2024-09-18T10:29:09.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.356+0000 7f459cae3700 1 -- 172.21.15.142:0/3692845085 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45980678a0 con 0x7f459808ab00 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459cae3700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 msgr2=0x7f459802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459cae3700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 0x7f459802a240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f459802b3c0 tx=0x7f4594021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459401ab70 con 0x7f459808ab00 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 10 monclient: dump: 2024-09-18T10:29:09.368 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4569 (secure 0 0 0) 0x7f4594033420 con 0x7f459808ab00 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:09.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 msgr2=0x7f459802a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 0x7f459802a240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f459802b3c0 tx=0x7f4594021000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 msgr2=0x7f459802a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f458cff9700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459808ab00 0x7f459802a240 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f459802b3c0 tx=0x7f4594021000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 194, 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:29:09.370 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.371 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.372 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:29:09.373 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.374 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:29:09.375 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.376 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.377 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.378 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.379 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.380 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.381 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:29:09.382 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.383 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:29:09.383 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:09.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4584035a70 msgr2=0x7f4584037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4584035a70 0x7f4584037f20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f45880096a0 tx=0x7f45880091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 10 monclient: shutdown 2024-09-18T10:29:09.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 msgr2=0x7f4598089a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f459000bee0 tx=0x7f45900096e0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459d2e4700 1 -- 172.21.15.142:0/3692845085 reap_dead start 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 shutdown_connections 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 --2- 172.21.15.142:0/3692845085 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f459809c410 0x7f4598089a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 >> 172.21.15.142:0/3692845085 conn(0x7f4598070ef0 msgr2=0x7f459809f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 shutdown_connections 2024-09-18T10:29:09.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.360+0000 7f459dae5700 1 -- 172.21.15.142:0/3692845085 wait complete. 2024-09-18T10:29:09.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:29:09.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-18T10:29:09.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.488+0000 7f0547068700 10 monclient: get_monmap_and_config 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.488+0000 7f0547068700 10 monclient: build_initial_monmap 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient: monmap: 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:09.495764+0000 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:09.495764+0000 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.498 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:09.499 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:09.499 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:09.499 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:09.499 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:09.499 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 Processor -- start 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 -- start start 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient: init 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f054009ae60 0x7f054006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): picked mon.noname-a con 0x7f054009ae60 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 0x7f054009b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): picked mon.noname-b con 0x7f054009cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f054009b770 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0540064a30 con 0x7f054009b770 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054009cb90 con 0x7f054009ae60 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054009e2d0 con 0x7f054009cdd0 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0547068700 10 monclient(hunting): authenticate will time out at 2038.421536s 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0545865700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 msgr2=0x7f054009b230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:09.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f054009ae60 0x7f054006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0545865700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 0x7f054009b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f054009ae60 0x7f054006b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45686/0 (socket says 172.21.15.142:45686) 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 -- 172.21.15.142:0/4272549013 learned_addr learned my addr 172.21.15.142:0/4272549013 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): get_auth_request con 0x7f054009ae60 auth_method 0 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient(hunting): handle_auth_done global_id 224181 payload 274 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 -- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 msgr2=0x7f054009b230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 --2- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 0x7f054009b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 -- 172.21.15.142:0/4272549013 >> v1:172.21.15.150:6789/0 conn(0x7f054009b770 legacy=0x7f0540132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient: _finish_hunting 0 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 monclient: found mon.noname-a 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.492+0000 7f0546066700 1 -- 172.21.15.142:0/4272549013 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054009bbf0 con 0x7f054009ae60 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0546066700 1 -- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009ae60 msgr2=0x7f054006b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0546066700 1 --2- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009ae60 0x7f054006b120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f053c0099b0 tx=0x7f053c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bcd18e67d3ea6c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 1 -- 172.21.15.142:0/4272549013 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c00bb50 con 0x7f054009ae60 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: dump: 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.507 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: _finish_auth 0 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 1 -- 172.21.15.142:0/4272549013 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c00bcb0 con 0x7f054009ae60 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 5 monclient: authenticate success, global_id 224181 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 1 -- 172.21.15.142:0/4272549013 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011670 con 0x7f054009ae60 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 4 set_mon_vals no callback set 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.508 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.509 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0545064700 10 monclient: dump: 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.513 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: shutdown 2024-09-18T10:29:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009ae60 msgr2=0x7f054006b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.514 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009ae60 0x7f054006b120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f053c0099b0 tx=0x7f053c009250 comp rx=0 tx=0).stop 2024-09-18T10:29:09.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- 172.21.15.142:0/4272549013 shutdown_connections 2024-09-18T10:29:09.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009cdd0 0x7f054009b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- 172.21.15.142:0/4272549013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009ae60 0x7f054006b120 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- 172.21.15.142:0/4272549013 >> 172.21.15.142:0/4272549013 conn(0x7f0540071ae0 msgr2=0x7f05400a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:09.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- 172.21.15.142:0/4272549013 shutdown_connections 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- 172.21.15.142:0/4272549013 wait complete. 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: build_initial_monmap 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: monmap: 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:09.524 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 Processor -- start 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- start start 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: init 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009ae60 0x7f0540141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): picked mon.noname-c con 0x7f054009ae60 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:09.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054009b770 0x7f0540142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): picked mon.noname-b con 0x7f054009b770 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 0x7f054011edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): picked mon.noname-a con 0x7f054009cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054009db70 con 0x7f054009cdd0 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054006b7b0 con 0x7f054009b770 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0540122670 con 0x7f054009ae60 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0547068700 10 monclient(hunting): authenticate will time out at 2038.427977s 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.496+0000 7f0546066700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009ae60 msgr2=0x7f0540141f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:09.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546066700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009ae60 0x7f0540141f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 0x7f054011edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 0x7f054011edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:45696/0 (socket says 172.21.15.142:45696) 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 learned_addr learned my addr 172.21.15.142:0/1062300770 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): get_auth_request con 0x7f054009cdd0 auth_method 0 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054009b770 0x7f0540142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): get_auth_request con 0x7f054009b770 auth_method 0 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.527 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient(hunting): handle_auth_done global_id 224184 payload 824 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009ae60 msgr2=0x7f0540141f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f054009ae60 0x7f0540141f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054009b770 msgr2=0x7f0540142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054009b770 0x7f0540142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient: _finish_hunting 0 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 monclient: found mon.noname-a 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:09.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0094f0 con 0x7f054009cdd0 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0545865700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054009b770 0x7f0540142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 msgr2=0x7f054011edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0546867700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 0x7f054011edc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f053000c5b0 tx=0x7f0530007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05300117a0 con 0x7f054009cdd0 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: dump: 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.529 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 5 monclient: authenticate success, global_id 224184 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 10 monclient: _renew_subs 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0530011900 con 0x7f054009cdd0 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05401223f0 con 0x7f054009cdd0 2024-09-18T10:29:09.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530011cc0 con 0x7f054009cdd0 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f05367fc700 10 monclient: dump: 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.531 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 10 monclient: _renew_subs 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0544863700 4 set_mon_vals no callback set 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.500+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0540123620 con 0x7f054009cdd0 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0547068700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0547068700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 0x7f0540029d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0540122ae0 con 0x7f054011fad0 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 0x7f0540029d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient: get_auth_request con 0x7f054011fad0 auth_method 0 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0530012630 con 0x7f054009cdd0 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f05367fc700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f052c035a70 0x7f052c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0530048a10 con 0x7f054009cdd0 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 10 monclient(hunting): handle_auth_done global_id 194527 payload 824 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0546066700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0540064680 con 0x7f054011fad0 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0545865700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f052c035a70 0x7f052c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.504+0000 7f0545865700 10 monclient: get_auth_request con 0x7f052c035a70 auth_method 0 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f0546066700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 msgr2=0x7f0540029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f0546066700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 0x7f0540029d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f053c015000 tx=0x7f053c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c011e40 con 0x7f054011fad0 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 10 monclient: dump: 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.534 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f0545865700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f052c035a70 0x7f052c037f20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f05400a1a60 tx=0x7f0538009760 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f053c01c450 con 0x7f054011fad0 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 msgr2=0x7f0540029d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 0x7f0540029d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f053c015000 tx=0x7f053c016000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 msgr2=0x7f0540029d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:09.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.508+0000 7f05367fc700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fad0 0x7f0540029d00 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f053c015000 tx=0x7f053c016000 comp rx=0 tx=0).stop 2024-09-18T10:29:09.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0547068700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:29:09.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0547068700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 0x7f054002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0547068700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054011b9e0 con 0x7f054011fea0 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 0x7f054002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient: get_auth_request con 0x7f054011fea0 auth_method 0 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 10 monclient(hunting): handle_auth_done global_id 194530 payload 824 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.520+0000 7f0546066700 1 -- 172.21.15.142:0/1062300770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0540122390 con 0x7f054011fea0 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0546066700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 msgr2=0x7f054002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0546066700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 0x7f054002a240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f054002b3c0 tx=0x7f053c01f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:09.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053c01a840 con 0x7f054011fea0 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 10 monclient: dump: 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:09.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4737 (secure 0 0 0) 0x7f053c01a9a0 con 0x7f054011fea0 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 msgr2=0x7f054002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 0x7f054002a240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f054002b3c0 tx=0x7f053c01f040 comp rx=0 tx=0).stop 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 msgr2=0x7f054002a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f05367fc700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f054011fea0 0x7f054002a240 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f054002b3c0 tx=0x7f053c01f040 comp rx=0 tx=0).stop 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:09.540 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 194, 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: 1 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 0, 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.541 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:29:09.542 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:29:09.543 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:09.544 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.545 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.546 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.547 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.548 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:29:09.549 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:09.550 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.551 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 5 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:29:09.552 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:29:09.553 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f052c035a70 msgr2=0x7f052c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f052c035a70 0x7f052c037f20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f05400a1a60 tx=0x7f0538009760 comp rx=0 tx=0).stop 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 10 monclient: shutdown 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 msgr2=0x7f054011edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 --2- 172.21.15.142:0/1062300770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f054009cdd0 0x7f054011edc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f053000c5b0 tx=0x7f0530007f60 comp rx=0 tx=0).stop 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 reap_dead start 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0546867700 1 -- 172.21.15.142:0/1062300770 reap_dead start 2024-09-18T10:29:09.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 shutdown_connections 2024-09-18T10:29:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 >> 172.21.15.142:0/1062300770 conn(0x7f0540071ae0 msgr2=0x7f054006cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 shutdown_connections 2024-09-18T10:29:09.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:09.524+0000 7f0547068700 1 -- 172.21.15.142:0/1062300770 wait complete. 2024-09-18T10:29:09.556 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-18T10:29:09.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=110 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 110 -ne 110 ]] 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ export CEPH_ARGS= 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ CEPH_ARGS= 2024-09-18T10:29:10.861 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph auth del client.bar 2024-09-18T10:29:10.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: get_monmap_and_config 2024-09-18T10:29:10.963 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: build_initial_monmap 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: monmap: 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:29:10.962018+0000 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:29:10.962018+0000 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:10.964 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 Processor -- start 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 -- start start 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: init 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 0x7f46dc06c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.967 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-b con 0x7f46dc06eb40 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f46dc06a4d0 0x7f46dc0a2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-a con 0x7f46dc06a4d0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46dc09c4c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc06a290 con 0x7f46dc09c4c0 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46dc06bf00 con 0x7f46dc06a4d0 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc09bf10 con 0x7f46dc06eb40 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.959+0000 7f46e4407700 10 monclient(hunting): authenticate will time out at 2033.055779s 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e21a3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 msgr2=0x7f46dc06c600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e21a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 0x7f46dc06c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:10.968 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e29a4700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f46dc09c4c0 0x7f46dc09c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52556/0 (socket says 172.21.15.150:52556) 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e29a4700 1 -- 172.21.15.150:0/2523092584 learned_addr learned my addr 172.21.15.150:0/2523092584 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 --2- 172.21.15.150:0/2523092584 >> v2:172.21.15.142:3300/0 conn(0x7f46dc06a4d0 0x7f46dc0a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): get_auth_request con 0x7f46dc06a4d0 auth_method 0 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient(hunting): handle_auth_done global_id 224187 payload 274 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 -- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 msgr2=0x7f46dc06c600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 --2- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 0x7f46dc06c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 -- 172.21.15.150:0/2523092584 >> v1:172.21.15.150:6789/0 conn(0x7f46dc09c4c0 legacy=0x7f46dc09c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:10.969 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient: _finish_hunting 0 2024-09-18T10:29:10.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 monclient: found mon.noname-a 2024-09-18T10:29:10.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:10.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:10.970 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 -- 172.21.15.150:0/2523092584 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc09d050 con 0x7f46dc06a4d0 2024-09-18T10:29:10.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 -- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc06a4d0 msgr2=0x7f46dc0a2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:10.972 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.963+0000 7f46e19a2700 1 --2- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc06a4d0 0x7f46dc0a2520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0098e0 tx=0x7f46cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4158566197be264f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 1 -- 172.21.15.150:0/2523092584 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00ba70 con 0x7f46dc06a4d0 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: dump: 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:10.973 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: _finish_auth 0 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 5 monclient: authenticate success, global_id 224187 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 1 -- 172.21.15.150:0/2523092584 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc00bbf0 con 0x7f46dc06a4d0 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 4 set_mon_vals no callback set 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 1 -- 172.21.15.150:0/2523092584 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc011420 con 0x7f46dc06a4d0 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e11a1700 10 monclient: dump: 2024-09-18T10:29:10.974 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 10 monclient: shutdown 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc06a4d0 msgr2=0x7f46dc0a2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 --2- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc06a4d0 0x7f46dc0a2520 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0098e0 tx=0x7f46cc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.975 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- 172.21.15.150:0/2523092584 shutdown_connections 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 --2- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc06a4d0 0x7f46dc0a2520 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 --2- 172.21.15.150:0/2523092584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06eb40 0x7f46dc06c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- 172.21.15.150:0/2523092584 >> 172.21.15.150:0/2523092584 conn(0x7f46dc071d90 msgr2=0x7f46dc0a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- 172.21.15.150:0/2523092584 shutdown_connections 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- 172.21.15.150:0/2523092584 wait complete. 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 10 monclient: build_initial_monmap 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 10 monclient: monmap: 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.976 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 Processor -- start 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 1 -- start start 2024-09-18T10:29:10.977 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.967+0000 7f46e4407700 10 monclient: init 2024-09-18T10:29:10.981 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 0x7f46dc1343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-c con 0x7f46dc06a4d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-b con 0x7f46dc06eb40 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 0x7f46dc11e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): picked mon.noname-a con 0x7f46dc09c4c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc06ad20 con 0x7f46dc09c4c0 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc121ac0 con 0x7f46dc06eb40 2024-09-18T10:29:10.982 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc1217e0 con 0x7f46dc06a4d0 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e4407700 10 monclient(hunting): authenticate will time out at 2033.064306s 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e21a3700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 msgr2=0x7f46dc1343e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e21a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 0x7f46dc1343e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53712/0 (socket says 172.21.15.150:53712) 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 -- 172.21.15.150:0/3423954317 learned_addr learned my addr 172.21.15.150:0/3423954317 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 0x7f46dc11e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): get_auth_request con 0x7f46dc06eb40 auth_method 0 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:10.983 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): get_auth_request con 0x7f46dc09c4c0 auth_method 0 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient(hunting): handle_auth_done global_id 194533 payload 824 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 -- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 msgr2=0x7f46dc1343e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:10.984 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 0x7f46dc1343e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 -- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 msgr2=0x7f46dc11e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 0x7f46dc11e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient: _finish_hunting 0 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 monclient: found mon.noname-b 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009420 con 0x7f46dc06eb40 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e29a4700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 0x7f46dc11e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 -- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 msgr2=0x7f46dc11dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46e19a2700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00b3c0 tx=0x7f46cc016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc011e30 con 0x7f46dc06eb40 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:10.985 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46d27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.971+0000 7f46d27fc700 10 monclient: dump: 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:10.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 5 monclient: authenticate success, global_id 194533 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc020070 con 0x7f46dc06eb40 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _renew_subs 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc142620 con 0x7f46dc06eb40 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc01b680 con 0x7f46dc06eb40 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 10 monclient: dump: 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:10.987 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e09a0700 4 set_mon_vals no callback set 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _renew_subs 2024-09-18T10:29:10.988 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc136c10 con 0x7f46dc06eb40 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f46cc022030 con 0x7f46dc06eb40 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc072e80 con 0x7f46dc06eb40 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 0x7f46c0037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f46cc04adf0 con 0x7f46dc06eb40 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e21a3700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 0x7f46c0037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:10.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.975+0000 7f46e21a3700 10 monclient: get_auth_request con 0x7f46c0035a70 auth_method 0 2024-09-18T10:29:10.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.979+0000 7f46e21a3700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 0x7f46c0037f20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f46d8003980 tx=0x7f46d8006e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:10.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.987+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f46cc04e030 con 0x7f46dc06eb40 2024-09-18T10:29:10.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.987+0000 7f46d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:10.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:10.987+0000 7f46d27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:11.214 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.211+0000 7f46e4407700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.211+0000 7f46e4407700 10 monclient: _send_command 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.211+0000 7f46e4407700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.215 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.211+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f46dc072e80 con 0x7f46dc06eb40 2024-09-18T10:29:11.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.219+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f46cc01ba70 con 0x7f46dc06eb40 2024-09-18T10:29:11.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.219+0000 7f46d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.222 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.219+0000 7f46d27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.371+0000 7f46e4407700 10 start_mon_command cmd=[{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.371+0000 7f46e4407700 10 monclient: _send_command 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.371+0000 7f46e4407700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.377 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.371+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bar"} v 0) v1 -- 0x7f46dc072e80 con 0x7f46dc06eb40 2024-09-18T10:29:11.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.375+0000 7f46d27fc700 1 -- 172.21.15.150:0/3423954317 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bar"}]=0 v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f46cc02cb30 con 0x7f46dc06eb40 2024-09-18T10:29:11.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.375+0000 7f46d27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth del", "entity": "client.bar"}] 2024-09-18T10:29:11.379 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.375+0000 7f46d27fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:29:11.381 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 msgr2=0x7f46c0037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 0x7f46c0037f20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f46d8003980 tx=0x7f46d8006e00 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 10 monclient: shutdown 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 msgr2=0x7f46dc11dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00b3c0 tx=0x7f46cc016000 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 shutdown_connections 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f46c0035a70 0x7f46c0037f20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46dc09c4c0 0x7f46dc11e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46dc06eb40 0x7f46dc11dee0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 --2- 172.21.15.150:0/3423954317 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46dc06a4d0 0x7f46dc1343e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 >> 172.21.15.150:0/3423954317 conn(0x7f46dc071d90 msgr2=0x7f46dc135510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:11.382 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 shutdown_connections 2024-09-18T10:29:11.383 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.379+0000 7f46e4407700 1 -- 172.21.15.150:0/3423954317 wait complete. 2024-09-18T10:29:11.391 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13 2024-09-18T10:29:11.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 2024-09-18T10:29:11.392 INFO:tasks.workunit.client.0.smithi150.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:29:11.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=13 2024-09-18T10:29:11.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:29:11.392 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status 2024-09-18T10:29:11.484 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=13 2024-09-18T10:29:11.484 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 13 -ne 13 ]] 2024-09-18T10:29:11.484 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:29:11.484 INFO:tasks.workunit.client.0.smithi150.stderr:+ rm /tmp/cephtest-mon-caps-madness.bazar.keyring /tmp/cephtest-mon-caps-madness.foo.keyring /tmp/cephtest-mon-caps-madness.bar.keyring 2024-09-18T10:29:11.485 INFO:tasks.workunit.client.0.smithi150.stderr:+ cat 2024-09-18T10:29:11.485 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph auth import -i - 2024-09-18T10:29:11.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: get_monmap_and_config 2024-09-18T10:29:11.574 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: build_initial_monmap 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: monmap: 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:29:11.572963+0000 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:29:11.572963+0000 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:11.575 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:11.576 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:11.576 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:11.576 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:11.576 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 Processor -- start 2024-09-18T10:29:11.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 -- start start 2024-09-18T10:29:11.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: init 2024-09-18T10:29:11.578 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc27009fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-b con 0x7fc27009f910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc27009bcc0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc27013b8e0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:11.579 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27006ac60 con 0x7fc27009bcc0 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27006bef0 con 0x7fc27013b8e0 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700a1020 con 0x7fc27009f910 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc275fed700 10 monclient(hunting): authenticate will time out at 2033.666616s 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc274feb700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 msgr2=0x7fc27009fce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc274feb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc27009fce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc2757ec700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc27013b8e0 0x7fc27013dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52570/0 (socket says 172.21.15.150:52570) 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.571+0000 7fc2757ec700 1 -- 172.21.15.150:0/604422353 learned_addr learned my addr 172.21.15.150:0/604422353 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:29:11.580 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 1 -- 172.21.15.150:0/604422353 >> v1:172.21.15.150:6789/0 conn(0x7fc27013b8e0 legacy=0x7fc27013dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 -- 172.21.15.150:0/604422353 >> v1:172.21.15.142:6789/0 conn(0x7fc27009bcc0 legacy=0x7fc2700a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727156282 0 0) 0x7fc27006bef0 con 0x7fc27013b8e0 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): my global_id is 194542 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258003570 con 0x7fc27013b8e0 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.0 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495036467 0 0) 0x7fc27006ac60 con 0x7fc27009bcc0 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.591 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): my global_id is 224196 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc27006bef0 con 0x7fc27009bcc0 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503556456 0 0) 0x7fc258003570 con 0x7fc27013b8e0 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 msgr2=0x7fc27009fce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc27009fce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 >> v1:172.21.15.142:6789/0 conn(0x7fc27009bcc0 legacy=0x7fc2700a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2700a16e0 con 0x7fc27013b8e0 2024-09-18T10:29:11.592 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc275fed700 5 monclient: authenticate success, global_id 194542 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fc2600017a0 con 0x7fc27013b8e0 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc275fed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: dump: 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:11.593 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 >> v1:172.21.15.150:6789/0 conn(0x7fc27013b8e0 legacy=0x7fc27013dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc258003c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): picked mon.c con 0x7fc27009f910 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 0x7fc258004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): picked mon.b con 0x7fc258004260 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:11.594 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc258004f50 0x7fc25800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): picked mon.a con 0x7fc258004f50 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700a1020 con 0x7fc258004f50 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2580056a0 con 0x7fc258004260 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800bff0 con 0x7fc27009f910 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc274feb700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 msgr2=0x7fc258003c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc275fed700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc274feb700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc258003c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 0x7fc258004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:11.595 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): get_auth_request con 0x7fc258004260 auth_method 0 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc258004f50 0x7fc25800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 10 monclient(hunting): get_auth_request con 0x7fc258004f50 auth_method 0 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc2757ec700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient(hunting): handle_auth_done global_id 194542 payload 293 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 msgr2=0x7fc258003c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:11.596 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc27009f910 0x7fc258003c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc258004f50 msgr2=0x7fc25800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc258004f50 0x7fc25800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 monclient: found mon.b 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 1 -- 172.21.15.150:0/604422353 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258003570 con 0x7fc258004260 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 10 monclient: _finish_auth 0 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.575+0000 7fc26ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26ffff700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 msgr2=0x7fc258004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26ffff700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 0x7fc258004810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26800c2a0 tx=0x7fc268001ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4ac6f0e96afdfeb server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cbd0 con 0x7fc258004260 2024-09-18T10:29:11.597 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: dump: 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:11.598 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26800cd30 con 0x7fc258004260 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 4 set_mon_vals no callback set 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 1 -- 172.21.15.150:0/604422353 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cea0 con 0x7fc258004260 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc26f7fe700 10 monclient: dump: 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.599 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 10 monclient: shutdown 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 -- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 msgr2=0x7fc258004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 --2- 172.21.15.150:0/604422353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc258004260 0x7fc258004810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26800c2a0 tx=0x7fc268001ac0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc2757ec700 1 -- 172.21.15.150:0/604422353 reap_dead start 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 -- 172.21.15.150:0/604422353 shutdown_connections 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 -- 172.21.15.150:0/604422353 >> 172.21.15.150:0/604422353 conn(0x7fc2700a18e0 msgr2=0x7fc2700a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 -- 172.21.15.150:0/604422353 shutdown_connections 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 1 -- 172.21.15.150:0/604422353 wait complete. 2024-09-18T10:29:11.600 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 10 monclient: build_initial_monmap 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.579+0000 7fc275fed700 10 monclient: monmap: 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 Processor -- start 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 -- start start 2024-09-18T10:29:11.601 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient: init 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 0x7fc2701234e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-a con 0x7fc270123110 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 0x7fc270123df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-c con 0x7fc270123a20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): picked mon.noname-b con 0x7fc270149d70 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27006e550 con 0x7fc270123110 2024-09-18T10:29:11.602 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270126b80 con 0x7fc270149d70 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc270126810 con 0x7fc270123a20 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc275fed700 10 monclient(hunting): authenticate will time out at 2033.677122s 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc26ffff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 msgr2=0x7fc270123df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc26ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 0x7fc270123df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53746/0 (socket says 172.21.15.150:53746) 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 -- 172.21.15.150:0/2236324036 learned_addr learned my addr 172.21.15.150:0/2236324036 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): get_auth_request con 0x7fc270149d70 auth_method 0 2024-09-18T10:29:11.603 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 0x7fc2701234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 10 monclient(hunting): get_auth_request con 0x7fc270123110 auth_method 0 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient(hunting): handle_auth_done global_id 194545 payload 824 2024-09-18T10:29:11.604 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 -- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 msgr2=0x7fc270123df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 0x7fc270123df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 -- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 msgr2=0x7fc2701234e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 0x7fc2701234e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient: _finish_hunting 0 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 monclient: found mon.noname-b 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc2757ec700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270126530 con 0x7fc270149d70 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.583+0000 7fc274feb700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 0x7fc2701234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc2757ec700 1 -- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 msgr2=0x7fc27014c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc2757ec700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc258004790 tx=0x7fc2700a1870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:11.605 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260010730 con 0x7fc270149d70 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: dump: 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.606 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc260010890 con 0x7fc270149d70 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 5 monclient: authenticate success, global_id 194545 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260010c50 con 0x7fc270149d70 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 10 monclient: dump: 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:11.607 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _renew_subs 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26effd700 4 set_mon_vals no callback set 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc270124a70 con 0x7fc270149d70 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _renew_subs 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.608 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc270125640 con 0x7fc270149d70 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fc26001c3e0 con 0x7fc270149d70 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 0x7fc25c037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fc260048a80 con 0x7fc270149d70 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2701243c0 con 0x7fc270149d70 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc274feb700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 0x7fc25c037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.587+0000 7fc274feb700 10 monclient: get_auth_request con 0x7fc25c035a70 auth_method 0 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.591+0000 7fc274feb700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 0x7fc25c037f20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc264003980 tx=0x7fc264008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.595+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fc26001ce40 con 0x7fc270149d70 2024-09-18T10:29:11.610 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.595+0000 7fc26cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:11.611 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.595+0000 7fc26cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:11.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.811+0000 7fc275fed700 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-18T10:29:11.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.811+0000 7fc275fed700 10 monclient: _send_command 2 [{"prefix": "auth import"}] 2024-09-18T10:29:11.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.811+0000 7fc275fed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.814 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.811+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fc27006dad0 con 0x7fc270149d70 2024-09-18T10:29:11.816 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:11.808+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 auth entities have invalid capabilities (AUTH_BAD_CAPS) 2024-09-18T10:29:11.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.819+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth import"}]=0 v35) v1 ==== 59+0+0 (secure 0 0 0) 0x7fc260021070 con 0x7fc270149d70 2024-09-18T10:29:11.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.819+0000 7fc26cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth import"}] 2024-09-18T10:29:11.822 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.819+0000 7fc26cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:11.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.971+0000 7fc275fed700 10 start_mon_command cmd=[{"prefix": "auth import"}] 2024-09-18T10:29:11.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.971+0000 7fc275fed700 10 monclient: _send_command 3 [{"prefix": "auth import"}] 2024-09-18T10:29:11.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.971+0000 7fc275fed700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:11.976 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.971+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth import"} v 0) v1 -- 0x7fc27006dad0 con 0x7fc270149d70 2024-09-18T10:29:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc26cff9700 1 -- 172.21.15.150:0/2236324036 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "auth import"}]=0 v36) v1 ==== 59+0+0 (secure 0 0 0) 0x7fc2600112e0 con 0x7fc270149d70 2024-09-18T10:29:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc26cff9700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth import"}] 2024-09-18T10:29:11.986 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc26cff9700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 msgr2=0x7fc25c037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 0x7fc25c037f20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc264003980 tx=0x7fc264008040 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 10 monclient: shutdown 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 msgr2=0x7fc27014c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc258004790 tx=0x7fc2700a1870 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 shutdown_connections 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fc25c035a70 0x7fc25c037f20 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc270149d70 0x7fc27014c150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc270123a20 0x7fc270123df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 --2- 172.21.15.150:0/2236324036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc270123110 0x7fc2701234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:11.989 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 >> 172.21.15.150:0/2236324036 conn(0x7fc2700a18e0 msgr2=0x7fc27009b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 shutdown_connections 2024-09-18T10:29:11.990 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:11.983+0000 7fc275fed700 1 -- 172.21.15.150:0/2236324036 wait complete. 2024-09-18T10:29:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph health 2024-09-18T10:29:12.000 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep AUTH_BAD_CAP 2024-09-18T10:29:12.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: get_monmap_and_config 2024-09-18T10:29:12.085 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: build_initial_monmap 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: monmap: 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:29:12.084247+0000 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:29:12.084247+0000 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:12.086 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:12.087 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 Processor -- start 2024-09-18T10:29:12.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 -- start start 2024-09-18T10:29:12.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: init 2024-09-18T10:29:12.089 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f810809e4e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 0x7f8108069720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-b con 0x7f810809efc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8108137510 0x7f81081398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-c con 0x7f8108137510 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080a03d0 con 0x7f810809e4e0 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810809f390 con 0x7f8108137510 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810806a7d0 con 0x7f810809efc0 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:12.090 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810d30b700 10 monclient(hunting): authenticate will time out at 2034.177458s 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f81077fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 msgr2=0x7f8108069720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f81077fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 0x7f8108069720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8108137510 0x7f81081398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8108137510 0x7f81081398f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53762/0 (socket says 172.21.15.150:53762) 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f8107fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f810809e4e0 0x7f810809e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.150:59190/0 (socket says 172.21.15.150:59190) 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 -- 172.21.15.150:0/3504980933 learned_addr learned my addr 172.21.15.150:0/3504980933 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): get_auth_request con 0x7f8108137510 auth_method 0 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:12.091 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient(hunting): handle_auth_done global_id 194548 payload 274 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 -- 172.21.15.150:0/3504980933 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 msgr2=0x7f8108069720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 --2- 172.21.15.150:0/3504980933 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 0x7f8108069720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 -- 172.21.15.150:0/3504980933 >> v1:172.21.15.142:6789/0 conn(0x7f810809e4e0 legacy=0x7f810809e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient: _finish_hunting 0 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 monclient: found mon.noname-c 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.092 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.083+0000 7f810cb0a700 1 -- 172.21.15.150:0/3504980933 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810809fd10 con 0x7f8108137510 2024-09-18T10:29:12.095 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810cb0a700 1 -- 172.21.15.150:0/3504980933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8108137510 msgr2=0x7f81081398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810cb0a700 1 --2- 172.21.15.150:0/3504980933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8108137510 0x7f81081398f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8100009f50 tx=0x7f8100009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92818912a2997f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 1 -- 172.21.15.150:0/3504980933 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810000e070 con 0x7f8108137510 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: dump: 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.096 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: _finish_auth 0 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 1 -- 172.21.15.150:0/3504980933 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810000bc80 con 0x7f8108137510 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 5 monclient: authenticate success, global_id 194548 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 1 -- 172.21.15.150:0/3504980933 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100012430 con 0x7f8108137510 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:12.097 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 4 set_mon_vals no callback set 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f8106ffd700 10 monclient: dump: 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.098 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: shutdown 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- 172.21.15.150:0/3504980933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8108137510 msgr2=0x7f81081398f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- 172.21.15.150:0/3504980933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8108137510 0x7f81081398f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8100009f50 tx=0x7f8100009840 comp rx=0 tx=0).stop 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- 172.21.15.150:0/3504980933 shutdown_connections 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- 172.21.15.150:0/3504980933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8108137510 0x7f81081398f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- 172.21.15.150:0/3504980933 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809efc0 0x7f8108069720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- 172.21.15.150:0/3504980933 >> 172.21.15.150:0/3504980933 conn(0x7f8108071e40 msgr2=0x7f81080a3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- 172.21.15.150:0/3504980933 shutdown_connections 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- 172.21.15.150:0/3504980933 wait complete. 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: build_initial_monmap 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: monmap: 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:12.099 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 Processor -- start 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 -- start start 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: init 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:12.100 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 0x7f810811eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-c con 0x7f810809e4e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-b con 0x7f810809efc0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 0x7f810811f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient(hunting): picked mon.noname-a con 0x7f8108137510 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.087+0000 7f810d30b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081224a0 con 0x7f8108137510 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810806e380 con 0x7f810809efc0 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:12.101 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810806db10 con 0x7f810809e4e0 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient(hunting): authenticate will time out at 2034.184032s 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f8107fff700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 msgr2=0x7f810811eb50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53770/0 (socket says 172.21.15.150:53770) 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 -- 172.21.15.150:0/3935979628 learned_addr learned my addr 172.21.15.150:0/3935979628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): get_auth_request con 0x7f810809efc0 auth_method 0 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f8107fff700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 0x7f810811eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:12.102 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 0x7f810811f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 10 monclient(hunting): get_auth_request con 0x7f8108137510 auth_method 0 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient(hunting): handle_auth_done global_id 194551 payload 824 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 -- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 msgr2=0x7f810811eb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 0x7f810811eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 -- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 msgr2=0x7f810811f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 0x7f810811f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:12.103 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:12.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 monclient: found mon.noname-b 2024-09-18T10:29:12.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:12.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100009ae0 con 0x7f810809efc0 2024-09-18T10:29:12.104 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810cb0a700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 0x7f810811f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:12.105 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 -- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 msgr2=0x7f810811f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b7e0 tx=0x7f80f0009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81077fe700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b7e0 tx=0x7f80f0009220 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7f80f0015070 mon_map magic: 0 v1 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f0015070 con 0x7f810809efc0 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 10 monclient: dump: 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.106 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 10 monclient: _finish_auth 0 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f0003e20 con 0x7f810809efc0 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 5 monclient: authenticate success, global_id 194551 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient: _renew_subs 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108121370 con 0x7f810809efc0 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f80fffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient: _renew_subs 2024-09-18T10:29:12.107 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f81067fc700 4 set_mon_vals no callback set 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.091+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108137230 con 0x7f810809efc0 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f0006380 con 0x7f810809efc0 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f810d30b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 10 monclient: dump: 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:12.108 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f80f0006730 con 0x7f810809efc0 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f810d30b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f810d30b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81081226a0 con 0x7f810809efc0 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f80fffff700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 0x7f80e8037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f8107fff700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 0x7f80e8037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.095+0000 7f8107fff700 10 monclient: get_auth_request con 0x7f80e8035af0 auth_method 0 2024-09-18T10:29:12.109 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.099+0000 7f8107fff700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 0x7f80e8037fa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80f8004910 tx=0x7f80f8007b50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:12.110 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.103+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f80f0048a60 con 0x7f810809efc0 2024-09-18T10:29:12.718 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.715+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f80f0010440 con 0x7f810809efc0 2024-09-18T10:29:12.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.715+0000 7f80fffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:12.719 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.715+0000 7f80fffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:12.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f810d30b700 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-18T10:29:12.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f810d30b700 10 monclient: _send_command 2 [{"prefix": "health"}] 2024-09-18T10:29:12.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f810d30b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:12.904 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f810804c430 con 0x7f810809efc0 2024-09-18T10:29:12.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+126 (secure 0 0 0) 0x7f80f0018310 con 0x7f810809efc0 2024-09-18T10:29:12.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f80fffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health"}] 2024-09-18T10:29:12.905 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:12.899+0000 7f80fffff700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:13.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 10 start_mon_command cmd=[{"prefix": "health"}] 2024-09-18T10:29:13.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 10 monclient: _send_command 3 [{"prefix": "health"}] 2024-09-18T10:29:13.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.022 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health"} v 0) v1 -- 0x7f810806a0c0 con 0x7f810809efc0 2024-09-18T10:29:13.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f80fffff700 1 -- 172.21.15.150:0/3935979628 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "health"}]=0 v0) v1 ==== 54+0+126 (secure 0 0 0) 0x7f80f00184f0 con 0x7f810809efc0 2024-09-18T10:29:13.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f80fffff700 10 monclient: handle_mon_command_ack 3 [{"prefix": "health"}] 2024-09-18T10:29:13.023 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f80fffff700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:29:13.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 msgr2=0x7f80e8037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 0x7f80e8037fa0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80f8004910 tx=0x7f80f8007b50 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 10 monclient: shutdown 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 msgr2=0x7f810811f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f80f000b7e0 tx=0x7f80f0009220 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 shutdown_connections 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f80e8035af0 0x7f80e8037fa0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8108137510 0x7f810811f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f810809efc0 0x7f810811f090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 --2- 172.21.15.150:0/3935979628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f810809e4e0 0x7f810811eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 >> 172.21.15.150:0/3935979628 conn(0x7f8108071e40 msgr2=0x7f81080a19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:13.026 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 shutdown_connections 2024-09-18T10:29:13.027 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.019+0000 7f810d30b700 1 -- 172.21.15.150:0/3935979628 wait complete. 2024-09-18T10:29:13.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph health detail 2024-09-18T10:29:13.035 INFO:tasks.workunit.client.0.smithi150.stderr:+ grep client.bad 2024-09-18T10:29:13.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 10 monclient: get_monmap_and_config 2024-09-18T10:29:13.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 10 monclient: build_initial_monmap 2024-09-18T10:29:13.124 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 10 monclient: monmap: 2024-09-18T10:29:13.124 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:29:13.123019+0000 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:29:13.123019+0000 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:13.125 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 1 Processor -- start 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 1 -- start start 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.119+0000 7fce8f59e700 10 monclient: init 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce9009fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fce9009f6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce9009bcc0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce90135300 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9006ce70 con 0x7fce9009bcc0 2024-09-18T10:29:13.127 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9006d150 con 0x7fce90135300 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce900a1790 con 0x7fce9009f6e0 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 10 monclient(hunting): authenticate will time out at 2035.216519s 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8ed9d700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fce90135300 0x7fce901376e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52582/0 (socket says 172.21.15.150:52582) 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8ed9d700 1 -- 172.21.15.150:0/1417172385 learned_addr learned my addr 172.21.15.150:0/1417172385 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8e59c700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 msgr2=0x7fce9009fab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:13.128 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8e59c700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce9009fab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8ed9d700 1 -- 172.21.15.150:0/1417172385 >> v1:172.21.15.150:6789/0 conn(0x7fce90135300 legacy=0x7fce901376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016685415 0 0) 0x7fce9006d150 con 0x7fce90135300 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient(hunting): my global_id is 194554 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce78003570 con 0x7fce90135300 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8dd9b700 1 -- 172.21.15.150:0/1417172385 >> v1:172.21.15.142:6789/0 conn(0x7fce9009bcc0 legacy=0x7fce9009fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2195575213 0 0) 0x7fce78003570 con 0x7fce90135300 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 msgr2=0x7fce9009fab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce9009fab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.132 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 >> v1:172.21.15.142:6789/0 conn(0x7fce9009bcc0 legacy=0x7fce9009fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: _finish_hunting 0 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 monclient: found mon.noname-c-legacy 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.150:6789/0 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce900a1260 con 0x7fce90135300 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: _finish_auth 0 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 5 monclient: authenticate success, global_id 194554 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2633518091 0 0) 0x7fce88001be0 con 0x7fce90135300 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: dump: 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] but i'm connected to v1:172.21.15.150:6789/0, reconnecting 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:13.134 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.123+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 >> v1:172.21.15.150:6789/0 conn(0x7fce90135300 legacy=0x7fce901376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce78003a60 0x7fce78003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): picked mon.a con 0x7fce78003a60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): picked mon.b con 0x7fce78004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce7800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): picked mon.c con 0x7fce9009f6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780055f0 con 0x7fce78003a60 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7800bfa0 con 0x7fce78004810 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7800c0e0 con 0x7fce9009f6e0 2024-09-18T10:29:13.135 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient(hunting): _finish_auth 0 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8f59e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8ed9d700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 msgr2=0x7fce7800ba60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): get_auth_request con 0x7fce78004810 auth_method 0 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8ed9d700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce7800ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce78003a60 0x7fce78003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:13.136 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 10 monclient(hunting): get_auth_request con 0x7fce78003a60 auth_method 0 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_done global_id 194554 payload 293 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 msgr2=0x7fce7800ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:13.137 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9009f6e0 0x7fce7800ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce78003a60 msgr2=0x7fce78003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce78003a60 0x7fce78003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient: _finish_hunting 0 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 monclient: found mon.b 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 -- 172.21.15.150:0/1417172385 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78003570 con 0x7fce78004810 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 10 monclient: _finish_auth 0 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:13.138 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8e59c700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce78003a60 0x7fce78003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 msgr2=0x7fce78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce78004c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce84001bf0 tx=0x7fce84007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d3b6690d8c3a8f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84001d50 con 0x7fce78004810 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: dump: 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.147 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce840070d0 con 0x7fce78004810 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8f59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:13.148 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8f59e700 4 set_mon_vals no callback set 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 1 -- 172.21.15.150:0/1417172385 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84007490 con 0x7fce78004810 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.127+0000 7fce8d59a700 10 monclient: dump: 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.149 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: shutdown 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 msgr2=0x7fce78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 --2- 172.21.15.150:0/1417172385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce78004c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce84001bf0 tx=0x7fce84007d10 comp rx=0 tx=0).stop 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8ed9d700 1 -- 172.21.15.150:0/1417172385 reap_dead start 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- 172.21.15.150:0/1417172385 shutdown_connections 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- 172.21.15.150:0/1417172385 >> 172.21.15.150:0/1417172385 conn(0x7fce900a18e0 msgr2=0x7fce900a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- 172.21.15.150:0/1417172385 shutdown_connections 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- 172.21.15.150:0/1417172385 wait complete. 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: build_initial_monmap 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: monmap: 2024-09-18T10:29:13.150 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 Processor -- start 2024-09-18T10:29:13.151 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- start start 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: init 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-b con 0x7fce78004810 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9011f0b0 0x7fce9011f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-a con 0x7fce9011f0b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 0x7fce901477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): picked mon.noname-c con 0x7fce9011f9c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9011fd90 con 0x7fce9011f0b0 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.152 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9006e740 con 0x7fce78004810 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9006b980 con 0x7fce9011f9c0 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8f59e700 10 monclient(hunting): authenticate will time out at 2035.227508s 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8ed9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 msgr2=0x7fce901477c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53778/0 (socket says 172.21.15.150:53778) 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8ed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 0x7fce901477c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 1 -- 172.21.15.150:0/2662340369 learned_addr learned my addr 172.21.15.150:0/2662340369 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): get_auth_request con 0x7fce78004810 auth_method 0 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.153 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9011f0b0 0x7fce9011f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8dd9b700 10 monclient(hunting): get_auth_request con 0x7fce9011f0b0 auth_method 0 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.131+0000 7fce8dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 10 monclient(hunting): handle_auth_done global_id 194557 payload 824 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 -- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 msgr2=0x7fce901477c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:13.154 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 0x7fce901477c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 -- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9011f0b0 msgr2=0x7fce9011f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9011f0b0 0x7fce9011f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 monclient: found mon.noname-b 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce90120430 con 0x7fce78004810 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 -- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 msgr2=0x7fce9011eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8e59c700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce8000d860 tx=0x7fce80008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80003e20 con 0x7fce78004810 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:13.155 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: dump: 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.156 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.157 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.157 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.157 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.157 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.157 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 5 monclient: authenticate success, global_id 194557 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce800050d0 con 0x7fce78004810 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 10 monclient: _renew_subs 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.158 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce90121f60 con 0x7fce78004810 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80005490 con 0x7fce78004810 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce7e7fc700 10 monclient: dump: 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.159 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.160 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.160 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.160 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.160 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.160 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8cd99700 4 set_mon_vals no callback set 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 10 monclient: _renew_subs 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.135+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce90121000 con 0x7fce78004810 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fce8001c460 con 0x7fce78004810 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce90122400 con 0x7fce78004810 2024-09-18T10:29:13.161 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce7e7fc700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 0x7fce70037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fce80048f90 con 0x7fce78004810 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 0x7fce70037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.139+0000 7fce8dd9b700 10 monclient: get_auth_request con 0x7fce70035a70 auth_method 0 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.143+0000 7fce8dd9b700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 0x7fce70037f20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fce84007820 tx=0x7fce84015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.147+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fce8000faf0 con 0x7fce78004810 2024-09-18T10:29:13.162 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.147+0000 7fce7e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.163 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.147+0000 7fce7e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:13.374 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce8f59e700 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce8f59e700 10 monclient: _send_command 2 [{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce8f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce901225d0 con 0x7fce78004810 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1208 (secure 0 0 0) 0x7fce8000f310 con 0x7fce78004810 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce7e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.375 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.371+0000 7fce7e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:13.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce8f59e700 10 start_mon_command cmd=[{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce8f59e700 10 monclient: _send_command 3 [{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce8f59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.519 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fce9004c430 con 0x7fce78004810 2024-09-18T10:29:13.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce7e7fc700 1 -- 172.21.15.150:0/2662340369 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1208 (secure 0 0 0) 0x7fce8000f4f0 con 0x7fce78004810 2024-09-18T10:29:13.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce7e7fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "health", "detail": "detail"}] 2024-09-18T10:29:13.520 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.515+0000 7fce7e7fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 msgr2=0x7fce70037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 0x7fce70037f20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fce84007820 tx=0x7fce84015000 comp rx=0 tx=0).stop 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 10 monclient: shutdown 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 msgr2=0x7fce9011eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce8000d860 tx=0x7fce80008040 comp rx=0 tx=0).stop 2024-09-18T10:29:13.522 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 shutdown_connections 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fce70035a70 0x7fce70037f20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce9011f9c0 0x7fce901477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce9011f0b0 0x7fce9011f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 --2- 172.21.15.150:0/2662340369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce78004810 0x7fce9011eb70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 >> 172.21.15.150:0/2662340369 conn(0x7fce900a18e0 msgr2=0x7fce9006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 shutdown_connections 2024-09-18T10:29:13.523 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.519+0000 7fce8f59e700 1 -- 172.21.15.150:0/2662340369 wait complete. 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad mds capability parse failed, stopped at 'also does not parse' of 'also does not parse' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad mon capability parse failed, stopped at 'this is wrong' of 'this is wrong' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stdout: client.bad osd capability parse failed, stopped at 'does not parse' of 'does not parse' 2024-09-18T10:29:13.531 INFO:tasks.workunit.client.0.smithi150.stderr:+ ceph auth rm client.bad 2024-09-18T10:29:13.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: get_monmap_and_config 2024-09-18T10:29:13.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: build_initial_monmap 2024-09-18T10:29:13.621 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: monmap: 2024-09-18T10:29:13.622 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:13.622 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.622 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:29:13.620222+0000 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:29:13.620222+0000 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:13.623 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 Processor -- start 2024-09-18T10:29:13.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 -- start start 2024-09-18T10:29:13.625 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: init 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56ec0a6500 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56ec0a4300 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f56ec0a4700 0x7f56ec13af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f56ec0a4700 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec06ae90 con 0x7f56ec0a6500 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56ec0a68d0 con 0x7f56ec0a4300 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec06be80 con 0x7f56ec0a4700 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56eb59e700 10 monclient(hunting): authenticate will time out at 2035.713478s 2024-09-18T10:29:13.626 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56e9d9b700 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f56ec0a4300 0x7f56ec1377f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:52588/0 (socket says 172.21.15.150:52588) 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56e9d9b700 1 -- 172.21.15.150:0/1116347789 learned_addr learned my addr 172.21.15.150:0/1116347789 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 1 --2- 172.21.15.150:0/1116347789 >> v2:172.21.15.142:3300/0 conn(0x7f56ec0a4700 0x7f56ec13af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): get_auth_request con 0x7f56ec0a4700 auth_method 0 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient(hunting): handle_auth_done global_id 224211 payload 274 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 1 -- 172.21.15.150:0/1116347789 >> v1:172.21.15.150:6789/0 conn(0x7f56ec0a4300 legacy=0x7f56ec1377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 1 -- 172.21.15.150:0/1116347789 >> v1:172.21.15.142:6789/0 conn(0x7f56ec0a6500 legacy=0x7f56ec0a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:13.627 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:13.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 1 monclient: found mon.noname-a 2024-09-18T10:29:13.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:13.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:13.628 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.619+0000 7f56ead9d700 1 -- 172.21.15.150:0/1116347789 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec0a5b60 con 0x7f56ec0a4700 2024-09-18T10:29:13.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56ead9d700 1 -- 172.21.15.150:0/1116347789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4700 msgr2=0x7f56ec13af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56ead9d700 1 --2- 172.21.15.150:0/1116347789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4700 0x7f56ec13af20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009e60 tx=0x7f56e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d358b5e17cd6d7f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 1 -- 172.21.15.150:0/1116347789 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400e070 con 0x7f56ec0a4700 2024-09-18T10:29:13.631 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: dump: 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.632 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: _finish_auth 0 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 5 monclient: authenticate success, global_id 224211 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 1 -- 172.21.15.150:0/1116347789 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4012430 con 0x7f56ec0a4700 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 4 set_mon_vals no callback set 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 1 -- 172.21.15.150:0/1116347789 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40127d0 con 0x7f56ec0a4700 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e959a700 10 monclient: dump: 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.633 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: shutdown 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- 172.21.15.150:0/1116347789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4700 msgr2=0x7f56ec13af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 --2- 172.21.15.150:0/1116347789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4700 0x7f56ec13af20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009e60 tx=0x7f56e4009780 comp rx=0 tx=0).stop 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- 172.21.15.150:0/1116347789 shutdown_connections 2024-09-18T10:29:13.634 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 --2- 172.21.15.150:0/1116347789 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4700 0x7f56ec13af20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- 172.21.15.150:0/1116347789 >> 172.21.15.150:0/1116347789 conn(0x7f56ec071960 msgr2=0x7f56ec0a5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- 172.21.15.150:0/1116347789 shutdown_connections 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- 172.21.15.150:0/1116347789 wait complete. 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: build_initial_monmap 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: monmap: 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 0 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 0.000000 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:created 0.000000 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:13.635 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 Processor -- start 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- start start 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: init 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f56ec0a4300 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f56ec0a4700 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 0x7f56ec1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f56ec0a6500 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:13.636 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec122f60 con 0x7f56ec0a4300 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1229a0 con 0x7f56ec0a4700 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec1204c0 con 0x7f56ec0a6500 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56eb59e700 10 monclient(hunting): authenticate will time out at 2035.719615s 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56ead9d700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 msgr2=0x7f56ec1414e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56ead9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 0x7f56ec1414e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53782/0 (socket says 172.21.15.150:53782) 2024-09-18T10:29:13.637 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 1 -- 172.21.15.150:0/8473873 learned_addr learned my addr 172.21.15.150:0/8473873 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 10 monclient(hunting): get_auth_request con 0x7f56ec0a4700 auth_method 0 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.623+0000 7f56e9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56ea59c700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56ea59c700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 msgr2=0x7f56ec13df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56ea59c700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 msgr2=0x7f56ec13df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56ea59c700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56ea59c700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient(hunting): handle_auth_done global_id 194560 payload 824 2024-09-18T10:29:13.638 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 msgr2=0x7f56ec1414e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 0x7f56ec1414e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 msgr2=0x7f56ec13df70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 monclient: found mon.noname-b 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.639 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e4009a20 con 0x7f56ec0a4700 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 msgr2=0x7f56ec13d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e9d9b700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00b5e0 tx=0x7f56dc009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc011840 con 0x7f56ec0a4700 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: dump: 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.641 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 5 monclient: authenticate success, global_id 194560 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 10 monclient: _renew_subs 2024-09-18T10:29:13.642 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc0119c0 con 0x7f56ec0a4700 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec1226c0 con 0x7f56ec0a4700 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e8d99700 4 set_mon_vals no callback set 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc011d80 con 0x7f56ec0a4700 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56e27fc700 10 monclient: dump: 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:epoch 1 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:13.643 INFO:tasks.workunit.client.0.smithi150.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:election_strategy: 1 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 10 monclient: _renew_subs 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.627+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec13db60 con 0x7f56ec0a4700 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f56dc010590 con 0x7f56ec0a4700 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56e27fc700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 0x7f56cc037f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56eb59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56eb59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.644 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56eb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f56dc0490c0 con 0x7f56ec0a4700 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec04c430 con 0x7f56ec0a4700 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56ea59c700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 0x7f56cc037f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56ea59c700 10 monclient: get_auth_request con 0x7f56cc035a70 auth_method 0 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.631+0000 7f56ea59c700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 0x7f56cc037f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f56d4007fd0 tx=0x7f56d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.639+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f56dc015070 con 0x7f56ec0a4700 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.639+0000 7f56e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:13.645 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.639+0000 7f56e27fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:13.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.863+0000 7f56eb59e700 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:13.865 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.863+0000 7f56eb59e700 10 monclient: _send_command 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:13.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.863+0000 7f56eb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:13.866 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.863+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7f56ec0a56e0 con 0x7f56ec0a4700 2024-09-18T10:29:13.873 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.871+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v37) v1 ==== 79+0+0 (secure 0 0 0) 0x7f56dc00f4c0 con 0x7f56ec0a4700 2024-09-18T10:29:13.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.871+0000 7f56e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:13.874 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:13.871+0000 7f56e27fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:14.019 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.015+0000 7f56eb59e700 10 start_mon_command cmd=[{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:14.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.015+0000 7f56eb59e700 10 monclient: _send_command 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:14.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.015+0000 7f56eb59e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:14.020 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.015+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth rm", "entity": "client.bad"} v 0) v1 -- 0x7f56ec0008d0 con 0x7f56ec0a4700 2024-09-18T10:29:14.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56e27fc700 1 -- 172.21.15.150:0/8473873 <== mon.1 v2:172.21.15.150:3300/0 8 ==== mon_command_ack([{"prefix": "auth rm", "entity": "client.bad"}]=0 v37) v1 ==== 79+0+0 (secure 0 0 0) 0x7f56dc00f2e0 con 0x7f56ec0a4700 2024-09-18T10:29:14.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56e27fc700 10 monclient: handle_mon_command_ack 3 [{"prefix": "auth rm", "entity": "client.bad"}] 2024-09-18T10:29:14.021 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56e27fc700 10 monclient: _finish_command 3 = system:0 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 msgr2=0x7f56cc037f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 0x7f56cc037f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f56d4007fd0 tx=0x7f56d4009040 comp rx=0 tx=0).stop 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 10 monclient: shutdown 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 msgr2=0x7f56ec13d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:14.024 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00b5e0 tx=0x7f56dc009120 comp rx=0 tx=0).stop 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 shutdown_connections 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f56cc035a70 0x7f56cc037f20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f56ec0a6500 0x7f56ec1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56ec0a4700 0x7f56ec13d620 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 --2- 172.21.15.150:0/8473873 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f56ec0a4300 0x7f56ec13df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 >> 172.21.15.150:0/8473873 conn(0x7f56ec071960 msgr2=0x7f56ec0a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 shutdown_connections 2024-09-18T10:29:14.025 INFO:tasks.workunit.client.0.smithi150.stderr:2024-09-18T10:29:14.019+0000 7f56eb59e700 1 -- 172.21.15.150:0/8473873 wait complete. 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stdout:ceph auth health | grep AUTH_BAD_CAP 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ expect 'ceph auth health | grep AUTH_BAD_CAP' 1 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ cmd='ceph auth health | grep AUTH_BAD_CAP' 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ expected_ret=1 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo ceph auth health '|' grep AUTH_BAD_CAP 2024-09-18T10:29:14.033 INFO:tasks.workunit.client.0.smithi150.stderr:+ eval ceph auth health '|' grep AUTH_BAD_CAP 2024-09-18T10:29:14.353 INFO:tasks.workunit.client.0.smithi150.stderr:+ ret=1 2024-09-18T10:29:14.353 INFO:tasks.workunit.client.0.smithi150.stderr:+ [[ 1 -ne 1 ]] 2024-09-18T10:29:14.353 INFO:tasks.workunit.client.0.smithi150.stderr:+ return 0 2024-09-18T10:29:14.353 INFO:tasks.workunit.client.0.smithi150.stdout:OK 2024-09-18T10:29:14.354 INFO:tasks.workunit.client.0.smithi150.stderr:+ echo OK 2024-09-18T10:29:14.355 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:29:14.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:29:14.368 INFO:tasks.workunit:Stopping ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh'] on client.0... 2024-09-18T10:29:14.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T10:29:14.604 DEBUG:teuthology.parallel:result is None 2024-09-18T10:29:14.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T10:29:14.613 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T10:29:14.613 DEBUG:teuthology.orchestra.run.smithi150:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T10:29:14.627 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T10:29:14.627 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-18T10:29:14.637 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-18T10:29:29.557 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-18T10:29:29.557 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-18T10:29:29.557 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T10:29:29.558 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T10:29:29.560 INFO:tasks.ceph.mon.c:Started 2024-09-18T10:29:29.560 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:29:29.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:29.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: get_monmap_and_config 2024-09-18T10:29:29.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: build_initial_monmap 2024-09-18T10:29:29.662 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: monmap: 2024-09-18T10:29:29.662 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:29.657749+0000 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:29.657749+0000 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 Processor -- start 2024-09-18T10:29:29.663 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 -- start start 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: init 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f7409bed0 0x7f2f74099240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-a con 0x7f2f7409bed0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 0x7f2f7409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-c con 0x7f2f7409c2a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74138a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-b con 0x7f2f74136690 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.664 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f74098c00 con 0x7f2f7409bed0 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7409de00 con 0x7f2f7409c2a0 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7409d880 con 0x7f2f74136690 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f79857700 10 monclient(hunting): authenticate will time out at 2058.583258s 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f737fe700 1 -- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 msgr2=0x7f2f74138a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.142:3301/0 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f7409bed0 0x7f2f74099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f737fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2f7409bed0 0x7f2f74099240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43588/0 (socket says 172.21.15.142:43588) 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 -- 172.21.15.142:0/282416819 learned_addr learned my addr 172.21.15.142:0/282416819 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:29.665 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): get_auth_request con 0x7f2f7409bed0 auth_method 0 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_done global_id 224217 payload 274 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 -- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 msgr2=0x7f2f74138a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f727fc700 1 --2- 172.21.15.142:0/282416819 >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 0x7f2f7409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 -- 172.21.15.142:0/282416819 >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 msgr2=0x7f2f7409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/282416819 >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 0x7f2f7409c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:29.666 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 monclient: found mon.noname-a 2024-09-18T10:29:29.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:29.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f72ffd700 1 -- 172.21.15.142:0/282416819 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7406f770 con 0x7f2f7409bed0 2024-09-18T10:29:29.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f727fc700 10 monclient: get_auth_request con 0x7f2f7409c2a0 auth_method 0 2024-09-18T10:29:29.667 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.651+0000 7f2f727fc700 1 --2- 172.21.15.142:0/282416819 >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 0x7f2f7409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f72ffd700 1 -- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 msgr2=0x7f2f74099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74099240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f74095140 tx=0x7f2f64009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578936d284b8f0bc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 1 -- 172.21.15.142:0/282416819 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6400ba70 con 0x7f2f7409bed0 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: dump: 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.674 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 5 monclient: authenticate success, global_id 224217 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 1 -- 172.21.15.142:0/282416819 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6400bbd0 con 0x7f2f7409bed0 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:29.675 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 4 set_mon_vals no callback set 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 1 -- 172.21.15.142:0/282416819 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64011420 con 0x7f2f7409bed0 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f71ffb700 10 monclient: dump: 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.676 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 10 monclient: shutdown 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 -- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 msgr2=0x7f2f74099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 --2- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74099240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f74095140 tx=0x7f2f64009250 comp rx=0 tx=0).stop 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 -- 172.21.15.142:0/282416819 shutdown_connections 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 --2- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 --2- 172.21.15.142:0/282416819 >> v2:172.21.15.150:3300/0 conn(0x7f2f7409c2a0 0x7f2f7409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 --2- 172.21.15.142:0/282416819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74099240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.677 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 -- 172.21.15.142:0/282416819 >> 172.21.15.142:0/282416819 conn(0x7f2f74071820 msgr2=0x7f2f740a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 -- 172.21.15.142:0/282416819 shutdown_connections 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.655+0000 7f2f79857700 1 -- 172.21.15.142:0/282416819 wait complete. 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient: build_initial_monmap 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient: monmap: 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:29.678 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 Processor -- start 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 -- start start 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient: init 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-a con 0x7f2f7409bed0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f7409c2a0 0x7f2f741462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-b con 0x7f2f7409c2a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): picked mon.noname-c con 0x7f2f74136690 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:29.679 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74126e70 con 0x7f2f7409bed0 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74093c30 con 0x7f2f7409c2a0 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74127ad0 con 0x7f2f74136690 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f79857700 10 monclient(hunting): authenticate will time out at 2058.589061s 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43616/0 (socket says 172.21.15.142:43616) 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 -- 172.21.15.142:0/3851568695 learned_addr learned my addr 172.21.15.142:0/3851568695 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): get_auth_request con 0x7f2f7409bed0 auth_method 0 2024-09-18T10:29:29.680 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f7409c2a0 0x7f2f741462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): get_auth_request con 0x7f2f7409c2a0 auth_method 0 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 10 monclient(hunting): get_auth_request con 0x7f2f74136690 auth_method 0 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:29.681 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient(hunting): handle_auth_done global_id 224220 payload 824 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 msgr2=0x7f2f74091400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 msgr2=0x7f2f74091400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 msgr2=0x7f2f74091400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f737fe700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f7409c2a0 msgr2=0x7f2f741462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:29.682 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f7409c2a0 0x7f2f741462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 monclient: found mon.noname-a 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.659+0000 7f2f72ffd700 1 -- 172.21.15.142:0/3851568695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f640094f0 con 0x7f2f7409bed0 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f72ffd700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 msgr2=0x7f2f74145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f72ffd700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6401a420 con 0x7f2f7409bed0 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: dump: 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:29.683 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: _finish_auth 0 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 5 monclient: authenticate success, global_id 224220 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _renew_subs 2024-09-18T10:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f741268b0 con 0x7f2f7409bed0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _renew_subs 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f74097ee0 con 0x7f2f7409bed0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6401a5c0 con 0x7f2f7409bed0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6401a940 con 0x7f2f7409bed0 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f717fa700 4 set_mon_vals no callback set 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 10 monclient: dump: 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540050b0 con 0x7f2f7409bed0 2024-09-18T10:29:29.686 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f2f6402e3f0 con 0x7f2f7409bed0 2024-09-18T10:29:29.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 0x7f2f60037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:29.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2f6404a700 con 0x7f2f7409bed0 2024-09-18T10:29:29.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f727fc700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 0x7f2f60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:29.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.663+0000 7f2f727fc700 10 monclient: get_auth_request con 0x7f2f60035ac0 auth_method 0 2024-09-18T10:29:29.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.667+0000 7f2f727fc700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 0x7f2f60037f70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2f74127360 tx=0x7f2f68006f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:29.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.671+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2f640253f0 con 0x7f2f7409bed0 2024-09-18T10:29:29.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.671+0000 7f2f5affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:29.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.671+0000 7f2f5affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:29.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.891+0000 7f2f79857700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:29.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.891+0000 7f2f79857700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:29.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.891+0000 7f2f79857700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:29.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.891+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2f540058e0 con 0x7f2f7409bed0 2024-09-18T10:29:29.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f5affd700 1 -- 172.21.15.142:0/3851568695 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7f2f64015030 con 0x7f2f7409bed0 2024-09-18T10:29:29.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f5affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:29.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f5affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:29.903 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 msgr2=0x7f2f60037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 0x7f2f60037f70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2f74127360 tx=0x7f2f68006f90 comp rx=0 tx=0).stop 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 10 monclient: shutdown 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 msgr2=0x7f2f74145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f64006fd0 tx=0x7f2f64017000 comp rx=0 tx=0).stop 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 shutdown_connections 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2f60035ac0 0x7f2f60037f70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2f74136690 0x7f2f74091400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f7409c2a0 0x7f2f741462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 --2- 172.21.15.142:0/3851568695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2f7409bed0 0x7f2f74145d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:29.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 >> 172.21.15.142:0/3851568695 conn(0x7f2f74071820 msgr2=0x7f2f740973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:29.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 shutdown_connections 2024-09-18T10:29:29.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:29.895+0000 7f2f79857700 1 -- 172.21.15.142:0/3851568695 wait complete. 2024-09-18T10:29:32.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:33.028 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.019+0000 7fad5e8fc700 10 monclient: get_monmap_and_config 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.019+0000 7fad5e8fc700 10 monclient: build_initial_monmap 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.019+0000 7fad5e8fc700 10 monclient: monmap: 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:33.026737+0000 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:33.026737+0000 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:33.029 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 Processor -- start 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 -- start start 2024-09-18T10:29:33.030 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient: init 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad580a1cf0 0x7fad5812f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-c con 0x7fad580a1cf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad5812f810 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 0x7fad58139420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-b con 0x7fad580a02d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5809fb90 con 0x7fad5812f810 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.033 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5809d990 con 0x7fad580a1cf0 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5809dc70 con 0x7fad580a02d0 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e8fc700 10 monclient(hunting): authenticate will time out at 2061.953603s 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 0x7fad58139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d0f9700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fad5812f810 0x7fad5809f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47144/0 (socket says 172.21.15.142:47144) 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d0f9700 1 -- 172.21.15.142:0/1542817242 learned_addr learned my addr 172.21.15.142:0/1542817242 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 10 monclient(hunting): get_auth_request con 0x7fad580a02d0 auth_method 0 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> v2:172.21.15.150:3300/0 conn(0x7fad580a1cf0 0x7fad5812f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.034 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request con 0x7fad580a1cf0 auth_method 0 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 msgr2=0x7fad58139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 msgr2=0x7fad58139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 0x7fad58139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5e0fb700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 0x7fad58139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_done global_id 194572 payload 274 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 msgr2=0x7fad58139420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-18T10:29:33.035 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a02d0 0x7fad58139420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> v1:172.21.15.142:6789/0 conn(0x7fad5812f810 legacy=0x7fad5809f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient: _finish_hunting 0 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 monclient: found mon.noname-c 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:29:33.036 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.023+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580a20c0 con 0x7fad580a1cf0 2024-09-18T10:29:33.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 msgr2=0x7fad5812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:33.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 0x7fad5812f030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad58071e60 tx=0x7fad540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5562835517afd3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:33.039 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 msgr2=0x7fad5812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 msgr2=0x7fad5812f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 0x7fad5812f030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad58071e60 tx=0x7fad540091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 0x7fad5812f030 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad58071e60 tx=0x7fad540091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient: ms_handle_reset current mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 msgr2=0x7fad5812f030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 0x7fad5812f030 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient: _add_conns ranks=[3,1] 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 1 --2- 172.21.15.142:0/1542817242 >> v2:172.21.15.142:3300/0 conn(0x7fad44003550 0x7fad440059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient(hunting): picked mon.noname-a con 0x7fad44003550 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad440062f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:33.040 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4400a250 con 0x7fad440062f0 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4400b410 con 0x7fad44003550 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5c8f8700 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> v2:172.21.15.142:3300/0 conn(0x7fad44003550 0x7fad440059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request con 0x7fad44003550 auth_method 0 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-18T10:29:33.041 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient(hunting): handle_auth_done global_id 194572 payload 293 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> v1:172.21.15.150:6789/0 conn(0x7fad440062f0 legacy=0x7fad440087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient: _finish_hunting 0 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 monclient: found mon.noname-a 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.027+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4400b640 con 0x7fad44003550 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5d8fa700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 msgr2=0x7fad440059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 0x7fad440059c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad540093f0 tx=0x7fad54017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f265877205c213e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54020070 con 0x7fad44003550 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: dump: 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:33.042 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: _finish_auth 0 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 5 monclient: authenticate success, global_id 194572 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5400b9b0 con 0x7fad44003550 2024-09-18T10:29:33.043 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 4 set_mon_vals no callback set 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 0x7fad440059c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad540093f0 tx=0x7fad54017000 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fad5401a630 mon_map magic: 0 v1 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: shutdown 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 msgr2=0x7fad440059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 0x7fad440059c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fad540093f0 tx=0x7fad54017000 comp rx=0 tx=0).stop 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e0fb700 1 -- 172.21.15.142:0/1542817242 reap_dead start 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 1 -- 172.21.15.142:0/1542817242 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7fad5401a630 con 0x7fad44003550 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5c8f8700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- 172.21.15.142:0/1542817242 shutdown_connections 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad44003550 0x7fad440059c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.044 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/1542817242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad580a1cf0 0x7fad5812f030 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- 172.21.15.142:0/1542817242 >> 172.21.15.142:0/1542817242 conn(0x7fad58071820 msgr2=0x7fad580a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:33.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- 172.21.15.142:0/1542817242 shutdown_connections 2024-09-18T10:29:33.045 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- 172.21.15.142:0/1542817242 wait complete. 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: build_initial_monmap 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: monmap: 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:33.050 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 Processor -- start 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- start start 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: init 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-c con 0x7fad580a1cf0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 0x7fad58142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-b con 0x7fad44003550 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): picked mon.noname-a con 0x7fad5806bb30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5806fa20 con 0x7fad5806bb30 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5809cdc0 con 0x7fad44003550 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad580a5470 con 0x7fad580a1cf0 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e8fc700 10 monclient(hunting): authenticate will time out at 2061.963772s 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e0fb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5d8fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.031+0000 7fad5e0fb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38404/0 (socket says 172.21.15.142:38404) 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 -- 172.21.15.142:0/4214751450 learned_addr learned my addr 172.21.15.142:0/4214751450 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): get_auth_request con 0x7fad5806bb30 auth_method 0 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.052 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request con 0x7fad580a1cf0 auth_method 0 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 0x7fad58142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 10 monclient(hunting): get_auth_request con 0x7fad44003550 auth_method 0 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:33.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 msgr2=0x7fad58142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 msgr2=0x7fad58142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient(hunting): handle_auth_done global_id 224223 payload 824 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 msgr2=0x7fad58142150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 msgr2=0x7fad58142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 0x7fad58142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient: _finish_hunting 0 2024-09-18T10:29:33.054 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 monclient: found mon.noname-a 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 -- 172.21.15.142:0/4214751450 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5806f320 con 0x7fad5806bb30 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5d0f9700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 0x7fad58142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 msgr2=0x7fad5806bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:33.056 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e0fb700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad44009fe0 tx=0x7fad500097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5000ed50 con 0x7fad5806bb30 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: dump: 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:33.057 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50014420 con 0x7fad5806bb30 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 5 monclient: authenticate success, global_id 224223 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 10 monclient: _renew_subs 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad581300b0 con 0x7fad5806bb30 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:33.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad500147e0 con 0x7fad5806bb30 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4dffb700 10 monclient: dump: 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:33.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad4ffff700 4 set_mon_vals no callback set 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 10 monclient: _renew_subs 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.035+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5806d9a0 con 0x7fad5806bb30 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fad50014c10 con 0x7fad5806bb30 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5e8fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5e8fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5e8fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5806ded0 con 0x7fad5806bb30 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad4dffb700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 0x7fad3c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fad50049590 con 0x7fad5806bb30 2024-09-18T10:29:33.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 0x7fad3c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.039+0000 7fad5d8fa700 10 monclient: get_auth_request con 0x7fad3c035ac0 auth_method 0 2024-09-18T10:29:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.043+0000 7fad5d8fa700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 0x7fad3c037f70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fad54002000 tx=0x7fad5401d070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.047+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fad50009880 con 0x7fad5806bb30 2024-09-18T10:29:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.047+0000 7fad4dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:33.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.047+0000 7fad4dffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad5e8fc700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad5e8fc700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad5e8fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad58072a80 con 0x7fad5806bb30 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad4dffb700 1 -- 172.21.15.142:0/4214751450 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1535 (secure 0 0 0) 0x7fad5001b310 con 0x7fad5806bb30 2024-09-18T10:29:33.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad4dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:33.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.271+0000 7fad4dffb700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:33.281 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":194,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 msgr2=0x7fad3c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 0x7fad3c037f70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fad54002000 tx=0x7fad5401d070 comp rx=0 tx=0).stop 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 10 monclient: shutdown 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 msgr2=0x7fad5806bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad44009fe0 tx=0x7fad500097f0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 shutdown_connections 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fad3c035ac0 0x7fad3c037f70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad5806bb30 0x7fad5806bf60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad44003550 0x7fad58142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 --2- 172.21.15.142:0/4214751450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fad580a1cf0 0x7fad58142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:33.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 >> 172.21.15.142:0/4214751450 conn(0x7fad58071820 msgr2=0x7fad58138f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:33.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 shutdown_connections 2024-09-18T10:29:33.284 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:33.275+0000 7fad5e8fc700 1 -- 172.21.15.142:0/4214751450 wait complete. 2024-09-18T10:29:36.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:36.407 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: get_monmap_and_config 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: monmap: 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:36.405405+0000 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:36.405405+0000 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:36.408 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:36.409 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:36.409 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:36.409 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 Processor -- start 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 -- start start 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: init 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab6009a7f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab600993f0 0x7fab6013b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-c con 0x7fab600993f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab6013bb30 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6009abc0 con 0x7fab6013bb30 2024-09-18T10:29:36.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6009d950 con 0x7fab6009a7f0 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab60099810 con 0x7fab600993f0 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient(hunting): authenticate will time out at 2065.330976s 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab6553b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fab6013bb30 0x7fab6013df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47160/0 (socket says 172.21.15.142:47160) 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab6553b700 1 -- 172.21.15.142:0/2242280347 learned_addr learned my addr 172.21.15.142:0/2242280347 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 --2- 172.21.15.142:0/2242280347 >> v2:172.21.15.150:3300/0 conn(0x7fab600993f0 0x7fab6013b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): get_auth_request con 0x7fab600993f0 auth_method 0 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient(hunting): handle_auth_done global_id 194578 payload 274 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 -- 172.21.15.142:0/2242280347 >> v1:172.21.15.150:6789/0 conn(0x7fab6009a7f0 legacy=0x7fab60098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 -- 172.21.15.142:0/2242280347 >> v1:172.21.15.142:6789/0 conn(0x7fab6013bb30 legacy=0x7fab6013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 monclient: found mon.noname-c 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 -- 172.21.15.142:0/2242280347 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6009db90 con 0x7fab600993f0 2024-09-18T10:29:36.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 -- 172.21.15.142:0/2242280347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab600993f0 msgr2=0x7fab6013b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5ffff700 1 --2- 172.21.15.142:0/2242280347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab600993f0 0x7fab6013b5f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab50009e60 tx=0x7fab50009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89738221d6fd1ab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 1 -- 172.21.15.142:0/2242280347 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5000e070 con 0x7fab600993f0 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: dump: 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.418 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 5 monclient: authenticate success, global_id 194578 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 1 -- 172.21.15.142:0/2242280347 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab50012430 con 0x7fab600993f0 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 1 -- 172.21.15.142:0/2242280347 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab500127d0 con 0x7fab600993f0 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab5f7fe700 10 monclient: dump: 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.419 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:36.420 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 4 set_mon_vals no callback set 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 10 monclient: shutdown 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 -- 172.21.15.142:0/2242280347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab600993f0 msgr2=0x7fab6013b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.403+0000 7fab65d3c700 1 --2- 172.21.15.142:0/2242280347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab600993f0 0x7fab6013b5f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab50009e60 tx=0x7fab50009780 comp rx=0 tx=0).stop 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- 172.21.15.142:0/2242280347 shutdown_connections 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 --2- 172.21.15.142:0/2242280347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab600993f0 0x7fab6013b5f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- 172.21.15.142:0/2242280347 >> 172.21.15.142:0/2242280347 conn(0x7fab600a1fe0 msgr2=0x7fab60070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- 172.21.15.142:0/2242280347 shutdown_connections 2024-09-18T10:29:36.421 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- 172.21.15.142:0/2242280347 wait complete. 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient: monmap: 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:36.422 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:36.423 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 Processor -- start 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- start start 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient: init 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:36.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-a con 0x7fab600993f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 0x7fab6006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-b con 0x7fab6009a7f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab6013bb30 0x7fab6006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): picked mon.noname-c con 0x7fab6013bb30 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006f970 con 0x7fab600993f0 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006d920 con 0x7fab6009a7f0 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006bdb0 con 0x7fab6013bb30 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab65d3c700 10 monclient(hunting): authenticate will time out at 2065.339476s 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38418/0 (socket says 172.21.15.142:38418) 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 1 -- 172.21.15.142:0/3200765719 learned_addr learned my addr 172.21.15.142:0/3200765719 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): get_auth_request con 0x7fab600993f0 auth_method 0 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab6013bb30 0x7fab6006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab64d3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): get_auth_request con 0x7fab6013bb30 auth_method 0 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab5ffff700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 0x7fab6006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab5ffff700 10 monclient(hunting): get_auth_request con 0x7fab6009a7f0 auth_method 0 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.407+0000 7fab6553b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab6553b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 10 monclient(hunting): handle_auth_done global_id 224226 payload 824 2024-09-18T10:29:36.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 -- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab6013bb30 msgr2=0x7fab6006a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab6013bb30 0x7fab6006a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 -- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 msgr2=0x7fab6006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 0x7fab6006a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 monclient: found mon.noname-a 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 -- 172.21.15.142:0/3200765719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab50009a20 con 0x7fab600993f0 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5ffff700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 0x7fab6006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 -- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 msgr2=0x7fab6006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab64d3a700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c5d0 tx=0x7fab54007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab540117a0 con 0x7fab600993f0 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: dump: 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab54011900 con 0x7fab600993f0 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 5 monclient: authenticate success, global_id 224226 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 10 monclient: _renew_subs 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6006b910 con 0x7fab600993f0 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab54011cc0 con 0x7fab600993f0 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5cff9700 10 monclient: dump: 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.438 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab5effd700 4 set_mon_vals no callback set 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 10 monclient: _renew_subs 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.411+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6006e890 con 0x7fab600993f0 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fab54012630 con 0x7fab600993f0 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab5cff9700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 0x7fab4c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab65d3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab65d3c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab65d3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fab54048d90 con 0x7fab600993f0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60072a80 con 0x7fab600993f0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab5ffff700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 0x7fab4c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.415+0000 7fab5ffff700 10 monclient: get_auth_request con 0x7fab4c035ac0 auth_method 0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.419+0000 7fab5ffff700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 0x7fab4c037f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fab500099a0 tx=0x7fab50018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.423+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fab54011e20 con 0x7fab600993f0 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.423+0000 7fab5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.423+0000 7fab5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:36.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab65d3c700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:36.655 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab65d3c700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:36.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab65d3c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fab6009d700 con 0x7fab600993f0 2024-09-18T10:29:36.656 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab5cff9700 1 -- 172.21.15.142:0/3200765719 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fab540102e0 con 0x7fab600993f0 2024-09-18T10:29:36.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab5cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:36.657 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.647+0000 7fab5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:36.657 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":196,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 msgr2=0x7fab4c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 0x7fab4c037f70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fab500099a0 tx=0x7fab50018000 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 10 monclient: shutdown 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 msgr2=0x7fab6006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fab5400c5d0 tx=0x7fab54007f60 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 shutdown_connections 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fab4c035ac0 0x7fab4c037f70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab6013bb30 0x7fab6006a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab6009a7f0 0x7fab6006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 --2- 172.21.15.142:0/3200765719 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab600993f0 0x7fab6006e350 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.659 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 >> 172.21.15.142:0/3200765719 conn(0x7fab600a1fe0 msgr2=0x7fab6009f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:36.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 shutdown_connections 2024-09-18T10:29:36.660 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.651+0000 7fab65d3c700 1 -- 172.21.15.142:0/3200765719 wait complete. 2024-09-18T10:29:36.670 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:29:36.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-18T10:29:36.670 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: get_monmap_and_config 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: monmap: 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:36.774620+0000 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:36.774620+0000 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:36.777 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:36.778 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:36.778 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:36.778 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:36.778 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 Processor -- start 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 -- start start 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: init 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbd009cdd0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbbd0134a50 0x7fbbd0136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-a con 0x7fbbd0134a50 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 0x7fbbd0139750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-b con 0x7fbbd0137370 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:36.781 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd009d1a0 con 0x7fbbd009cdd0 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd009d780 con 0x7fbbd0134a50 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd009dd40 con 0x7fbbd0137370 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbd6025700 10 monclient(hunting): authenticate will time out at 2065.700898s 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 0x7fbbd0139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbbd0134a50 0x7fbbd0136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcf7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbbd009cdd0 0x7fbbd009e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47174/0 (socket says 172.21.15.142:47174) 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/2611689407 learned_addr learned my addr 172.21.15.142:0/2611689407 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): get_auth_request con 0x7fbbd0134a50 auth_method 0 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.782 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): get_auth_request con 0x7fbbd0137370 auth_method 0 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_done global_id 224229 payload 274 2024-09-18T10:29:36.783 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 -- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 msgr2=0x7fbbd0139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 0x7fbbd0139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 -- 172.21.15.142:0/2611689407 >> v1:172.21.15.142:6789/0 conn(0x7fbbd009cdd0 legacy=0x7fbbd009e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 monclient: found mon.noname-a 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbc7fff700 1 -- 172.21.15.142:0/2611689407 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd00a6710 con 0x7fbbd0134a50 2024-09-18T10:29:36.784 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.771+0000 7fbbcffff700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 0x7fbbd0139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbc7fff700 1 -- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0134a50 msgr2=0x7fbbd0136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbc7fff700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0134a50 0x7fbbd0136e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098e0 tx=0x7fbbb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0a9670900c1194 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.788 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 1 -- 172.21.15.142:0/2611689407 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800b850 con 0x7fbbd0134a50 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: dump: 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.789 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 5 monclient: authenticate success, global_id 224229 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 1 -- 172.21.15.142:0/2611689407 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb800b9b0 con 0x7fbbd0134a50 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 4 set_mon_vals no callback set 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 1 -- 172.21.15.142:0/2611689407 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800bd50 con 0x7fbbd0134a50 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbceffd700 10 monclient: dump: 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.791 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.792 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.793 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.793 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.793 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.793 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.793 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 10 monclient: shutdown 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0134a50 msgr2=0x7fbbd0136e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0134a50 0x7fbbd0136e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098e0 tx=0x7fbbb80091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- 172.21.15.142:0/2611689407 shutdown_connections 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd0137370 0x7fbbd0139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 --2- 172.21.15.142:0/2611689407 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0134a50 0x7fbbd0136e30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- 172.21.15.142:0/2611689407 >> 172.21.15.142:0/2611689407 conn(0x7fbbd0071680 msgr2=0x7fbbd00a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- 172.21.15.142:0/2611689407 shutdown_connections 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- 172.21.15.142:0/2611689407 wait complete. 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 10 monclient: monmap: 2024-09-18T10:29:36.794 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:36.795 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 Processor -- start 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 1 -- start start 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.775+0000 7fbbd6025700 10 monclient: init 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd009cdd0 0x7fbbd006b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.796 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-c con 0x7fbbd009cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd0134a50 0x7fbbd006b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-b con 0x7fbbd0134a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 0x7fbbd006bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): picked mon.noname-a con 0x7fbbd0137370 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.797 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd00a3760 con 0x7fbbd0137370 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd006ebd0 con 0x7fbbd0134a50 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd006ce70 con 0x7fbbd009cdd0 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd009cdd0 0x7fbbd006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient(hunting): authenticate will time out at 2065.708400s 2024-09-18T10:29:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd009cdd0 0x7fbbd006b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37934/0 (socket says 172.21.15.142:37934) 2024-09-18T10:29:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/1251498287 learned_addr learned my addr 172.21.15.142:0/1251498287 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): get_auth_request con 0x7fbbd009cdd0 auth_method 0 2024-09-18T10:29:36.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 0x7fbbd006bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): get_auth_request con 0x7fbbd0137370 auth_method 0 2024-09-18T10:29:36.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd0134a50 0x7fbbd006b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): get_auth_request con 0x7fbbd0134a50 auth_method 0 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient(hunting): handle_auth_done global_id 224232 payload 824 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd009cdd0 msgr2=0x7fbbd006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbbd009cdd0 0x7fbbd006b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd0134a50 msgr2=0x7fbbd006b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbd0134a50 0x7fbbd006b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 monclient: found mon.noname-a 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc8009a20 con 0x7fbbd0137370 2024-09-18T10:29:36.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 msgr2=0x7fbbd006bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbcffff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 0x7fbbd006bde0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009f10 tx=0x7fbbc800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.810 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc800cd80 con 0x7fbbd0137370 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: dump: 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.811 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc800f040 con 0x7fbbd0137370 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 5 monclient: authenticate success, global_id 224232 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _renew_subs 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb8009420 con 0x7fbbd0137370 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc80146d0 con 0x7fbbd0137370 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbce7fc700 4 set_mon_vals no callback set 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbc77fe700 10 monclient: dump: 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _renew_subs 2024-09-18T10:29:36.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.779+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd006da00 con 0x7fbbd0137370 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbd6025700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbd6025700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 0x7fbbd00997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd00648c0 con 0x7fbbd006c320 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fbbc8003020 con 0x7fbbd0137370 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 0x7fbbd00997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient: get_auth_request con 0x7fbbd006c320 auth_method 0 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbc77fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbbb0035b40 0x7fbbb0037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fbbc80497c0 con 0x7fbbd0137370 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_done global_id 224235 payload 824 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbd00696a0 con 0x7fbbd006c320 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbc7fff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbbb0035b40 0x7fbbb0037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.783+0000 7fbbc7fff700 10 monclient: get_auth_request con 0x7fbbb0035b40 auth_method 0 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 msgr2=0x7fbbd00997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbcf7fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 0x7fbbd00997c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000eac0 tx=0x7fbbc0004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc000cbb0 con 0x7fbbd006c320 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 10 monclient: dump: 2024-09-18T10:29:36.815 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fbbc000cd10 con 0x7fbbd006c320 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc7fff700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbbb0035b40 0x7fbbb0037ff0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80030c0 tx=0x7fbbb8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 msgr2=0x7fbbd00997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 0x7fbbd00997c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000eac0 tx=0x7fbbc0004a80 comp rx=0 tx=0).stop 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 msgr2=0x7fbbd00997c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.787+0000 7fbbc77fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd006c320 0x7fbbd00997c0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000eac0 tx=0x7fbbc0004a80 comp rx=0 tx=0).stop 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbd6025700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbd6025700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 0x7fbbd009a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbd6025700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd006d580 con 0x7fbbd0099d00 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 0x7fbbd009a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient: get_auth_request con 0x7fbbd0099d00 auth_method 0 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 10 monclient(hunting): handle_auth_done global_id 224238 payload 824 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.795+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/1251498287 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbbd006ed90 con 0x7fbbd0099d00 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbcf7fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 msgr2=0x7fbbd009a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbcf7fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 0x7fbbd009a100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0018000 tx=0x7fbbc0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0013b40 con 0x7fbbd0099d00 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 10 monclient: dump: 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 <== mon.0 v2:172.21.15.142:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4580 (secure 0 0 0) 0x7fbbc0025070 con 0x7fbbd0099d00 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:29:36.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:36.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 msgr2=0x7fbbd009a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 0x7fbbd009a100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0018000 tx=0x7fbbc0016000 comp rx=0 tx=0).stop 2024-09-18T10:29:36.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 msgr2=0x7fbbd009a100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:36.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbc77fe700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0099d00 0x7fbbd009a100 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0018000 tx=0x7fbbc0016000 comp rx=0 tx=0).stop 2024-09-18T10:29:36.820 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "leader", 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 196, 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:36.821 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:36.822 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:36.823 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:29:36.824 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:36.825 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:29:36.826 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:36.827 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.828 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:36.829 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.830 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:36.831 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.832 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 3 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:29:36.833 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:36.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbbb0035b40 msgr2=0x7fbbb0037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fbbb0035b40 0x7fbbb0037ff0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80030c0 tx=0x7fbbb8016000 comp rx=0 tx=0).stop 2024-09-18T10:29:36.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 10 monclient: shutdown 2024-09-18T10:29:36.835 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 msgr2=0x7fbbd006bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 --2- 172.21.15.142:0/1251498287 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbbd0137370 0x7fbbd006bde0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009f10 tx=0x7fbbc800b040 comp rx=0 tx=0).stop 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 reap_dead start 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbcffff700 1 -- 172.21.15.142:0/1251498287 reap_dead start 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 shutdown_connections 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 >> 172.21.15.142:0/1251498287 conn(0x7fbbd0071680 msgr2=0x7fbbd0136610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 shutdown_connections 2024-09-18T10:29:36.836 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.799+0000 7fbbd6025700 1 -- 172.21.15.142:0/1251498287 wait complete. 2024-09-18T10:29:36.837 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:29:36.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-18T10:29:36.837 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-18T10:29:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.943+0000 7f8515d49700 10 monclient: get_monmap_and_config 2024-09-18T10:29:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.943+0000 7f8515d49700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.943+0000 7f8515d49700 10 monclient: monmap: 2024-09-18T10:29:36.953 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:36.951071+0000 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:36.951071+0000 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:36.954 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:36.955 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:36.955 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:36.955 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 Processor -- start 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- start start 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: init 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f851006f3b0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85100990d0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 0x7f8510137da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-b con 0x7f85100994f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:36.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851009c330 con 0x7f85100990d0 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851009c530 con 0x7f851006f3b0 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851009b380 con 0x7f85100994f0 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient(hunting): authenticate will time out at 2065.876985s 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8507fff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f85100990d0 0x7f8510134670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47176/0 (socket says 172.21.15.142:47176) 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8507fff700 1 -- 172.21.15.142:0/2993290538 learned_addr learned my addr 172.21.15.142:0/2993290538 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 --2- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 0x7f8510137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): get_auth_request con 0x7f85100994f0 auth_method 0 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient(hunting): handle_auth_done global_id 234110 payload 274 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 -- 172.21.15.142:0/2993290538 >> v1:172.21.15.150:6789/0 conn(0x7f851006f3b0 legacy=0x7f85100989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 -- 172.21.15.142:0/2993290538 >> v1:172.21.15.142:6789/0 conn(0x7f85100990d0 legacy=0x7f8510134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 monclient: found mon.noname-b 2024-09-18T10:29:36.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:36.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 -- 172.21.15.142:0/2993290538 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851009c0f0 con 0x7f85100994f0 2024-09-18T10:29:36.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 -- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 msgr2=0x7f8510137da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515548700 1 --2- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 0x7f8510137da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e90 tx=0x7f8508009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce34b4569387c882 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 1 -- 172.21.15.142:0/2993290538 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800e070 con 0x7f85100994f0 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: dump: 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.974 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 5 monclient: authenticate success, global_id 234110 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 1 -- 172.21.15.142:0/2993290538 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8508012430 con 0x7f85100994f0 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 4 set_mon_vals no callback set 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 1 -- 172.21.15.142:0/2993290538 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85080127d0 con 0x7f85100994f0 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f85077fe700 10 monclient: dump: 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.976 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: shutdown 2024-09-18T10:29:36.977 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:36.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 msgr2=0x7f8510137da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 --2- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 0x7f8510137da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e90 tx=0x7f8508009780 comp rx=0 tx=0).stop 2024-09-18T10:29:36.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- 172.21.15.142:0/2993290538 shutdown_connections 2024-09-18T10:29:36.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 --2- 172.21.15.142:0/2993290538 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f85100994f0 0x7f8510137da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- 172.21.15.142:0/2993290538 >> 172.21.15.142:0/2993290538 conn(0x7f8510070ef0 msgr2=0x7f85100712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- 172.21.15.142:0/2993290538 shutdown_connections 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- 172.21.15.142:0/2993290538 wait complete. 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: build_initial_monmap 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: monmap: 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:36.979 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 Processor -- start 2024-09-18T10:29:36.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 1 -- start start 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.951+0000 7f8515d49700 10 monclient: init 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 0x7f8510089870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-c con 0x7f851006f3b0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100990d0 0x7f8510089db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-b con 0x7f85100990d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:36.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 0x7f851008a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): picked mon.noname-a con 0x7f85100994f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851006fa20 con 0x7f85100994f0 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8510096f80 con 0x7f85100990d0 2024-09-18T10:29:36.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851008be70 con 0x7f851006f3b0 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515d49700 10 monclient(hunting): authenticate will time out at 2065.884725s 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 0x7f851008a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 0x7f8510089870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 0x7f8510089870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37960/0 (socket says 172.21.15.142:37960) 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 0x7f851008a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38472/0 (socket says 172.21.15.142:38472) 2024-09-18T10:29:36.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 -- 172.21.15.142:0/3673780106 learned_addr learned my addr 172.21.15.142:0/3673780106 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): get_auth_request con 0x7f851006f3b0 auth_method 0 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): get_auth_request con 0x7f85100994f0 auth_method 0 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.984 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100990d0 0x7f8510089db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.985 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 10 monclient(hunting): get_auth_request con 0x7f85100990d0 auth_method 0 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient(hunting): handle_auth_done global_id 234113 payload 824 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100990d0 msgr2=0x7f8510089db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100990d0 0x7f8510089db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.986 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 msgr2=0x7f851008a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 0x7f851008a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient: _finish_hunting 0 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 monclient: found mon.noname-c 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508009a20 con 0x7f851006f3b0 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85100990d0 0x7f8510089db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:36.987 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8515548700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f85100994f0 0x7f851008a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 msgr2=0x7f8510089870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.955+0000 7f8514d47700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 0x7f8510089870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c5d0 tx=0x7f850c007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c0117a0 con 0x7f851006f3b0 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: dump: 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.988 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.989 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 5 monclient: authenticate success, global_id 234113 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 monclient: _renew_subs 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c011900 con 0x7f851006f3b0 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851008b2e0 con 0x7f851006f3b0 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:36.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c011cc0 con 0x7f851006f3b0 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8506ffd700 4 set_mon_vals no callback set 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 10 monclient: dump: 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:36.991 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 monclient: _renew_subs 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8510094d70 con 0x7f851006f3b0 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85100648a0 con 0x7f851008a830 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f850c012630 con 0x7f851006f3b0 2024-09-18T10:29:36.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient: get_auth_request con 0x7f851008a830 auth_method 0 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 10 monclient(hunting): handle_auth_done global_id 234116 payload 824 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8510072a80 con 0x7f851008a830 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 msgr2=0x7f851006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 msgr2=0x7f851006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:36.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84fc035b10 0x7f84fc037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f850000e740 tx=0x7f8500009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f850c048ca0 con 0x7f851006f3b0 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515548700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84fc035b10 0x7f84fc037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.959+0000 7f8515548700 10 monclient: get_auth_request con 0x7f84fc035b10 auth_method 0 2024-09-18T10:29:36.994 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:36.967+0000 7f8515548700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84fc035b10 0x7f84fc037fc0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e90 tx=0x7f8508018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: get_auth_request con 0x7f851008a830 auth_method 0 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: _init_auth method 2 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: _init_auth already have auth, reseting 2024-09-18T10:29:37.172 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: handle_auth_reply_more payload 9 2024-09-18T10:29:37.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-18T10:29:37.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 10 monclient: handle_auth_done global_id 234116 payload 293 2024-09-18T10:29:37.173 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 msgr2=0x7f851006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.174 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.163+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f850000e740 tx=0x7f8500009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8500010b90 con 0x7f851008a830 2024-09-18T10:29:37.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:37.178 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 10 monclient: dump: 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.179 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8500010cf0 con 0x7f851008a830 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 msgr2=0x7f851006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f850000e740 tx=0x7f8500009040 comp rx=0 tx=0).stop 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 msgr2=0x7f851006c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:37.180 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.171+0000 7f8504ff9700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851008a830 0x7f851006c370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:29:37.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 0x7f851006ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.187 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851008ac00 con 0x7f851006c8b0 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 0x7f851006ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient: get_auth_request con 0x7f851006c8b0 auth_method 0 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.188 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 10 monclient(hunting): handle_auth_done global_id 234119 payload 824 2024-09-18T10:29:37.189 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f85100678a0 con 0x7f851006c8b0 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 msgr2=0x7f851006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8507fff700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 0x7f851006ccb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f850001c000 tx=0x7f850001d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000159e0 con 0x7f851006c8b0 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:37.190 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 10 monclient: dump: 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.191 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.192 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 <== mon.2 v2:172.21.15.142:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4408 (secure 0 0 0) 0x7f8500010980 con 0x7f851006c8b0 2024-09-18T10:29:37.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:29:37.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:37.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 msgr2=0x7f851006ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 0x7f851006ccb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f850001c000 tx=0x7f850001d000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 msgr2=0x7f851006ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:37.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8504ff9700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006c8b0 0x7f851006ccb0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f850001c000 tx=0x7f850001d000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.194 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:37.194 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:37.194 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:37.194 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:29:37.194 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 196, 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.195 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.196 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:29:37.197 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:29:37.198 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:29:37.198 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:29:37.198 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:29:37.198 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:37.198 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:29:37.199 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.200 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.201 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.202 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:37.203 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.204 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.205 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.206 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.207 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.208 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:29:37.209 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.209 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.209 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.209 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:29:37.209 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:37.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84fc035b10 msgr2=0x7f84fc037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f84fc035b10 0x7f84fc037fc0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8508009e90 tx=0x7f8508018000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 10 monclient: shutdown 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 msgr2=0x7f8510089870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 --2- 172.21.15.142:0/3673780106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f851006f3b0 0x7f8510089870 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c5d0 tx=0x7f850c007f60 comp rx=0 tx=0).stop 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515548700 1 -- 172.21.15.142:0/3673780106 reap_dead start 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515548700 1 -- 172.21.15.142:0/3673780106 reap_dead start 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 shutdown_connections 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 >> 172.21.15.142:0/3673780106 conn(0x7f8510070ef0 msgr2=0x7f8510096720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:37.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 shutdown_connections 2024-09-18T10:29:37.213 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.183+0000 7f8515d49700 1 -- 172.21.15.142:0/3673780106 wait complete. 2024-09-18T10:29:37.213 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:29:37.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-18T10:29:37.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-18T10:29:37.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: get_monmap_and_config 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: build_initial_monmap 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: monmap: 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:37.323152+0000 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:37.323152+0000 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:37.326 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 Processor -- start 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 -- start start 2024-09-18T10:29:37.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: init 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff53009abd0 0x7ff5300960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): picked mon.noname-a con 0x7ff53009abd0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff53009b740 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff53009c5e0 0x7ff530137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): picked mon.noname-c con 0x7ff53009c5e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:37.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53009dc90 con 0x7ff53009b740 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53009b500 con 0x7ff53009abd0 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53009afa0 con 0x7ff53009c5e0 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff536280700 10 monclient(hunting): authenticate will time out at 2066.248668s 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff53009abd0 0x7ff5300960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff534a7d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff53009b740 0x7ff53009bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47178/0 (socket says 172.21.15.142:47178) 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff534a7d700 1 -- 172.21.15.142:0/1565782375 learned_addr learned my addr 172.21.15.142:0/1565782375 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): get_auth_request con 0x7ff53009abd0 auth_method 0 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 1 --2- 172.21.15.142:0/1565782375 >> v2:172.21.15.150:3300/0 conn(0x7ff53009c5e0 0x7ff530137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): get_auth_request con 0x7ff53009c5e0 auth_method 0 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff535a7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient(hunting): handle_auth_done global_id 224244 payload 274 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 -- 172.21.15.142:0/1565782375 >> v2:172.21.15.150:3300/0 conn(0x7ff53009c5e0 msgr2=0x7ff530137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 --2- 172.21.15.142:0/1565782375 >> v2:172.21.15.150:3300/0 conn(0x7ff53009c5e0 0x7ff530137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 -- 172.21.15.142:0/1565782375 >> v1:172.21.15.142:6789/0 conn(0x7ff53009b740 legacy=0x7ff53009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient: _finish_hunting 0 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 monclient: found mon.noname-a 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.319+0000 7ff53527e700 1 -- 172.21.15.142:0/1565782375 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53006f810 con 0x7ff53009abd0 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 1 -- 172.21.15.142:0/1565782375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009abd0 msgr2=0x7ff5300960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 1 --2- 172.21.15.142:0/1565782375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009abd0 0x7ff5300960e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff530076de0 tx=0x7ff52c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c370f027c98cfae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.338 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 1 -- 172.21.15.142:0/1565782375 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c011840 con 0x7ff53009abd0 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: dump: 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.339 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: _finish_auth 0 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 1 -- 172.21.15.142:0/1565782375 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52c0119c0 con 0x7ff53009abd0 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 5 monclient: authenticate success, global_id 224244 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 4 set_mon_vals no callback set 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 1 -- 172.21.15.142:0/1565782375 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c011d80 con 0x7ff53009abd0 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff527fff700 10 monclient: dump: 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.341 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: shutdown 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- 172.21.15.142:0/1565782375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009abd0 msgr2=0x7ff5300960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- 172.21.15.142:0/1565782375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009abd0 0x7ff5300960e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff530076de0 tx=0x7ff52c0091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- 172.21.15.142:0/1565782375 shutdown_connections 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- 172.21.15.142:0/1565782375 >> v2:172.21.15.150:3300/0 conn(0x7ff53009c5e0 0x7ff530137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- 172.21.15.142:0/1565782375 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009abd0 0x7ff5300960e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- 172.21.15.142:0/1565782375 >> 172.21.15.142:0/1565782375 conn(0x7ff530070ef0 msgr2=0x7ff5300712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- 172.21.15.142:0/1565782375 shutdown_connections 2024-09-18T10:29:37.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- 172.21.15.142:0/1565782375 wait complete. 2024-09-18T10:29:37.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: build_initial_monmap 2024-09-18T10:29:37.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: monmap: 2024-09-18T10:29:37.343 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:37.343 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:37.343 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:37.344 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 Processor -- start 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- start start 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: init 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:29:37.345 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff53009abd0 0x7ff530141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): picked mon.noname-c con 0x7ff53009abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): picked mon.noname-a con 0x7ff53009b740 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53009c5e0 0x7ff5301421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): picked mon.noname-b con 0x7ff53009c5e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.346 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53008fdd0 con 0x7ff53009b740 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53009f460 con 0x7ff53009c5e0 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff530091310 con 0x7ff53009abd0 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff536280700 10 monclient(hunting): authenticate will time out at 2066.255332s 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.347 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff53009abd0 0x7ff530141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38482/0 (socket says 172.21.15.142:38482) 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 1 -- 172.21.15.142:0/1843988071 learned_addr learned my addr 172.21.15.142:0/1843988071 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): get_auth_request con 0x7ff53009b740 auth_method 0 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.348 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): get_auth_request con 0x7ff53009abd0 auth_method 0 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.349 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff535a7f700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53009c5e0 0x7ff5301421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff535a7f700 10 monclient(hunting): get_auth_request con 0x7ff53009c5e0 auth_method 0 2024-09-18T10:29:37.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff535a7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff535a7f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.350 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff535a7f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.323+0000 7ff534a7d700 10 monclient(hunting): handle_auth_done global_id 224247 payload 824 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff53009abd0 msgr2=0x7ff530141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.351 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff53009abd0 0x7ff530141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53009c5e0 msgr2=0x7ff5301421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53009c5e0 0x7ff5301421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 monclient: found mon.noname-a 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.352 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c009420 con 0x7ff53009b740 2024-09-18T10:29:37.353 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff535a7f700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53009c5e0 0x7ff5301421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 msgr2=0x7ff530141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff534a7d700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff52000c5e0 tx=0x7ff520004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff520005d40 con 0x7ff53009b740 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: dump: 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.362 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: _finish_auth 0 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 5 monclient: authenticate success, global_id 224247 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5200070d0 con 0x7ff53009b740 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff520007490 con 0x7ff53009b740 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5257fa700 10 monclient: dump: 2024-09-18T10:29:37.363 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 monclient: _renew_subs 2024-09-18T10:29:37.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff5277fe700 4 set_mon_vals no callback set 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53009d240 con 0x7ff53009b740 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 monclient: _renew_subs 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff530092640 con 0x7ff53009b740 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 0x7ff53008a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.365 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.327+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5300648a0 con 0x7ff530089ff0 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 0x7ff53008a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient: get_auth_request con 0x7ff530089ff0 auth_method 0 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff534a7d700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff52000c5e0 tx=0x7ff520004a80 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7ff52001b3b0 mgrmap(e 5) v1 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff52001b3b0 con 0x7ff53009b740 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.366 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff514035b10 0x7ff514037fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 10 monclient(hunting): handle_auth_done global_id 194596 payload 824 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff530072a80 con 0x7ff530089ff0 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff535a7f700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff514035b10 0x7ff514037fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff535a7f700 10 monclient: get_auth_request con 0x7ff514035b10 auth_method 0 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 msgr2=0x7ff53008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff53527e700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 0x7ff53008a3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff52c016000 tx=0x7ff52c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c00be50 con 0x7ff530089ff0 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.335+0000 7ff5257fa700 10 monclient: dump: 2024-09-18T10:29:37.367 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff535a7f700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff514035b10 0x7ff514037fc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff5300916c0 tx=0x7ff52800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ff52c020040 con 0x7ff530089ff0 2024-09-18T10:29:37.368 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 msgr2=0x7ff53008a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 0x7ff53008a3c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff52c016000 tx=0x7ff52c017000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 msgr2=0x7ff53008a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.339+0000 7ff5257fa700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff530089ff0 0x7ff53008a3c0 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff52c016000 tx=0x7ff52c017000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff536280700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff536280700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 0x7ff53008dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff536280700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53008f8b0 con 0x7ff53008a900 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 0x7ff53008dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient: get_auth_request con 0x7ff53008a900 auth_method 0 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 10 monclient(hunting): handle_auth_done global_id 194599 payload 824 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.347+0000 7ff53527e700 1 -- 172.21.15.142:0/1843988071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff5300678a0 con 0x7ff53008a900 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff53527e700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 msgr2=0x7ff53008dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff53527e700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 0x7ff53008dd00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff53008f770 tx=0x7ff52c021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c01bb50 con 0x7ff53008a900 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 10 monclient: dump: 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.370 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.1 v2:172.21.15.150:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4748 (secure 0 0 0) 0x7ff52c026400 con 0x7ff53008a900 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 msgr2=0x7ff53008dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 0x7ff53008dd00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff53008f770 tx=0x7ff52c021000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 msgr2=0x7ff53008dd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff53008a900 0x7ff53008dd00 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff53008f770 tx=0x7ff52c021000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:37.372 INFO:teuthology.orchestra.run.smithi142.stdout: "state": "peon", 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: "election_epoch": 196, 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum": [ 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: 0, 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: 1, 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: 2 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_age": 2, 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:37.373 INFO:teuthology.orchestra.run.smithi142.stdout: "required_con": "2449958755906961412", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "required_mon": [ 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.374 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_con": "4540138320759226367", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quorum_mon": [ 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.375 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: "outside_quorum": [], 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: "extra_probe_peers": [], 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: "sync_provider": [], 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: "monmap": { 2024-09-18T10:29:37.376 INFO:teuthology.orchestra.run.smithi142.stdout: "epoch": 1, 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "fsid": "bfa92b4c-6f79-422f-bfaa-515c68f959fb", 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "modified": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "created": "2024-09-18T10:08:19.331585Z", 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release": 17, 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "min_mon_release_name": "quincy", 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "election_strategy": 1, 2024-09-18T10:29:37.377 INFO:teuthology.orchestra.run.smithi142.stdout: "disallowed_leaders: ": "", 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false, 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "tiebreaker_mon": "", 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "removed_ranks: ": "", 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "features": { 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "persistent": [ 2024-09-18T10:29:37.378 INFO:teuthology.orchestra.run.smithi142.stdout: "kraken", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "luminous", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "mimic", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "osdmap-prune", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "nautilus", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "octopus", 2024-09-18T10:29:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: "pacific", 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: "elector-pinging", 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: "quincy" 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: "optional": [] 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.380 INFO:teuthology.orchestra.run.smithi142.stdout: "mons": [ 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 0, 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "a", 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.381 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3300", 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.382 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789", 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.383 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6789/0", 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6789/0", 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.384 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.385 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 1, 2024-09-18T10:29:37.385 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "b", 2024-09-18T10:29:37.385 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.385 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.386 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.386 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.386 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:3300", 2024-09-18T10:29:37.386 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.386 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789", 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.387 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.150:6789/0", 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.150:6789/0", 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.388 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.389 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.389 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.389 INFO:teuthology.orchestra.run.smithi142.stdout: "rank": 2, 2024-09-18T10:29:37.389 INFO:teuthology.orchestra.run.smithi142.stdout: "name": "c", 2024-09-18T10:29:37.389 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addrs": { 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: "addrvec": [ 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v2", 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:3301", 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.390 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: "type": "v1", 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790", 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: "nonce": 0 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.391 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: "addr": "172.21.15.142:6790/0", 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: "public_addr": "172.21.15.142:6790/0", 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: "priority": 0, 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: "weight": 0, 2024-09-18T10:29:37.392 INFO:teuthology.orchestra.run.smithi142.stdout: "crush_location": "{}" 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: "feature_map": { 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: "mon": [ 2024-09-18T10:29:37.393 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.394 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.394 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.394 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:37.394 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.394 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: "osd": [ 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 2 2024-09-18T10:29:37.395 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: "client": [ 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.396 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 4 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: ], 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: "mgr": [ 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: { 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-18T10:29:37.397 INFO:teuthology.orchestra.run.smithi142.stdout: "release": "luminous", 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout: "num": 1 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout: } 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout: }, 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout: "stretch_mode": false 2024-09-18T10:29:37.398 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:37.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff514035b10 msgr2=0x7ff514037fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff514035b10 0x7ff514037fc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff5300916c0 tx=0x7ff52800b040 comp rx=0 tx=0).stop 2024-09-18T10:29:37.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 10 monclient: shutdown 2024-09-18T10:29:37.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 msgr2=0x7ff530141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 --2- 172.21.15.142:0/1843988071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff53009b740 0x7ff530141ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff52000c5e0 tx=0x7ff520004a80 comp rx=0 tx=0).stop 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff535a7f700 1 -- 172.21.15.142:0/1843988071 reap_dead start 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff5257fa700 1 -- 172.21.15.142:0/1843988071 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (unknown 0 0 0) 0x7ff5200484b0 con 0x7ff53009b740 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff535a7f700 1 -- 172.21.15.142:0/1843988071 reap_dead start 2024-09-18T10:29:37.402 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 shutdown_connections 2024-09-18T10:29:37.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 >> 172.21.15.142:0/1843988071 conn(0x7ff530070ef0 msgr2=0x7ff53009de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:37.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 shutdown_connections 2024-09-18T10:29:37.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.351+0000 7ff536280700 1 -- 172.21.15.142:0/1843988071 wait complete. 2024-09-18T10:29:37.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-18T10:29:37.404 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-18T10:29:37.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-18T10:29:37.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.511+0000 7feb751b2700 10 monclient: get_monmap_and_config 2024-09-18T10:29:37.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.511+0000 7feb751b2700 10 monclient: build_initial_monmap 2024-09-18T10:29:37.517 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.511+0000 7feb751b2700 10 monclient: monmap: 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:37.515419+0000 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:37.515419+0000 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:37.518 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:37.519 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 Processor -- start 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 -- start start 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient: init 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-18T10:29:37.522 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7feb70132030 0x7feb70132400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-a con 0x7feb70132030 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-b con 0x7feb7009ece0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb7009ae60 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb70132940 con 0x7feb7009ae60 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7009f960 con 0x7feb70132030 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb700a1a90 con 0x7feb7009ece0 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb751b2700 10 monclient(hunting): authenticate will time out at 2066.441102s 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.523 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37982/0 (socket says 172.21.15.142:37982) 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 -- 172.21.15.142:0/2493795242 learned_addr learned my addr 172.21.15.142:0/2493795242 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): get_auth_request con 0x7feb7009ece0 auth_method 0 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb6ffff700 1 --2- 172.21.15.142:0/2493795242 >> v2:172.21.15.142:3300/0 conn(0x7feb70132030 0x7feb70132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb6ffff700 10 monclient(hunting): get_auth_request con 0x7feb70132030 auth_method 0 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.524 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient(hunting): handle_auth_done global_id 234125 payload 274 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 -- 172.21.15.142:0/2493795242 >> v2:172.21.15.142:3300/0 conn(0x7feb70132030 msgr2=0x7feb70132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 --2- 172.21.15.142:0/2493795242 >> v2:172.21.15.142:3300/0 conn(0x7feb70132030 0x7feb70132400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 -- 172.21.15.142:0/2493795242 >> v1:172.21.15.142:6789/0 conn(0x7feb7009ae60 legacy=0x7feb7009b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 monclient: found mon.noname-b 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:37.525 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.515+0000 7feb67fff700 1 -- 172.21.15.142:0/2493795242 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb700a17b0 con 0x7feb7009ece0 2024-09-18T10:29:37.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb67fff700 1 -- 172.21.15.142:0/2493795242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 msgr2=0x7feb7009f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.526 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb67fff700 1 --2- 172.21.15.142:0/2493795242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb580099b0 tx=0x7feb58009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d7206d65c761d5c server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 1 -- 172.21.15.142:0/2493795242 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb5800ba70 con 0x7feb7009ece0 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: dump: 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.532 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 1 -- 172.21.15.142:0/2493795242 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5800bbd0 con 0x7feb7009ece0 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 5 monclient: authenticate success, global_id 234125 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 1 -- 172.21.15.142:0/2493795242 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58011420 con 0x7feb7009ece0 2024-09-18T10:29:37.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 4 set_mon_vals no callback set 2024-09-18T10:29:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb6f7fe700 10 monclient: dump: 2024-09-18T10:29:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.534 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 10 monclient: shutdown 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 -- 172.21.15.142:0/2493795242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 msgr2=0x7feb7009f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.535 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 --2- 172.21.15.142:0/2493795242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feb580099b0 tx=0x7feb58009250 comp rx=0 tx=0).stop 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 -- 172.21.15.142:0/2493795242 shutdown_connections 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 --2- 172.21.15.142:0/2493795242 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb7009ece0 0x7feb7009f0b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 --2- 172.21.15.142:0/2493795242 >> v2:172.21.15.142:3300/0 conn(0x7feb70132030 0x7feb70132400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 -- 172.21.15.142:0/2493795242 >> 172.21.15.142:0/2493795242 conn(0x7feb700a5700 msgr2=0x7feb700a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 -- 172.21.15.142:0/2493795242 shutdown_connections 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.519+0000 7feb751b2700 1 -- 172.21.15.142:0/2493795242 wait complete. 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient: build_initial_monmap 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient: monmap: 2024-09-18T10:29:37.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:37.537 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 Processor -- start 2024-09-18T10:29:37.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 -- start start 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient: init 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-a con 0x7feb7009ae60 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 0x7feb701310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-b con 0x7feb7009ece0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): picked mon.noname-c con 0x7feb70132030 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb700a4840 con 0x7feb7009ae60 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7011e4e0 con 0x7feb7009ece0 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7006f420 con 0x7feb70132030 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb751b2700 10 monclient(hunting): authenticate will time out at 2066.451341s 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38508/0 (socket says 172.21.15.142:38508) 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 1 -- 172.21.15.142:0/638725142 learned_addr learned my addr 172.21.15.142:0/638725142 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): get_auth_request con 0x7feb7009ae60 auth_method 0 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:37990/0 (socket says 172.21.15.142:37990) 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): get_auth_request con 0x7feb70132030 auth_method 0 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.541 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb749b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb67fff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 0x7feb701310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb67fff700 10 monclient(hunting): get_auth_request con 0x7feb7009ece0 auth_method 0 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb67fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.523+0000 7feb67fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 10 monclient(hunting): handle_auth_done global_id 224253 payload 824 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 -- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 msgr2=0x7feb7011bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 -- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 msgr2=0x7feb701310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.542 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 0x7feb701310e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 monclient: found mon.noname-a 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 -- 172.21.15.142:0/638725142 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580094f0 con 0x7feb7009ae60 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb67fff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 0x7feb701310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb749b1700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 -- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 msgr2=0x7feb7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:37.543 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6ffff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d890 tx=0x7feb60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60012800 con 0x7feb7009ae60 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: dump: 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.545 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb60012960 con 0x7feb7009ae60 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 5 monclient: authenticate success, global_id 224253 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 10 monclient: _renew_subs 2024-09-18T10:29:37.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7011e200 con 0x7feb7009ae60 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60012ce0 con 0x7feb7009ae60 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6cff9700 10 monclient: dump: 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:37.547 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 10 monclient: _renew_subs 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb6effd700 4 set_mon_vals no callback set 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.527+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7011d2a0 con 0x7feb7009ae60 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb751b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7feb60014640 con 0x7feb7009ae60 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb751b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb751b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7011e6a0 con 0x7feb7009ae60 2024-09-18T10:29:37.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb6cff9700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 0x7feb50037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7feb60049630 con 0x7feb7009ae60 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb67fff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 0x7feb50037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb67fff700 10 monclient: get_auth_request con 0x7feb50035ac0 auth_method 0 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.531+0000 7feb67fff700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 0x7feb50037f70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7feb580099b0 tx=0x7feb58016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.539+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7feb60012e40 con 0x7feb7009ae60 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.539+0000 7feb6cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:37.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.539+0000 7feb6cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:37.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.759+0000 7feb751b2700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-18T10:29:37.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.759+0000 7feb751b2700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-18T10:29:37.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.759+0000 7feb751b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:37.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.759+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7feb70072a80 con 0x7feb7009ae60 2024-09-18T10:29:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.763+0000 7feb6cff9700 1 -- 172.21.15.142:0/638725142 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7feb6001c6e0 con 0x7feb7009ae60 2024-09-18T10:29:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.763+0000 7feb6cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-18T10:29:37.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.763+0000 7feb6cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=841827859,config=3140514009,health=2145234240,kv=4261215878,logm=1143406895}) 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=7,logm=54} crc {auth=3184709575,config=3140514009,health=909117543,kv=4261215878,logm=1940211425}) 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=841827859,config=3140514009,health=2145234240,kv=4261215878,logm=1143406895}) 2024-09-18T10:29:37.770 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.763+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=3921356430,config=3140514009,health=2145234240,kv=3032515489,logm=906476237}) 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102307170}) 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2153859398}) 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102307170}) 2024-09-18T10:29:37.771 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733536352}) 2024-09-18T10:29:37.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 msgr2=0x7feb50037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 0x7feb50037f70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7feb580099b0 tx=0x7feb58016000 comp rx=0 tx=0).stop 2024-09-18T10:29:37.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 10 monclient: shutdown 2024-09-18T10:29:37.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 msgr2=0x7feb7006e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb6000d890 tx=0x7feb60008040 comp rx=0 tx=0).stop 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 shutdown_connections 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7feb50035ac0 0x7feb50037f70 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7feb70132030 0x7feb7011bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb7009ece0 0x7feb701310e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 --2- 172.21.15.142:0/638725142 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb7009ae60 0x7feb7006e9f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 >> 172.21.15.142:0/638725142 conn(0x7feb700a5700 msgr2=0x7feb700a4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 shutdown_connections 2024-09-18T10:29:37.773 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:37.767+0000 7feb751b2700 1 -- 172.21.15.142:0/638725142 wait complete. 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241313370}) 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298388280}) 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241313370}) 2024-09-18T10:29:37.774 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.767+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3455256207}) 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455306615}) 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101811280}) 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455306615}) 2024-09-18T10:29:37.775 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501591587}) 2024-09-18T10:29:37.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936468861}) 2024-09-18T10:29:37.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3370047500}) 2024-09-18T10:29:37.777 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936468861}) 2024-09-18T10:29:37.778 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.771+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1968628729}) 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129353104}) 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408381683}) 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129353104}) 2024-09-18T10:29:37.779 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022329781}) 2024-09-18T10:29:37.780 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.780 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763156970}) 2024-09-18T10:29:37.780 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=129646298}) 2024-09-18T10:29:37.781 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.781 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763156970}) 2024-09-18T10:29:37.781 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.775+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345486240}) 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067335027}) 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557660882}) 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067335027}) 2024-09-18T10:29:37.782 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3187026751}) 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943584211}) 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967627698}) 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943584211}) 2024-09-18T10:29:37.783 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2137987118}) 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243351606}) 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413307514}) 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243351606}) 2024-09-18T10:29:37.784 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1560527595}) 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444703348}) 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034159635}) 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444703348}) 2024-09-18T10:29:37.786 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.779+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511442641}) 2024-09-18T10:29:37.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-18T10:29:37.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198615896}) 2024-09-18T10:29:37.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178147881}) 2024-09-18T10:29:37.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.788 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198615896}) 2024-09-18T10:29:37.789 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.783+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004020734}) 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944050033}) 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1068027517}) 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=944050033}) 2024-09-18T10:29:37.790 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459235568}) 2024-09-18T10:29:37.791 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.791 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540441291}) 2024-09-18T10:29:37.791 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118690328}) 2024-09-18T10:29:37.791 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.791 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1540441291}) 2024-09-18T10:29:37.792 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3446636889}) 2024-09-18T10:29:37.792 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.792 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211182987}) 2024-09-18T10:29:37.792 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560362836}) 2024-09-18T10:29:37.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211182987}) 2024-09-18T10:29:37.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=694545066}) 2024-09-18T10:29:37.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.787+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.793 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984863325}) 2024-09-18T10:29:37.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3283110770}) 2024-09-18T10:29:37.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984863325}) 2024-09-18T10:29:37.794 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107780543}) 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785568769}) 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490440843}) 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785568769}) 2024-09-18T10:29:37.795 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=123905212}) 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007599288}) 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4238603120}) 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3007599288}) 2024-09-18T10:29:37.796 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167693094}) 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529475060}) 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955768084}) 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529475060}) 2024-09-18T10:29:37.797 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.791+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876114836}) 2024-09-18T10:29:37.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32532148}) 2024-09-18T10:29:37.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119237229}) 2024-09-18T10:29:37.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.798 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32532148}) 2024-09-18T10:29:37.799 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.795+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428073156}) 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327771490}) 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184202236}) 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327771490}) 2024-09-18T10:29:37.807 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916676816}) 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040311632}) 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751951634}) 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040311632}) 2024-09-18T10:29:37.808 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911406708}) 2024-09-18T10:29:37.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289686286}) 2024-09-18T10:29:37.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156261437}) 2024-09-18T10:29:37.809 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289686286}) 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.803+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3193046173}) 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868774338}) 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509464111}) 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.810 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868774338}) 2024-09-18T10:29:37.811 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095082567}) 2024-09-18T10:29:37.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4215867989}) 2024-09-18T10:29:37.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769397376}) 2024-09-18T10:29:37.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.812 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4215867989}) 2024-09-18T10:29:37.813 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708103270}) 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029459929}) 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=499166957}) 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029459929}) 2024-09-18T10:29:37.814 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.807+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780395289}) 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001234110}) 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=593361463}) 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001234110}) 2024-09-18T10:29:37.815 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825876579}) 2024-09-18T10:29:37.816 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299077660}) 2024-09-18T10:29:37.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103409826}) 2024-09-18T10:29:37.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=299077660}) 2024-09-18T10:29:37.817 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981210957}) 2024-09-18T10:29:37.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295903255}) 2024-09-18T10:29:37.819 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497023522}) 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295903255}) 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.811+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168863575}) 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657198759}) 2024-09-18T10:29:37.820 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=562711146}) 2024-09-18T10:29:37.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657198759}) 2024-09-18T10:29:37.821 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=356161701}) 2024-09-18T10:29:37.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.822 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630469307}) 2024-09-18T10:29:37.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3743201130}) 2024-09-18T10:29:37.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630469307}) 2024-09-18T10:29:37.823 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.815+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259933114}) 2024-09-18T10:29:37.828 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022406421}) 2024-09-18T10:29:37.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188128727}) 2024-09-18T10:29:37.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022406421}) 2024-09-18T10:29:37.829 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.823+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414423363}) 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997852885}) 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2947505668}) 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997852885}) 2024-09-18T10:29:37.830 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573914547}) 2024-09-18T10:29:37.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872454414}) 2024-09-18T10:29:37.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109959299}) 2024-09-18T10:29:37.831 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872454414}) 2024-09-18T10:29:37.832 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2295433346}) 2024-09-18T10:29:37.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020600840}) 2024-09-18T10:29:37.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=182240069}) 2024-09-18T10:29:37.833 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020600840}) 2024-09-18T10:29:37.834 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.827+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=621296656}) 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443302445}) 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288861184}) 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443302445}) 2024-09-18T10:29:37.835 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057708477}) 2024-09-18T10:29:37.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899901305}) 2024-09-18T10:29:37.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248827258}) 2024-09-18T10:29:37.836 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899901305}) 2024-09-18T10:29:37.837 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.831+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783529687}) 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911561762}) 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768734399}) 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911561762}) 2024-09-18T10:29:37.838 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1334058455}) 2024-09-18T10:29:37.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2802235160}) 2024-09-18T10:29:37.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843296590}) 2024-09-18T10:29:37.839 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2802235160}) 2024-09-18T10:29:37.840 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932909228}) 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542411786}) 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607835838}) 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542411786}) 2024-09-18T10:29:37.841 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.835+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=816329713}) 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261993057}) 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2897971558}) 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261993057}) 2024-09-18T10:29:37.842 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414413828}) 2024-09-18T10:29:37.843 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099962851}) 2024-09-18T10:29:37.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282358335}) 2024-09-18T10:29:37.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099962851}) 2024-09-18T10:29:37.844 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499568138}) 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235483859}) 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174005591}) 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4235483859}) 2024-09-18T10:29:37.845 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.839+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522288231}) 2024-09-18T10:29:37.846 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785421859}) 2024-09-18T10:29:37.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250996372}) 2024-09-18T10:29:37.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785421859}) 2024-09-18T10:29:37.847 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905497273}) 2024-09-18T10:29:37.848 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891430235}) 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=208572095}) 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891430235}) 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3391124707}) 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572180398}) 2024-09-18T10:29:37.849 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770077670}) 2024-09-18T10:29:37.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=572180398}) 2024-09-18T10:29:37.850 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.843+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656282465}) 2024-09-18T10:29:37.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143103909}) 2024-09-18T10:29:37.851 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115145703}) 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143103909}) 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016310837}) 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541431003}) 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287109383}) 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.852 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541431003}) 2024-09-18T10:29:37.853 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616564338}) 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813137974}) 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=100513565}) 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813137974}) 2024-09-18T10:29:37.854 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194859987}) 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518234173}) 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707085721}) 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518234173}) 2024-09-18T10:29:37.855 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.847+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143716770}) 2024-09-18T10:29:37.856 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810929075}) 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442072374}) 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810929075}) 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247654184}) 2024-09-18T10:29:37.857 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3313801392}) 2024-09-18T10:29:37.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541691740}) 2024-09-18T10:29:37.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3313801392}) 2024-09-18T10:29:37.858 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939603667}) 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872179541}) 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129564548}) 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872179541}) 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.851+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501537541}) 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.860 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233724943}) 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949626194}) 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4233724943}) 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534216665}) 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486878444}) 2024-09-18T10:29:37.861 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2443680143}) 2024-09-18T10:29:37.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486878444}) 2024-09-18T10:29:37.862 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3082830378}) 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266279690}) 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861212744}) 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266279690}) 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.855+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371096736}) 2024-09-18T10:29:37.863 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604446022}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=665475302}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604446022}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335440988}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000325861}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827509181}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000325861}) 2024-09-18T10:29:37.864 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553077074}) 2024-09-18T10:29:37.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168932988}) 2024-09-18T10:29:37.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481899958}) 2024-09-18T10:29:37.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.865 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168932988}) 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205417328}) 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565134337}) 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046096248}) 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565134337}) 2024-09-18T10:29:37.866 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.859+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477666809}) 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151796564}) 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469075072}) 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151796564}) 2024-09-18T10:29:37.867 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2432587709}) 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337774422}) 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669081136}) 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337774422}) 2024-09-18T10:29:37.868 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.863+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303834218}) 2024-09-18T10:29:37.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952500767}) 2024-09-18T10:29:37.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640853197}) 2024-09-18T10:29:37.983 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952500767}) 2024-09-18T10:29:37.984 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1761962217}) 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938942584}) 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3654643590}) 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938942584}) 2024-09-18T10:29:37.986 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.979+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979333769}) 2024-09-18T10:29:37.987 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.987 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549264210}) 2024-09-18T10:29:37.987 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349458082}) 2024-09-18T10:29:37.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549264210}) 2024-09-18T10:29:37.988 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015558540}) 2024-09-18T10:29:37.989 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107490100}) 2024-09-18T10:29:37.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.983+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2141520171}) 2024-09-18T10:29:37.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107490100}) 2024-09-18T10:29:37.990 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318685125}) 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4046742369}) 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138879541}) 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4046742369}) 2024-09-18T10:29:37.992 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476418676}) 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.987+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255997520}) 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256606103}) 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255997520}) 2024-09-18T10:29:37.994 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448284569}) 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786244565}) 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1060798733}) 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786244565}) 2024-09-18T10:29:37.996 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360452208}) 2024-09-18T10:29:37.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.997 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170015207}) 2024-09-18T10:29:37.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627454952}) 2024-09-18T10:29:37.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170015207}) 2024-09-18T10:29:37.998 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.991+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231808551}) 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849556493}) 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441586084}) 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849556493}) 2024-09-18T10:29:37.999 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=672645256}) 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589159316}) 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3626797179}) 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589159316}) 2024-09-18T10:29:38.001 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.995+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2841603124}) 2024-09-18T10:29:38.004 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539429982}) 2024-09-18T10:29:38.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769199625}) 2024-09-18T10:29:38.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.005 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539429982}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396250076}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686295008}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2695727337}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686295008}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2177259021}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065344854}) 2024-09-18T10:29:38.006 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664320129}) 2024-09-18T10:29:38.007 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.007 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065344854}) 2024-09-18T10:29:38.007 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:37.999+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704296330}) 2024-09-18T10:29:38.008 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.008 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290991754}) 2024-09-18T10:29:38.008 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1694196715}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290991754}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977390670}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101466117}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645178357}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101466117}) 2024-09-18T10:29:38.009 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842910213}) 2024-09-18T10:29:38.011 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815308777}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089187077}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815308777}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.003+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1883104653}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546385241}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180476998}) 2024-09-18T10:29:38.012 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546385241}) 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451564710}) 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557829334}) 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461645265}) 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557829334}) 2024-09-18T10:29:38.013 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663229338}) 2024-09-18T10:29:38.014 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.014 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513233377}) 2024-09-18T10:29:38.014 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494909828}) 2024-09-18T10:29:38.014 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.015 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513233377}) 2024-09-18T10:29:38.015 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.007+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342272742}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=336772224}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956440364}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=336772224}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341436909}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148108345}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084961121}) 2024-09-18T10:29:38.017 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.018 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148108345}) 2024-09-18T10:29:38.018 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.011+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=436169626}) 2024-09-18T10:29:38.019 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.019 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713424336}) 2024-09-18T10:29:38.019 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581666654}) 2024-09-18T10:29:38.019 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.020 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713424336}) 2024-09-18T10:29:38.020 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054517496}) 2024-09-18T10:29:38.021 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.021 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551608552}) 2024-09-18T10:29:38.022 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366752504}) 2024-09-18T10:29:38.022 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.022 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551608552}) 2024-09-18T10:29:38.022 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.015+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758809320}) 2024-09-18T10:29:38.024 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.024 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948432331}) 2024-09-18T10:29:38.024 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1970848535}) 2024-09-18T10:29:38.024 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.024 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948432331}) 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862742513}) 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237894956}) 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915732755}) 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237894956}) 2024-09-18T10:29:38.025 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412365965}) 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844642817}) 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3317199076}) 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844642817}) 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.019+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016107540}) 2024-09-18T10:29:38.028 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813520589}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3982136947}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813520589}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2175615082}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177431946}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081788655}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177431946}) 2024-09-18T10:29:38.029 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733229956}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292950460}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670014292}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292950460}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.023+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2592586101}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570731302}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285554387}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570731302}) 2024-09-18T10:29:38.031 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115767493}) 2024-09-18T10:29:38.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334293017}) 2024-09-18T10:29:38.032 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382962875}) 2024-09-18T10:29:38.033 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.033 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334293017}) 2024-09-18T10:29:38.033 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319885555}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439274699}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244219274}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439274699}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.027+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801377685}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3930681708}) 2024-09-18T10:29:38.035 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1114548677}) 2024-09-18T10:29:38.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3930681708}) 2024-09-18T10:29:38.036 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169646295}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258998446}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386892644}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258998446}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.031+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766690588}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297566071}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161051626}) 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.038 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3297566071}) 2024-09-18T10:29:38.039 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277539604}) 2024-09-18T10:29:38.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.040 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522383685}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000582334}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3522383685}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792032886}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210870540}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465099251}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1210870540}) 2024-09-18T10:29:38.041 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.035+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=804489171}) 2024-09-18T10:29:38.042 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.042 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031817963}) 2024-09-18T10:29:38.042 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806890197}) 2024-09-18T10:29:38.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031817963}) 2024-09-18T10:29:38.043 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874158561}) 2024-09-18T10:29:38.044 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.045 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787498542}) 2024-09-18T10:29:38.045 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=233141665}) 2024-09-18T10:29:38.045 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787498542}) 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932800241}) 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878533021}) 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079559599}) 2024-09-18T10:29:38.046 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.047 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878533021}) 2024-09-18T10:29:38.047 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988999841}) 2024-09-18T10:29:38.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.050 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579607757}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512597704}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579607757}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.039+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618999606}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003869995}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3304183795}) 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.051 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4003869995}) 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878190424}) 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693404677}) 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4205065962}) 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693404677}) 2024-09-18T10:29:38.052 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681608035}) 2024-09-18T10:29:38.053 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.053 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=718523264,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=568253033}) 2024-09-18T10:29:38.053 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=84,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=1289370434,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=2396623012}) 2024-09-18T10:29:38.053 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.053 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=718523264,mdsmap=1531151339,mgr=3316658068,mgr_command_descs=47663135,mgr_metadata=2738435100,mgrstat=568253033}) 2024-09-18T10:29:38.054 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.043+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=3} crc {logm=157229911,mdsmap=1531151339,mgr=2679965908,mgr_command_descs=47663135,mgrstat=105665762}) 2024-09-18T10:29:38.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=6,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=941861239,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=2526516998}) 2024-09-18T10:29:38.055 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=6,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=52} crc {mgrstat=941861239,mon_config_key=620546692,monmap=4028308432,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=3823785925,osdmap=2059883535}) 2024-09-18T10:29:38.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-18T10:29:38.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=6,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=53} crc {mgrstat=941861239,mon_config_key=620546692,monmap=3505902565,osd_metadata=1440571423,osd_pg_creating=69081760,osd_snap=454132730,osdmap=2526516998}) 2024-09-18T10:29:38.056 INFO:tasks.ceph.mon.a.smithi142.stderr:2024-09-18T10:29:38.051+0000 7fd11d786700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=6,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=52} crc {mgrstat=941861239,mon_config_key=620546692,monmap=3505902565,osd_metadata=2166527243,osd_pg_creating=69081760,osd_snap=454132730,osdmap=1360768017}) 2024-09-18T10:29:38.787 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-18T10:29:38.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-18T10:29:38.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: get_monmap_and_config 2024-09-18T10:29:38.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: build_initial_monmap 2024-09-18T10:29:38.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: monmap: 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:38.897383+0000 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:38.897383+0000 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:38.900 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:38.901 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 Processor -- start 2024-09-18T10:29:38.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 -- start start 2024-09-18T10:29:38.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: init 2024-09-18T10:29:38.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:38.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-b con 0x7f4b2009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b200967a0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c4f0 0x7f4b201375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-c con 0x7f4b2009c4f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b20095880 con 0x7f4b200967a0 2024-09-18T10:29:38.904 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2009aeb0 con 0x7f4b2009c4f0 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2006f6d0 con 0x7f4b2009aae0 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b25679700 10 monclient(hunting): authenticate will time out at 2067.822888s 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b177fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4b200967a0 0x7f4b2009bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47202/0 (socket says 172.21.15.142:47202) 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38004/0 (socket says 172.21.15.142:38004) 2024-09-18T10:29:38.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b177fe700 1 -- 172.21.15.142:0/1901415021 learned_addr learned my addr 172.21.15.142:0/1901415021 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request con 0x7f4b2009aae0 auth_method 0 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 1 --2- 172.21.15.142:0/1901415021 >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c4f0 0x7f4b201375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): get_auth_request con 0x7f4b2009c4f0 auth_method 0 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:38.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:38.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_done global_id 234131 payload 274 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1901415021 >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c4f0 msgr2=0x7f4b201375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/1901415021 >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c4f0 0x7f4b201375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1901415021 >> v1:172.21.15.142:6789/0 conn(0x7f4b200967a0 legacy=0x7f4b2009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 monclient: found mon.noname-b 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.895+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1901415021 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200955a0 con 0x7f4b2009aae0 2024-09-18T10:29:38.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1901415021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 msgr2=0x7f4b20096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/1901415021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b100098e0 tx=0x7f4b100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c9227682ca068f server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/1901415021 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000b850 con 0x7f4b2009aae0 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: dump: 2024-09-18T10:29:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:38.918 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/1901415021 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1000b9b0 con 0x7f4b2009aae0 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 5 monclient: authenticate success, global_id 234131 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 1 -- 172.21.15.142:0/1901415021 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000bd50 con 0x7f4b2009aae0 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 4 set_mon_vals no callback set 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b1f7fe700 10 monclient: dump: 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:38.919 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 10 monclient: shutdown 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 -- 172.21.15.142:0/1901415021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 msgr2=0x7f4b20096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 --2- 172.21.15.142:0/1901415021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4b100098e0 tx=0x7f4b100091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 -- 172.21.15.142:0/1901415021 shutdown_connections 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 --2- 172.21.15.142:0/1901415021 >> v2:172.21.15.150:3300/0 conn(0x7f4b2009c4f0 0x7f4b201375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 --2- 172.21.15.142:0/1901415021 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009aae0 0x7f4b20096260 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 -- 172.21.15.142:0/1901415021 >> 172.21.15.142:0/1901415021 conn(0x7f4b200a1fe0 msgr2=0x7f4b20070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 -- 172.21.15.142:0/1901415021 shutdown_connections 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.899+0000 7f4b25679700 1 -- 172.21.15.142:0/1901415021 wait complete. 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient: build_initial_monmap 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient: monmap: 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:38.921 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 Processor -- start 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 -- start start 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient: init 2024-09-18T10:29:38.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-a con 0x7f4b200967a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009aae0 0x7f4b20141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-b con 0x7f4b2009aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 0x7f4b201420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): picked mon.noname-c con 0x7f4b2009c4f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20091cd0 con 0x7f4b200967a0 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2006f900 con 0x7f4b2009aae0 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20091650 con 0x7f4b2009c4f0 2024-09-18T10:29:38.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b25679700 10 monclient(hunting): authenticate will time out at 2067.830758s 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 0x7f4b201420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38510/0 (socket says 172.21.15.142:38510) 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 0x7f4b201420f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38014/0 (socket says 172.21.15.142:38014) 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1747140186 learned_addr learned my addr 172.21.15.142:0/1747140186 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request con 0x7f4b200967a0 auth_method 0 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:38.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): get_auth_request con 0x7f4b2009c4f0 auth_method 0 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009aae0 0x7f4b20141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:38.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b24e78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): get_auth_request con 0x7f4b2009aae0 auth_method 0 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient(hunting): handle_auth_done global_id 224256 payload 824 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 msgr2=0x7f4b201420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 0x7f4b201420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009aae0 msgr2=0x7f4b20141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:38.926 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009aae0 0x7f4b20141bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 monclient: found mon.noname-a 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.903+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1747140186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10009420 con 0x7f4b200967a0 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1ffff700 1 -- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 msgr2=0x7f4b20141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1ffff700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4b100093f0 tx=0x7f4b10016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10011b40 con 0x7f4b200967a0 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: dump: 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:38.927 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 5 monclient: authenticate success, global_id 224256 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b10020070 con 0x7f4b200967a0 2024-09-18T10:29:38.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 10 monclient: _renew_subs 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2009a2f0 con 0x7f4b200967a0 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1001b450 con 0x7f4b200967a0 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:38.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1cff9700 10 monclient: dump: 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:38.930 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 10 monclient: _renew_subs 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b20091370 con 0x7f4b200967a0 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.907+0000 7f4b1effd700 4 set_mon_vals no callback set 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b25679700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b25679700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b25679700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b20072a80 con 0x7f4b200967a0 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f4b10022030 con 0x7f4b200967a0 2024-09-18T10:29:38.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b1cff9700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 0x7f4b00037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f4b1004aac0 con 0x7f4b200967a0 2024-09-18T10:29:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b177fe700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 0x7f4b00037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:38.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b177fe700 10 monclient: get_auth_request con 0x7f4b00035ac0 auth_method 0 2024-09-18T10:29:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.911+0000 7f4b177fe700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 0x7f4b00037f70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007880 tx=0x7f4b08007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.915+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f4b1001ae00 con 0x7f4b200967a0 2024-09-18T10:29:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.915+0000 7f4b1cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:38.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:38.915+0000 7f4b1cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:39.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.139+0000 7f4b25679700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-18T10:29:39.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.139+0000 7f4b25679700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:39.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.139+0000 7f4b25679700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:39.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.139+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b20072a80 con 0x7f4b200967a0 2024-09-18T10:29:39.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b1cff9700 1 -- 172.21.15.142:0/1747140186 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7f4b1001a9e0 con 0x7f4b200967a0 2024-09-18T10:29:39.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b1cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-18T10:29:39.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b1cff9700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:39.147 INFO:teuthology.orchestra.run.smithi142.stdout:{"election_epoch":196,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","modified":"2024-09-18T10:08:19.331585Z","created":"2024-09-18T10:08:19.331585Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:3300","nonce":0},{"type":"v1","addr":"172.21.15.150:6789","nonce":0}]},"addr":"172.21.15.150:6789/0","public_addr":"172.21.15.150:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3301","nonce":0},{"type":"v1","addr":"172.21.15.142:6790","nonce":0}]},"addr":"172.21.15.142:6790/0","public_addr":"172.21.15.142:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 msgr2=0x7f4b00037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 0x7f4b00037f70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007880 tx=0x7f4b08007f90 comp rx=0 tx=0).stop 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 10 monclient: shutdown 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 msgr2=0x7f4b20141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4b100093f0 tx=0x7f4b10016000 comp rx=0 tx=0).stop 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 shutdown_connections 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f4b00035ac0 0x7f4b00037f70 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4b2009c4f0 0x7f4b201420f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b2009aae0 0x7f4b20141bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 --2- 172.21.15.142:0/1747140186 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b200967a0 0x7f4b20141670 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 >> 172.21.15.142:0/1747140186 conn(0x7f4b200a1fe0 msgr2=0x7f4b2009fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 shutdown_connections 2024-09-18T10:29:39.150 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.143+0000 7f4b25679700 1 -- 172.21.15.142:0/1747140186 wait complete. 2024-09-18T10:29:39.164 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-18T10:29:39.165 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T10:29:39.173 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T10:29:39.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:29:39.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: get_monmap_and_config 2024-09-18T10:29:39.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: monmap: 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:39.277463+0000 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:39.277463+0000 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:39.280 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:39.281 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:39.281 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:39.281 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 Processor -- start 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 -- start start 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: init 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-18T10:29:39.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6cd809bd80 0x7f6cd809a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-c con 0x7f6cd809bd80 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cd809a740 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 0x7f6cd8139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd81376a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8068ff0 con 0x7f6cd809a740 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd8099b00 con 0x7f6cd809bd80 2024-09-18T10:29:39.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd809e5f0 con 0x7f6cd81376a0 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cde1f0700 10 monclient(hunting): authenticate will time out at 2068.203343s 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd6ffd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f6cd809a740 0x7f6cd809c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47206/0 (socket says 172.21.15.142:47206) 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 1 --2- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 0x7f6cd8139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/4149109757 learned_addr learned my addr 172.21.15.142:0/4149109757 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): get_auth_request con 0x7f6cd81376a0 auth_method 0 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd77fe700 1 --2- 172.21.15.142:0/4149109757 >> v2:172.21.15.150:3300/0 conn(0x7f6cd809bd80 0x7f6cd809a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd77fe700 10 monclient(hunting): get_auth_request con 0x7f6cd809bd80 auth_method 0 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.275+0000 7f6cd77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 10 monclient(hunting): handle_auth_done global_id 234137 payload 274 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 -- 172.21.15.142:0/4149109757 >> v2:172.21.15.150:3300/0 conn(0x7f6cd809bd80 msgr2=0x7f6cd809a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 --2- 172.21.15.142:0/4149109757 >> v2:172.21.15.150:3300/0 conn(0x7f6cd809bd80 0x7f6cd809a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 -- 172.21.15.142:0/4149109757 >> v1:172.21.15.142:6789/0 conn(0x7f6cd809a740 legacy=0x7f6cd809c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 monclient: found mon.noname-b 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 -- 172.21.15.142:0/4149109757 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd809ea80 con 0x7f6cd81376a0 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 -- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 msgr2=0x7f6cd8139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd7fff700 1 --2- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 0x7f6cd8139a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009e60 tx=0x7f6cd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949846c6603e6cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 1 -- 172.21.15.142:0/4149109757 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000e070 con 0x7f6cd81376a0 2024-09-18T10:29:39.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: dump: 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.299 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 5 monclient: authenticate success, global_id 234137 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 1 -- 172.21.15.142:0/4149109757 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000bd70 con 0x7f6cd81376a0 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 4 set_mon_vals no callback set 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 1 -- 172.21.15.142:0/4149109757 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0012680 con 0x7f6cd81376a0 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:39.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cd67fc700 10 monclient: dump: 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 10 monclient: shutdown 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 1 -- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 msgr2=0x7f6cd8139a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 0x7f6cd8139a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009e60 tx=0x7f6cd0009780 comp rx=0 tx=0).stop 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 1 -- 172.21.15.142:0/4149109757 shutdown_connections 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/4149109757 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd81376a0 0x7f6cd8139a80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.279+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/4149109757 >> v2:172.21.15.150:3300/0 conn(0x7f6cd809bd80 0x7f6cd809a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- 172.21.15.142:0/4149109757 >> 172.21.15.142:0/4149109757 conn(0x7f6cd80a24f0 msgr2=0x7f6cd80a28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- 172.21.15.142:0/4149109757 shutdown_connections 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- 172.21.15.142:0/4149109757 wait complete. 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient: monmap: 2024-09-18T10:29:39.302 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 Processor -- start 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- start start 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient: init 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd809a740 0x7f6cd812f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-b con 0x7f6cd809a740 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-c con 0x7f6cd809bd80 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd81376a0 0x7f6cd8096900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): picked mon.noname-a con 0x7f6cd81376a0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8098c40 con 0x7f6cd81376a0 2024-09-18T10:29:39.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80976f0 con 0x7f6cd809a740 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80988d0 con 0x7f6cd809bd80 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cde1f0700 10 monclient(hunting): authenticate will time out at 2068.211551s 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38032/0 (socket says 172.21.15.142:38032) 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/3207750140 learned_addr learned my addr 172.21.15.142:0/3207750140 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:39.305 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): get_auth_request con 0x7f6cd809bd80 auth_method 0 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd7fff700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd81376a0 0x7f6cd8096900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd77fe700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd809a740 0x7f6cd812f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd7fff700 10 monclient(hunting): get_auth_request con 0x7f6cd81376a0 auth_method 0 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd77fe700 10 monclient(hunting): get_auth_request con 0x7f6cd809a740 auth_method 0 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.283+0000 7f6cd6ffd700 10 monclient(hunting): handle_auth_done global_id 234140 payload 824 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd809a740 msgr2=0x7f6cd812f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd809a740 0x7f6cd812f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd81376a0 msgr2=0x7f6cd8096900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd81376a0 0x7f6cd8096900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.307 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 monclient: found mon.noname-c 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/3207750140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd0009a20 con 0x7f6cd809bd80 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 -- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 msgr2=0x7f6cd80963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd6ffd700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800b5e0 tx=0x7f6cc8009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8011840 con 0x7f6cd809bd80 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: dump: 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.308 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 5 monclient: authenticate success, global_id 234140 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 10 monclient: _renew_subs 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc80119c0 con 0x7f6cd809bd80 2024-09-18T10:29:39.309 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8130240 con 0x7f6cd809bd80 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8011d80 con 0x7f6cd809bd80 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 10 monclient: dump: 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.310 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cd5ffb700 4 set_mon_vals no callback set 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 10 monclient: _renew_subs 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8097970 con 0x7f6cd809bd80 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.287+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6cc8010590 con 0x7f6cd809bd80 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cde1f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.311 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cde1f0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cde1f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cbf7fe700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 0x7f6cc4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8098a90 con 0x7f6cd809bd80 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f6cc8049150 con 0x7f6cd809bd80 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cd77fe700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 0x7f6cc4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cd77fe700 10 monclient: get_auth_request con 0x7f6cc4035ac0 auth_method 0 2024-09-18T10:29:39.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.291+0000 7f6cd77fe700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 0x7f6cc4037f70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009d40 tx=0x7f6ccc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.295+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f6cc800f4c0 con 0x7f6cd809bd80 2024-09-18T10:29:39.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.295+0000 7f6cbf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.295+0000 7f6cbf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:39.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.459+0000 7f6cde1f0700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:39.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.459+0000 7f6cde1f0700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:39.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.459+0000 7f6cde1f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.459+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cd80696a0 con 0x7f6cd809bd80 2024-09-18T10:29:39.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.463+0000 7f6cbf7fe700 1 -- 172.21.15.142:0/3207750140 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+21200 (secure 0 0 0) 0x7f6cc8015070 con 0x7f6cd809bd80 2024-09-18T10:29:39.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.463+0000 7f6cbf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:39.466 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.463+0000 7f6cbf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:39.467 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:39.467 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":264,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:18:28.413385+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:13:27.979439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":182,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T10:09:00.830395+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-18T10:13:58.026157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-18T10:13:58.878550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"912b7d9d-08f4-4308-9252-03c9211c6cd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.358800+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.344702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.5","mappings":[{"from":0,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 msgr2=0x7f6cc4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 0x7f6cc4037f70 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009d40 tx=0x7f6ccc008040 comp rx=0 tx=0).stop 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 10 monclient: shutdown 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 msgr2=0x7f6cd80963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800b5e0 tx=0x7f6cc8009120 comp rx=0 tx=0).stop 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 shutdown_connections 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f6cc4035ac0 0x7f6cc4037f70 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6cd81376a0 0x7f6cd8096900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.470 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6cd809bd80 0x7f6cd80963c0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6cc800b5e0 tx=0x7f6cc8009120 comp rx=0 tx=0).stop 2024-09-18T10:29:39.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 --2- 172.21.15.142:0/3207750140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6cd809a740 0x7f6cd812f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 >> 172.21.15.142:0/3207750140 conn(0x7f6cd80a24f0 msgr2=0x7f6cd80a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 shutdown_connections 2024-09-18T10:29:39.471 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.467+0000 7f6cde1f0700 1 -- 172.21.15.142:0/3207750140 wait complete. 2024-09-18T10:29:39.487 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T10:29:39.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: get_monmap_and_config 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: monmap: 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:39.593004+0000 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:39.593004+0000 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.595 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:39.596 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.597 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 Processor -- start 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 -- start start 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: init 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-b con 0x7f24a809d920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24a80953c0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24a80957d0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a809b0b0 con 0x7f24a80953c0 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a809a7e0 con 0x7f24a80957d0 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a806fba0 con 0x7f24a809d920 2024-09-18T10:29:39.598 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24ad645700 10 monclient(hunting): authenticate will time out at 2068.518499s 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f24a80953c0 0x7f24a8096cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47218/0 (socket says 172.21.15.142:47218) 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a77fe700 1 -- 172.21.15.142:0/3573632907 learned_addr learned my addr 172.21.15.142:0/3573632907 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 1 --2- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): get_auth_request con 0x7f24a809d920 auth_method 0 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient(hunting): handle_auth_done global_id 234143 payload 274 2024-09-18T10:29:39.599 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 1 -- 172.21.15.142:0/3573632907 >> v1:172.21.15.150:6789/0 conn(0x7f24a80957d0 legacy=0x7f24a80973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 1 -- 172.21.15.142:0/3573632907 >> v1:172.21.15.142:6789/0 conn(0x7f24a80953c0 legacy=0x7f24a8096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 1 monclient: found mon.noname-b 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.600 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.591+0000 7f24a7fff700 1 -- 172.21.15.142:0/3573632907 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a809aaf0 con 0x7f24a809d920 2024-09-18T10:29:39.601 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a7fff700 1 -- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 msgr2=0x7f24a806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a7fff700 1 --2- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a806f660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009910 tx=0x7f24a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8285bf55ab9460a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 1 -- 172.21.15.142:0/3573632907 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000ba10 con 0x7f24a809d920 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: dump: 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.602 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 5 monclient: authenticate success, global_id 234143 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 1 -- 172.21.15.142:0/3573632907 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a000bb70 con 0x7f24a809d920 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 4 set_mon_vals no callback set 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 1 -- 172.21.15.142:0/3573632907 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0011420 con 0x7f24a809d920 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.603 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24a6ffd700 10 monclient: dump: 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 10 monclient: shutdown 2024-09-18T10:29:39.604 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 -- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 msgr2=0x7f24a806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 --2- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a806f660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009910 tx=0x7f24a00091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 -- 172.21.15.142:0/3573632907 shutdown_connections 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 --2- 172.21.15.142:0/3573632907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a806f660 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 -- 172.21.15.142:0/3573632907 >> 172.21.15.142:0/3573632907 conn(0x7f24a80a1fe0 msgr2=0x7f24a8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 -- 172.21.15.142:0/3573632907 shutdown_connections 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 1 -- 172.21.15.142:0/3573632907 wait complete. 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.595+0000 7f24ad645700 10 monclient: monmap: 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:39.605 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 Processor -- start 2024-09-18T10:29:39.606 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 -- start start 2024-09-18T10:29:39.607 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient: init 2024-09-18T10:29:39.610 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-a con 0x7f24a80953c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24a80957d0 0x7f24a808a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-b con 0x7f24a80957d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): picked mon.noname-c con 0x7f24a809d920 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a808e0e0 con 0x7f24a80953c0 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a808d510 con 0x7f24a80957d0 2024-09-18T10:29:39.611 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a808d230 con 0x7f24a809d920 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ad645700 10 monclient(hunting): authenticate will time out at 2068.524858s 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38530/0 (socket says 172.21.15.142:38530) 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38052/0 (socket says 172.21.15.142:38052) 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 1 -- 172.21.15.142:0/23242340 learned_addr learned my addr 172.21.15.142:0/23242340 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): get_auth_request con 0x7f24a80953c0 auth_method 0 2024-09-18T10:29:39.612 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): get_auth_request con 0x7f24a809d920 auth_method 0 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24a80957d0 0x7f24a808a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.613 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): get_auth_request con 0x7f24a80957d0 auth_method 0 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient(hunting): handle_auth_done global_id 234146 payload 824 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 -- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24a80957d0 msgr2=0x7f24a808a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24a80957d0 0x7f24a808a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.614 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 -- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 msgr2=0x7f24a8089b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 monclient: found mon.noname-c 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.616 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 -- 172.21.15.142:0/23242340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a0009420 con 0x7f24a809d920 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24a7fff700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 -- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 msgr2=0x7f24a808a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.599+0000 7f24ace44700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f249800c4c0 tx=0x7f2498007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249800ce30 con 0x7f24a809d920 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: dump: 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.617 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 5 monclient: authenticate success, global_id 234146 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24980117a0 con 0x7f24a809d920 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _renew_subs 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a808cfb0 con 0x7f24a809d920 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498011b60 con 0x7f24a809d920 2024-09-18T10:29:39.618 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 10 monclient: dump: 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24a67fc700 4 set_mon_vals no callback set 2024-09-18T10:29:39.619 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _renew_subs 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a808bff0 con 0x7f24a809d920 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a808b4b0 con 0x7f24a809d920 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f249801c460 con 0x7f24a809d920 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 0x7f2488037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f2498048b90 con 0x7f24a809d920 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24a7fff700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 0x7f2488037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.603+0000 7f24a7fff700 10 monclient: get_auth_request con 0x7f2488035ac0 auth_method 0 2024-09-18T10:29:39.620 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.607+0000 7f24a7fff700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 0x7f2488037f70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f24a000bb40 tx=0x7f24a0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.611+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f2498012780 con 0x7f24a809d920 2024-09-18T10:29:39.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.611+0000 7f249ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.621 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.611+0000 7f249ffff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:39.777 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.771+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24a8072a80 con 0x7f2488035ac0 2024-09-18T10:29:39.786 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f249ffff700 1 -- 172.21.15.142:0/23242340 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97482 (secure 0 0 0) 0x7f24a8072a80 con 0x7f2488035ac0 2024-09-18T10:29:39.788 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:39.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 msgr2=0x7f2488037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 0x7f2488037f70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f24a000bb40 tx=0x7f24a0016000 comp rx=0 tx=0).stop 2024-09-18T10:29:39.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 10 monclient: shutdown 2024-09-18T10:29:39.789 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 msgr2=0x7f24a808a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f249800c4c0 tx=0x7f2498007ea0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 shutdown_connections 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f2488035ac0 0x7f2488037f70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f24a809d920 0x7f24a808a5a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24a80957d0 0x7f24a808a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 --2- 172.21.15.142:0/23242340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24a80953c0 0x7f24a8089b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 >> 172.21.15.142:0/23242340 conn(0x7f24a80a1fe0 msgr2=0x7f24a809f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.790 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 shutdown_connections 2024-09-18T10:29:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.783+0000 7f24ad645700 1 -- 172.21.15.142:0/23242340 wait complete. 2024-09-18T10:29:39.791 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:29:39.802 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":876,"stamp":"2024-09-18T10:29:39.131981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820536,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410504,"statfs":{"total":579820584960,"available":577956356096,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003770"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":174,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733987+0000","last_change":"2024-09-18T10:15:10.262207+0000","last_active":"2024-09-18T10:18:23.733987+0000","last_peered":"2024-09-18T10:18:23.733987+0000","last_clean":"2024-09-18T10:18:23.733987+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:18:23.733987+0000","last_undegraded":"2024-09-18T10:18:23.733987+0000","last_fullsized":"2024-09-18T10:18:23.733987+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:41:30.693246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735070+0000","last_change":"2024-09-18T10:15:10.259663+0000","last_active":"2024-09-18T10:18:23.735070+0000","last_peered":"2024-09-18T10:18:23.735070+0000","last_clean":"2024-09-18T10:18:23.735070+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:18:23.735070+0000","last_undegraded":"2024-09-18T10:18:23.735070+0000","last_fullsized":"2024-09-18T10:18:23.735070+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:34.758954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860612+0000","last_change":"2024-09-18T10:15:10.263309+0000","last_active":"2024-09-18T10:18:24.860612+0000","last_peered":"2024-09-18T10:18:24.860612+0000","last_clean":"2024-09-18T10:18:24.860612+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:18:24.860612+0000","last_undegraded":"2024-09-18T10:18:24.860612+0000","last_fullsized":"2024-09-18T10:18:24.860612+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:44:24.067636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734913+0000","last_change":"2024-09-18T10:15:10.262000+0000","last_active":"2024-09-18T10:18:23.734913+0000","last_peered":"2024-09-18T10:18:23.734913+0000","last_clean":"2024-09-18T10:18:23.734913+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:18:23.734913+0000","last_undegraded":"2024-09-18T10:18:23.734913+0000","last_fullsized":"2024-09-18T10:18:23.734913+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:45:34.980736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":132,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735082+0000","last_change":"2024-09-18T10:15:12.231443+0000","last_active":"2024-09-18T10:18:23.735082+0000","last_peered":"2024-09-18T10:18:23.735082+0000","last_clean":"2024-09-18T10:18:23.735082+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:18:23.735082+0000","last_undegraded":"2024-09-18T10:18:23.735082+0000","last_fullsized":"2024-09-18T10:18:23.735082+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:21.077963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662983+0000","last_change":"2024-09-18T10:15:10.258259+0000","last_active":"2024-09-18T10:18:24.662983+0000","last_peered":"2024-09-18T10:18:24.662983+0000","last_clean":"2024-09-18T10:18:24.662983+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:18:24.662983+0000","last_undegraded":"2024-09-18T10:18:24.662983+0000","last_fullsized":"2024-09-18T10:18:24.662983+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:43.637146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150425+0000","last_change":"2024-09-18T10:15:10.268743+0000","last_active":"2024-09-18T10:18:24.150425+0000","last_peered":"2024-09-18T10:18:24.150425+0000","last_clean":"2024-09-18T10:18:24.150425+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:18:24.150425+0000","last_undegraded":"2024-09-18T10:18:24.150425+0000","last_fullsized":"2024-09-18T10:18:24.150425+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:29:46.935586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662856+0000","last_change":"2024-09-18T10:15:10.258566+0000","last_active":"2024-09-18T10:18:24.662856+0000","last_peered":"2024-09-18T10:18:24.662856+0000","last_clean":"2024-09-18T10:18:24.662856+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:18:24.662856+0000","last_undegraded":"2024-09-18T10:18:24.662856+0000","last_fullsized":"2024-09-18T10:18:24.662856+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:34:16.243392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":115,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734740+0000","last_change":"2024-09-18T10:15:10.258870+0000","last_active":"2024-09-18T10:18:23.734740+0000","last_peered":"2024-09-18T10:18:23.734740+0000","last_clean":"2024-09-18T10:18:23.734740+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:18:23.734740+0000","last_undegraded":"2024-09-18T10:18:23.734740+0000","last_fullsized":"2024-09-18T10:18:23.734740+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:54:33.116712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149784+0000","last_change":"2024-09-18T10:15:10.253346+0000","last_active":"2024-09-18T10:18:24.149784+0000","last_peered":"2024-09-18T10:18:24.149784+0000","last_clean":"2024-09-18T10:18:24.149784+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:18:24.149784+0000","last_undegraded":"2024-09-18T10:18:24.149784+0000","last_fullsized":"2024-09-18T10:18:24.149784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:56.046228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860584+0000","last_change":"2024-09-18T10:15:10.261745+0000","last_active":"2024-09-18T10:18:24.860584+0000","last_peered":"2024-09-18T10:18:24.860584+0000","last_clean":"2024-09-18T10:18:24.860584+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:18:24.860584+0000","last_undegraded":"2024-09-18T10:18:24.860584+0000","last_fullsized":"2024-09-18T10:18:24.860584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:40.372105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662648+0000","last_change":"2024-09-18T10:15:10.253445+0000","last_active":"2024-09-18T10:18:24.662648+0000","last_peered":"2024-09-18T10:18:24.662648+0000","last_clean":"2024-09-18T10:18:24.662648+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:18:24.662648+0000","last_undegraded":"2024-09-18T10:18:24.662648+0000","last_fullsized":"2024-09-18T10:18:24.662648+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:54:33.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":86,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662787+0000","last_change":"2024-09-18T10:15:10.264746+0000","last_active":"2024-09-18T10:18:24.662787+0000","last_peered":"2024-09-18T10:18:24.662787+0000","last_clean":"2024-09-18T10:18:24.662787+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:18:24.662787+0000","last_undegraded":"2024-09-18T10:18:24.662787+0000","last_fullsized":"2024-09-18T10:18:24.662787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:06:12.470999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733793+0000","last_change":"2024-09-18T10:15:10.262348+0000","last_active":"2024-09-18T10:18:23.733793+0000","last_peered":"2024-09-18T10:18:23.733793+0000","last_clean":"2024-09-18T10:18:23.733793+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:18:23.733793+0000","last_undegraded":"2024-09-18T10:18:23.733793+0000","last_fullsized":"2024-09-18T10:18:23.733793+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:28:23.247111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860518+0000","last_change":"2024-09-18T10:15:11.672918+0000","last_active":"2024-09-18T10:18:24.860518+0000","last_peered":"2024-09-18T10:18:24.860518+0000","last_clean":"2024-09-18T10:18:24.860518+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:18:24.860518+0000","last_undegraded":"2024-09-18T10:18:24.860518+0000","last_fullsized":"2024-09-18T10:18:24.860518+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:56:53.310516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662547+0000","last_change":"2024-09-18T10:15:10.260417+0000","last_active":"2024-09-18T10:18:24.662547+0000","last_peered":"2024-09-18T10:18:24.662547+0000","last_clean":"2024-09-18T10:18:24.662547+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:18:24.662547+0000","last_undegraded":"2024-09-18T10:18:24.662547+0000","last_fullsized":"2024-09-18T10:18:24.662547+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:13:26.409756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":333,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149870+0000","last_change":"2024-09-18T10:15:10.257790+0000","last_active":"2024-09-18T10:18:24.149870+0000","last_peered":"2024-09-18T10:18:24.149870+0000","last_clean":"2024-09-18T10:18:24.149870+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:18:24.149870+0000","last_undegraded":"2024-09-18T10:18:24.149870+0000","last_fullsized":"2024-09-18T10:18:24.149870+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:50:37.166570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734844+0000","last_change":"2024-09-18T10:15:10.272763+0000","last_active":"2024-09-18T10:18:23.734844+0000","last_peered":"2024-09-18T10:18:23.734844+0000","last_clean":"2024-09-18T10:18:23.734844+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:18:23.734844+0000","last_undegraded":"2024-09-18T10:18:23.734844+0000","last_fullsized":"2024-09-18T10:18:23.734844+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:14.189522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149867+0000","last_change":"2024-09-18T10:15:10.254634+0000","last_active":"2024-09-18T10:18:24.149867+0000","last_peered":"2024-09-18T10:18:24.149867+0000","last_clean":"2024-09-18T10:18:24.149867+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:18:24.149867+0000","last_undegraded":"2024-09-18T10:18:24.149867+0000","last_fullsized":"2024-09-18T10:18:24.149867+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:33:08.149535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860416+0000","last_change":"2024-09-18T10:15:10.258271+0000","last_active":"2024-09-18T10:18:24.860416+0000","last_peered":"2024-09-18T10:18:24.860416+0000","last_clean":"2024-09-18T10:18:24.860416+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:18:24.860416+0000","last_undegraded":"2024-09-18T10:18:24.860416+0000","last_fullsized":"2024-09-18T10:18:24.860416+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:18.769125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734817+0000","last_change":"2024-09-18T10:15:10.267445+0000","last_active":"2024-09-18T10:18:23.734817+0000","last_peered":"2024-09-18T10:18:23.734817+0000","last_clean":"2024-09-18T10:18:23.734817+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:18:23.734817+0000","last_undegraded":"2024-09-18T10:18:23.734817+0000","last_fullsized":"2024-09-18T10:18:23.734817+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:02:06.660884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":239,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662357+0000","last_change":"2024-09-18T10:15:12.228339+0000","last_active":"2024-09-18T10:18:24.662357+0000","last_peered":"2024-09-18T10:18:24.662357+0000","last_clean":"2024-09-18T10:18:24.662357+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:18:24.662357+0000","last_undegraded":"2024-09-18T10:18:24.662357+0000","last_fullsized":"2024-09-18T10:18:24.662357+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150004+0000","last_change":"2024-09-18T10:15:10.263155+0000","last_active":"2024-09-18T10:18:24.150004+0000","last_peered":"2024-09-18T10:18:24.150004+0000","last_clean":"2024-09-18T10:18:24.150004+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:18:24.150004+0000","last_undegraded":"2024-09-18T10:18:24.150004+0000","last_fullsized":"2024-09-18T10:18:24.150004+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:11:15.186223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.663037+0000","last_change":"2024-09-18T10:14:59.172586+0000","last_active":"2024-09-18T10:18:24.663037+0000","last_peered":"2024-09-18T10:18:24.663037+0000","last_clean":"2024-09-18T10:18:24.663037+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:18:24.663037+0000","last_undegraded":"2024-09-18T10:18:24.663037+0000","last_fullsized":"2024-09-18T10:18:24.663037+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:50.899590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150325+0000","last_change":"2024-09-18T10:15:10.258203+0000","last_active":"2024-09-18T10:18:24.150325+0000","last_peered":"2024-09-18T10:18:24.150325+0000","last_clean":"2024-09-18T10:18:24.150325+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:18:24.150325+0000","last_undegraded":"2024-09-18T10:18:24.150325+0000","last_fullsized":"2024-09-18T10:18:24.150325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:55.852806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.57399999999999995},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.268},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.61899999999999999,"5min":0.48999999999999999,"15min":0.46100000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.68200000000000005},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.54100000000000004,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.081},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.88},"min":{"1min":0.70499999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.875},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.82399999999999995,"15min":0.90000000000000002},"min":{"1min":0.73499999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.609},"last":0.873}]},{"osd":4,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.86099999999999999},"min":{"1min":0.76900000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.214,"5min":1.214,"15min":1.242},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.64400000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 18 10:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84799999999999998,"15min":0.878},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.099,"5min":1.099,"15min":1.3740000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.995,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.73899999999999999,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":1.393,"5min":1.393,"15min":1.829},"last":0.83799999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607804,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.49099999999999999,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64600000000000002},"min":{"1min":0.52200000000000002,"5min":0.438,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":1.1359999999999999,"15min":1.528},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.66900000000000004},"min":{"1min":0.56200000000000006,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.91000000000000003,"5min":0.999,"15min":1.0549999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68799999999999994,"15min":0.72099999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85299999999999998,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":0.98299999999999998,"5min":1.2589999999999999,"15min":1.377},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.875,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.627,"15min":0.48899999999999999},"max":{"1min":1.0509999999999999,"5min":1.163,"15min":1.5589999999999999},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.871},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.45100000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.5960000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90300000000000002,"15min":0.90600000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98099999999999998,"5min":1.3,"15min":1.49},"last":0.879}]},{"osd":5,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91700000000000004,"15min":0.89700000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.028,"5min":1.423,"15min":1.4590000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.53300000000000003},"max":{"1min":1.331,"5min":1.331,"15min":1.639},"last":0.93100000000000005}]}]},{"osd":5,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.78800000000000003},"min":{"1min":0.61099999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98499999999999999,"5min":1.0489999999999999,"15min":1.101},"last":0.505},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86099999999999999,"15min":0.85699999999999998},"min":{"1min":0.73499999999999999,"5min":0.504,"15min":0.504},"max":{"1min":1.125,"5min":1.222,"15min":2.4460000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.52600000000000002},"max":{"1min":1.077,"5min":1.091,"15min":1.4430000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.92100000000000004,"15min":0.877},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.399,"5min":1.399,"15min":2.3799999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.83299999999999996},"min":{"1min":0.60399999999999998,"5min":0.60099999999999998,"15min":0.55700000000000005},"max":{"1min":1.034,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.59799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72199999999999998,"15min":0.72599999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":1.1319999999999999,"15min":1.2210000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.65300000000000002,"5min":0.56699999999999995,"15min":0.45700000000000002},"max":{"1min":1.0169999999999999,"5min":1.3600000000000001,"15min":1.556},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.98699999999999999,"15min":1.0149999999999999},"min":{"1min":0.78700000000000003,"5min":0.69599999999999995,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":1.2370000000000001,"15min":1.653},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73499999999999999,"15min":0.78100000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.88900000000000001,"5min":1.165,"15min":2.327},"last":0.70799999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76800000000000002,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":0.92900000000000005,"5min":1.044,"15min":1.347},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.85199999999999998,"15min":0.85899999999999999},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":1.232}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":1.117}]}]},{"osd":3,"up_from":11,"seq":47244640508,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.627},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.91500000000000004}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":0.84299999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640507,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.59099999999999997},"min":{"1min":0.376,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.64200000000000002},"min":{"1min":0.433,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.96099999999999997},"last":0.60699999999999998}]},{"osd":2,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63300000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.86399999999999999,"5min":0.96999999999999997,"15min":1.1459999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66700000000000004,"15min":0.68400000000000005},"min":{"1min":0.50800000000000001,"5min":0.45100000000000001,"15min":0.41799999999999998},"max":{"1min":0.755,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.92000000000000004,"15min":0.91000000000000003},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.96799999999999997,"15min":0.93500000000000005},"min":{"1min":0.83399999999999996,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.0860000000000001,"5min":1.27,"15min":1.347},"last":0.96399999999999997}]},{"osd":4,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.68200000000000005,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.0600000000000001,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.871},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.022,"5min":1.0669999999999999,"15min":1.3899999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.878,"15min":0.89800000000000002},"min":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.099,"5min":1.1950000000000001,"15min":1.333},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":0.94899999999999995},"min":{"1min":0.71099999999999997,"5min":0.69899999999999995,"15min":0.64500000000000002},"max":{"1min":1.149,"5min":1.252,"15min":1.446},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:29:39.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:29:39.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: get_monmap_and_config 2024-09-18T10:29:39.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: monmap: 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:39.910387+0000 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:39.910387+0000 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:39.913 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:39.914 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 Processor -- start 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 -- start start 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: init 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.916 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-b con 0x7fe30009aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3000967a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe30009c4f0 0x7fe3001375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-a con 0x7fe30009c4f0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe300095880 con 0x7fe3000967a0 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30009aeb0 con 0x7fe30009c4f0 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006f6d0 con 0x7fe30009aae0 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.907+0000 7fe306aa0700 10 monclient(hunting): authenticate will time out at 2068.835976s 2024-09-18T10:29:39.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38068/0 (socket says 172.21.15.142:38068) 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe30009c4f0 0x7fe3001375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 -- 172.21.15.142:0/1131877213 learned_addr learned my addr 172.21.15.142:0/1131877213 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): get_auth_request con 0x7fe30009aae0 auth_method 0 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): get_auth_request con 0x7fe30009c4f0 auth_method 0 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient(hunting): handle_auth_done global_id 234149 payload 274 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 -- 172.21.15.142:0/1131877213 >> v2:172.21.15.142:3300/0 conn(0x7fe30009c4f0 msgr2=0x7fe3001375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 --2- 172.21.15.142:0/1131877213 >> v2:172.21.15.142:3300/0 conn(0x7fe30009c4f0 0x7fe3001375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 -- 172.21.15.142:0/1131877213 >> v1:172.21.15.150:6789/0 conn(0x7fe3000967a0 legacy=0x7fe30009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.919 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 monclient: found mon.noname-b 2024-09-18T10:29:39.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:39.920 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 -- 172.21.15.142:0/1131877213 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000955a0 con 0x7fe30009aae0 2024-09-18T10:29:39.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 -- 172.21.15.142:0/1131877213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 msgr2=0x7fe300096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe305a9e700 1 --2- 172.21.15.142:0/1131877213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00099b0 tx=0x7fe2f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ebbbef694ff1095 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 1 -- 172.21.15.142:0/1131877213 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0009ea0 con 0x7fe30009aae0 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 10 monclient: dump: 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.925 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.926 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe304a9c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe306aa0700 5 monclient: authenticate success, global_id 234149 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.911+0000 7fe306aa0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 1 -- 172.21.15.142:0/1131877213 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f000ba70 con 0x7fe30009aae0 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 4 set_mon_vals no callback set 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 1 -- 172.21.15.142:0/1131877213 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000be30 con 0x7fe30009aae0 2024-09-18T10:29:39.927 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe304a9c700 10 monclient: dump: 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.928 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 10 monclient: shutdown 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:39.929 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- 172.21.15.142:0/1131877213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 msgr2=0x7fe300096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1131877213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00099b0 tx=0x7fe2f0009250 comp rx=0 tx=0).stop 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- 172.21.15.142:0/1131877213 shutdown_connections 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1131877213 >> v2:172.21.15.142:3300/0 conn(0x7fe30009c4f0 0x7fe3001375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1131877213 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe30009aae0 0x7fe300096260 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- 172.21.15.142:0/1131877213 >> 172.21.15.142:0/1131877213 conn(0x7fe3000a1fe0 msgr2=0x7fe300070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- 172.21.15.142:0/1131877213 shutdown_connections 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- 172.21.15.142:0/1131877213 wait complete. 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 10 monclient: build_initial_monmap 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 10 monclient: monmap: 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:39.930 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.931 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 Processor -- start 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 1 -- start start 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.915+0000 7fe306aa0700 10 monclient: init 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-c con 0x7fe3000967a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 0x7fe3000943c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-b con 0x7fe30009aae0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 0x7fe300098900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.932 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): picked mon.noname-a con 0x7fe30009c4f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000915e0 con 0x7fe30009c4f0 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30006f300 con 0x7fe30009aae0 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000976f0 con 0x7fe3000967a0 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient(hunting): authenticate will time out at 2068.844569s 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 0x7fe300098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.933 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38078/0 (socket says 172.21.15.142:38078) 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 1 -- 172.21.15.142:0/1509403141 learned_addr learned my addr 172.21.15.142:0/1509403141 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): get_auth_request con 0x7fe3000967a0 auth_method 0 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.934 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): get_auth_request con 0x7fe30009c4f0 auth_method 0 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.935 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 0x7fe3000943c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 10 monclient(hunting): get_auth_request con 0x7fe30009aae0 auth_method 0 2024-09-18T10:29:39.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:39.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:39.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:39.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient(hunting): handle_auth_done global_id 224268 payload 824 2024-09-18T10:29:39.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 -- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 msgr2=0x7fe3001417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.937 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 -- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 msgr2=0x7fe3000943c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 0x7fe3000943c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient: _finish_hunting 0 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 monclient: found mon.noname-a 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 -- 172.21.15.142:0/1509403141 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00094f0 con 0x7fe30009c4f0 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe305a9e700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:39.938 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30529d700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 0x7fe3000943c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:39.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 -- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 msgr2=0x7fe300098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:39.942 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe30629f700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 0x7fe300098900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009e50 tx=0x7fe2f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f800cd60 con 0x7fe30009c4f0 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: dump: 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.943 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.944 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.944 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.944 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.944 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.944 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 5 monclient: authenticate success, global_id 224268 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient: _renew_subs 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f800f040 con 0x7fe30009c4f0 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3001426c0 con 0x7fe30009c4f0 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80146d0 con 0x7fe30009c4f0 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:39.945 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f5ffb700 10 monclient: dump: 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:39.946 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe306aa0700 10 monclient: _renew_subs 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.919+0000 7fe2f7fff700 4 set_mon_vals no callback set 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30009a480 con 0x7fe30009c4f0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300072a80 con 0x7fe30009c4f0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe2f8014a40 con 0x7fe30009c4f0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe2f5ffb700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe2f804a0f0 con 0x7fe30009c4f0 2024-09-18T10:29:39.947 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe305a9e700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:39.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.923+0000 7fe305a9e700 10 monclient: get_auth_request con 0x7fe2e4035ac0 auth_method 0 2024-09-18T10:29:39.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.927+0000 7fe305a9e700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00030a0 tx=0x7fe2f0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:39.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.931+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fe2f8012340 con 0x7fe30009c4f0 2024-09-18T10:29:39.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.931+0000 7fe2f5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:39.948 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:39.931+0000 7fe2f5ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:40.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.095+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3001426c0 con 0x7fe2e4035ac0 2024-09-18T10:29:40.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe2f5ffb700 1 -- 172.21.15.142:0/1509403141 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97482 (secure 0 0 0) 0x7fe3001426c0 con 0x7fe2e4035ac0 2024-09-18T10:29:40.108 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:40.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 msgr2=0x7fe2e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00030a0 tx=0x7fe2f0016000 comp rx=0 tx=0).stop 2024-09-18T10:29:40.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 10 monclient: shutdown 2024-09-18T10:29:40.110 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 msgr2=0x7fe300098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 0x7fe300098900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009e50 tx=0x7fe2f800b040 comp rx=0 tx=0).stop 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 shutdown_connections 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe2e4035ac0 0x7fe2e4037f70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe30009c4f0 0x7fe300098900 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe30009aae0 0x7fe3000943c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 --2- 172.21.15.142:0/1509403141 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe3000967a0 0x7fe3001417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 >> 172.21.15.142:0/1509403141 conn(0x7fe3000a1fe0 msgr2=0x7fe30009fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 shutdown_connections 2024-09-18T10:29:40.111 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.103+0000 7fe306aa0700 1 -- 172.21.15.142:0/1509403141 wait complete. 2024-09-18T10:29:40.112 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:29:40.123 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":876,"stamp":"2024-09-18T10:29:39.131981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820536,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410504,"statfs":{"total":579820584960,"available":577956356096,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003770"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":174,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733987+0000","last_change":"2024-09-18T10:15:10.262207+0000","last_active":"2024-09-18T10:18:23.733987+0000","last_peered":"2024-09-18T10:18:23.733987+0000","last_clean":"2024-09-18T10:18:23.733987+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:18:23.733987+0000","last_undegraded":"2024-09-18T10:18:23.733987+0000","last_fullsized":"2024-09-18T10:18:23.733987+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:41:30.693246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735070+0000","last_change":"2024-09-18T10:15:10.259663+0000","last_active":"2024-09-18T10:18:23.735070+0000","last_peered":"2024-09-18T10:18:23.735070+0000","last_clean":"2024-09-18T10:18:23.735070+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:18:23.735070+0000","last_undegraded":"2024-09-18T10:18:23.735070+0000","last_fullsized":"2024-09-18T10:18:23.735070+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:34.758954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860612+0000","last_change":"2024-09-18T10:15:10.263309+0000","last_active":"2024-09-18T10:18:24.860612+0000","last_peered":"2024-09-18T10:18:24.860612+0000","last_clean":"2024-09-18T10:18:24.860612+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:18:24.860612+0000","last_undegraded":"2024-09-18T10:18:24.860612+0000","last_fullsized":"2024-09-18T10:18:24.860612+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:44:24.067636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734913+0000","last_change":"2024-09-18T10:15:10.262000+0000","last_active":"2024-09-18T10:18:23.734913+0000","last_peered":"2024-09-18T10:18:23.734913+0000","last_clean":"2024-09-18T10:18:23.734913+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:18:23.734913+0000","last_undegraded":"2024-09-18T10:18:23.734913+0000","last_fullsized":"2024-09-18T10:18:23.734913+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:45:34.980736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":132,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735082+0000","last_change":"2024-09-18T10:15:12.231443+0000","last_active":"2024-09-18T10:18:23.735082+0000","last_peered":"2024-09-18T10:18:23.735082+0000","last_clean":"2024-09-18T10:18:23.735082+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:18:23.735082+0000","last_undegraded":"2024-09-18T10:18:23.735082+0000","last_fullsized":"2024-09-18T10:18:23.735082+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:21.077963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662983+0000","last_change":"2024-09-18T10:15:10.258259+0000","last_active":"2024-09-18T10:18:24.662983+0000","last_peered":"2024-09-18T10:18:24.662983+0000","last_clean":"2024-09-18T10:18:24.662983+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:18:24.662983+0000","last_undegraded":"2024-09-18T10:18:24.662983+0000","last_fullsized":"2024-09-18T10:18:24.662983+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:43.637146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150425+0000","last_change":"2024-09-18T10:15:10.268743+0000","last_active":"2024-09-18T10:18:24.150425+0000","last_peered":"2024-09-18T10:18:24.150425+0000","last_clean":"2024-09-18T10:18:24.150425+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:18:24.150425+0000","last_undegraded":"2024-09-18T10:18:24.150425+0000","last_fullsized":"2024-09-18T10:18:24.150425+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:29:46.935586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662856+0000","last_change":"2024-09-18T10:15:10.258566+0000","last_active":"2024-09-18T10:18:24.662856+0000","last_peered":"2024-09-18T10:18:24.662856+0000","last_clean":"2024-09-18T10:18:24.662856+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:18:24.662856+0000","last_undegraded":"2024-09-18T10:18:24.662856+0000","last_fullsized":"2024-09-18T10:18:24.662856+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:34:16.243392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":115,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734740+0000","last_change":"2024-09-18T10:15:10.258870+0000","last_active":"2024-09-18T10:18:23.734740+0000","last_peered":"2024-09-18T10:18:23.734740+0000","last_clean":"2024-09-18T10:18:23.734740+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:18:23.734740+0000","last_undegraded":"2024-09-18T10:18:23.734740+0000","last_fullsized":"2024-09-18T10:18:23.734740+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:54:33.116712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149784+0000","last_change":"2024-09-18T10:15:10.253346+0000","last_active":"2024-09-18T10:18:24.149784+0000","last_peered":"2024-09-18T10:18:24.149784+0000","last_clean":"2024-09-18T10:18:24.149784+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:18:24.149784+0000","last_undegraded":"2024-09-18T10:18:24.149784+0000","last_fullsized":"2024-09-18T10:18:24.149784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:56.046228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860584+0000","last_change":"2024-09-18T10:15:10.261745+0000","last_active":"2024-09-18T10:18:24.860584+0000","last_peered":"2024-09-18T10:18:24.860584+0000","last_clean":"2024-09-18T10:18:24.860584+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:18:24.860584+0000","last_undegraded":"2024-09-18T10:18:24.860584+0000","last_fullsized":"2024-09-18T10:18:24.860584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:40.372105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662648+0000","last_change":"2024-09-18T10:15:10.253445+0000","last_active":"2024-09-18T10:18:24.662648+0000","last_peered":"2024-09-18T10:18:24.662648+0000","last_clean":"2024-09-18T10:18:24.662648+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:18:24.662648+0000","last_undegraded":"2024-09-18T10:18:24.662648+0000","last_fullsized":"2024-09-18T10:18:24.662648+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:54:33.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":86,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662787+0000","last_change":"2024-09-18T10:15:10.264746+0000","last_active":"2024-09-18T10:18:24.662787+0000","last_peered":"2024-09-18T10:18:24.662787+0000","last_clean":"2024-09-18T10:18:24.662787+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:18:24.662787+0000","last_undegraded":"2024-09-18T10:18:24.662787+0000","last_fullsized":"2024-09-18T10:18:24.662787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:06:12.470999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733793+0000","last_change":"2024-09-18T10:15:10.262348+0000","last_active":"2024-09-18T10:18:23.733793+0000","last_peered":"2024-09-18T10:18:23.733793+0000","last_clean":"2024-09-18T10:18:23.733793+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:18:23.733793+0000","last_undegraded":"2024-09-18T10:18:23.733793+0000","last_fullsized":"2024-09-18T10:18:23.733793+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:28:23.247111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860518+0000","last_change":"2024-09-18T10:15:11.672918+0000","last_active":"2024-09-18T10:18:24.860518+0000","last_peered":"2024-09-18T10:18:24.860518+0000","last_clean":"2024-09-18T10:18:24.860518+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:18:24.860518+0000","last_undegraded":"2024-09-18T10:18:24.860518+0000","last_fullsized":"2024-09-18T10:18:24.860518+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:56:53.310516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662547+0000","last_change":"2024-09-18T10:15:10.260417+0000","last_active":"2024-09-18T10:18:24.662547+0000","last_peered":"2024-09-18T10:18:24.662547+0000","last_clean":"2024-09-18T10:18:24.662547+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:18:24.662547+0000","last_undegraded":"2024-09-18T10:18:24.662547+0000","last_fullsized":"2024-09-18T10:18:24.662547+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:13:26.409756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":333,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149870+0000","last_change":"2024-09-18T10:15:10.257790+0000","last_active":"2024-09-18T10:18:24.149870+0000","last_peered":"2024-09-18T10:18:24.149870+0000","last_clean":"2024-09-18T10:18:24.149870+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:18:24.149870+0000","last_undegraded":"2024-09-18T10:18:24.149870+0000","last_fullsized":"2024-09-18T10:18:24.149870+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:50:37.166570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734844+0000","last_change":"2024-09-18T10:15:10.272763+0000","last_active":"2024-09-18T10:18:23.734844+0000","last_peered":"2024-09-18T10:18:23.734844+0000","last_clean":"2024-09-18T10:18:23.734844+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:18:23.734844+0000","last_undegraded":"2024-09-18T10:18:23.734844+0000","last_fullsized":"2024-09-18T10:18:23.734844+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:14.189522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149867+0000","last_change":"2024-09-18T10:15:10.254634+0000","last_active":"2024-09-18T10:18:24.149867+0000","last_peered":"2024-09-18T10:18:24.149867+0000","last_clean":"2024-09-18T10:18:24.149867+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:18:24.149867+0000","last_undegraded":"2024-09-18T10:18:24.149867+0000","last_fullsized":"2024-09-18T10:18:24.149867+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:33:08.149535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860416+0000","last_change":"2024-09-18T10:15:10.258271+0000","last_active":"2024-09-18T10:18:24.860416+0000","last_peered":"2024-09-18T10:18:24.860416+0000","last_clean":"2024-09-18T10:18:24.860416+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:18:24.860416+0000","last_undegraded":"2024-09-18T10:18:24.860416+0000","last_fullsized":"2024-09-18T10:18:24.860416+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:18.769125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734817+0000","last_change":"2024-09-18T10:15:10.267445+0000","last_active":"2024-09-18T10:18:23.734817+0000","last_peered":"2024-09-18T10:18:23.734817+0000","last_clean":"2024-09-18T10:18:23.734817+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:18:23.734817+0000","last_undegraded":"2024-09-18T10:18:23.734817+0000","last_fullsized":"2024-09-18T10:18:23.734817+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:02:06.660884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":239,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662357+0000","last_change":"2024-09-18T10:15:12.228339+0000","last_active":"2024-09-18T10:18:24.662357+0000","last_peered":"2024-09-18T10:18:24.662357+0000","last_clean":"2024-09-18T10:18:24.662357+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:18:24.662357+0000","last_undegraded":"2024-09-18T10:18:24.662357+0000","last_fullsized":"2024-09-18T10:18:24.662357+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150004+0000","last_change":"2024-09-18T10:15:10.263155+0000","last_active":"2024-09-18T10:18:24.150004+0000","last_peered":"2024-09-18T10:18:24.150004+0000","last_clean":"2024-09-18T10:18:24.150004+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:18:24.150004+0000","last_undegraded":"2024-09-18T10:18:24.150004+0000","last_fullsized":"2024-09-18T10:18:24.150004+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:11:15.186223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.663037+0000","last_change":"2024-09-18T10:14:59.172586+0000","last_active":"2024-09-18T10:18:24.663037+0000","last_peered":"2024-09-18T10:18:24.663037+0000","last_clean":"2024-09-18T10:18:24.663037+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:18:24.663037+0000","last_undegraded":"2024-09-18T10:18:24.663037+0000","last_fullsized":"2024-09-18T10:18:24.663037+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:50.899590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150325+0000","last_change":"2024-09-18T10:15:10.258203+0000","last_active":"2024-09-18T10:18:24.150325+0000","last_peered":"2024-09-18T10:18:24.150325+0000","last_clean":"2024-09-18T10:18:24.150325+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:18:24.150325+0000","last_undegraded":"2024-09-18T10:18:24.150325+0000","last_fullsized":"2024-09-18T10:18:24.150325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:55.852806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.57399999999999995},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.268},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.61899999999999999,"5min":0.48999999999999999,"15min":0.46100000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.68200000000000005},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.54100000000000004,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.081},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.88},"min":{"1min":0.70499999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.875},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.82399999999999995,"15min":0.90000000000000002},"min":{"1min":0.73499999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.609},"last":0.873}]},{"osd":4,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.86099999999999999},"min":{"1min":0.76900000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.214,"5min":1.214,"15min":1.242},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.64400000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 18 10:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84799999999999998,"15min":0.878},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.099,"5min":1.099,"15min":1.3740000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.995,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.73899999999999999,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":1.393,"5min":1.393,"15min":1.829},"last":0.83799999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607804,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.49099999999999999,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64600000000000002},"min":{"1min":0.52200000000000002,"5min":0.438,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":1.1359999999999999,"15min":1.528},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.66900000000000004},"min":{"1min":0.56200000000000006,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.91000000000000003,"5min":0.999,"15min":1.0549999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68799999999999994,"15min":0.72099999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85299999999999998,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":0.98299999999999998,"5min":1.2589999999999999,"15min":1.377},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.875,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.627,"15min":0.48899999999999999},"max":{"1min":1.0509999999999999,"5min":1.163,"15min":1.5589999999999999},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.871},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.45100000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.5960000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90300000000000002,"15min":0.90600000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98099999999999998,"5min":1.3,"15min":1.49},"last":0.879}]},{"osd":5,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91700000000000004,"15min":0.89700000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.028,"5min":1.423,"15min":1.4590000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.53300000000000003},"max":{"1min":1.331,"5min":1.331,"15min":1.639},"last":0.93100000000000005}]}]},{"osd":5,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.78800000000000003},"min":{"1min":0.61099999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98499999999999999,"5min":1.0489999999999999,"15min":1.101},"last":0.505},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86099999999999999,"15min":0.85699999999999998},"min":{"1min":0.73499999999999999,"5min":0.504,"15min":0.504},"max":{"1min":1.125,"5min":1.222,"15min":2.4460000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.52600000000000002},"max":{"1min":1.077,"5min":1.091,"15min":1.4430000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.92100000000000004,"15min":0.877},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.399,"5min":1.399,"15min":2.3799999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.83299999999999996},"min":{"1min":0.60399999999999998,"5min":0.60099999999999998,"15min":0.55700000000000005},"max":{"1min":1.034,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.59799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72199999999999998,"15min":0.72599999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":1.1319999999999999,"15min":1.2210000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.65300000000000002,"5min":0.56699999999999995,"15min":0.45700000000000002},"max":{"1min":1.0169999999999999,"5min":1.3600000000000001,"15min":1.556},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.98699999999999999,"15min":1.0149999999999999},"min":{"1min":0.78700000000000003,"5min":0.69599999999999995,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":1.2370000000000001,"15min":1.653},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73499999999999999,"15min":0.78100000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.88900000000000001,"5min":1.165,"15min":2.327},"last":0.70799999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76800000000000002,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":0.92900000000000005,"5min":1.044,"15min":1.347},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.85199999999999998,"15min":0.85899999999999999},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":1.232}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":1.117}]}]},{"osd":3,"up_from":11,"seq":47244640508,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.627},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.91500000000000004}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":0.84299999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640507,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.59099999999999997},"min":{"1min":0.376,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.64200000000000002},"min":{"1min":0.433,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.96099999999999997},"last":0.60699999999999998}]},{"osd":2,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63300000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.86399999999999999,"5min":0.96999999999999997,"15min":1.1459999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66700000000000004,"15min":0.68400000000000005},"min":{"1min":0.50800000000000001,"5min":0.45100000000000001,"15min":0.41799999999999998},"max":{"1min":0.755,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.92000000000000004,"15min":0.91000000000000003},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.96799999999999997,"15min":0.93500000000000005},"min":{"1min":0.83399999999999996,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.0860000000000001,"5min":1.27,"15min":1.347},"last":0.96399999999999997}]},{"osd":4,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.68200000000000005,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.0600000000000001,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.871},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.022,"5min":1.0669999999999999,"15min":1.3899999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.878,"15min":0.89800000000000002},"min":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.099,"5min":1.1950000000000001,"15min":1.333},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":0.94899999999999995},"min":{"1min":0.71099999999999997,"5min":0.69899999999999995,"15min":0.64500000000000002},"max":{"1min":1.149,"5min":1.252,"15min":1.446},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:29:40.125 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T10:29:40.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: get_monmap_and_config 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: build_initial_monmap 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: monmap: 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.235 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:40.233047+0000 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:40.233047+0000 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:40.236 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:40.237 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 Processor -- start 2024-09-18T10:29:40.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 -- start start 2024-09-18T10:29:40.248 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: init 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-b con 0x7f0b7409aae0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b740967a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b7409c4f0 0x7f0b741375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-c con 0x7f0b7409c4f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b74095880 con 0x7f0b740967a0 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.249 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7409aeb0 con 0x7f0b7409c4f0 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7406f6d0 con 0x7f0b7409aae0 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b78fe3700 10 monclient(hunting): authenticate will time out at 2069.158466s 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38086/0 (socket says 172.21.15.142:38086) 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 -- 172.21.15.142:0/2529623446 learned_addr learned my addr 172.21.15.142:0/2529623446 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): get_auth_request con 0x7f0b7409aae0 auth_method 0 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b73fff700 1 --2- 172.21.15.142:0/2529623446 >> v2:172.21.15.150:3300/0 conn(0x7f0b7409c4f0 0x7f0b741375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.250 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b73fff700 10 monclient(hunting): get_auth_request con 0x7f0b7409c4f0 auth_method 0 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b73fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient(hunting): handle_auth_done global_id 234155 payload 274 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 -- 172.21.15.142:0/2529623446 >> v2:172.21.15.150:3300/0 conn(0x7f0b7409c4f0 msgr2=0x7f0b741375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 --2- 172.21.15.142:0/2529623446 >> v2:172.21.15.150:3300/0 conn(0x7f0b7409c4f0 0x7f0b741375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 -- 172.21.15.142:0/2529623446 >> v1:172.21.15.150:6789/0 conn(0x7f0b740967a0 legacy=0x7f0b7409bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:40.251 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 monclient: found mon.noname-b 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.231+0000 7f0b737fe700 1 -- 172.21.15.142:0/2529623446 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740955a0 con 0x7f0b7409aae0 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b737fe700 1 -- 172.21.15.142:0/2529623446 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 msgr2=0x7f0b74096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b737fe700 1 --2- 172.21.15.142:0/2529623446 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b680099b0 tx=0x7f0b68009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a559de46c8dfb67 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 1 -- 172.21.15.142:0/2529623446 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6800ba70 con 0x7f0b7409aae0 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: dump: 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.252 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 5 monclient: authenticate success, global_id 234155 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 1 -- 172.21.15.142:0/2529623446 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6800bbd0 con 0x7f0b7409aae0 2024-09-18T10:29:40.253 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 4 set_mon_vals no callback set 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 1 -- 172.21.15.142:0/2529623446 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68011420 con 0x7f0b7409aae0 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b727fc700 10 monclient: dump: 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.254 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 10 monclient: shutdown 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 -- 172.21.15.142:0/2529623446 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 msgr2=0x7f0b74096260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/2529623446 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0b680099b0 tx=0x7f0b68009250 comp rx=0 tx=0).stop 2024-09-18T10:29:40.255 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 -- 172.21.15.142:0/2529623446 shutdown_connections 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/2529623446 >> v2:172.21.15.150:3300/0 conn(0x7f0b7409c4f0 0x7f0b741375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/2529623446 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409aae0 0x7f0b74096260 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 -- 172.21.15.142:0/2529623446 >> 172.21.15.142:0/2529623446 conn(0x7f0b740a1fe0 msgr2=0x7f0b74070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 -- 172.21.15.142:0/2529623446 shutdown_connections 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 -- 172.21.15.142:0/2529623446 wait complete. 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 10 monclient: build_initial_monmap 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 10 monclient: monmap: 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:40.256 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.235+0000 7f0b78fe3700 1 Processor -- start 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 -- start start 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient: init 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b740967a0 0x7f0b7413f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-b con 0x7f0b740967a0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:40.257 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 0x7f0b7413fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-a con 0x7f0b7409aae0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 0x7f0b741400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): picked mon.noname-c con 0x7f0b7409c4f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7406ecd0 con 0x7f0b7409aae0 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7408db60 con 0x7f0b740967a0 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7406f8d0 con 0x7f0b7409c4f0 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b78fe3700 10 monclient(hunting): authenticate will time out at 2069.164859s 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 0x7f0b741400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.258 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 0x7f0b741400d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38090/0 (socket says 172.21.15.142:38090) 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 1 -- 172.21.15.142:0/3511740703 learned_addr learned my addr 172.21.15.142:0/3511740703 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): get_auth_request con 0x7f0b7409c4f0 auth_method 0 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 0x7f0b7413fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.259 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b740967a0 0x7f0b7413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): get_auth_request con 0x7f0b7409aae0 auth_method 0 2024-09-18T10:29:40.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.260 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): get_auth_request con 0x7f0b740967a0 auth_method 0 2024-09-18T10:29:40.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.261 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.262 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient(hunting): handle_auth_done global_id 224271 payload 824 2024-09-18T10:29:40.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 -- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 msgr2=0x7f0b741400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.263 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 0x7f0b741400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 -- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b740967a0 msgr2=0x7f0b7413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b740967a0 0x7f0b7413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:40.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 monclient: found mon.noname-a 2024-09-18T10:29:40.264 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 -- 172.21.15.142:0/3511740703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680094f0 con 0x7f0b7409aae0 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 -- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 msgr2=0x7f0b7413fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.239+0000 7f0b72ffd700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 0x7f0b7413fb90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c890 tx=0x7f0b640026a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64012840 con 0x7f0b7409aae0 2024-09-18T10:29:40.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:40.266 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: dump: 2024-09-18T10:29:40.266 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.266 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.266 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.267 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.267 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.267 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.267 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.267 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.268 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.268 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:40.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:40.268 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 5 monclient: authenticate success, global_id 224271 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _renew_subs 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b640129c0 con 0x7f0b7409aae0 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7408e1b0 con 0x7f0b7409aae0 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:40.269 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64012d80 con 0x7f0b7409aae0 2024-09-18T10:29:40.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b717fa700 4 set_mon_vals no callback set 2024-09-18T10:29:40.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:40.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 10 monclient: dump: 2024-09-18T10:29:40.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _renew_subs 2024-09-18T10:29:40.272 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7408f110 con 0x7f0b7409aae0 2024-09-18T10:29:40.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7408f7d0 con 0x7f0b7409aae0 2024-09-18T10:29:40.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f0b64011590 con 0x7f0b7409aae0 2024-09-18T10:29:40.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 0x7f0b60037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f0b64049290 con 0x7f0b7409aae0 2024-09-18T10:29:40.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b737fe700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 0x7f0b60037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.274 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.243+0000 7f0b737fe700 10 monclient: get_auth_request con 0x7f0b60035ac0 auth_method 0 2024-09-18T10:29:40.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.247+0000 7f0b737fe700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 0x7f0b60037f70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0b680099b0 tx=0x7f0b68016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.251+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f0b6400a8f0 con 0x7f0b7409aae0 2024-09-18T10:29:40.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.251+0000 7f0b5b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.251+0000 7f0b5b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:40.417 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.411+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b74072a80 con 0x7f0b60035ac0 2024-09-18T10:29:40.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.419+0000 7f0b5b7fe700 1 -- 172.21.15.142:0/3511740703 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97482 (secure 0 0 0) 0x7f0b74072a80 con 0x7f0b60035ac0 2024-09-18T10:29:40.426 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 msgr2=0x7f0b60037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 0x7f0b60037f70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0b680099b0 tx=0x7f0b68016000 comp rx=0 tx=0).stop 2024-09-18T10:29:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 10 monclient: shutdown 2024-09-18T10:29:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:40.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 msgr2=0x7f0b7413fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 0x7f0b7413fb90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400c890 tx=0x7f0b640026a0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 shutdown_connections 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f0b60035ac0 0x7f0b60037f70 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0b7409c4f0 0x7f0b741400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b7409aae0 0x7f0b7413fb90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 --2- 172.21.15.142:0/3511740703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b740967a0 0x7f0b7413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 >> 172.21.15.142:0/3511740703 conn(0x7f0b740a1fe0 msgr2=0x7f0b7406c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 shutdown_connections 2024-09-18T10:29:40.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.423+0000 7f0b78fe3700 1 -- 172.21.15.142:0/3511740703 wait complete. 2024-09-18T10:29:40.430 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:29:40.442 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":876,"stamp":"2024-09-18T10:29:39.131981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820536,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410504,"statfs":{"total":579820584960,"available":577956356096,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003770"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":174,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733987+0000","last_change":"2024-09-18T10:15:10.262207+0000","last_active":"2024-09-18T10:18:23.733987+0000","last_peered":"2024-09-18T10:18:23.733987+0000","last_clean":"2024-09-18T10:18:23.733987+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:18:23.733987+0000","last_undegraded":"2024-09-18T10:18:23.733987+0000","last_fullsized":"2024-09-18T10:18:23.733987+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:41:30.693246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735070+0000","last_change":"2024-09-18T10:15:10.259663+0000","last_active":"2024-09-18T10:18:23.735070+0000","last_peered":"2024-09-18T10:18:23.735070+0000","last_clean":"2024-09-18T10:18:23.735070+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:18:23.735070+0000","last_undegraded":"2024-09-18T10:18:23.735070+0000","last_fullsized":"2024-09-18T10:18:23.735070+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:34.758954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860612+0000","last_change":"2024-09-18T10:15:10.263309+0000","last_active":"2024-09-18T10:18:24.860612+0000","last_peered":"2024-09-18T10:18:24.860612+0000","last_clean":"2024-09-18T10:18:24.860612+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:18:24.860612+0000","last_undegraded":"2024-09-18T10:18:24.860612+0000","last_fullsized":"2024-09-18T10:18:24.860612+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:44:24.067636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734913+0000","last_change":"2024-09-18T10:15:10.262000+0000","last_active":"2024-09-18T10:18:23.734913+0000","last_peered":"2024-09-18T10:18:23.734913+0000","last_clean":"2024-09-18T10:18:23.734913+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:18:23.734913+0000","last_undegraded":"2024-09-18T10:18:23.734913+0000","last_fullsized":"2024-09-18T10:18:23.734913+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:45:34.980736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":132,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735082+0000","last_change":"2024-09-18T10:15:12.231443+0000","last_active":"2024-09-18T10:18:23.735082+0000","last_peered":"2024-09-18T10:18:23.735082+0000","last_clean":"2024-09-18T10:18:23.735082+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:18:23.735082+0000","last_undegraded":"2024-09-18T10:18:23.735082+0000","last_fullsized":"2024-09-18T10:18:23.735082+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:21.077963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662983+0000","last_change":"2024-09-18T10:15:10.258259+0000","last_active":"2024-09-18T10:18:24.662983+0000","last_peered":"2024-09-18T10:18:24.662983+0000","last_clean":"2024-09-18T10:18:24.662983+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:18:24.662983+0000","last_undegraded":"2024-09-18T10:18:24.662983+0000","last_fullsized":"2024-09-18T10:18:24.662983+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:43.637146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150425+0000","last_change":"2024-09-18T10:15:10.268743+0000","last_active":"2024-09-18T10:18:24.150425+0000","last_peered":"2024-09-18T10:18:24.150425+0000","last_clean":"2024-09-18T10:18:24.150425+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:18:24.150425+0000","last_undegraded":"2024-09-18T10:18:24.150425+0000","last_fullsized":"2024-09-18T10:18:24.150425+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:29:46.935586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662856+0000","last_change":"2024-09-18T10:15:10.258566+0000","last_active":"2024-09-18T10:18:24.662856+0000","last_peered":"2024-09-18T10:18:24.662856+0000","last_clean":"2024-09-18T10:18:24.662856+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:18:24.662856+0000","last_undegraded":"2024-09-18T10:18:24.662856+0000","last_fullsized":"2024-09-18T10:18:24.662856+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:34:16.243392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":115,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734740+0000","last_change":"2024-09-18T10:15:10.258870+0000","last_active":"2024-09-18T10:18:23.734740+0000","last_peered":"2024-09-18T10:18:23.734740+0000","last_clean":"2024-09-18T10:18:23.734740+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:18:23.734740+0000","last_undegraded":"2024-09-18T10:18:23.734740+0000","last_fullsized":"2024-09-18T10:18:23.734740+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:54:33.116712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149784+0000","last_change":"2024-09-18T10:15:10.253346+0000","last_active":"2024-09-18T10:18:24.149784+0000","last_peered":"2024-09-18T10:18:24.149784+0000","last_clean":"2024-09-18T10:18:24.149784+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:18:24.149784+0000","last_undegraded":"2024-09-18T10:18:24.149784+0000","last_fullsized":"2024-09-18T10:18:24.149784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:56.046228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860584+0000","last_change":"2024-09-18T10:15:10.261745+0000","last_active":"2024-09-18T10:18:24.860584+0000","last_peered":"2024-09-18T10:18:24.860584+0000","last_clean":"2024-09-18T10:18:24.860584+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:18:24.860584+0000","last_undegraded":"2024-09-18T10:18:24.860584+0000","last_fullsized":"2024-09-18T10:18:24.860584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:40.372105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662648+0000","last_change":"2024-09-18T10:15:10.253445+0000","last_active":"2024-09-18T10:18:24.662648+0000","last_peered":"2024-09-18T10:18:24.662648+0000","last_clean":"2024-09-18T10:18:24.662648+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:18:24.662648+0000","last_undegraded":"2024-09-18T10:18:24.662648+0000","last_fullsized":"2024-09-18T10:18:24.662648+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:54:33.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":86,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662787+0000","last_change":"2024-09-18T10:15:10.264746+0000","last_active":"2024-09-18T10:18:24.662787+0000","last_peered":"2024-09-18T10:18:24.662787+0000","last_clean":"2024-09-18T10:18:24.662787+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:18:24.662787+0000","last_undegraded":"2024-09-18T10:18:24.662787+0000","last_fullsized":"2024-09-18T10:18:24.662787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:06:12.470999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733793+0000","last_change":"2024-09-18T10:15:10.262348+0000","last_active":"2024-09-18T10:18:23.733793+0000","last_peered":"2024-09-18T10:18:23.733793+0000","last_clean":"2024-09-18T10:18:23.733793+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:18:23.733793+0000","last_undegraded":"2024-09-18T10:18:23.733793+0000","last_fullsized":"2024-09-18T10:18:23.733793+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:28:23.247111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860518+0000","last_change":"2024-09-18T10:15:11.672918+0000","last_active":"2024-09-18T10:18:24.860518+0000","last_peered":"2024-09-18T10:18:24.860518+0000","last_clean":"2024-09-18T10:18:24.860518+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:18:24.860518+0000","last_undegraded":"2024-09-18T10:18:24.860518+0000","last_fullsized":"2024-09-18T10:18:24.860518+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:56:53.310516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662547+0000","last_change":"2024-09-18T10:15:10.260417+0000","last_active":"2024-09-18T10:18:24.662547+0000","last_peered":"2024-09-18T10:18:24.662547+0000","last_clean":"2024-09-18T10:18:24.662547+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:18:24.662547+0000","last_undegraded":"2024-09-18T10:18:24.662547+0000","last_fullsized":"2024-09-18T10:18:24.662547+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:13:26.409756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":333,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149870+0000","last_change":"2024-09-18T10:15:10.257790+0000","last_active":"2024-09-18T10:18:24.149870+0000","last_peered":"2024-09-18T10:18:24.149870+0000","last_clean":"2024-09-18T10:18:24.149870+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:18:24.149870+0000","last_undegraded":"2024-09-18T10:18:24.149870+0000","last_fullsized":"2024-09-18T10:18:24.149870+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:50:37.166570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734844+0000","last_change":"2024-09-18T10:15:10.272763+0000","last_active":"2024-09-18T10:18:23.734844+0000","last_peered":"2024-09-18T10:18:23.734844+0000","last_clean":"2024-09-18T10:18:23.734844+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:18:23.734844+0000","last_undegraded":"2024-09-18T10:18:23.734844+0000","last_fullsized":"2024-09-18T10:18:23.734844+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:14.189522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149867+0000","last_change":"2024-09-18T10:15:10.254634+0000","last_active":"2024-09-18T10:18:24.149867+0000","last_peered":"2024-09-18T10:18:24.149867+0000","last_clean":"2024-09-18T10:18:24.149867+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:18:24.149867+0000","last_undegraded":"2024-09-18T10:18:24.149867+0000","last_fullsized":"2024-09-18T10:18:24.149867+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:33:08.149535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860416+0000","last_change":"2024-09-18T10:15:10.258271+0000","last_active":"2024-09-18T10:18:24.860416+0000","last_peered":"2024-09-18T10:18:24.860416+0000","last_clean":"2024-09-18T10:18:24.860416+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:18:24.860416+0000","last_undegraded":"2024-09-18T10:18:24.860416+0000","last_fullsized":"2024-09-18T10:18:24.860416+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:18.769125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734817+0000","last_change":"2024-09-18T10:15:10.267445+0000","last_active":"2024-09-18T10:18:23.734817+0000","last_peered":"2024-09-18T10:18:23.734817+0000","last_clean":"2024-09-18T10:18:23.734817+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:18:23.734817+0000","last_undegraded":"2024-09-18T10:18:23.734817+0000","last_fullsized":"2024-09-18T10:18:23.734817+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:02:06.660884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":239,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662357+0000","last_change":"2024-09-18T10:15:12.228339+0000","last_active":"2024-09-18T10:18:24.662357+0000","last_peered":"2024-09-18T10:18:24.662357+0000","last_clean":"2024-09-18T10:18:24.662357+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:18:24.662357+0000","last_undegraded":"2024-09-18T10:18:24.662357+0000","last_fullsized":"2024-09-18T10:18:24.662357+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150004+0000","last_change":"2024-09-18T10:15:10.263155+0000","last_active":"2024-09-18T10:18:24.150004+0000","last_peered":"2024-09-18T10:18:24.150004+0000","last_clean":"2024-09-18T10:18:24.150004+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:18:24.150004+0000","last_undegraded":"2024-09-18T10:18:24.150004+0000","last_fullsized":"2024-09-18T10:18:24.150004+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:11:15.186223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.663037+0000","last_change":"2024-09-18T10:14:59.172586+0000","last_active":"2024-09-18T10:18:24.663037+0000","last_peered":"2024-09-18T10:18:24.663037+0000","last_clean":"2024-09-18T10:18:24.663037+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:18:24.663037+0000","last_undegraded":"2024-09-18T10:18:24.663037+0000","last_fullsized":"2024-09-18T10:18:24.663037+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:50.899590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150325+0000","last_change":"2024-09-18T10:15:10.258203+0000","last_active":"2024-09-18T10:18:24.150325+0000","last_peered":"2024-09-18T10:18:24.150325+0000","last_clean":"2024-09-18T10:18:24.150325+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:18:24.150325+0000","last_undegraded":"2024-09-18T10:18:24.150325+0000","last_fullsized":"2024-09-18T10:18:24.150325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:55.852806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.57399999999999995},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.268},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.61899999999999999,"5min":0.48999999999999999,"15min":0.46100000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.93000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.68200000000000005},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.54100000000000004,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.081},"last":0.56499999999999995}]},{"osd":3,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.88},"min":{"1min":0.70499999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.875},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.82399999999999995,"15min":0.90000000000000002},"min":{"1min":0.73499999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.609},"last":0.873}]},{"osd":4,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.86099999999999999},"min":{"1min":0.76900000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.214,"5min":1.214,"15min":1.242},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.64400000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Wed Sep 18 10:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84799999999999998,"15min":0.878},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.099,"5min":1.099,"15min":1.3740000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.995,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.73899999999999999,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":1.393,"5min":1.393,"15min":1.829},"last":0.83799999999999997}]}]},{"osd":1,"up_from":12,"seq":51539607804,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.49099999999999999,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64600000000000002},"min":{"1min":0.52200000000000002,"5min":0.438,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":1.1359999999999999,"15min":1.528},"last":0.59099999999999997}]},{"osd":2,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.66900000000000004},"min":{"1min":0.56200000000000006,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.91000000000000003,"5min":0.999,"15min":1.0549999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68799999999999994,"15min":0.72099999999999997},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.42599999999999999},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.85299999999999998,"15min":0.85799999999999998},"min":{"1min":0.71399999999999997,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":0.98299999999999998,"5min":1.2589999999999999,"15min":1.377},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.875,"15min":0.84999999999999998},"min":{"1min":0.64700000000000002,"5min":0.627,"15min":0.48899999999999999},"max":{"1min":1.0509999999999999,"5min":1.163,"15min":1.5589999999999999},"last":1.0309999999999999}]},{"osd":4,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.871},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.45100000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.5960000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90300000000000002,"15min":0.90600000000000003},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98099999999999998,"5min":1.3,"15min":1.49},"last":0.879}]},{"osd":5,"last update":"Wed Sep 18 10:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91700000000000004,"15min":0.89700000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.028,"5min":1.423,"15min":1.4590000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.71699999999999997,"5min":0.69299999999999995,"15min":0.53300000000000003},"max":{"1min":1.331,"5min":1.331,"15min":1.639},"last":0.93100000000000005}]}]},{"osd":5,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.78800000000000003},"min":{"1min":0.61099999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98499999999999999,"5min":1.0489999999999999,"15min":1.101},"last":0.505},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86099999999999999,"15min":0.85699999999999998},"min":{"1min":0.73499999999999999,"5min":0.504,"15min":0.504},"max":{"1min":1.125,"5min":1.222,"15min":2.4460000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.52600000000000002},"max":{"1min":1.077,"5min":1.091,"15min":1.4430000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.92100000000000004,"15min":0.877},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.399,"5min":1.399,"15min":2.3799999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.83299999999999996},"min":{"1min":0.60399999999999998,"5min":0.60099999999999998,"15min":0.55700000000000005},"max":{"1min":1.034,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.59799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72199999999999998,"15min":0.72599999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":1.1319999999999999,"15min":1.2210000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.65300000000000002,"5min":0.56699999999999995,"15min":0.45700000000000002},"max":{"1min":1.0169999999999999,"5min":1.3600000000000001,"15min":1.556},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.98699999999999999,"15min":1.0149999999999999},"min":{"1min":0.78700000000000003,"5min":0.69599999999999995,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":1.2370000000000001,"15min":1.653},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73499999999999999,"15min":0.78100000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.88900000000000001,"5min":1.165,"15min":2.327},"last":0.70799999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76800000000000002,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":0.92900000000000005,"5min":1.044,"15min":1.347},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.85199999999999998,"15min":0.85899999999999999},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":1.232}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":1.117}]}]},{"osd":3,"up_from":11,"seq":47244640508,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.627},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.91500000000000004}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":0.84299999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640507,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.59099999999999997},"min":{"1min":0.376,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.83899999999999997,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.64200000000000002},"min":{"1min":0.433,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.96099999999999997},"last":0.60699999999999998}]},{"osd":2,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.63300000000000001,"15min":0.65000000000000002},"min":{"1min":0.40999999999999998,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.86399999999999999,"5min":0.96999999999999997,"15min":1.1459999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.66700000000000004,"15min":0.68400000000000005},"min":{"1min":0.50800000000000001,"5min":0.45100000000000001,"15min":0.41799999999999998},"max":{"1min":0.755,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.92000000000000004,"15min":0.91000000000000003},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0149999999999999,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.96799999999999997,"15min":0.93500000000000005},"min":{"1min":0.83399999999999996,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.0860000000000001,"5min":1.27,"15min":1.347},"last":0.96399999999999997}]},{"osd":4,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85499999999999998,"15min":0.84599999999999997},"min":{"1min":0.68200000000000005,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.0600000000000001,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.871},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.022,"5min":1.0669999999999999,"15min":1.3899999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Wed Sep 18 10:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.878,"15min":0.89800000000000002},"min":{"1min":0.70599999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.099,"5min":1.1950000000000001,"15min":1.333},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.90400000000000003,"15min":0.94899999999999995},"min":{"1min":0.71099999999999997,"5min":0.69899999999999995,"15min":0.64500000000000002},"max":{"1min":1.149,"5min":1.252,"15min":1.446},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:29:40.444 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: get_monmap_and_config 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: build_initial_monmap 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: monmap: 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:40.547900+0000 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:40.547900+0000 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:40.551 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:40.551 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:40.551 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:40.551 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:40.551 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 Processor -- start 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 -- start start 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: init 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8800a0270 0x7fd8800a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): picked mon.noname-c con 0x7fd8800a0270 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:40.554 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd88009cf70 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd880099120 0x7fd8800a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): picked mon.noname-a con 0x7fd880099120 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88009fd50 con 0x7fd88009cf70 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88006a7a0 con 0x7fd880099120 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88009d4d0 con 0x7fd8800a0270 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:40.555 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd884817700 10 monclient(hunting): authenticate will time out at 2069.473300s 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd880099120 0x7fd8800a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87dd9b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fd88009cf70 0x7fd8800a0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:47230/0 (socket says 172.21.15.142:47230) 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87dd9b700 1 -- 172.21.15.142:0/1821552567 learned_addr learned my addr 172.21.15.142:0/1821552567 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): get_auth_request con 0x7fd880099120 auth_method 0 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 1 --2- 172.21.15.142:0/1821552567 >> v2:172.21.15.150:3300/0 conn(0x7fd8800a0270 0x7fd8800a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.556 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): get_auth_request con 0x7fd8800a0270 auth_method 0 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_done global_id 224274 payload 274 2024-09-18T10:29:40.557 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 -- 172.21.15.142:0/1821552567 >> v2:172.21.15.150:3300/0 conn(0x7fd8800a0270 msgr2=0x7fd8800a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 --2- 172.21.15.142:0/1821552567 >> v2:172.21.15.150:3300/0 conn(0x7fd8800a0270 0x7fd8800a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 -- 172.21.15.142:0/1821552567 >> v1:172.21.15.142:6789/0 conn(0x7fd88009cf70 legacy=0x7fd8800a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 monclient: found mon.noname-a 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.547+0000 7fd87ed9d700 1 -- 172.21.15.142:0/1821552567 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88009e040 con 0x7fd880099120 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87ed9d700 1 -- 172.21.15.142:0/1821552567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 msgr2=0x7fd8800a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:40.558 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87ed9d700 1 --2- 172.21.15.142:0/1821552567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd8800a1290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd878009e60 tx=0x7fd878009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbdf44260ea18b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 1 -- 172.21.15.142:0/1821552567 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87800e070 con 0x7fd880099120 2024-09-18T10:29:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: dump: 2024-09-18T10:29:40.564 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: _finish_auth 0 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:40.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 5 monclient: authenticate success, global_id 224274 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 1 -- 172.21.15.142:0/1821552567 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87800bd30 con 0x7fd880099120 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 4 set_mon_vals no callback set 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 1 -- 172.21.15.142:0/1821552567 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87800bea0 con 0x7fd880099120 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd87d59a700 10 monclient: dump: 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.566 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: shutdown 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- 172.21.15.142:0/1821552567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 msgr2=0x7fd8800a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- 172.21.15.142:0/1821552567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd8800a1290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd878009e60 tx=0x7fd878009780 comp rx=0 tx=0).stop 2024-09-18T10:29:40.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- 172.21.15.142:0/1821552567 shutdown_connections 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- 172.21.15.142:0/1821552567 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd8800a1290 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- 172.21.15.142:0/1821552567 >> v2:172.21.15.150:3300/0 conn(0x7fd8800a0270 0x7fd8800a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- 172.21.15.142:0/1821552567 >> 172.21.15.142:0/1821552567 conn(0x7fd880071820 msgr2=0x7fd8800a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- 172.21.15.142:0/1821552567 shutdown_connections 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- 172.21.15.142:0/1821552567 wait complete. 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: build_initial_monmap 2024-09-18T10:29:40.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: monmap: 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:40.569 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 Processor -- start 2024-09-18T10:29:40.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- start start 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: init 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): picked mon.noname-a con 0x7fd880099120 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 0x7fd880138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.571 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): picked mon.noname-b con 0x7fd8800a0270 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 0x7fd8801394b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): picked mon.noname-c con 0x7fd88008f0a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88009bc20 con 0x7fd880099120 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88008e540 con 0x7fd8800a0270 2024-09-18T10:29:40.572 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800975b0 con 0x7fd88008f0a0 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.551+0000 7fd884817700 10 monclient(hunting): authenticate will time out at 2069.479933s 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38562/0 (socket says 172.21.15.142:38562) 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 -- 172.21.15.142:0/4233906408 learned_addr learned my addr 172.21.15.142:0/4233906408 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:40.573 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): get_auth_request con 0x7fd880099120 auth_method 0 2024-09-18T10:29:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 0x7fd8801394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.574 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): get_auth_request con 0x7fd88008f0a0 auth_method 0 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.575 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 0x7fd880138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 10 monclient(hunting): get_auth_request con 0x7fd8800a0270 auth_method 0 2024-09-18T10:29:40.576 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient(hunting): handle_auth_done global_id 224277 payload 824 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 -- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 msgr2=0x7fd8801394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 0x7fd8801394b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 -- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 msgr2=0x7fd880138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 0x7fd880138f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient: _finish_hunting 0 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 monclient: found mon.noname-a 2024-09-18T10:29:40.577 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 -- 172.21.15.142:0/4233906408 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878009a20 con 0x7fd880099120 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87dd9b700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 0x7fd880138f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87ed9d700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 0x7fd8801394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 -- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 msgr2=0x7fd880138a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd87e59c700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd86800ec80 tx=0x7fd8680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86800ca00 con 0x7fd880099120 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: dump: 2024-09-18T10:29:40.578 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:40.579 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 5 monclient: authenticate success, global_id 224277 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86800cb60 con 0x7fd880099120 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 10 monclient: _renew_subs 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800902e0 con 0x7fd880099120 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868013470 con 0x7fd880099120 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd8767fc700 10 monclient: dump: 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:40.580 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd877fff700 4 set_mon_vals no callback set 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 10 monclient: _renew_subs 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.581 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.555+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd880090e60 con 0x7fd880099120 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd868013860 con 0x7fd880099120 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd884817700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd884817700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd884817700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800906e0 con 0x7fd880099120 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd8767fc700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 0x7fd860037ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd8680492b0 con 0x7fd880099120 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd87dd9b700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 0x7fd860037ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:40.582 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.559+0000 7fd87dd9b700 10 monclient: get_auth_request con 0x7fd860035b40 auth_method 0 2024-09-18T10:29:40.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.563+0000 7fd87dd9b700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 0x7fd860037ff0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd88006a5b0 tx=0x7fd870009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:40.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.567+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd8680236b0 con 0x7fd880099120 2024-09-18T10:29:40.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.567+0000 7fd8767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:40.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.567+0000 7fd8767fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:40.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.731+0000 7fd884817700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:40.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.731+0000 7fd884817700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:40.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.731+0000 7fd884817700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:40.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.731+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8800696a0 con 0x7fd880099120 2024-09-18T10:29:40.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd8767fc700 1 -- 172.21.15.142:0/4233906408 <== mon.0 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v264) v1 ==== 74+0+21200 (secure 0 0 0) 0x7fd8680232f0 con 0x7fd880099120 2024-09-18T10:29:40.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd8767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-18T10:29:40.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd8767fc700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:29:40.740 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:40.740 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":264,"fsid":"bfa92b4c-6f79-422f-bfaa-515c68f959fb","created":"2024-09-18T10:08:48.184643+0000","modified":"2024-09-18T10:18:28.413385+0000","last_up_change":"2024-09-18T10:08:59.791843+0000","last_in_change":"2024-09-18T10:13:27.979439+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":182,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-18T10:08:58.836373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-18T10:09:00.830395+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"fooset","create_time":"2024-09-18T10:13:58.026157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":10,"pool_name":"barset","create_time":"2024-09-18T10:13:58.878550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"5aa843cd-ccc8-4450-87dd-76691eac5d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":20975},{"type":"v1","addr":"172.21.15.142:6809","nonce":20975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":20975},{"type":"v1","addr":"172.21.15.142:6811","nonce":20975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":20975},{"type":"v1","addr":"172.21.15.142:6815","nonce":20975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":20975},{"type":"v1","addr":"172.21.15.142:6813","nonce":20975}]},"public_addr":"172.21.15.142:6809/20975","cluster_addr":"172.21.15.142:6811/20975","heartbeat_back_addr":"172.21.15.142:6815/20975","heartbeat_front_addr":"172.21.15.142:6813/20975","state":["exists","up"]},{"osd":1,"uuid":"9bedda37-ff2e-47a0-9fa0-1b62b9b934cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":20974},{"type":"v1","addr":"172.21.15.142:6801","nonce":20974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":20974},{"type":"v1","addr":"172.21.15.142:6803","nonce":20974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":20974},{"type":"v1","addr":"172.21.15.142:6807","nonce":20974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":20974},{"type":"v1","addr":"172.21.15.142:6805","nonce":20974}]},"public_addr":"172.21.15.142:6801/20974","cluster_addr":"172.21.15.142:6803/20974","heartbeat_back_addr":"172.21.15.142:6807/20974","heartbeat_front_addr":"172.21.15.142:6805/20974","state":["exists","up"]},{"osd":2,"uuid":"1097bd95-2d11-48af-b727-df403e6983a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":20976},{"type":"v1","addr":"172.21.15.142:6817","nonce":20976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":20976},{"type":"v1","addr":"172.21.15.142:6819","nonce":20976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":20976},{"type":"v1","addr":"172.21.15.142:6823","nonce":20976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":20976},{"type":"v1","addr":"172.21.15.142:6821","nonce":20976}]},"public_addr":"172.21.15.142:6817/20976","cluster_addr":"172.21.15.142:6819/20976","heartbeat_back_addr":"172.21.15.142:6823/20976","heartbeat_front_addr":"172.21.15.142:6821/20976","state":["exists","up"]},{"osd":3,"uuid":"d9381f04-413d-4382-a0b1-b73fca6b0d11","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":21119},{"type":"v1","addr":"172.21.15.150:6801","nonce":21119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":21119},{"type":"v1","addr":"172.21.15.150:6803","nonce":21119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":21119},{"type":"v1","addr":"172.21.15.150:6807","nonce":21119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":21119},{"type":"v1","addr":"172.21.15.150:6805","nonce":21119}]},"public_addr":"172.21.15.150:6801/21119","cluster_addr":"172.21.15.150:6803/21119","heartbeat_back_addr":"172.21.15.150:6807/21119","heartbeat_front_addr":"172.21.15.150:6805/21119","state":["exists","up"]},{"osd":4,"uuid":"65c1f281-564d-4628-8323-7b9adf16621f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":21118},{"type":"v1","addr":"172.21.15.150:6819","nonce":21118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":21118},{"type":"v1","addr":"172.21.15.150:6821","nonce":21118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":21118},{"type":"v1","addr":"172.21.15.150:6825","nonce":21118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":21118},{"type":"v1","addr":"172.21.15.150:6823","nonce":21118}]},"public_addr":"172.21.15.150:6819/21118","cluster_addr":"172.21.15.150:6821/21118","heartbeat_back_addr":"172.21.15.150:6825/21118","heartbeat_front_addr":"172.21.15.150:6823/21118","state":["exists","up"]},{"osd":5,"uuid":"6d9435f3-4ce9-40fc-90c2-2432f38deacd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":21120},{"type":"v1","addr":"172.21.15.150:6811","nonce":21120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":21120},{"type":"v1","addr":"172.21.15.150:6813","nonce":21120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":21120},{"type":"v1","addr":"172.21.15.150:6817","nonce":21120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":21120},{"type":"v1","addr":"172.21.15.150:6815","nonce":21120}]},"public_addr":"172.21.15.150:6811/21120","cluster_addr":"172.21.15.150:6813/21120","heartbeat_back_addr":"172.21.15.150:6817/21120","heartbeat_front_addr":"172.21.15.150:6815/21120","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"912b7d9d-08f4-4308-9252-03c9211c6cd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.545224+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.358800+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.344702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.380226+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.555695+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T10:08:57.498049+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.5","mappings":[{"from":0,"to":1}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"ec-foo-ssd":{"crush-device-class":"ssd","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 msgr2=0x7fd860037ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 0x7fd860037ff0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd88006a5b0 tx=0x7fd870009040 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 10 monclient: shutdown 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 msgr2=0x7fd880138a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd86800ec80 tx=0x7fd8680091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 shutdown_connections 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd860035b40 0x7fd860037ff0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd88008f0a0 0x7fd8801394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8800a0270 0x7fd880138f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 --2- 172.21.15.142:0/4233906408 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd880099120 0x7fd880138a30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:40.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.735+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 >> 172.21.15.142:0/4233906408 conn(0x7fd880071820 msgr2=0x7fd88009b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:40.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.739+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 shutdown_connections 2024-09-18T10:29:40.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:40.739+0000 7fd884817700 1 -- 172.21.15.142:0/4233906408 wait complete. 2024-09-18T10:29:41.753 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T10:29:41.754 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:41.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 10 monclient: get_monmap_and_config 2024-09-18T10:29:41.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 10 monclient: build_initial_monmap 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 10 monclient: monmap: 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:41.877562+0000 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:41.877562+0000 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:41.880 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 1 Processor -- start 2024-09-18T10:29:41.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 1 -- start start 2024-09-18T10:29:41.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.875+0000 7f5854c6e700 10 monclient: init 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-b con 0x7f5850095d20 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f585009cdf0 0x7f585009d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.884 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-c con 0x7f585009cdf0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f585009d700 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585009ac30 con 0x7f585009d700 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58500960f0 con 0x7f585009cdf0 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585009a600 con 0x7f5850095d20 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5854c6e700 10 monclient(hunting): authenticate will time out at 2070.804656s 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.885 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38106/0 (socket says 172.21.15.142:38106) 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 -- 172.21.15.142:0/52441843 learned_addr learned my addr 172.21.15.142:0/52441843 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): get_auth_request con 0x7f5850095d20 auth_method 0 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 1 --2- 172.21.15.142:0/52441843 >> v2:172.21.15.150:3300/0 conn(0x7f585009cdf0 0x7f585009d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): get_auth_request con 0x7f585009cdf0 auth_method 0 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:41.886 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient(hunting): handle_auth_done global_id 234164 payload 274 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 -- 172.21.15.142:0/52441843 >> v2:172.21.15.150:3300/0 conn(0x7f585009cdf0 msgr2=0x7f585009d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 --2- 172.21.15.142:0/52441843 >> v2:172.21.15.150:3300/0 conn(0x7f585009cdf0 0x7f585009d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 -- 172.21.15.142:0/52441843 >> v1:172.21.15.150:6789/0 conn(0x7f585009d700 legacy=0x7f5850135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 monclient: found mon.noname-b 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:41.887 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.879+0000 7f584f7fe700 1 -- 172.21.15.142:0/52441843 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585009a320 con 0x7f5850095d20 2024-09-18T10:29:41.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584f7fe700 1 -- 172.21.15.142:0/52441843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 msgr2=0x7f5850099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:41.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584f7fe700 1 --2- 172.21.15.142:0/52441843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58400099b0 tx=0x7f5840009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8bc65148e32243b server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 1 -- 172.21.15.142:0/52441843 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840009ea0 con 0x7f5850095d20 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: dump: 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:41.893 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: _finish_auth 0 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 5 monclient: authenticate success, global_id 234164 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 1 -- 172.21.15.142:0/52441843 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584000ba70 con 0x7f5850095d20 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 4 set_mon_vals no callback set 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 1 -- 172.21.15.142:0/52441843 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584000be30 con 0x7f5850095d20 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:41.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f584effd700 10 monclient: dump: 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 10 monclient: shutdown 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:41.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- 172.21.15.142:0/52441843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 msgr2=0x7f5850099af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 --2- 172.21.15.142:0/52441843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f58400099b0 tx=0x7f5840009250 comp rx=0 tx=0).stop 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- 172.21.15.142:0/52441843 shutdown_connections 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 --2- 172.21.15.142:0/52441843 >> v2:172.21.15.150:3300/0 conn(0x7f585009cdf0 0x7f585009d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 --2- 172.21.15.142:0/52441843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5850095d20 0x7f5850099af0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- 172.21.15.142:0/52441843 >> 172.21.15.142:0/52441843 conn(0x7f58500a2400 msgr2=0x7f58500a27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- 172.21.15.142:0/52441843 shutdown_connections 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- 172.21.15.142:0/52441843 wait complete. 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 10 monclient: build_initial_monmap 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 10 monclient: monmap: 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:41.896 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 Processor -- start 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 1 -- start start 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.883+0000 7f5854c6e700 10 monclient: init 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:41.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 0x7f5850070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-a con 0x7f5850095d20 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585009cdf0 0x7f585008ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-b con 0x7f585009cdf0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585009d700 0x7f585008bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): picked mon.noname-c con 0x7f585009d700 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850098b40 con 0x7f5850095d20 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850071af0 con 0x7f585009cdf0 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585008ce50 con 0x7f585009d700 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:41.898 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5854c6e700 10 monclient(hunting): authenticate will time out at 2070.812708s 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585009d700 0x7f585008bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 0x7f5850070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 0x7f5850070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:38570/0 (socket says 172.21.15.142:38570) 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585009d700 0x7f585008bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38118/0 (socket says 172.21.15.142:38118) 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 -- 172.21.15.142:0/4125125017 learned_addr learned my addr 172.21.15.142:0/4125125017 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): get_auth_request con 0x7f5850095d20 auth_method 0 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): get_auth_request con 0x7f585009d700 auth_method 0 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:41.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585009cdf0 0x7f585008ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): get_auth_request con 0x7f585009cdf0 auth_method 0 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:41.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f5846ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient(hunting): handle_auth_done global_id 224280 payload 824 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585009d700 msgr2=0x7f585008bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f585009d700 0x7f585008bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585009cdf0 msgr2=0x7f585008ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585009cdf0 0x7f585008ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 monclient: found mon.noname-a 2024-09-18T10:29:41.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:41.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:41.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.887+0000 7f584f7fe700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400094f0 con 0x7f5850095d20 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f584f7fe700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 msgr2=0x7f5850070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f584f7fe700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 0x7f5850070fe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5840006fd0 tx=0x7f5840016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840006c00 con 0x7f5850095d20 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: dump: 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.905 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: _finish_auth 0 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 5 monclient: authenticate success, global_id 224280 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5840020070 con 0x7f5850095d20 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840011c70 con 0x7f5850095d20 2024-09-18T10:29:41.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 10 monclient: _renew_subs 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585008e030 con 0x7f5850095d20 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f584e7fc700 4 set_mon_vals no callback set 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: dump: 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:41.907 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 10 monclient: _renew_subs 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585008d0d0 con 0x7f5850095d20 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f585008e1f0 con 0x7f5850095d20 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f5840022030 con 0x7f5850095d20 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5830035ac0 0x7f5830037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f584004ab30 con 0x7f5850095d20 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f583003ea40 0x7f5830040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5830041580 con 0x7f583003ea40 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=264) v2 ==== 24+0+0 (secure 0 0 0) 0x7f584004f320 con 0x7f5850095d20 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5847fff700 10 monclient: handle_get_version_reply finishing 1 version 264 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5846ffd700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5830035ac0 0x7f5830037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f5846ffd700 10 monclient: get_auth_request con 0x7f5830035ac0 auth_method 0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f584ffff700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f583003ea40 0x7f5830040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.891+0000 7f584ffff700 10 monclient: get_auth_request con 0x7f583003ea40 auth_method 0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.895+0000 7f5846ffd700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5830035ac0 0x7f5830037f70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5838009db0 tx=0x7f58380091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.895+0000 7f584ffff700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f583003ea40 0x7f5830040eb0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:41.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.899+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== osd.0 v2:172.21.15.142:6808/20975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5830041580 con 0x7f583003ea40 2024-09-18T10:29:41.921 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.915+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 --> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f585008c700 con 0x7f583003ea40 2024-09-18T10:29:41.922 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5847fff700 1 -- 172.21.15.142:0/4125125017 <== osd.0 v2:172.21.15.142:6808/20975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f585008c700 con 0x7f583003ea40 2024-09-18T10:29:41.922 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:41.922 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:41.922 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:41.923 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f583003ea40 msgr2=0x7f5830040eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:6808/20975,v1:172.21.15.142:6809/20975] conn(0x7f583003ea40 0x7f5830040eb0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5830035ac0 msgr2=0x7f5830037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f5830035ac0 0x7f5830037f70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5838009db0 tx=0x7f58380091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 10 monclient: shutdown 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 msgr2=0x7f5850070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 --2- 172.21.15.142:0/4125125017 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5850095d20 0x7f5850070fe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5840006fd0 tx=0x7f5840016000 comp rx=0 tx=0).stop 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f584ffff700 1 -- 172.21.15.142:0/4125125017 reap_dead start 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 shutdown_connections 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 >> 172.21.15.142:0/4125125017 conn(0x7f58500a2400 msgr2=0x7f5850098d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:41.924 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 shutdown_connections 2024-09-18T10:29:41.925 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:41.919+0000 7f5854c6e700 1 -- 172.21.15.142:0/4125125017 wait complete. 2024-09-18T10:29:41.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T10:29:42.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: get_monmap_and_config 2024-09-18T10:29:42.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.051 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: monmap: 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:42.049019+0000 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:42.049019+0000 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:42.052 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:42.053 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 Processor -- start 2024-09-18T10:29:42.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 -- start start 2024-09-18T10:29:42.053 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: init 2024-09-18T10:29:42.057 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-b con 0x7fec6809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fec6809db80 0x7fec6809c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-a con 0x7fec6809db80 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 0x7fec68096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-c con 0x7fec6809c950 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6806fa60 con 0x7fec6809db80 2024-09-18T10:29:42.058 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec680958c0 con 0x7fec6809c950 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec68095ef0 con 0x7fec6809abd0 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6fc17700 10 monclient(hunting): authenticate will time out at 2070.974599s 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fec6809db80 0x7fec6809c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38120/0 (socket says 172.21.15.142:38120) 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 -- 172.21.15.142:0/2497824489 learned_addr learned my addr 172.21.15.142:0/2497824489 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): get_auth_request con 0x7fec6809abd0 auth_method 0 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.059 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): get_auth_request con 0x7fec6809db80 auth_method 0 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 0x7fec68096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.060 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 10 monclient(hunting): get_auth_request con 0x7fec6809c950 auth_method 0 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_done global_id 234170 payload 274 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 -- 172.21.15.142:0/2497824489 >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 msgr2=0x7fec68096870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 0x7fec68096870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 -- 172.21.15.142:0/2497824489 >> v2:172.21.15.142:3300/0 conn(0x7fec6809db80 msgr2=0x7fec6809c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.142:3300/0 conn(0x7fec6809db80 0x7fec6809c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 monclient: found mon.noname-b 2024-09-18T10:29:42.061 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6d9b3700 1 -- 172.21.15.142:0/2497824489 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6809afa0 con 0x7fec6809abd0 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.047+0000 7fec6e1b4700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 0x7fec68096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6d9b3700 1 -- 172.21.15.142:0/2497824489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 msgr2=0x7fec6809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/2497824489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec6806fdb0 tx=0x7fec58009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14cc18fe73c9cc0d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 1 -- 172.21.15.142:0/2497824489 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58009e70 con 0x7fec6809abd0 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: dump: 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.062 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 5 monclient: authenticate success, global_id 234170 2024-09-18T10:29:42.063 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 1 -- 172.21.15.142:0/2497824489 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5800ba70 con 0x7fec6809abd0 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 1 -- 172.21.15.142:0/2497824489 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800be30 con 0x7fec6809abd0 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 4 set_mon_vals no callback set 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6c9b1700 10 monclient: dump: 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.064 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 10 monclient: shutdown 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 -- 172.21.15.142:0/2497824489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 msgr2=0x7fec6809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 --2- 172.21.15.142:0/2497824489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fec6806fdb0 tx=0x7fec58009250 comp rx=0 tx=0).stop 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 -- 172.21.15.142:0/2497824489 shutdown_connections 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.150:3300/0 conn(0x7fec6809c950 0x7fec68096870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.065 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 --2- 172.21.15.142:0/2497824489 >> v2:172.21.15.142:3300/0 conn(0x7fec6809db80 0x7fec6809c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 --2- 172.21.15.142:0/2497824489 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec6809bed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 -- 172.21.15.142:0/2497824489 >> 172.21.15.142:0/2497824489 conn(0x7fec680a14e0 msgr2=0x7fec680a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 -- 172.21.15.142:0/2497824489 shutdown_connections 2024-09-18T10:29:42.066 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.051+0000 7fec6fc17700 1 -- 172.21.15.142:0/2497824489 wait complete. 2024-09-18T10:29:42.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.072 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient: monmap: 2024-09-18T10:29:42.072 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.073 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 Processor -- start 2024-09-18T10:29:42.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 -- start start 2024-09-18T10:29:42.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient: init 2024-09-18T10:29:42.076 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-c con 0x7fec6809abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec6809c950 0x7fec68141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-a con 0x7fec6809c950 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec6809db80 0x7fec681421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): picked mon.noname-b con 0x7fec6809db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6809a5a0 con 0x7fec6809c950 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6809a370 con 0x7fec6809db80 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec68097430 con 0x7fec6809abd0 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6fc17700 10 monclient(hunting): authenticate will time out at 2070.982861s 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:38134/0 (socket says 172.21.15.142:38134) 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 -- 172.21.15.142:0/518479886 learned_addr learned my addr 172.21.15.142:0/518479886 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): get_auth_request con 0x7fec6809abd0 auth_method 0 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.078 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec6809c950 0x7fec68141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): get_auth_request con 0x7fec6809c950 auth_method 0 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec6809db80 0x7fec681421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): get_auth_request con 0x7fec6809db80 auth_method 0 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.079 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d1b2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6e1b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient(hunting): handle_auth_done global_id 234173 payload 824 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 -- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec6809db80 msgr2=0x7fec681421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec6809db80 0x7fec681421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 -- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec6809c950 msgr2=0x7fec68141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.080 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec6809c950 0x7fec68141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 monclient: found mon.noname-c 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.055+0000 7fec6d9b3700 1 -- 172.21.15.142:0/518479886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec580094f0 con 0x7fec6809abd0 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6d9b3700 1 -- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 msgr2=0x7fec68141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6d9b3700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec68096030 tx=0x7fec58017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58006a90 con 0x7fec6809abd0 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.081 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: dump: 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.082 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec580118c0 con 0x7fec6809abd0 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 5 monclient: authenticate success, global_id 234173 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec58011c80 con 0x7fec6809abd0 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _renew_subs 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec67fff700 4 set_mon_vals no callback set 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec68090300 con 0x7fec6809abd0 2024-09-18T10:29:42.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 10 monclient: dump: 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.084 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _renew_subs 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec680914b0 con 0x7fec6809abd0 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fec58023070 con 0x7fec6809abd0 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.059+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec680696a0 con 0x7fec6809abd0 2024-09-18T10:29:42.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.063+0000 7fec65ffb700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 0x7fec54037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.063+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fec5804a120 con 0x7fec6809abd0 2024-09-18T10:29:42.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.063+0000 7fec6d1b2700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 0x7fec54037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.063+0000 7fec6d1b2700 10 monclient: get_auth_request con 0x7fec54035ac0 auth_method 0 2024-09-18T10:29:42.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.063+0000 7fec6d1b2700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 0x7fec54037f70 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fec68097610 tx=0x7fec5c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.071+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fec58023320 con 0x7fec6809abd0 2024-09-18T10:29:42.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.071+0000 7fec65ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.071+0000 7fec65ffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:42.236 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.231+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fec68091960 con 0x7fec54035ac0 2024-09-18T10:29:42.237 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.231+0000 7fec65ffb700 1 -- 172.21.15.142:0/518479886 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fec68091960 con 0x7fec54035ac0 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 msgr2=0x7fec54037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 0x7fec54037f70 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fec68097610 tx=0x7fec5c00b040 comp rx=0 tx=0).stop 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 10 monclient: shutdown 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 msgr2=0x7fec68141760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec68096030 tx=0x7fec58017000 comp rx=0 tx=0).stop 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 shutdown_connections 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fec54035ac0 0x7fec54037f70 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec6809db80 0x7fec681421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fec6809c950 0x7fec68141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.240 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 --2- 172.21.15.142:0/518479886 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fec6809abd0 0x7fec68141760 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 >> 172.21.15.142:0/518479886 conn(0x7fec680a14e0 msgr2=0x7fec6809fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 shutdown_connections 2024-09-18T10:29:42.241 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.235+0000 7fec6fc17700 1 -- 172.21.15.142:0/518479886 wait complete. 2024-09-18T10:29:42.241 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T10:29:42.250 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T10:29:42.251 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:42.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: get_monmap_and_config 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: monmap: 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:42.361518+0000 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:42.361518+0000 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:42.364 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:42.365 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 Processor -- start 2024-09-18T10:29:42.369 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 -- start start 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: init 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc28132260 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdc2809ee30 0x7fdc2809f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-a con 0x7fdc2809ee30 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-b con 0x7fdc28137360 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc280a1530 con 0x7fdc28132260 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.370 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc28132d40 con 0x7fdc2809ee30 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc280a00f0 con 0x7fdc28137360 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2db9e700 10 monclient(hunting): authenticate will time out at 2071.287466s 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdc2809ee30 0x7fdc2809f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35384/0 (socket says 172.21.15.142:35384) 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 1 -- 172.21.15.142:0/1820845647 learned_addr learned my addr 172.21.15.142:0/1820845647 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): get_auth_request con 0x7fdc2809ee30 auth_method 0 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.371 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): get_auth_request con 0x7fdc28137360 auth_method 0 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc2d39d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.359+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 10 monclient(hunting): handle_auth_done global_id 234176 payload 274 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 -- 172.21.15.142:0/1820845647 >> v2:172.21.15.142:3300/0 conn(0x7fdc2809ee30 msgr2=0x7fdc2809f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.372 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/1820845647 >> v2:172.21.15.142:3300/0 conn(0x7fdc2809ee30 0x7fdc2809f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 -- 172.21.15.142:0/1820845647 >> v1:172.21.15.150:6789/0 conn(0x7fdc28132260 legacy=0x7fdc28132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 monclient: found mon.noname-b 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 -- 172.21.15.142:0/1820845647 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280a1b60 con 0x7fdc28137360 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 -- 172.21.15.142:0/1820845647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 msgr2=0x7fdc28139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/1820845647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009f20 tx=0x7fdc20009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c14807fbbb434e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 1 -- 172.21.15.142:0/1820845647 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2000e070 con 0x7fdc28137360 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:42.373 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: dump: 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 5 monclient: authenticate success, global_id 234176 2024-09-18T10:29:42.374 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 1 -- 172.21.15.142:0/1820845647 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2000bbe0 con 0x7fdc28137360 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 4 set_mon_vals no callback set 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 1 -- 172.21.15.142:0/1820845647 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc200092e0 con 0x7fdc28137360 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc277fe700 10 monclient: dump: 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.375 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 10 monclient: shutdown 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 -- 172.21.15.142:0/1820845647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 msgr2=0x7fdc28139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/1820845647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009f20 tx=0x7fdc20009840 comp rx=0 tx=0).stop 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 -- 172.21.15.142:0/1820845647 shutdown_connections 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/1820845647 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28137360 0x7fdc28139740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/1820845647 >> v2:172.21.15.142:3300/0 conn(0x7fdc2809ee30 0x7fdc2809f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.376 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 -- 172.21.15.142:0/1820845647 >> 172.21.15.142:0/1820845647 conn(0x7fdc280a5700 msgr2=0x7fdc280a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 -- 172.21.15.142:0/1820845647 shutdown_connections 2024-09-18T10:29:42.377 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.363+0000 7fdc2db9e700 1 -- 172.21.15.142:0/1820845647 wait complete. 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient: monmap: 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:42.382 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 Processor -- start 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 -- start start 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient: init 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 0x7fdc2808ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-a con 0x7fdc2809ee30 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28132260 0x7fdc2806d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-c con 0x7fdc28132260 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.383 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc28137360 0x7fdc2806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): picked mon.noname-b con 0x7fdc28137360 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc28089430 con 0x7fdc2809ee30 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2809e7b0 con 0x7fdc28137360 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2809fad0 con 0x7fdc28132260 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2db9e700 10 monclient(hunting): authenticate will time out at 2071.293531s 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 0x7fdc2808ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.384 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 0x7fdc2808ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34116/0 (socket says 172.21.15.142:34116) 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 -- 172.21.15.142:0/2700778718 learned_addr learned my addr 172.21.15.142:0/2700778718 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): get_auth_request con 0x7fdc2809ee30 auth_method 0 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28132260 0x7fdc2806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): get_auth_request con 0x7fdc28132260 auth_method 0 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.385 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc28137360 0x7fdc2806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 10 monclient(hunting): get_auth_request con 0x7fdc28137360 auth_method 0 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient(hunting): handle_auth_done global_id 224292 payload 824 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28132260 msgr2=0x7fdc2806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.386 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdc28132260 0x7fdc2806d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc28137360 msgr2=0x7fdc2806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc28137360 0x7fdc2806de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 monclient: found mon.noname-a 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2cb9c700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20009ae0 con 0x7fdc2809ee30 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.367+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc28137360 0x7fdc2806de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2cb9c700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 msgr2=0x7fdc2808ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2cb9c700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 0x7fdc2808ba10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00c600 tx=0x7fdc1c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0117a0 con 0x7fdc2809ee30 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.387 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: dump: 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.388 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 5 monclient: authenticate success, global_id 224292 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 10 monclient: _renew_subs 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc280899f0 con 0x7fdc2809ee30 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c011900 con 0x7fdc2809ee30 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c011cc0 con 0x7fdc2809ee30 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.389 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: dump: 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.390 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 10 monclient: _renew_subs 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc26ffd700 4 set_mon_vals no callback set 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2806f3c0 con 0x7fdc2809ee30 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdc1c012630 con 0x7fdc2809ee30 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc28072a80 con 0x7fdc2809ee30 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc14035af0 0x7fdc14037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdc1c048a90 con 0x7fdc2809ee30 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7fdc1403ebe0 0x7fdc14041050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc14041720 con 0x7fdc1403ebe0 2024-09-18T10:29:42.391 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=264) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc1c01c460 con 0x7fdc2809ee30 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc24ff9700 10 monclient: handle_get_version_reply finishing 1 version 264 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc27fff700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc14035af0 0x7fdc14037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc27fff700 10 monclient: get_auth_request con 0x7fdc14035af0 auth_method 0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7fdc1403ebe0 0x7fdc14041050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.371+0000 7fdc2d39d700 10 monclient: get_auth_request con 0x7fdc1403ebe0 auth_method 0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.375+0000 7fdc27fff700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc14035af0 0x7fdc14037fa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdc2806f090 tx=0x7fdc180091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.379+0000 7fdc2d39d700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7fdc1403ebe0 0x7fdc14041050 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.392 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.379+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== osd.1 v2:172.21.15.142:6800/20974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdc14041720 con 0x7fdc1403ebe0 2024-09-18T10:29:42.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.395+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 --> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdc2808bf50 con 0x7fdc1403ebe0 2024-09-18T10:29:42.401 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc24ff9700 1 -- 172.21.15.142:0/2700778718 <== osd.1 v2:172.21.15.142:6800/20974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdc2808bf50 con 0x7fdc1403ebe0 2024-09-18T10:29:42.402 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:42.402 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:42.402 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7fdc1403ebe0 msgr2=0x7fdc14041050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:6800/20974,v1:172.21.15.142:6801/20974] conn(0x7fdc1403ebe0 0x7fdc14041050 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc14035af0 msgr2=0x7fdc14037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdc14035af0 0x7fdc14037fa0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdc2806f090 tx=0x7fdc180091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 10 monclient: shutdown 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 msgr2=0x7fdc2808ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 --2- 172.21.15.142:0/2700778718 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc2809ee30 0x7fdc2808ba10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00c600 tx=0x7fdc1c007f60 comp rx=0 tx=0).stop 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2d39d700 1 -- 172.21.15.142:0/2700778718 reap_dead start 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 shutdown_connections 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 >> 172.21.15.142:0/2700778718 conn(0x7fdc280a5700 msgr2=0x7fdc2809cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.403 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 shutdown_connections 2024-09-18T10:29:42.404 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.399+0000 7fdc2db9e700 1 -- 172.21.15.142:0/2700778718 wait complete. 2024-09-18T10:29:42.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: get_monmap_and_config 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: monmap: 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:42.525858+0000 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:42.525858+0000 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.528 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:42.529 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 1 Processor -- start 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 1 -- start start 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: init 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d6009cdd0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.523+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d60134a50 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9d60138550 0x7f9d6013a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-c con 0x7f9d60138550 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:42.532 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6009d1a0 con 0x7f9d60134a50 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6009d780 con 0x7f9d6009cdd0 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6009dd40 con 0x7f9d60138550 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d66d9d700 10 monclient(hunting): authenticate will time out at 2071.452114s 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d5ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9d60134a50 0x7f9d60136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40280/0 (socket says 172.21.15.142:40280) 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d5ffff700 1 -- 172.21.15.142:0/1838259539 learned_addr learned my addr 172.21.15.142:0/1838259539 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 --2- 172.21.15.142:0/1838259539 >> v2:172.21.15.150:3300/0 conn(0x7f9d60138550 0x7f9d6013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): get_auth_request con 0x7f9d60138550 auth_method 0 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.533 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient(hunting): handle_auth_done global_id 194650 payload 274 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 -- 172.21.15.142:0/1838259539 >> v1:172.21.15.150:6789/0 conn(0x7f9d6009cdd0 legacy=0x7f9d6009e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 -- 172.21.15.142:0/1838259539 >> v1:172.21.15.142:6789/0 conn(0x7f9d60134a50 legacy=0x7f9d60136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 monclient: found mon.noname-c 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:29:42.534 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 -- 172.21.15.142:0/1838259539 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600a6710 con 0x7f9d60138550 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 -- 172.21.15.142:0/1838259539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60138550 msgr2=0x7f9d6013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.527+0000 7f9d6533a700 1 --2- 172.21.15.142:0/1838259539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60138550 0x7f9d6013a930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009e60 tx=0x7f9d58009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a53193ab9ee828b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 1 -- 172.21.15.142:0/1838259539 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800e070 con 0x7f9d60138550 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: dump: 2024-09-18T10:29:42.536 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 5 monclient: authenticate success, global_id 194650 2024-09-18T10:29:42.537 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 1 -- 172.21.15.142:0/1838259539 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d58012430 con 0x7f9d60138550 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 4 set_mon_vals no callback set 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 1 -- 172.21.15.142:0/1838259539 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d580127d0 con 0x7f9d60138550 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d5f7fe700 10 monclient: dump: 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.538 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 10 monclient: shutdown 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1838259539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60138550 msgr2=0x7f9d6013a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1838259539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60138550 0x7f9d6013a930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009e60 tx=0x7f9d58009780 comp rx=0 tx=0).stop 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1838259539 shutdown_connections 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1838259539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60138550 0x7f9d6013a930 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1838259539 >> 172.21.15.142:0/1838259539 conn(0x7f9d60071680 msgr2=0x7f9d600a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1838259539 shutdown_connections 2024-09-18T10:29:42.539 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1838259539 wait complete. 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 10 monclient: monmap: 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:42.540 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 Processor -- start 2024-09-18T10:29:42.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 1 -- start start 2024-09-18T10:29:42.544 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.531+0000 7f9d66d9d700 10 monclient: init 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 0x7f9d6006f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-a con 0x7f9d6009cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60134a50 0x7f9d6006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-b con 0x7f9d60134a50 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): picked mon.noname-c con 0x7f9d60138550 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d600a3320 con 0x7f9d6009cdd0 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6006bcc0 con 0x7f9d60134a50 2024-09-18T10:29:42.545 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6006d550 con 0x7f9d60138550 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient(hunting): authenticate will time out at 2071.460352s 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 0x7f9d6006f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35406/0 (socket says 172.21.15.142:35406) 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 1 -- 172.21.15.142:0/1525908076 learned_addr learned my addr 172.21.15.142:0/1525908076 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): get_auth_request con 0x7f9d60138550 auth_method 0 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): get_auth_request con 0x7f9d6009cdd0 auth_method 0 2024-09-18T10:29:42.546 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60134a50 0x7f9d6006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): get_auth_request con 0x7f9d60134a50 auth_method 0 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.547 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient(hunting): handle_auth_done global_id 224295 payload 824 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 -- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 msgr2=0x7f9d6006b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 -- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60134a50 msgr2=0x7f9d6006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60134a50 0x7f9d6006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 monclient: found mon.noname-a 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.548 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 -- 172.21.15.142:0/1525908076 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d58009a20 con 0x7f9d6009cdd0 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d6533a700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 -- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 msgr2=0x7f9d6006f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d64b39700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 0x7f9d6006f610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00c510 tx=0x7f9d4c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00ce80 con 0x7f9d6009cdd0 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: dump: 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.549 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 5 monclient: authenticate success, global_id 224295 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0117a0 con 0x7f9d6009cdd0 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _renew_subs 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6006d830 con 0x7f9d6009cdd0 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.550 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c011b60 con 0x7f9d6009cdd0 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5e7fc700 4 set_mon_vals no callback set 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d5cff9700 10 monclient: dump: 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.551 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _renew_subs 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.535+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6006dbc0 con 0x7f9d6009cdd0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d66d9d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d66d9d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d66d9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d600696a0 con 0x7f9d6009cdd0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9d4c01c460 con 0x7f9d6009cdd0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d5cff9700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 0x7f9d44037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9d4c048b70 con 0x7f9d6009cdd0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d5ffff700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 0x7f9d44037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.552 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.539+0000 7f9d5ffff700 10 monclient: get_auth_request con 0x7f9d44035ac0 auth_method 0 2024-09-18T10:29:42.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.543+0000 7f9d5ffff700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 0x7f9d44037f70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d60137c10 tx=0x7f9d54007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.547+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9d4c002090 con 0x7f9d6009cdd0 2024-09-18T10:29:42.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.547+0000 7f9d5cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:42.553 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.547+0000 7f9d5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:42.712 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.707+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d6006d9c0 con 0x7f9d44035ac0 2024-09-18T10:29:42.713 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.707+0000 7f9d5cff9700 1 -- 172.21.15.142:0/1525908076 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9d6006d9c0 con 0x7f9d44035ac0 2024-09-18T10:29:42.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 msgr2=0x7f9d44037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 0x7f9d44037f70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9d60137c10 tx=0x7f9d54007f90 comp rx=0 tx=0).stop 2024-09-18T10:29:42.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 10 monclient: shutdown 2024-09-18T10:29:42.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.716 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 msgr2=0x7f9d6006f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 0x7f9d6006f610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00c510 tx=0x7f9d4c007ea0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 shutdown_connections 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9d44035ac0 0x7f9d44037f70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d60138550 0x7f9d6006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d60134a50 0x7f9d6006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 --2- 172.21.15.142:0/1525908076 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d6009cdd0 0x7f9d6006f610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 >> 172.21.15.142:0/1525908076 conn(0x7f9d60071680 msgr2=0x7f9d60139f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 shutdown_connections 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.711+0000 7f9d66d9d700 1 -- 172.21.15.142:0/1525908076 wait complete. 2024-09-18T10:29:42.717 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T10:29:42.728 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T10:29:42.728 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: get_monmap_and_config 2024-09-18T10:29:42.838 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: monmap: 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:42.836617+0000 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:42.836617+0000 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:42.839 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.841 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 Processor -- start 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 -- start start 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: init 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f887406a740 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f887412ef00 0x7f887412f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): picked mon.noname-c con 0x7f887412ef00 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f887412f840 0x7f88741363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): picked mon.noname-a con 0x7f887412f840 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88740972d0 con 0x7f887406a740 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887406da40 con 0x7f887412f840 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.842 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887406e870 con 0x7f887412ef00 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f8879276700 10 monclient(hunting): authenticate will time out at 2071.762496s 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f887412f840 0x7f88741363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f887412f840 0x7f88741363e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34132/0 (socket says 172.21.15.142:34132) 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 -- 172.21.15.142:0/3587511839 learned_addr learned my addr 172.21.15.142:0/3587511839 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): get_auth_request con 0x7f887412f840 auth_method 0 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 1 --2- 172.21.15.142:0/3587511839 >> v2:172.21.15.150:3300/0 conn(0x7f887412ef00 0x7f887412f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.843 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): get_auth_request con 0x7f887412ef00 auth_method 0 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient(hunting): handle_auth_done global_id 224298 payload 274 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 -- 172.21.15.142:0/3587511839 >> v2:172.21.15.150:3300/0 conn(0x7f887412ef00 msgr2=0x7f887412f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 --2- 172.21.15.142:0/3587511839 >> v2:172.21.15.150:3300/0 conn(0x7f887412ef00 0x7f887412f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 -- 172.21.15.142:0/3587511839 >> v1:172.21.15.150:6789/0 conn(0x7f887406a740 legacy=0x7f8874096bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 monclient: found mon.noname-a 2024-09-18T10:29:42.844 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.845 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.835+0000 7f88737fe700 1 -- 172.21.15.142:0/3587511839 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887406e590 con 0x7f887412f840 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88737fe700 1 -- 172.21.15.142:0/3587511839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412f840 msgr2=0x7f88741363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88737fe700 1 --2- 172.21.15.142:0/3587511839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412f840 0x7f88741363e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f887409e1f0 tx=0x7f886c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326673687eac8cb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 1 -- 172.21.15.142:0/3587511839 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c00e070 con 0x7f887412f840 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: dump: 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.849 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 5 monclient: authenticate success, global_id 224298 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 1 -- 172.21.15.142:0/3587511839 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c00bc30 con 0x7f887412f840 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:42.850 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 4 set_mon_vals no callback set 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 1 -- 172.21.15.142:0/3587511839 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0092e0 con 0x7f887412f840 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f88727fc700 10 monclient: dump: 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.851 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 10 monclient: shutdown 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 -- 172.21.15.142:0/3587511839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412f840 msgr2=0x7f88741363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 --2- 172.21.15.142:0/3587511839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412f840 0x7f88741363e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f887409e1f0 tx=0x7f886c009840 comp rx=0 tx=0).stop 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 -- 172.21.15.142:0/3587511839 shutdown_connections 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 --2- 172.21.15.142:0/3587511839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412f840 0x7f88741363e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 --2- 172.21.15.142:0/3587511839 >> v2:172.21.15.150:3300/0 conn(0x7f887412ef00 0x7f887412f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 -- 172.21.15.142:0/3587511839 >> 172.21.15.142:0/3587511839 conn(0x7f8874071b20 msgr2=0x7f88740a1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 -- 172.21.15.142:0/3587511839 shutdown_connections 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 1 -- 172.21.15.142:0/3587511839 wait complete. 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 10 monclient: build_initial_monmap 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.839+0000 7f8879276700 10 monclient: monmap: 2024-09-18T10:29:42.852 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 Processor -- start 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 -- start start 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient: init 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:29:42.853 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f887412ef00 0x7f887412ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): picked mon.noname-b con 0x7f887412ef00 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f887412f840 0x7f8874119860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): picked mon.noname-c con 0x7f887412f840 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 0x7f8874119da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): picked mon.noname-a con 0x7f887412e300 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887411be60 con 0x7f887412e300 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887411d550 con 0x7f887412ef00 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887406de00 con 0x7f887412f840 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:42.854 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8879276700 10 monclient(hunting): authenticate will time out at 2071.769123s 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 0x7f8874119da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f887412f840 0x7f8874119860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 0x7f8874119da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34146/0 (socket says 172.21.15.142:34146) 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 learned_addr learned my addr 172.21.15.142:0/3692115398 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): get_auth_request con 0x7f887412e300 auth_method 0 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): get_auth_request con 0x7f887412f840 auth_method 0 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f887412ef00 0x7f887412ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.855 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): get_auth_request con 0x7f887412ef00 auth_method 0 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f886bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f8872ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:42.856 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient(hunting): handle_auth_done global_id 224301 payload 824 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f887412f840 msgr2=0x7f8874119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f887412f840 0x7f8874119860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f887412ef00 msgr2=0x7f887412ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f887412ef00 0x7f887412ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 monclient: found mon.noname-a 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.857 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.843+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c009ae0 con 0x7f887412e300 2024-09-18T10:29:42.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 msgr2=0x7f8874119da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:42.859 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f88737fe700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 0x7f8874119da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f886c00b6c0 tx=0x7f886c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c0092e0 con 0x7f887412e300 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: dump: 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.860 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: _finish_auth 0 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 5 monclient: authenticate success, global_id 224301 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c0070e0 con 0x7f887412e300 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c01c660 con 0x7f887412e300 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 10 monclient: _renew_subs 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88740a0db0 con 0x7f887412e300 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f88717fa700 4 set_mon_vals no callback set 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:42.861 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: dump: 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 10 monclient: _renew_subs 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887411b950 con 0x7f887412e300 2024-09-18T10:29:42.862 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88740696a0 con 0x7f887412e300 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f886c019070 con 0x7f887412e300 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8854035af0 0x7f8854037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f886c04a340 con 0x7f887412e300 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7f885403ec60 0x7f88540410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88540417a0 con 0x7f885403ec60 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=264) v2 ==== 24+0+0 (secure 0 0 0) 0x7f886c04c8d0 con 0x7f887412e300 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886affd700 10 monclient: handle_get_version_reply finishing 1 version 264 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8872ffd700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8854035af0 0x7f8854037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f8872ffd700 10 monclient: get_auth_request con 0x7f8854035af0 auth_method 0 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886bfff700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7f885403ec60 0x7f88540410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.847+0000 7f886bfff700 10 monclient: get_auth_request con 0x7f885403ec60 auth_method 0 2024-09-18T10:29:42.863 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.851+0000 7f8872ffd700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8854035af0 0x7f8854037fa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8864003a30 tx=0x7f8864007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.855+0000 7f886bfff700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7f885403ec60 0x7f88540410d0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:42.864 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.855+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== osd.2 v2:172.21.15.142:6816/20976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f88540417a0 con 0x7f885403ec60 2024-09-18T10:29:42.878 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 --> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f887411b440 con 0x7f885403ec60 2024-09-18T10:29:42.879 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f886affd700 1 -- 172.21.15.142:0/3692115398 <== osd.2 v2:172.21.15.142:6816/20976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f887411b440 con 0x7f885403ec60 2024-09-18T10:29:42.879 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:42.879 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:42.879 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:42.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7f885403ec60 msgr2=0x7f88540410d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.880 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:6816/20976,v1:172.21.15.142:6817/20976] conn(0x7f885403ec60 0x7f88540410d0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8854035af0 msgr2=0x7f8854037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f8854035af0 0x7f8854037fa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8864003a30 tx=0x7f8864007ea0 comp rx=0 tx=0).stop 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 10 monclient: shutdown 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 msgr2=0x7f8874119da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 --2- 172.21.15.142:0/3692115398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f887412e300 0x7f8874119da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f886c00b6c0 tx=0x7f886c017000 comp rx=0 tx=0).stop 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f88737fe700 1 -- 172.21.15.142:0/3692115398 reap_dead start 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 shutdown_connections 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 >> 172.21.15.142:0/3692115398 conn(0x7f887406a740 msgr2=0x7f887409e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 shutdown_connections 2024-09-18T10:29:42.881 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.875+0000 7f8879276700 1 -- 172.21.15.142:0/3692115398 wait complete. 2024-09-18T10:29:42.892 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T10:29:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: get_monmap_and_config 2024-09-18T10:29:43.002 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: monmap: 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:43.000493+0000 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:43.000493+0000 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:43.003 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 Processor -- start 2024-09-18T10:29:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 -- start start 2024-09-18T10:29:43.005 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: init 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e0809c2c0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e08134880 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0813a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-b con 0x7f9e08138380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080a6620 con 0x7f9e08134880 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080a1c70 con 0x7f9e0809c2c0 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0809ab70 con 0x7f9e08138380 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0eaaa700 10 monclient(hunting): authenticate will time out at 2071.926419s 2024-09-18T10:29:43.006 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0d2a7700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9e08134880 0x7f9e08136c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40282/0 (socket says 172.21.15.142:40282) 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0d2a7700 1 -- 172.21.15.142:0/1419062888 learned_addr learned my addr 172.21.15.142:0/1419062888 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 1 --2- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0813a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): get_auth_request con 0x7f9e08138380 auth_method 0 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_done global_id 234188 payload 274 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/1419062888 >> v1:172.21.15.150:6789/0 conn(0x7f9e0809c2c0 legacy=0x7f9e080999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.007 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/1419062888 >> v1:172.21.15.142:6789/0 conn(0x7f9e08134880 legacy=0x7f9e08136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 1 monclient: found mon.noname-b 2024-09-18T10:29:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.008 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:42.999+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/1419062888 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0809c690 con 0x7f9e08138380 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 msgr2=0x7f9e0813a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0e2a9700 1 --2- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0813a760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009910 tx=0x7f9e000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611b931a2e407f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 1 -- 172.21.15.142:0/1419062888 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0000ba70 con 0x7f9e08138380 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: dump: 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.010 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 1 -- 172.21.15.142:0/1419062888 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0000bbd0 con 0x7f9e08138380 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 5 monclient: authenticate success, global_id 234188 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:43.011 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 1 -- 172.21.15.142:0/1419062888 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00011420 con 0x7f9e08138380 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 4 set_mon_vals no callback set 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0caa6700 10 monclient: dump: 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.012 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 10 monclient: shutdown 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 msgr2=0x7f9e0813a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0813a760 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009910 tx=0x7f9e000091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/1419062888 shutdown_connections 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/1419062888 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0813a760 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/1419062888 >> 172.21.15.142:0/1419062888 conn(0x7f9e080a5720 msgr2=0x7f9e080a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/1419062888 shutdown_connections 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/1419062888 wait complete. 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.003+0000 7f9e0eaaa700 10 monclient: monmap: 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:43.013 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 Processor -- start 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 -- start start 2024-09-18T10:29:43.014 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient: init 2024-09-18T10:29:43.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:43.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-a con 0x7f9e0809c2c0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:43.018 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e08134880 0x7f9e0806b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-b con 0x7f9e08134880 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): picked mon.noname-c con 0x7f9e08138380 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0806e8e0 con 0x7f9e0809c2c0 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0806f5f0 con 0x7f9e08134880 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0806d8e0 con 0x7f9e08138380 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0eaaa700 10 monclient(hunting): authenticate will time out at 2071.933310s 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.019 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35422/0 (socket says 172.21.15.142:35422) 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34158/0 (socket says 172.21.15.142:34158) 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/2351823540 learned_addr learned my addr 172.21.15.142:0/2351823540 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): get_auth_request con 0x7f9e08138380 auth_method 0 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): get_auth_request con 0x7f9e0809c2c0 auth_method 0 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e08134880 0x7f9e0806b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): get_auth_request con 0x7f9e08134880 auth_method 0 2024-09-18T10:29:43.020 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0d2a7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient(hunting): handle_auth_done global_id 234191 payload 824 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e08134880 msgr2=0x7f9e0806b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.021 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e08134880 0x7f9e0806b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 msgr2=0x7f9e0809f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 monclient: found mon.noname-c 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/2351823540 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00009420 con 0x7f9e08138380 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.007+0000 7f9e0daa8700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0e2a9700 1 -- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 msgr2=0x7f9e0806bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0e2a9700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e000093f0 tx=0x7f9e00016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00011e10 con 0x7f9e08138380 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.022 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: dump: 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.023 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e00020070 con 0x7f9e08138380 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 5 monclient: authenticate success, global_id 234191 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _renew_subs 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0001b8b0 con 0x7f9e08138380 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0806e600 con 0x7f9e08138380 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 10 monclient: dump: 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.024 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfffff700 4 set_mon_vals no callback set 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _renew_subs 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0806cb10 con 0x7f9e08138380 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08072a80 con 0x7f9e08138380 2024-09-18T10:29:43.025 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f9e00022030 con 0x7f9e08138380 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 0x7f9df4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f9e0004aca0 con 0x7f9e08138380 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0daa8700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 0x7f9df4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.011+0000 7f9e0daa8700 10 monclient: get_auth_request con 0x7f9df4035ac0 auth_method 0 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.015+0000 7f9e0daa8700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 0x7f9df4037f70 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9df8002e00 tx=0x7f9df8007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.019+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f9e00010980 con 0x7f9e08138380 2024-09-18T10:29:43.026 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.019+0000 7f9dfdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.027 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.019+0000 7f9dfdffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:43.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.187+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e08072a80 con 0x7f9df4035ac0 2024-09-18T10:29:43.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.187+0000 7f9dfdffb700 1 -- 172.21.15.142:0/2351823540 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9e08072a80 con 0x7f9df4035ac0 2024-09-18T10:29:43.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 msgr2=0x7f9df4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 0x7f9df4037f70 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9df8002e00 tx=0x7f9df8007ea0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.196 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 10 monclient: shutdown 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 msgr2=0x7f9e0806bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e000093f0 tx=0x7f9e00016000 comp rx=0 tx=0).stop 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 shutdown_connections 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f9df4035ac0 0x7f9df4037f70 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e08138380 0x7f9e0806bf30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e08134880 0x7f9e0806b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 --2- 172.21.15.142:0/2351823540 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e0809c2c0 0x7f9e0809f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 >> 172.21.15.142:0/2351823540 conn(0x7f9e080a5720 msgr2=0x7f9e080a4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 shutdown_connections 2024-09-18T10:29:43.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.191+0000 7f9e0eaaa700 1 -- 172.21.15.142:0/2351823540 wait complete. 2024-09-18T10:29:43.198 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T10:29:43.212 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T10:29:43.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: get_monmap_and_config 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: monmap: 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:43.316943+0000 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:43.316943+0000 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:43.319 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:43.320 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:43.320 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:43.320 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:43.320 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 Processor -- start 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 -- start start 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: init 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60901320a0 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:43.322 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f609009f4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f609009f8e0 0x7f609013ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): picked mon.noname-c con 0x7f609009f8e0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f609009d090 con 0x7f609009f4f0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900a16b0 con 0x7f60901320a0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6090132b80 con 0x7f609009f8e0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6098651700 10 monclient(hunting): authenticate will time out at 2072.243149s 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6095bec700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f609009f4f0 0x7f60901373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40296/0 (socket says 172.21.15.142:40296) 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6095bec700 1 -- 172.21.15.142:0/1073889948 learned_addr learned my addr 172.21.15.142:0/1073889948 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 1 --2- 172.21.15.142:0/1073889948 >> v2:172.21.15.150:3300/0 conn(0x7f609009f8e0 0x7f609013ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): get_auth_request con 0x7f609009f8e0 auth_method 0 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.323 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.315+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 10 monclient(hunting): handle_auth_done global_id 194665 payload 274 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 -- 172.21.15.142:0/1073889948 >> v1:172.21.15.150:6789/0 conn(0x7f60901320a0 legacy=0x7f6090132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 -- 172.21.15.142:0/1073889948 >> v1:172.21.15.142:6789/0 conn(0x7f609009f4f0 legacy=0x7f60901373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 monclient: found mon.noname-c 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.324 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 -- 172.21.15.142:0/1073889948 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609009dce0 con 0x7f609009f8e0 2024-09-18T10:29:43.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 -- 172.21.15.142:0/1073889948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 msgr2=0x7f609013ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6096bee700 1 --2- 172.21.15.142:0/1073889948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f609013ab20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6080009e60 tx=0x7f6080009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25fad51bd58f56b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 1 -- 172.21.15.142:0/1073889948 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608000e070 con 0x7f609009f8e0 2024-09-18T10:29:43.325 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: dump: 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.326 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6098651700 5 monclient: authenticate success, global_id 194665 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 1 -- 172.21.15.142:0/1073889948 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080012430 con 0x7f609009f8e0 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6098651700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6098651700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f6098651700 4 set_mon_vals no callback set 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 1 -- 172.21.15.142:0/1073889948 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60800127d0 con 0x7f609009f8e0 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.319+0000 7f60953eb700 10 monclient: dump: 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.327 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.328 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.331 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: shutdown 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- 172.21.15.142:0/1073889948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 msgr2=0x7f609013ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 --2- 172.21.15.142:0/1073889948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f609013ab20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6080009e60 tx=0x7f6080009780 comp rx=0 tx=0).stop 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- 172.21.15.142:0/1073889948 shutdown_connections 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 --2- 172.21.15.142:0/1073889948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f609013ab20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- 172.21.15.142:0/1073889948 >> 172.21.15.142:0/1073889948 conn(0x7f6090071680 msgr2=0x7f60900a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- 172.21.15.142:0/1073889948 shutdown_connections 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- 172.21.15.142:0/1073889948 wait complete. 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: monmap: 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.332 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 Processor -- start 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- start start 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: init 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:43.333 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 0x7f609012f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): picked mon.noname-a con 0x7f609009f4f0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f6090141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): picked mon.noname-b con 0x7f609009f8e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 0x7f6090141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): picked mon.noname-c con 0x7f60901320a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609012e8a0 con 0x7f609009f4f0 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609009d220 con 0x7f609009f8e0 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60901319c0 con 0x7f60901320a0 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6098651700 10 monclient(hunting): authenticate will time out at 2072.251666s 2024-09-18T10:29:43.334 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 0x7f609012f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 0x7f609012f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34162/0 (socket says 172.21.15.142:34162) 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 1 -- 172.21.15.142:0/1938269629 learned_addr learned my addr 172.21.15.142:0/1938269629 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): get_auth_request con 0x7f609009f4f0 auth_method 0 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6096bee700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 0x7f6090141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6096bee700 10 monclient(hunting): get_auth_request con 0x7f60901320a0 auth_method 0 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6096bee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6096bee700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f6096bee700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.335 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.323+0000 7f60963ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f6090141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 10 monclient(hunting): get_auth_request con 0x7f609009f8e0 auth_method 0 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 msgr2=0x7f6090141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 msgr2=0x7f6090141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient(hunting): handle_auth_done global_id 234194 payload 824 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f6090141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-18T10:29:43.336 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 msgr2=0x7f6090141620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6095bec700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f6090141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f609009f8e0 0x7f6090141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 msgr2=0x7f609012f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 0x7f609012f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 monclient: found mon.noname-c 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080009a20 con 0x7f60901320a0 2024-09-18T10:29:43.337 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f60963ed700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f609009f4f0 0x7f609012f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:43.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 msgr2=0x7f6090141b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6096bee700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 0x7f6090141b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f609006a4a0 tx=0x7f6080018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.339 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080026430 con 0x7f60901320a0 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: dump: 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.340 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 5 monclient: authenticate success, global_id 234194 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6080021070 con 0x7f60901320a0 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080026820 con 0x7f60901320a0 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 10 monclient: _renew_subs 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609009c720 con 0x7f60901320a0 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608ffff700 4 set_mon_vals no callback set 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f608e7fc700 10 monclient: dump: 2024-09-18T10:29:43.341 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 10 monclient: _renew_subs 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.327+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609012d310 con 0x7f60901320a0 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f6098651700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.342 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f609012e390 con 0x7f60901320a0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f6080023030 con 0x7f60901320a0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f607c035ac0 0x7f607c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f608004b750 con 0x7f60901320a0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f607c03ebb0 0x7f607c041020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f607c0416f0 con 0x7f607c03ebb0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=264) v2 ==== 24+0+0 (secure 0 0 0) 0x7f608004dd60 con 0x7f60901320a0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f608e7fc700 10 monclient: handle_get_version_reply finishing 1 version 264 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f60963ed700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f607c035ac0 0x7f607c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f60963ed700 10 monclient: get_auth_request con 0x7f607c035ac0 auth_method 0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f6095bec700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f607c03ebb0 0x7f607c041020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.331+0000 7f6095bec700 10 monclient: get_auth_request con 0x7f607c03ebb0 auth_method 0 2024-09-18T10:29:43.343 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.335+0000 7f60963ed700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f607c035ac0 0x7f607c037f70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6088007fd0 tx=0x7f6088007f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.335+0000 7f6095bec700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f607c03ebb0 0x7f607c041020 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.344 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.339+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== osd.3 v2:172.21.15.150:6800/21119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f607c0416f0 con 0x7f607c03ebb0 2024-09-18T10:29:43.359 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 --> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f60901422e0 con 0x7f607c03ebb0 2024-09-18T10:29:43.360 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f608e7fc700 1 -- 172.21.15.142:0/1938269629 <== osd.3 v2:172.21.15.150:6800/21119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f60901422e0 con 0x7f607c03ebb0 2024-09-18T10:29:43.361 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:43.361 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:43.361 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f607c03ebb0 msgr2=0x7f607c041020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6800/21119,v1:172.21.15.150:6801/21119] conn(0x7f607c03ebb0 0x7f607c041020 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f607c035ac0 msgr2=0x7f607c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f607c035ac0 0x7f607c037f70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6088007fd0 tx=0x7f6088007f60 comp rx=0 tx=0).stop 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 10 monclient: shutdown 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 msgr2=0x7f6090141b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6098651700 1 --2- 172.21.15.142:0/1938269629 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f60901320a0 0x7f6090141b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f609006a4a0 tx=0x7f6080018000 comp rx=0 tx=0).stop 2024-09-18T10:29:43.362 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.355+0000 7f6096bee700 1 -- 172.21.15.142:0/1938269629 reap_dead start 2024-09-18T10:29:43.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.359+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 shutdown_connections 2024-09-18T10:29:43.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.359+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 >> 172.21.15.142:0/1938269629 conn(0x7f6090071680 msgr2=0x7f6090130560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.359+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 shutdown_connections 2024-09-18T10:29:43.363 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.359+0000 7f6098651700 1 -- 172.21.15.142:0/1938269629 wait complete. 2024-09-18T10:29:43.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: get_monmap_and_config 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: monmap: 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.479 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:43.477099+0000 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:43.477099+0000 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:43.480 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 Processor -- start 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 -- start start 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: init 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 0x7f7dc409e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.483 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-b con 0x7f7dc409cdd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7dc40a02e0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 0x7f7dc4139a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-a con 0x7f7dc40a0dc0 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc409da60 con 0x7f7dc40a02e0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc409d1a0 con 0x7f7dc40a0dc0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc409e980 con 0x7f7dc409cdd0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dca342700 10 monclient(hunting): authenticate will time out at 2072.403402s 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 0x7f7dc409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc37fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f7dc40a02e0 0x7f7dc40a06b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40312/0 (socket says 172.21.15.142:40312) 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 0x7f7dc4139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.484 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc37fe700 1 -- 172.21.15.142:0/3044297215 learned_addr learned my addr 172.21.15.142:0/3044297215 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): get_auth_request con 0x7f7dc409cdd0 auth_method 0 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): get_auth_request con 0x7f7dc40a0dc0 auth_method 0 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.475+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_done global_id 234197 payload 274 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 -- 172.21.15.142:0/3044297215 >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 msgr2=0x7f7dc4139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 --2- 172.21.15.142:0/3044297215 >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 0x7f7dc4139a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 -- 172.21.15.142:0/3044297215 >> v1:172.21.15.142:6789/0 conn(0x7f7dc40a02e0 legacy=0x7f7dc40a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 monclient: found mon.noname-b 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 -- 172.21.15.142:0/3044297215 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc409d780 con 0x7f7dc409cdd0 2024-09-18T10:29:43.487 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc88df700 1 --2- 172.21.15.142:0/3044297215 >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 0x7f7dc4139a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:43.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 -- 172.21.15.142:0/3044297215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 msgr2=0x7f7dc409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc3fff700 1 --2- 172.21.15.142:0/3044297215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 0x7f7dc409e440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7db40098e0 tx=0x7f7db40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be3121361e3e474 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 1 -- 172.21.15.142:0/3044297215 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400b850 con 0x7f7dc409cdd0 2024-09-18T10:29:43.491 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: dump: 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.492 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dca342700 5 monclient: authenticate success, global_id 234197 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dca342700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 1 -- 172.21.15.142:0/3044297215 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400b9b0 con 0x7f7dc409cdd0 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dca342700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 1 -- 172.21.15.142:0/3044297215 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400bd50 con 0x7f7dc409cdd0 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dca342700 4 set_mon_vals no callback set 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.479+0000 7f7dc2ffd700 10 monclient: dump: 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.493 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: shutdown 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- 172.21.15.142:0/3044297215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 msgr2=0x7f7dc409e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- 172.21.15.142:0/3044297215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 0x7f7dc409e440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7db40098e0 tx=0x7f7db40091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- 172.21.15.142:0/3044297215 shutdown_connections 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- 172.21.15.142:0/3044297215 >> v2:172.21.15.142:3300/0 conn(0x7f7dc40a0dc0 0x7f7dc4139a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.494 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- 172.21.15.142:0/3044297215 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc409cdd0 0x7f7dc409e440 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- 172.21.15.142:0/3044297215 >> 172.21.15.142:0/3044297215 conn(0x7f7dc4071680 msgr2=0x7f7dc40a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- 172.21.15.142:0/3044297215 shutdown_connections 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- 172.21.15.142:0/3044297215 wait complete. 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: monmap: 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:43.495 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 Processor -- start 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- start start 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: init 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 0x7f7dc4141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-a con 0x7f7dc409cdd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 0x7f7dc4142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-b con 0x7f7dc40a02e0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 0x7f7dc408cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): picked mon.noname-c con 0x7f7dc40a0dc0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.496 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc408ea80 con 0x7f7dc409cdd0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4091740 con 0x7f7dc40a02e0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc409c100 con 0x7f7dc40a0dc0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dca342700 10 monclient(hunting): authenticate will time out at 2072.410912s 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 0x7f7dc408cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 0x7f7dc4141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 0x7f7dc408cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35452/0 (socket says 172.21.15.142:35452) 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 1 -- 172.21.15.142:0/3668503550 learned_addr learned my addr 172.21.15.142:0/3668503550 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): get_auth_request con 0x7f7dc40a0dc0 auth_method 0 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 0x7f7dc4141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34186/0 (socket says 172.21.15.142:34186) 2024-09-18T10:29:43.497 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): get_auth_request con 0x7f7dc409cdd0 auth_method 0 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 0x7f7dc4142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): get_auth_request con 0x7f7dc40a02e0 auth_method 0 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.498 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc88df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient(hunting): handle_auth_done global_id 194668 payload 824 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 -- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 msgr2=0x7f7dc408cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 0x7f7dc408cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 -- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 msgr2=0x7f7dc4141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 0x7f7dc4141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 monclient: found mon.noname-b 2024-09-18T10:29:43.499 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.500 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.483+0000 7f7dc37fe700 1 -- 172.21.15.142:0/3668503550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc009a20 con 0x7f7dc40a02e0 2024-09-18T10:29:43.501 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dc37fe700 1 -- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 msgr2=0x7f7dc4142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dc37fe700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 0x7f7dc4142210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c560 tx=0x7f7dac007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac005d40 con 0x7f7dc40a02e0 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: dump: 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.502 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac0070d0 con 0x7f7dc40a02e0 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 5 monclient: authenticate success, global_id 194668 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _renew_subs 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dac007490 con 0x7f7dc40a02e0 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db4009420 con 0x7f7dc40a02e0 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dc27fc700 4 set_mon_vals no callback set 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.503 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dbbfff700 10 monclient: dump: 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _renew_subs 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.504 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc40904b0 con 0x7f7dc40a02e0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.487+0000 7f7dca342700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4090230 con 0x7f7dc40a02e0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f7dac01b3b0 con 0x7f7dc40a02e0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dbbfff700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 0x7f7da4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f7dac0483a0 con 0x7f7dc40a02e0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dc3fff700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 0x7f7da4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dc3fff700 10 monclient: get_auth_request con 0x7f7da4035ac0 auth_method 0 2024-09-18T10:29:43.505 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.491+0000 7f7dc3fff700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 0x7f7da4037f70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7db40030c0 tx=0x7f7db4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.499+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f7dac019890 con 0x7f7dc40a02e0 2024-09-18T10:29:43.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.499+0000 7f7dbbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.506 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.499+0000 7f7dbbfff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:43.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.663+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7dc408f620 con 0x7f7da4035ac0 2024-09-18T10:29:43.668 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.663+0000 7f7dbbfff700 1 -- 172.21.15.142:0/3668503550 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7dc408f620 con 0x7f7da4035ac0 2024-09-18T10:29:43.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 msgr2=0x7f7da4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 0x7f7da4037f70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7db40030c0 tx=0x7f7db4016000 comp rx=0 tx=0).stop 2024-09-18T10:29:43.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 10 monclient: shutdown 2024-09-18T10:29:43.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.671 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 msgr2=0x7f7dc4142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 0x7f7dc4142210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00c560 tx=0x7f7dac007f90 comp rx=0 tx=0).stop 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 shutdown_connections 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f7da4035ac0 0x7f7da4037f70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7dc40a0dc0 0x7f7dc408cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dc40a02e0 0x7f7dc4142210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 --2- 172.21.15.142:0/3668503550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc409cdd0 0x7f7dc4141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 >> 172.21.15.142:0/3668503550 conn(0x7f7dc4071680 msgr2=0x7f7dc40a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 shutdown_connections 2024-09-18T10:29:43.672 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.667+0000 7f7dca342700 1 -- 172.21.15.142:0/3668503550 wait complete. 2024-09-18T10:29:43.673 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T10:29:43.682 INFO:tasks.ceph:Scrubbing osd.4 2024-09-18T10:29:43.683 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:43.792 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: get_monmap_and_config 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: monmap: 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:43.790322+0000 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:43.790322+0000 2024-09-18T10:29:43.793 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 Processor -- start 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 -- start start 2024-09-18T10:29:43.794 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: init 2024-09-18T10:29:43.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.798 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 0x7fe8d0098c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-b con 0x7fe8d009a6e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8d0094830 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-a con 0x7fe8d009c570 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d009d9f0 con 0x7fe8d0094830 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d0099660 con 0x7fe8d009c570 2024-09-18T10:29:43.799 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0099920 con 0x7fe8d009a6e0 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d6fbc700 10 monclient(hunting): authenticate will time out at 2072.715801s 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d67bb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.787+0000 7fe8d5fba700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 0x7fe8d0098c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34196/0 (socket says 172.21.15.142:34196) 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 1 -- 172.21.15.142:0/2422185069 learned_addr learned my addr 172.21.15.142:0/2422185069 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): get_auth_request con 0x7fe8d009c570 auth_method 0 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.800 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): get_auth_request con 0x7fe8d009a6e0 auth_method 0 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_done global_id 234203 payload 274 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 -- 172.21.15.142:0/2422185069 >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 msgr2=0x7fe8d013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 --2- 172.21.15.142:0/2422185069 >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.801 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 -- 172.21.15.142:0/2422185069 >> v1:172.21.15.150:6789/0 conn(0x7fe8d0094830 legacy=0x7fe8d009be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 monclient: found mon.noname-b 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 -- 172.21.15.142:0/2422185069 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0099c00 con 0x7fe8d009a6e0 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d67bb700 1 --2- 172.21.15.142:0/2422185069 >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 -- 172.21.15.142:0/2422185069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 msgr2=0x7fe8d0098c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d5fba700 1 --2- 172.21.15.142:0/2422185069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 0x7fe8d0098c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40098e0 tx=0x7fe8c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a2b749d8d7a98a server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 1 -- 172.21.15.142:0/2422185069 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c400b850 con 0x7fe8d009a6e0 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: dump: 2024-09-18T10:29:43.802 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d6fbc700 5 monclient: authenticate success, global_id 234203 2024-09-18T10:29:43.803 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d6fbc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 1 -- 172.21.15.142:0/2422185069 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c400b9b0 con 0x7fe8d009a6e0 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d6fbc700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d6fbc700 4 set_mon_vals no callback set 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 1 -- 172.21.15.142:0/2422185069 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c400bd50 con 0x7fe8d009a6e0 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.791+0000 7fe8d4fb8700 10 monclient: dump: 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.804 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: shutdown 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/2422185069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 msgr2=0x7fe8d0098c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/2422185069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 0x7fe8d0098c00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40098e0 tx=0x7fe8c40091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/2422185069 shutdown_connections 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/2422185069 >> v2:172.21.15.142:3300/0 conn(0x7fe8d009c570 0x7fe8d013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.805 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/2422185069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d009a6e0 0x7fe8d0098c00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/2422185069 >> 172.21.15.142:0/2422185069 conn(0x7fe8d00a1470 msgr2=0x7fe8d00a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/2422185069 shutdown_connections 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/2422185069 wait complete. 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: monmap: 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:43.806 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:43.807 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 Processor -- start 2024-09-18T10:29:43.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- start start 2024-09-18T10:29:43.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: init 2024-09-18T10:29:43.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.812 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8d0094830 0x7fe8d0138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-b con 0x7fe8d0094830 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8d009c570 0x7fe8d006b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-a con 0x7fe8d009c570 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 0x7fe8d006b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): picked mon.noname-c con 0x7fe8d0138ca0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d006f590 con 0x7fe8d009c570 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0146a00 con 0x7fe8d0094830 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d006f3e0 con 0x7fe8d0138ca0 2024-09-18T10:29:43.813 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d6fbc700 10 monclient(hunting): authenticate will time out at 2072.722817s 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8d009c570 0x7fe8d006b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 0x7fe8d006b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8d009c570 0x7fe8d006b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34204/0 (socket says 172.21.15.142:34204) 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 1 -- 172.21.15.142:0/1931166079 learned_addr learned my addr 172.21.15.142:0/1931166079 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): get_auth_request con 0x7fe8d0138ca0 auth_method 0 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): get_auth_request con 0x7fe8d009c570 auth_method 0 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.814 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8d0094830 0x7fe8d0138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d57b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): get_auth_request con 0x7fe8d0094830 auth_method 0 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.815 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d5fba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient(hunting): handle_auth_done global_id 234206 payload 824 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8d0094830 msgr2=0x7fe8d0138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8d0094830 0x7fe8d0138760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8d009c570 msgr2=0x7fe8d006b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8d009c570 0x7fe8d006b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 monclient: found mon.noname-c 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.795+0000 7fe8d67bb700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c8009ae0 con 0x7fe8d0138ca0 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d67bb700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 msgr2=0x7fe8d006b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.816 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d67bb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 0x7fe8d006b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009f10 tx=0x7fe8c800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8018070 con 0x7fe8d0138ca0 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: dump: 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.817 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 5 monclient: authenticate success, global_id 234206 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 10 monclient: _renew_subs 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c800f040 con 0x7fe8d0138ca0 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c4009420 con 0x7fe8d0138ca0 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c800cda0 con 0x7fe8d0138ca0 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:43.818 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c1ffb700 10 monclient: dump: 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8c37fe700 4 set_mon_vals no callback set 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 10 monclient: _renew_subs 2024-09-18T10:29:43.819 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d006d890 con 0x7fe8d0138ca0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.799+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8d0072a80 con 0x7fe8d0138ca0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d67bb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 0x7fe8d006b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009f10 tx=0x7fe8c800b040 comp rx=0 tx=0).queue_received will delay after 1.72666e+09 on 0x7fe8c8048df0 osd_map(264..264 src has 239..264) v4 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fe8c8003020 con 0x7fe8d0138ca0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8c1ffb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8b4035af0 0x7fe8b4037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fe8c8048df0 con 0x7fe8d0138ca0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8c1ffb700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7fe8b403eb60 0x7fe8b4040fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8b40416a0 con 0x7fe8b403eb60 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d5fba700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8b4035af0 0x7fe8b4037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d5fba700 10 monclient: get_auth_request con 0x7fe8b4035af0 auth_method 0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d57b9700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7fe8b403eb60 0x7fe8b4040fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.820 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d57b9700 10 monclient: get_auth_request con 0x7fe8b403eb60 auth_method 0 2024-09-18T10:29:43.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.803+0000 7fe8d5fba700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8b4035af0 0x7fe8b4037fa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00a1380 tx=0x7fe8c4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.807+0000 7fe8d57b9700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7fe8b403eb60 0x7fe8b4040fd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.821 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.807+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== osd.4 v2:172.21.15.150:6818/21118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe8b40416a0 con 0x7fe8b403eb60 2024-09-18T10:29:43.830 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 --> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe8d01394e0 con 0x7fe8b403eb60 2024-09-18T10:29:43.831 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== osd.4 v2:172.21.15.150:6818/21118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8d01394e0 con 0x7fe8b403eb60 2024-09-18T10:29:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:43.831 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:43.831 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7fe8b403eb60 msgr2=0x7fe8b4040fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6818/21118,v1:172.21.15.150:6819/21118] conn(0x7fe8b403eb60 0x7fe8b4040fd0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8b4035af0 msgr2=0x7fe8b4037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fe8b4035af0 0x7fe8b4037fa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00a1380 tx=0x7fe8c4016000 comp rx=0 tx=0).stop 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 10 monclient: shutdown 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 20 monclient: shutdown canceling and discarding version request 1 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.833 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 msgr2=0x7fe8d006b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 --2- 172.21.15.142:0/1931166079 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8d0138ca0 0x7fe8d006b990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe8c8009f10 tx=0x7fe8c800b040 comp rx=0 tx=0).stop 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8c1ffb700 1 -- 172.21.15.142:0/1931166079 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=264) v2 ==== 24+0+0 (unknown 0 0 0) 0x7fe8c804b340 con 0x7fe8d0138ca0 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8c1ffb700 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=264) v2 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d67bb700 1 -- 172.21.15.142:0/1931166079 reap_dead start 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 shutdown_connections 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 >> 172.21.15.142:0/1931166079 conn(0x7fe8d00a1470 msgr2=0x7fe8d0139880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 shutdown_connections 2024-09-18T10:29:43.834 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.827+0000 7fe8d6fbc700 1 -- 172.21.15.142:0/1931166079 wait complete. 2024-09-18T10:29:43.843 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-18T10:29:43.956 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 10 monclient: get_monmap_and_config 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 10 monclient: monmap: 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:43.954702+0000 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:43.954702+0000 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:43.957 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:43.958 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:43.958 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:43.958 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 1 Processor -- start 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 1 -- start start 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.951+0000 7fd216519700 10 monclient: init 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd21009c760 0x7fd21009af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): picked mon.noname-c con 0x7fd21009c760 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd21009b490 addr v1:172.21.15.150:6789/0 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd21009d240 0x7fd210135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): picked mon.noname-a con 0x7fd21009d240 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd21009a850 con 0x7fd21009b490 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd210099290 con 0x7fd21009d240 2024-09-18T10:29:43.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd210098f30 con 0x7fd21009c760 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd216519700 10 monclient(hunting): authenticate will time out at 2072.880475s 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd21009d240 0x7fd210135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd21009d240 0x7fd210135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34214/0 (socket says 172.21.15.142:34214) 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 -- 172.21.15.142:0/2997662512 learned_addr learned my addr 172.21.15.142:0/2997662512 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): get_auth_request con 0x7fd21009d240 auth_method 0 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 1 --2- 172.21.15.142:0/2997662512 >> v2:172.21.15.150:3300/0 conn(0x7fd21009c760 0x7fd21009af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): get_auth_request con 0x7fd21009c760 auth_method 0 2024-09-18T10:29:43.962 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient(hunting): handle_auth_done global_id 224322 payload 274 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 -- 172.21.15.142:0/2997662512 >> v2:172.21.15.150:3300/0 conn(0x7fd21009c760 msgr2=0x7fd21009af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 --2- 172.21.15.142:0/2997662512 >> v2:172.21.15.150:3300/0 conn(0x7fd21009c760 0x7fd21009af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 -- 172.21.15.142:0/2997662512 >> v1:172.21.15.150:6789/0 conn(0x7fd21009b490 legacy=0x7fd21009cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 monclient: found mon.noname-a 2024-09-18T10:29:43.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:43.964 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 -- 172.21.15.142:0/2997662512 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21009a290 con 0x7fd21009d240 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 -- 172.21.15.142:0/2997662512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009d240 msgr2=0x7fd210135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.955+0000 7fd214ab6700 1 --2- 172.21.15.142:0/2997662512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009d240 0x7fd210135b80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd21009dd40 tx=0x7fd208009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640529df0dece64d server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 1 -- 172.21.15.142:0/2997662512 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20800e070 con 0x7fd21009d240 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: dump: 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.968 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 5 monclient: authenticate success, global_id 224322 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 1 -- 172.21.15.142:0/2997662512 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20800bc30 con 0x7fd21009d240 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:43.969 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 4 set_mon_vals no callback set 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 1 -- 172.21.15.142:0/2997662512 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2080092e0 con 0x7fd21009d240 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20effd700 10 monclient: dump: 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.970 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: shutdown 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- 172.21.15.142:0/2997662512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009d240 msgr2=0x7fd210135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- 172.21.15.142:0/2997662512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009d240 0x7fd210135b80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd21009dd40 tx=0x7fd208009840 comp rx=0 tx=0).stop 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- 172.21.15.142:0/2997662512 shutdown_connections 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- 172.21.15.142:0/2997662512 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009d240 0x7fd210135b80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- 172.21.15.142:0/2997662512 >> v2:172.21.15.150:3300/0 conn(0x7fd21009c760 0x7fd21009af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- 172.21.15.142:0/2997662512 >> 172.21.15.142:0/2997662512 conn(0x7fd210071b20 msgr2=0x7fd2100a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- 172.21.15.142:0/2997662512 shutdown_connections 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- 172.21.15.142:0/2997662512 wait complete. 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: build_initial_monmap 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: monmap: 2024-09-18T10:29:43.971 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 Processor -- start 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- start start 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: init 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:43.972 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): picked mon.noname-a con 0x7fd21009c760 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21009d240 0x7fd2100942f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): picked mon.noname-c con 0x7fd21009d240 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 0x7fd21008df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): picked mon.noname-b con 0x7fd210090620 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2100a1440 con 0x7fd21009c760 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210091b60 con 0x7fd210090620 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd210091510 con 0x7fd21009d240 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:43.973 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd216519700 10 monclient(hunting): authenticate will time out at 2072.887382s 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34220/0 (socket says 172.21.15.142:34220) 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 1 -- 172.21.15.142:0/1477442492 learned_addr learned my addr 172.21.15.142:0/1477442492 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): get_auth_request con 0x7fd21009c760 auth_method 0 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21009d240 0x7fd2100942f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 0x7fd21008df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): get_auth_request con 0x7fd210090620 auth_method 0 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.974 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): get_auth_request con 0x7fd21009d240 auth_method 0 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd214ab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.959+0000 7fd20f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd20f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:43.975 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 10 monclient(hunting): handle_auth_done global_id 194677 payload 824 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 -- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21009d240 msgr2=0x7fd2100942f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21009d240 0x7fd2100942f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 -- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 msgr2=0x7fd210141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 10 monclient: _finish_hunting 0 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 monclient: found mon.noname-b 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 -- 172.21.15.142:0/1477442492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd208009ae0 con 0x7fd210090620 2024-09-18T10:29:43.976 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd20ffff700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:43.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 -- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 msgr2=0x7fd21008df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:43.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd214ab6700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 0x7fd21008df00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd208014000 tx=0x7fd208017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208012b30 con 0x7fd210090620 2024-09-18T10:29:43.978 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: dump: 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: _finish_auth 0 2024-09-18T10:29:43.979 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 5 monclient: authenticate success, global_id 194677 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208012cf0 con 0x7fd210090620 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20801d670 con 0x7fd210090620 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd1f7fff700 10 monclient: dump: 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:43.980 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 10 monclient: _renew_subs 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2100917f0 con 0x7fd210090620 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd20e7fc700 4 set_mon_vals no callback set 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 10 monclient: _renew_subs 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.963+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd210141f20 con 0x7fd210090620 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd216519700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd216519700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.981 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd216519700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.150:3300/0 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21008fa10 con 0x7fd210090620 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd20801da20 con 0x7fd210090620 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd1f7fff700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 0x7fd1fc037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd20804a890 con 0x7fd210090620 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd20ffff700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 0x7fd1fc037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.967+0000 7fd20ffff700 10 monclient: get_auth_request con 0x7fd1fc035ac0 auth_method 0 2024-09-18T10:29:43.982 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.971+0000 7fd20ffff700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 0x7fd1fc037f70 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd204009fd0 tx=0x7fd2040091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:43.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.979+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mon.1 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd208012e60 con 0x7fd210090620 2024-09-18T10:29:43.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.979+0000 7fd1f7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:43.983 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:43.979+0000 7fd1f7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:44.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.139+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd210091ca0 con 0x7fd1fc035ac0 2024-09-18T10:29:44.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd1f7fff700 1 -- 172.21.15.142:0/1477442492 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd210091ca0 con 0x7fd1fc035ac0 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 msgr2=0x7fd1fc037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 0x7fd1fc037f70 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd204009fd0 tx=0x7fd2040091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 10 monclient: shutdown 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 msgr2=0x7fd21008df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 0x7fd21008df00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd208014000 tx=0x7fd208017000 comp rx=0 tx=0).stop 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 shutdown_connections 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd1fc035ac0 0x7fd1fc037f70 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd210090620 0x7fd21008df00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21009d240 0x7fd2100942f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 --2- 172.21.15.142:0/1477442492 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd21009c760 0x7fd210141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 >> 172.21.15.142:0/1477442492 conn(0x7fd21009b490 msgr2=0x7fd210087fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 shutdown_connections 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.143+0000 7fd216519700 1 -- 172.21.15.142:0/1477442492 wait complete. 2024-09-18T10:29:44.149 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 4 to deep-scrub 2024-09-18T10:29:44.159 INFO:tasks.ceph:Scrubbing osd.5 2024-09-18T10:29:44.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-18T10:29:44.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: get_monmap_and_config 2024-09-18T10:29:44.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.270 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: monmap: 2024-09-18T10:29:44.270 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:44.268375+0000 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:44.268375+0000 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:44.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 Processor -- start 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 -- start start 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: init 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f15f01320a0 0x7f15f0132470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-c con 0x7f15f01320a0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15f009f4f0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-b con 0x7f15f009f8e0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f009d090 con 0x7f15f009f4f0 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f00a16b0 con 0x7f15f01320a0 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f009dce0 con 0x7f15f009f8e0 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15f4b8b700 10 monclient(hunting): authenticate will time out at 2073.194406s 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35504/0 (socket says 172.21.15.142:35504) 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 -- 172.21.15.142:0/1416271212 learned_addr learned my addr 172.21.15.142:0/1416271212 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): get_auth_request con 0x7f15f009f8e0 auth_method 0 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.278 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15ee59c700 1 --2- 172.21.15.142:0/1416271212 >> v2:172.21.15.150:3300/0 conn(0x7f15f01320a0 0x7f15f0132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15ee59c700 10 monclient(hunting): get_auth_request con 0x7f15f01320a0 auth_method 0 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15ee59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15ee59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient(hunting): handle_auth_done global_id 234212 payload 274 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 -- 172.21.15.142:0/1416271212 >> v2:172.21.15.150:3300/0 conn(0x7f15f01320a0 msgr2=0x7f15f0132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 --2- 172.21.15.142:0/1416271212 >> v2:172.21.15.150:3300/0 conn(0x7f15f01320a0 0x7f15f0132470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 -- 172.21.15.142:0/1416271212 >> v1:172.21.15.142:6789/0 conn(0x7f15f009f4f0 legacy=0x7f15f00a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 monclient: found mon.noname-b 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.279 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.267+0000 7f15eed9d700 1 -- 172.21.15.142:0/1416271212 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f009da00 con 0x7f15f009f8e0 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15eed9d700 1 -- 172.21.15.142:0/1416271212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 msgr2=0x7f15f00a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15eed9d700 1 --2- 172.21.15.142:0/1416271212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009f20 tx=0x7f15e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ddd29488e0f0cdc server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 1 -- 172.21.15.142:0/1416271212 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e800e070 con 0x7f15f009f8e0 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: dump: 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.280 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 5 monclient: authenticate success, global_id 234212 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 1 -- 172.21.15.142:0/1416271212 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e800bc90 con 0x7f15f009f8e0 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 1 -- 172.21.15.142:0/1416271212 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e80092e0 con 0x7f15f009f8e0 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 4 set_mon_vals no callback set 2024-09-18T10:29:44.281 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15ed59a700 10 monclient: dump: 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.282 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 10 monclient: shutdown 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 -- 172.21.15.142:0/1416271212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 msgr2=0x7f15f00a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/1416271212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009f20 tx=0x7f15e8009840 comp rx=0 tx=0).stop 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 -- 172.21.15.142:0/1416271212 shutdown_connections 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/1416271212 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f009f8e0 0x7f15f00a09e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/1416271212 >> v2:172.21.15.150:3300/0 conn(0x7f15f01320a0 0x7f15f0132470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 -- 172.21.15.142:0/1416271212 >> 172.21.15.142:0/1416271212 conn(0x7f15f0071680 msgr2=0x7f15f00a58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 -- 172.21.15.142:0/1416271212 shutdown_connections 2024-09-18T10:29:44.283 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.271+0000 7f15f4b8b700 1 -- 172.21.15.142:0/1416271212 wait complete. 2024-09-18T10:29:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient: monmap: 2024-09-18T10:29:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.289 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 Processor -- start 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 -- start start 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient: init 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.290 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f009f4f0 0x7f15f012f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-b con 0x7f15f009f4f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 0x7f15f009ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-a con 0x7f15f009f8e0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 0x7f15f009b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): picked mon.noname-c con 0x7f15f01320a0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f012ebc0 con 0x7f15f009f8e0 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f012e600 con 0x7f15f009f4f0 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f012e8e0 con 0x7f15f01320a0 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.291 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15f4b8b700 10 monclient(hunting): authenticate will time out at 2073.202675s 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 0x7f15f009ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 0x7f15f009ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34226/0 (socket says 172.21.15.142:34226) 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 1 -- 172.21.15.142:0/4037990167 learned_addr learned my addr 172.21.15.142:0/4037990167 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 0x7f15f009b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): get_auth_request con 0x7f15f009f8e0 auth_method 0 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f009f4f0 0x7f15f012f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): get_auth_request con 0x7f15f01320a0 auth_method 0 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.292 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): get_auth_request con 0x7f15f009f4f0 auth_method 0 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.293 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient(hunting): handle_auth_done global_id 234215 payload 824 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f009f4f0 msgr2=0x7f15f012f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15f009f4f0 0x7f15f012f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 msgr2=0x7f15f009ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 0x7f15f009ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 monclient: found mon.noname-c 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15eed9d700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e8009ae0 con 0x7f15f01320a0 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.275+0000 7f15edd9b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15f009f8e0 0x7f15f009ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15eed9d700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 msgr2=0x7f15f009b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15eed9d700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 0x7f15f009b090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f15f006a4a0 tx=0x7f15e8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.294 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8012b00 con 0x7f15f01320a0 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: dump: 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.295 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e8012c60 con 0x7f15f01320a0 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 5 monclient: authenticate success, global_id 234215 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 10 monclient: _renew_subs 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f012d3c0 con 0x7f15f01320a0 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e801d420 con 0x7f15f01320a0 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 10 monclient: dump: 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.296 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15ecd99700 4 set_mon_vals no callback set 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 10 monclient: _renew_subs 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f009c470 con 0x7f15f01320a0 2024-09-18T10:29:44.297 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f15e801d850 con 0x7f15f01320a0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15f00696a0 con 0x7f15f01320a0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.279+0000 7f15de7fc700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15d0035af0 0x7f15d0037fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f15e804a4d0 con 0x7f15f01320a0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15de7fc700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f15d003ebe0 0x7f15d0041050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15d0041720 con 0x7f15d003ebe0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15ee59c700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15d0035af0 0x7f15d0037fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15ee59c700 10 monclient: get_auth_request con 0x7f15d0035af0 auth_method 0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15edd9b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f15d003ebe0 0x7f15d0041050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15edd9b700 10 monclient: get_auth_request con 0x7f15d003ebe0 auth_method 0 2024-09-18T10:29:44.298 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.283+0000 7f15ee59c700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15d0035af0 0x7f15d0037fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f15e4006fd0 tx=0x7f15e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.287+0000 7f15edd9b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f15d003ebe0 0x7f15d0041050 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.287+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== osd.5 v2:172.21.15.150:6810/21120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f15d0041720 con 0x7f15d003ebe0 2024-09-18T10:29:44.312 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.307+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 --> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f15f009b810 con 0x7f15d003ebe0 2024-09-18T10:29:44.313 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.307+0000 7f15de7fc700 1 -- 172.21.15.142:0/4037990167 <== osd.5 v2:172.21.15.150:6810/21120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f15f009b810 con 0x7f15d003ebe0 2024-09-18T10:29:44.313 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-09-18T10:29:44.314 INFO:teuthology.orchestra.run.smithi142.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T10:29:44.314 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f15d003ebe0 msgr2=0x7f15d0041050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6810/21120,v1:172.21.15.150:6811/21120] conn(0x7f15d003ebe0 0x7f15d0041050 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15d0035af0 msgr2=0x7f15d0037fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f15d0035af0 0x7f15d0037fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f15e4006fd0 tx=0x7f15e4008040 comp rx=0 tx=0).stop 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 10 monclient: shutdown 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 20 monclient: shutdown canceling and discarding version request 1 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 msgr2=0x7f15f009b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 --2- 172.21.15.142:0/4037990167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15f01320a0 0x7f15f009b090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f15f006a4a0 tx=0x7f15e8017000 comp rx=0 tx=0).stop 2024-09-18T10:29:44.315 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15eed9d700 1 -- 172.21.15.142:0/4037990167 reap_dead start 2024-09-18T10:29:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 shutdown_connections 2024-09-18T10:29:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 >> 172.21.15.142:0/4037990167 conn(0x7f15f0071680 msgr2=0x7f15f012b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 shutdown_connections 2024-09-18T10:29:44.316 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.311+0000 7f15f4b8b700 1 -- 172.21.15.142:0/4037990167 wait complete. 2024-09-18T10:29:44.326 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-18T10:29:44.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: get_monmap_and_config 2024-09-18T10:29:44.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: monmap: 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:44.433428+0000 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:44.433428+0000 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:44.436 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:44.437 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:44.437 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:44.437 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 Processor -- start 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 -- start start 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: init 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcd806f3b0 addr v1:172.21.15.142:6789/0 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 0x7fdcd8134670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.439 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-b con 0x7fdcd80990d0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdcd80994c0 0x7fdcd8136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-c con 0x7fdcd80994c0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd809c330 con 0x7fdcd806f3b0 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd809b380 con 0x7fdcd80994c0 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd809b030 con 0x7fdcd80990d0 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcdc88c700 10 monclient(hunting): authenticate will time out at 2073.359742s 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcd77fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdcd806f3b0 0x7fdcd80989c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:40332/0 (socket says 172.21.15.142:40332) 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcd6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 0x7fdcd8134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.431+0000 7fdcd77fe700 1 -- 172.21.15.142:0/1563901330 learned_addr learned my addr 172.21.15.142:0/1563901330 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): get_auth_request con 0x7fdcd80990d0 auth_method 0 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 1 --2- 172.21.15.142:0/1563901330 >> v2:172.21.15.150:3300/0 conn(0x7fdcd80994c0 0x7fdcd8136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.441 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request con 0x7fdcd80994c0 auth_method 0 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_done global_id 234218 payload 274 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 -- 172.21.15.142:0/1563901330 >> v2:172.21.15.150:3300/0 conn(0x7fdcd80994c0 msgr2=0x7fdcd8136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 --2- 172.21.15.142:0/1563901330 >> v2:172.21.15.150:3300/0 conn(0x7fdcd80994c0 0x7fdcd8136bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 -- 172.21.15.142:0/1563901330 >> v1:172.21.15.142:6789/0 conn(0x7fdcd806f3b0 legacy=0x7fdcd80989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 monclient: found mon.noname-b 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:29:44.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 -- 172.21.15.142:0/1563901330 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd809c530 con 0x7fdcd80990d0 2024-09-18T10:29:44.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 -- 172.21.15.142:0/1563901330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 msgr2=0x7fdcd8134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd6ffd700 1 --2- 172.21.15.142:0/1563901330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 0x7fdcd8134670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098e0 tx=0x7fdccc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab135285d6d32968 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 1 -- 172.21.15.142:0/1563901330 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00b850 con 0x7fdcd80990d0 2024-09-18T10:29:44.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.450 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: dump: 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.451 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcdc88c700 5 monclient: authenticate success, global_id 234218 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcdc88c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 1 -- 172.21.15.142:0/1563901330 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc00b9b0 con 0x7fdcd80990d0 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcdc88c700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 1 -- 172.21.15.142:0/1563901330 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00bd50 con 0x7fdcd80990d0 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcdc88c700 4 set_mon_vals no callback set 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.435+0000 7fdcd67fc700 10 monclient: dump: 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.452 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: shutdown 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- 172.21.15.142:0/1563901330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 msgr2=0x7fdcd8134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/1563901330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 0x7fdcd8134670 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098e0 tx=0x7fdccc0091a0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- 172.21.15.142:0/1563901330 shutdown_connections 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/1563901330 >> v2:172.21.15.150:3300/0 conn(0x7fdcd80994c0 0x7fdcd8136bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.453 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/1563901330 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80990d0 0x7fdcd8134670 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- 172.21.15.142:0/1563901330 >> 172.21.15.142:0/1563901330 conn(0x7fdcd8070ef0 msgr2=0x7fdcd80712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- 172.21.15.142:0/1563901330 shutdown_connections 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- 172.21.15.142:0/1563901330 wait complete. 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: monmap: 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.454 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 Processor -- start 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- start start 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: init 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-a con 0x7fdcd806f3b0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:44.455 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcd80990d0 0x7fdcd813f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-b con 0x7fdcd80990d0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80994c0 0x7fdcd813fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): picked mon.noname-c con 0x7fdcd80994c0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806fa20 con 0x7fdcd806f3b0 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd8097300 con 0x7fdcd80990d0 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd808f7a0 con 0x7fdcd80994c0 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.456 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcdc88c700 10 monclient(hunting): authenticate will time out at 2073.366761s 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34242/0 (socket says 172.21.15.142:34242) 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 -- 172.21.15.142:0/100188131 learned_addr learned my addr 172.21.15.142:0/100188131 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request con 0x7fdcd806f3b0 auth_method 0 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80994c0 0x7fdcd813fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcd80990d0 0x7fdcd813f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request con 0x7fdcd80994c0 auth_method 0 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.457 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): get_auth_request con 0x7fdcd80990d0 auth_method 0 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.458 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_done global_id 224331 payload 824 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 -- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80994c0 msgr2=0x7fdcd813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80994c0 0x7fdcd813fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 -- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcd80990d0 msgr2=0x7fdcd813f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcd80990d0 0x7fdcd813f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 monclient: found mon.noname-a 2024-09-18T10:29:44.459 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.439+0000 7fdcd77fe700 1 -- 172.21.15.142:0/100188131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc009420 con 0x7fdcd806f3b0 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcd77fe700 1 -- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 msgr2=0x7fdcd809c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcd77fe700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c5e0 tx=0x7fdcc8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8005d40 con 0x7fdcd806f3b0 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: dump: 2024-09-18T10:29:44.460 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc80070d0 con 0x7fdcd806f3b0 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 5 monclient: authenticate success, global_id 224331 2024-09-18T10:29:44.461 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _renew_subs 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd808f1e0 con 0x7fdcd806f3b0 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8007490 con 0x7fdcd806f3b0 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 10 monclient: dump: 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.462 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcd57fa700 4 set_mon_vals no callback set 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _renew_subs 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81401c0 con 0x7fdcd806f3b0 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fdcc801b3b0 con 0x7fdcd806f3b0 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.443+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8072a80 con 0x7fdcd806f3b0 2024-09-18T10:29:44.463 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.447+0000 7fdcbf7fe700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 0x7fdcc4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.447+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fdcc8048490 con 0x7fdcd806f3b0 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.447+0000 7fdcd6ffd700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 0x7fdcc4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.447+0000 7fdcd6ffd700 10 monclient: get_auth_request con 0x7fdcc4035ac0 auth_method 0 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.447+0000 7fdcd6ffd700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 0x7fdcc4037f70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0093f0 tx=0x7fdccc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.455+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fdcc8019870 con 0x7fdcd806f3b0 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.455+0000 7fdcbf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.464 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.455+0000 7fdcbf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:44.627 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.623+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcd808f3d0 con 0x7fdcc4035ac0 2024-09-18T10:29:44.628 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.623+0000 7fdcbf7fe700 1 -- 172.21.15.142:0/100188131 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdcd808f3d0 con 0x7fdcc4035ac0 2024-09-18T10:29:44.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 msgr2=0x7fdcc4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.631 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 0x7fdcc4037f70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0093f0 tx=0x7fdccc016000 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 10 monclient: shutdown 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 msgr2=0x7fdcd809c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c5e0 tx=0x7fdcc8004a80 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 shutdown_connections 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fdcc4035ac0 0x7fdcc4037f70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcd80994c0 0x7fdcd813fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdcd80990d0 0x7fdcd813f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 --2- 172.21.15.142:0/100188131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcd806f3b0 0x7fdcd809c7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 >> 172.21.15.142:0/100188131 conn(0x7fdcd8070ef0 msgr2=0x7fdcd80a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 shutdown_connections 2024-09-18T10:29:44.632 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.627+0000 7fdcdc88c700 1 -- 172.21.15.142:0/100188131 wait complete. 2024-09-18T10:29:44.633 INFO:teuthology.orchestra.run.smithi142.stderr:instructed osd(s) 5 to deep-scrub 2024-09-18T10:29:44.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 10 monclient: get_monmap_and_config 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 10 monclient: monmap: 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.745 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:29:44.742691+0000 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:29:44.742691+0000 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 1 Processor -- start 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 1 -- start start 2024-09-18T10:29:44.746 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.739+0000 7fd8c507f700 10 monclient: init 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 0x7fd8c0098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8c009a7f0 addr v2:172.21.15.150:3300/0 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c0099220 0x7fd8c009bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8c0099220 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd8c009c320 0x7fd8c009c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8c009c320 addr v2:172.21.15.142:3300/0 2024-09-18T10:29:44.751 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c009cc50 con 0x7fd8c009c320 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c009abc0 con 0x7fd8c009a7f0 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c009d950 con 0x7fd8c0099220 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c507f700 10 monclient(hunting): authenticate will time out at 2073.668279s 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd8c009c320 0x7fd8c009c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd8c009c320 0x7fd8c009c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:34250/0 (socket says 172.21.15.142:34250) 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c0099220 0x7fd8c009bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 -- 172.21.15.142:0/1867630785 learned_addr learned my addr 172.21.15.142:0/1867630785 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): get_auth_request con 0x7fd8c009c320 auth_method 0 2024-09-18T10:29:44.752 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): get_auth_request con 0x7fd8c0099220 auth_method 0 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 1 --2- 172.21.15.142:0/1867630785 >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 0x7fd8c0098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 10 monclient(hunting): get_auth_request con 0x7fd8c009a7f0 auth_method 0 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.753 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient(hunting): handle_auth_done global_id 224334 payload 274 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 -- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c0099220 msgr2=0x7fd8c009bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c0099220 0x7fd8c009bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 -- 172.21.15.142:0/1867630785 >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 msgr2=0x7fd8c0098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1867630785 >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 0x7fd8c0098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 monclient: found mon.noname-a 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.754 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 -- 172.21.15.142:0/1867630785 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c009db90 con 0x7fd8c009c320 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8bffff700 1 --2- 172.21.15.142:0/1867630785 >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 0x7fd8c0098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 -- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 msgr2=0x7fd8c009c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c009c710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0098a60 tx=0x7fd8b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727acef86f6abb01 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 1 -- 172.21.15.142:0/1867630785 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8009e70 con 0x7fd8c009c320 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 10 monclient: dump: 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.755 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.743+0000 7fd8beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 1 -- 172.21.15.142:0/1867630785 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b800ba70 con 0x7fd8c009c320 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 1 -- 172.21.15.142:0/1867630785 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b800be30 con 0x7fd8c009c320 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 5 monclient: authenticate success, global_id 224334 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: get_monmap_and_config success 2024-09-18T10:29:44.756 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 4 set_mon_vals no callback set 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8beffd700 10 monclient: dump: 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.757 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: shutdown 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 msgr2=0x7fd8c009c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c009c710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0098a60 tx=0x7fd8b8009250 comp rx=0 tx=0).stop 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- 172.21.15.142:0/1867630785 shutdown_connections 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c009c710 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1867630785 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c0099220 0x7fd8c009bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1867630785 >> v2:172.21.15.150:3300/0 conn(0x7fd8c009a7f0 0x7fd8c0098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- 172.21.15.142:0/1867630785 >> 172.21.15.142:0/1867630785 conn(0x7fd8c00a1fe0 msgr2=0x7fd8c0070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- 172.21.15.142:0/1867630785 shutdown_connections 2024-09-18T10:29:44.758 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- 172.21.15.142:0/1867630785 wait complete. 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: build_initial_monmap 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: monmap: 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.762 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 Processor -- start 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- start start 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: init 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: _reopen_session rank -1 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 0x7fd8c0145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-b con 0x7fd8c0099220 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 0x7fd8c01418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-c con 0x7fd8c009a7f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:29:44.763 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c0142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): picked mon.noname-a con 0x7fd8c009c320 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0127000 con 0x7fd8c009c320 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0126940 con 0x7fd8c0099220 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c006e380 con 0x7fd8c009a7f0 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): _renew_subs 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.747+0000 7fd8c507f700 10 monclient(hunting): authenticate will time out at 2073.675788s 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c0142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 0x7fd8c01418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 0x7fd8c0145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.764 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 0x7fd8c01418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:35558/0 (socket says 172.21.15.142:35558) 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 0x7fd8c0145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.142:48046/0 (socket says 172.21.15.142:48046) 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 1 -- 172.21.15.142:0/1642473877 learned_addr learned my addr 172.21.15.142:0/1642473877 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): get_auth_request con 0x7fd8c009c320 auth_method 0 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): get_auth_request con 0x7fd8c009a7f0 auth_method 0 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): get_auth_request con 0x7fd8c0099220 auth_method 0 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:29:44.765 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient(hunting): handle_auth_done global_id 224337 payload 824 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 -- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 msgr2=0x7fd8c01418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 0x7fd8c01418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.766 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 -- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 msgr2=0x7fd8c0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 0x7fd8c0145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient: _finish_hunting 0 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 monclient: found mon.noname-a 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 -- 172.21.15.142:0/1642473877 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b80094f0 con 0x7fd8c009c320 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 -- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 msgr2=0x7fd8c0142260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c487e700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c0142260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd8c009d6b0 tx=0x7fd8b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b801f070 con 0x7fd8c009c320 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:29:44.767 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: dump: 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: _finish_auth 0 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 5 monclient: authenticate success, global_id 224337 2024-09-18T10:29:44.768 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b8011a60 con 0x7fd8c009c320 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8011e00 con 0x7fd8c009c320 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8a7fff700 10 monclient: dump: 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:29:44.769 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 10 monclient: _renew_subs 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c006d820 con 0x7fd8c009c320 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8bdffb700 4 set_mon_vals no callback set 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 10 monclient: _renew_subs 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.751+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c006e9d0 con 0x7fd8c009c320 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8c507f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8c507f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8c507f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c0072a80 con 0x7fd8c009c320 2024-09-18T10:29:44.770 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7fd8b8023070 con 0x7fd8c009c320 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8a7fff700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 0x7fd8ac037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7fd8b8049e30 con 0x7fd8c009c320 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8bffff700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 0x7fd8ac037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.755+0000 7fd8bffff700 10 monclient: get_auth_request con 0x7fd8ac035ac0 auth_method 0 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.759+0000 7fd8bffff700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 0x7fd8ac037f70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4006f00 tx=0x7fd8b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.763+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7fd8b802c540 con 0x7fd8c009c320 2024-09-18T10:29:44.771 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.763+0000 7fd8a7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:29:44.772 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.763+0000 7fd8a7fff700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:29:44.928 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.923+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8c009d730 con 0x7fd8ac035ac0 2024-09-18T10:29:44.936 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.931+0000 7fd8a7fff700 1 -- 172.21.15.142:0/1642473877 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97476 (secure 0 0 0) 0x7fd8c009d730 con 0x7fd8ac035ac0 2024-09-18T10:29:44.938 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 msgr2=0x7fd8ac037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 0x7fd8ac037f70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4006f00 tx=0x7fd8b4008040 comp rx=0 tx=0).stop 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 10 monclient: shutdown 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 msgr2=0x7fd8c0142260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:29:44.940 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c0142260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd8c009d6b0 tx=0x7fd8b8017000 comp rx=0 tx=0).stop 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 shutdown_connections 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7fd8ac035ac0 0x7fd8ac037f70 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8c009c320 0x7fd8c0142260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8c009a7f0 0x7fd8c01418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 --2- 172.21.15.142:0/1642473877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8c0099220 0x7fd8c0145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 >> 172.21.15.142:0/1642473877 conn(0x7fd8c00a1fe0 msgr2=0x7fd8c00907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 shutdown_connections 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:29:44.935+0000 7fd8c507f700 1 -- 172.21.15.142:0/1642473877 wait complete. 2024-09-18T10:29:44.941 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:29:44.954 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":878,"stamp":"2024-09-18T10:29:43.133294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820536,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410504,"statfs":{"total":579820584960,"available":577956356096,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003836"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":174,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733987+0000","last_change":"2024-09-18T10:15:10.262207+0000","last_active":"2024-09-18T10:18:23.733987+0000","last_peered":"2024-09-18T10:18:23.733987+0000","last_clean":"2024-09-18T10:18:23.733987+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:18:23.733987+0000","last_undegraded":"2024-09-18T10:18:23.733987+0000","last_fullsized":"2024-09-18T10:18:23.733987+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:41:30.693246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735070+0000","last_change":"2024-09-18T10:15:10.259663+0000","last_active":"2024-09-18T10:18:23.735070+0000","last_peered":"2024-09-18T10:18:23.735070+0000","last_clean":"2024-09-18T10:18:23.735070+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:18:23.735070+0000","last_undegraded":"2024-09-18T10:18:23.735070+0000","last_fullsized":"2024-09-18T10:18:23.735070+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:34.758954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":158,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860612+0000","last_change":"2024-09-18T10:15:10.263309+0000","last_active":"2024-09-18T10:18:24.860612+0000","last_peered":"2024-09-18T10:18:24.860612+0000","last_clean":"2024-09-18T10:18:24.860612+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:18:24.860612+0000","last_undegraded":"2024-09-18T10:18:24.860612+0000","last_fullsized":"2024-09-18T10:18:24.860612+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:44:24.067636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":91,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734913+0000","last_change":"2024-09-18T10:15:10.262000+0000","last_active":"2024-09-18T10:18:23.734913+0000","last_peered":"2024-09-18T10:18:23.734913+0000","last_clean":"2024-09-18T10:18:23.734913+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:18:23.734913+0000","last_undegraded":"2024-09-18T10:18:23.734913+0000","last_fullsized":"2024-09-18T10:18:23.734913+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:45:34.980736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":132,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.735082+0000","last_change":"2024-09-18T10:15:12.231443+0000","last_active":"2024-09-18T10:18:23.735082+0000","last_peered":"2024-09-18T10:18:23.735082+0000","last_clean":"2024-09-18T10:18:23.735082+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:18:23.735082+0000","last_undegraded":"2024-09-18T10:18:23.735082+0000","last_fullsized":"2024-09-18T10:18:23.735082+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:21.077963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662983+0000","last_change":"2024-09-18T10:15:10.258259+0000","last_active":"2024-09-18T10:18:24.662983+0000","last_peered":"2024-09-18T10:18:24.662983+0000","last_clean":"2024-09-18T10:18:24.662983+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:18:24.662983+0000","last_undegraded":"2024-09-18T10:18:24.662983+0000","last_fullsized":"2024-09-18T10:18:24.662983+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:43.637146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":90,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150425+0000","last_change":"2024-09-18T10:15:10.268743+0000","last_active":"2024-09-18T10:18:24.150425+0000","last_peered":"2024-09-18T10:18:24.150425+0000","last_clean":"2024-09-18T10:18:24.150425+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:18:24.150425+0000","last_undegraded":"2024-09-18T10:18:24.150425+0000","last_fullsized":"2024-09-18T10:18:24.150425+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:29:46.935586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662856+0000","last_change":"2024-09-18T10:15:10.258566+0000","last_active":"2024-09-18T10:18:24.662856+0000","last_peered":"2024-09-18T10:18:24.662856+0000","last_clean":"2024-09-18T10:18:24.662856+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:18:24.662856+0000","last_undegraded":"2024-09-18T10:18:24.662856+0000","last_fullsized":"2024-09-18T10:18:24.662856+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:34:16.243392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":115,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734740+0000","last_change":"2024-09-18T10:15:10.258870+0000","last_active":"2024-09-18T10:18:23.734740+0000","last_peered":"2024-09-18T10:18:23.734740+0000","last_clean":"2024-09-18T10:18:23.734740+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:18:23.734740+0000","last_undegraded":"2024-09-18T10:18:23.734740+0000","last_fullsized":"2024-09-18T10:18:23.734740+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:54:33.116712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":112,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149784+0000","last_change":"2024-09-18T10:15:10.253346+0000","last_active":"2024-09-18T10:18:24.149784+0000","last_peered":"2024-09-18T10:18:24.149784+0000","last_clean":"2024-09-18T10:18:24.149784+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:18:24.149784+0000","last_undegraded":"2024-09-18T10:18:24.149784+0000","last_fullsized":"2024-09-18T10:18:24.149784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:24:56.046228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":126,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860584+0000","last_change":"2024-09-18T10:15:10.261745+0000","last_active":"2024-09-18T10:18:24.860584+0000","last_peered":"2024-09-18T10:18:24.860584+0000","last_clean":"2024-09-18T10:18:24.860584+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:18:24.860584+0000","last_undegraded":"2024-09-18T10:18:24.860584+0000","last_fullsized":"2024-09-18T10:18:24.860584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:39:40.372105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662648+0000","last_change":"2024-09-18T10:15:10.253445+0000","last_active":"2024-09-18T10:18:24.662648+0000","last_peered":"2024-09-18T10:18:24.662648+0000","last_clean":"2024-09-18T10:18:24.662648+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:18:24.662648+0000","last_undegraded":"2024-09-18T10:18:24.662648+0000","last_fullsized":"2024-09-18T10:18:24.662648+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:54:33.914408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":86,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662787+0000","last_change":"2024-09-18T10:15:10.264746+0000","last_active":"2024-09-18T10:18:24.662787+0000","last_peered":"2024-09-18T10:18:24.662787+0000","last_clean":"2024-09-18T10:18:24.662787+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:18:24.662787+0000","last_undegraded":"2024-09-18T10:18:24.662787+0000","last_fullsized":"2024-09-18T10:18:24.662787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:06:12.470999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":181,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.733793+0000","last_change":"2024-09-18T10:15:10.262348+0000","last_active":"2024-09-18T10:18:23.733793+0000","last_peered":"2024-09-18T10:18:23.733793+0000","last_clean":"2024-09-18T10:18:23.733793+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:18:23.733793+0000","last_undegraded":"2024-09-18T10:18:23.733793+0000","last_fullsized":"2024-09-18T10:18:23.733793+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:28:23.247111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860518+0000","last_change":"2024-09-18T10:15:11.672918+0000","last_active":"2024-09-18T10:18:24.860518+0000","last_peered":"2024-09-18T10:18:24.860518+0000","last_clean":"2024-09-18T10:18:24.860518+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:18:24.860518+0000","last_undegraded":"2024-09-18T10:18:24.860518+0000","last_fullsized":"2024-09-18T10:18:24.860518+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:59.442484+0000","last_clean_scrub_stamp":"2024-09-18T10:13:59.442484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:56:53.310516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":85,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662547+0000","last_change":"2024-09-18T10:15:10.260417+0000","last_active":"2024-09-18T10:18:24.662547+0000","last_peered":"2024-09-18T10:18:24.662547+0000","last_clean":"2024-09-18T10:18:24.662547+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:18:24.662547+0000","last_undegraded":"2024-09-18T10:18:24.662547+0000","last_fullsized":"2024-09-18T10:18:24.662547+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:13:58.430771+0000","last_clean_scrub_stamp":"2024-09-18T10:13:58.430771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:13:26.409756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":333,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149870+0000","last_change":"2024-09-18T10:15:10.257790+0000","last_active":"2024-09-18T10:18:24.149870+0000","last_peered":"2024-09-18T10:18:24.149870+0000","last_clean":"2024-09-18T10:18:24.149870+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:18:24.149870+0000","last_undegraded":"2024-09-18T10:18:24.149870+0000","last_fullsized":"2024-09-18T10:18:24.149870+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:08:59.791843+0000","last_clean_scrub_stamp":"2024-09-18T10:08:59.791843+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:50:37.166570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734844+0000","last_change":"2024-09-18T10:15:10.272763+0000","last_active":"2024-09-18T10:18:23.734844+0000","last_peered":"2024-09-18T10:18:23.734844+0000","last_clean":"2024-09-18T10:18:23.734844+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:18:23.734844+0000","last_undegraded":"2024-09-18T10:18:23.734844+0000","last_fullsized":"2024-09-18T10:18:23.734844+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:13:14.189522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.149867+0000","last_change":"2024-09-18T10:15:10.254634+0000","last_active":"2024-09-18T10:18:24.149867+0000","last_peered":"2024-09-18T10:18:24.149867+0000","last_clean":"2024-09-18T10:18:24.149867+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:18:24.149867+0000","last_undegraded":"2024-09-18T10:18:24.149867+0000","last_fullsized":"2024-09-18T10:18:24.149867+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:33:08.149535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.860416+0000","last_change":"2024-09-18T10:15:10.258271+0000","last_active":"2024-09-18T10:18:24.860416+0000","last_peered":"2024-09-18T10:18:24.860416+0000","last_clean":"2024-09-18T10:18:24.860416+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:18:24.860416+0000","last_undegraded":"2024-09-18T10:18:24.860416+0000","last_fullsized":"2024-09-18T10:18:24.860416+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:34:18.769125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:23.734817+0000","last_change":"2024-09-18T10:15:10.267445+0000","last_active":"2024-09-18T10:18:23.734817+0000","last_peered":"2024-09-18T10:18:23.734817+0000","last_clean":"2024-09-18T10:18:23.734817+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:18:23.734817+0000","last_undegraded":"2024-09-18T10:18:23.734817+0000","last_fullsized":"2024-09-18T10:18:23.734817+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:02:06.660884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":239,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.662357+0000","last_change":"2024-09-18T10:15:12.228339+0000","last_active":"2024-09-18T10:18:24.662357+0000","last_peered":"2024-09-18T10:18:24.662357+0000","last_clean":"2024-09-18T10:18:24.662357+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:18:24.662357+0000","last_undegraded":"2024-09-18T10:18:24.662357+0000","last_fullsized":"2024-09-18T10:18:24.662357+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150004+0000","last_change":"2024-09-18T10:15:10.263155+0000","last_active":"2024-09-18T10:18:24.150004+0000","last_peered":"2024-09-18T10:18:24.150004+0000","last_clean":"2024-09-18T10:18:24.150004+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:18:24.150004+0000","last_undegraded":"2024-09-18T10:18:24.150004+0000","last_fullsized":"2024-09-18T10:18:24.150004+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:11:15.186223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.663037+0000","last_change":"2024-09-18T10:14:59.172586+0000","last_active":"2024-09-18T10:18:24.663037+0000","last_peered":"2024-09-18T10:18:24.663037+0000","last_clean":"2024-09-18T10:18:24.663037+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:18:24.663037+0000","last_undegraded":"2024-09-18T10:18:24.663037+0000","last_fullsized":"2024-09-18T10:18:24.663037+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:32:50.899590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-18T10:18:24.150325+0000","last_change":"2024-09-18T10:15:10.258203+0000","last_active":"2024-09-18T10:18:24.150325+0000","last_peered":"2024-09-18T10:18:24.150325+0000","last_clean":"2024-09-18T10:18:24.150325+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:18:24.150325+0000","last_undegraded":"2024-09-18T10:18:24.150325+0000","last_fullsized":"2024-09-18T10:18:24.150325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:23:55.852806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.57399999999999995},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.372},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.268},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.73199999999999998,"15min":0.73599999999999999},"min":{"1min":0.61899999999999999,"5min":0.48999999999999999,"15min":0.46100000000000002},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.47799999999999998}]},{"osd":1,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.68200000000000005},"min":{"1min":0.47599999999999998,"5min":0.438,"15min":0.438},"max":{"1min":0.79900000000000004,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.5},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.64800000000000002,"15min":0.66000000000000003},"min":{"1min":0.54100000000000004,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":1.081},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82199999999999995,"15min":0.88},"min":{"1min":0.70499999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.875},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.82399999999999995,"15min":0.90000000000000002},"min":{"1min":0.73499999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.609},"last":0.89900000000000002}]},{"osd":4,"last update":"Wed Sep 18 10:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.86099999999999999},"min":{"1min":0.76900000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.214,"5min":1.214,"15min":1.242},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.67700000000000005,"5min":0.66300000000000003,"15min":0.64400000000000002},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":0.85499999999999998}]},{"osd":5,"last update":"Wed Sep 18 10:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.84799999999999998,"15min":0.878},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.099,"5min":1.099,"15min":1.3740000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.995,"5min":0.94899999999999995,"15min":0.99199999999999999},"min":{"1min":0.73899999999999999,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":1.393,"5min":1.393,"15min":1.829},"last":0.76900000000000002}]}]},{"osd":1,"up_from":12,"seq":51539607805,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.46500000000000002,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":0.76800000000000002,"5min":1.079,"15min":1.079},"last":0.442},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38800000000000001},"max":{"1min":0.753,"5min":1.1359999999999999,"15min":1.528},"last":0.56999999999999995}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.999,"15min":1.0549999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67900000000000005,"15min":0.71699999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93799999999999994,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.50700000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.72199999999999998,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":1.008,"5min":1.2589999999999999,"15min":1.377},"last":0.751},{"interface":"front","average":{"1min":0.87,"5min":0.876,"15min":0.85099999999999998},"min":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.48899999999999999},"max":{"1min":1.0309999999999999,"5min":1.163,"15min":1.5589999999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.89500000000000002,"15min":0.875},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45100000000000001},"max":{"1min":1.127,"5min":1.222,"15min":1.5960000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89800000000000002,"15min":0.90400000000000003},"min":{"1min":0.73799999999999999,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98199999999999998,"5min":1.3,"15min":1.49},"last":0.79500000000000004}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91500000000000004,"15min":0.89400000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0129999999999999,"5min":1.423,"15min":1.423},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.91400000000000003},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.53300000000000003},"max":{"1min":1.038,"5min":1.331,"15min":1.639},"last":0.83699999999999997}]}]},{"osd":5,"up_from":11,"seq":47244640506,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303252,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068588,"statfs":{"total":96636764160,"available":96326234112,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81299999999999994,"15min":0.78800000000000003},"min":{"1min":0.61099999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98499999999999999,"5min":1.0489999999999999,"15min":1.101},"last":0.505},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86099999999999999,"15min":0.85699999999999998},"min":{"1min":0.73499999999999999,"5min":0.504,"15min":0.504},"max":{"1min":1.125,"5min":1.222,"15min":2.4460000000000002},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.90800000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.52600000000000002},"max":{"1min":1.077,"5min":1.091,"15min":1.4430000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.92100000000000004,"15min":0.877},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.399,"5min":1.399,"15min":2.3799999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.79300000000000004,"15min":0.83299999999999996},"min":{"1min":0.60399999999999998,"5min":0.60099999999999998,"15min":0.55700000000000005},"max":{"1min":1.034,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.89000000000000001},"min":{"1min":0.59799999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72199999999999998,"15min":0.72599999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.42499999999999999},"max":{"1min":0.92300000000000004,"5min":1.1319999999999999,"15min":1.2210000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.65300000000000002,"5min":0.56699999999999995,"15min":0.45700000000000002},"max":{"1min":1.0169999999999999,"5min":1.3600000000000001,"15min":1.556},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Sep 18 10:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.98699999999999999,"15min":1.0149999999999999},"min":{"1min":0.78700000000000003,"5min":0.69599999999999995,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":1.2370000000000001,"15min":1.653},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73499999999999999,"15min":0.78100000000000003},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.88900000000000001,"5min":1.165,"15min":2.327},"last":0.70799999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.76800000000000002,"15min":0.79000000000000004},"min":{"1min":0.59999999999999998,"5min":0.58499999999999996,"15min":0.50700000000000001},"max":{"1min":0.92900000000000005,"5min":1.044,"15min":1.347},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.85199999999999998,"15min":0.85899999999999999},"min":{"1min":0.65100000000000002,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0640000000000001,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":1.232}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":1.1619999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":1.117}]}]},{"osd":3,"up_from":11,"seq":47244640508,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.627},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.76100000000000001}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":0.57199999999999995}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.91500000000000004}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":0.84299999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640508,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303768,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068072,"statfs":{"total":96636764160,"available":96325705728,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.80000000000000004,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.47299999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.1459999999999999},"last":0.495},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65800000000000003,"15min":0.68000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92700000000000005,"15min":0.91200000000000003},"min":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0920000000000001,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97399999999999998,"15min":0.93799999999999994},"min":{"1min":0.85199999999999998,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.1970000000000001,"5min":1.27,"15min":1.347},"last":0.87}]},{"osd":4,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85199999999999998,"15min":0.84599999999999997},"min":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.125,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86899999999999999,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.3899999999999999},"last":0.93600000000000005}]},{"osd":5,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.871,"15min":0.89600000000000002},"min":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.1719999999999999,"5min":1.1950000000000001,"15min":1.333},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90100000000000002,"15min":0.94599999999999995},"min":{"1min":0.74199999999999999,"5min":0.69899999999999995,"15min":0.66400000000000003},"max":{"1min":1.2270000000000001,"5min":1.252,"15min":1.446},"last":0.83199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:29:44.959 INFO:tasks.ceph:pgid 10.4 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 10.5 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 10.6 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 10.7 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.960 INFO:tasks.ceph:pgid 10.0 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 10.1 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 10.2 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 10.3 last_scrub_stamp 2024-09-18T10:13:59.442484+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-18T10:13:58.430771+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=13, tm_sec=58, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T10:08:59.791843+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=8, tm_sec=59, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.961 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:29:44.962 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T10:30:04.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:30:05.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.071+0000 7f7608f78700 10 monclient: get_monmap_and_config 2024-09-18T10:30:05.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.071+0000 7f7608f78700 10 monclient: build_initial_monmap 2024-09-18T10:30:05.077 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.071+0000 7f7608f78700 10 monclient: monmap: 2024-09-18T10:30:05.077 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:30:05.075445+0000 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:30:05.075445+0000 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:30:05.078 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 Processor -- start 2024-09-18T10:30:05.083 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 -- start start 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient: init 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 0x7f760409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-b con 0x7f760409abd0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f760409db80 0x7f760409c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-c con 0x7f760409db80 addr v2:172.21.15.150:3300/0 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f760409c950 0x7f7604096870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-a con 0x7f760409c950 addr v2:172.21.15.142:3300/0 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.084 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760406fa60 con 0x7f760409c950 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76040958c0 con 0x7f760409db80 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7604095ef0 con 0x7f760409abd0 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7608f78700 10 monclient(hunting): authenticate will time out at 2094.000931s 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f760409c950 0x7f7604096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 0x7f760409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f760409c950 0x7f7604096870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48702/0 (socket says 172.21.15.142:48702) 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 -- 172.21.15.142:0/3609114434 learned_addr learned my addr 172.21.15.142:0/3609114434 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): get_auth_request con 0x7f760409c950 auth_method 0 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 0x7f760409bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:50586/0 (socket says 172.21.15.142:50586) 2024-09-18T10:30:05.085 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 1 --2- 172.21.15.142:0/3609114434 >> v2:172.21.15.150:3300/0 conn(0x7f760409db80 0x7f760409c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): get_auth_request con 0x7f760409abd0 auth_method 0 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): get_auth_request con 0x7f760409db80 auth_method 0 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.086 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient(hunting): handle_auth_done global_id 224340 payload 274 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 -- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 msgr2=0x7f760409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 --2- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 0x7f760409bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 -- 172.21.15.142:0/3609114434 >> v2:172.21.15.150:3300/0 conn(0x7f760409db80 msgr2=0x7f760409c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 --2- 172.21.15.142:0/3609114434 >> v2:172.21.15.150:3300/0 conn(0x7f760409db80 0x7f760409c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient: _finish_hunting 0 2024-09-18T10:30:05.087 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 monclient: found mon.noname-a 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.075+0000 7f7602d9d700 1 -- 172.21.15.142:0/3609114434 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760409afa0 con 0x7f760409c950 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7602d9d700 1 -- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409c950 msgr2=0x7f7604096870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7602d9d700 1 --2- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409c950 0x7f7604096870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76040a23c0 tx=0x7f75fc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87f0a72b7155d37 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 1 -- 172.21.15.142:0/3609114434 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc00e070 con 0x7f760409c950 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: dump: 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:05.088 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: _finish_auth 0 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 5 monclient: authenticate success, global_id 224340 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 1 -- 172.21.15.142:0/3609114434 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc00bb80 con 0x7f760409c950 2024-09-18T10:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 10 monclient: get_monmap_and_config success 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 4 set_mon_vals no callback set 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 1 -- 172.21.15.142:0/3609114434 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0092e0 con 0x7f760409c950 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f760159a700 10 monclient: dump: 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.090 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 10 monclient: shutdown 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409c950 msgr2=0x7f7604096870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 --2- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409c950 0x7f7604096870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f76040a23c0 tx=0x7f75fc009840 comp rx=0 tx=0).stop 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- 172.21.15.142:0/3609114434 shutdown_connections 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 --2- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409c950 0x7f7604096870 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 --2- 172.21.15.142:0/3609114434 >> v2:172.21.15.150:3300/0 conn(0x7f760409db80 0x7f760409c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 --2- 172.21.15.142:0/3609114434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409abd0 0x7f760409bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- 172.21.15.142:0/3609114434 >> 172.21.15.142:0/3609114434 conn(0x7f76040a14e0 msgr2=0x7f76040a18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:05.091 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- 172.21.15.142:0/3609114434 shutdown_connections 2024-09-18T10:30:05.092 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- 172.21.15.142:0/3609114434 wait complete. 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 10 monclient: build_initial_monmap 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 10 monclient: monmap: 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:05.097 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 Processor -- start 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 1 -- start start 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.079+0000 7f7608f78700 10 monclient: init 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-a con 0x7f760409abd0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 0x7f7604094810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-c con 0x7f760409c950 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f760409db80 0x7f760406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): picked mon.noname-b con 0x7f760409db80 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604126fa0 con 0x7f760409abd0 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604097c90 con 0x7f760409db80 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760406eb20 con 0x7f760409c950 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7608f78700 10 monclient(hunting): authenticate will time out at 2094.009171s 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48718/0 (socket says 172.21.15.142:48718) 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 1 -- 172.21.15.142:0/1018978881 learned_addr learned my addr 172.21.15.142:0/1018978881 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): get_auth_request con 0x7f760409abd0 auth_method 0 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 0x7f7604094810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): get_auth_request con 0x7f760409c950 auth_method 0 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f760409db80 0x7f760406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): get_auth_request con 0x7f760409db80 auth_method 0 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:05.101 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7602d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient(hunting): handle_auth_done global_id 234233 payload 824 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 -- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f760409db80 msgr2=0x7f760406b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f760409db80 0x7f760406b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 -- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 msgr2=0x7f76040942d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient: _finish_hunting 0 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 monclient: found mon.noname-c 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.142:3301/0 2024-09-18T10:30:05.102 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f7601d9b700 1 -- 172.21.15.142:0/1018978881 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc009ae0 con 0x7f760409c950 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.083+0000 7f760259c700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7601d9b700 1 -- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 msgr2=0x7f7604094810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7601d9b700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 0x7f7604094810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75f400ea80 tx=0x7f75f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40095d0 con 0x7f760409c950 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: dump: 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:05.103 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: _finish_auth 0 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 5 monclient: authenticate success, global_id 234233 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _renew_subs 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760406d1a0 con 0x7f760409c950 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f400ca00 con 0x7f760409c950 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:05.104 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7600d99700 4 set_mon_vals no callback set 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _renew_subs 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760406d860 con 0x7f760409c950 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f400cdc0 con 0x7f760409c950 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 10 monclient: dump: 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:05.105 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.142:3301/0 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f75f4013680 con 0x7f760409c950 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76040696a0 con 0x7f760409c950 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 0x7f75e4037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f75f4049140 con 0x7f760409c950 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f760259c700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 0x7f75e4037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.087+0000 7f760259c700 10 monclient: get_auth_request con 0x7f75e4035ac0 auth_method 0 2024-09-18T10:30:05.106 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.091+0000 7f760259c700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 0x7f75e4037f70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:05.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.095+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f75f40236b0 con 0x7f760409c950 2024-09-18T10:30:05.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.099+0000 7f75fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:05.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.099+0000 7f75fa7fc700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:30:05.265 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.259+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7604126cc0 con 0x7f75e4035ac0 2024-09-18T10:30:05.273 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.267+0000 7f75fa7fc700 1 -- 172.21.15.142:0/1018978881 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+97758 (secure 0 0 0) 0x7f7604126cc0 con 0x7f75e4035ac0 2024-09-18T10:30:05.275 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:30:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 msgr2=0x7f75e4037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 0x7f75e4037f70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec0091a0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.276 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 10 monclient: shutdown 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 msgr2=0x7f7604094810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 0x7f7604094810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75f400ea80 tx=0x7f75f40091a0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 shutdown_connections 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f75e4035ac0 0x7f75e4037f70 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f760409db80 0x7f760406b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760409c950 0x7f7604094810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 --2- 172.21.15.142:0/1018978881 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f760409abd0 0x7f76040942d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 >> 172.21.15.142:0/1018978881 conn(0x7f76040a14e0 msgr2=0x7f760409f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 shutdown_connections 2024-09-18T10:30:05.277 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:05.271+0000 7f7608f78700 1 -- 172.21.15.142:0/1018978881 wait complete. 2024-09-18T10:30:05.278 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:30:05.290 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-18T10:30:05.140858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820512,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410528,"statfs":{"total":579820584960,"available":577956380672,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004280"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":185,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.182470+0000","last_change":"2024-09-18T10:29:51.182344+0000","last_active":"2024-09-18T10:29:51.182470+0000","last_peered":"2024-09-18T10:29:51.182470+0000","last_clean":"2024-09-18T10:29:51.182470+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:29:51.182470+0000","last_undegraded":"2024-09-18T10:29:51.182470+0000","last_fullsized":"2024-09-18T10:29:51.182470+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.182019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.182019+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.182019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:38.316819+0000","scrub_duration":0.0039790019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":102,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:55.073494+0000","last_change":"2024-09-18T10:29:55.073414+0000","last_active":"2024-09-18T10:29:55.073494+0000","last_peered":"2024-09-18T10:29:55.073494+0000","last_clean":"2024-09-18T10:29:55.073494+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:29:55.073494+0000","last_undegraded":"2024-09-18T10:29:55.073494+0000","last_fullsized":"2024-09-18T10:29:55.073494+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:55.073177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:55.073177+0000","last_clean_scrub_stamp":"2024-09-18T10:29:55.073177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:39:45.579691+0000","scrub_duration":0.0032654250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":169,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:45.219039+0000","last_change":"2024-09-18T10:29:45.218956+0000","last_active":"2024-09-18T10:29:45.219039+0000","last_peered":"2024-09-18T10:29:45.219039+0000","last_clean":"2024-09-18T10:29:45.219039+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:29:45.219039+0000","last_undegraded":"2024-09-18T10:29:45.219039+0000","last_fullsized":"2024-09-18T10:29:45.219039+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:45.218733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:45.218733+0000","last_clean_scrub_stamp":"2024-09-18T10:29:45.218733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:34:35.140002+0000","scrub_duration":0.0038162840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":102,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.114436+0000","last_change":"2024-09-18T10:29:51.114362+0000","last_active":"2024-09-18T10:29:51.114436+0000","last_peered":"2024-09-18T10:29:51.114436+0000","last_clean":"2024-09-18T10:29:51.114436+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:29:51.114436+0000","last_undegraded":"2024-09-18T10:29:51.114436+0000","last_fullsized":"2024-09-18T10:29:51.114436+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.114155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.114155+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.114155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:16:39.912194+0000","scrub_duration":0.00349906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":143,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:59.091861+0000","last_change":"2024-09-18T10:29:59.091774+0000","last_active":"2024-09-18T10:29:59.091861+0000","last_peered":"2024-09-18T10:29:59.091861+0000","last_clean":"2024-09-18T10:29:59.091861+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:29:59.091861+0000","last_undegraded":"2024-09-18T10:29:59.091861+0000","last_fullsized":"2024-09-18T10:29:59.091861+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:59.091557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:59.091557+0000","last_clean_scrub_stamp":"2024-09-18T10:29:59.091557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:36.166378+0000","scrub_duration":0.003394163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":118,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.073436+0000","last_change":"2024-09-18T10:29:49.073323+0000","last_active":"2024-09-18T10:29:49.073436+0000","last_peered":"2024-09-18T10:29:49.073436+0000","last_clean":"2024-09-18T10:29:49.073436+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:29:49.073436+0000","last_undegraded":"2024-09-18T10:29:49.073436+0000","last_fullsized":"2024-09-18T10:29:49.073436+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.073106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.073106+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.073106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:14:09.851749+0000","scrub_duration":0.0034174990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":101,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:45.221737+0000","last_change":"2024-09-18T10:29:45.221645+0000","last_active":"2024-09-18T10:29:45.221737+0000","last_peered":"2024-09-18T10:29:45.221737+0000","last_clean":"2024-09-18T10:29:45.221737+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:29:45.221737+0000","last_undegraded":"2024-09-18T10:29:45.221737+0000","last_fullsized":"2024-09-18T10:29:45.221737+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:45.221449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:45.221449+0000","last_clean_scrub_stamp":"2024-09-18T10:29:45.221449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:09:15.687705+0000","scrub_duration":0.003519777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:56.054450+0000","last_change":"2024-09-18T10:29:56.054369+0000","last_active":"2024-09-18T10:29:56.054450+0000","last_peered":"2024-09-18T10:29:56.054450+0000","last_clean":"2024-09-18T10:29:56.054450+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:29:56.054450+0000","last_undegraded":"2024-09-18T10:29:56.054450+0000","last_fullsized":"2024-09-18T10:29:56.054450+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:56.054169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:56.054169+0000","last_clean_scrub_stamp":"2024-09-18T10:29:56.054169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:31:57.167450+0000","scrub_duration":0.0036297059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":126,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:47.142271+0000","last_change":"2024-09-18T10:29:47.142164+0000","last_active":"2024-09-18T10:29:47.142271+0000","last_peered":"2024-09-18T10:29:47.142271+0000","last_clean":"2024-09-18T10:29:47.142271+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:29:47.142271+0000","last_undegraded":"2024-09-18T10:29:47.142271+0000","last_fullsized":"2024-09-18T10:29:47.142271+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:47.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:47.141958+0000","last_clean_scrub_stamp":"2024-09-18T10:29:47.141958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:22:29.529273+0000","scrub_duration":0.003581089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":123,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:56.294228+0000","last_change":"2024-09-18T10:29:56.294134+0000","last_active":"2024-09-18T10:29:56.294228+0000","last_peered":"2024-09-18T10:29:56.294228+0000","last_clean":"2024-09-18T10:29:56.294228+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:29:56.294228+0000","last_undegraded":"2024-09-18T10:29:56.294228+0000","last_fullsized":"2024-09-18T10:29:56.294228+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:56.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:56.293936+0000","last_clean_scrub_stamp":"2024-09-18T10:29:56.293936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:38.727118+0000","scrub_duration":0.0034417409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":137,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.690938+0000","last_change":"2024-09-18T10:29:49.690859+0000","last_active":"2024-09-18T10:29:49.690938+0000","last_peered":"2024-09-18T10:29:49.690938+0000","last_clean":"2024-09-18T10:29:49.690938+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:29:49.690938+0000","last_undegraded":"2024-09-18T10:29:49.690938+0000","last_fullsized":"2024-09-18T10:29:49.690938+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.690661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.690661+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.690661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:08:01.146815+0000","scrub_duration":0.0036244139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:58.992750+0000","last_change":"2024-09-18T10:29:58.992666+0000","last_active":"2024-09-18T10:29:58.992750+0000","last_peered":"2024-09-18T10:29:58.992750+0000","last_clean":"2024-09-18T10:29:58.992750+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:29:58.992750+0000","last_undegraded":"2024-09-18T10:29:58.992750+0000","last_fullsized":"2024-09-18T10:29:58.992750+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:58.992467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:58.992467+0000","last_clean_scrub_stamp":"2024-09-18T10:29:58.992467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:02:37.866959+0000","scrub_duration":0.0030699339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":97,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:55.097270+0000","last_change":"2024-09-18T10:29:55.097184+0000","last_active":"2024-09-18T10:29:55.097270+0000","last_peered":"2024-09-18T10:29:55.097270+0000","last_clean":"2024-09-18T10:29:55.097270+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:29:55.097270+0000","last_undegraded":"2024-09-18T10:29:55.097270+0000","last_fullsized":"2024-09-18T10:29:55.097270+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:55.096976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:55.096976+0000","last_clean_scrub_stamp":"2024-09-18T10:29:55.096976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:42.920133+0000","scrub_duration":0.0036152390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":192,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:44.219576+0000","last_change":"2024-09-18T10:29:44.219488+0000","last_active":"2024-09-18T10:29:44.219576+0000","last_peered":"2024-09-18T10:29:44.219576+0000","last_clean":"2024-09-18T10:29:44.219576+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:29:44.219576+0000","last_undegraded":"2024-09-18T10:29:44.219576+0000","last_fullsized":"2024-09-18T10:29:44.219576+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:44.219192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:44.219192+0000","last_clean_scrub_stamp":"2024-09-18T10:29:44.219192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:23.294288+0000","scrub_duration":0.0030681720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":99,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.238692+0000","last_change":"2024-09-18T10:29:51.238602+0000","last_active":"2024-09-18T10:29:51.238692+0000","last_peered":"2024-09-18T10:29:51.238692+0000","last_clean":"2024-09-18T10:29:51.238692+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:29:51.238692+0000","last_undegraded":"2024-09-18T10:29:51.238692+0000","last_fullsized":"2024-09-18T10:29:51.238692+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.238417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.238417+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.238417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:48:05.601926+0000","scrub_duration":0.003583433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:50.120399+0000","last_change":"2024-09-18T10:29:50.120300+0000","last_active":"2024-09-18T10:29:50.120399+0000","last_peered":"2024-09-18T10:29:50.120399+0000","last_clean":"2024-09-18T10:29:50.120399+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:29:50.120399+0000","last_undegraded":"2024-09-18T10:29:50.120399+0000","last_fullsized":"2024-09-18T10:29:50.120399+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:50.120093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:50.120093+0000","last_clean_scrub_stamp":"2024-09-18T10:29:50.120093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:05:27.565756+0000","scrub_duration":0.0034736680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"264'74","reported_seq":346,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:53.189845+0000","last_change":"2024-09-18T10:29:53.189775+0000","last_active":"2024-09-18T10:29:53.189845+0000","last_peered":"2024-09-18T10:29:53.189845+0000","last_clean":"2024-09-18T10:29:53.189845+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:29:53.189845+0000","last_undegraded":"2024-09-18T10:29:53.189845+0000","last_fullsized":"2024-09-18T10:29:53.189845+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'74","last_scrub_stamp":"2024-09-18T10:29:53.189579+0000","last_deep_scrub":"264'74","last_deep_scrub_stamp":"2024-09-18T10:29:53.189579+0000","last_clean_scrub_stamp":"2024-09-18T10:29:53.189579+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:39.106255+0000","scrub_duration":0.0094258559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"264'2","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.111710+0000","last_change":"2024-09-18T10:29:49.111606+0000","last_active":"2024-09-18T10:29:49.111710+0000","last_peered":"2024-09-18T10:29:49.111710+0000","last_clean":"2024-09-18T10:29:49.111710+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:29:49.111710+0000","last_undegraded":"2024-09-18T10:29:49.111710+0000","last_fullsized":"2024-09-18T10:29:49.111710+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'2","last_scrub_stamp":"2024-09-18T10:29:49.111335+0000","last_deep_scrub":"264'2","last_deep_scrub_stamp":"2024-09-18T10:29:49.111335+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.111335+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:43:18.072502+0000","scrub_duration":0.0063368210000000003,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":287,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:44.242136+0000","last_change":"2024-09-18T10:29:44.242047+0000","last_active":"2024-09-18T10:29:44.242136+0000","last_peered":"2024-09-18T10:29:44.242136+0000","last_clean":"2024-09-18T10:29:44.242136+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:29:44.242136+0000","last_undegraded":"2024-09-18T10:29:44.242136+0000","last_fullsized":"2024-09-18T10:29:44.242136+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:44.241720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:44.241720+0000","last_clean_scrub_stamp":"2024-09-18T10:29:44.241720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.129303+0000","scrub_duration":0.003522074,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.275452+0000","last_change":"2024-09-18T10:29:49.275386+0000","last_active":"2024-09-18T10:29:49.275452+0000","last_peered":"2024-09-18T10:29:49.275452+0000","last_clean":"2024-09-18T10:29:49.275452+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:29:49.275452+0000","last_undegraded":"2024-09-18T10:29:49.275452+0000","last_fullsized":"2024-09-18T10:29:49.275452+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.275204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.275204+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.275204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:28.923431+0000","scrub_duration":0.00290369,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"264'3","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:46.150056+0000","last_change":"2024-09-18T10:29:46.149987+0000","last_active":"2024-09-18T10:29:46.150056+0000","last_peered":"2024-09-18T10:29:46.150056+0000","last_clean":"2024-09-18T10:29:46.150056+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:29:46.150056+0000","last_undegraded":"2024-09-18T10:29:46.150056+0000","last_fullsized":"2024-09-18T10:29:46.150056+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'3","last_scrub_stamp":"2024-09-18T10:29:46.149793+0000","last_deep_scrub":"264'3","last_deep_scrub_stamp":"2024-09-18T10:29:46.149793+0000","last_clean_scrub_stamp":"2024-09-18T10:29:46.149793+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:14:43.812460+0000","scrub_duration":0.006735843,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":240,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:18:29.462651+0000","last_change":"2024-09-18T10:15:12.228339+0000","last_active":"2024-09-18T10:18:29.462651+0000","last_peered":"2024-09-18T10:18:29.462651+0000","last_clean":"2024-09-18T10:18:29.462651+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:18:29.462651+0000","last_undegraded":"2024-09-18T10:18:29.462651+0000","last_fullsized":"2024-09-18T10:18:29.462651+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:09:00.831183+0000","last_clean_scrub_stamp":"2024-09-18T10:09:00.831183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:52.233099+0000","last_change":"2024-09-18T10:29:52.233010+0000","last_active":"2024-09-18T10:29:52.233099+0000","last_peered":"2024-09-18T10:29:52.233099+0000","last_clean":"2024-09-18T10:29:52.233099+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:29:52.233099+0000","last_undegraded":"2024-09-18T10:29:52.233099+0000","last_fullsized":"2024-09-18T10:29:52.233099+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:52.232808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:52.232808+0000","last_clean_scrub_stamp":"2024-09-18T10:29:52.232808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:19:01.369564+0000","scrub_duration":0.0033643459999999998,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:47.043374+0000","last_change":"2024-09-18T10:29:47.043284+0000","last_active":"2024-09-18T10:29:47.043374+0000","last_peered":"2024-09-18T10:29:47.043374+0000","last_clean":"2024-09-18T10:29:47.043374+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:29:47.043374+0000","last_undegraded":"2024-09-18T10:29:47.043374+0000","last_fullsized":"2024-09-18T10:29:47.043374+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:47.043063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:47.043063+0000","last_clean_scrub_stamp":"2024-09-18T10:29:47.043063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:33:03.128406+0000","scrub_duration":0.003472569,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:59.337315+0000","last_change":"2024-09-18T10:29:59.337217+0000","last_active":"2024-09-18T10:29:59.337315+0000","last_peered":"2024-09-18T10:29:59.337315+0000","last_clean":"2024-09-18T10:29:59.337315+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:29:59.337315+0000","last_undegraded":"2024-09-18T10:29:59.337315+0000","last_fullsized":"2024-09-18T10:29:59.337315+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:59.337006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:59.337006+0000","last_clean_scrub_stamp":"2024-09-18T10:29:59.337006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:06.732291+0000","scrub_duration":0.0032446089999999999,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54300000000000004,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.41299999999999998,"15min":0.372},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":1.268},"last":0.504},{"interface":"front","average":{"1min":0.747,"5min":0.73299999999999998,"15min":0.74099999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.67100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72199999999999998,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.41199999999999998},"max":{"1min":0.80300000000000005,"5min":0.98999999999999999,"15min":1.081},"last":0.67200000000000004}]},{"osd":3,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81399999999999995,"15min":0.873},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96099999999999997,"5min":1.1879999999999999,"15min":1.875},"last":1.081},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.83299999999999996,"15min":0.89700000000000002},"min":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.048,"5min":1.2150000000000001,"15min":1.609},"last":0.83699999999999997}]},{"osd":4,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.86299999999999999},"min":{"1min":0.65300000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.0349999999999999,"5min":1.214,"15min":1.242},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92100000000000004,"15min":0.88900000000000001},"min":{"1min":0.70499999999999996,"5min":0.67700000000000005,"15min":0.64400000000000002},"max":{"1min":1.0960000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.74299999999999999,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.121,"5min":1.121,"15min":1.3740000000000001},"last":1.28},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94199999999999995,"15min":0.97899999999999998},"min":{"1min":0.76900000000000002,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":0.996,"5min":1.393,"15min":1.704},"last":1.504}]}]},{"osd":1,"up_from":12,"seq":51539607809,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.46500000000000002,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":0.76800000000000002,"5min":1.079,"15min":1.079},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38800000000000001},"max":{"1min":0.753,"5min":1.1359999999999999,"15min":1.528},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.999,"15min":1.0549999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67900000000000005,"15min":0.71699999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93799999999999994,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.71399999999999997}]},{"osd":3,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.72199999999999998,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":1.008,"5min":1.2589999999999999,"15min":1.377},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.876,"15min":0.85099999999999998},"min":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.48899999999999999},"max":{"1min":1.0309999999999999,"5min":1.163,"15min":1.5589999999999999},"last":0.93700000000000006}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.89500000000000002,"15min":0.875},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45100000000000001},"max":{"1min":1.127,"5min":1.222,"15min":1.5960000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89800000000000002,"15min":0.90400000000000003},"min":{"1min":0.73799999999999999,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98199999999999998,"5min":1.3,"15min":1.49},"last":0.98399999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91500000000000004,"15min":0.89400000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0129999999999999,"5min":1.423,"15min":1.423},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.91400000000000003},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.53300000000000003},"max":{"1min":1.038,"5min":1.331,"15min":1.639},"last":0.748}]}]},{"osd":5,"up_from":11,"seq":47244640511,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.78400000000000003},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0089999999999999,"5min":1.0489999999999999,"15min":1.0569999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.85099999999999998,"15min":0.85699999999999998},"min":{"1min":0.53600000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.018,"5min":1.222,"15min":2.4460000000000002},"last":0.88800000000000001}]},{"osd":1,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.4430000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93200000000000005,"15min":0.88100000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.53100000000000003},"max":{"1min":1.278,"5min":1.399,"15min":2.3799999999999999},"last":0.92100000000000004}]},{"osd":2,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79600000000000004,"15min":0.82899999999999996},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0129999999999999,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.83499999999999996,"15min":0.88},"min":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.96999999999999997,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.78500000000000003}]},{"osd":3,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42499999999999999},"max":{"1min":0.96399999999999997,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.1819999999999999},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.84999999999999998,"15min":0.88200000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.45700000000000002},"max":{"1min":1.2110000000000001,"5min":1.3600000000000001,"15min":1.556},"last":1.034}]},{"osd":4,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.95599999999999996,"15min":1.0069999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.095,"5min":1.2370000000000001,"15min":1.653},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72399999999999998,"15min":0.77600000000000002},"min":{"1min":0.58099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.05,"5min":1.103,"15min":2.327},"last":1.0620000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640513,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.50700000000000001},"max":{"1min":0.90100000000000002,"5min":1.044,"15min":1.347},"last":0.753},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85499999999999998},"min":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":1.0169999999999999}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":0.83499999999999996}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":1.0429999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640513,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303764,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068076,"statfs":{"total":96636764160,"available":96325709824,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":1.1659999999999999}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.755},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.871}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":1.03}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.96899999999999997}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.995},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":1.3799999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640513,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303764,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068076,"statfs":{"total":96636764160,"available":96325709824,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.80000000000000004,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.47299999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.82899999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.1459999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65800000000000003,"15min":0.68000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92700000000000005,"15min":0.91200000000000003},"min":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0920000000000001,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97399999999999998,"15min":0.93799999999999994},"min":{"1min":0.85199999999999998,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.1970000000000001,"5min":1.27,"15min":1.347},"last":0.96599999999999997}]},{"osd":4,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85199999999999998,"15min":0.84599999999999997},"min":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.125,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":1.01},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86899999999999999,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.3899999999999999},"last":1.085}]},{"osd":5,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.871,"15min":0.89600000000000002},"min":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.1719999999999999,"5min":1.1950000000000001,"15min":1.333},"last":1.046},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90100000000000002,"15min":0.94599999999999995},"min":{"1min":0.74199999999999999,"5min":0.69899999999999995,"15min":0.66400000000000003},"max":{"1min":1.2270000000000001,"5min":1.252,"15min":1.446},"last":1.244}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:30:05.293 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-18T10:09:00.831183+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=9, tm_sec=0, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=10, tm_min=29, tm_sec=40, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T10:30:05.293 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T10:30:25.294 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T10:30:25.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.402+0000 7f77bee5f700 10 monclient: get_monmap_and_config 2024-09-18T10:30:25.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.402+0000 7f77bee5f700 10 monclient: build_initial_monmap 2024-09-18T10:30:25.409 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.402+0000 7f77bee5f700 10 monclient: monmap: 2024-09-18T10:30:25.409 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:25.409 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:30:25.407365+0000 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:30:25.407365+0000 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:30:25.410 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 Processor -- start 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 -- start start 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient: init 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77b809d920 addr v1:172.21.15.142:6789/0 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77b806f660 0x7f77b8096cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.413 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-c con 0x7f77b806f660 addr v2:172.21.15.150:3300/0 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 0x7f77b8136590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-b con 0x7f77b80971f0 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b809b0b0 con 0x7f77b809d920 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b8095ad0 con 0x7f77b806f660 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b809a7e0 con 0x7f77b80971f0 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bee5f700 10 monclient(hunting): authenticate will time out at 2114.332872s 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bde5d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f77b809d920 0x7f77b80953c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:42526/0 (socket says 172.21.15.142:42526) 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bde5d700 1 -- 172.21.15.142:0/586585023 learned_addr learned my addr 172.21.15.142:0/586585023 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 0x7f77b8136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.414 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): get_auth_request con 0x7f77b80971f0 auth_method 0 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 1 --2- 172.21.15.142:0/586585023 >> v2:172.21.15.150:3300/0 conn(0x7f77b806f660 0x7f77b8096cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): get_auth_request con 0x7f77b806f660 auth_method 0 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.415 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77bd65c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient(hunting): handle_auth_done global_id 234236 payload 274 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 -- 172.21.15.142:0/586585023 >> v2:172.21.15.150:3300/0 conn(0x7f77b806f660 msgr2=0x7f77b8096cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 --2- 172.21.15.142:0/586585023 >> v2:172.21.15.150:3300/0 conn(0x7f77b806f660 0x7f77b8096cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 -- 172.21.15.142:0/586585023 >> v1:172.21.15.142:6789/0 conn(0x7f77b809d920 legacy=0x7f77b80953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient: _finish_hunting 0 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 monclient: found mon.noname-b 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.142:3301/0 2024-09-18T10:30:25.416 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.406+0000 7f77be65e700 1 -- 172.21.15.142:0/586585023 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b809aaf0 con 0x7f77b80971f0 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 -- 172.21.15.142:0/586585023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 msgr2=0x7f77b8136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 --2- 172.21.15.142:0/586585023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 0x7f77b8136590 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009e60 tx=0x7f77b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9683597cb8c53b11 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 1 -- 172.21.15.142:0/586585023 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b000e070 con 0x7f77b80971f0 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: dump: 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.422 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: _finish_auth 0 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 1 -- 172.21.15.142:0/586585023 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b000bd70 con 0x7f77b80971f0 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 5 monclient: authenticate success, global_id 234236 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: get_monmap_and_config success 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 1 -- 172.21.15.142:0/586585023 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0012660 con 0x7f77b80971f0 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 4 set_mon_vals no callback set 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bce5b700 10 monclient: dump: 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.424 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: shutdown 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- 172.21.15.142:0/586585023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 msgr2=0x7f77b8136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- 172.21.15.142:0/586585023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 0x7f77b8136590 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009e60 tx=0x7f77b0009780 comp rx=0 tx=0).stop 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- 172.21.15.142:0/586585023 shutdown_connections 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- 172.21.15.142:0/586585023 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b80971f0 0x7f77b8136590 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- 172.21.15.142:0/586585023 >> v2:172.21.15.150:3300/0 conn(0x7f77b806f660 0x7f77b8096cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- 172.21.15.142:0/586585023 >> 172.21.15.142:0/586585023 conn(0x7f77b80a1fe0 msgr2=0x7f77b8070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- 172.21.15.142:0/586585023 shutdown_connections 2024-09-18T10:30:25.425 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- 172.21.15.142:0/586585023 wait complete. 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: build_initial_monmap 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: monmap: 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.426 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 Processor -- start 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- start start 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: init 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b806f660 0x7f77b8094740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-c con 0x7f77b806f660 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 0x7f77b8093cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-b con 0x7f77b80971f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): picked mon.noname-a con 0x7f77b809d920 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:30:25.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b8094200 con 0x7f77b809d920 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80a07d0 con 0x7f77b80971f0 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b808d230 con 0x7f77b806f660 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bee5f700 10 monclient(hunting): authenticate will time out at 2114.339240s 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b806f660 0x7f77b8094740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46120/0 (socket says 172.21.15.142:46120) 2024-09-18T10:30:25.428 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 -- 172.21.15.142:0/3742296828 learned_addr learned my addr 172.21.15.142:0/3742296828 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): get_auth_request con 0x7f77b809d920 auth_method 0 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): get_auth_request con 0x7f77b806f660 auth_method 0 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 0x7f77b8093cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.429 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bde5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 10 monclient(hunting): get_auth_request con 0x7f77b80971f0 auth_method 0 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient(hunting): handle_auth_done global_id 224346 payload 824 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 -- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b806f660 msgr2=0x7f77b8094740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b806f660 0x7f77b8094740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 -- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 msgr2=0x7f77b8093cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 0x7f77b8093cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient: _finish_hunting 0 2024-09-18T10:30:25.430 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 monclient: found mon.noname-a 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77be65e700 1 -- 172.21.15.142:0/3742296828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0009a20 con 0x7f77b809d920 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.410+0000 7f77bd65c700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 0x7f77b8093cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77be65e700 1 -- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 msgr2=0x7f77b8089690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77be65e700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77b809a920 tx=0x7f77b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00070d0 con 0x7f77b809d920 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: dump: 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.431 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: _finish_auth 0 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 5 monclient: authenticate success, global_id 224346 2024-09-18T10:30:25.432 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b0021070 con 0x7f77b809d920 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 monclient: _renew_subs 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b808cc70 con 0x7f77b809d920 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0026430 con 0x7f77b809d920 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 10 monclient: dump: 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.433 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.434 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 monclient: _renew_subs 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b808bd10 con 0x7f77b809d920 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77af7fe700 4 set_mon_vals no callback set 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7f77b0023030 con 0x7f77b809d920 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.414+0000 7f77bee5f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77bee5f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b8072a80 con 0x7f77b809d920 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77adffb700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 0x7f779c037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.435 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7f77b004b280 con 0x7f77b809d920 2024-09-18T10:30:25.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77bde5d700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 0x7f779c037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77bde5d700 10 monclient: get_auth_request con 0x7f779c035ac0 auth_method 0 2024-09-18T10:30:25.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.418+0000 7f77bde5d700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 0x7f779c037f70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f77b4007880 tx=0x7f77b4007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.426+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7f77b001c420 con 0x7f77b809d920 2024-09-18T10:30:25.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.426+0000 7f77adffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.426+0000 7f77adffb700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:30:25.583 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.578+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 --> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77b808bf90 con 0x7f779c035ac0 2024-09-18T10:30:25.591 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.586+0000 7f77adffb700 1 -- 172.21.15.142:0/3742296828 <== mgr.4100 v2:172.21.15.150:6808/20862 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+98022 (secure 0 0 0) 0x7f77b808bf90 con 0x7f779c035ac0 2024-09-18T10:30:25.593 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 msgr2=0x7f779c037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 0x7f779c037f70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f77b4007880 tx=0x7f77b4007f90 comp rx=0 tx=0).stop 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 10 monclient: shutdown 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 msgr2=0x7f77b8089690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77b809a920 tx=0x7f77b0018000 comp rx=0 tx=0).stop 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 shutdown_connections 2024-09-18T10:30:25.595 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7f779c035ac0 0x7f779c037f70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77b809d920 0x7f77b8089690 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80971f0 0x7f77b8093cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 --2- 172.21.15.142:0/3742296828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77b806f660 0x7f77b8094740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 >> 172.21.15.142:0/3742296828 conn(0x7f77b80a1fe0 msgr2=0x7f77b809fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 shutdown_connections 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.590+0000 7f77bee5f700 1 -- 172.21.15.142:0/3742296828 wait complete. 2024-09-18T10:30:25.596 INFO:teuthology.orchestra.run.smithi142.stderr:dumped all 2024-09-18T10:30:25.608 INFO:teuthology.orchestra.run.smithi142.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-09-18T10:30:25.147310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1820512,"kb_used_data":11440,"kb_used_omap":9,"kb_used_meta":1809014,"kb_avail":564410528,"statfs":{"total":579820584960,"available":577956380672,"internally_reserved":0,"allocated":11714560,"data_stored":9341422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1852430352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003730"},"pg_stats":[{"pgid":"10.4","version":"0'0","reported_seq":185,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.182470+0000","last_change":"2024-09-18T10:29:51.182344+0000","last_active":"2024-09-18T10:29:51.182470+0000","last_peered":"2024-09-18T10:29:51.182470+0000","last_clean":"2024-09-18T10:29:51.182470+0000","last_became_active":"2024-09-18T10:15:10.261808+0000","last_became_peered":"2024-09-18T10:15:10.261808+0000","last_unstale":"2024-09-18T10:29:51.182470+0000","last_undegraded":"2024-09-18T10:29:51.182470+0000","last_fullsized":"2024-09-18T10:29:51.182470+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.182019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.182019+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.182019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:48:38.316819+0000","scrub_duration":0.0039790019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":102,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:55.073494+0000","last_change":"2024-09-18T10:29:55.073414+0000","last_active":"2024-09-18T10:29:55.073494+0000","last_peered":"2024-09-18T10:29:55.073494+0000","last_clean":"2024-09-18T10:29:55.073494+0000","last_became_active":"2024-09-18T10:15:10.259380+0000","last_became_peered":"2024-09-18T10:15:10.259380+0000","last_unstale":"2024-09-18T10:29:55.073494+0000","last_undegraded":"2024-09-18T10:29:55.073494+0000","last_fullsized":"2024-09-18T10:29:55.073494+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:55.073177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:55.073177+0000","last_clean_scrub_stamp":"2024-09-18T10:29:55.073177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:39:45.579691+0000","scrub_duration":0.0032654250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":169,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:45.219039+0000","last_change":"2024-09-18T10:29:45.218956+0000","last_active":"2024-09-18T10:29:45.219039+0000","last_peered":"2024-09-18T10:29:45.219039+0000","last_clean":"2024-09-18T10:29:45.219039+0000","last_became_active":"2024-09-18T10:15:10.262993+0000","last_became_peered":"2024-09-18T10:15:10.262993+0000","last_unstale":"2024-09-18T10:29:45.219039+0000","last_undegraded":"2024-09-18T10:29:45.219039+0000","last_fullsized":"2024-09-18T10:29:45.219039+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:45.218733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:45.218733+0000","last_clean_scrub_stamp":"2024-09-18T10:29:45.218733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:34:35.140002+0000","scrub_duration":0.0038162840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":102,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.114436+0000","last_change":"2024-09-18T10:29:51.114362+0000","last_active":"2024-09-18T10:29:51.114436+0000","last_peered":"2024-09-18T10:29:51.114436+0000","last_clean":"2024-09-18T10:29:51.114436+0000","last_became_active":"2024-09-18T10:15:10.261674+0000","last_became_peered":"2024-09-18T10:15:10.261674+0000","last_unstale":"2024-09-18T10:29:51.114436+0000","last_undegraded":"2024-09-18T10:29:51.114436+0000","last_fullsized":"2024-09-18T10:29:51.114436+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.114155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.114155+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.114155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:16:39.912194+0000","scrub_duration":0.00349906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":143,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:59.091861+0000","last_change":"2024-09-18T10:29:59.091774+0000","last_active":"2024-09-18T10:29:59.091861+0000","last_peered":"2024-09-18T10:29:59.091861+0000","last_clean":"2024-09-18T10:29:59.091861+0000","last_became_active":"2024-09-18T10:15:12.230601+0000","last_became_peered":"2024-09-18T10:15:12.230601+0000","last_unstale":"2024-09-18T10:29:59.091861+0000","last_undegraded":"2024-09-18T10:29:59.091861+0000","last_fullsized":"2024-09-18T10:29:59.091861+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:59.091557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:59.091557+0000","last_clean_scrub_stamp":"2024-09-18T10:29:59.091557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:36.166378+0000","scrub_duration":0.003394163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":118,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.073436+0000","last_change":"2024-09-18T10:29:49.073323+0000","last_active":"2024-09-18T10:29:49.073436+0000","last_peered":"2024-09-18T10:29:49.073436+0000","last_clean":"2024-09-18T10:29:49.073436+0000","last_became_active":"2024-09-18T10:15:10.257895+0000","last_became_peered":"2024-09-18T10:15:10.257895+0000","last_unstale":"2024-09-18T10:29:49.073436+0000","last_undegraded":"2024-09-18T10:29:49.073436+0000","last_fullsized":"2024-09-18T10:29:49.073436+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.073106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.073106+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.073106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:14:09.851749+0000","scrub_duration":0.0034174990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":101,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:45.221737+0000","last_change":"2024-09-18T10:29:45.221645+0000","last_active":"2024-09-18T10:29:45.221737+0000","last_peered":"2024-09-18T10:29:45.221737+0000","last_clean":"2024-09-18T10:29:45.221737+0000","last_became_active":"2024-09-18T10:15:10.268136+0000","last_became_peered":"2024-09-18T10:15:10.268136+0000","last_unstale":"2024-09-18T10:29:45.221737+0000","last_undegraded":"2024-09-18T10:29:45.221737+0000","last_fullsized":"2024-09-18T10:29:45.221737+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:45.221449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:45.221449+0000","last_clean_scrub_stamp":"2024-09-18T10:29:45.221449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:09:15.687705+0000","scrub_duration":0.003519777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:56.054450+0000","last_change":"2024-09-18T10:29:56.054369+0000","last_active":"2024-09-18T10:29:56.054450+0000","last_peered":"2024-09-18T10:29:56.054450+0000","last_clean":"2024-09-18T10:29:56.054450+0000","last_became_active":"2024-09-18T10:15:10.257077+0000","last_became_peered":"2024-09-18T10:15:10.257077+0000","last_unstale":"2024-09-18T10:29:56.054450+0000","last_undegraded":"2024-09-18T10:29:56.054450+0000","last_fullsized":"2024-09-18T10:29:56.054450+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:56.054169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:56.054169+0000","last_clean_scrub_stamp":"2024-09-18T10:29:56.054169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:31:57.167450+0000","scrub_duration":0.0036297059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":126,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:47.142271+0000","last_change":"2024-09-18T10:29:47.142164+0000","last_active":"2024-09-18T10:29:47.142271+0000","last_peered":"2024-09-18T10:29:47.142271+0000","last_clean":"2024-09-18T10:29:47.142271+0000","last_became_active":"2024-09-18T10:15:10.258291+0000","last_became_peered":"2024-09-18T10:15:10.258291+0000","last_unstale":"2024-09-18T10:29:47.142271+0000","last_undegraded":"2024-09-18T10:29:47.142271+0000","last_fullsized":"2024-09-18T10:29:47.142271+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:47.141958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:47.141958+0000","last_clean_scrub_stamp":"2024-09-18T10:29:47.141958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:22:29.529273+0000","scrub_duration":0.003581089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":123,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:56.294228+0000","last_change":"2024-09-18T10:29:56.294134+0000","last_active":"2024-09-18T10:29:56.294228+0000","last_peered":"2024-09-18T10:29:56.294228+0000","last_clean":"2024-09-18T10:29:56.294228+0000","last_became_active":"2024-09-18T10:15:10.252856+0000","last_became_peered":"2024-09-18T10:15:10.252856+0000","last_unstale":"2024-09-18T10:29:56.294228+0000","last_undegraded":"2024-09-18T10:29:56.294228+0000","last_fullsized":"2024-09-18T10:29:56.294228+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:56.293936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:56.293936+0000","last_clean_scrub_stamp":"2024-09-18T10:29:56.293936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:05:38.727118+0000","scrub_duration":0.0034417409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":137,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.690938+0000","last_change":"2024-09-18T10:29:49.690859+0000","last_active":"2024-09-18T10:29:49.690938+0000","last_peered":"2024-09-18T10:29:49.690938+0000","last_clean":"2024-09-18T10:29:49.690938+0000","last_became_active":"2024-09-18T10:15:10.261439+0000","last_became_peered":"2024-09-18T10:15:10.261439+0000","last_unstale":"2024-09-18T10:29:49.690938+0000","last_undegraded":"2024-09-18T10:29:49.690938+0000","last_fullsized":"2024-09-18T10:29:49.690938+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.690661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.690661+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.690661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:08:01.146815+0000","scrub_duration":0.0036244139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:58.992750+0000","last_change":"2024-09-18T10:29:58.992666+0000","last_active":"2024-09-18T10:29:58.992750+0000","last_peered":"2024-09-18T10:29:58.992750+0000","last_clean":"2024-09-18T10:29:58.992750+0000","last_became_active":"2024-09-18T10:15:10.252811+0000","last_became_peered":"2024-09-18T10:15:10.252811+0000","last_unstale":"2024-09-18T10:29:58.992750+0000","last_undegraded":"2024-09-18T10:29:58.992750+0000","last_fullsized":"2024-09-18T10:29:58.992750+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:58.992467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:58.992467+0000","last_clean_scrub_stamp":"2024-09-18T10:29:58.992467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:02:37.866959+0000","scrub_duration":0.0030699339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":97,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:55.097270+0000","last_change":"2024-09-18T10:29:55.097184+0000","last_active":"2024-09-18T10:29:55.097270+0000","last_peered":"2024-09-18T10:29:55.097270+0000","last_clean":"2024-09-18T10:29:55.097270+0000","last_became_active":"2024-09-18T10:15:10.264378+0000","last_became_peered":"2024-09-18T10:15:10.264378+0000","last_unstale":"2024-09-18T10:29:55.097270+0000","last_undegraded":"2024-09-18T10:29:55.097270+0000","last_fullsized":"2024-09-18T10:29:55.097270+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:55.096976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:55.096976+0000","last_clean_scrub_stamp":"2024-09-18T10:29:55.096976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:08:42.920133+0000","scrub_duration":0.0036152390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":192,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:44.219576+0000","last_change":"2024-09-18T10:29:44.219488+0000","last_active":"2024-09-18T10:29:44.219576+0000","last_peered":"2024-09-18T10:29:44.219576+0000","last_clean":"2024-09-18T10:29:44.219576+0000","last_became_active":"2024-09-18T10:15:10.262017+0000","last_became_peered":"2024-09-18T10:15:10.262017+0000","last_unstale":"2024-09-18T10:29:44.219576+0000","last_undegraded":"2024-09-18T10:29:44.219576+0000","last_fullsized":"2024-09-18T10:29:44.219576+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:44.219192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:44.219192+0000","last_clean_scrub_stamp":"2024-09-18T10:29:44.219192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:23.294288+0000","scrub_duration":0.0030681720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":99,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:51.238692+0000","last_change":"2024-09-18T10:29:51.238602+0000","last_active":"2024-09-18T10:29:51.238692+0000","last_peered":"2024-09-18T10:29:51.238692+0000","last_clean":"2024-09-18T10:29:51.238692+0000","last_became_active":"2024-09-18T10:15:11.672348+0000","last_became_peered":"2024-09-18T10:15:11.672348+0000","last_unstale":"2024-09-18T10:29:51.238692+0000","last_undegraded":"2024-09-18T10:29:51.238692+0000","last_fullsized":"2024-09-18T10:29:51.238692+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":126,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:51.238417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:51.238417+0000","last_clean_scrub_stamp":"2024-09-18T10:29:51.238417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:48:05.601926+0000","scrub_duration":0.003583433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":96,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:50.120399+0000","last_change":"2024-09-18T10:29:50.120300+0000","last_active":"2024-09-18T10:29:50.120399+0000","last_peered":"2024-09-18T10:29:50.120399+0000","last_clean":"2024-09-18T10:29:50.120399+0000","last_became_active":"2024-09-18T10:15:10.259961+0000","last_became_peered":"2024-09-18T10:15:10.259961+0000","last_unstale":"2024-09-18T10:29:50.120399+0000","last_undegraded":"2024-09-18T10:29:50.120399+0000","last_fullsized":"2024-09-18T10:29:50.120399+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:50.120093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:50.120093+0000","last_clean_scrub_stamp":"2024-09-18T10:29:50.120093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:05:27.565756+0000","scrub_duration":0.0034736680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"264'74","reported_seq":346,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:53.189845+0000","last_change":"2024-09-18T10:29:53.189775+0000","last_active":"2024-09-18T10:29:53.189845+0000","last_peered":"2024-09-18T10:29:53.189845+0000","last_clean":"2024-09-18T10:29:53.189845+0000","last_became_active":"2024-09-18T10:15:10.257452+0000","last_became_peered":"2024-09-18T10:15:10.257452+0000","last_unstale":"2024-09-18T10:29:53.189845+0000","last_undegraded":"2024-09-18T10:29:53.189845+0000","last_fullsized":"2024-09-18T10:29:53.189845+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'74","last_scrub_stamp":"2024-09-18T10:29:53.189579+0000","last_deep_scrub":"264'74","last_deep_scrub_stamp":"2024-09-18T10:29:53.189579+0000","last_clean_scrub_stamp":"2024-09-18T10:29:53.189579+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:01:39.106255+0000","scrub_duration":0.0094258559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"264'2","reported_seq":274,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.111710+0000","last_change":"2024-09-18T10:29:49.111606+0000","last_active":"2024-09-18T10:29:49.111710+0000","last_peered":"2024-09-18T10:29:49.111710+0000","last_clean":"2024-09-18T10:29:49.111710+0000","last_became_active":"2024-09-18T10:15:10.262725+0000","last_became_peered":"2024-09-18T10:15:10.262725+0000","last_unstale":"2024-09-18T10:29:49.111710+0000","last_undegraded":"2024-09-18T10:29:49.111710+0000","last_fullsized":"2024-09-18T10:29:49.111710+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'2","last_scrub_stamp":"2024-09-18T10:29:49.111335+0000","last_deep_scrub":"264'2","last_deep_scrub_stamp":"2024-09-18T10:29:49.111335+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.111335+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:43:18.072502+0000","scrub_duration":0.0063368210000000003,"objects_trimmed":0,"snaptrim_duration":0.0022918579999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":287,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:44.242136+0000","last_change":"2024-09-18T10:29:44.242047+0000","last_active":"2024-09-18T10:29:44.242136+0000","last_peered":"2024-09-18T10:29:44.242136+0000","last_clean":"2024-09-18T10:29:44.242136+0000","last_became_active":"2024-09-18T10:15:10.254207+0000","last_became_peered":"2024-09-18T10:15:10.254207+0000","last_unstale":"2024-09-18T10:29:44.242136+0000","last_undegraded":"2024-09-18T10:29:44.242136+0000","last_fullsized":"2024-09-18T10:29:44.242136+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:44.241720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:44.241720+0000","last_clean_scrub_stamp":"2024-09-18T10:29:44.241720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.129303+0000","scrub_duration":0.003522074,"objects_trimmed":0,"snaptrim_duration":0.00061021000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:49.275452+0000","last_change":"2024-09-18T10:29:49.275386+0000","last_active":"2024-09-18T10:29:49.275452+0000","last_peered":"2024-09-18T10:29:49.275452+0000","last_clean":"2024-09-18T10:29:49.275452+0000","last_became_active":"2024-09-18T10:15:10.257667+0000","last_became_peered":"2024-09-18T10:15:10.257667+0000","last_unstale":"2024-09-18T10:29:49.275452+0000","last_undegraded":"2024-09-18T10:29:49.275452+0000","last_fullsized":"2024-09-18T10:29:49.275452+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:49.275204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:49.275204+0000","last_clean_scrub_stamp":"2024-09-18T10:29:49.275204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:32:28.923431+0000","scrub_duration":0.00290369,"objects_trimmed":0,"snaptrim_duration":0.00064360999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"264'3","reported_seq":268,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:46.150056+0000","last_change":"2024-09-18T10:29:46.149987+0000","last_active":"2024-09-18T10:29:46.150056+0000","last_peered":"2024-09-18T10:29:46.150056+0000","last_clean":"2024-09-18T10:29:46.150056+0000","last_became_active":"2024-09-18T10:15:10.260690+0000","last_became_peered":"2024-09-18T10:15:10.260690+0000","last_unstale":"2024-09-18T10:29:46.150056+0000","last_undegraded":"2024-09-18T10:29:46.150056+0000","last_fullsized":"2024-09-18T10:29:46.150056+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'3","last_scrub_stamp":"2024-09-18T10:29:46.149793+0000","last_deep_scrub":"264'3","last_deep_scrub_stamp":"2024-09-18T10:29:46.149793+0000","last_clean_scrub_stamp":"2024-09-18T10:29:46.149793+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:14:43.812460+0000","scrub_duration":0.006735843,"objects_trimmed":0,"snaptrim_duration":0.0020439590000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":250,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:30:04.076597+0000","last_change":"2024-09-18T10:30:04.076533+0000","last_active":"2024-09-18T10:30:04.076597+0000","last_peered":"2024-09-18T10:30:04.076597+0000","last_clean":"2024-09-18T10:30:04.076597+0000","last_became_active":"2024-09-18T10:15:12.227653+0000","last_became_peered":"2024-09-18T10:15:12.227653+0000","last_unstale":"2024-09-18T10:30:04.076597+0000","last_undegraded":"2024-09-18T10:30:04.076597+0000","last_fullsized":"2024-09-18T10:30:04.076597+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:30:04.076321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:30:04.076321+0000","last_clean_scrub_stamp":"2024-09-18T10:30:04.076321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:04:52.302699+0000","scrub_duration":0.0032831090000000002,"objects_trimmed":0,"snaptrim_duration":0.00061341300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":100,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:52.233099+0000","last_change":"2024-09-18T10:29:52.233010+0000","last_active":"2024-09-18T10:29:52.233099+0000","last_peered":"2024-09-18T10:29:52.233099+0000","last_clean":"2024-09-18T10:29:52.233099+0000","last_became_active":"2024-09-18T10:15:10.262800+0000","last_became_peered":"2024-09-18T10:15:10.262800+0000","last_unstale":"2024-09-18T10:29:52.233099+0000","last_undegraded":"2024-09-18T10:29:52.233099+0000","last_fullsized":"2024-09-18T10:29:52.233099+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:52.232808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:52.232808+0000","last_clean_scrub_stamp":"2024-09-18T10:29:52.232808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:19:01.369564+0000","scrub_duration":0.0033643459999999998,"objects_trimmed":0,"snaptrim_duration":0.00072818100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":118,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:47.043374+0000","last_change":"2024-09-18T10:29:47.043284+0000","last_active":"2024-09-18T10:29:47.043374+0000","last_peered":"2024-09-18T10:29:47.043374+0000","last_clean":"2024-09-18T10:29:47.043374+0000","last_became_active":"2024-09-18T10:14:59.172164+0000","last_became_peered":"2024-09-18T10:14:59.172164+0000","last_unstale":"2024-09-18T10:29:47.043374+0000","last_undegraded":"2024-09-18T10:29:47.043374+0000","last_fullsized":"2024-09-18T10:29:47.043374+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:47.043063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:47.043063+0000","last_clean_scrub_stamp":"2024-09-18T10:29:47.043063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T13:33:03.128406+0000","scrub_duration":0.003472569,"objects_trimmed":0,"snaptrim_duration":0.00063124100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-18T10:29:59.337315+0000","last_change":"2024-09-18T10:29:59.337217+0000","last_active":"2024-09-18T10:29:59.337315+0000","last_peered":"2024-09-18T10:29:59.337315+0000","last_clean":"2024-09-18T10:29:59.337315+0000","last_became_active":"2024-09-18T10:15:10.257264+0000","last_became_peered":"2024-09-18T10:15:10.257264+0000","last_unstale":"2024-09-18T10:29:59.337315+0000","last_undegraded":"2024-09-18T10:29:59.337315+0000","last_fullsized":"2024-09-18T10:29:59.337315+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T10:29:59.337006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T10:29:59.337006+0000","last_clean_scrub_stamp":"2024-09-18T10:29:59.337006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:18:06.732291+0000","scrub_duration":0.0032446089999999999,"objects_trimmed":0,"snaptrim_duration":0.0010025679999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":6},{"poolid":9,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54300000000000004,"15min":0.57099999999999995},"min":{"1min":0.42199999999999999,"5min":0.41299999999999998,"15min":0.372},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":1.268},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.73299999999999998,"15min":0.74099999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.46100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98499999999999999,"15min":1.522},"last":0.88900000000000001}]},{"osd":1,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.58299999999999996,"15min":0.67100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72199999999999998,"5min":0.86799999999999999,"15min":1.3979999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.41199999999999998},"max":{"1min":0.80300000000000005,"5min":0.98999999999999999,"15min":1.081},"last":0.70899999999999996}]},{"osd":3,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.81399999999999995,"15min":0.873},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96099999999999997,"5min":1.1879999999999999,"15min":1.875},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.83299999999999996,"15min":0.89700000000000002},"min":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.048,"5min":1.2150000000000001,"15min":1.609},"last":0.76500000000000001}]},{"osd":4,"last update":"Wed Sep 18 10:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.88,"15min":0.86299999999999999},"min":{"1min":0.65300000000000002,"5min":0.58599999999999997,"15min":0.57899999999999996},"max":{"1min":1.0349999999999999,"5min":1.214,"15min":1.242},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.92100000000000004,"15min":0.88900000000000001},"min":{"1min":0.70499999999999996,"5min":0.67700000000000005,"15min":0.64400000000000002},"max":{"1min":1.0960000000000001,"5min":1.4910000000000001,"15min":1.8200000000000001},"last":1.0169999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.85999999999999999,"15min":0.879},"min":{"1min":0.74299999999999999,"5min":0.69599999999999995,"15min":0.66100000000000003},"max":{"1min":1.121,"5min":1.121,"15min":1.3740000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94199999999999995,"15min":0.97899999999999998},"min":{"1min":0.76900000000000002,"5min":0.72699999999999998,"15min":0.70299999999999996},"max":{"1min":0.996,"5min":1.393,"15min":1.704},"last":0.98899999999999999}]}]},{"osd":1,"up_from":12,"seq":51539607813,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.46500000000000002,"5min":0.435,"15min":0.30199999999999999},"max":{"1min":0.76800000000000002,"5min":1.079,"15min":1.079},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.64100000000000001,"15min":0.64500000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.38800000000000001},"max":{"1min":0.753,"5min":1.1359999999999999,"15min":1.528},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.65600000000000003,"15min":0.66700000000000004},"min":{"1min":0.52200000000000002,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.77400000000000002,"5min":0.999,"15min":1.0549999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67900000000000005,"15min":0.71699999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93799999999999994,"5min":0.95299999999999996,"15min":1.1160000000000001},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.72199999999999998,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":1.008,"5min":1.2589999999999999,"15min":1.377},"last":0.879},{"interface":"front","average":{"1min":0.87,"5min":0.876,"15min":0.85099999999999998},"min":{"1min":0.66300000000000003,"5min":0.64700000000000002,"15min":0.48899999999999999},"max":{"1min":1.0309999999999999,"5min":1.163,"15min":1.5589999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.89500000000000002,"15min":0.875},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.45100000000000001},"max":{"1min":1.127,"5min":1.222,"15min":1.5960000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.89800000000000002,"15min":0.90400000000000003},"min":{"1min":0.73799999999999999,"5min":0.69599999999999995,"15min":0.66200000000000003},"max":{"1min":0.98199999999999998,"5min":1.3,"15min":1.49},"last":0.85699999999999998}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91500000000000004,"15min":0.89400000000000002},"min":{"1min":0.72799999999999998,"5min":0.72299999999999998,"15min":0.57399999999999995},"max":{"1min":1.0129999999999999,"5min":1.423,"15min":1.423},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90700000000000003,"15min":0.91400000000000003},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.53300000000000003},"max":{"1min":1.038,"5min":1.331,"15min":1.639},"last":0.95899999999999996}]}]},{"osd":5,"up_from":11,"seq":47244640515,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303248,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068592,"statfs":{"total":96636764160,"available":96326238208,"internally_reserved":0,"allocated":1777664,"data_stored":1380607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.80700000000000005,"15min":0.78400000000000003},"min":{"1min":0.505,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0089999999999999,"5min":1.0489999999999999,"15min":1.0569999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.85099999999999998,"15min":0.85699999999999998},"min":{"1min":0.53600000000000003,"5min":0.504,"15min":0.504},"max":{"1min":1.018,"5min":1.222,"15min":2.4460000000000002},"last":0.97699999999999998}]},{"osd":1,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.079,"5min":1.079,"15min":1.4430000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93200000000000005,"15min":0.88100000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.53100000000000003},"max":{"1min":1.278,"5min":1.399,"15min":2.3799999999999999},"last":0.95499999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79600000000000004,"15min":0.82899999999999996},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0129999999999999,"5min":1.2789999999999999,"15min":1.3169999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.83499999999999996,"15min":0.88},"min":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.96999999999999997,"5min":1.3140000000000001,"15min":1.3149999999999999},"last":0.76200000000000001}]},{"osd":3,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.42499999999999999},"max":{"1min":0.96399999999999997,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.84999999999999998,"15min":0.88200000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.45700000000000002},"max":{"1min":1.2110000000000001,"5min":1.3600000000000001,"15min":1.556},"last":0.86799999999999999}]},{"osd":4,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.95599999999999996,"15min":1.0069999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.095,"5min":1.2370000000000001,"15min":1.653},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72399999999999998,"15min":0.77600000000000002},"min":{"1min":0.58099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.05,"5min":1.103,"15min":2.327},"last":0.83199999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640517,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303244,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068596,"statfs":{"total":96636764160,"available":96326242304,"internally_reserved":0,"allocated":1773568,"data_stored":1380588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:40 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.60399999999999998,"5min":0.59999999999999998,"15min":0.50700000000000001},"max":{"1min":0.90100000000000002,"5min":1.044,"15min":1.347},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.85899999999999999,"15min":0.85499999999999998},"min":{"1min":0.54100000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.997,"5min":1.2030000000000001,"15min":1.3879999999999999},"last":0.93200000000000005}]},{"osd":1,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.83099999999999996,"15min":0.79300000000000004},"min":{"1min":0.63600000000000001,"5min":0.57499999999999996,"15min":0.54800000000000004},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.81200000000000006,"15min":0.83699999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.89700000000000002,"5min":1.1779999999999999,"15min":2.0339999999999998},"last":0.83399999999999996}]},{"osd":2,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.79600000000000004,"15min":0.80600000000000005},"min":{"1min":0.65200000000000002,"5min":0.61599999999999999,"15min":0.57499999999999996},"max":{"1min":0.98099999999999998,"5min":1.052,"15min":1.3180000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.0229999999999999,"15min":0.98099999999999998},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.59599999999999997},"max":{"1min":1.232,"5min":1.671,"15min":1.871},"last":0.97599999999999998}]},{"osd":3,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.47399999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.2789999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":1.1639999999999999,"15min":1.976},"last":0.69899999999999995}]},{"osd":5,"last update":"Wed Sep 18 10:29:34 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.002,"15min":0.96799999999999997},"min":{"1min":0.69099999999999995,"5min":0.623,"15min":0.53600000000000003},"max":{"1min":1.1459999999999999,"5min":1.6259999999999999,"15min":2.085},"last":0.877},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.82599999999999996,"15min":0.871},"min":{"1min":0.623,"5min":0.56599999999999995,"15min":0.39200000000000002},"max":{"1min":1.117,"5min":1.1859999999999999,"15min":1.3089999999999999},"last":0.64900000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640517,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303764,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068076,"statfs":{"total":96636764160,"available":96325709824,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.78600000000000003,"15min":0.80300000000000005},"min":{"1min":0.57099999999999995,"5min":0.53300000000000003,"15min":0.52400000000000002},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":1.5089999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89200000000000002,"15min":0.89700000000000002},"min":{"1min":0.66100000000000003,"5min":0.47799999999999998,"15min":0.443},"max":{"1min":1.131,"5min":1.2170000000000001,"15min":1.4610000000000001},"last":0.84899999999999998}]},{"osd":1,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78900000000000003,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.51100000000000001},"max":{"1min":0.93500000000000005,"5min":1.0920000000000001,"15min":1.21},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2,"5min":1.2,"15min":1.3700000000000001},"last":0.89500000000000002}]},{"osd":2,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83599999999999997,"15min":0.83099999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.1950000000000001,"5min":1.353,"15min":1.353},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.86699999999999999,"15min":0.879},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.54500000000000004},"max":{"1min":1.242,"5min":1.242,"15min":1.425},"last":0.79900000000000004}]},{"osd":4,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.73299999999999998,"15min":0.72199999999999998},"min":{"1min":0.56499999999999995,"5min":0.52800000000000002,"15min":0.40000000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.371},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.76800000000000002,"15min":0.75600000000000001},"min":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.34599999999999997},"max":{"1min":1.266,"5min":1.29,"15min":1.29},"last":0.61199999999999999}]},{"osd":5,"last update":"Wed Sep 18 10:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.85499999999999998,"15min":0.82299999999999995},"min":{"1min":0.70399999999999996,"5min":0.58099999999999996,"15min":0.45200000000000001},"max":{"1min":0.96899999999999997,"5min":1.2470000000000001,"15min":1.3260000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.93000000000000005,"15min":0.93899999999999995},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.53600000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.5600000000000001},"last":0.72899999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640517,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303764,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94068076,"statfs":{"total":96636764160,"available":96325709824,"internally_reserved":0,"allocated":2306048,"data_stored":1909516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":0.80000000000000004,"5min":0.86599999999999999,"15min":1.0629999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.47299999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.495,"5min":0.40100000000000002,"15min":0.34899999999999998},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":1.1459999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65800000000000003,"15min":0.68000000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92700000000000005,"15min":0.91200000000000003},"min":{"1min":0.69399999999999995,"5min":0.65200000000000002,"15min":0.50800000000000001},"max":{"1min":1.0920000000000001,"5min":1.2310000000000001,"15min":1.2430000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97399999999999998,"15min":0.93799999999999994},"min":{"1min":0.85199999999999998,"5min":0.71499999999999997,"15min":0.61399999999999999},"max":{"1min":1.1970000000000001,"5min":1.27,"15min":1.347},"last":0.93899999999999995}]},{"osd":4,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.85199999999999998,"15min":0.84599999999999997},"min":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.54200000000000004},"max":{"1min":1.125,"5min":1.1830000000000001,"15min":1.2729999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.86899999999999999,"15min":0.87},"min":{"1min":0.71999999999999997,"5min":0.68100000000000005,"15min":0.55300000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.3899999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Wed Sep 18 10:29:39 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.871,"15min":0.89600000000000002},"min":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.60899999999999999},"max":{"1min":1.1719999999999999,"5min":1.1950000000000001,"15min":1.333},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90100000000000002,"15min":0.94599999999999995},"min":{"1min":0.74199999999999999,"5min":0.69899999999999995,"15min":0.66400000000000003},"max":{"1min":1.2270000000000001,"5min":1.252,"15min":1.446},"last":0.82999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T10:30:25.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T10:30:25.717 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: get_monmap_and_config 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: build_initial_monmap 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: monmap: 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:30:25.716055+0000 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:30:25.716055+0000 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:0: v1:172.21.15.142:6789/0 mon.noname-a-legacy 2024-09-18T10:30:25.718 INFO:teuthology.orchestra.run.smithi142.stderr:1: v1:172.21.15.150:6789/0 mon.noname-c-legacy 2024-09-18T10:30:25.719 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-b 2024-09-18T10:30:25.719 INFO:teuthology.orchestra.run.smithi142.stderr:3: v2:172.21.15.142:3300/0 mon.noname-a 2024-09-18T10:30:25.719 INFO:teuthology.orchestra.run.smithi142.stderr:4: v2:172.21.15.150:3300/0 mon.noname-c 2024-09-18T10:30:25.719 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 Processor -- start 2024-09-18T10:30:25.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 -- start start 2024-09-18T10:30:25.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: init 2024-09-18T10:30:25.721 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff17412f1f0 addr v1:172.21.15.150:6789/0 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1740a11d0 0x7ff17409cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff1740a11d0 addr v2:172.21.15.150:3300/0 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff17409d120 addr v1:172.21.15.142:6789/0 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1740a1ba0 con 0x7ff17409d120 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff17412fcd0 con 0x7ff17412f1f0 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff174068020 con 0x7ff1740a11d0 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff173fff700 10 monclient(hunting): authenticate will time out at 2114.642072s 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1737fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff17409d120 0x7ff174139340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:42534/0 (socket says 172.21.15.142:42534) 2024-09-18T10:30:25.722 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1737fe700 1 -- 172.21.15.142:0/1893504739 learned_addr learned my addr 172.21.15.142:0/1893504739 (peer_addr_for_me v1:172.21.15.142:0/0) 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 1 --2- 172.21.15.142:0/1893504739 >> v2:172.21.15.150:3300/0 conn(0x7ff1740a11d0 0x7ff17409cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): get_auth_request con 0x7ff1740a11d0 auth_method 0 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient(hunting): handle_auth_done global_id 194710 payload 274 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 1 -- 172.21.15.142:0/1893504739 >> v1:172.21.15.150:6789/0 conn(0x7ff17412f1f0 legacy=0x7ff17412f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 1 -- 172.21.15.142:0/1893504739 >> v1:172.21.15.142:6789/0 conn(0x7ff17409d120 legacy=0x7ff174139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-18T10:30:25.723 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient: _finish_hunting 0 2024-09-18T10:30:25.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 1 monclient: found mon.noname-c 2024-09-18T10:30:25.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:25.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.150:3300/0 2024-09-18T10:30:25.724 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.714+0000 7ff1727fc700 1 -- 172.21.15.142:0/1893504739 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740a1830 con 0x7ff1740a11d0 2024-09-18T10:30:25.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff1727fc700 1 -- 172.21.15.142:0/1893504739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1740a11d0 msgr2=0x7ff17409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:25.725 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff1727fc700 1 --2- 172.21.15.142:0/1893504739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1740a11d0 0x7ff17409cbe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff1640098e0 tx=0x7ff1640091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c609a3c62da4c38 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 1 -- 172.21.15.142:0/1893504739 <== mon.1 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16400ba70 con 0x7ff1740a11d0 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: dump: 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.726 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: _finish_auth 0 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 5 monclient: authenticate success, global_id 194710 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 1 -- 172.21.15.142:0/1893504739 <== mon.1 v2:172.21.15.150:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16400bbd0 con 0x7ff1740a11d0 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 10 monclient: get_monmap_and_config success 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 1 -- 172.21.15.142:0/1893504739 <== mon.1 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164011420 con 0x7ff1740a11d0 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 4 set_mon_vals no callback set 2024-09-18T10:30:25.727 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff171ffb700 10 monclient: dump: 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.728 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 10 monclient: shutdown 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 -- 172.21.15.142:0/1893504739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1740a11d0 msgr2=0x7ff17409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 --2- 172.21.15.142:0/1893504739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1740a11d0 0x7ff17409cbe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff1640098e0 tx=0x7ff1640091a0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 -- 172.21.15.142:0/1893504739 shutdown_connections 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 --2- 172.21.15.142:0/1893504739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1740a11d0 0x7ff17409cbe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 -- 172.21.15.142:0/1893504739 >> 172.21.15.142:0/1893504739 conn(0x7ff1740a5700 msgr2=0x7ff1740a5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 -- 172.21.15.142:0/1893504739 shutdown_connections 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 1 -- 172.21.15.142:0/1893504739 wait complete. 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 10 monclient: build_initial_monmap 2024-09-18T10:30:25.729 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.718+0000 7ff173fff700 10 monclient: monmap: 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 0 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 0.000000 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:created 0.000000 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 0 (unknown) 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.noname-a 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.noname-b 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.noname-c 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 Processor -- start 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 -- start start 2024-09-18T10:30:25.730 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient: init 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient: _reopen_session rank -1 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff17409d120 0x7ff174139910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-c con 0x7ff17409d120 addr [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-a con 0x7ff1740a11d0 addr [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-09-18T10:30:25.736 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17412f1f0 0x7ff174131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): picked mon.noname-b con 0x7ff17412f1f0 addr [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff174133500 con 0x7ff1740a11d0 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff174135700 con 0x7ff17412f1f0 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): start opening mon connection 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17409f5c0 con 0x7ff17409d120 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): _renew_subs 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff173fff700 10 monclient(hunting): authenticate will time out at 2114.648870s 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:46126/0 (socket says 172.21.15.142:46126) 2024-09-18T10:30:25.737 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff17409d120 0x7ff174139910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 -- 172.21.15.142:0/1787505170 learned_addr learned my addr 172.21.15.142:0/1787505170 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): get_auth_request con 0x7ff1740a11d0 auth_method 0 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): get_auth_request con 0x7ff17409d120 auth_method 0 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17412f1f0 0x7ff174131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.738 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): get_auth_request con 0x7ff17412f1f0 auth_method 0 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): _init_auth method 2 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff172ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient(hunting): handle_auth_done global_id 224349 payload 824 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 -- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff17409d120 msgr2=0x7ff174139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.739 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff17409d120 0x7ff174139910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 -- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17412f1f0 msgr2=0x7ff174131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17412f1f0 0x7ff174131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient: _finish_hunting 0 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 monclient: found mon.noname-a 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.722+0000 7ff1727fc700 1 -- 172.21.15.142:0/1787505170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff164009420 con 0x7ff1740a11d0 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff1727fc700 1 -- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 msgr2=0x7ff174131320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff1727fc700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1640030c0 tx=0x7ff164016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff164011420 con 0x7ff1740a11d0 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-18T10:30:25.740 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: dump: 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: _finish_auth 0 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-18T10:30:25.741 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 5 monclient: authenticate success, global_id 224349 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff164020070 con 0x7ff1740a11d0 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: handle_config config(0 keys) v1 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16401b660 con 0x7ff1740a11d0 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _renew_subs 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1740a4b50 con 0x7ff1740a11d0 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff1717fa700 4 set_mon_vals no callback set 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 10 monclient: dump: 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:epoch 1 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:fsid bfa92b4c-6f79-422f-bfaa-515c68f959fb 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:last_changed 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.742 INFO:teuthology.orchestra.run.smithi142.stderr:created 2024-09-18T10:08:19.331585+0000 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:min_mon_release 17 (quincy) 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:election_strategy: 1 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:0: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.a 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:1: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.b 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _renew_subs 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff174133000 con 0x7ff1740a11d0 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.743 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff174133000 con 0x7ff1740a11d0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 42798+0+0 (secure 0 0 0) 0x7ff164022030 con 0x7ff1740a11d0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 0x7ff160037f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 5 ==== osd_map(264..264 src has 239..264) v4 ==== 7729+0+0 (secure 0 0 0) 0x7ff16404b150 con 0x7ff1740a11d0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff172ffd700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 0x7ff160037f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.726+0000 7ff172ffd700 10 monclient: get_auth_request con 0x7ff160035ac0 auth_method 0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.730+0000 7ff172ffd700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 0x7ff160037f70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff1680039a0 tx=0x7ff168007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.734+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169563 (secure 0 0 0) 0x7ff1640105b0 con 0x7ff1740a11d0 2024-09-18T10:30:25.744 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.734+0000 7ff15affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-18T10:30:25.745 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.734+0000 7ff15affd700 10 monclient: _finish_command 1 = system:0 2024-09-18T10:30:25.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.898+0000 7ff173fff700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T10:30:25.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.898+0000 7ff173fff700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T10:30:25.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.898+0000 7ff173fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.142:3300/0 2024-09-18T10:30:25.903 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.898+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff174133000 con 0x7ff1740a11d0 2024-09-18T10:30:25.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff1640105b0 con 0x7ff1740a11d0 2024-09-18T10:30:25.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff15affd700 10 monclient: handle_config config(1 keys) v1 2024-09-18T10:30:25.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff1717fa700 4 set_mon_vals no callback set 2024-09-18T10:30:25.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff1717fa700 10 set_mon_vals mon_health_to_clog = false 2024-09-18T10:30:25.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff15affd700 1 -- 172.21.15.142:0/1787505170 <== mon.0 v2:172.21.15.142:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff16401ab30 con 0x7ff1740a11d0 2024-09-18T10:30:25.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff15affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-18T10:30:25.909 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.902+0000 7ff15affd700 10 monclient: _finish_command 2 = system:0 2024-09-18T10:30:25.911 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 msgr2=0x7ff160037f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 0x7ff160037f70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff1680039a0 tx=0x7ff168007ea0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 10 monclient: shutdown 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 msgr2=0x7ff174131320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1640030c0 tx=0x7ff164016000 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 shutdown_connections 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:6808/20862,v1:172.21.15.150:6809/20862] conn(0x7ff160035ac0 0x7ff160037f70 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff17412f1f0 0x7ff174131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1740a11d0 0x7ff174131320 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 --2- 172.21.15.142:0/1787505170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff17409d120 0x7ff174139910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-18T10:30:25.912 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 >> 172.21.15.142:0/1787505170 conn(0x7ff1740a5700 msgr2=0x7ff17413a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-18T10:30:25.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 shutdown_connections 2024-09-18T10:30:25.913 INFO:teuthology.orchestra.run.smithi142.stderr:2024-09-18T10:30:25.906+0000 7ff173fff700 1 -- 172.21.15.142:0/1787505170 wait complete. 2024-09-18T10:30:25.923 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T10:30:25.923 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T10:30:25.923 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T10:30:25.923 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.004 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T10:30:26.004 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T10:30:26.005 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.079 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T10:30:26.080 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T10:30:26.080 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.156 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T10:30:26.157 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T10:30:26.157 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.234 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T10:30:26.234 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-18T10:30:26.235 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.321 INFO:tasks.ceph.osd.4:Stopped 2024-09-18T10:30:26.322 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-18T10:30:26.322 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.395 INFO:tasks.ceph.osd.5:Stopped 2024-09-18T10:30:26.395 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T10:30:26.395 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T10:30:26.395 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.424 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T10:30:26.424 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T10:30:26.425 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T10:30:26.425 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.445 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T10:30:26.445 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T10:30:26.445 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.462 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T10:30:26.462 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T10:30:26.462 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T10:30:26.485 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T10:30:26.485 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T10:30:26.486 DEBUG:teuthology.orchestra.run.smithi142:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2024-09-18T10:30:26.509 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:30:26.510 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T10:30:26.842 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:30:26.843 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T10:30:27.185 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:30:27.186 DEBUG:teuthology.orchestra.run.smithi142:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T10:30:27.367 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:30:27.368 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T10:30:27.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:30:27.548 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-18T10:30:27.874 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:30:27.874 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-18T10:30:28.049 INFO:tasks.ceph:Archiving mon data... 2024-09-18T10:30:28.051 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/data/mon.a.tgz 2024-09-18T10:30:28.051 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-09-18T10:30:28.058 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.A7AEG8Sdn2 2024-09-18T10:30:28.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.A7AEG8Sdn2 2024-09-18T10:30:28.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.A7AEG8Sdn2 2024-09-18T10:30:28.274 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.A7AEG8Sdn2 is 805KB 2024-09-18T10:30:28.338 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.A7AEG8Sdn2 2024-09-18T10:30:28.345 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/data/mon.c.tgz 2024-09-18T10:30:28.345 DEBUG:teuthology.orchestra.run.smithi142:> mktemp 2024-09-18T10:30:28.399 INFO:teuthology.orchestra.run.smithi142.stdout:/tmp/tmp.juzUM5NgII 2024-09-18T10:30:28.399 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.juzUM5NgII 2024-09-18T10:30:28.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0666 /tmp/tmp.juzUM5NgII 2024-09-18T10:30:28.559 DEBUG:teuthology.orchestra.remote:smithi142:/tmp/tmp.juzUM5NgII is 801KB 2024-09-18T10:30:28.628 DEBUG:teuthology.orchestra.run.smithi142:> rm -fr /tmp/tmp.juzUM5NgII 2024-09-18T10:30:28.634 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/data/mon.b.tgz 2024-09-18T10:30:28.634 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-09-18T10:30:28.641 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.TDDHm3Tq5J 2024-09-18T10:30:28.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.TDDHm3Tq5J 2024-09-18T10:30:28.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.TDDHm3Tq5J 2024-09-18T10:30:28.813 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.TDDHm3Tq5J is 802KB 2024-09-18T10:30:28.838 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.TDDHm3Tq5J 2024-09-18T10:30:28.845 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T10:30:28.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T10:30:28.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T10:30:28.905 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T10:30:28.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T10:30:28.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T10:30:28.937 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T10:30:28.939 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T10:30:28.945 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T10:30:28.947 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi142/crash 2024-09-18T10:30:28.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T10:30:28.966 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi150/crash 2024-09-18T10:30:28.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T10:30:28.984 INFO:tasks.ceph:Compressing logs... 2024-09-18T10:30:28.984 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T10:30:29.011 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T10:30:29.045 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23554.log 2024-09-18T10:30:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31566.log 2024-09-18T10:30:29.046 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25544.log 2024-09-18T10:30:29.047 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31701.log 2024-09-18T10:30:29.047 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31566.log: /var/log/ceph/ceph-client.admin.25544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34632.log 2024-09-18T10:30:29.048 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.23554.log.gz 2024-09-18T10:30:29.048 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27838.log 2024-09-18T10:30:29.048 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31566.log.gz 2024-09-18T10:30:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31701.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.25544.log.gz 2024-09-18T10:30:29.049 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30659.log 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34632.log: /var/log/ceph/ceph-client.admin.27838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30415.log 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31701.log.gz 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34292.log 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34632.log.gz 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30659.log: 89.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.27838.log.gz-5 2024-09-18T10:30:29.050 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.28535.log 2024-09-18T10:30:29.051 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30415.log: /var/log/ceph/ceph-client.admin.34292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29589.log 2024-09-18T10:30:29.051 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30659.log.gz 2024-09-18T10:30:29.051 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30415.log.gz 2024-09-18T10:30:29.053 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28562.log 2024-09-18T10:30:29.053 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33790.log 2024-09-18T10:30:29.053 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.34292.log.gz 2024-09-18T10:30:29.053 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29589.log: /var/log/ceph/ceph-client.admin.28562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26266.log 2024-09-18T10:30:29.054 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32423.log 2024-09-18T10:30:29.054 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28535.log.gz 2024-09-18T10:30:29.054 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29589.log.gz 2024-09-18T10:30:29.054 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28562.log.gz 2024-09-18T10:30:29.054 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25571.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.26266.log.gz 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30064.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32423.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33790.log.gz 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25571.log: /var/log/ceph/ceph-client.admin.30064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33412.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.32423.log.gz 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30064.log.gz 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26401.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32666.log 2024-09-18T10:30:29.057 INFO:teuthology.orchestra.run.smithi142.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.25571.log.gz 2024-09-18T10:30:29.061 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27346.log 2024-09-18T10:30:29.061 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26401.log: /var/log/ceph/ceph-client.admin.32666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25334.log 2024-09-18T10:30:29.061 INFO:teuthology.orchestra.run.smithi142.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.33412.log.gz 2024-09-18T10:30:29.061 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32666.log.gz 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26401.log.gz 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33385.log 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33655.log 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35080.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27346.log.gz 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27021.log 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25334.log.gz -- replaced with /var/log/ceph/ceph-client.admin.33385.log.gz 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.062 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30875.log 2024-09-18T10:30:29.063 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33655.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-09-18T10:30:29.063 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28163.log 2024-09-18T10:30:29.063 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27021.log: /var/log/ceph/ceph-client.admin.30875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34551.log 2024-09-18T10:30:29.063 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33655.log.gz 2024-09-18T10:30:29.063 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.28163.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26562.log 2024-09-18T10:30:29.064 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30875.log.gz 2024-09-18T10:30:29.064 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27021.log.gz 2024-09-18T10:30:29.064 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35540.log 2024-09-18T10:30:29.064 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28163.log.gz 2024-09-18T10:30:29.064 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26562.log: /var/log/ceph/ceph-client.admin.28940.log 2024-09-18T10:30:29.261 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25379.log 2024-09-18T10:30:29.261 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23943.log 2024-09-18T10:30:29.261 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29556.log 2024-09-18T10:30:29.261 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26987.log 2024-09-18T10:30:29.261 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23943.log: /var/log/ceph/ceph-client.admin.29556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28688.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25379.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.26987.log: -- /var/log/ceph/ceph-client.admin.30579.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29556.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23943.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26009.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28688.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24970.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.26987.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30579.log: /var/log/ceph/ceph-client.admin.26009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31686.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28228.log 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24970.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26009.log.gz 2024-09-18T10:30:29.262 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30579.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30954.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31686.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.24970.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.25934.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28228.log: /var/log/ceph/ceph-client.admin.30954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29318.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24229.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26661.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28228.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30954.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29318.log: /var/log/ceph/ceph-client.admin.24229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29756.log 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.25934.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.foo.31686.log.gz 2024-09-18T10:30:29.263 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28080.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24229.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29975.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27782.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26661.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29756.log: /var/log/ceph/ceph-client.admin.29975.log: gzip -5 90.4%--verbose -- /var/log/ceph/ceph-client.admin.26161.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29318.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28080.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21021.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29975.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29756.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25559.log 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27782.log.gz 2024-09-18T10:30:29.264 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30729.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.21021.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26161.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28562.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25559.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.21021.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30404.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31121.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28562.log: /var/log/ceph/ceph-client.admin.30404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30604.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.25559.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30729.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30454.log 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30404.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28562.log.gz 2024-09-18T10:30:29.265 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31121.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27266.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25859.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30604.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25456.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30454.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31542.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27266.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24015.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25456.log: /var/log/ceph/ceph-client.admin.31542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24920.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25859.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29925.log 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24015.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31542.log.gz 2024-09-18T10:30:29.266 INFO:teuthology.orchestra.run.smithi150.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.25456.log.gz/var/log/ceph/ceph-client.admin.24920.log: 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.31926.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.24015.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31876.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29925.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.31926.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.24920.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28358.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31876.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29925.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.bar.31926.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25663.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31876.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26134.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28358.log: /var/log/ceph/ceph-client.admin.25663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25147.log 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28358.log.gz 2024-09-18T10:30:29.267 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28306.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25663.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28176.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26134.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29143.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25147.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26242.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28306.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28176.log: /var/log/ceph/ceph-client.admin.29143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31355.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28176.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29243.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29143.log.gz 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31280.log 2024-09-18T10:30:29.268 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31355.log: 87.6%/var/log/ceph/ceph-client.admin.29243.log: -- replaced with /var/log/ceph/ceph-client.admin.26242.log.gz 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25303.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31355.log.gz 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21080.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31280.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29243.log.gz 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.25303.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26611.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27580.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31280.log.gz 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.21080.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25303.log.gz 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24391.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:gzip 88.8%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.21080.log.gz-- 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30125.log 2024-09-18T10:30:29.269 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27580.log: /var/log/ceph/ceph-client.admin.24391.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26611.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23918.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27580.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29850.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30125.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.24391.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25227.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23918.log: /var/log/ceph/ceph-client.admin.29850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31255.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30125.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26687.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.23918.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29850.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28003.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29354.log 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25227.log.gz 2024-09-18T10:30:29.270 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31255.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30629.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26687.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27214.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29354.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28003.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26839.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29581.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27214.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29354.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25884.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30629.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26839.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27214.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26413.log 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29581.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26839.log.gz 2024-09-18T10:30:29.271 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30250.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25884.log: /var/log/ceph/ceph-client.admin.26413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29012.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29581.log.gz 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.31972.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25884.log.gz 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30250.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26413.log.gz 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29706.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30250.log.gz 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24086.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.31972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20995.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29706.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29012.log.gz 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26342.log 2024-09-18T10:30:29.272 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.bar.31972.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29706.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.20995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27291.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.24086.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27425.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20995.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26342.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25584.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31567.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27425.log: /var/log/ceph/ceph-client.admin.25584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29456.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27291.log.gz 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29040.log 2024-09-18T10:30:29.273 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27425.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25584.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27400.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29456.log: 88.9% -- replaced with /var/log/ceph/ceph-client.foo.31567.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31405.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31096.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27400.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29456.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31405.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.29040.log.gz-5 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:--verbose -- /var/log/ceph/ceph-client.admin.26517.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31517.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31096.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27400.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31405.log.gz 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31230.log 2024-09-18T10:30:29.274 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27087.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31096.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.26517.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24738.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31230.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31517.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26109.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27087.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31230.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26588.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24895.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27087.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30150.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24738.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26109.log.gz 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28663.log 2024-09-18T10:30:29.275 INFO:teuthology.orchestra.run.smithi150.stderr:gzip 88.2%-5 -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:--verbose -- /var/log/ceph/ceph-client.admin.26789.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24895.log: /var/log/ceph/ceph-client.admin.28663.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26588.log.gz 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28459.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 88.8%--verbose -- replaced with /var/log/ceph/ceph-client.admin.24895.log.gz-- 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28434.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26789.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28663.log.gz 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29093.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26789.log.gz 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30829.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29093.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28459.log.gz 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24616.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29531.log 2024-09-18T10:30:29.276 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29093.log.gz 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28434.log.gz 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27350.log /var/log/ceph/ceph-client.admin.24616.log: 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28512.log 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29531.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30829.log.gz 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27350.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24616.log.gz 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29531.log.gz 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.27350.log.gz-5 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25404.log /var/log/ceph/ceph-client.admin.28512.log: 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31592.log 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.25404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25354.log 2024-09-18T10:30:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28512.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27450.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31592.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25404.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24422.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.foo.31592.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30379.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25509.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25354.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30379.log: /var/log/ceph/ceph-client.admin.25738.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27450.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31205.log 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24422.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30379.log.gz 2024-09-18T10:30:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29631.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25738.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25509.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27375.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25200.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25738.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29631.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31205.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27475.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27375.log: /var/log/ceph/ceph-client.admin.25200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27832.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29631.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27375.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27475.log: 88.6%/var/log/ceph/ceph-client.admin.27832.log: -- replaced with /var/log/ceph/ceph-client.admin.25200.log.gz 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25095.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T10:30:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29875.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27832.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27475.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25020.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5%/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25984.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25095.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24279.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24204.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25534.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25020.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24279.log: /var/log/ceph/ceph-client.admin.24204.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25984.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24279.log.gz 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24663.log 2024-09-18T10:30:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24204.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31146.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32223.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25534.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31146.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24663.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32223.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31146.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26440.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28984.log 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30929.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32223.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26440.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-09-18T10:30:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26440.log.gz 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27930.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25909.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30779.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28984.log.gz 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25909.log: gzip 87.7%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.27930.log.gz-- 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29405.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29068.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30779.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25909.log.gz 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29405.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30779.log.gz 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29405.log.gz 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27139.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.29068.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28255.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26388.log 2024-09-18T10:30:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27139.log: 90.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29068.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31071.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27139.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31437.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28255.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30479.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26388.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31437.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31071.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31437.log.gz 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26937.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28850.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30479.log: /var/log/ceph/ceph-client.admin.26937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31380.log 2024-09-18T10:30:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25174.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30479.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26084.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31380.log: -- replaced with /var/log/ceph/ceph-client.admin.26937.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29900.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25174.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28850.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28537.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31380.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26084.log: /var/log/ceph/ceph-client.admin.29900.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25174.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26084.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28638.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29900.log.gz 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24526.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28931.log 2024-09-18T10:30:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28638.log: gzip /var/log/ceph/ceph-client.admin.24526.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24134.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28537.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28931.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24526.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28931.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27012.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28333.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27554.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27012.log: 88.5%/var/log/ceph/ceph-client.admin.28333.log: -- replaced with /var/log/ceph/ceph-client.admin.24134.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29781.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27012.log.gz 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30025.log 2024-09-18T10:30:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27554.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28333.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30704.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29781.log: /var/log/ceph/ceph-client.admin.30025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23992.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27554.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30704.log: 90.2% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30025.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29781.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.23992.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.23842.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25045.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29430.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30704.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23992.log.gz 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25688.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25252.log 2024-09-18T10:30:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25045.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.23842.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28203.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30225.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25252.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.25045.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25688.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28203.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25252.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29656.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28203.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24333.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30301.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30225.log.gz 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32133.log 2024-09-18T10:30:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24333.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29656.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27323.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.24333.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32133.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30301.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29118.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27323.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32133.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.29379.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27707.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30979.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29118.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27323.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27707.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26565.log 2024-09-18T10:30:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29118.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25638.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30979.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27707.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27905.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27500.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30979.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25638.log: /var/log/ceph/ceph-client.admin.27905.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.26565.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24572.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25638.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27905.log.gz 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30175.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25070.log 2024-09-18T10:30:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27953.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27500.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30175.log: /var/log/ceph/ceph-client.admin.25070.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24572.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26215.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30854.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30175.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27953.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25070.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26215.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27953.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26215.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27062.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24820.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30854.log: /var/log/ceph/ceph-client.admin.27062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25713.log 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30854.log.gz 2024-09-18T10:30:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24820.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.27062.log.gz-5 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:--verbose -- /var/log/ceph/ceph-client.admin.28744.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.25713.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28822.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24820.log.gz 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28281.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25761.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25713.log.gz 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29218.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24503.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28822.log.gz 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25761.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28281.log.gz 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24038.log 2024-09-18T10:30:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31663.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1%/var/log/ceph/ceph-client.admin.24503.log: -- replaced with /var/log/ceph/ceph-client.admin.25761.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28878.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29218.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24503.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31663.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24038.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.foo.31663.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27978.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26636.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23867.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30050.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28878.log.gz/var/log/ceph/ceph-client.admin.26636.log: 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27630.log 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27978.log.gz 2024-09-18T10:30:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29268.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26636.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30050.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23867.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26365.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29268.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30050.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27630.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.32060.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29268.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28587.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24594.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.32060.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.26365.log.gz 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.32158.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29193.log 2024-09-18T10:30:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24594.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28587.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.32158.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24594.log.gz 92.2% 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.bar.32060.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 83.3% -- replaced with /var/log/ceph/ceph-client.bar.32158.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24182.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 /var/log/ceph/ceph-client.admin.29193.log: --verbose -- /var/log/ceph/ceph-client.admin.23969.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24157.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24182.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29193.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27880.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23969.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28055.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24182.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24157.log: /var/log/ceph/ceph-client.admin.27880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31046.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23969.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24845.log 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28055.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24157.log.gz 2024-09-18T10:30:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31046.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27880.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28055.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29168.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27732.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31046.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30354.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24845.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27732.log: /var/log/ceph/ceph-client.admin.30354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25809.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30654.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27732.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23892.log 2024-09-18T10:30:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25809.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26265.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26188.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.23892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24995.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24638.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26188.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23892.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26265.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31617.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26188.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26814.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31617.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24995.log.gz 2024-09-18T10:30:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26814.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24638.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26059.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.foo.31617.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30804.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26814.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28128.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27757.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32172.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28128.log: /var/log/ceph/ceph-client.admin.27757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31640.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26059.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28128.log.gz 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26713.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.foo.31864.log 2024-09-18T10:30:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1%/var/log/ceph/ceph-client.foo.31640.log: -- replaced with /var/log/ceph/ceph-client.admin.27757.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25121.log 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32172.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.foo.31640.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.foo.31864.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26713.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.foo.31864.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26315.log 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30200.log 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25121.log: /var/log/ceph/ceph-client.admin.26315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27189.log 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25121.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30754.log 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30200.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26315.log.gz 2024-09-18T10:30:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29481.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27189.log: /var/log/ceph/ceph-client.admin.30754.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30200.log.gz 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24789.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26290.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27189.log.gz 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29481.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30754.log.gz 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27657.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.26290.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28797.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29481.log.gz 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24789.log.gz 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29950.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.32037.log 2024-09-18T10:30:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26290.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bazar.31462.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27657.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29950.log: /var/log/ceph/ceph-client.bar.32037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26034.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28797.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28029.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29950.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bazar.31462.log: 88.3% -- replaced with /var/log/ceph/ceph-client.bar.32037.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28409.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28029.log: /var/log/ceph/ceph-client.admin.30904.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26034.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.bazar.31462.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28029.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24061.log 2024-09-18T10:30:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28409.log: /var/log/ceph/ceph-client.admin.32197.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32274.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32197.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28409.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27682.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24061.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26887.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32197.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24688.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32274.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.31949.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:gzip /var/log/ceph/ceph-client.admin.24688.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30679.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27682.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26887.log.gz 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.31949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28484.log 2024-09-18T10:30:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24688.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.bar.31949.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30075.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20938.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: 89.1%/var/log/ceph/ceph-client.admin.30075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26764.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30679.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28484.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31839.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.20938.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30075.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26764.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.20938.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26764.log.gz 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24449.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26739.log 2024-09-18T10:30:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31839.log: /var/log/ceph/ceph-client.admin.24449.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31839.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26465.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.24449.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26490.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26739.log: gzip /var/log/ceph/ceph-client.admin.26465.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27164.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26739.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24304.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26465.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27164.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26490.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27603.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27164.log.gz 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28716.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31305.log 2024-09-18T10:30:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27603.log: /var/log/ceph/ceph-client.admin.28716.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24304.log.gz 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25834.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27603.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28716.log.gz 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31305.log: 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25430.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31172.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25834.log: /var/log/ceph/ceph-client.admin.25430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28906.log 88.9% 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31305.log.gz 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31172.log: 90.1% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25430.log.gz -- replaced with /var/log/ceph/ceph-client.admin.25834.log.gz 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31172.log.gz 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26912.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24254.log 2024-09-18T10:30:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.4.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28906.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26912.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.26912.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.bar.31901.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25786.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.bar.31901.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24254.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26540.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25786.log: 89.0% -- replaced with /var/log/ceph/ceph-client.bar.31901.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25786.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25959.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32249.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26540.log: /var/log/ceph/ceph-client.admin.25959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30329.log 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.26540.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.32249.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25959.log.gz 2024-09-18T10:30:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27037.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30429.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30329.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32249.log.gz 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28105.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30329.log.gz 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30429.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27037.log.gz 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26862.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30429.log.gz 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28105.log: /var/log/ceph/ceph-client.admin.24364.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.26862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24480.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29293.log 2024-09-18T10:30:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28105.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24364.log: /var/log/ceph/ceph-client.admin.24480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24364.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.26862.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24480.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25278.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29293.log: /var/log/ceph/ceph-client.admin.25278.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29293.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28613.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25278.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25611.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30554.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30504.log 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25611.log: /var/log/ceph/ceph-client.admin.30554.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28613.log.gz 2024-09-18T10:30:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25611.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30275.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30554.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26962.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28772.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30275.log: /var/log/ceph/ceph-client.admin.26962.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30504.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30275.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24870.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26962.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28772.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28772.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28956.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25482.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24870.log: /var/log/ceph/ceph-client.admin.28956.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24870.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24109.log 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28956.log.gz 2024-09-18T10:30:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29606.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30100.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.25482.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27807.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29606.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.24109.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30100.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29606.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30000.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30100.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27807.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27807.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24713.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24549.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30000.log: /var/log/ceph/ceph-client.admin.24713.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30000.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24713.log.gz 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28151.log 2024-09-18T10:30:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24763.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29681.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28151.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.24549.log.gz 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27855.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28151.log.gz/var/log/ceph/ceph-client.admin.29681.log: 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24763.log.gz 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29681.log.gz 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29506.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27239.log 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29731.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27855.log.gz 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29506.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-09-18T10:30:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29506.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28384.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27239.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27239.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21052.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25329.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30879.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.21052.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28384.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.25329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27114.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.21052.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.30879.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25329.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24945.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30879.log.gz 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.27114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20968.log 2024-09-18T10:30:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.24945.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27114.log.gz 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.20968.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24945.log.gz 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.20968.log.gz 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30529.log 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.31330.log: /var/log/ceph/ceph-client.admin.30529.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz 2024-09-18T10:30:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30529.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35540.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34551.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29346.log 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35540.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34902.log 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.26562.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28940.log: /var/log/ceph/ceph-client.admin.29346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31728.log 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25762.log 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34902.log: /var/log/ceph/ceph-client.admin.31728.log: gzip -5 --verbose -- 90.9%/var/log/ceph/ceph-client.admin.23700.log 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28940.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29346.log.gz 2024-09-18T10:30:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30280.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6%/var/log/ceph/ceph-client.admin.25762.log: -- replaced with /var/log/ceph/ceph-client.admin.31728.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34902.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17242.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23700.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25762.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32053.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30280.log: /var/log/ceph/ceph.tmp-client.admin.17242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32693.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17242.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23700.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35703.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32053.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30280.log.gz 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 /var/log/ceph/ceph-client.admin.32693.log: --verbose -- /var/log/ceph/ceph-client.admin.32937.log 2024-09-18T10:30:29.331 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34659.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32053.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32693.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24903.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32937.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35703.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26131.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24903.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32937.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34659.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31307.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24903.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26131.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24795.log 2024-09-18T10:30:29.332 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31512.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30983.log: /var/log/ceph/ceph-client.admin.24795.log: gzip -5 89.6%--verbose -- /var/log/ceph/ceph-client.admin.24930.log -- replaced with /var/log/ceph/ceph-client.admin.26131.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.31307.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.31512.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25172.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.24795.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26832.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24930.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31512.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29727.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25172.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24930.log.gz 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32215.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28832.log 2024-09-18T10:30:29.333 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.25172.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26832.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29183.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29727.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24633.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28832.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.32215.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29129.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29183.log: /var/log/ceph/ceph-client.admin.24633.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28832.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34076.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30226.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24633.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29129.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24849.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34076.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29183.log.gz 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31280.log 2024-09-18T10:30:29.334 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30226.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34076.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24280.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24849.log: gzip -5 /var/log/ceph/ceph-client.admin.31280.log: --verbose -- /var/log/ceph/ceph-client.admin.23692.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30226.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.24849.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28994.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24280.log: 89.8%/var/log/ceph/ceph-client.admin.23692.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.31280.log.gz-5 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.34442.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.24280.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35324.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35188.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.23692.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33180.log 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35324.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28994.log.gz 2024-09-18T10:30:29.335 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28859.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35188.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34442.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35324.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26239.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33180.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35188.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29265.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28859.log: /var/log/ceph/ceph-client.admin.26239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27238.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.33180.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28859.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.26239.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.27238.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34524.log 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29265.log.gz 2024-09-18T10:30:29.336 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25442.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25870.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34524.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27238.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33709.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25870.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.34524.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34376.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.25442.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24523.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.25870.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24334.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33709.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33913.log 2024-09-18T10:30:29.337 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24523.log: /var/log/ceph/ceph-client.admin.24334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23868.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33709.log.gz 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35161.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24523.log.gz 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24334.log.gz 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31064.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33913.log.gz 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.23868.log.gz 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35215.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31064.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35161.log.gz-5 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.27400.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24496.log 2024-09-18T10:30:29.338 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35215.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31064.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25995.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35486.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24496.log: gzip 89.8%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.27400.log.gz-- 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33493.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25995.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.24496.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34686.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32134.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25995.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33493.log: /var/log/ceph/ceph-client.admin.34686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29156.log 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35486.log.gz 2024-09-18T10:30:29.339 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30145.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32134.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34686.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33493.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32134.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31891.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30145.log: /var/log/ceph/ceph-client.admin.31809.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29156.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29821.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30145.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31891.log: /var/log/ceph/ceph-client.admin.29821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33072.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27292.log 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31891.log.gz 2024-09-18T10:30:29.340 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29821.log.gz 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31647.log 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33072.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33736.log 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27292.log: /var/log/ceph/ceph-client.admin.31647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33682.log 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33072.log.gz 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27292.log.gz 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30037.log 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33736.log: gzip -5 /var/log/ceph/ceph-client.admin.33682.log: --verbose -- /var/log/ceph/ceph-client.admin.24199.log 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31647.log.gz 2024-09-18T10:30:29.341 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30037.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.33736.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33682.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30037.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33126.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34578.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30172.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.24199.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29237.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33126.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27526.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33330.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29237.log: /var/log/ceph/ceph-client.admin.27526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34103.log 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34578.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30172.log.gz 2024-09-18T10:30:29.342 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28643.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33330.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29237.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25600.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34103.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27526.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28643.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34103.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27811.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25600.log: /var/log/ceph/ceph-client.admin.29643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31037.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28643.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31253.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27811.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.25600.log.gz 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.31037.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24253.log 2024-09-18T10:30:29.343 INFO:teuthology.orchestra.run.smithi142.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27427.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31253.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.27811.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35459.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24253.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31253.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28805.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27427.log: gzip /var/log/ceph/ceph-client.admin.35459.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28001.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24253.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28805.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35459.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.27427.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28805.log.gz 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28589.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27662.log 2024-09-18T10:30:29.344 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28001.log: /var/log/ceph/ceph-client.admin.28589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26751.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28001.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33601.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27662.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28589.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31837.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23866.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33601.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27662.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25517.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31837.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.26751.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25226.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33601.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33264.log 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31837.log.gz 2024-09-18T10:30:29.345 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25627.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.23866.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25226.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25517.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.25226.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.25627.log: -- /var/log/ceph/ceph-client.admin.25928.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose 88.6%-- /var/log/ceph/ceph-client.admin.29875.log: /var/log/ceph/ceph-client.admin.28109.log -- replaced with /var/log/ceph/ceph-client.admin.33264.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25627.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31091.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25928.log: /var/log/ceph/ceph-client.admin.28109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31199.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26670.log 2024-09-18T10:30:29.346 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31091.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.25928.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28109.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26535.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31199.log: /var/log/ceph/ceph-client.admin.26670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26805.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31091.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31199.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27865.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26535.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26670.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25199.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34157.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27865.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26535.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32531.log 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26805.log.gz 2024-09-18T10:30:29.347 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34157.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27865.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32531.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.25199.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31755.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34157.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 /var/log/ceph/ceph-client.admin.34265.log: --verbose -- /var/log/ceph/ceph-client.admin.34022.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32531.log.gz-5 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.31539.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31755.log: gzip -5 --verbose -- 88.4%/var/log/ceph/ceph-client.admin.23605.log /var/log/ceph/ceph-client.admin.34022.log: -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.31539.log: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31755.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34022.log.gz 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34848.log 2024-09-18T10:30:29.348 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23605.log: /var/log/ceph/ceph-mon.c.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31539.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.23605.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34848.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28244.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35432.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23502.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.34848.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35432.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32991.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30091.log 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28244.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23502.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35432.log.gz 2024-09-18T10:30:29.349 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27460.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32991.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.23502.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35107.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28190.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32991.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27460.log: /var/log/ceph/ceph-client.admin.35107.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30091.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30632.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28190.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.27460.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35107.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28190.log.gz 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24469.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27947.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23659.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24469.log: /var/log/ceph/ceph-client.admin.27947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31145.log 2024-09-18T10:30:29.350 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30632.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28454.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23659.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.24469.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28400.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27947.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31145.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.23659.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28454.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31145.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28454.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31674.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29373.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33045.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32802.log 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29373.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28400.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31674.log.gz 2024-09-18T10:30:29.351 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33045.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29373.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33045.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29021.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31593.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26859.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32802.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23865.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31593.log: gzip 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29021.log.gz-5 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.23705.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26859.log: /var/log/ceph/ceph-client.admin.23865.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31593.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30253.log 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.26859.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23865.log.gz 2024-09-18T10:30:29.352 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30523.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34049.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26439.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30523.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.23705.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34049.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30253.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30523.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34049.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29102.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30307.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26439.log: /var/log/ceph/ceph-client.admin.29102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30902.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32080.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26439.log.gz 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20973.log 2024-09-18T10:30:29.353 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29102.log.gz 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30902.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30307.log.gz 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.32080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29075.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 89.8%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.30902.log.gz-- 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30767.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.20973.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32080.log.gz 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25843.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29075.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20973.log.gz 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27608.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30767.log: /var/log/ceph/ceph-client.admin.25843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27554.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.27608.log: 90.6%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.29075.log.gz-- 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27920.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30767.log.gz 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27893.log 2024-09-18T10:30:29.354 INFO:teuthology.orchestra.run.smithi142.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.25843.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28778.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27608.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27920.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27554.log.gz/var/log/ceph/ceph-client.admin.27893.log: 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33466.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28778.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27893.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26293.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32856.log: -- replaced with /var/log/ceph/ceph-client.admin.27920.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25654.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33466.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28778.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30118.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26293.log: /var/log/ceph/ceph-client.admin.25654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27635.log 2024-09-18T10:30:29.355 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33466.log.gz 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32188.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26293.log.gz 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30118.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32477.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25654.log.gz/var/log/ceph/ceph-client.admin.27635.log: 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24550.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.30118.log.gz-5 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.27581.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32477.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27635.log.gz 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32188.log.gz 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24550.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32477.log.gz 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35351.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27716.log 2024-09-18T10:30:29.356 INFO:teuthology.orchestra.run.smithi142.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.24550.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 /var/log/ceph/ceph-client.admin.35351.log: --verbose -- /var/log/ceph/ceph-client.admin.29793.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34130.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.27581.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27716.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35351.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25013.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28670.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27716.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28481.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25013.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29793.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26778.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34130.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28481.log: /var/log/ceph/ceph-client.admin.32774.log 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.25013.log.gz 2024-09-18T10:30:29.357 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28670.log.gz 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31010.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30605.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9%/var/log/ceph/ceph-client.admin.32774.log: -- replaced with /var/log/ceph/ceph-client.admin.28481.log.gz 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.26778.log.gz-5 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31010.log: --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32774.log.gz 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32639.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35567.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34794.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31010.log.gz 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32450.log 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30605.log.gz 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz-5 2024-09-18T10:30:29.358 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.27157.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35567.log: /var/log/ceph/ceph-client.admin.32450.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32639.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27974.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35567.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29427.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31782.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32450.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27974.log: /var/log/ceph/ceph-client.admin.29427.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27157.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24388.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27974.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29902.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31782.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29427.log.gz 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24068.log 2024-09-18T10:30:29.359 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33574.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31782.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29902.log: /var/log/ceph/ceph-client.admin.24068.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24388.log.gz-5 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.31864.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26320.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33574.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29902.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24068.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30442.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32910.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26320.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33574.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30821.log 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30442.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31864.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32910.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26320.log.gz 2024-09-18T10:30:29.360 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34875.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24442.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30442.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32107.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34875.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.32910.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30821.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27770.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34875.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29319.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32107.log: /var/log/ceph/ceph-client.admin.27770.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.24442.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.35053.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24024.log 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32107.log.gz -- replaced with /var/log/ceph/ceph-client.admin.27770.log.gz 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.361 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29319.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28305.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29319.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24024.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30848.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35053.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28305.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.24024.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28305.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23867.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25253.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25145.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23867.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30848.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30956.log 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25253.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23867.log.gz 2024-09-18T10:30:29.362 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26951.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25253.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34346.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30956.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25145.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30713.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28697.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34346.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30956.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23632.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.26951.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28427.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28697.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34346.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30713.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30469.log 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28697.log.gz 2024-09-18T10:30:29.363 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.23632.log.gz/var/log/ceph/ceph-client.admin.28427.log: 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32612.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31945.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28427.log.gz 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31999.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30469.log.gz 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.32612.log: -- /var/log/ceph/ceph-client.admin.34238.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31431.log: /var/log/ceph/ceph-client.admin.34238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24741.log 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32612.log.gz 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 88.7%-5 -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz--verbose 2024-09-18T10:30:29.364 INFO:teuthology.orchestra.run.smithi142.stderr:-- /var/log/ceph/ceph-client.admin.26913.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34238.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24741.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.24741.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26507.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26077.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26913.log: /var/log/ceph/ceph-client.admin.26507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34998.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28724.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26077.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.26913.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26507.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24012.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30496.log 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28724.log: -- replaced with /var/log/ceph/ceph-client.admin.26077.log.gz 2024-09-18T10:30:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 89.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34998.log.gz-- 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35378.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24012.log: /var/log/ceph/ceph-client.admin.30496.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24012.log.gz 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28724.log.gz 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26104.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:gzip 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30496.log.gz-5 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:--verbose -- /var/log/ceph/ceph-client.admin.35405.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24822.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26104.log: /var/log/ceph/ceph-client.admin.35405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28055.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:gzip /var/log/ceph/ceph-client.admin.24822.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34740.log 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.26104.log.gz 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35405.log.gz 2024-09-18T10:30:29.366 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33547.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.24822.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24986.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28055.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33547.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34740.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28886.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33547.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31458.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23703.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28886.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.24986.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32883.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31458.log: /var/log/ceph/ceph-client.admin.23703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28136.log 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28886.log.gz 2024-09-18T10:30:29.367 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31458.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32883.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.23703.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32883.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33940.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24307.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32269.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33940.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28136.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25051.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24307.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33940.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26724.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25897.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9%/var/log/ceph/ceph-client.admin.25051.log: -- replaced with /var/log/ceph/ceph-client.admin.24307.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32269.log.gz 2024-09-18T10:30:29.368 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7%/var/log/ceph/ceph-client.admin.25897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26886.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25051.log.gz 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.26724.log.gz 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34605.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26185.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.25897.log.gz 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26886.log: /var/log/ceph/ceph-client.admin.34605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35296.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29048.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26185.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.26886.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24415.log 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34605.log.gz 2024-09-18T10:30:29.369 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35296.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26185.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35296.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26050.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29048.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35676.log: /var/log/ceph/ceph-client.admin.25361.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25789.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.24415.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26050.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35676.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23863.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33018.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26050.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25789.log: /var/log/ceph/ceph-client.admin.23863.log: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.25361.log.gz 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.25415.log 2024-09-18T10:30:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33018.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25789.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.23863.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33153.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23528.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25415.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33018.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24361.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.25415.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23702.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23528.log: /var/log/ceph/ceph-client.admin.24361.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33153.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30199.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23528.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23702.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.24361.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.23702.log.gz 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24074.log 2024-09-18T10:30:29.371 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35269.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24604.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24074.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30199.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27130.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35269.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24604.log: -- replaced with /var/log/ceph/ceph-client.admin.24074.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29481.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35269.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27130.log: /var/log/ceph/ceph-client.admin.35648.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24604.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21075.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27130.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31620.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26616.log 2024-09-18T10:30:29.372 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.21075.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29481.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25708.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24172.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.21075.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26616.log: /var/log/ceph/ceph-client.admin.25708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33358.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31620.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25681.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.26616.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25117.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33358.log: -- replaced with /var/log/ceph/ceph-client.admin.25708.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24172.log.gz 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33439.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31972.log 2024-09-18T10:30:29.373 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25117.log: 89.5% 92.8% -- replaced with /var/log/ceph/ceph-client.admin.33358.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25681.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33439.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25117.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35242.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30794.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33439.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31972.log: /var/log/ceph/ceph-client.admin.35242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29616.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29848.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30794.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35242.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31972.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30550.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30794.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29562.log 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8%/var/log/ceph/ceph-client.admin.29848.log: -- replaced with /var/log/ceph/ceph-client.admin.29616.log.gz 2024-09-18T10:30:29.374 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30550.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29848.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30550.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33234.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32396.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26697.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33234.log: /var/log/ceph/ceph-client.admin.32396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34713.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29562.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26697.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33234.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34713.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32396.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.26697.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34713.log.gz 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-18T10:30:29.375 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30577.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34767.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30686.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32720.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30577.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28028.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31118.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32720.log: -- replaced with /var/log/ceph/ceph-client.admin.30577.log.gz 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30686.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27319.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28028.log: /var/log/ceph/ceph-client.admin.31118.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.32720.log.gz 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28028.log.gz 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29454.log 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31118.log.gz 2024-09-18T10:30:29.376 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29454.log: /var/log/ceph/ceph-client.admin.28913.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.0.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27319.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29454.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24714.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27211.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28913.log: /var/log/ceph/ceph-client.admin.24714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27049.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27211.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28913.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.24714.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27184.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27211.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27049.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27049.log.gz 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31485.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32299.log 2024-09-18T10:30:29.377 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27184.log: /var/log/ceph/ceph-client.admin.31485.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27184.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26023.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31485.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32299.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32299.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35728.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.26023.log: -- /var/log/ceph/ceph-client.admin.28372.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35728.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.26023.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35728.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33820.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.28372.log: -- /var/log/ceph/ceph-client.admin.31226.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33820.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28372.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29400.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33820.log.gz 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31172.log 2024-09-18T10:30:29.378 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29400.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31226.log.gz 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27265.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29400.log.gz 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31172.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31172.log.gz 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27103.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33099.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27265.log: /var/log/ceph/ceph-client.admin.27103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27265.log.gz/var/log/ceph/ceph-client.admin.33099.log: 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24660.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27103.log.gz 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33099.log.gz 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33628.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24876.log 2024-09-18T10:30:29.379 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34821.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24876.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33628.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.24660.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33763.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.24876.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34821.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34821.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27076.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25735.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35026.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27076.log: 89.9%/var/log/ceph/ceph-client.admin.25735.log: -- replaced with /var/log/ceph/ceph-client.admin.33763.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27076.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24226.log 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.25735.log.gz 2024-09-18T10:30:29.380 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35026.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35026.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29956.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35621.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24226.log: /var/log/ceph/ceph-client.admin.29956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34932.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24226.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35621.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29956.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24687.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35621.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30740.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24687.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34932.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25307.log 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24687.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T10:30:29.381 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30740.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30740.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34319.log 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23706.log 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25307.log: /var/log/ceph/ceph-client.admin.34319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33207.log 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30388.log 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23706.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.25307.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9%/var/log/ceph/ceph-client.admin.33207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28217.log 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34319.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23706.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24957.log /var/log/ceph/ceph-client.admin.30388.log: 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33207.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28217.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30388.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28217.log.gz 2024-09-18T10:30:29.382 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31334.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33520.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25280.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31334.log: /var/log/ceph/ceph-client.admin.33520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30010.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr: 89.5% 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24957.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31334.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25280.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33968.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33520.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30010.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25280.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30010.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31361.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32829.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.31361.log: /var/log/ceph/ceph-client.admin.32829.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33968.log.gz 2024-09-18T10:30:29.383 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26374.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31361.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8%/var/log/ceph/ceph-client.admin.35594.log: -- replaced with /var/log/ceph/ceph-client.admin.32829.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28751.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32747.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35513.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28751.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.26374.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28271.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28751.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.35513.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32747.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35513.log.gz 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20853.log 2024-09-18T10:30:29.384 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32585.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28271.log: /var/log/ceph/ceph-client.admin.20853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28082.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32585.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28271.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32964.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.20853.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4%/var/log/ceph/ceph-client.admin.28082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35134.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32585.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24048.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32964.log: 89.5%/var/log/ceph/ceph-client.admin.35134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28082.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24048.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32964.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35134.log.gz 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32558.log 2024-09-18T10:30:29.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- 88.9%/var/log/ceph/ceph-client.admin.30361.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24048.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34496.log: /var/log/ceph/ceph-client.admin.32558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23864.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27373.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30361.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32558.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.23864.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23864.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25388.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28616.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.27373.log: /var/log/ceph/ceph-client.admin.25388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26212.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25816.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28616.log: 90.7%/var/log/ceph/ceph-client.admin.26212.log: -- replaced with /var/log/ceph/ceph-client.admin.27373.log.gz 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28616.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29697.log 2024-09-18T10:30:29.386 INFO:teuthology.orchestra.run.smithi142.stderr: 90.9% 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25388.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28508.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.26212.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.25816.log: /var/log/ceph/ceph-client.admin.29697.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25816.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24768.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29697.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29929.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30334.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32026.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29929.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28508.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29535.log 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24768.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.30334.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29929.log.gz 2024-09-18T10:30:29.387 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32026.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30334.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32026.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26158.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27689.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26589.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26158.log: gzip /var/log/ceph/ceph-client.admin.27689.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29670.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29535.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26158.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26589.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27689.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.26589.log.gz 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29508.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32161.log 2024-09-18T10:30:29.388 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29670.log: /var/log/ceph/ceph-client.admin.29508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33995.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32161.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29670.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29508.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24014.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32161.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24577.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.33995.log: /var/log/ceph/ceph-client.admin.24014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26643.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.24577.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33995.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28967.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26643.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.24014.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-osd.2.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.24577.log.gz 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34211.log 2024-09-18T10:30:29.389 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26643.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.28967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26347.log 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32365.log 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.34211.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28967.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.26347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29292.log 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34211.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.32365.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.26347.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32365.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29210.log 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33886.log 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/ceph-client.admin.29210.log: 88.7%/var/log/ceph/ceph-client.admin.33886.log: -- replaced with /var/log/ceph/ceph-client.admin.29292.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29210.log.gz 2024-09-18T10:30:29.390 INFO:teuthology.orchestra.run.smithi142.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33886.log.gz 2024-09-18T10:30:29.391 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T10:30:29.520 INFO:teuthology.orchestra.run.smithi142.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T10:30:29.635 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T10:30:30.045 INFO:teuthology.orchestra.run.smithi150.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-18T10:30:30.050 INFO:teuthology.orchestra.run.smithi142.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T10:30:30.119 INFO:teuthology.orchestra.run.smithi150.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-18T10:30:30.193 INFO:teuthology.orchestra.run.smithi142.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T10:30:30.201 INFO:teuthology.orchestra.run.smithi142.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T10:30:30.201 INFO:teuthology.orchestra.run.smithi142.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T10:30:30.202 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-09-18T10:30:30.202 INFO:teuthology.orchestra.run.smithi142.stderr:real 0m1.189s 2024-09-18T10:30:30.203 INFO:teuthology.orchestra.run.smithi142.stderr:user 0m4.887s 2024-09-18T10:30:30.203 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m0.239s 2024-09-18T10:30:30.205 INFO:teuthology.orchestra.run.smithi150.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T10:30:30.207 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-18T10:30:30.207 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m1.174s 2024-09-18T10:30:30.208 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m3.958s 2024-09-18T10:30:30.208 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.241s 2024-09-18T10:30:30.208 INFO:tasks.ceph:Archiving logs... 2024-09-18T10:30:30.208 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi142/log 2024-09-18T10:30:30.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T10:30:31.327 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi150/log 2024-09-18T10:30:31.328 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T10:30:32.183 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T10:30:32.194 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T10:30:32.194 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T10:30:32.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T10:30:32.261 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T10:30:32.272 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T10:30:32.272 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T10:30:32.275 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T10:30:32.296 INFO:teuthology.orchestra.run.smithi142.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T10:30:32.296 INFO:teuthology.orchestra.run.smithi142.stdout:============================================================================== 2024-09-18T10:30:32.297 INFO:teuthology.orchestra.run.smithi142.stdout:*hv01.front.sepi 67.205.162.81 3 u 1 64 377 0.124 2.066 0.243 2024-09-18T10:30:32.297 INFO:teuthology.orchestra.run.smithi142.stdout:+hv02.front.sepi 63.231.80.2 3 u 14 64 377 0.090 0.523 0.201 2024-09-18T10:30:32.297 INFO:teuthology.orchestra.run.smithi142.stdout:+hv03.front.sepi 74.6.168.72 3 u 64 64 377 0.089 3.016 0.208 2024-09-18T10:30:32.297 INFO:teuthology.orchestra.run.smithi142.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-18T10:30:32.298 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T10:30:32.298 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-09-18T10:30:32.298 INFO:teuthology.orchestra.run.smithi150.stdout:*hv01.front.sepi 67.205.162.81 3 u 3 128 377 0.059 1.377 0.106 2024-09-18T10:30:32.299 INFO:teuthology.orchestra.run.smithi150.stdout:+hv02.front.sepi 63.231.80.2 3 u 78 128 377 0.092 -0.213 0.172 2024-09-18T10:30:32.299 INFO:teuthology.orchestra.run.smithi150.stdout:+hv03.front.sepi 74.6.168.72 3 u 129 128 377 0.101 2.509 0.203 2024-09-18T10:30:32.299 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-18T10:30:32.299 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T10:30:32.308 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T10:30:32.309 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T10:30:32.350 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T10:30:32.383 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T10:30:32.416 INFO:teuthology.task.internal:Duration was 1792.816306 seconds 2024-09-18T10:30:32.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T10:30:32.450 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T10:30:32.450 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T10:30:32.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T10:30:32.506 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T10:30:32.506 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-09-18T10:30:32.506 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T10:30:32.518 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-09-18T10:30:32.518 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T10:30:32.530 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T10:30:32.530 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T10:30:32.563 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T10:30:32.601 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T10:30:32.602 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T10:30:32.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T10:30:32.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T10:30:32.726 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T10:30:32.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T10:30:32.742 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T10:30:32.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T10:30:32.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:32.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:32.799 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-09-18T10:30:32.816 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-09-18T10:30:32.828 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:30:32.833 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:30:32.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T10:30:32.852 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T10:30:32.853 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi142 2024-09-18T10:30:32.853 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T10:30:32.878 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910840/remote/smithi150 2024-09-18T10:30:32.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T10:30:32.902 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T10:30:32.902 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T10:30:32.919 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T10:30:32.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T10:30:32.957 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T10:30:32.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T10:30:32.986 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T10:30:32.986 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T10:30:32.988 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T10:30:32.994 INFO:teuthology.orchestra.run.smithi142.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 10:30 /home/ubuntu/cephtest 2024-09-18T10:30:32.996 INFO:teuthology.orchestra.run.smithi150.stdout: 924110 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 10:30 /home/ubuntu/cephtest 2024-09-18T10:30:32.998 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T10:30:33.019 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T10:30:33.090 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} duration: 1792.8163058757782 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T10:30:33.090 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T10:30:33.170 INFO:teuthology.run:pass